Can't deploy diskEncryptionSets

Bicep version
0.4.1272
Describe the bug
It's impossible to deploy a VM with a diskEncryptionSet and create the encryptionSet itself in the process. The diskEncryptionSet needs a reference to a key in a Key Vault, therefore there needs to be an entry in the Key Vault access policy that grants the managed identity of the diskEncryptionSet access to the vault. This policy on the other hand can only be created, when the id of the diskEncryptionSet's managed identity is known, which is after it is created. So it's impossible to create the diskEncryptionSet and it's not possible to tell bicep to ignore the one step that failed and continue deploying the template which would already help.
To Reproduce
Use the following resources to try to create a diskEncryptionSet:
resource diskEncryptionSet 'Microsoft.Compute/diskEncryptionSets@2021-08-01' = {
  name: diskEncryptionSetName
  location: location
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    activeKey: {
      sourceVault: {
        id: keyVault.id
      }
      keyUrl: keyVaultKey.properties.keyUriWithVersion
    }
    encryptionType: 'EncryptionAtRestWithCustomerKey'
    rotationToLatestKeyVersionEnabled: true
  }
}

resource keyVault 'Microsoft.KeyVault/vaults@2021-11-01-preview' = {
  name: keyVaultName
  location: location
  properties: {
    sku: {
      family: 'A'
      name: 'standard'
    }
    tenantId: tenant().tenantId
    accessPolicies: []
    enabledForDeployment: true
    enabledForDiskEncryption: true
    enabledForTemplateDeployment: true
  }
}

resource keyVaultKey 'Microsoft.KeyVault/vaults/keys@2021-11-01-preview' = {
  parent: keyVault
  name: keyVaultKeyName
  properties: {
    attributes: {
      enabled: true
    }
    kty: 'RSA'
    keySize: 2048
  }
}

resource keyVaultAccessPolicy 'Microsoft.KeyVault/vaults/accessPolicies@2021-11-01-preview' = {
  name: 'replace'
  parent: keyVault
  properties: {
    accessPolicies: [
      {
        objectId: diskEncryptionSet.identity.principalId
        tenantId: tenant().tenantId
        permissions: {
          keys: [
            'get'
            'wrapKey'
            'unwrapKey'
          ]
          certificates: []
          secrets: []
        }
      }
    ]
  }
}

This is widely known limitation with System Managed Identities. It is present in other resources as well. Basically chicken and egg problem. The only way of overcoming that is if the resource supports user assigned managed identity. In that case you can create the identity, assign it Key Vault permission and add to the resource that will use it. So if a resource does not support system assigned managed identity ask the team responsible for the service to support that scenario.

To use a user assigned managed identity was my first idea as well, but it's unfortunately not supported with diskEncryptionSets.
The product team definitely has work to do, but the Bicep team can also help with this. If there would be a flag to continue on error, the problem would already be mitigated, as you could deploy the diskEncryptionSet first, ignore the error, deploy the accessPolicy and then the vm.
I think this is a fundamental flaw in the whole ARM system, as it should be possible to deploy multi-step and multi deployment templates (like template spec) and control the error behavior of each one individual.

@itpropro The problem is that this is not a Bicep issue or ARM issue. If there is something like continue on error for this it needs to be done by the team who owns the RP. Which means they better offer user assigned identity support instead of creating workaround.

@slavizh for me it's clearly an ARM issue, as there is no guidance or standard for RPs to implement these types of situations and user managed identities were implemented after the problem still existed for months.
Also, implementing some kind of "continue on error" functionality, would be the best solution currently, as it would help all people having this problem now with a workaround instead of waiting month for a fix. From a customer perspective, waiting for each RP to implement user managed identities is definitely not the way to go.

I am a bit confused on this one. The accessPolicy should be dependent on the diskEncryptionSet. Why isn't the system-assigned identity available at that point? Shouldn't the diskEncryptionSet not report completed until that is done @bmoore-msft / @slavizh
/ @brwilkinson?  And @brwilkinson, does using Azure RBAC instead of Access Policies help with this at all?
I've heard of similar cases where the resource that is creating the system-assigned identity actually needs a reference to its own identity, and the same resource needs to be deployed a second time, but I don't think this is the same case. FWIW, in that case, the workaround is to redeploy the resource in a module.
@itpropro -- "continue on error" is not something we would entertain. This would have negative implications for idempotency and introduce opportunities to make the code less reliable. RPs need to be pushed to ensure their APIs work well using declarative tools.

@itpropro - what error are you getting?  the typical chicken/egg problem around CMK encryption doesn't affect DES deployment the same way it does say, storageAccounts.  The DES can be created and configured with a key but access to the vault is only needed when you want to encrypt something.  So a workflow like this should work:

create vault
create key
create des with keyuri
assign role to give des access to vault
encrypt...

For step 4 - azure RBAC will work better than access policies if your keyvault lifecycle is shared (i.e. multiple deployments of the vault from different templates).

The following works fine for me.
I don't use Access Policies anymore, I have moved to RBAC for KV permissions, however I am not sure that would be required, it's just a recommendation and my own preference
https://docs.microsoft.com/en-us/azure/key-vault/general/rbac-guide?tabs=azurepowershell
I don't believe the policy is being used when it is originally created, so It doesn't appear to be any chicken or egg here?
resource KV 'Microsoft.KeyVault/vaults@2021-06-01-preview' existing = {
  name: 'ACU1-BRW-AOA-T5-kvApp01'
}

resource key 'Microsoft.KeyVault/vaults/keys@2021-11-01-preview' = {
  name: 'mykey1'
  parent: KV
  properties: {
    attributes: {
      enabled: true
    }
    kty: 'RSA'
    keySize: 2048
  }
}

resource mytestencryptionset 'Microsoft.Compute/diskEncryptionSets@2021-08-01' = {
  name: 'mytestencryptionset'
  location: resourceGroup().location
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    activeKey: {
      keyUrl: key.properties.keyUriWithVersion
      sourceVault: {
        id: KV.id
      }
    }
    encryptionType: 'EncryptionAtRestWithCustomerKey'
  }
}

// Key Vault Crypto Service Encryption User
resource roleAssignment 'Microsoft.Authorization/roleAssignments@2020-10-01-preview' = {
  name: guid(mytestencryptionset.id,KV.id)
  scope: KV
  properties: {
    principalId: mytestencryptionset.identity.principalId
    roleDefinitionId: resourceId('Microsoft.Authorization/roleDefinitions', 'e147488a-f6f5-4113-8e2d-b22465e65bf6')
  }
}

@alex-frankel to be honest I have not tried this specific scenario but it is quite often that when a resource needs system assigned identity for specific feature that identity requires to have permissions already in order to use features on the resource. In this case it is chicken and egg problem as you cannot create the resource with system assigned identity and configure that specific feature. It will fail because the identity does not have the permissions. In those cases you can either run two deployments - first create the resource with the identity and assign the needed permissions. Second deployment that does all from first + configure the feature that requires the system identity. Of course needles to say that this approach is not sufficient for CI/CD / IaC thus the user assigned identity support is needed, in order to create the identity, assign permissions use it on the resource and configure the feature that requires it in a single deployment. Keep in mind that you can only get the object ID of system assigned identity only after it is created on the resource.

@bmoore-msft and @alex-frankel  in theory the steps by @bmoore-msft are fine, but in Azure, step 3 doesn't work with ARM templates or bicep. The problem is that the DES can't be created (it can, but ARM reports it as failed, which is why the only solution currently is a continue on error behavior) which leads to the whole deployment to be cancelled. The reason is that the DES checks for permissions on the key during creation time, which are not set yet, because the object id of the DES that is needed for the kv access policy is firstly accessible after creation of the DES.
I also can't do this with two deployments consistently, as there would have to be some way (which is not if I am not mistaken) to define in the ARM/bicep template to do specific actions only if the target resource exists. Then I would still run into the error and the first deployment would fail, as the DES is dependent on the access policy and the access policy is dependent on the DES, but if there is something like a check if the resource exists, I could on second try reverse the dependencies so that the access policy gets created first with the DES managed identity of the DES from the failed deployment.
That's classic chicken and egg and I am really frustrated that there are no solutions for this except the idea that @brwilkinson had to completely drop Access Policies.
The problem is that that is not possible for many customers and I think dropping such an essential part of the product can't be the only solution to such a fundamental problem.

hmmm @itpropro Let me do a repro with Access Policies. It seems odd that one would work and the other would not work...

Here is the exact same repro as my last one, however with access policies.
This also appears to work just fine? ✅

I commented out the RBAC items
I changed the access policies to /add instead of replace

// resource KV 'Microsoft.KeyVault/vaults@2021-06-01-preview' existing = {
//   name: 'ACU1-BRW-AOA-T5-kvApp01'
// }

resource KV 'Microsoft.KeyVault/vaults@2021-11-01-preview' = {
  name: 'ACU1-BRW-AOA-T5-kvApp04'
  location: resourceGroup().location
  properties: {
    sku: {
      family: 'A'
      name: 'standard'
    }
    tenantId: tenant().tenantId
    accessPolicies: []
    enabledForDeployment: true
    enabledForDiskEncryption: true
    enabledForTemplateDeployment: true
  }
}

resource key 'Microsoft.KeyVault/vaults/keys@2021-11-01-preview' = {
  name: 'mykey1'
  parent: KV
  properties: {
    attributes: {
      enabled: true
    }
    kty: 'RSA'
    keySize: 2048
  }
}

resource mytestencryptionset 'Microsoft.Compute/diskEncryptionSets@2021-08-01' = {
  name: 'mytestencryptionset2'
  location: resourceGroup().location
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    activeKey: {
      keyUrl: key.properties.keyUriWithVersion
      sourceVault: {
        id: KV.id
      }
    }
    encryptionType: 'EncryptionAtRestWithCustomerKey'
  }
}

resource keyVaultAccessPolicy 'Microsoft.KeyVault/vaults/accessPolicies@2021-11-01-preview' = {
  name: 'add'
  parent: KV
  properties: {
    accessPolicies: [
      {
        objectId: mytestencryptionset.identity.principalId
        tenantId: tenant().tenantId
        permissions: {
          keys: [
            'get'
            'wrapKey'
            'unwrapKey'
          ]
          certificates: []
          secrets: []
        }
      }
    ]
  }
}

// // Key Vault Crypto Service Encryption User
// resource roleAssignment 'Microsoft.Authorization/roleAssignments@2020-10-01-preview' = {
//   name: guid(mytestencryptionset.id,KV.id)
//   scope: KV
//   properties: {
//     principalId: mytestencryptionset.identity.principalId
//     roleDefinitionId: resourceId('Microsoft.Authorization/roleDefinitions', 'e147488a-f6f5-4113-8e2d-b22465e65bf6')
//   }
// }

I do believe that I have worked out what your issue is however... This is a long-standing well-known issue with keyvault Access Policies. Each time you deploy the the Keyvault, the Access Policies will be wiped out.

So this will work fine if you only have a single diskEncryptionSet, however if you have more and you attempt to redeploy the Keyvault, then each time you redeploy the keyvault, it wipes the policies for the previous system assigned identities, so only the latest one would be added and it breaks the older ones that were not deployed in the current template.

The fix for this is, have ALL disk access policies that use a particular KeyVault, defined the same template, so they all get added back as an access policy each time it was deployed
OR a better idea

Don't redeploy the keyvault, change it to an existing resource, the same as my first example

OR the best idea

Moved to using RBAC on they vault. This became available several years ago, I recommend you make the jump.

this empty array is what wipes out the previous policies.

Hey @brwilkinson, that was one of my first ideas when I tried to find a fix, but that all doesn't fix the problem that the DES fails to deploy if it has no access to the Key Vault. The issues with the wiped out policies is indeed well known, which is why I can either deploy the KV with empty policies and deploy the right one afterwards or I use add/replace for that. The result is the same, the policy is there and working as long as the DES existed beforehand and is referenced like that. If the DES is created in the same template, it will not work, because the DES will fail, which means that the access policy step is never reached.

Hi @itpropro I am still trying to repro a failure, however I cannot do so.
Can you test the above code sample, then let me know if it fails.
Also try to provide me a sample that does fail for you.

Hey @brwilkinson, that was one of my first ideas when I tried to find a fix, but that all doesn't fix the problem that the DES fails to deploy if it has no access to the Key Vault.

@itpropro - better yet, if you have a repro can you share?  Not only does the DES not need access to the vault to deploy, I don't think the vault even needs to exist for deployment/creation.  Obviously, it does [need to exist and have access] if anything uses the DES and encryption is needed, but that's a distinct operation from the PUT that creates the DES.
I'm wondering if you're actually using the DES in the template that you have a replication issue with permissions - which would be easy to test by just deploying the same template again after the failure.

Hi @itpropro
Did you make any progress on this one?

Hello @brwilkinson,
the implementation seems to have changed. I have replayed one of the older failed deployments in the deployment history without any adjustments, and now, the DES seems not to be dependent on accessing the Key Vault anymore.
Thanks for addressing this.

Thanks @itpropro glad things are working for you. 👍

The following works fine for me.
I don't use Access Policies anymore, I have moved to RBAC for KV permissions, however I am not sure that would be required, it's just a recommendation and my own preference
https://docs.microsoft.com/en-us/azure/key-vault/general/rbac-guide?tabs=azurepowershell
I don't believe the policy is being used when it is originally created, so It doesn't appear to be any chicken or egg here?
resource KV 'Microsoft.KeyVault/vaults@2021-06-01-preview' existing = {
  name: 'ACU1-BRW-AOA-T5-kvApp01'
}

resource key 'Microsoft.KeyVault/vaults/keys@2021-11-01-preview' = {
  name: 'mykey1'
  parent: KV
  properties: {
    attributes: {
      enabled: true
    }
    kty: 'RSA'
    keySize: 2048
  }
}

resource mytestencryptionset 'Microsoft.Compute/diskEncryptionSets@2021-08-01' = {
  name: 'mytestencryptionset'
  location: resourceGroup().location
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    activeKey: {
      keyUrl: key.properties.keyUriWithVersion
      sourceVault: {
        id: KV.id
      }
    }
    encryptionType: 'EncryptionAtRestWithCustomerKey'
  }
}

// Key Vault Crypto Service Encryption User
resource roleAssignment 'Microsoft.Authorization/roleAssignments@2020-10-01-preview' = {
  name: guid(mytestencryptionset.id,KV.id)
  scope: KV
  properties: {
    principalId: mytestencryptionset.identity.principalId
    roleDefinitionId: resourceId('Microsoft.Authorization/roleDefinitions', 'e147488a-f6f5-4113-8e2d-b22465e65bf6')
  }
}

This does not work in most cases as the roleAssignment would fail since diskEncryptionSet takes time to create the resource. By that time we are trying to create the roleAssignment and it fails . I just tried it .
I cannot find a way to wait for diskEncryptionSet to be created before running the roleAssignment.
Best solution would be for diskEncryptionSet to support User Managed Identities.

Hello @brwilkinson, the implementation seems to have changed. I have replayed one of the older failed deployments in the deployment history without any adjustments, and now, the DES seems not to be dependent on accessing the Key Vault anymore. Thanks for addressing this.

I doubt this will work . If you delete all the resources and try again it will fail again.

Yeah it still fails every time, but the first deployment. There is still a fundamental issue with how bicep works here. I ended up creating a custom script action that checks for existence of the Key Vault and deploys the respective resources.
This will stay the only possible solution to this problem until either Bicep introduces the ability to check for resource existence or if Azure ARM changes internally.

Yeah it still fails every time, but the first deployment. There is still a fundamental issue with how bicep works here. I ended up creating a custom script action that checks for existence of the Key Vault and deploys the respective resources. This will stay the only possible solution to this problem until either Bicep introduces the ability to check for resource existence or if Azure ARM supports user assigned identities for everything internally.

Absolutely. The issue is still valid and needs to be fixed by MS .

@imshinto - it would be great if you could provide a bicep snippet that reproduces the problem you're running into...

It's the same with access policies. The problem is still that the disk encryption set needs access to the Key Vault secret (except for the first creation) which is evaluated on every change.
It's the same thing that causes the well known problem that you cannot idempotently manage a Azure Key Vault with Bicep/ARM without having the old access policies overwritten.
If you first create a Key Vault and add some access policies to it and you want to modify them later, it will delete your policies, as the policies parameter is mandatory on the Key Vault object.
This is currently only solvable by implementing your own logic that checks if the Key Vault (or respective other resource) exists already. Until there is a change in the ARM provider to support user assigned identities everywhere and remove the requirement for a Key Vault to provide the polices parameter, this can only be solved with custom logic.
Here is the necessary code for a module I use to check for existence in case anyone has this problem until it is fixed:
existenceCheck module
param location string
param keyVaultName string
param managedIdentityId string
param forceUpdateTag string = newGuid()

resource existenceCheck 'Microsoft.Resources/deploymentScripts@2020-10-01' = {
  name: 'existenceCheck'
  kind: 'AzurePowerShell'
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
      '${managedIdentityId}': {}
    }
  }
  location: location
  properties: {
    retentionInterval: 'PT1H'
    forceUpdateTag: forceUpdateTag
    azPowerShellVersion: '7.0'
    arguments: '-keyVaultName ${keyVaultName} -resourceGroupName ${resourceGroup().name} -subscriptionId ${subscription().subscriptionId}'
    scriptContent: '''
      param([string]$keyVaultName, [string]$resourceGroupName, [string]$subscriptionId)
      $null = Set-AzContext -Subscription $subscriptionId
      Write-Output "Checking for existence of key vault '$keyVaultName' in resource group '$resourceGroupName'"
      $keyVault = Get-AzKeyVault -VaultName $keyVaultName -ResourceGroupName $resourceGroupName
      $DeploymentScriptOutputs = @{}
      $DeploymentScriptOutputs['keyVaultExists'] = !!$keyVault
    '''
  }
}

output keyVaultExists bool = existenceCheck.properties.outputs.keyVaultExists

Key Vault module
param location string
param keyVaultName string
param keyVaultExists bool

resource keyVault 'Microsoft.KeyVault/vaults@2021-11-01-preview' = if (!keyVaultExists) {
  name: keyVaultName
  location: location
  properties: {
    sku: {
      family: 'A'
      name: 'standard'
    }
    accessPolicies: []
  }
}

resource existingKeyVault 'Microsoft.KeyVault/vaults@2021-11-01-preview' existing = if (keyVaultExists ) {
  name: keyVaultName
}

output keyVaultId string = keyVaultExists ? existingKeyVault.id : keyVault.id

This can then be used together:
module keyVaultExistenceCheck 'modules/keyVaultExistenceCheck.bicep' = {
  name: 'keyVaultExistenceCheck'
  scope: keyVaultResourceGroup
  params: {
    location: location
    keyVaultName: keyVaultName
    managedIdentityId: managedIdentityId
  }
}

module keyVault 'modules/keyVault.bicep' = {
  name: keyVaultName
  scope: keyVaultResourceGroup
  params: {
    keyVaultName: keyVaultName
    location: location
    keyVaultExists: keyVaultExistenceCheck.outputs.keyVaultExists
  }
}

You can then manage the access policies with a separate module:
param keyVaultName string
param diskEncryptionSetPrincipalId string

resource keyVaultAccessPolicy 'Microsoft.KeyVault/vaults/accessPolicies@2021-11-01-preview' = {
  name: '${keyVaultName}/add'
  properties: {
    accessPolicies: [
      {
        objectId: diskEncryptionSetPrincipalId
        tenantId: tenant().tenantId
        permissions: {
          keys: [
            'get'
            'wrapKey'
            'unwrapKey'
          ]
          certificates: []
          secrets: []
        }
      }
    ]
  }
}

This issue results from having no state on the one hand and having a lot of Azure resources that cannot be deployed idempotent. I really hope some kind of existence check will be available soon, as I stumbled upon this problem in many projects already...

@imshinto

Here , when the roleAssignment is executed, the diskEncryptionSet is not created yet (The creation is in progress). Hence it fails. Even the re-triggering of deployment will fail mentioning the diskEncryptionSet not having access to keyVault(since there is no permissions yet)

In the greenfield case here it's not that the DES is not created but the MSI has not propagated through AAD - so the roleAssignment will not find the objectId in some cases.  You need to add the principalType property to the roleAssignment and that will force it's creation, e.g.
resource roleAssignment 'Microsoft.Authorization/roleAssignments@2020-10-01-preview' = {
  name: guid(diskEncryptionSet.id,kv.id)
  scope: kv
  properties: {
    principalId: diskEncryptionSet.identity.principalId
    roleDefinitionId: resourceId('Microsoft.Authorization/roleDefinitions', 'e147488a-f6f5-4113-8e2d-b22465e65bf6')
    principalType: 'ServicePrincipal'
  }
}

If that doesn't work for you maybe add the entire template and the error that you're seeing...
@itpropro
When using accessPolicies instead of roleAssignments, there isn't really a good solution to the propagation problem other than a deploymentScript or using RBAC.  UserAssigned identities will also work but that's not always an option. This scenario is one of the reasons why the resources with their own access controls are starting to enable RBAC for access control.

@imshinto

Here , when the roleAssignment is executed, the diskEncryptionSet is not created yet (The creation is in progress). Hence it fails. Even the re-triggering of deployment will fail mentioning the diskEncryptionSet not having access to keyVault(since there is no permissions yet)

In the greenfield case here it's not that the DES is not created but the MSI has not propagated through AAD - so the roleAssignment will not find the objectId in some cases. You need to add the principalType property to the roleAssignment and that will force it's creation, e.g.
resource roleAssignment 'Microsoft.Authorization/roleAssignments@2020-10-01-preview' = {
  name: guid(diskEncryptionSet.id,kv.id)
  scope: kv
  properties: {
    principalId: diskEncryptionSet.identity.principalId
    roleDefinitionId: resourceId('Microsoft.Authorization/roleDefinitions', 'e147488a-f6f5-4113-8e2d-b22465e65bf6')
    principalType: 'ServicePrincipal'
  }
}

If that doesn't work for you maybe add the entire template and the error that you're seeing...
@itpropro
When using accessPolicies instead of roleAssignments, there isn't really a good solution to the propagation problem other than a deploymentScript or using RBAC. UserAssigned identities will also work but that's not always an option. This scenario is one of the reasons why the resources with their own access controls are starting to enable RBAC for access control.

Adding the principalType worked. The role assignment was created successfully. But just after that I have a VM creation which uses this diskEncryptionSet. But since the role assignment had to wait , the VM creation went in first and i got an RBAC error
{     "status": "Failed",     "error": {         "code": "KeyVaultAccessForbidden",         "message": "Unable to access key vault resource 'xxxx' to enable encryption at rest. Please grant get, wrap and unwrap key permissions to disk encryption set 'xxxxx-des'. Please visit https://aka.ms/keyvaultaccessssecmk for more information."     } }
Is there a way to make the VM creation wait until the roleAssignment is completed.

No, but there is a discussion here https://github.com/Azure/bicep/issues/1013 that basically says, dependsOn should report the correct state, otherwise you should open a issue (for role assignments in this case) to fix the dependsOn behavior to only report success after the assignment is really created.
As a workaround, you could deploy some dummy resource like Microsoft.Resources/deploymentScripts with just a Start-Sleep in it. This way you don't even need to provide a managed identity and you can just put this in between the steps and refer it with dependsOn.

@imshinto

Is there a way to make the VM creation wait until the roleAssignment is completed.

Assuming you take the same template, and deploy it again right away and it works, then you're likely running into a replication delay... the roleAssignment is created but hasn't replicated to all regions yet.  dependsOn doesn't really "fix" the issue because the dependency is only waiting for the resource to be created, not replicated.
If you have other resources to deploy that you can squeeze between the two, that's usually sufficient, replication usually takes seconds but can take minutes.  Aside from that - as @itpropro mentioned a deploymentScript, while heavyweight can also introduce a delay.

Has anyone found the solution to the problem of assigning access policies to KV for disk encryption set? I cannot change existing KV.

If you cannot modify the vault, the only option would be to use RBAC on the vault instead of accessPolicies...  somehow you have to be able to give the new principal permission to it...

Hi everyone, as this issue is still open I thought I'd try my luck here.
I'm using a key vault that has RBAC enabled, which does on the other side mean I cannot use any access policies on it.
Now, at first I also encountered the issue described here, that after the initial deployment of the bicep resource of the disk encryption set, the disk encryption set has no access to the key vault. To mitigate/test I tried to rewrite the logic using azure cli.
However, after then successfully deploying the disk encryption set I receive a similar error when actually trying to encrypt the disk of a linux vm.
The following scripts are used:
#!/usr/bin/env bash
set -e

az disk-encryption-set create --key-url $KEY_URL \
                              --name encryption-set\
                              --resource-group $RESOURCE_GROUP \
                              --auto-rotation true \
                              --encryption-type EncryptionAtRestWithCustomerKey \
                              --location $LOCATION \
                              --mi-system-assigned true \
                              --source-vault $KEYVAULT_ID \
                              --tags $TAGS

$MANAGED_IDENTITY=$(az disk-encryption-set list -g $RESOURCE_GROUP --query ["[0].identity.principalId"] -o tsv)
$DISK_ENCRYPTION_SET=$(az disk-encryption-set list -g $RESOURCE_GROUP --query ["[0].id"] -o tsv)

az role assignment create --role "Key Vault Crypto Service Encryption User" --assignee $MANAGED_IDENTITY --scope $KEYVAULT_ID

az disk update --disk-encryption-set $DISK_ENCRYPTION_SET --ids $DISK_ID --encryption-type EncryptionAtRestWithCustomerKey

So what I am doing is actively assign the necessary role to the system managed identity of the DES. In my understanding the DES now should be able to access the key vault.
However when running the last command (az disk update) I receive the following error message:
(KeyVaultAccessForbidden) Unable to access key vault resource 'https://xyz-keyvault.vault.azure.net/keys/vm-disk-encyption-key/****' to enable encryption at rest. Please grant get, wrap and unwrap key permissions to disk encryption set 'encryption-set'. Please visit https://aka.ms/keyvaultaccessssecmk for more information.
Code: KeyVaultAccessForbidden
Message: Unable to access key vault resource 'https://xyz-keyvault.vault.azure.net/keys/vm-disk-encyption-key/****' to enable encryption at rest. Please grant get, wrap and unwrap key permissions to disk encryption set 'encryption-set'. Please visit https://aka.ms/keyvaultaccessssecmk for more information.

I do not see anything I could additionally do, because access policies are not possible in RBAC key vaults, and the DES already has all rights it needs.
Thanks in advance for any help!

@eliasmueller you can assign one of the following roles since you are using RBAC not access policies.
Get-AzRoleDefinition | where name -match vault | foreach Name
Key Vault Contributor
Key Vault Administrator
Key Vault Crypto User
Key Vault Secrets Officer
Key Vault Secrets User
Key Vault Certificates Officer
Key Vault Reader
Key Vault Crypto Service Encryption User
Key Vault Crypto Officer

You can assign this before you deploy the keyvault by assigning it on the Resource Group where the keyvault will be deployed to OR you can also assign on the Keyvault OR even on individual keys/secrets/certs... I recommend to just assign it on the Resource group, given it's not common to deploy more than 1 vault per RG.

I would like to close this one out. however @itpropro if you are able to confirm you have this working now?
As I mentioned I would recommend moving to RBAC, however I have tested both and able to get both working.

I would like to close this one out. however @itpropro if you are able to confirm you have this working now?
As I mentioned I would recommend moving to RBAC, however I have tested both and able to get both working.

Using RBAC, everything works, but I can still reproduce the same error with the old model. I will not use anything else than RBAC anyway, so I see this as another reason to use granular permissions now ;)

Thank you @itpropro for confirming.

Adding a note on this one: New versions of the api support user assigned managed identity.
so you can easily pre create and pre assign the user identity with access to the vault.
resource mytestencryptionset 'Microsoft.Compute/diskEncryptionSets@2022-07-02' = {
  name: 'mytestencryptionset'
  location: resourceGroup().location
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
      '${UAI.id}': {}
    }
  }


Upgrade to .NET 4.6.1

Upgrade to .NET 4.6.1 which can be use with .NET Standard library 2.0
PowerShell script tool to upgrade the .NET framework to all projects and

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: aaronamm sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

/cc @BenedekFarkas

@sebastienros  @BenedekFarkas  I just found for using C# 6 feature, you don't need to upgrade to a newer version of .NET. However, I will let you decide if you want to keep this PR or not because PowerShell script to change targeting .NET still useful.
Thanks.

Related issue: #8007, which mentions that 4.6.2 would also be nice for HTTP/2 support.
@sebastienros judging from the .NET documentation 4.6 would sufficient for that - on the other hand is there a reason not to upgrade to 4.6.2 or even higher? Hosting could be an issue if they are behind with the Windows or Windows Server version, right?

According to this (https://docs.microsoft.com/en-us/dotnet/framework/get-started/system-requirements), as long as the server has Windows Server 2008 R2, they can go up to .NET Framework 4.7.2 (maybe 4.7.3 also?).

@BenedekFarkas  @zakaluka  Sorry for late reply since I was on a vacation.
I am happy to help to update to any versino.
Please let me the your decision.
For me,  it would be nice, if we can start using a class library that supports only .NET standard library 2.0 up.
Making a .NET standard library is a great idea to maintain only one code base and works to all platform.
It is very useful for opensource source projects.

Hey, sorry for replying just now!
@aaronamm the PS script is really nice - I was doing these kinds of bulk updates with search and replace in Notepad++. :)
Maybe we can improve it a bit by processing the XML, but it might not be necessary. The concept, however is nice as we can do the same for LangVersion as well for example.
@zakaluka thanks for looking into the framework compatibility - I think we should be OK with 4.7.2 (so we support from Windows Server 2008 R2 onwards), but I'd like @sebastienros' opinion on it.

@BenedekFarkas  I think I can do it.
I will convert the PowerShell script to process XML in a PowerShell way and update the project to .NET 4.7.2.

4.6.1 approved


perf: Avoid making an extra heap copy in DecodeBytes

Deserializing bytes right now is making a new heap copy of the slice. In IAVL v1, with legacy nodes, over a 100 block period this appeared to be 0.3% of the CPU time. (Plus heap / memory pressure)
This is only needed if the input bytes are mutated, but this is not the case in any usage of this, as each read is coming from a new GetNode or cache result. GetNode returns new heap-data for the value, and the cache entries do not mutate. Hence we can save on CPU time + heap pressure with this.

@ValarDragon could we close it since it is included in #877?

Sure!


What about pangocairo crate?

Is it planned? Or is it already exist and I didn't found it?
Or not even as a crate, but as a feature.

I know pango (which has a binding), I know cairo (which also has a binding), I didn't know pangocairo was a thing.

I'm speaking about this: https://developer.gnome.org/pango/stable/pango-Cairo-Rendering.html
As far as I understand, it's part of pango. On Gentoo I have /usr/include/pango-1.0/pango/pangocairo.h as part of pango package.
Looks like pango crate doesn't have such bindings. Probably title is a bit misleading.

Hum strange. Have you check the master branch or the crate one?

Yes. I didn't found anything about cairo here: https://github.com/gtk-rs/sys/blob/master/pango-sys/src/lib.rs

Then we haven't done it yet. Motivated to add it? :)

All I can found is: https://github.com/gtk-rs/gir-files/blob/master/PangoCairo-1.0.gir
Yes, why not. Any readme about how to do this? As far as I understand you are using codegen for this.

Yes, take a look at gir's README (which is a bit short...). Normally you should be ok with it but don't hesitate to ask if you can't find out.

Well... I didn't really understand what I need to do.
As far as I understand, I should create gir-pangocairo.toml in gtk-rs/sys/conf/, because gir-files contains it as separate file. So it's really should be a create. Also there are an actual library called libpangocairo, which is part of the pango package.
Then I should hit make, but I get an error:
% make
gir/target/release/gir -c conf/gir-pangocairo.toml -o /media/data/Programming/Sources/gtk/sys/pangocairo-sys -d gir-files
thread 'main' panicked at 'src/parser.rs:659 Missing alias target's c:type in gir-files/freetype2-2.0.gir:12:7', src/parser.rs:53
note: Run with `RUST_BACKTRACE=1` for a backtrace.
make: *** [Makefile:9: pangocairo-sys/src/lib.rs] Error 10

Looks like it didn't like this line: https://github.com/gtk-rs/gir-files/blob/master/freetype2-2.0.gir#L12

And that's the moment when we summon @EPashkin to say: there is a panic! 😨

Looks like freetype2-2.0.gir is invalid. It should look like this:
<type name="int32" c:type="int"/>

but it's
<type name="int32"/>

Then I have to fix PangoCairo-1.0.gir
-        <type c:type="PangoFcFontMap"/>
+        <type name="FontMap" c:type="PangoFcFontMap"/>

Now I'm getting:
% make
gir/target/release/gir -c conf/gir-pangocairo.toml -o /media/data/Programming/Sources/gtk/sys/pangocairo-sys -d gir-files
thread 'main' panicked at 'Incomplete library, unresolved: ["freetype2.int32"]', src/library_postprocessing.rs:41
note: Run with `RUST_BACKTRACE=1` for a backtrace.
make: *** [Makefile:9: pangocairo-sys/src/lib.rs] Error 101

After patching freetype2-2.0.gir:
-      <type name="int32"/>
+      <type name="Int32" c:type="gint"/>

I got it working and I get a pangocairo-sys dir.
What next? I should send a pull request with it and then manually create bindings in a new git repo, like pango itself?

And I still can build it, because it depend on: cairo-sys, freetype2-sys.

Currently I against patching gir-files crate, so better not include pangocairo-sys to sys crate.
int32 IMHO better fixed on gir side by adding its type (gint as can be 64bits)
Don't know how to fix PangoFcFontMap issue as no such type in Pango-1.0.gir (only found PangoFontMap).

Seems PangoCairo-sys will be written manually (maybe with using generated code as template).
I also found https://github.com/jpernst/pangocairo-sys on crates.

I see. Then I will try to write it manually.
Yes, I saw this crate, but it has binding not only for pangocairo and I don't think it can be mixed with existing cairo and pango crates. And I want a crate, which will depend on gtk-rs libs.

So:

I have created pangocairo-sys crate, which works fine.
Then I have created pangocairo crate, which depend on pangocairo-sys, cairo and pango. Empty library has been build without errors.
Now I have to create an actual wrapper, to pangocairo-sys methods.

And there are problems. The root function looks like this:
pub fn pango_cairo_create_layout(cr: *mut cairo::cairo_t) -> *mut pango::PangoLayout;

so I have to write something like this:
pub fn create_layout(content: &cairo::Content) -> pango::Layout {
    unsafe { from_glib_full(ffi::pango_cairo_create_layout(content.to_glib_none().0)) }
}

but:

I get an error:the method `to_glib_none` exists but the following trait bounds were not satisfied: `cairo::Content : glib::translate::ToGlibPtr<_>`

There are no pango::Layout in pango crate.

Content is enum
May be you mean Context.
And yes, no PangoLayout

Yes, my mistake. So I should implement/wait for PangoLayout?

I added generation of layout in https://github.com/EPashkin/pango/tree/add_generation
Commit not full due my git error (It can't init submodules).
Makefile copied from working project so IMHO it run normally.
For generating I used fixed gir from https://github.com/EPashkin/gir/tree/fix_for_pango

So I should wait until it will be merged to the main tree?

I prefer leave this branch to you :wink:
Feel free to use it in your PR to pango.
At minimum better deignore Layout::get_alignment by adding "Pango.Alignment", to generate=[

I did not think that adding a simple wrapper like pangocairo will be such pain.
Yours branch contain an auto folder which should be manually ported to the widget folder. Am I correct?

Not, auto folder is primary, widget in pango crates or files in src in other gir-connected crates used only for part that can't properly generated by gir.
In your case I think you need add generation pango classes that needed for pangocairo crate and try minimize commented functions in generated code (if it works without problem, ex. "Pango.Rectangle" currently can't be used/generated).

Well, from on side I still have no idea what to do, but from another, I really need this crate...
Is there are any plans for making it or it not even planned?

Is I remember, no certain plans about applying gir to pango crate, mostly due gir deficiencies.
I don't use pango, so I don't sure what classes needed first, that why I propose that you add PR with classes or other definition that needed for pangocairo.
If you don't want or don't have time do it, then I add PR on weekend with minimal changes from current add_generation branch

The problem is that I didn't know what I need too. I have C code that I should port to Rust. It uses pangocairo. As far as I can see it uses almost all pangocairo methods.
I can write, I hope so, a wrapper around ffi, like pango and cairo itself, but all this generator stuff are beyond my understanding.

I mean if your pangocairo method will use some pango definition (as with Layout) you can add it (or try) with gir just by adding it definition to generate array in Gir.toml and run gir on it.

minimize commented functions in generated code

Manually?

I have finally figure out how to build your repo.
So I've added Pango.Alignment and got this error:
error[E0432]: unresolved import `glib_ffi`
 --> src/auto/enums.rs:5:5
  |
5 | use glib_ffi;
  |     ^^^^^^^^ no `glib_ffi` in the root

As I understand it's also should be fixed via gir?

I've added
[dependencies.glib-sys]
git = "https://github.com/gtk-rs/sys"
version = "0.3.2"

and not it saying that it's useless:
warning: unused import: `glib_ffi`, #[warn(unused_imports)] on by default
 --> src/auto/enums.rs:5:5
  |
5 | use glib_ffi;
  |     ^^^^^^^^

warning: unused import: `glib::error::ErrorDomain`, #[warn(unused_imports)] on by default
 --> src/auto/enums.rs:6:5
  |
6 | use glib::error::ErrorDomain;
  |     ^^^^^^^^^^^^^^^^^^^^^^^^

So it's shouldn't be generated at first place?

Finally I made my create_layout method from pangocairo working.
Then what? I should add all methods from pangocairo while adding missing types to pango's Gir.toml? And if it will work - I'll have to create a pull request to yours pango fork?

Seems you solved extern crate glib_sys as glib_ffi; problem.
About adding PR: you need add it to https://github.com/gtk-rs/pango
If you can, remove lines in lib.rs like pub use ffi::PangoGravity as Gravity; and add generate these enums and flags. After that you need add many to_glib`from_glibto files inwidgets` dir.

Ok.
Next problem:
I've added "Pango.FontMap" to the Gir.toml, because I need this struct, and it generates all the data, but under #[cfg(feature = "v1_4")]. And the problem is that there is no v1_4 in pango_sys.
pango crate itself has v1_38 and v1_42.
I've added v1_4 = ["v1_38", "pango-sys/v1_38"] manually to pango's Cargo.toml and it's works for now.
How it should be solved?

Change\add min_cfg_version = "1.30" in pango\Gir.toml
I forgot to do that.

Ok. Fixed.
Where should I implement ToGlibPtr/FromGlibPtr traits? Not in the auto dir, presumably. In lib.rs?

For what class?

Yeap. Looks like it already implemented in by glib_wrapper!, because I'm getting
conflicting implementations of trait `glib::translate::ToGlibPtr<'_, *mut ffi::PangoLayout>` for type `&auto::layout::Layout`:

But if I remove it - I'm getting:
error[E0277]: the trait bound `pango::Layout: glib::translate::ToGlibPtr<'_, *mut pango_sys::PangoContext>` is not satisfied
  --> src/lib.rs:35:74
   |
35 |     unsafe { ffi::pango_cairo_update_context(cr.to_glib_none().0, layout.to_glib_none().0); };
   |                                                                          ^^^^^^^^^^^^ the trait `glib::translate::ToGlibPtr<'_, *mut pango_sys::PangoContext>` is not implemented for `pango::Layout`
   |
   = help: the following implementations were found:
   = help:   <pango::Layout as glib::translate::ToGlibPtr<'a, *mut glib::object::GObject>>
   = help:   <pango::Layout as glib::translate::ToGlibPtr<'a, *const pango_sys::PangoLayout>>
   = help:   <pango::Layout as glib::translate::ToGlibPtr<'a, *mut pango_sys::PangoLayout>>

The method itself:
pub fn show_layout(cr: &cairo::Context, layout: &pango::Layout) {
    unsafe { ffi::pango_cairo_update_context(cr.to_glib_none().0, layout.to_glib_none().0); };
}

Wait. I've misspelled the method... Forget about it.

Now I'm getting:
error: linking with `cc` failed: exit code: 1
  |

... a lot of text
 "-l" "pango-1.0" "-l" "gobject-2.0" "-l" "glib-2.0" "-l" "cairo" "-l" "pango-1.0" "-l" "gobject-2.0" "-l" "glib-2.0" "-l" "gobject-2.0" "-l" "glib-2.0" "-l" "glib-2.0" "-l" "util" "-l" "dl" "-l" "rt" "-l" "pthread" "-l" "gcc_s" "-l" "pthread" "-l" "c" "-l" "m" "-l" "rt" "-l" "util"
  = note: /media/data/Programming/Projects/rust/resvg/tools/rendersvg/target/debug/deps/libpangocairo-a39c28cc6685b440.rlib(pangocairo-a39c28cc6685b440.0.o): In function `pangocairo::create_layout':
/media/data/Programming/Projects/rust/pangocairo/src/lib.rs:27: undefined reference to `pango_cairo_create_layout'
collect2: error: ld returned 1 exit status

I understand that some libraries are missing during linking, but I've selected them in build.rs.
Here is pangocairo-sys/build.rs:
extern crate pkg_config;

use pkg_config::{Config, Error};
use std::env;
use std::io::prelude::*;
use std::io;
use std::process;

fn main() {
    if let Err(s) = find() {
        let _ = writeln!(io::stderr(), "{}", s);
        process::exit(1);
    }
}

fn find() -> Result<(), Error> {
    let package_name = "pango";
    let shared_libs = ["pango-1.0", "pangocairo-1.0", "pango-1.0", "pangoft2-1.0"];
    let version = "1.38";
    // let version = if cfg!(feature = "v1_42") {
    //     "1.42"
    // } else if cfg!(feature = "v1_38") {
    //     "1.38"
    // } else if cfg!(feature = "v1_36_7") {
    //     "1.36.7"
    // } else if cfg!(feature = "v1_34") {
    //     "1.34"
    // } else if cfg!(feature = "v1_32_4") {
    //     "1.32.4"
    // } else if cfg!(feature = "v1_32") {
    //     "1.32"
    // } else if cfg!(feature = "v1_31") {
    //     "1.31"
    // } else {
    //     "1.30"
    // };

    if let Ok(lib_dir) = env::var("GTK_LIB_DIR") {
        for lib_ in shared_libs.iter() {
            println!("cargo:rustc-link-lib=dylib={}", lib_);
        }
        println!("cargo:rustc-link-search=native={}", lib_dir);
        return Ok(())
    }

    let target = env::var("TARGET").unwrap();
    let hardcode_shared_libs = target.contains("windows");

    let mut config = Config::new();
    config.atleast_version(version);
    if hardcode_shared_libs {
        config.cargo_metadata(false);
    }
    match config.probe(package_name) {
        Ok(library) => {
            if hardcode_shared_libs {
                for lib_ in shared_libs.iter() {
                    println!("cargo:rustc-link-lib=dylib={}", lib_);
                }
                for path in library.link_paths.iter() {
                    println!("cargo:rustc-link-search=native={}", path.to_str().unwrap());
                }
            }
            Ok(())
        }
        Err(Error::EnvNoPkgConfig(_)) | Err(Error::Command { .. }) => {
            for lib_ in shared_libs.iter() {
                println!("cargo:rustc-link-lib=dylib={}", lib_);
            }
            Ok(())
        }
        Err(err) => Err(err),
    }
}

and Here is pangocairo-sys/Cargo.toml:
[package]
name = "pangocairo-sys"
version = "0.1.0"
authors = ["Reizner Evgeniy<EMAIL_ADDRESS>build = "build.rs"
license = "MIT"
description = "Rust bindings for the PangoCairo library"
repository = "https://github.com/razrfalcon/pangocairo-sys"
keywords = ["pango", "cairo", "ffi"]

[dependencies]
libc = "0.2"
cairo-sys-rs = { git = "https://github.com/gtk-rs/cairo" }
pango-sys = { git = "https://github.com/gtk-rs/sys" }

[dependencies.glib-sys]
git = "https://github.com/gtk-rs/sys"

[dependencies.gobject-sys]
git = "https://github.com/gtk-rs/sys"

[build-dependencies]
pkg-config = "0.3.7"

[lib]
name = "pangocairo_sys"

In build, package_name = "pangocairo"; ?
I don't see -l "pangocairo" in command line, may be you need minimize shared_libs

D'oh. Fixed by a proper package_name.

And again.
I've added "Pango.FontDescription" to the Gir.toml and removed one from widgets.
Now pango is failed to build:
error[E0277]: the trait bound `auto::font_description::FontDescription: glib::translate::FromGlibPtr<*const ffi::PangoFontDescription>` is not satisfied
  --> src/auto/layout.rs:86:13
   |
86 |             from_glib_none(ffi::pango_layout_get_font_description(self.to_glib_none().0))
   |             ^^^^^^^^^^^^^^ the trait `glib::translate::FromGlibPtr<*const ffi::PangoFontDescription>` is not implemented for `auto::font_description::FontDescription`
   |
   = help: the following implementations were found:
   = help:   <auto::font_description::FontDescription as glib::translate::FromGlibPtr<*mut ffi::PangoFontDescription>>
   = note: required because of the requirements on the impl of `glib::translate::FromGlibPtr<*const ffi::PangoFontDescription>` for `std::option::Option<auto::font_description::FontDescription>`
   = note: required by `glib::translate::from_glib_none`

error: aborting due to previous error

Method itself:
    pub fn get_font_description(&self) -> Option<FontDescription> {
        unsafe {
            from_glib_none(ffi::pango_layout_get_font_description(self.to_glib_none().0))
        }
    }

Problem in that sys says that pango_layout_get_font_description return *const ffi::PangoFontDescription, and generated defined only for *mut ffi::PangoFontDescription.
You need change in sys or add impl FromGlibPtr<*const ffi::PangoFontDescription> for FontDescription {.
Also I don't sure that you need generate classes in widgets if it work without it.

So I should add it to auto/layout.rs? But it will be overwritten.

Also I don't sure that you need generate classes in widgets if it work without it.
Not really understand it. I do not generate anything in widgets. I have removed font_description.rs from widgets, because it was generated in auto.

I mean that maybe you don't need generating any of widgets as it already present in crate.
But if it really need add something to generated crate, you can add module font_description.rs near lib.rs (or in case of pango in widgets) where be only impl FromGlibPtr for generated FontDescription

The problem is that in this case I will have to edit layout.rs, because it uses FontDescription. So in either cases I have to do manual corrections after each gen.

Seems there some misunderstanding. As I see call pango_layout_get_font_description don't compiled due FontDescription don't implement FromGlibPtr<*const ffi::PangoFontDescription> (only implement *mut ...) and you need add it somewhere. This can be added in separate module with  same name  font_description.rs but not in auto dir

Don't see reason to change generated layout.rs.

Also I missed your reason for generating "Pango.FontDescription".
If you need uncomment generated pango_layout_get_font_description, you can add "pango_layout_get_font_description" to manual array, without removing widgets\font_description.rs

Don't see reason to change generated layout.rs.

Because all changes will be missed after next gen.
How exactly should I implement this trait?
impl FromGlibPtr<*const ffi::PangoFontDescription> for FontDescription {
    unsafe fn from_glib_none(ptr: *const ffi::PangoFontDescription) -> Self {
        let tmp = ffi::pango_font_description_copy(ptr);
        assert!(!tmp.is_null());
        FontDescription(tmp)
    }

    unsafe fn from_glib_full(ptr: *const ffi::PangoFontDescription) -> Self {
        assert!(!ptr.is_null());
        FontDescription(ptr)
    }
}

error[E0308]: mismatched types
  --> src/lib.rs:29:25
   |
29 |         FontDescription(tmp)
   |                         ^^^ expected struct `glib::boxed::Boxed`, found *-ptr
   |
   = note: expected type `glib::boxed::Boxed<ffi::PangoFontDescription, auto::font_description::MemoryManager>`
   = note:    found type `*mut ffi::PangoFontDescription`

error[E0308]: mismatched types
  --> src/lib.rs:34:25
   |
34 |         FontDescription(ptr)
   |                         ^^^ expected struct `glib::boxed::Boxed`, found *-ptr
   |
   = note: expected type `glib::boxed::Boxed<ffi::PangoFontDescription, auto::font_description::MemoryManager>`
   = note:    found type `*const ffi::PangoFontDescription`

I've tried using widgets/font_description.rs, but I still got the same error. There is no particular motivation, I'm just trying to make it work.

Strange, I added to manual, added impl for const * with only notable changes in from_glib_full and it works fine.
No changes in auto\layout.rs.
    unsafe fn from_glib_full(ptr: *const ffi::PangoFontDescription) -> Self {
        assert!(!ptr.is_null());
        FontDescription(ptr as *mut _)
    }

*no changes other that generated

manual is in Gir.toml? Doesn't seems like it changed anything.
It's works for now. In the end I removed FontDescription from Gir.toml and edited auto/layout.rs manually.

Yes,
manual = [
    "Pango.FontDescription",
]

with removing it from generate, add impl to widgets/fond_description.rs and remove auto/font_description.rs.
I have no idea what you changed in auto/layout.rs

Cool. It works.
You sad:

you can add "pango_layout_get_font_description" to manual array

So I put pango_layout_get_font_description in manual. Which was wrong. Now everything works fine, without any changes in auto.

Sorry, I was copied wrong text, fixed it, but don't say about it.

Finally! I drew some text. Mission accomplished.
Will try to prepare a pull request tomorrow.

@RazrFalcon, I merged gir's PR with pango-related fixes.
Please, upgrade gir submodule to it. First line of generated file look // This file was generated by gir (aacfc92) from gir-files (71d73f0)

Done?.. It's my first pull request, so I could easily mess it up.

pango is done for now (I'll probably add new structs when I need them). But what to do with pangocairo-sys and pangocairo?
Should I publish them as two separate repos in my profile or in gtk project? Should I publish them oi n crates?

I prefer that it published in your profile (may be even only one crate as https://crates.io/crates/cairo-rs).
@GuillaumeGomez, what you think?

Ok. By one crate you mean that sys will be inside?

Yes.

Only problem with publishing your crate that we don't have planned date to publish pango, so at first it may depends not on crate but on git.

It's not a problem. I'm not planning to release my main project in crates any time soon.

How to reexport "defines" from sys? I need this define: https://github.com/gtk-rs/sys/blob/master/pango-sys/src/lib.rs#L573
Currently done by adding pub use ffi::PANGO_SCALE as SCALE; to the libs.rs.

Second problem. I need Rectangle. But adding "Pango.Rectangle",to manual doesn't work, because currently Rectangle is a trait. The object itself is ffi::PangoRectangle.
How is this should be solved?
PS: maybe we should create another issue for this tasks, because it's strictly pango crate problems.

Constant really can just reexported, but name better rustified like Scale.
As gir currently don't support direct record, you can only rewrite rectangle same as https://github.com/gtk-rs/gdk/blob/master/src/rectangle.rs

About new issue, it depend of you need of fast merging https://github.com/gtk-rs/pango/pull/54.
We can just move there and mark it as WIP.

I was wrong about constant name, SCALE is fine.

Ok. I will use this Rectangle implementations as a reference.

it depend of you need of fast merging
I'm not in rush. I'm using a local copy for now.

I'm fine with @RazrFalcon publishing the crate on his own, it's his code after all. We're just very glad to have his support for the gtk-rs project. ;)

@RazrFalcon, can you add link to your pangocairo git and close this Issue?

I do not publish it yet. I will do it later today.
But should it be closed? My crate will depend on git crates, and I can't release it on creates.io like this. Only when pango with my changes will be released to creates too.

Done: https://github.com/RazrFalcon/pangocairo-rs
Suggestions are welcome.
All the commented code in src/lib.rs can't be implemented now, because there are no such types in pango/cairo yet. But it's already supported all I need.

Done https://crates.io/crates/pangocairo !


Setting for TAB_CLI_EXE_MAC for default install with spaces in path

Neither escaping spaces nor quoted paths seem to work for TAB_CLI_EXE_MAC. What settings should be used for macOS?
$ ls -l /Applications/Tableau\ Desktop\ 2019.3.app/Contents/MacOS/tabquerytool
-rwxr-xr-x  1 root  wheel  22736 Aug 22 03:27 /Applications/Tableau Desktop 2019.3.app/Contents/MacOS/tabquerytool

$ ls -l "/Applications/Tableau Desktop 2019.3.app/Contents/MacOS/tabquerytool"
-rwxr-xr-x  1 root  wheel  22736 Aug 22 03:27 /Applications/Tableau Desktop 2019.3.app/Contents/MacOS/tabquerytool

TAB_CLI_EXE_MAC = /Applications/Tableau\ Desktop\ 2019.3.app/Contents/MacOS/tabquerytool

(tableau-sdk)$ python -m tdvt.tdvt --run pg --verbose

Checking generated logical setup files...
Could not find tabquery at [/Applications/Tableau\ Desktop\ 2019.3.app/Contents/MacOS/tabquerytool]
Could not find Tabquerycli.

TAB_CLI_EXE_MAC = "/Applications/Tableau Desktop 2019.3.app/Contents/MacOS/tabquerytool"

(tableau-sdk)$ python -m tdvt.tdvt --run pg --verbose

Reading tdvt ini file tabquerycli path is ["/Applications/Tableau Desktop 2019.3.app/Contents/MacOS/tabquerytool"]
Checking generated logical setup files...
Could not find tabquery at ["/Applications/Tableau Desktop 2019.3.app/Contents/MacOS/tabquerytool"]
Could not find Tabquerycli.

Hi, I think it will work with just the plain value, no escaping or quoting is needed, just type it in directly.

You might want to try changing the value in connector-plugin-sdk/config/tdvt/tdvt_override.ini.
The following works for me on Mac:
TAB_CLI_EXE_X64 = tabquerytool.exe
TAB_CLI_EXE_MAC = /Applications/Tableau Desktop 2019.2.app/Contents/MacOS/tabquerytool
```

Thanks - that worked for me.
TAB_CLI_EXE_MAC = /Applications/Tableau Desktop 2019.3.app/Contents/MacOS/tabquerytool


registryadmin scripts

Add start-sdk publish command which can potentially replace the Haskell implementation of registry-publish upload

closing in favor of #2426


Sync .github/workflows directory with azure-sdk-tools for PR 7098

Sync .github/workflows directory with azure-sdk-tools for PR https://github.com/Azure/azure-sdk-tools/pull/7098  See eng/common workflow

I'm going to override check enforcer since nothing will run for this change.

/check-enforcer override


Payload for sensors in group?

Trying to define a group called "Watch Movie" that should turn off lights and roll down the movie screen. What is supposed to be set as payload? I think "ON" and "OFF" are working for Switches but for Dimmers setting a value 0 to 100 gives errors. This is for RFLink sensors, I guessing that SET_LEVEL isn't implemented?

@Redferne for me it works as expected. I do not have RF_LINK sensors, I just checked with MySensors

I think "ON" and "OFF"

You have set 1 for ON and 0 for OFF.

This is for RFLink sensors, I guessing that SET_LEVEL isn't implemented?

RF_LINK, SET_LEVEL implemented as Percentage in MyController.

Oh. I figured out the problem. Sending SET_LEVEL=0 to a Dimmer does not equal to OFF. It is the lowest possible dim level. To turn off a Dimmer you need to send OFF.  So 0-100% needs to be mapped in 17 possible states. OFF = 0%, 0=6%, 1=12%, 2=19%.... 15=100%

@Redferne ok, I will remap this one.

@Redferne Updated on SNAPSHOT version.

Yes. Now I looks much better, now to the next issue 😄
I have defined a group of RFLink sensors (switches and dimmers) but cannot get them all to get to the correct state. There seems to be a synchronization problem on the serialization of the RFLink serial port. If I only have one or two sensors in the group, it works ok. But in the log below I have 4 and RFLink responds CMD UNKNOWN even when the command syntax is correct:
2017-01-07 11:03:57,807 DEBUG [Acme.Utils.ThreadPool(4)-PooledThread: Acme.Serve.Serve$ServeConnection@1fda3a2] [org.mycontroller.standalone.gateway.serialport.SerialPortMonitoringThread:135] RawMessage(gatewayId=35, data=10;NewKaku;03eb3168;1;OFF;\r\n, subData=null, isTxMessage=false, networkType=RF_LINK, timestamp=null) sent to Gateway[Name:RFLink, NetworkType:RF_LINK] 2017-01-07 11:03:57,847 DEBUG [Thread-36] [org.mycontroller.standalone.gateway.serialport.SerialDataListenerjSerialComm:64] Received a message:[20;F0;CMD UNKNOWN;\r] 2017-01-07 11:03:57,860 DEBUG [Acme.Utils.ThreadPool(4)-PooledThread: Acme.Serve.Serve$ServeConnection@1fda3a2] [org.mycontroller.standalone.gateway.serialport.SerialPortMonitoringThread:135] RawMessage(gatewayId=35, data=10;NewKaku;01876f42;1;OFF;\r\n, subData=null, isTxMessage=false, networkType=RF_LINK, timestamp=null) sent to Gateway[Name:RFLink, NetworkType:RF_LINK] org.mycontroller.standalone.message.RawMessageException: NodeEui can]not be NULL. Message:[20;F0;CMD UNKNOWN; 2017-01-07 11:03:57,941 DEBUG [Acme.Utils.ThreadPool(4)-PooledThread: Acme.Serve.Serve$ServeConnection@1fda3a2] [org.mycontroller.standalone.gateway.serialport.SerialPortMonitoringThread:135] RawMessage(gatewayId=35, data=10;NewKaku;0205c46e;1;OFF;\r\n, subData=null, isTxMessage=false, networkType=RF_LINK, timestamp=null) sent to Gateway[Name:RFLink, NetworkType:RF_LINK] 2017-01-07 11:03:57,993 DEBUG [Acme.Utils.ThreadPool(4)-PooledThread: Acme.Serve.Serve$ServeConnection@1fda3a2] [org.mycontroller.standalone.gateway.serialport.SerialPortMonitoringThread:135] RawMessage(gatewayId=35, data=10;NewKaku;0362c6c8;1;OFF;\r\n, subData=null, isTxMessage=false, networkType=RF_LINK, timestamp=null) sent to Gateway[Name:RFLink, NetworkType:RF_LINK] 2017-01-07 11:03:58,768 DEBUG [Thread-36] [org.mycontroller.standalone.gateway.serialport.SerialDataListenerjSerialComm:64] Received a message:[20;F1;OK;\r] 2017-01-07 11:03:59,669 DEBUG [Thread-36] [org.mycontroller.standalone.gateway.serialport.SerialDataListenerjSerialComm:64] Received a message:[20;F2;OK;\r] 2017-01-07 11:04:00,565 DEBUG [Thread-36] [org.mycontroller.standalone.gateway.serialport.SerialDataListenerjSerialComm:64] Received a message:[20;F3;OK;\r]
I'm not sure that it's possible to use multiple TX threads to RFLink, my understanding is that you need to wait for the response (OK/CMD UNKOWN) before sending next. I tried experimenting with the "TX message processing delay" but it didn't seem to work for RFLink?
Thanks!

@Redferne Let me address this soon.

@Redferne I guess this issue was addressed with Tx delay time


Hashmap breaks when minified

I've noticed that when hashmap is minified, this.type doesn't work correctly. The reason is the automatic chaining which does the followin check:
if (fn.toString().indexOf('return ') === -1) {
    proto[method] = chain(fn);
}

In minified code, it seems type is transformed into this:
type: function(t) {var e = Object.prototype.toString.call(t), n = e.slice(8, -1).toLowerCase();return"domwindow" !== n || t ? n : t + ""}
Because there's no space between return and "domwindow", the check causes type to be chained, returning this instead of the correct return value.
I propose that you explicitly list the names of the methods you want to chain instead of doing checks based on the structure of the function. But why did you have the space after return in that string to check anyway?

This update should fix the problem: https://github.com/flesler/hashmap/pull/24

I saw the PR before this, I think we just have the PR.
Anyway, we could also change return  to return. But in all truth, while I like the nifty auto-detection, it's simply less code to just hardcode the names.
I'll comment on the PR about changes


KAFKA-7240: -total metrics in Streams are incorrect

Changes:

Add org.apache.kafka.streams.processor.internals.metrics.CumulativeCount analogous to Count, but not a SampledStat
Use CumulativeCount for -total metrics in streams instead of Count

Testing strategy:

Add a test in StreamsMetricsImplTest which fails on old, incorrect behavior

The contribution is my original work and I license the work to the project under the project's open source license.
Committer Checklist (excluded from commit message)

[x] Verify design and implementation
[x] Verify test coverage and CI build status
[x] Verify documentation (including upgrade notes)

@vvcephei @guozhangwang @mjsax for review

While working on this PR, I noticed a couple of other streams metrics related code that may be incorrect. I could include fixes but that may be out of the scope of this PR. If there's a better place for this discussion like the mailing list or a new JIRA ticket let me know.

I believe this call to streamsMetrics.processTimeSensor.record is redudant, because process time is already measured in each iteration of the loop in processAndMaybeCommit. I don't fully understand what's going on here though, so I may have missed something.

In StreamsMetricsThreadImpl, commitTimeSensor, processTimeSensor, and punctuateTimeSensor latency and rate/total metrics are not consistent with each other.
Each call to record for those sensors is of the form commitTimeSensor.record(computeLatency() / (double) committed, timerStartedMs), i.e. the value being recorded is the average latency per task commited in this example. But the rate/total metrics will be measuring one event per call to commitTimeSensor.record, when it should probably be measuring committed events per call. One fix would be to move the -rate and -total metrics to a different sensor just for throughput, and call record like commitThroughputSensor.record(committed, timerStartedMs).
The other sensors in StreamsMetricsThreadImpl are fine, they are all called like sensor.record(computeLatency(), timerStartedMs).

retest this please

Regarding the other issues you pointed out, do you mind making separate PRs? As far as I'm concerned, they don't need Jira issues filed, you can just entitle the PR "MINOR: blah blah blah". I'm just thinking that if you also add tests for those metrics to perform properly (which we should have had all along :/ ), then it'll be a little hard for reviewers to follow 3 different threads of change.
For #1, yeah, that looks redundant, and in fact, the commitTimeSensor also looks to be redundantly recorded in processAndMaybeCommit and maybeCommit.
For #2, I believe you are also correct, and I think your proposed solution is appropriate.

The test failure is unrelated.
It's possible to re-run them by saying this in a comment:
Retest this, please.

Oh, and I forgot to say thank you for your bug report and fix!

@vvcephei My original changes were made before last Friday, but I had to rebase and fix some conflicts after #5450 was merged. I see more was changed in that PR that I originally realized, for example I see the new method addInvocationRateAndCount is called in many places.
What are the implications of my change to that method (using CumulativeCount)? Is it still correct for state store metrics?

@lendle I'll check out your PR and dig through the usages to double check, but it looks right to me.
I think we can depend on the following invariant: Count should be used only inside of a Rate and CumulativeCount should be used only outside of a Rate. As long as this invariant holds, we should be good.
Also, as long as all callers to addInvocationRateAndCount actually want an invocation rate and count, I think we're also ok.

@vvcephei Yes that was my concern. Thanks for confirming.

Hey @lendle, Just wanted to check in. Once you address @bbejeck's comment, we can merge this PR.

Sorry, just saw this. I have limited access to the internet for the next
few days, I can finish this early next week.
On Mon, Aug 13, 2018, 10:09 AM John Roesler<EMAIL_ADDRESS>wrote:

Hey @lendle https://github.com/lendle, Just wanted to check in. Once
you address @bbejeck https://github.com/bbejeck's comment, we can merge
this PR.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/kafka/pull/5467#issuecomment-412552014, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAxxKyx4rl47fy7GKId1QVf95llqz9zAks5uQZaigaJpZM4VxLhs
.

Oh, also it looks like this now conflicts with trunk. You'll have to rebase when you get back to it.

@mjsax This PR is basically ready to go, except for:

needs to be rebased
still needs to address Bill's nitpick (or not, since he said he can go either way on it)

retest this please

@guozhangwang / @mjsax ,
I believe this PR is ready to merge.


secret search broken

secret search loops through all files in all projects, but data structure for results is only made to store results of one project. either change the loop or do the proper way and do project storage. latter is preferred so results don't always have to be searched on each project switch (but needs more logic to stay in sync!)
alternatively switch to a manual search mode - search could start taking a while on many regexes and big projects

still not optimal, but should at least work now as expected


Unit test failing after big merge

I just merged [this big PR](https://github.com/oceanprotocol/df-py/pull/166.
Unit tests were passing locally. But they failed remotely.
Aim of this issue: get all tests passing remotely.

Could you try to pull latest barge and run the unit tests locally? New version of barge returns an empty list of approved tokens.

I just checked, I have the latest version of barge


Deep sleep entry: UART flush does not have a timeout (IDFGH-1331)

Environment

Module or chip used:  ESP32-PICO-D4
IDF version: v4.0-dev-771-gc9bf3a3b1

Problem Description
The flush_uarts() function within components/esp32/sleep_modes.c does not have any kind of timeout mechanism.  If you have a UART with flow control enabled, and the remote side is blocking UART traffic with flow control, the deep sleep code can hang indefinitely waiting for the UART to flush which will never flush.
Expected Behavior
That the deep sleep call would not hang indefinitely in this situation.  One of the following could be implemented to adhere to this:

Device times out waiting for the UART to flush and continues into deep sleep
Device times out waiting for the UART to flush and aborts entering deep sleep
Entering deep sleep does not attempt to flush the UART state and leaves any of these device consistency questions to the user of the SDK.

Actual Behavior
Device loops forever waiting for the UART to flush.
Steps to reproduce

Setup a UART with flow control
Assert the FLOW control line into the UART module (you could probably do this via the mux by attaching the input flow control line inside the UART to a mux 1 line)
Send some data into the UART while the flow control line is asserted
Invoke deep sleep

Discussion on methods for fixing
In our code I've mitigated this by turning off hardware flow control on all of the UARTs before entering deep sleep to ensure it doesn't get hung-up due to the other end of the UART, but it begs the higher level discussion on what the correct behavior in this case should be.  I'm not sure which is better (between the three options I posed in the expected behavior above), and I'm not sure what an appropriate timeout would be.  It feels like these are use-case dependent, so it feels to me that it'd be better to leave any flushing to the SDK user instead of assuming that the SDK user wants these flushed, however, that'd break expectations folks already have with the current SDK (and potentially several examples).
(A warning could be printed in the deep sleep entry code that there is pending UART data if the SDK didn't try to flush the data, but this, too, would need to be flushed so it's a catch-22.)
Related issues

#2145, commit 10c2cddddf329dff8fbe109e14fd00501d18e0c2

Hi, @tim-nordell-nimbelink
Can you help test the following fix? you need to change the API in esp32/rom/uart.h
static inline void IRAM_ATTR uart_tx_wait_idle(uint8_t uart_no) {
    uint32_t status;
    uint32_t flow_ctrl_mask = READ_PERI_REG(UART_CONF0_REG(uart_no)) & UART_TX_FLOW_EN_M;
    CLEAR_PERI_REG_MASK(UART_CONF0_REG(uart_no), flow_ctrl_mask);
    SET_PERI_REG_MASK(UART_FLOW_CONF_REG(uart_no), UART_FORCE_XON_M);
    do {
        status = READ_PERI_REG(UART_STATUS_REG(uart_no));
        /* either tx count or state is non-zero */
    } while ((status & (UART_ST_UTX_OUT_M | UART_TXFIFO_CNT_M)) != 0);
    SET_PERI_REG_MASK(UART_CONF0_REG(uart_no), flow_ctrl_mask);
}

If this fix is OK, we will apply it in the IDF release.
thanks !!

Hi @koobest -
I don't think universally disabling flow control is a good solution.  It happens to be okay for our product, but I don't know if it'd be okay for all folks.  It also wouldn't be appropriate to modify this ROM function to disable flow control - there are uses of it in the current SDK that occur before shutting down the main CPU and it would introduce a bug into the codebase inadvertently turning off flow control onto folks who are already using the function and expect flow control to remain enabled.
Here's a sample that I had coded with an eye toward a pull request, but I wasn't sure what approach would be acceptable to espressif (which is why I had opened a ticket instead of providing a patch):
static void IRAM_ATTR flush_uarts()
{
    int idle_status = 0;

    // Loop for up to ~100ms waiting for all UARTs to flush.
    for (int i = 0; i < 100; ++i)
    {
        // Update idle status for each UART
        for (int j = 0; j < UART_NUM_MAX; ++j)
        {
            if (!(idle_status & (1 << j)))
            {
                uint32_t status = READ_PERI_REG(UART_STATUS_REG(j));
                status &= ((UART_ST_UTX_OUT_M | UART_TXFIFO_CNT_M));
                idle_status |= (status == 0) << j;
            }
        }

        // If all 3 UARTs are idle, break
        if (idle_status == ((1 << UART_NUM_MAX) - 1))
        {
            break;
        }

        // Otherwise wait a little bit.
        ets_delay_us(1000);
    }
}

That approach would work with a maximum timeout of ~100ms.  However, I'm not convinced that would satisfy all use cases, and certainly it wouldn't address current users of uart_tx_wait_idle(...) - it looks like there are 16 invocations in the core codebase (and it looks like many assume that CONFIG_CONSOLE_UART_NUM is valid which it isn't if CONFIG_CONSOLE_UART_NONE=y).  Another option would be to add a second function which accepts a timeout parameter similar to uart_tx_wait_idle(...).  However... "esp/rom/uart.h" seems like it's supposed to be only for replacing functions that are in the ROM code that are broken, not for introducing new ones...
-- Tim

I think this can be solved in a combination of two changes:

On entry to deep sleep, don't wait for UART FIFO to be empty if flow control is enabled and UART state is not "transmitting".
The behavior on entering deep sleep can be made run-time configurable. The default would be the same as now, to wait for UART FIFO to be empty (taking into account pt. 1). User could request to not wait for UART FIFO by calling a new function prior to entering deep sleep (something like esp_sleep_wait_for_uart(ESP_SLEEP_NOWAIT);). This will also address similar feature requests for faster entry into deep sleep, which we had before.

Hi @igrr, @koobest  -
I think it'd be okay to check if the UART state is "not transmitting", but I wasn't sure if the UART could go into that state between characters while it was running.  If one is doing the UART idle state check (and it doesn't get into TX_IDLE regardless of the configuration of the UART such as someone setting UART_TX_IDLE_NUM to non-zero), then you probably wouldn't need to check to see if flow control is enabled and it could be a universal check across all users.  The TRM wasn't clear on when those state transitions occur from that state register.
If the NOWAIT option isn't enabled by the user, I think it'd make sense to emit a warning on the console UART that it didn't flush UART X.  Not all users will have a debugger attached to pinpoint where an issue is.
I think the esp_restart_noos() function probably should be changed, too, since it could run into the same issue with flow control upon someone issuing a restart call to the system.  I think the other usages of uart_tx_wait_idle(...) are okay in the system but I haven't exhaustively checked.
-- Tim

Hi @igrr, @koobest -
I did some testing with the following code:
    auto p = uart_port_t::UART_NUM_1;
    uart_set_hw_flow_ctrl(p, uart_hw_flowcontrol_t::UART_HW_FLOWCTRL_DISABLE, 0);
    uart_set_baudrate(p, 2400);
    uart_write_bytes(p, "123\r\n", 5);

    uint8_t transitions[32];
    uint8_t last_mode = 0;

    int spin = 100000;
    int i;
    for(i=0;i<sizeof(transitions);)
    {
        uint32_t new_mode = (READ_PERI_REG(UART_STATUS_REG(p)) & UART_ST_UTX_OUT_M) >> UART_ST_UTX_OUT_S;
        if(new_mode != last_mode)
        {
            last_mode = transitions[i++] = new_mode;
            spin = 100000;
        } else {
            if(--spin == 0)
            {
                break;
            }
        }
    }

    printf("%i transitions; current mode: %i\n", i,
                (READ_PERI_REG(UART_STATUS_REG(p)) & UART_ST_UTX_OUT_M) >> UART_ST_UTX_OUT_S);
    for(int j=0;j<i;++j)
    {
        printf("Transition[%i] = %i\n", j, transitions[j]);
    }

and I get something like the following for output:
32 transitions; current mode: 3
Transition[0] = 1
Transition[1] = 3
Transition[2] = 4
Transition[3] = 5
Transition[4] = 6
Transition[5] = 7
Transition[6] = 8
Transition[7] = 9
Transition[8] = 11
Transition[9] = 1
Transition[10] = 2
Transition[11] = 3
Transition[12] = 4
Transition[13] = 5
Transition[14] = 6
Transition[15] = 7
Transition[16] = 8
Transition[17] = 9
Transition[18] = 11
Transition[19] = 1
Transition[20] = 2
Transition[21] = 3
Transition[22] = 4
Transition[23] = 5
Transition[24] = 6
Transition[25] = 7
Transition[26] = 8
Transition[27] = 9
Transition[28] = 11
Transition[29] = 1
Transition[30] = 2
Transition[31] = 3

If flow control is enabled (and blocked continuously by the other side), I get:
0 transitions; current mode: 0

So, for the case of flushing all data possible without waiting on flow control, it seems like we could modify the check to only validate that the serial mode is 0.  E.g. something like:
static void IRAM_ATTR flush_uarts()
{
    for (int i = 0; i < 3; ++i) {
        while (REG_GET_FIELD(UART_STATUS_REG(i), UART_ST_UTX_OUT) != 0) {
            ;
        }
}

which would look very similar to the suspend_uarts() function, but lacking the set bit call.
I still don't think it's a great solution for the general use-case (I think a good faith attempt with a timeout is better), but it's not bad.  Some uC's that might be companions to the ESP32 only have 1-byte FIFOs (for instance, any in the STM32L4 series) and would hit the idle case even while it's flushing.
(The STM32L4 series will assert flow control briefly after every character received, so this patch would be incompatible with that series for instance, but the example I gave earlier with a timeout wouldn't be incompatible with that series of microcontrollers.  Here's the diagram from the STM32L476 datasheet as an example where a purely idle state detection would be broken on the ESP32 side:

I don't know how many other microcontrollers behave this way that someone might choose to connect to the ESP32.  We do have a product where the ESP32 is hooked up to a STM32L476 with flow control lines.  I would hit this issue on that product though if it is implemented in the proposed manner, but not with the current implementation.)
-- Tim

Since uart_tx_wait_idle(uint8_t uart_no) has parameters uart_no, can we use bitmaps to specify whether flow control needs to be turned off? @igrr
bit 6~bit0 -> uart_number,
bit 7 -> controls whether to close the flow control.


Include database in health checks

We should not return ready until we have established communication with the database. This would only apply to the Cassandra backend currently.

This is kind of done. The service isn't ready until database access either succeeds or times out. I don't know that failing to start just because the database isn't ready is the right solution.


Messages sent don't show up

Description
I followed readme instructions and the demo didn't work. I can send messages in the bottom text field but nothing shows up in either client's incoming messages area. I want feedback about what's happening and help to debug.
Screenshot

First terminal:
init() using default .env config init() connecting to existing topic MirrorConsensusTopicQuery() 0.0.28583 TopicMessageSubmitTransaction() {"operatorAccount":"0.0.47502376","client":"0nBjh_DeGHjpoLVSAAAA","message":"Hola"} TopicMessageSubmitTransaction() {"operatorAccount":"0.0.47502376","client":"0nBjh_DeGHjpoLVSAAAA","message":"Hello?"}
Second terminal:
init() using default .env config init() connecting to existing topic MirrorConsensusTopicQuery() 0.0.28583 TopicMessageSubmitTransaction() {"operatorAccount":"0.0.47502376","client":"bbNExYrAOZzjmcuXAAAA","message":"xyz"}
Steps to reproduce
Follow instructions
Run two instances in two term windows
Try to send messages
Receive nothing in either browser window
Additional context
No response
Hedera network
testnet
Version
7/4/22 master clone
Operating system
No response

hey @mowliv -- just set this up and it worked just fine. I was able to repro your issue only when not ever initiating a new topic.
ie the first time you run it, you do need to run "create new topic" rather than simply connecting to it. Could you try that?

Hi Helena - thanks for your message. Yes, if I create a new topic in one term and then reference that in the other terminal, it works! That's fun. But if I choose to connect to existing topic, and I hit enter to default to .env file (which shows TOPIC_ID=0.0.28583) then it fails. Perhaps instructions could be improved.


gtdb taxonomy build

Expected Behavior
map queries to GTDB then get taxonomy lca of best hits
Current Behavior
-all taxonomy results are "unclassified" when mapping to GTDB
-the gtdb_mapping file has only a single column (0..256163114) which may be the problem
-I think the gtdb.lookup table may be wrong.
Steps to Reproduce (for bugs)
-download gtdb.tar.gz
mmseqs createdb tardb gtdb
-execute shell commands in databases.sh to create nodes.dmp, names.dmp, mapping_genomes (these files all seem to be fine)
mmseqs createtaxdb gtdb tmp/taxdb --ncbi-tax-dump tmp/taxonomy --tax-mapping-file "tmp/taxonomy/mapping_genomes" --tax-mapping-mode 1
MMseqs Output (for bugs)
-no errors or warnings
createtaxdb gtdb tmp/taxdb --ncbi-tax-dump tmp/taxonomy --tax-mapping-file tmp/taxonomy/mapping_genomes --tax-mapping-mode 1

MMseqs Version:        	14.7e284
NCBI tax dump directory	tmp/taxonomy
Taxonomy mapping file  	tmp/taxonomy/mapping_genomes
Taxonomy mapping mode  	1
Taxonomy db mode       	1
Threads                	28
Verbosity              	3

if I understand correctly, the gtdb.lookup file should have 3 columns: proteinNumber (0.. 256163114), proteinID, sourceGenomeNumber :
0       NZ_CP009512.1_1 1
1       NZ_CP009512.1_33        1
2       NZ_CP009512.1_65        1
3       NZ_CP009512.1_97        1
4       NZ_CP009512.1_129       1
5       NZ_CP009512.1_161       1
6       NZ_CP009512.1_193       1
7       NZ_CP009512.1_225       1
8       NZ_CP009512.1_257       1
9       NZ_CP009512.1_289       1
10      NZ_CP009512.1_321       1

however, my gtdb.lookup file has entries from different genomes pointing to the same source genome number. e.g.
0       NZ_CP009512.1_1 1
100     NZ_CP009512.1_3201      1
101     NZ_CP009512.1_3233      1
102180795       NZ_JAAOIZ010000001.1_5  1  #should point to source genome 65537
102180796       NZ_JAAOIZ010000001.1_37 1  #should point to source genome 65537
102181002       JAJXZV010000318.1_2     2  #should point to source genome 65538
102181003       JAJXZV010000021.1_4     2  #should point to source genome 65538
104066383       NC_007181.1_8   2
104066384       NC_007181.1_40  2

the highest source genome number found in the gtdb.lookup file is 65535, and then it seems to start over at 1 duplicating ids...?
Your Environment
MMseqs2 Version: 14.7e284
tried with both latest binary download and conda install
Linux my.local 3.10.0-1160.76.1.el7.x86_64 #1 SMP Wed Aug 10 16:21:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

hmm, I just noticed that the --max-seq-len search parameter is 65535...

fyi in case anyone else is having this problem... I was able to successfully run the GTDB taxonomy after manually re-making the gtdb.lookup file and a gtdb.taxid.mapping file, and running createtaxdb with this taxid mapping file.

@mlcoleman
I have the same problem after executing the shell commands in download.sh manually.
For the gtdb.lookup file, did you just fix the third column to not start at 1 again after 65535? How did you create and use the gtdb.taxid.mapping file (do you mean adding the tax-ids to gtdb_mapping which has only 1 column now?):
I would make a protein-id vs genome table and then (fasta headers from protein_faa_reps) and merge with the proteins IDs in the new gtdb.lookup and the tax-ids in tmp/taxonomy/mapping_genomes

I encountered the same issue while working with mmseqs version 14-7e284 and GTDB r214, using a Linux server (Ubuntu 22.04 LTS; CPU with SSE/AVX2 support and 512 GB RAM). Although there is a lot of overlap with the post by @mlcoleman previously and the discussion by @mherold1 , I include some details below about the problems I ran into and how I addressed them.
Problem description
After making a seqdb for the GTDB, there seems to be a collision of genome IDs between different genomes in the seqdb's gdtb.lookup file. As mentioned by @mlcoleman , these ID collisions seem to disrupt mmseqs taxonomy workflows downstream.
Steps to reproduce

Download gtdb.tar.gz as done in the data/workflow/databases.sh script (or you can just run: wget -o gdtb.tar.gz https://data.ace.uq.edu.au/public/gtdb/data/releases/latest/genomic_files_reps/gtdb_proteins_aa_reps.tar.gz)
Import gtdb.tar.gz as a tardb via mmseqs tar2db gtdb.tar.gz tardb --tar-include 'faa.gz$'
Run mmseqs createdb tardb gtdb

mmseqs createdb finishes without error, but the resulting gtdb.lookup file seems to assign the wrong genome IDs (resulting in ID collisions) for any genomes with an ID over 65535. Meanwhile, the correct IDs seem to be assigned in gtdb.source.
For example:
Last few lines of gtdb.source (tail gtdb.source):
85195   RS_GCF_014201515.1
85196   RS_GCF_003977665.1
85197   RS_GCF_007922615.2
85198   GB_GCA_002389765.1
85199   RS_GCF_001440475.1
85200   GB_GCA_015059865.1
85201   GB_GCA_900546895.1
85202   GB_GCA_017621015.1
85203   GB_GCA_015486425.1
85204   GB_GCA_934565415.1

It seems like the left column is the genome ID, and the right column is the genome name in the GTDB. Everything seems OK with this file.
Last few lines of gtdb.lookup (tail gtdb.lookup)
256163105       CAKTHQ010000047.1_16    19668
256163106       CAKTHQ010000050.1_5     19668
256163107       CAKTHQ010000052.1_8     19668
256163108       CAKTHQ010000056.1_1     19668
256163109       CAKTHQ010000058.1_10    19668
256163110       CAKTHQ010000061.1_12    19668
256163111       CAKTHQ010000065.1_4     19668
256163112       CAKTHQ010000070.1_6     19668
256163113       CAKTHQ010000075.1_3     19668
256163114       CAKTHQ010000082.1_2     19668

It seems like the left column is the ORF ID, the middle column is the contig name, and the right column is the genome ID.
This file seems problematic. I confirmed (via the NCBI website) that contig CAKTHQ010000082.1 (from which the ORFs were derived in gtdb.lookup above) comes from assembly GCA_934565415.1. This assembly is shown on the last line of gtdb.source above and has an ID of 85204 in gtdb.source. However, in gtdb.lookup, ORFs on the contig seem to have the erroneous genome ID of 19668, which matches another genome in gtdb.source. I assume that the numbers on the right side of the gtdb.lookup shown above should be 85204.
Although not shown above, the genome ID counter seems to re-start in gtdb.lookup (at 0) after 65535. Note, for example, that 19668 + 1 + 65535 = 85204.
Potential causes
The issue seems to have something to do with the mmseqs createdb step, but I have not looked into the code for this command. As noted by @mlcoleman , it's interesting that 65535 is the same as the --max-seq-len param.
Consequences
If the default GTDB seqdb is used to make a taxdb (without correcting the gtdb.lookup file), then taxonomy results become ambiguous or incorrect. For example, here are the taxonomies of some contigs run through mmseqs easy-taxonomy with the default GTDB seqtaxdb:
contig_1        2       superkingdom    Bacteria        77      69      59      0.850   d_Bacteria
contig_2        2       superkingdom    Bacteria        66      61      54      0.890   d_Bacteria
contig_3        2       superkingdom    Bacteria        54      47      45      0.960   d_Bacteria
contig_4        2       superkingdom    Bacteria        53      51      47      0.920   d_Bacteria
contig_5        2       superkingdom    Bacteria        52      48      44      0.910   d_Bacteria
contig_6        2       superkingdom    Bacteria        57      50      46      0.930   d_Bacteria
contig_7        2       superkingdom    Bacteria        59      54      52      0.980   d_Bacteria

If the gtdb.lookup file is corrected (as described under "Possible workaround" below), the classifications match what I would expect for the input contigs. (Sorry, I don't show the corrected classification data for these contigs here because I am working with unpublished data at the moment.)
Possible workaround
As discussed by @mlcoleman , after running mmseqs createdb tardb gtdb, I was able to fix the gtdb.lookup file so that the genome IDs in this file corresponded to the genome IDs used in gtdb.source. (Unlike @mherold1 , I set the genome IDs to start from 0, because this was the format used in gtdb.source.)
To map the genome IDs in gtdb.source to those in gtdb.lookup, I first made a table mapping each ORF ID to each genome ID via the gtdb.tar.gz tarball, as follows:
Create the following bash script, get_orf_names.sh:
#!/usr/bin/env bash
# get_orf_names.sh
# Helper script to get ORF names from a gtdb_proteins_aa_reps.tar.gz file
# This script expects STDIN to be a gzipped FAA file for one genome; $1 should be the filename or path of the FAA file
# Output is a tab-separated table (no header) with two columns: genome name and ORF name
set -euo pipefail

tar_filename=$1

genome_name="${tar_filename##*/}"
genome_name="${genome_name%_protein.faa.gz}"

#zcat - | \
pigz -cd -p 8 - | \
  awk -v gname="${genome_name}" \
    '{ if ($0 ~ /^>/) { gsub("^>", ""); print gname "\t" $1 }}'

Give this exec permissions: chmod 755 get_orf_names.sh.
Then run:
tar -xzf gtdb.tar.gz \
    --wildcards "*_protein.faa.gz" \
    --to-command './get_orf_names.sh $TAR_FILENAME' \
  > genome_orf_mapping.tsv

Format of the resulting file (head genome_orf_mapping.tsv):
RS_GCF_000970205.1      NZ_CP009512.1_1
RS_GCF_000970205.1      NZ_CP009512.1_2
RS_GCF_000970205.1      NZ_CP009512.1_3
RS_GCF_000970205.1      NZ_CP009512.1_4
RS_GCF_000970205.1      NZ_CP009512.1_5
RS_GCF_000970205.1      NZ_CP009512.1_6
RS_GCF_000970205.1      NZ_CP009512.1_7
RS_GCF_000970205.1      NZ_CP009512.1_8
RS_GCF_000970205.1      NZ_CP009512.1_9
RS_GCF_000970205.1      NZ_CP009512.1_10

You can then do a JOIN operation between the gtdb.lookup table, the genome_orf_mapping.tsv table, and the gtdb.source table to link the genome IDs in gtdb.source to gtdb.lookup. Save an output file with the same format as gdtb.lookup but with the genome IDs from gtdb.source. Use this new file to replace the old gtdb.lookup. Here is some rough python/pandas code to do this (note this is slow and uses ~100 GB of RAM):
import pandas as pd

lookup_data = pd.read_csv('gtdb.lookup', sep='\t', header=None)
lookup_data.columns = ['gene-id','gene-name','genome-id']

source_data = pd.read_csv('gtdb.source', sep='\t', header=None)
source_data.columns = ['genome-id','genome-name']

genome_orf_mapping = pd.read_csv('genome_orf_mapping.tsv', sep='\t', header=None)
genome_orf_mapping.columns = ['genome-name','gene-name']

# Perform the JOIN
lookup_data_corrected = lookup_data.rename(columns={'genome-id':'genome-id-old'})\
  .merge(genome_orf_mapping, on='gene-name', how='left', validate='m:1')\
  .merge(source_data, on='genome-name', how='left', validate='m:1')

lookup_data_gene_corrected.drop(columns=['genome-id-old','genome-name'])\
  .to_csv('gtdb.lookup.corrected', sep='\t', header=None, index=False)
# Replace gtdb.lookup with this newly saved file

After making this fix, I could run mmseqs createtaxdb and get a taxdb that seemed to work fine with the mmseqs easy-taxonomy pipeline, matching my expected results (as mentioned under "Consequences" above). Maybe because I corrected gdtb.lookup before making the taxdb, I didn't need to edit any other mapping files as mentioned by @mlcoleman .
I can't guarantee this workaround is 100% correct because I do not understand all the database files generated by mmseqs. However, the workaround seems to do the trick for my data for the time being. Hope this helps before a longer-term solution is ready.

@milot-mirdita Thanks for the quick response and for the data type fix! I tested the latest master branch, and the gtdb.lookup file seems to be correct now after running mmseqs databases gtdb results tmp. Taxonomy results look as expected when running mmseqs easy-taxonomy using this updated GTDB database. (I did not test the awk code you included, but thanks for writing this.)
I think it's OK to close this issue now. Thanks again.


Remove global polyfills

Currently library includes polyfill for built-ins like Promise, Symbol or Object.assign.
But according to the Vue CLI docs:

When using Vue CLI to build a library or Web Components, it is recommended to pass useBuiltIns: false to @vue/babel-preset-app to disable automatic polyfill injection. This ensures you don't include unnecessary polyfills in your code, as it should be the responsibility of the consuming app to include polyfills.

We can cut up to 20kb of JS from the bundle
TODO:

[x] PR
[x] Remove polyfills
[x] Describe needed polyfills in docs
[x] Release

Released as part of v1.0.0


"Skip to main content" link is showing up with tab navigation on Hamburger

As part of accessibility testing, when trying to click on the hamburger icon using the tab button "skip to main content" link is showing instead of the Hamburger icon in the browsers Chrome, Safari, Edge.
Please see the screenshot below.

This is intentionally the first thing that you can tab to. Pressing tab a second time will bring you to the hamburger menu. Closing as working as intended. Please feel free to reopen if you feel there is more to discuss.