Azure Salesforce Provisioning not syncing SF Profiles to AD Roles

We have Salesforce Provisioning set up in Azure.  When selecting a "Role" in Azure, only the standard SF Profiles are in the list to be selected.  The custom SF profiles are not in the list.
There is another thread that says that Azure doesn't consume SF Profiles, only SF Roles. That is not true. The Roles in Azure are 1:1 with the standard SF Profiles.  Unfortunately the custom SF profiles are not being synchronized.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 624be0a5-9ab6-ec59-e7a2-74585cb0571e
Version Independent ID: a00a09c8-f0c5-2bfe-75aa-6183346af405
Content: Tutorial: Configure Salesforce for automatic user provisioning with Azure Active Directory
Content Source: articles/active-directory/saas-apps/salesforce-provisioning-tutorial.md
Service: active-directory
GitHub Login: @jeevansd
Microsoft Alias: jeedes

Following. I am seeing the same issue - only standard, internal profiles are being imported.  Can provisioning not be used for external Salesforce users?

I am following this as well, We just have the same problem with a Salesforce Production is not taking custom Profiles however this is working on Salesforce  Sandbox. Is there any update on this? our Salesforce Provisioning is stuck

I'm having the same issue using Remedyforce. any updates on this?

@jeevansd: in post https://github.com/MicrosoftDocs/azure-docs/issues/13103 you mention the custom profiles are being read when the provisioning is enabled. What if there are new profiles being created in Salesforce after  enabling the provisioning?
We are having this issue where we've been using the provisioning for a while now and have now created a few new Salesforce profiles. However, when going back to Azure to be able to assign Groups to these Profiles, we cannot find these newly created Salesforce profiles.

Having same issue in a production environment. @jeevansd was this issue ever resolved or workaround provided in order to have newly created salesforce profiles appear in Azure?

Never resolved. I’m still having a problem.
Dexter Jackson
VP, Information Technology Services
RXR Realty
625 RXR Plaza
Uniondale, New York 11556
T:<PHONE_NUMBER>
<EMAIL_ADDRESS>www.rxrrealty.com
Confidentiality Notice: The information contained in this e-mail and any attachments may be legally privileged and confidential. If you are not an intended recipient, you should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person.
From: juliancasablanca<EMAIL_ADDRESS>Sent: Monday, March 4, 2019 4:40 PM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: Jackson, Dexter<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] Azure Salesforce Provisioning not syncing SF Profiles to AD Roles (#22223)
Having same issue in a production environment. @jeevansdhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_jeevansd&d=DwMCaQ&c=rGIfYmk-1z7cfZPhPNJ2NA&r=xFXf11WonGCmJqods-1nHd6KZb07-uu4YYY6mxqhn1k&m=ttRMGiYQI03MCPiueNEubcUd8He44iTvNGMjIQ8nnNs&s=6nEJW8sqymd4E0ERKxSU8MciXEWOts5JbKb2mmiSh98&e= was this issue ever resolved or workaround provided in order to have newly created salesforce profiles appear in Azure?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_MicrosoftDocs_azure-2Ddocs_issues_22223-23issuecomment-2D469432137&d=DwMCaQ&c=rGIfYmk-1z7cfZPhPNJ2NA&r=xFXf11WonGCmJqods-1nHd6KZb07-uu4YYY6mxqhn1k&m=ttRMGiYQI03MCPiueNEubcUd8He44iTvNGMjIQ8nnNs&s=R-OBf-vU5XjySgcb-tew9GsvW1hi2tyHDkJK8UiiJDo&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AtaLTfCSYF5LqPdVR0deTiIdSCQSpdwEks5vTZKagaJpZM4Z8Wt-5F&d=DwMCaQ&c=rGIfYmk-1z7cfZPhPNJ2NA&r=xFXf11WonGCmJqods-1nHd6KZb07-uu4YYY6mxqhn1k&m=ttRMGiYQI03MCPiueNEubcUd8He44iTvNGMjIQ8nnNs&s=EJGpLfq3smPwUIHYlAxGjhjGMb0YEbc2AunwWADDAbw&e=.

A workaround I could think of is to recreate a whole new App in Azure. However it would require significant rework to redo the whole mapping again, point the users to this new app and then restart the user provisioning again.
Seems like a lot of effort for something Microsoft should be fixing.

We already had all profiles in place when we created the app in Azure. I
think something must have broken recently if it used to work.
On Mon, Mar 4, 2019, 6:00 PM fabienturcotte<EMAIL_ADDRESS>wrote:

A workaround I could think of is to recreate a whole new App in Azure.
However it would require significant rework to redo the whole mapping
again, point the users to this new app and then restart the user
provisioning again.
Seems like a lot of effort for something Microsoft should be fixing.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/azure-docs/issues/22223#issuecomment-469457083,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABfSamSQxtus4_E5cvWcJDIcVuqECf8Eks5vTaWogaJpZM4Z8Wt_
.

@henslecd  and @sweetcaoz We are workign with our engineering team on this to get the details on this issue.

As of today this is the synchronization details ( Enterprise applications - All applicationshttps://aad.portal.azure.com/ SalesForce– Provisioning)
Summary
We have synchronized 705 object(s) of type User to User.
Synchronization was last run on Mon Apr 01 2019 17:07:51 GMT-0400 (Eastern Daylight Time)
Most recent full synchronization was completed Mon Apr 01 2019 17:07:51 GMT-0400 (Eastern Daylight Time)
We completed the first full synchronization on Thu Feb 14 2019 18:08:26 GMT-0500 (Eastern Standard Time)
Errors
Which is correct as to the amount of AD users, but for some reason not all of the users are showing in SF which leads me to a problem syncing.
Please advise.
Dexter Jackson
VP, Information Technology Services
RXR Realty
625 RXR Plaza
Uniondale, New York 11556
T:<PHONE_NUMBER>
<EMAIL_ADDRESS>www.rxrrealty.com
Confidentiality Notice: The information contained in this e-mail and any attachments may be legally privileged and confidential. If you are not an intended recipient, you should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person.
From: Jeevan Desarda<EMAIL_ADDRESS>Sent: Monday, April 1, 2019 6:40 PM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: Jackson, Dexter<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] Azure Salesforce Provisioning not syncing SF Profiles to AD Roles (#22223)
@henslecdhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_henslecd&d=DwMCaQ&c=rGIfYmk-1z7cfZPhPNJ2NA&r=xFXf11WonGCmJqods-1nHd6KZb07-uu4YYY6mxqhn1k&m=4IKTLZhW4ejOBCxEKaKQPmgsWx8LmPQAOXC0cxyrm6I&s=yedDcZTwgCHXoCNa1ByRBlU6YQJtNzBuXHCkPN2zmgA&e= and @sweetcaozhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_sweetcaoz&d=DwMCaQ&c=rGIfYmk-1z7cfZPhPNJ2NA&r=xFXf11WonGCmJqods-1nHd6KZb07-uu4YYY6mxqhn1k&m=4IKTLZhW4ejOBCxEKaKQPmgsWx8LmPQAOXC0cxyrm6I&s=ML8eIqOknRHaoGMLxUhKIWPMp7-ZnVRN3D5pdCTeBI8&e= We are workign with our engineering team on this to get the details on this issue.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_MicrosoftDocs_azure-2Ddocs_issues_22223-23issuecomment-2D478772954&d=DwMCaQ&c=rGIfYmk-1z7cfZPhPNJ2NA&r=xFXf11WonGCmJqods-1nHd6KZb07-uu4YYY6mxqhn1k&m=4IKTLZhW4ejOBCxEKaKQPmgsWx8LmPQAOXC0cxyrm6I&s=_NR0gVP-EtmuOHroaSF128CNcm1nOu5tuyVPUllwP2E&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AtaLTULOv50mPHhMQ8u46IerD5k-2DWphqks5vcoq9gaJpZM4Z8Wt-5F&d=DwMCaQ&c=rGIfYmk-1z7cfZPhPNJ2NA&r=xFXf11WonGCmJqods-1nHd6KZb07-uu4YYY6mxqhn1k&m=4IKTLZhW4ejOBCxEKaKQPmgsWx8LmPQAOXC0cxyrm6I&s=alvjy_DDNYSEG6Iim3N40gIYiCNN0uL7B8tlWHJU20U&e=.

This is also an issue for my organization. Is this functionality actually supported and is currently not working? Or is this more of a feature that will be added in the future?

This is also an issue for my organization.  Please advise if the syncing of profiles and roles between Azure and Salesforce is currently supported.  Has anyone been able to successfully sync a custom attribute?  When does Microsoft plan on having a resolution for this matter?

I hope this helps.  I was able to get up and running using these settings.
Azure Active Directory/ Enterprise Applications/select app – Salesforce or Remedyforce / Provisioning / Mapping
<EMAIL_ADDRESS>Click on Synchronize Azure Active Directory Users to Salesforce.com
The two attributes I’m using are highlighted
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Dexter Jackson
VP, Information Technology Services
RXR Realty
625 RXR Plaza
Uniondale, New York 11556
T:<PHONE_NUMBER>
<EMAIL_ADDRESS>www.rxrrealty.com
Confidentiality Notice: The information contained in this e-mail and any attachments may be legally privileged and confidential. If you are not an intended recipient, you should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person.
From: SalesforeUnicorn<EMAIL_ADDRESS>Sent: Thursday, April 11, 2019 1:42 PM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: Jackson, Dexter<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] Azure Salesforce Provisioning not syncing SF Profiles to AD Roles (#22223)
This is also an issue for my organization. Please advise if the syncing of profiles and roles between Azure and Salesforce is currently supported. Has anyone been able to successfully sync a custom attribute? When does Microsoft plan on having a resolution for this matter?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_MicrosoftDocs_azure-2Ddocs_issues_22223-23issuecomment-2D482221272&d=DwMFaQ&c=rGIfYmk-1z7cfZPhPNJ2NA&r=xFXf11WonGCmJqods-1nHd6KZb07-uu4YYY6mxqhn1k&m=oWVlpsmo3pzVE01wibn1bgX-DKXLlDqC3LiWRtNyTQE&s=KjutIPkZNXpmOs2ToTG_o5DXnAj3XV8bsZTN7DGkzP0&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AtaLTaX-5Fs1qlNU5MhIkLTRvo-2DEdYHULbks5vf3PjgaJpZM4Z8Wt-5F&d=DwMFaQ&c=rGIfYmk-1z7cfZPhPNJ2NA&r=xFXf11WonGCmJqods-1nHd6KZb07-uu4YYY6mxqhn1k&m=oWVlpsmo3pzVE01wibn1bgX-DKXLlDqC3LiWRtNyTQE&s=vsO_pqOMXdNYA10_dJa-12hDm7UKpUg0oWQ4HxdZ2o4&e=.

Thank you so much!
Kimberly Cobb | Salesforce Administrator
Office:<PHONE_NUMBER> | Mobile:<PHONE_NUMBER>
1920 Main Street, Suite 850, Irvine, CA 92614
CoreVest Financehttps://www.corevestfinance.com/ | LinkedInhttps://www.linkedin.com/company/corevestfinance | Facebookhttps://www.facebook.com/corevest | Twitterhttps://www.twitter.com/corevestfinance | Instagramhttps://www.instagram.com/corevest | YouTubehttps://www.youtube.com/channel/UC5m3JKszv2lYRkpp7q_nQdA
From: DJackson625<EMAIL_ADDRESS>Sent: Thursday, April 11, 2019 11:10 AM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: Kimberly Cobb<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] Azure Salesforce Provisioning not syncing SF Profiles to AD Roles (#22223)
I hope this helps. I was able to get up and running using these settings.
Azure Active Directory/ Enterprise Applications/select app – Salesforce or Remedyforce / Provisioning / Mapping
<EMAIL_ADDRESS>Click on Synchronize Azure Active Directory Users to Salesforce.com
The two attributes I’m using are highlighted
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Dexter Jackson
VP, Information Technology Services
RXR Realty
625 RXR Plaza
Uniondale, New York 11556
T:<PHONE_NUMBER>
djackson<EMAIL_ADDRESS>www.rxrrealty.comhttp://www.rxrrealty.com
Confidentiality Notice: The information contained in this e-mail and any attachments may be legally privileged and confidential. If you are not an intended recipient, you should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person.
From: SalesforeUnicorn <notifications<EMAIL_ADDRESS>Sent: Thursday, April 11, 2019 1:42 PM
To: MicrosoftDocs/azure-docs <azure-docs<EMAIL_ADDRESS>Cc: Jackson, Dexter <djackson<EMAIL_ADDRESS>Comment <comment<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] Azure Salesforce Provisioning not syncing SF Profiles to AD Roles (#22223)
This is also an issue for my organization. Please advise if the syncing of profiles and roles between Azure and Salesforce is currently supported. Has anyone been able to successfully sync a custom attribute? When does Microsoft plan on having a resolution for this matter?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_MicrosoftDocs_azure-2Ddocs_issues_22223-23issuecomment-2D482221272&d=DwMFaQ&c=rGIfYmk-1z7cfZPhPNJ2NA&r=xFXf11WonGCmJqods-1nHd6KZb07-uu4YYY6mxqhn1k&m=oWVlpsmo3pzVE01wibn1bgX-DKXLlDqC3LiWRtNyTQE&s=KjutIPkZNXpmOs2ToTG_o5DXnAj3XV8bsZTN7DGkzP0&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AtaLTaX-5Fs1qlNU5MhIkLTRvo-2DEdYHULbks5vf3PjgaJpZM4Z8Wt-5F&d=DwMFaQ&c=rGIfYmk-1z7cfZPhPNJ2NA&r=xFXf11WonGCmJqods-1nHd6KZb07-uu4YYY6mxqhn1k&m=oWVlpsmo3pzVE01wibn1bgX-DKXLlDqC3LiWRtNyTQE&s=vsO_pqOMXdNYA10_dJa-12hDm7UKpUg0oWQ4HxdZ2o4&e=.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/azure-docs/issues/22223#issuecomment-482234353, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AvPN5d1o9ehmCgC4CtLKgdMSTkLKkkE3ks5vf3pzgaJpZM4Z8Wt_.

@DJackson625
Could you please clarify the attributes that you had highlighted? I can't see the picture and I'm having a similar issue.
Thanks!

Azure Active Directory  admin/Enterprise Applications/salesforce/provisioning/mappings/
Click on the field “Synchronize Azure Active Directory Users to salesforce.com”
Enabled should be set to yes
Source object
User
Source object scope
All records
Target Object Actions
*place check marks on the following:
Create
Update
Delete
Attribute Mappings                        Saleforce.com Attribute     Matching
userPrincipalName                         UserName                                      1
SingelAppRoleAssignment             ProfileName
Click save
Dexter Jackson
VP, Information Technology Services
RXR Realty
625 RXR Plaza
Uniondale, New York 11556
T:<PHONE_NUMBER>
<EMAIL_ADDRESS>www.rxrrealty.com
Confidentiality Notice: The information contained in this e-mail and any attachments may be legally privileged and confidential. If you are not an intended recipient, you should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person.
From: dustinlcraig<EMAIL_ADDRESS>Sent: Monday, April 22, 2019 3:56 PM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: Jackson, Dexter<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] Azure Salesforce Provisioning not syncing SF Profiles to AD Roles (#22223)
@DJackson625https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_DJackson625&d=DwMCaQ&c=rGIfYmk-1z7cfZPhPNJ2NA&r=xFXf11WonGCmJqods-1nHd6KZb07-uu4YYY6mxqhn1k&m=qJQnoqh_SoWUOzMQCWY8_HvXz11otza5YrvmXhB3NAA&s=EHO8QDd3bZELeeKw2rP25_umMPS0W2A4GSEwZZHjYJ0&e=
Could you please clarify the attributes that you had highlighted? I can't see the picture and I'm having a similar issue.
Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_MicrosoftDocs_azure-2Ddocs_issues_22223-23issuecomment-2D485531103&d=DwMCaQ&c=rGIfYmk-1z7cfZPhPNJ2NA&r=xFXf11WonGCmJqods-1nHd6KZb07-uu4YYY6mxqhn1k&m=qJQnoqh_SoWUOzMQCWY8_HvXz11otza5YrvmXhB3NAA&s=jbRu9TwpZRiyMd-S3Hj2cmeNgEhsBPJ6CLmU59IOSQA&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ALLIWTM4XE4SGVUHD5IW4J3PRYJ37ANCNFSM4GPRNN7Q&d=DwMCaQ&c=rGIfYmk-1z7cfZPhPNJ2NA&r=xFXf11WonGCmJqods-1nHd6KZb07-uu4YYY6mxqhn1k&m=qJQnoqh_SoWUOzMQCWY8_HvXz11otza5YrvmXhB3NAA&s=A3L79dZ87TwuS9HYZamu96AAreFtf8JiMBSztiuuKCE&e=.

Unfortunately, for us, using the mappings as @DJackson625 suggested doesn't import the custom profiles/roles from Salesforce into Azure AD.
Did anyone got lucky adding other mappings?

Working with a client on this, and following the additional suggestions by @Archehandoro, we got the SF profiles to sync to Azure Roles - great!
A totally unexpected behavior is that Azure AD also created 1181 +NEW+ Chatter Free users in the SF Org - one for every account in AD. 😵 I deactivated all of the new users, with XL Connector, but this was a bit of a hairball in a production Org. Any thoughts as to how to preclude this from happening? I honestly didn't expect that a sync would create new users in SF unless specified.

Working with a client on this, and following the additional suggestions by @Archehandoro, we got the SF profiles to sync to Azure Roles - great!
A totally unexpected behavior is that Azure AD also created 1181 +NEW+ Chatter Free users in the SF Org - one for every account in AD.  I deactivated all of the new users, with XL Connector, but this was a bit of a hairball in a production Org. Any thoughts as to how to preclude this from happening? I honestly didn't expect that a sync would create new users in SF unless specified.

Sorry to hear that! Yeah, by syncing if the user to be provisioned isn't on SF then the account gets created. If the account exists, and something has changed, it gets updated. I should have specified that better in my previous comment.
Within the "Users and groups" of the Salesforce app in Azure AD, did you add all the org?

How can get custom profiles if I use SAML SSO and no provisioning?
Under users and groups I have to choose a profile but only a couple of profiles are shown. No custom profiles.
I'd rather have Salesforce manage this because everyone is already setup there.

How can get custom profiles if I use SAML SSO and no provisioning?
Under users and groups I have to choose a profile but only a couple of profiles are shown. No custom profiles.
I'd rather have Salesforce manage this because everyone is already setup there.

I believe if you don't use provisioning custom profiles aren't needed. You are only matching objects in Azure AD to Salesforce users. By doing this you need to manage users profiles directly in Salesforce.

How can get custom profiles if I use SAML SSO and no provisioning?
Under users and groups I have to choose a profile but only a couple of profiles are shown. No custom profiles.
I'd rather have Salesforce manage this because everyone is already setup there.

I believe if you don't use provisioning custom profiles aren't needed. You are only matching objects in Azure AD to Salesforce users. By doing this you need to manage users profiles directly in Salesforce.

Yes, I want to manage user profiles directly in Salesforce. However, I'm forced to select a role (so a Salesforce profile) in Azure AD when using SSO.
How do I get around this?

Hi, Then, the role you put your users in on the Azure AD side becomes irrelevant. It will only be used to grant access to the application. The Group you create can be associated with Standard User or other - it won't matter, as you don't use JIT or other attributes in the SAML response.Hope this helps, Sent from my Samsung Galaxy smartphone.
-------- Original message --------From: syswpit<EMAIL_ADDRESS>Date: 2019-07-15  07:48  (GMT-05:00) To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: fabienturcotte<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] Azure Salesforce Provisioning not syncing SF Profiles to AD Roles (#22223)
How can get custom profiles if I use SAML SSO and no provisioning?
Under users and groups I have to choose a profile but only a couple of profiles are shown. No custom profiles.
I'd rather have Salesforce manage this because everyone is already setup there.
I believe if you don't use provisioning custom profiles aren't needed. You are only matching objects in Azure AD to Salesforce users. By doing this you need to manage users profiles directly in Salesforce.
Yes, I want to manage user profiles directly in Salesforce. However, I'm forced to select a role (so a Salesforce profile) in Azure AD when using SSO.
How do I get around this?
—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or mute the thread.
[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/MicrosoftDocs/azure-docs/issues/22223?email_source=notifications\u0026email_token=ALHXNJT6PBRRNIUOO7WKBS3P7RPXFA5CNFSM4GPRNN72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ5OIQI#issuecomment-511370305",
"url": "https://github.com/MicrosoftDocs/azure-docs/issues/22223?email_source=notifications\u0026email_token=ALHXNJT6PBRRNIUOO7WKBS3P7RPXFA5CNFSM4GPRNN72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ5OIQI#issuecomment-511370305",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]

Hi, Then, the role you put your users in on the Azure AD side becomes irrelevant. It will only be used to grant access to the application. The Group you create can be associated with Standard User or other - it won't matter, as you don't use JIT or other attributes in the SAML response.Hope this helps,

Are you saying that if I select Standard User for everyone in AD, the Administrators will still be Administrators in Salesforce?

Are you saying that if I select Standard User for everyone in AD, the Administrators will still be Administrators in Salesforce?

@syswpit I can confirm, that's the case.

For this issue please raise the support ticket with Microsoft and they can able to help you. I am closing this thread now.

#please-close


feat: add cargo features

Close #25

✔ Checklist:

[x] Commit messages have been written in Conventional Commits format;
[x] This PR has been added to CHANGELOG.md (at the top of the list);
[x] Tests for the changes have been added (for bug fixes / features).

Codecov Report

Merging #26 (54fc192) into main (77710de) will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main      #26      +/-   ##
==========================================
+ Coverage   94.36%   94.39%   +0.02%
==========================================
  Files           9        9
  Lines         568      571       +3
==========================================
+ Hits          536      539       +3
  Misses         32       32

Impacted Files
Coverage Δ

src/lib.rs
96.17% <ø> (+0.05%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 77710de...54fc192. Read the comment docs.


PR for #65

https://github.com/istio/issues/issues/65
also includes PR for #523 as I can't figure out an easier way to separate it.

Jenkins job istio.github.io/presubmit passed

Jenkins job istio.github.io/presubmit passed

github issue ? I don't see anything in the commit


Add copy button

Add a "copy" button that can copy the current code block into the clipboard, per this feature request https://github.com/yusanshi/embed-like-gist/issues/9

Adds clipboard.js to the 404.html page

I'm not opposed to using something like the 404.html page uses versus embedding a new script.

Adds "Show Copy Button" checkbox to the main form
When "show copy button" is on, a "copy" button appears in the embedded code block.

Clicking the button copies the contents of the code block to the clipboard (via clipboard.js)

Dark theme:

Light them:

TODO:

Style button (or make a link)
Use a "copy" icon for the button?

Hi @cfurrow ,
Thanks for your work!
I tested it and had found an issue, that is, when the code is embedded on third-party pages, then clipboard.js is in fact not imported (I just added the Test part to README.md and you can try the 4th of that part).
To make it work, it seems that we should do something like https://github.com/yusanshi/embed-like-gist/blob/master/embed.js#L76-L87.
However, to me, on our own pages (i.e., 404.html), we can import any libraries we want but on others' pages (i.e., the pages where the code is embedded), I prefer to import the least libraries for two reasons:

speed up the loading.
influence (or pollute, honestly) the target page less.

I believe the highlight.js and highlightjs-line-numbers.js are required but not clipboard.js.
To copy the text we can write it ourselves (https://github.com/yusanshi/embed-like-gist/blob/master/404.html#L294-L323). To show a popover on copying I believe implementing it with pure CSS is also feasible.

 I forgot to test this on an actual page! I wanted to do a little more cleanup on this anyway (that's why it's still left in draft mode), but I will definitely use the copy code you already had within the test page, and embed it in the JS.
Thanks!

I'm running into an odd issue on my "test" page where I have three embed.js examples:

Test 1 (embed.js)

Test 2 (highlightjs badge)

Test 3 (wave.js)

When everything is fully rendered, the leading whitespace seems to be removed completely, and I've tracked it down to happening somewhere inside of highlightjs-line-numbers.min.js.
Have you seen this before? It doesn't seem to happen on the test page. I tried a few times with just one embed.js example on my test page, and it still removes the leading whitespace when line numbers are enabled.
With Line Numbers ❌

Without Line Numbers ✅

I use this example and it works well:
<!DOCTYPE html>

    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Test</title>

Test 1 (With line numbers)

Test 2 (Without line numbers)

Chrome v89 on Ubuntu

Firefox v87 on Ubuntu

What's the OS and browser you're using?
And could you inspect the page and check whether there is a white-space: pre style?

https://developer.mozilla.org/en-US/docs/Web/CSS/white-space

Thanks for that info. Your test file seems to work for me as well. I am not sure what may have been wrong with my example, as it was a bare-bones HTML page on the Brave Browser (Chromium) on macOS. I even saw the same issue with Safari.
Now that it seems to work and the whitespace correct, I'm closer to wrapping up this PR and getting it reviewed.

Thanks again for all the time you've provided in helping me work through this issue, @yusanshi. I'm wondering if you could elaborate on your prior comment:

To show a popover on copying I believe implementing it with pure CSS is also feasible.

Are you referring to potentially rendering the "copy" button as a pseudo-element? Or did you have something else in mind versus what I've done, which is a lot of JS DOM manipulation/element creation.

Hi @cfurrow ,
After googling I realized that popover with pseudo-element, pseudo-class and some JavaScript isn't a good idea. A simple implementation doesn't look well, and a beautiful one may need a lot of work considering the compatibility on different OS and browsers.
Well, GitHub doesn't show the popover, too. I think currently our background color changing on :active is enouth 😆😆

In my commits just now, I mainly adjusted some CSS attributes, including font, padding, etc. You can check it and after that, I think it's ready for mering.
Thanks for your work!

Those changes look good to me! Thanks for taking the time to tweak it.
Looks good to merge.


Get Laurenti AdultHemOrgans dataset ready for DCP2

Dataset/group this task is for:

AdultHemOrgans: Transcriptomic characterisation of haematopoietic stem and progenitor cells from human adult bone marrow, spleen and peripheral blood
This issues is an updated version of https://github.com/HumanCellAtlas/hca-data-wrangling/issues/422
which contains the remaining tasks of https://github.com/HumanCellAtlas/hca-data-wrangling/issues/394 .
Wrangler responsible for this dataset/lab:

Ray
Description of the task:

[x] fix cross-entity validation errors: 10x_has_more_than_2_files.adoc; ensure_lane_index.adoc
[x] upload data files
[x] check linking
[x] validate data files
[x] submit dataset to Production

Acceptance criteria for the task:

[x] all the tasks above complete

@rays22 Make sure the dataset does not contain living donors. If so, this dataset cannot be pushed to DCP2

@rays22 Make sure the dataset does not contain living donors. If so, this dataset cannot be pushed to DCP2

O

Updated HCA metadata spreadsheet to include

publication information for preprint PPR110213
Title = {Quantitative and molecular differences distinguish adult human medullary and extramedullary haematopoietic stem and progenitor cell landscapes},
Author = {Mende, Nicole and Bastos, Hugo and Santoro, Antonella and Sham, Kendig and Mahbubani, Krishnaa and Curd, Abbie and Takizawa, Hitoshi and Wilson, Nicola and Göttgens, Bertie and Saeb-Parsy, Kourosh and Laurenti, Elisa},
DOI = 10.1101/2<IP_ADDRESS>9753
Publisher = {bioRxiv}, Year = {2020},
BioStudies and BioSamples accession identifiers
S-SUBS4
"accession" : "SAMEA6646090", "alias" : "207B",
"accession" : "SAMEA6646089", "alias" : "313C",
"accession" : "SAMEA6646087", "alias" : "207B_bone_marrow",
"accession" : "SAMEA6646094", "alias" : "207B_peripheral_blood",
"accession" : "SAMEA6646095", "alias" : "207B_spleen",
"accession" : "SAMEA6646091", "alias" : "313C_bone_marrow",
"accession" : "SAMEA6646093", "alias" : "313C_peripheral_blood",
"accession" : "SAMEA6646096", "alias" : "313C_spleen",
"accession" : "SAMEA6646097", "alias" : "207B_bone_marrow_cells",
"accession" : "SAMEA6646098", "alias" : "207B_peripheral_blood_cells",
"accession" : "SAMEA6646088", "alias" : "207B_spleen_cells",
"accession" : "SAMEA6646086", "alias" : "313C_bone_marrow_cells",
"accession" : "SAMEA6646085", "alias" : "313C_peripheral_blood_cells",
"accession" : "SAMEA6646092", "alias" : "313C_spleen_cells",

Uploaded the updated metadata spreadsheet to Ingest.

Jul 1, 2020, 10:22:23 PM	609a2401-246d-470f-a89e-6dfc38f3c7a9	Valid

Uploaded fastq.gz sequencing data files to Ingest upload area

s3://org-hca-data-archive-upload-prod/609a2401-246d-470f-a89e-6dfc38f3c7a9/.

Uploaded the latest HCA metadata spreadsheet
FASTQ-updated-2020-07-01.xlsx
to
https://drive.google.com/open?id=1wPFQvAVYX8_F5rlYlgFIxuI4w-Fmr8J-
.

[ ] I am waiting for the green light to submit the dataset in Ingest UI, because this dataset has already been archived, so it just needs to be sent/exported to DCP2.

Submitted.

Need to add ENA accessions.

This morning I made a submission to project
https://ui.ingest.archive.data.humancellatlas.org/projects/detail?uuid=af36b442-c96d-4708-99cc-b01c38dbd367
The submission is from upload Jul 1, 2020, 10:22:23 PM and it has status Exporting
Jul 1, 2020, 10:22:23 PM    609a2401-246d-470f-a89e-6dfc38f3c7a9    Exporting.

I need to add the ENA accessions to the metadata. I have tried to upload an update submission:
Jul 6, 2020, 4:30:27 PM db5fa8f5-5069-4aa0-a5f9-edc61f9120fd    Submitted

, but it does seem to have done only a partial update. I can not see any updates using Ingest UI concerning
INSDC project accession (project.insdc_study_accessions)
and
ENA run accessions (sequence_file.insdc_run_accessions).
If necessary, I could re-start the project/submission from scratch, but I can not delete submission 609a2401-246d-470f-a89e-6dfc38f3c7a9 that has the status Exporting.
I have the updated metadata spreadsheet and the data files, but I am unable to update the existing submission or delete and re-create this project from scratch with the updates.

The BioSamples sample_accession - ENA sample (experiment_accession and secondary_sample_accession) associations look incorrect. As a result, the run accessions are not linked to the appropriate samples.

The remaining tasks will be tracked in https://github.com/ebi-ait/hca-ebi-wrangler-central/issues/60.

I have uploaded the updated HCA spreadsheets with ENA accessions to
https://drive.google.com/open?id=1wPFQvAVYX8_F5rlYlgFIxuI4w-Fmr8J-
.

Spreadsheet with UUIDs (to make an update submission): FASTQ-updated-ENA_accessions_update.xlsx
Spreadsheet without UUIDs: FASTQ-updated-ENA_accessions.xlsx

Needs update as defined in #271

Update on this project was done "ad-hoc" I believe these changes can be made it the UI and then the project needs re-export using the 'metadata only' flag.
Project: AdultHemOrgans:5b5f05b7-2482-468d-b76d-8f68c04a7a47:
project.publications.url: "https://www.biorxiv.org/content/10.1101/2<IP_ADDRESS>9753v1"
project.contributors.name: "Ray,,stefancsik"
project.contributors.email<EMAIL_ADDRESS>project.contributors.institution: "EMBL-EBI"
project.contributors.laboratory: "Human Cell Atlas Data Coordination Platform"
project.contributors.adress: "Wellcome Genome Campus, Hinxton, Cambridge CB10 1SA"
updates indicated as per - https://github.com/ebi-ait/hca-ebi-wrangler-central/issues/260

I have updated the contributor list in Ingest UI.

This project has two submissions, the newer one being an update with additional data from the contributors. The Ingest status changed to Valid for the earlier submission.

I tried to do an export of the Valid submission and chose option Submit only metadata to the Human Cell Atlas.

The Ingest status of the Valid  submission changed to Exported.

I have checked the broad-dsp-monster-hca-prod-ebi-storage area, but I could not find any newly exported metadata file. I have checked the /prod/metadata/project/*.json file, but the contributor list is still the same as before the update.

In summary, I can find no evidence for a successful export of the updates and I am stuck.
@MightyAx , could you kindly look into it?

It seems that re-starting the Exporters yesterday has worked:

It seems that re-starting the Exporters yesterday has worked:

No, I am afraid that not solved my problems. There are two issues. The first one is that the Ingest status did change to Exported, despite that I have found no evidence of updated project metadata in broad-dsp-monster-hca-prod-ebi-storage. The second issue is the missing metadata update in the export bucket, which in turn, I believe, will prevent  any metadata update to show up in the HCA Data Browser.
Sorry if that was not clear in my earlier notes.
This morning I re-tried the export after doing some minor project metadata edits in Ingest for the project (just to allow re-export by Ingest) and chose the option Submit only metadata to the Human Cell Atlas again.
The submission status has changed to Exported (as previously).
Unfortunately, I can still see no evidence of any updated metadata for this project in broad-dsp-monster-hca-prod-ebi-storage.
@MightyAx , could you kindly double check if there is any evidence of metadata update in the export area that reflects the changes that I made in Ingest UI?
Some extra information to help troubleshooting. This project has two previous exports:

2020-11-12T20:09:50Z
2021-02-11T12:47:44Z

The dates might be important for the history of how Ingest exports worked at those times.
I am still blocked.

The exporter logs contain failures for envelopeUuid: 90513d30-5f47-473e-b180-c37a974fc03b which is the first of the two submissions listed chronologically.
Example:
ingest-exporter-69697f8488-59wm7 ingest-exporter  2021-04-08 12:58:24,369  - exporter.terra.terra_listener - ERROR in terra_listener.py:119 _experiment_message_handler(): Failed to export experiment message with body:
{
    "bundleUuid":"7237eeb1-d047-4296-83f5-f51b864a6b53",
    "versionTimestamp":"2020-07-15T15:56:40.181Z",
    "messageProtocol":null,
    "exportJobId":"606dc841dd9aab1232d14c00",
    "documentId":"5f0f273832d8c765c3fdfb16",
    "documentUuid":"7237eeb1-d047-4296-83f5-f51b864a6b53",
    "callbackLink":"/processes/5f0f273832d8c765c3fdfb16",
    "documentType":"Process",
    "envelopeId":"5f0f273132d8c765c3fdfa2f",
    "envelopeUuid":"90513d30-5f47-473e-b180-c37a974fc03b",
    "index":2,
    "total":32
}

Complete log for that instance of ingest-exporter:
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:08:11,850  - ingest.api.ingestapi - INFO in ingestapi.py:30 __init__(): using http://<IP_ADDRESS>:80 for ingest API
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:08:12,482  - manifest.receiver.ManifestReceiver - INFO in receiver.py:18 run(): Running ManifestReceiver
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:08:12,537  - ingest.api.ingestapi - INFO in ingestapi.py:30 __init__(): using http://<IP_ADDRESS>:80 for ingest API
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:08:12,869  - exporter.terra.terra_listener - INFO in terra_listener.py:106 _experiment_message_handler(): Received experiment message for process 8a9b7207-a859-466e-b47d-5f90a6d0e32e (index 13 for submission 2f03df05-521c-4115-8964-bdef07c5dbd2)
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:08:13,535  - exporter.terra.terra_exporter - INFO in terra_exporter.py:31 export(): The export data flag has been set to True
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:08:13,536  - exporter.terra.terra_exporter - INFO in terra_exporter.py:34 export(): Exporting data files..
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:08:16,078  - exporter.terra.gcs - INFO in gcs.py:156 _assert_job_complete(): Awaiting complete transfer for job transferJobs/606e1954dd9aab1232d14c02. Waiting 2 seconds.(total time waited: 0, max wait time: 21600 seconds)
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:08:18,181  - exporter.terra.gcs - INFO in gcs.py:156 _assert_job_complete(): Awaiting complete transfer for job transferJobs/606e1954dd9aab1232d14c02. Waiting 4 seconds.(total time waited: 2, max wait time: 21600 seconds)
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:08:22,281  - exporter.terra.gcs - INFO in gcs.py:156 _assert_job_complete(): Awaiting complete transfer for job transferJobs/606e1954dd9aab1232d14c02. Waiting 8 seconds.(total time waited: 6, max wait time: 21600 seconds)
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:08:30,379  - exporter.terra.gcs - INFO in gcs.py:156 _assert_job_complete(): Awaiting complete transfer for job transferJobs/606e1954dd9aab1232d14c02. Waiting 16 seconds.(total time waited: 14, max wait time: 21600 seconds)
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:08:46,489  - exporter.terra.gcs - INFO in gcs.py:156 _assert_job_complete(): Awaiting complete transfer for job transferJobs/606e1954dd9aab1232d14c02. Waiting 32 seconds.(total time waited: 30, max wait time: 21600 seconds)
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:09:18,626  - exporter.terra.gcs - INFO in gcs.py:156 _assert_job_complete(): Awaiting complete transfer for job transferJobs/606e1954dd9aab1232d14c02. Waiting 64 seconds.(total time waited: 62, max wait time: 21600 seconds)
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:10:22,810  - exporter.terra.gcs - INFO in gcs.py:156 _assert_job_complete(): Awaiting complete transfer for job transferJobs/606e1954dd9aab1232d14c02. Waiting 128 seconds.(total time waited: 126, max wait time: 21600 seconds)
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:12:30,989  - exporter.terra.gcs - INFO in gcs.py:156 _assert_job_complete(): Awaiting complete transfer for job transferJobs/606e1954dd9aab1232d14c02. Waiting 256 seconds.(total time waited: 254, max wait time: 21600 seconds)
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:16:47,191  - exporter.terra.gcs - INFO in gcs.py:156 _assert_job_complete(): Awaiting complete transfer for job transferJobs/606e1954dd9aab1232d14c02. Waiting 512 seconds.(total time waited: 510, max wait time: 21600 seconds)
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:19,391  - exporter.terra.terra_exporter - INFO in terra_exporter.py:37 export(): Exporting metadata..
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:23,355  - ingest.api.ingestapi - INFO in ingestapi.py:270 get_all(): GET schemas {"size": 20, "totalElements": 101, "totalPages": 6, "number": 1}
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:23,439  - ingest.api.ingestapi - INFO in ingestapi.py:270 get_all(): GET schemas {"size": 20, "totalElements": 101, "totalPages": 6, "number": 2}
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:23,548  - ingest.api.ingestapi - INFO in ingestapi.py:270 get_all(): GET schemas {"size": 20, "totalElements": 101, "totalPages": 6, "number": 3}
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:23,682  - ingest.api.ingestapi - INFO in ingestapi.py:270 get_all(): GET schemas {"size": 20, "totalElements": 101, "totalPages": 6, "number": 4}
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:23,797  - ingest.api.ingestapi - INFO in ingestapi.py:270 get_all(): GET schemas {"size": 20, "totalElements": 101, "totalPages": 6, "number": 5}
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:27,228  - exporter.terra.gcs - INFO in gcs.py:254 _assert_file_uploaded(): Verifying upload of blob prod/05be4f37-4506-429b-b112-506444507d62/metadata/donor_organism/7b6d6a81-5064-4af4-b33c-72bf7d56ecf8_2021-04-06T23:05:29.487000Z.json. Waiting for 0.2 seconds...
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:29,397  - ingest.api.ingestapi - INFO in ingestapi.py:270 get_all(): GET schemas {"size": 20, "totalElements": 101, "totalPages": 6, "number": 1}
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:29,611  - ingest.api.ingestapi - INFO in ingestapi.py:270 get_all(): GET schemas {"size": 20, "totalElements": 101, "totalPages": 6, "number": 2}
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:29,770  - ingest.api.ingestapi - INFO in ingestapi.py:270 get_all(): GET schemas {"size": 20, "totalElements": 101, "totalPages": 6, "number": 3}
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:29,887  - ingest.api.ingestapi - INFO in ingestapi.py:270 get_all(): GET schemas {"size": 20, "totalElements": 101, "totalPages": 6, "number": 4}
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:29,964  - ingest.api.ingestapi - INFO in ingestapi.py:270 get_all(): GET schemas {"size": 20, "totalElements": 101, "totalPages": 6, "number": 5}
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:30,500  - exporter.terra.terra_listener - INFO in terra_listener.py:108 _experiment_message_handler(): Exported experiment for process uuid 8a9b7207-a859-466e-b47d-5f90a6d0e32e (--index 13 --total 1543 --submission 2f03df05-521c-4115-8964-bdef07c5dbd2)
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:30,655  - exporter.terra.terra_listener - INFO in terra_listener.py:106 _experiment_message_handler(): Received experiment message for process 84e6191d-e8b6-4cd3-9c81-84b0b0be4b2e (index 4 for submission 90513d30-5f47-473e-b180-c37a974fc03b)
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:30,748  - exporter.terra.terra_exporter - INFO in terra_exporter.py:31 export(): The export data flag has been set to False
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:30,749  - exporter.terra.terra_exporter - INFO in terra_exporter.py:37 export(): Exporting metadata..
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:32,018  - exporter.terra.gcs - INFO in gcs.py:254 _assert_file_uploaded(): Verifying upload of blob prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json. Waiting for 0.2 seconds...
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:32,252  - exporter.terra.gcs - INFO in gcs.py:254 _assert_file_uploaded(): Verifying upload of blob prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json. Waiting for 0.4 seconds...
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:32,688  - exporter.terra.gcs - INFO in gcs.py:254 _assert_file_uploaded(): Verifying upload of blob prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json. Waiting for 0.8 seconds...
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:33,518  - exporter.terra.gcs - INFO in gcs.py:254 _assert_file_uploaded(): Verifying upload of blob prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json. Waiting for 1.6 seconds...
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:35,155  - exporter.terra.gcs - INFO in gcs.py:254 _assert_file_uploaded(): Verifying upload of blob prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json. Waiting for 3.2 seconds...
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:38,391  - exporter.terra.gcs - INFO in gcs.py:254 _assert_file_uploaded(): Verifying upload of blob prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json. Waiting for 6.4 seconds...
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:44,827  - exporter.terra.gcs - INFO in gcs.py:254 _assert_file_uploaded(): Verifying upload of blob prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json. Waiting for 12.8 seconds...
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:25:57,672  - exporter.terra.gcs - INFO in gcs.py:254 _assert_file_uploaded(): Verifying upload of blob prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json. Waiting for 25.6 seconds...
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:26:23,337  - exporter.terra.gcs - INFO in gcs.py:254 _assert_file_uploaded(): Verifying upload of blob prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json. Waiting for 51.2 seconds...
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:27:14,625  - exporter.terra.gcs - INFO in gcs.py:254 _assert_file_uploaded(): Verifying upload of blob prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json. Waiting for 102.4 seconds...
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:28:57,160  - exporter.terra.gcs - INFO in gcs.py:254 _assert_file_uploaded(): Verifying upload of blob prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json. Waiting for 204.8 seconds...
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:32:22,109  - exporter.terra.gcs - INFO in gcs.py:254 _assert_file_uploaded(): Verifying upload of blob prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json. Waiting for 409.6 seconds...
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:39:11,878  - exporter.terra.gcs - INFO in gcs.py:254 _assert_file_uploaded(): Verifying upload of blob prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json. Waiting for 819.2 seconds...
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 11:52:51,251  - exporter.terra.gcs - INFO in gcs.py:254 _assert_file_uploaded(): Verifying upload of blob prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json. Waiting for 1638.4 seconds...
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 12:20:09,824  - exporter.terra.gcs - INFO in gcs.py:254 _assert_file_uploaded(): Verifying upload of blob prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json. Waiting for 3276.8 seconds...
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 13:14:46,865  - exporter.terra.gcs - INFO in gcs.py:254 _assert_file_uploaded(): Verifying upload of blob prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json. Waiting for 6553.6 seconds...
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 13:14:46,865  - exporter.terra.terra_listener - ERROR in terra_listener.py:119 _experiment_message_handler(): Failed to export experiment message with body: {"bundleUuid":"84e6191d-e8b6-4cd3-9c81-84b0b0be4b2e","versionTimestamp":"2020-07-15T15:56:40.214Z","messageProtocol":null,"exportJobId":"606dc841dd9aab1232d14c00","documentId":"5f0f273832d8c765c3fdfb18","documentUuid":"84e6191d-e8b6-4cd3-9c81-84b0b0be4b2e","callbackLink":"/processes/5f0f273832d8c765c3fdfb18","documentType":"Process","envelopeId":"5f0f273132d8c765c3fdfa2f","envelopeUuid":"90513d30-5f47-473e-b180-c37a974fc03b","index":4,"total":32}
ingest-exporter-69697f8488-2qd2n ingest-exporter  2021-04-08 13:14:46,865  - exporter.terra.terra_listener - ERROR in terra_listener.py:120 _experiment_message_handler(): Could not verify completed upload for blob prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json within maximum wait time of 3600 seconds
ingest-exporter-69697f8488-2qd2n ingest-exporter Traceback (most recent call last):
ingest-exporter-69697f8488-2qd2n ingest-exporter   File "/app/exporter/terra/terra_listener.py", line 107, in _experiment_message_handler
ingest-exporter-69697f8488-2qd2n ingest-exporter     self.terra_exporter.export(exp.process_uuid, exp.submission_uuid, exp.experiment_uuid, exp.experiment_version, exp.job_id)
ingest-exporter-69697f8488-2qd2n ingest-exporter   File "/app/exporter/terra/terra_exporter.py", line 40, in export
ingest-exporter-69697f8488-2qd2n ingest-exporter     self.dcp_staging_client.write_metadatas(experiment_graph.nodes.get_nodes(), project.uuid)
ingest-exporter-69697f8488-2qd2n ingest-exporter   File "/app/exporter/terra/dcp_staging_client.py", line 71, in write_metadatas
ingest-exporter-69697f8488-2qd2n ingest-exporter     self.write_metadata(metadata, project_uuid)
ingest-exporter-69697f8488-2qd2n ingest-exporter   File "/app/exporter/terra/dcp_staging_client.py", line 88, in write_metadata
ingest-exporter-69697f8488-2qd2n ingest-exporter     self.write_file_descriptor(metadata, project_uuid)
ingest-exporter-69697f8488-2qd2n ingest-exporter   File "/app/exporter/terra/dcp_staging_client.py", line 100, in write_file_descriptor
ingest-exporter-69697f8488-2qd2n ingest-exporter     self.write_to_staging_bucket(dest_object_key, data_stream)
ingest-exporter-69697f8488-2qd2n ingest-exporter   File "/app/exporter/terra/dcp_staging_client.py", line 113, in write_to_staging_bucket
ingest-exporter-69697f8488-2qd2n ingest-exporter     self.gcs_storage.write(object_key, data_stream)
ingest-exporter-69697f8488-2qd2n ingest-exporter   File "/app/exporter/terra/gcs.py", line 209, in write
ingest-exporter-69697f8488-2qd2n ingest-exporter     self.assert_file_uploaded(object_key)
ingest-exporter-69697f8488-2qd2n ingest-exporter   File "/app/exporter/terra/gcs.py", line 239, in assert_file_uploaded
ingest-exporter-69697f8488-2qd2n ingest-exporter     return self._assert_file_uploaded(blob, one_hundred_milliseconds, one_hour_in_seconds)
ingest-exporter-69697f8488-2qd2n ingest-exporter   File "/app/exporter/terra/gcs.py", line 255, in _assert_file_uploaded
ingest-exporter-69697f8488-2qd2n ingest-exporter     return self._assert_file_uploaded(blob, new_sleep_time, max_sleep_time)
ingest-exporter-69697f8488-2qd2n ingest-exporter   File "/app/exporter/terra/gcs.py", line 255, in _assert_file_uploaded
ingest-exporter-69697f8488-2qd2n ingest-exporter     return self._assert_file_uploaded(blob, new_sleep_time, max_sleep_time)
ingest-exporter-69697f8488-2qd2n ingest-exporter   File "/app/exporter/terra/gcs.py", line 255, in _assert_file_uploaded
ingest-exporter-69697f8488-2qd2n ingest-exporter     return self._assert_file_uploaded(blob, new_sleep_time, max_sleep_time)
ingest-exporter-69697f8488-2qd2n ingest-exporter   [Previous line repeated 13 more times]
ingest-exporter-69697f8488-2qd2n ingest-exporter   File "/app/exporter/terra/gcs.py", line 243, in _assert_file_uploaded
ingest-exporter-69697f8488-2qd2n ingest-exporter     raise UploadPollingException(f'Could not verify completed upload for blob {blob.name} within maximum '
ingest-exporter-69697f8488-2qd2n ingest-exporter exporter.terra.gcs.UploadPollingException: Could not verify completed upload for blob prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json within maximum wait time of 3600 seconds

I'll be working through this example with @aaclan-ebi later.
Project UUID: 455b46e6-d8ea-4611-861e-de720a562ada
Staging Bucket: gs://broad-dsp-monster-hca-prod-ebi-storage/prod/455b46e6-d8ea-4611-861e-de720a562ada
Ingest UI: https://contribute.data.humancellatlas.org/projects/detail?uuid=455b46e6-d8ea-4611-861e-de720a562ada
Submission UUIDs: 90513d30-5f47-473e-b180-c37a974fc03b, cb156730-90b0-4b77-944c-bfc263204c61
[Export Logs submissionUuid= 90513d30-5f47-473e-b180-c37a974fc03b](https://api.ingest.archive.data.humancellatlas.org/exportJobs/search/find?submissionUuid= 90513d30-5f47-473e-b180-c37a974fc03b)
Export Logs submissionUuid=cb156730-90b0-4b77-944c-bfc263204c61
Not Exported since February

File might actually be here:
gs://broad-dsp-monster-hca-prod-ebi-storage/prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json

File is actually there but the metadata for export complete has not been added so the exporter can't verify that it exists.
The file was added before the exporter would add export_complete to files after the finished uploading.
Details here: https://github.com/ebi-ait/dcp-ingest-central/issues/175#issuecomment-779443533
Running this detects the issue: gsutil stat gs://broad-dsp-monster-hca-prod-ebi-storage/prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json gs://broad-dsp-monster-hca-prod-ebi-storage/prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json
    Creation time:          Thu, 12 Nov 2020 20:09:46 GMT
    Update time:            Thu, 12 Nov 2020 20:09:46 GMT
    Storage class:          STANDARD
    Content-Language:       en
    Content-Length:         623
    Content-Type:           application/json
    Hash (crc32c):          3IhKew==
    Hash (md5):             N0QDFMDLNmkzp4W4riAyqg==
    ETag:                   CJP00fLn/ewCEAE=
    Generation:<PHONE_NUMBER>607123
    Metageneration:         1

compared to a recently updated file you can see that Metadata exported, export_completed are missing:
    Creation time:          Thu, 12 Nov 2020 20:09:50 GMT
    Update time:            Mon, 15 Feb 2021 12:57:56 GMT
    Storage class:          STANDARD
    Content-Length:         5488
    Content-Type:           application/json
    Metadata:
        exported:           True
        export_completed:   True
    Hash (crc32c):          oNdgLg==
    Hash (md5):             fqG/bL7E66WjkMVanQELCA==
    ETag:                   CPuLtvTn/ewCEAM=
    Generation:<PHONE_NUMBER>345723
    Metageneration:         3

Running the following will fix the issue for this file
gsutil setmeta -h "x-goog-meta-export_completed:True" gs://broad-dsp-monster-hca-prod-ebi-storage/prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json
But there are at least 20 other files for this project and an unknown amount for other projects that also need this fix to ensure exports do not fail in the future.
I'm going to open a ticket to write a script to detect and remediate any metadata files that have not been marked as export_complete and where the update time is older that March 1st.

File might actually be here:
gs://broad-dsp-monster-hca-prod-ebi-storage/prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/5c226313-56df-4dfd-bc2b-0fc3871facc3_2020-07-15T15:56:37.218000Z.json

I am sorry but that does not look like the file I am looking for. I think you might have uncovered some other exporting issues/errors.
I would expect the updated and exported project metadata file with a more current date (the date and time of the latest export).
I had seen 3 project metadata files before I started the export of the latest (two) update attempts:
  5.36 KiB  2020-11-12T21:00:31Z  gs://broad-dsp-monster-hca-prod-ebi-storage/bk/metadata/project/455b46e6-d8ea-4611-861e-de720a562ada_2020-07-15T15:56:36.681000Z.json
  5.36 KiB  2020-11-12T20:09:50Z  gs://broad-dsp-monster-hca-prod-ebi-storage/prod/455b46e6-d8ea-4611-861e-de720a562ada/metadata/project/455b46e6-d8ea-4611-861e-de720a562ada_2020-07-15T15:56:36.681000Z.json
  4.93 KiB  2021-02-11T12:47:44Z  gs://broad-dsp-monster-hca-prod-ebi-storage/prod/metadata/project/455b46e6-d8ea-4611-861e-de720a562ada_2020-07-15T15:56:36.681000Z.json

I can see only the same 3 files after my update and export attempts.
None of these 3 files contain my updates which I have confirmed by looking into their contents.
I suspect that the metadata file
gs://broad-dsp-monster-hca-prod-ebi-storage/prod/455b46e6-d8ea-4611-861e-de720a562ada/metadata/project/455b46e6-d8ea-4611-861e-de720a562ada_2020-07-15T15:56:36.681000Z.json

has been edited/altered by hand after the 2020-07-15 export from Ingest for some reason, because it has content that was not in Ingest nor the spreadsheet that I used originally. Could that manual alteration/hack be the culprit that prevents correct exporting now by causing cloud file metadata inconsistencies?

I believe the exporter failing means that the exporter is never exporting the "updated" versions of the metadata.
I have a fix for the old metadata now, lets reasses the above once that has been run and the exporter jobs complete sucessfully.

@rays22 Can I update the metadata for the following 192 files so that the exporter knows they have been transferred?

I have no reason to object the updating of the file metadata if you think that it is necessary. I admit that it is beyond my understanding of the exporter to tell if there were any reasons/arguments against doing the updates.

The above files have been updated:
Files inspected:	1,138	Files found:	192	Files updated:	192
example file stat after update:
gsutil stat gs://broad-dsp-monster-hca-prod-ebi-storage/prod/455b46e6-d8ea-4611-861e-de720a562ada/descriptors/sequence_file/fe8351b7-dc0c-4b17-8ac5-4550245b62cb_2020-07-15T15:56:37.236000Z.json

    Creation time:          Thu, 12 Nov 2020 20:09:48 GMT
    Update time:            Mon, 12 Apr 2021 14:04:10 GMT
    Storage class:          STANDARD
    Content-Language:       en
    Content-Length:         623
    Content-Type:           application/json
    Metadata:
        export_completed:   True
    Hash (crc32c):          QNhtVA==
    Hash (md5):             3eRakQ6WRuPMDKo7wbU+mA==
    ETag:                   CPX20/Pn/ewCEAI=
    Generation:<PHONE_NUMBER>737397
    Metageneration:         2

The metadata I am referring to is the file metadata stored against the files (whether data files of metadata files) on the terra staging bucket.
The errors exporting the Laurenti project meant that none of the updates could be applied because the exporter could not verify that the files were present to be updated. Which would explain why you only found the original metadata in the bucket.
@rays22 I believe the updates have now all processed but I would appreciate it if you could confirm.
Any issues and we can start a new investigation

I can confirm that the project description metadata have been successfully exported to the Terra bucket.
Many thanks @MightyAx  for fixing the exporting errors.
I can also see exported links metadata updates. This correctly represents the metadata in Ingest. Let's hope that the Broad import team can import the links updates too.

Hi Ray. I'm really sorry about this, but did you also include the change to the project.publications.url to "https://www.biorxiv.org/content/10.1101/2<IP_ADDRESS>9753v1" when you made the updates? I had a look at the gsutil bucket and didn't see it there.

Thanks for spotting my oversight @Wkt8 .  I have added the missing project.publications.url as https://doi.org/10.1101/2<IP_ADDRESS>9753 and re-exported the updated metadata.


runtime: idle mark workers run when there's work to do

Currently idle mark workers always run for a whole Go scheduler quantum (10ms), even when more work comes in in the middle of that quantum (e.g., more runnable goroutines or incoming network packets). This is really bad for latency-sensitive systems.
The effect of this is particularly noticeable in RPC servers with very short handlers, such as in issue #16432. In these systems, all of the Ps have frequent, short idle periods. With the current idle worker behavior, these idle periods immediately trigger the idle worker, putting that P out of commission for 10ms. This cascades through the Ps until it reaches the point where some set of Ps is completely and continuously saturated.
We should fix idle workers to really have idle priority, returning control to the scheduler as soon as possible if there is other work the P could be doing.
Ideally, the idle worker would be able to check a simple flag on each iteration through the drain loop.
A simpler solution would be to only scan one object or root job at a time and then return to the scheduler. This is essentially how the background sweep currently achieves an "idle priority". This would have fairly high overhead for the idle worker, but it would accomplish the desired effect. We could batch objects into, say, units of 10k bytes, which would limit the idle worker to ~10 microseconds, while somewhat amortizing the cost of returning to the scheduler. I have a version of this prototyped (though with a 1ms time-based bound) at https://go-review.googlesource.com/c/24706/.
Another possible solution (from https://github.com/golang/go/issues/16432#issuecomment-235465053) would be to always keep one P "truly idle". If that P finds itself with work, it would signal some idle worker to exit immediately.
/cc @RLH @rhysh

I can apply the patch at https://go-review.googlesource.com/#/c/24706/1/src/runtime/mgcmark.go to an updated master source and test it.
Could it be useful for you?

@genez, thanks for offering. There's no harm in testing with that patch, but I'm not sure it would be useful.

I explored this problem more today. The rpc benchmark from gcbench shows this off particularly well. This benchmark fires up trivial client and server processes, which communicate over a set of TCP sockets. The client uses open-loop control to send requests according to a Poisson process, which models the behavior of real RPC systems.
This is what the execution trace looks like during a GC:

G18 and G19 are the background workers. You can see that they always run for a full quantum, and often run simultaneously, blocking both Ps. Also, even though there's only a little overall idle time, together they're clearly taking up much more than 25% of the total CPU. In the goroutines row, you can see the system regularly falling behind throughout the GC cycle, and in the network row, you can see it mostly falls back to the periodic sysmon network poll.
CL 30031 is an experiment that changes the idle worker to constantly poll for either run queue work or network wake-ups. With this CL, the trace looks very different (trace file):

In this trace, G8 and G9 are the background workers. There are still large regions where they're running the fractional worker, but there's only one at a time, and they add up to almost exactly 25% of the CPU. In the goroutines and network rows, you can see that it still falls behind at the beginning of GC and switches to the sysmon network poll. However, about 80ms in to the cycle, it catches up and things run quite smoothly for the rest of the cycle: the goroutine queue stays short, and network events are coming in rapidly.
There are a few extra colors in this trace. In the lower half of each proc row, red means "syscall" (which means user code in this case), teal means "assist", and blue means "idle worker". At the beginning of GC, when the system is falling behind, there are a lot of assists. In fact, this phase lasts about 70ms, which is right around the P99.9 and max latencies of this run. Eventually, however, the background workers get ahead, the assists stop, the runnable goroutine queue drains, we start having idle time again, and we start seeing the idle worker kicking in for very brief periods.

@aclements just as an additional data point, I got around to running the small micro benchmark that I wrote for: https://github.com/golang/go/issues/17341
So I ran it with few different versions of GO, something like:
Go1.7
GODEBUG=gctrace=1 ./net_gctest_GO17 -port=5001 -nroutines=30000 -sleep_t=3s -max_latency=30ms -total_qps=5000000 &> gcbench_5mln_go17.txt

Go: d70b0fe...
GODEBUG=gctrace=1 ./net_gctest_GOd70b0fe -port=5001 -nroutines=30000 -sleep_t=3s -max_latency=30ms -total_qps=5000000 &> gcbench_5mln_god70b0fe.txt

Go: 8ddf8d4ea...
GODEBUG=gctrace=1 ./net_gctest_GO8ddf8d4ea -port=5001 -nroutines=30000 -sleep_t=3s -max_latency=30ms -total_qps=5000000 &> gcbench_5mln_go8ddf8d4ea.txt

So each one does 5mln QPS total in the benchmark, and then I checked how many out of these 5mln QPS took more than 30ms.
cat gcbench_5mln_go17.txt | grep "STALL" | wc -l
12298

cat gcbench_5mln_god70b0fe.txt | grep "STALL" | wc -l
27679

cat gcbench_5mln_go8ddf8d4ea.txt | grep "STALL" | wc -l
30877

So overall it looks like, there are actually more requests that are slowed down/stuck than there were before.
Note that this experiment is running on a pretty beefy machine in general, so there is almost no CPU utilization, since it is only doing 10k QPS. Also as before the GC lines look something like:
gc 31 @314.612s 0%: 0.35+174+1.1 ms clock, 10+590/1383/3164+35 ms cpu, 1411->1415->723 MB, 1447 MB goal, 32 P
gc 32 @330.130s 0%: 0.32+174+1.4 ms clock, 10+595/1358/3004+45 ms cpu, 1411->1415->723 MB, 1447 MB goal, 32 P

So there is almost no STW gc time, it is all in the concurrent phase.
I can also try to augment the benchmark to to get the perf traces too if that will be helpful.

Actually looks like I made a mistake of not properly rebuilding things with GO master, so I have rebuilt it properly and thinks do look better with the patch compared to Go1.7:
Finishing Up. GO Version: devel +8ddf8d4 Sun Oct 30 20:27:14 2016 -0400,
QPS: 2000000, STALLS: 2301
Finishing Up. GO Version: go1.7.1, QPS: 2000000, STALLS: 6952
So there are almost 3 times less stalls with +8ddf8d4 compared to GO 1.7.

Thanks for the data points @zviadm. Just to clarify, do you mean commit d70b0fe specifically made the stalls worse (compared to d70b0fe^), or just that they got worse between go1.7.1 and d70b0fe?

I only did tests with Go1.7 and d70b0fe. So things just got worse sometime between Go 1.7.1 and d70b0fe.
One more data point, I ran a more real/complex application experiment with: 8ddf8d4. There is definitely improvement, but having lots of go routines is still quite problematic.
Here are some numbers from experiment for a RPC call (times are end to end measured from client side):

p75: ~0.5ms (which is great)
p95: ~27ms (which is quite bad for low latency service)
p99: ~100ms

The p75 times are great, however because of huge p95/p99 times, even average latency is at ~5ms.
This real application also has similar GC situation where actual STW times are <1-2ms. The concurrent part of GC is taking 100-150ms, and it still causes massive p95/p99 request spikes.

@zviadm, thanks for the data. Hopefully the commit I just pushed helps, but it sounds like there are other latency issues affecting your benchmark. This isn't too surprising; we're aware there are other issues. I also have an RPC microbenchmark that shows similar tail latency behavior to yours. I don't completely understand what's going on in my benchmark, but I've analyzed it enough to determine that it's something subtler than the GC just taking away long time slices.
You might want to follow #14812, though it's possible that issue has been fixed (waiting on confirmation). If it has, I'll open up a new tracking bug.


[KYUUBI #2081] add YARN_CONF_DIR to kyuubi server classpath

Why are the changes needed?

fix #2081
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before make a pull request

@SteNicholas has submitted a more reasonable pr, I will close this


gulp init not working on my mac

on running gulp init it says ---->This will walk you through creating a new React Native project in /Users/prnjls/Documents/reactnative/angular2-react-native-seed/dist/ngRnSeed.
But it does not do anything afterwards.
and on running gulp start.ios afterwards says ----> file.js
[16:48:27] Starting '!assets'...
[16:48:27] Starting 'watch'...
[16:48:27] Finished '!assets' after 63 ms
[16:48:27] Starting 'transpile'...
[16:48:31] Finished 'transpile' after 3.7 s
[16:48:31] Starting '!launch.ios'...
{ [Error: Command failed: /bin/sh -c react-native run-ios
Command run-ios unrecognized. Did you mean to run this inside a react-native project?
]
killed: false,
code: 1,
signal: null,
cmd: '/bin/sh -c react-native run-ios' }
[16:48:32] Finished '!launch.ios' after 348 ms

Hello,
this is what gulp init should do:
gulp init
[11:22:24] Using gulpfile ~/dev/github/react-native-renderer/gulpfile.js
[11:22:24] Starting 'clean'...
[11:22:24] Finished 'clean' after 39 ms
[11:22:24] Starting '!create'...
This will walk you through creating a new React Native project in /Users/mlaval/dev/github/react-native-renderer/dist/app/ngReactNative

Installing react-native package from npm...

Setting up new React Native app in /Users/mlaval/dev/github/react-native-renderer/dist/app/ngReactNative```

It takes a few minutes to execute everything.
Once it is done, you can do the `gulp start.ios`

By the way, the project has just been moved to: https://github.com/angular/react-native-renderer
See you there :)

Thanks did the trick .Will meet again at  https://github.com/angular/react-native-renderer


Automate refreshing data or refresh button!

Thank you for this great work,
It would be really good if you can keep the data or database's tables refreshing by it self
OR
if we can have like refresh button on each table and each SharedPreferences to refresh data!
Thank you in advance.

Yes, this will be a good feature to add in the new release. Thanks for the suggestion.
Thanks
Amit Shekhar

+1


Change DateDecodingStrategy & DateEncodingStrategy

It would be nice if the Dateformat would be configurable. Maybe an optional parameter on load and save functions? Currently I dont see any other option

hi @muellers7, would you mind describing what is the specific part of the library where we can add such functionality, please?

Hey @egzonpllana,
thanks for you reply.
First of all I have to admit that I'm pretty new to swift so maybe I misunderstand something.
My use case is that I want to load and save a json file that stores date values. From what I found, the following date formats are supported by JSONDecoder:
ISO-8601: .iso8601
Seconds since 1970: .secondsSince1970
Milliseconds since 1970: .millisecondsSince1970
Custom: .formatted(formatter)
Seconds since 2001 (default: .deferredToDate
JSONDecoder has a property "dateDecodingStrategy"
and JSONEncoder has a property "dateEncodingStrategy"
so it would be possible to change the signature of the load function to (changed part in bold):
func load<T: Decodable>(withFilename filename: String, atDirectory directory: CodableFilesDirectory = .defaultDirectory, **dateDecodingStrategy: JSONDecoder.DateDecodingStrategy = JSONDecoder.DateDecodingStrategy.deferredToDate**) throws -> T

And in addition it would be neccessary to instantiate the the JSONDecoder and set it in the function (
replace from line 115 in CodableFiles ):
    let jsonDecoder = JSONDecoder()
    jsonDecoder.dateDecodingStrategy = dateDecodingStrategy
    // Load the JSON data from the file and decode it to the desired type.
    let contentData = try Data(contentsOf: filePathURL)
    return try jsonDecoder.decode(T.self, from: contentData)

For save function it would be similar but we have to use the dateEncodingStrategy instead.
Greets
Stefan

Your suggestion makes sense Stefan, thanks for explaining it. I will implement this feature this weekend and on Monday it will be ready. I will add a new parameter to provide the Encoder or Decoder for the save and load methods, so the developer can provide more options with customized decoder/encoder not just the dateDecodingStrategy.
Let me know what you think, or if you have something else in mind. Thanks!

🚀  CodableFiles (1.1.1) successfully published
📅  March 3rd, 13:11
👨💻 Changes:

Added new parameter to provider Encoder or Decoder instance.
Link to changes (v1.1.0 vs v1.1.1)
https://github.com/egzonpllana/CodableFiles/compare/1.1.0...1.1.1

Thanks a lot!


Openapi filter

Description
This change filters the OpenApi so that the graphQL schema generates from it results much more slender.

/rebase

/merge


Adding LSystems demo by Andrea and Ross

This is LSystems demo from yesterday's London meetup by @andrea and @pezipink. I still need to clean this a bit & add some web page, but to make sure I do not forget to do this, here is a work-in-progress PR :-)

I wonder if this test failure is some error?

I adjusted a bit too much the times in the test and sometimes it's failing in one of the two CI servers (I think it never did on my machine). A new build usually fixes the problem but I'll try to have a deeper look the next time it fails... I don't even remember now which test exactly was failing 😅

This is now ready to merge :+1:
I sadly cannot release the docs myself because I'm getting #310, but when I comment out the Nunit sample, everything works perfectly. If you can publish new docs, that would be splendid! :tada: :shipit:

Fantastic! :clap: This is done with master, right? Then it may be better to release the new version before updating the docs, I'll probably do it tomorrow 👍

Yes, using master. Maybe that's why i had issues with List.rev (there is a workaround for that). Tomorrow sounds good!

Merge time! Which problems did you have with List.rev btw?

I was getting some JavaScript error in fable-core.js. I thought this might be because the project was just getting fable-core from NPM and so it was getting an older version.
I added a workaround for this:

https://github.com/fsprojects/Fable/blob/master/samples/browser/lsystem/lsystem.fsx#L21

I guess this can now be safely removed....

Ok, it'll be nice if you could remove it and check that everything is working. Also, the sample is now live and I'm still getting Maximum call stack size exceeded for iterations close to 10. This is not related with the trampoline you added to Async, right?
Thanks a lot for your help!

Will have a look!
No, the stack size exceeded is a normal stack overflow exception that you get for a lot of iterations - just because it involves very deep recursion.


Fixed case where error in reading partitions can lead to using empty configuration

If e.g. a partition is being removed at a specific moment, there can be a 404 error in reading partitions from Service Fabric. Errors in reading partitions led to trying to use the invalid results, which eventually led to using effectively empty configuration until the next configuration read time. This means that any requests coming to that Traefik instance will fail for a few seconds.
Instead if there is an error when reading the partitions, return error so that the configuration is not used. Verified in practice that the partition reading errors no longer lead to empty configuration after the code fix.
Error in plugin logs without the fix:
time="2023-11-27T02:27:07Z" level=info msg="failed to get partitions: Service Fabric responded with error code 404 FABRIC_E_PARTITION_NOT_FOUND to request https://localhost:19080/Applications//$/GetServices///$/GetPartitions/?api-version=3.0 with body &{0x302720 0xc000425240 0x517500}"
time="2023-11-27T02:27:07Z" level=info msg="runtime error: invalid memory address or nil pointer dereference"

@mmatur Could you please take a look at this and related pull requests #29 and #31? These fixes together should make Service Fabric plugin more reliable, allowing us to take the Traefik 2.x versions into production use.


Deprecated HttpClient service, added ClientInterface

Fixes #425

Q
A

Bug fix?
yes

New feature?
no

BC breaks?
no

Deprecations?
yes

Related tickets
fixes #425

Documentation
if this is a new feature, link to pull request in https://github.com/php-http/documentation that adds relevant documentation

License
MIT

What's in this PR?
It deprecates exposing the deprecated Http\Client\HttpClient and adds default service Psr\Http\Client\ClientInterface.
Why?
Example Usage
Checklist

[x] Updated CHANGELOG.md to describe BC breaks / deprecations | new feature | bugfix
[ ] Documentation pull request created (if not simply a bugfix)

To Do

[ ] If the PR is not complete but you want to discuss the approach, list what remains to be done here

or would you rather remove "package" from here?
this + add it as a comment in the xml

i solved it like this in FOSHttpCacheBundle: https://github.com/FriendsOfSymfony/FOSHttpCacheBundle/blob/ef1bab770fbb2fc3f0f6a3e9c66dceb1cde58822/src/DependencyInjection/FOSHttpCacheExtension.php#L286 with https://github.com/FriendsOfSymfony/FOSHttpCacheBundle/blob/2.x/src/Resources/config/user_context_legacy_sf6.xml and https://github.com/FriendsOfSymfony/FOSHttpCacheBundle/blob/2.x/src/Resources/config/user_context_legacy.xml
the idea is to extract the deprecated alias into its own service configuration file and have the extension include the variant that works for the symfony version that is used. (i would invert the naming to what we did in the bundle, something like httplug_alias.xml for the new syntax and httplug_alias_legacy.xml for old symfony versions)


Request CSV Import creates Requests in the system with no associated Event.

Need project/product owner feedback on this issue: cc @tonysurma @MisterJames @OhMcGoo
We want to make sure when adding a Request to the Request table, that we're associating an Event to the Request.
Request's w/out an associated Event will appear "orphaned" to our UI that is responsible for assigning Requests to Itineraries.
Most likely, the same lookup mechanism that will be used to match incoming Requests from our API to Events could be used for this import as well.

Yes @mgmccarthy same rules as API ingestion should be used for CSV ingestion

@tonysurma, don't know if you have any time to chat about how we make the association with Request to Event, but regardless of if we use the zip code or somehow use the region (which we would need to figure out how to represent in our system), but it would probably help me sumbmit something that makes sense.
Maybe pulling @MisterJames in on the convo would help... Skype (in person would be best), but we could always use Slack as well

paging @OhMcGoo and @schuback as we need to get the red cross view of how to do this assignment (how do we map a request from get a smoke alarm) to the right region/event/etc. in our system.  What happens now with GetASmokeAlarm.org for instance - how do they get from their system to the right folks at ARC?  Maybe they can duplicate that for us and we can just map regions to events or something...

@mgmccarthy @tonysurma I'm not sure that I'm opposed to orphaned requests, but there are two ways that come to mind immediately to solve this.

If we decide that orphaned requests are okay, we simply display a warning/alert at the top of every event page (for admins) and prompt them to "assign" the requests to the event. This could be done via a list with a filter set at the top/selection. This would be more work, but might allow finer-grained control.
If we prefer direct assignment, we could add an event drop down as part of the CSV import process. This would be trivial but would mean that all requests would have the same event assignment (which is likely okay).

@MisterJames +1 to option 2

Option 2 works for CSV but not API but like it.  It can even be optional.
for option 1 we can have orphan (but at least tagged to org per comment above/somewhere else) and I think we can do a variation of it where the Steve's create request buttons appear.

the discussion about not associating an incoming Request with an Event has been broken out into Issue #1591

@MisterJames @tonysurma @stevejgordon, I'm going to pick up the work on Request CSV Import.
To sum up, I'll be adding an Event drop-down to the UI which will allow a user to associate this "batch" of incoming requests with a given Event.
Unless the decision is to also funnel the CSV Import to the "orphaned Requests" Organization page and then let he user associate that Request with an Event for that org?
I don't see any input on this issue or Issue #1591 about what the final decision is.

@MisterJames @tonysurma @OhMcGoo since I'm pickup up work on this, I'm assuming:

this is an all-or-nothing operation. If one or more rows of the import fails, the entire import fails
the CSV import handles new requests only (inserts)

please "yay" or "nay" this.

1.) going to rely on @MisterJames thoughts but I think the answer is 'yay'
2.) yay
is that how it was implemented in #1641 and other PRs? can we close this or is there more work?  thanks for the help clarifying

I broke out the parts of this Issue surround the CSV import only to Issue#1654


[MBOX-3759] current time in millis

What:

CurrentTimeInMillis operator which returns the time in milliseconds since the UNIX epoch (January 1, 1970 00:00:00 UTC)

Why:

To extend stdlib and offer more possibilities

Example:

see unit tests https://github.com/allegro/json-logic-kmp/pull/81/files#diff-3216d5df711298888b8fdbb68d7cc4c27bb658365b039095767316e6ce54f9b3

Tests have been fixed and they work on both separately platforms


Dropdown: Append-to-body breaks dropup

When class="dropup" and dropdown-append-to-body are used together. The dropdown no longer dropsup.

@shyamal890, please be respectful of our time and processes by adhering to the issue template.  As we are all volunteers, we simply do not have the time to chase every incomplete issue entered by users.

closing due to lack of response.

http://plnkr.co/edit/9Mc7WQ7qUbbAgQDlIpIv?p=preview

@shlomil thanks for the plunker.  i'm not seeing what the issue is.  can you please post steps to recreate?  thanks.

Sorry, I thought the plunkr was clear. There are 3 examples in the above plunkr. All use class "dropup" which means it should display the dropdown menu above the button. It works fine in normal use (3rd button) but if you try to use the dropdown-append-to-body directive, it will display the menu below the button, ignoring the dropup class.

@shlomil, ah, I understand now. Reopening.  In general, it's a good idea to explain exactly what the issue is and exact steps to show us the issue.  As we are all volunteers, we simply do not have the time to chase everything down.  Thanks.

This is likely related to the $uibPosition service usage in the dropdown - it has no knowledge of the dropup class. We need to add logic to account for that, and position it appropriately in that situation.

Any word on when the fixes are going to be reviewed and, hopefully, accepted?


[4.4] style: update cs

Description
See #7820

update cs

Checklist:

[x] Securely signed commits
[] Component(s) with PHPDoc blocks, only if necessary or adds value
[] Unit testing, with >80% coverage
[] User guide updated
[x] Conforms to style guide

Thanks!


Lite: Fully_connected op bug fixed and code refactored

1:> TF_LITE_MACRO_DISPATCH --> remove this macro unused
2:> Unnecessary check removed as it is clearly evident from code above
3:> Shift negation is not required explicitly when computed, it should be binded with code where it is used, if needed
4:> Confusing comment removed
5:> Error message improvised
6:> Code refactored to accommodate multiple if() else ()
7:> Move data type check to one central place, to improve code readability
8:> 2 Bug fixed as part of above change

The code was prone to significant data loss, if user has given Input as int16 type which was explicitly type casted to int8/uint8.
Null pointer access when Bias is not present at EvalShuffledQuantized()

9:> If condition modified to imply quantized input

Please avoid coupling refactoring changes with actual behavior changes and/or bug fixes. Can you split this CL into different pieces accordingly (with the highest priority being the bug fix)? Thanks!

Please avoid coupling refactoring changes with actual behavior changes and/or bug fixes. Can you split this CL into different pieces accordingly (with the highest priority being the bug fix)? Thanks!

@jdduke : Thanks for your suggestion, i will keep in mind for my future PRs. Because the PR contains multiple changes, i have consolidated the list of changes above with descriptions, if that helps. Otherwise should i close the current PR and raise multiple PRs, or raise multiple commits in one PR? Please let me know your convenience, TIA!

Otherwise should i close the current PR and raise multiple PRs, or raise multiple commits in one PR? Please let me know your convenience, TIA!

Prefer multiple PRs. In practice, once this change gets approved, it is landed internally as an atomic commit. Better to have it split across multiple (real) commits than a single one.

Also, it makes it easier to review and land if it focuses on a single issue/fix.

Also, it makes it easier to review and land if it focuses on a single issue/fix.

@jdduke : I am closing this PR, to raise again with multiple PRs as suggested by you.
@shahzadlone : Thanks for your comments, i will consider those in my next PRs.

Closing to raise again.


bump up go version to 1.22 in setup-environment action

missed this one in https://github.com/lf-edge/eden/pull/1012 .

I grepped this things, did we got all the pokemons?

@uncleDecart pull master and search again, poor vscode :D

My apologies, that was old master indeed. LGTM


Provide some minimal git integration

Did you check if a similar feature request has already been reported?
Checked, not reported
Is your feature request related to a problem? Please describe.
Keeping old versions of the tasks (in case some task is accidentally deleted), easing synchronization of tasks among multiple computers.
Describe the solution you'd like
todo.txt files are line based text files. This is the ideal format for taking diffs and for storing in a revision control system, such as git. Would be great to have some form of git integration modeled after what is offered in the pass password manager (https://www.passwordstore.org/) and its QtPass frontend (https://github.com/IJHack/QtPass).
This would work as follows:

When sleek realizes that a configured todo file is in a directory under git control (i.e., with a .git subdir) and that git is available, activate git integration
When git integration is active if a todo file in a directory under revision control is modified, perform a commit in that dir. Alternatively, provide a commit button for the user to manually commit the file being worked upon to git
Provide a pull and a push button, resulting in git pull and git push.

+1

Unclear to me whether this will automatically be closed also in the sleek 2.x backlog. Please, keep it open there.

Your comment should be enough to keep it open.

At least for Unix-like systems, an easy solution is to use https://github.com/gitwatch/gitwatch. Not sure, but it should  work with Cygwin/WSL/Git Bash/etc.
Also Syncthing can be quickly configured to keep a bunch of backups (might be enough in many cases -- and this is what I'm using right now.)

Most important: has the existence of the issue spotted in arch been excluded in other distros using OCaml 5.x?


The way globs in elvis.config are interpreted generates duplicated analysis of files

Even after inaka/elvis_core#111, we still have certain conditions in which we analyze some files twice.
To replicate the issue, build a project with the following subfolders:

src

m1.erl
sub1

m2.erl
sub3

m3.erl

You can try with the following values for dirs:

["src/**"]: that won't analyze m1.erl.
["src", "src/**"]: that will analyze each module twice.
["src", "src/*", "src/*/*"]: that will analyze m1.erl once, m2.erl twice and m3.erl three times.

Related to https://github.com/inaka/elvis_core/pull/124/commits/54c090e47300537465f4b2a91cdfa155dfd4848a.

Fixed by inaka/elvis_core#124

Fixed by inaka/elvis_core#124

Nice. Shall we close it?


Questions about experiments

Hi, thanks for your great contributions.
I have some questions after reading your paper:

In Table 3 (GLUE benchmark), you present Size for pre-training but do not give the corresponding unit. I know that LAION-2B contains 2B image-text pairs. But what do 0.8B, 3.3B, 4,5000B mean for language models? The number of tokens or the disk space of text files?
In Table 3 (GLUE benchmark), the model with frozen LAION-2B-pretrained backbone (i.e., Meta-Transformer-B16_F) lags behind the SOTA performance by a large margin (similarly in Table 9 - video understanding, and Table 12 - graph data understanding). In terms of this, I mentioned in issue#49 that the released Meta-Transformer weights may give inferior representations, especially for modalities like text, video and graph (because the backbone is pre-trained on LAION-2B, not trained jointly on data across 12 modalities). Do I misunderstand or overlook sth? It seems that one would have to fine-tune the backbone to get more discriminative representations (e.g., Table 3, Meta-Transformer-B16_F vs. Meta-Transformer-B16_T).

Yes, 0.8B, 3.3B, and 4,5000B are pretraining tokens for these language models.
It's insightful of your observations. Yes, Meta-Transformer delivers limited performance in video, graph, and text understanding, even though it's outstanding in point cloud, time-series, audio, and IMU data understanding. It seems that finetuning will bring better performance. But, I think that the key design is how to perform data tokenization. Meanwhile, we found that an easy approach can effectively enhance the performance in time-series forecasting tasks.


AlphaFold Colab runs keep getting stuck at AMBER relaxation step

AlphaFold Colab runs keep getting stuck at AMBER relaxation step.  Runtime type: Python 3; hardware accelerator: T4 GPU.
AMBER relaxation: 83% 5/6 [elapsed: 30:18 remaining: 06:00]
WARNING:root:Warning: importing 'simtk.openmm' is deprecated.  Import 'openmm' instead.
Sometimes completes the run; other times it crashes.

.@rukibuki in my experience the run throws up this warning and continues; most of the time the run is completed, but sometimes it does not.  There is no way to tell.  But I have noticed a strong anti-correlation between the size of the system and completion of the run.

This appears to be an issue with the latest version of openmm. Where before it would exit once converged (or reached some maximum number of iterations), now it can keep going forever if clashes remain in structure...

Thank you @sokrypton for checking.  I think the issue is not too bad but can be frustrating when no results are generated repeatedly.  On one occasion, I tried to skip the minimization step but it still gave me several warnings including the fact that I should be minimizing the structure and then failed to yield a structure.  Unfortunately, I did not save the warnings it gave.

Is it possible you have a stale copy of the Colab notebook? We removed all imports via simtk in https://github.com/google-deepmind/alphafold/commit/955737f12cc4b27398fe2369417fc2cc79815eb6.

Are you running relaxation with GPU? Sometimes it will not work. Try to run the prediction with GPU, then relax with CPU. It may take longer, but always work for me (set --use_gpu_relax=False).


Replacing LockFreeArray with TBB's concurrent_vector

This PR replace current fix size (1024*1024) LockFreeArray with Intel's TBB concurrent_vector.

Coverage increased (+0.005%) to 76.851% when pulling 581cacb7d9e3bb6cd64c83305432f7774f8ebaa9 on zhaoguoquan94:tbb into ff418e34d3060e26abed3fc043f10a9da5a4161a on cmu-db:master.

Memory usage is now reduced to 347MB. It used to be 619MB (no query executed at all)

@zhaoguoquan94 A part of your changes from this PR are already merged in the master branch. But I explicitly reverted the lock_free_array changes. So you need to add that back. Also you have mixed multiple other commits in this PR. So here is what I would suggest. Create a new branch, cherrypick changes from this PR and raise a new PR.
In the future, always use the following commands to rebase a PR:
git remote add upstream git://github.com/cmu-db/peloton.git
git fetch upstream
git rebase -i upstream/master

@tcm-marcel

What does it make lock free? (before switching to the concurrent vector)

Previously the lock free is just a big std::array of size 1048576. There is an atomic int index and each time the user append something, the atomic index get incremented.

Do we still need the LockFreeArray abstraction after switching to the TBB concurrent vector?

Actually we don't need it but I suggest we keep it. As I described here #973 ,there are two use cases but they actually want different functionalities. I remove the LockFreeArray with tbb::concurrent_unordered_map manager.h because manager don't even insert (append) stuff. It just update the elements at specific index, so I replace it with tbb::concurrent_unordered_map. The reason that I should keep the LockFreeArray abstraction is that tbb::concurrent_vector has more functionalities than we need and documentation is not well formatted.  You need to click many links until you find the correct doc page. We can keep it so that new Peloton developers can use it just by looking at the header.
I agree with @poojanilangekar it's better to create a new PR since part of this PR has been merged into master. I'll close it and create a new PR asap.


Dependencies - replace pyiron_atomistics with pyiron_io

Is it possible to instead of loading the job using pr.load() to access the output of the job use pyiron-hdf5-format to access the job? The goal would be that only a single converter has to be written and the same output mapping can be used for all pyiron jobs like LAMMPS, SPHInX and VASP. If there are quantities missing in the HDF5 files of one or the other job, then pyiron-hdf5-format should handle the conversion. In addition, using  pyiron-hdf5-format should be faster as it does not require so many imports as pyiron_atomistics. In the long term the goal would be that both the GenericOutput and the pyiron table use pyiron-hdf5-format still that might take some more time.

pyiron_io is now archived - please use pyiron_dataclasses instead.