Toggle revision history by clicking on last modified

This is dissapointing since we only have metadata.
There is no way to link to a revision or show a diff.
The only thing that can be offered is an HTML download.
Doing a direct fetch of the HTML download runs into auth issues.
Perhaps there is a way to fix this, but I don't think it is very useful
to show old revisions without any diff.
There is an unofficial URL for getting a diff,
but it is a JSON output that needs rendering.
https://docs.google.com/document/d/{file.id}/showrevision?id={file.id}&end={rev.id}&start={rev.id}

The current UI works but does not looks good :) For example, it cannot collapse and the layout doesn't look nice.

I suggest that you can:

Use Callout to make it collapsible.
Use Stack to arrange your content with some paddings to make it looks nicer.

Thanks for the PR!

It is already designed to be collapsible by clicking on the same button, that is a bug when I transitioned the state out of redux.
That is fixed now. I also adjusted the spacing and added an hr border for visual separation.


Guard against state updates on unmounted component

Getting: Warning: Can't perform a React state update on an unmounted component on a component using withWindowSize that is being re-rendered within 100ms (default throttle).
This should fix that.

Hi @jaredpalmer , there's any plans to release it? Thanks


Add to existing trims for RC trim calculation

Right now if you've previously set RC trims, but then want to adjust them, you fly with the old trims active and then set a small incremental trim on your transmitter, but then when you call the calibrate_rc_trim service it ignores the old trim and sets only the incremental trim as your new total trim. That's probably not a great behavior.
We could just have users reset the trims to zero before recalibrating, but then you can't do incremental adjustments. So we should change it to having the new trim be added to whatever trim was active while you were just flying.
This is coming from the discussion at https://discuss.rosflight.org/discussion/20/questions-regarding-trimming.

@dpkoch I just changed the code to fix it but looks like someone already fixed that part. You can close this issue.


One of the IP in S1 cluster is not working

Issue Description
Get this error: connect ETIMEDOUT <IP_ADDRESS>:51234 when using s1 cluster node https://s1.ripple.com:51234/. This happens quite a lot so can you take a look.
Steps to Reproduce
You can try ping the IP:
curl -L -X POST '<IP_ADDRESS>:51234/' \
-H 'Content-Type: application/json' \
--data-raw '{
    "method": "ping",
    "params": [
        {
        }
    ]
}'

Where did you get the ip <IP_ADDRESS> from?

Hi, this is the reproduce code. I believe that the error is thrown by jayson
const jayson = require('jayson');
const Promise = require('bluebird');

async function getLedger(ledgerIndex) {
  const rpc = Promise.promisifyAll(jayson.client.https('https://s1.ripple.com:51234/'))
  const params = [{ ledger_index: ledgerIndex }];
  return (await rpc.requestAsync('ledger', params)).result;
}

getLedger('validated').catch(console.err);

// Log
// Error: connect ETIMEDOUT <IP_ADDRESS>:51234

Appreciate the report; while this is not the correct place to report issues with the cluster of servers operated by Ripple, I will pass this on to the team that maintains/operates the cluster and the DNS zone files.

Hi, I have switched to the https://xrpl.ws node, so far I haven't encountered this error anymore. I will close this now since I can no longer report the error if it happens again.


Timeout People Picker (AzureCP used) - reprocued with SP2019 and SE (not the case if we remove AzureCP)

Hello,
We noticed an issue with People Picker when AzureCP is installed and configured (same behavior on SP2019 and also SE).
It is about a timeout (only occurring the first try after some inactivity time  ... second try - to search people- usually is OK and super fast).
After this first search try (timing out) - a Red error message is dispalyed on the bottom of the search Windows (People Picker). The detail of the error message is below:
Sorry, we're having trouble reaching the server.
Log reviewed and also verbose mode activated ... we could see easily see the timeout Unexpected record (4000ms)
I would highly appreciate if you could have a look at that and investigate more.
I am available for some troubleshooting session (please reach out to me via e-mail).
Best Regards
Verbose	[AzureCP] Getting new access token for tenant 'noxxxx.onMicrosoft.com' on cloud instance 'AzurePublic' using client ID ...
Unexpected	[AzureCP] Queries on Azure AD tenant 'noxxxx.onMicrosoft.com' exceeded timeout of 4000 ms and were cancelled. ...
...
...
...
Medium	[AzureCP] Got 0 users/groups in 8942 ms from 'noxxxx.onMicrosoft.com' with input 'ham'
...
Verbose	[AzureCP] 26 entity(ies) to create after filtering

hi @senn00, there are 2 options to troubleshoot this timeout:

Analyzing SharePoint logs to find what operation exactly takes too long (it seems it is while getting the access token but this needs to be confirmed). For that, you can filter the logs on the message contains AzureCP: It will output the monitored scope messages that AzureCP records on key operations and show how long they take individually
Additionally, you can use Fiddler as a proxy of the SharePoint site to capture the traffic to Azure and understand what exactly takes too long. This is described in this page.

Hi Yvan
Thank you for the inputs …
Had in mind to proceed with more troubleshooting using Fiddler. Will update you.
BR
From: Yvan Duhamel @.>
Sent: Wednesday, July 13, 2022 11:53 AM
To: Yvand/AzureCP @.>
Cc: Ennaifar, Sofiane @.>; Mention @.>
Subject: Re: [Yvand/AzureCP] Timeout People Picker (AzureCP used) - reprocued with SP2019 and SE (not the case if we remove AzureCP) (Issue #155)
This Message is from an External Sender. Do not click links or open attachments unless you trust the sender.
hi @senn00https://urldefense.com/v3/__https:/github.com/senn00__;!!N3hqHg43uw!tTy3Ovo4C6sMG9BfhKUOKheZGOswHKXK6HS-0FP3Xy1oaLw0FEuaZVnUiXpGSUDViuxu98lSRDshjAQGLPJ__PSxcjEvIq4$, there are 2 options to troubleshoot this timeout:

Analyzing SharePoint logs to find what operation exactly takes too long (it seems it is while getting the access token but this needs to be confirmed). For that, you can filter the logs on the message contains AzureCP: It will output the monitored scope messages that AzureCP records on key operations and show how long they take individually
Additionally, you can use Fiddler as a proxy of the SharePoint site to capture the traffic to Azure and understand what exactly takes too long. This is described in this pagehttps://urldefense.com/v3/__https:/azurecp.yvand.net/docs/help/troubleshooting/*inspect-the-traffic-to-azure-ad__;Iw!!N3hqHg43uw!tTy3Ovo4C6sMG9BfhKUOKheZGOswHKXK6HS-0FP3Xy1oaLw0FEuaZVnUiXpGSUDViuxu98lSRDshjAQGLPJ__PSxkAUFfA8$.

—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/Yvand/AzureCP/issues/155*issuecomment-1183014590__;Iw!!N3hqHg43uw!tTy3Ovo4C6sMG9BfhKUOKheZGOswHKXK6HS-0FP3Xy1oaLw0FEuaZVnUiXpGSUDViuxu98lSRDshjAQGLPJ__PSxrCcvHQU$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A2AIWUKSNL45I56VPUYSZ2DVT2GY3ANCNFSM53HVN4QA__;!!N3hqHg43uw!tTy3Ovo4C6sMG9BfhKUOKheZGOswHKXK6HS-0FP3Xy1oaLw0FEuaZVnUiXpGSUDViuxu98lSRDshjAQGLPJ__PSxmlmTGks$.
You are receiving this because you were mentioned.Message ID<EMAIL_ADDRESS>

Below the logs (already collected a couple of days ago) … If I am not wrong, not really related to the token acquisition …
Timestamp                    Process                                      TID       Area                                      Category                        EventID Level                Message           Correlation
07/11/2022 14:24:22.43             w3wp.exe (GLCHBS-SP320923:0x32CC)           0x2174            AzureCP                        Core                               1337     Verbose            [AzureCP] Getting new access token for tenant 'XYZXYZ.onMicrosoft.com' on cloud instance 'AzurePublic' using client ID 989ccf59-3183-4bb9-ac21-901d752bc06a and a client secret.            46f64fa0-f129-e0a9-7ac6-e84d1e1496f9
07/11/2022 14:24:23.08             w3wp.exe (GLCHBS-SP320923:0x32CC)           0x2DB0            AzureCP                       Core                               1337     High     [AzureCP] Got new access token for tenant 'XYZXYZ.onMicrosoft.com' on cloud instance 'AzurePublic', valid for 1 hour(s) and retrieved in 662 ms
07/11/2022 14:24:53.44             w3wp.exe (GLCHBS-SP320923:0x32CC)           0x1B24            AzureCP                       Lookup                           1337     Unexpected      [AzureCP] Queries on Azure AD tenant 'XYZXYZ.onMicrosoft.com' exceeded timeout of 4000 ms and were cancelled. 47f64fa0-316e-e0a9-7ac6-e442a9c604a7
07/11/2022 14:24:53.44             w3wp.exe (GLCHBS-SP320923:0x32CC)           0x216C            AzureCP                       Lookup                           1337     Unexpected      [AzureCP] Queries on Azure AD tenant 'XYZXYZ.onMicrosoft.com' exceeded timeout of 4000 ms and were cancelled. 49f64fa0-6158-e0a9-7ac6-eada4c37a860
07/11/2022 14:24:53.44             w3wp.exe (GLCHBS-SP320923:0x32CC)           0x20B8            AzureCP                       Lookup                           1337     Unexpected      [AzureCP] Queries on Azure AD tenant 'XYZXYZ.onMicrosoft.com' exceeded timeout of 4000 ms and were cancelled. 50f64fa0-31b5-e0a9-7ac6-e3f1dd0fbf81
07/11/2022 14:24:53.44             w3wp.exe (GLCHBS-SP320923:0x32CC)           0x1B24            AzureCP                       Lookup                           1337     Medium [AzureCP] Got 0 users/groups in 8048 ms from 'XYZXYZ.onMicrosoft.com' with input 'lamng'         47f64fa0-316e-e0a9-7ac6-e442a9c604a7
07/11/2022 14:24:53.44             w3wp.exe (GLCHBS-SP320923:0x32CC)           0x216C            AzureCP                       Lookup                           1337     Medium [AzureCP] Got 0 users/groups in 31085 ms from 'XYZXYZ.onMicrosoft.com' with input 'erich'         49f64fa0-6158-e0a9-7ac6-eada4c37a860
07/11/2022 14:24:53.44             w3wp.exe (GLCHBS-SP320923:0x32CC)           0x20B8            AzureCP                       Lookup                           1337     Medium [AzureCP] Got 0 users/groups in 8532 ms from 'XYZXYZ.onMicrosoft.com' with input 'lamn'           50f64fa0-31b5-e0a9-7ac6-e3f1dd0fbf81
07/11/2022 14:24:53.44             w3wp.exe (GLCHBS-SP320923:0x32CC)           0x2EA0            AzureCP                       Lookup                           1337     Unexpected      [AzureCP] Queries on Azure AD tenant 'XYZXYZ.onMicrosoft.com' exceeded timeout of 4000 ms and were cancelled. 48f64fa0-e173-e0a9-7ac6-e9ca79d894c6
07/11/2022 14:24:53.44             w3wp.exe (GLCHBS-SP320923:0x32CC)           0x2EA0            AzureCP                       Lookup                           1337     Medium [AzureCP] Got 0 users/groups in 8942 ms from 'XYZXYZ.onMicrosoft.com' with input 'lam' 48f64fa0-e173-e0a9-7ac6-e9ca79d894c6
07/11/2022 14:24:53.44             w3wp.exe (GLCHBS-SP320923:0x32CC)           0x3658            AzureCP                       Lookup                           1337     Medium [AzureCP] Got 0 users/groups in 1128 ms from 'XYZXYZ.onMicrosoft.com' with input 'lamngi'        45f64fa0-11fb-e0a9-7ac6-e4a4e07ad89a
07/11/2022 14:24:53.44             w3wp.exe (GLCHBS-SP320923:0x32CC)           0x2A74            AzureCP                       Lookup                           1337     Medium [AzureCP] Got 26 users/groups in 1916 ms from 'XYZXYZ.onMicrosoft.com' with input 'lamn'           2ef54fa0-41d7-e0a9-7ac6-e7d54bfdc8da
07/11/2022 14:24:53.44             w3wp.exe (GLCHBS-SP320923:0x32CC)           0x3B5C            AzureCP                       Lookup                           1337     Medium [AzureCP] Got 55 users/groups in 2338 ms from 'XYZXYZ.onMicrosoft.com' with input 'lam' 47f64fa0-416b-e0a9-7ac6-e81bf02ce2b3
07/11/2022 14:24:53.44             w3wp.exe (GLCHBS-SP320923:0x32CC)           0x3CDC            AzureCP                       Lookup                           1337     Verbose            [AzureCP] 26 entity(ies) to create after filtering       50f64fa0-6183-e0a9-7ac6-e4d9958f4b52

Hi Yvan
Thank you for the inputs …
Had in mind to proceed with more troubleshooting using Fiddler. Will update you.
BR
hi @senn00https://urldefense.com/v3/__https:/github.com/senn00__;!!N3hqHg43uw!tTy3Ovo4C6sMG9BfhKUOKheZGOswHKXK6HS-0FP3Xy1oaLw0FEuaZVnUiXpGSUDViuxu98lSRDshjAQGLPJ__PSxcjEvIq4$, there are 2 options to troubleshoot this timeout:

Analyzing SharePoint logs to find what operation exactly takes too long (it seems it is while getting the access token but this needs to be confirmed). For that, you can filter the logs on the message contains AzureCP: It will output the monitored scope messages that AzureCP records on key operations and show how long they take individually
Additionally, you can use Fiddler as a proxy of the SharePoint site to capture the traffic to Azure and understand what exactly takes too long. This is described in this pagehttps://urldefense.com/v3/__https:/azurecp.yvand.net/docs/help/troubleshooting/*inspect-the-traffic-to-azure-ad__;Iw!!N3hqHg43uw!tTy3Ovo4C6sMG9BfhKUOKheZGOswHKXK6HS-0FP3Xy1oaLw0FEuaZVnUiXpGSUDViuxu98lSRDshjAQGLPJ__PSxkAUFfA8$.

—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/Yvand/AzureCP/issues/155*issuecomment-1183014590__;Iw!!N3hqHg43uw!tTy3Ovo4C6sMG9BfhKUOKheZGOswHKXK6HS-0FP3Xy1oaLw0FEuaZVnUiXpGSUDViuxu98lSRDshjAQGLPJ__PSxrCcvHQU$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A2AIWUKSNL45I56VPUYSZ2DVT2GY3ANCNFSM53HVN4QA__;!!N3hqHg43uw!tTy3Ovo4C6sMG9BfhKUOKheZGOswHKXK6HS-0FP3Xy1oaLw0FEuaZVnUiXpGSUDViuxu98lSRDshjAQGLPJ__PSxmlmTGks$.
You are receiving this because you were mentioned.Message ID<EMAIL_ADDRESS>

Well, can you open the issue in the browser and format the log in a way where it can actually be read? Because as is I just give up trying to make sense of it.
Also, the first "Leaving Monitored Scope" is missing the time taken information

Hi – Could prepare a CSV/XLSX file but not sure if I could attach it (or upload it) …
Well, can you open the issue in the browser and format the log in a way where it can actually be read? Because as is I just give up trying to make sense of it.
Also, the first "Leaving Monitored Scope" is missing the time taken information
—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/Yvand/AzureCP/issues/155*issuecomment-1185583651__;Iw!!N3hqHg43uw!vlbK-pd0YtskKubMyH0BYqr8RO196CSS8rO10ieG_KZMAhZ0j-A--5n7Lh24EIk6-gAol91LUpk3fDR9Kg8eNmNjgnozKvU$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A2AIWUJBLZXAOOKDYHJEVGTVUFWHDANCNFSM53HVN4QA__;!!N3hqHg43uw!vlbK-pd0YtskKubMyH0BYqr8RO196CSS8rO10ieG_KZMAhZ0j-A--5n7Lh24EIk6-gAol91LUpk3fDR9Kg8eNmNj3X5KfOk$.
You are receiving this because you were mentioned.Message ID<EMAIL_ADDRESS>

Dear Yvan
Did you have the change to have a look at the log (Excel file sent the 18th of the July) ?
Also, I had a look at the CAPI2 event logs … I have a feeling that this could be related to the Revocation check related to the call to graph.microsoft.com (a full extract of the log is below – see the bottom … an particularly : - Result The revocation function was unable to check revocation because the revocation server was offline.

System

Provider

[ Name]  Microsoft-Windows-CAPI2
[ Guid]  {5bbca4a8-b209-48dc-a8c7-b23d3e5216fb}
EventID 11
Version 0
Level 2
Task 11
Opcode 2
Keywords 0x4000000000000003

TimeCreated

[ SystemTime]  2022-07-25T07:59:02.256483700Z
EventRecordID 1834
Correlation

Execution

[ ProcessID]  20416
[ ThreadID]  20400
Channel Microsoft-Windows-CAPI2/Operational
Computer XYZ_hostname.XYZ_company.net

Security

[ UserID]  S-1-5-21-329068152-854245398-839522115-2118747

UserData

CertGetCertificateChain

Certificate

[ fileRef]<PHONE_NUMBER>0C5712DCB4C2DECFB36DEBE611DC5F.cer
[ subjectName]  graph.microsoft.com
ValidationTime 2022-07-25T07:57:14.515Z

AdditionalStore

Certificate

[ fileRef]  A8985D3A65E5E5C4B2D7D66D40C6DD2FB19C5436.cer
[ subjectName]  DigiCert Global Root CA

Certificate

[ fileRef]  67D147D5DAB7F28D663CA5B7A9568F087427B9F7.cer
[ subjectName]  thawte SHA256 SSL CA

Certificate

[ fileRef]  14B4ACF94434F7D0768D3EE48D188EFD0C29137A.cer
[ subjectName]  thawte Extended Validation SHA256 SSL CA

Certificate

[ fileRef]  D69B561148F01C77C54578C10926DF5B856976AD.cer
[ subjectName]  GlobalSign

Certificate

[ fileRef]  B0C2D2D13CDD56CDAA6AB6E2C04440BE4A429C75.cer
[ subjectName]  Microsoft RSA TLS CA 02

Certificate

[ fileRef]  B17967B8B9395E2BBBBEBD5BAF89EC9CE761B645.cer
[ subjectName]  Microsoft Azure Federated SSO Certificate

Certificate

[ fileRef]  F18B538D1BE903B6A6F056435B171589CAF36BF2.cer
[ subjectName]  thawte Primary Root CA - G3

Certificate

[ fileRef]  30E01761AB97E59A06B41EF20AF6F2DE7EF4F7B0.cer
[ subjectName]  Microsoft Azure TLS Issuing CA 06

Certificate

[ fileRef]  DF3C24F9BFD666761B268073FE06D1CC8D4F82A4.cer
[ subjectName]  DigiCert Global Root G2

Certificate

[ fileRef]  AADBBC22238FC401A127BB38DDF41DDB089EF012.cer
[ subjectName]  thawte Primary Root CA - G2

Certificate

[ fileRef]  8572A527B617508118534FDE77E37569023B54A0.cer
[ subjectName]  SharePoint Root Authority

Certificate

[ fileRef]  1973F307359BC96103035790BF75DD9151E83F99.cer
[ subjectName]  thawte EV SSL CA - G2

Certificate

[ fileRef]  68060CA074FF36C7E81B0B338D7E8376790ED020.cer
[ subjectName]  thawte EV SSL CA - G3

Certificate

[ fileRef]  B418B32DB3B8CF9FDFA19CC31216852FCC8286E3.cer
[ subjectName]  GlobalSign CloudSSL CA - SHA256 - G3

Certificate

[ fileRef]  91C6D6EE3E8AC86384E548C299295C756C817B81.cer
[ subjectName]  thawte Primary Root CA

EventOverflow

[ objectCount]  7
ExtendedKeyUsage

Flags

[ value]  40000000
[ CERT_CHAIN_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT]  true

ChainEngineInfo

[ context]  serial

CertificateChain

[ chainRef]  {AA234732-A47E-42F3-AB40-6801363EBD86}

TrustStatus

ErrorStatus

[ value]  1000040
[ CERT_TRUST_REVOCATION_STATUS_UNKNOWN]  true
[ CERT_TRUST_IS_OFFLINE_REVOCATION]  true

InfoStatus

[ value]  100
[ CERT_TRUST_HAS_PREFERRED_ISSUER]  true

ChainElement

Certificate

[ fileRef]<PHONE_NUMBER>0C5712DCB4C2DECFB36DEBE611DC5F.cer
[ subjectName]  graph.microsoft.com

SignatureAlgorithm

[ oid]  1.2.840.1135<IP_ADDRESS>
[ hashName]  SHA384
[ publicKeyName]  RSA

PublicKeyAlgorithm

[ oid]  1.2.840.1135<IP_ADDRESS>
[ publicKeyName]  RSA
[ publicKeyLength]  2048

TrustStatus

ErrorStatus

[ value]  1000040
[ CERT_TRUST_REVOCATION_STATUS_UNKNOWN]  true
[ CERT_TRUST_IS_OFFLINE_REVOCATION]  true

InfoStatus

[ value]  102
[ CERT_TRUST_HAS_KEY_MATCH_ISSUER]  true
[ CERT_TRUST_HAS_PREFERRED_ISSUER]  true

ApplicationUsage

Usage

[ oid]  <IP_ADDRESS>.<IP_ADDRESS>.2
[ name]  Client Authentication

Usage

[ oid]  <IP_ADDRESS>.<IP_ADDRESS>.1
[ name]  Server Authentication

IssuanceUsage

Usage

[ oid]  <IP_ADDRESS>.2.2

RevocationInfo

RevocationResult The revocation function was unable to check revocation because the revocation server was offline.

[ value]  80092013

ChainElement

Certificate

[ fileRef]  6C3AF02E7F269AA73AFD0EFF2A88A4A1F04ED1E5.cer
[ subjectName]  Microsoft Azure TLS Issuing CA 05

SignatureAlgorithm

[ oid]  1.2.840.1135<IP_ADDRESS>
[ hashName]  SHA384
[ publicKeyName]  RSA

PublicKeyAlgorithm

[ oid]  1.2.840.1135<IP_ADDRESS>
[ publicKeyName]  RSA
[ publicKeyLength]  4096

TrustStatus

ErrorStatus

[ value]  1000040
[ CERT_TRUST_REVOCATION_STATUS_UNKNOWN]  true
[ CERT_TRUST_IS_OFFLINE_REVOCATION]  true

InfoStatus

[ value]  102
[ CERT_TRUST_HAS_KEY_MATCH_ISSUER]  true
[ CERT_TRUST_HAS_PREFERRED_ISSUER]  true

ApplicationUsage

Usage

[ oid]  <IP_ADDRESS>.<IP_ADDRESS>.1
[ name]  Server Authentication

Usage

[ oid]  <IP_ADDRESS>.<IP_ADDRESS>.2
[ name]  Client Authentication

IssuanceUsage

Usage

[ oid]  <IP_ADDRESS>.2.1

Usage

[ oid]  <IP_ADDRESS>.2.2

RevocationInfo

RevocationResult The revocation function was unable to check revocation because the revocation server was offline.

[ value]  80092013

ChainElement

Certificate

[ fileRef]  DF3C24F9BFD666761B268073FE06D1CC8D4F82A4.cer
[ subjectName]  DigiCert Global Root G2

SignatureAlgorithm

[ oid]  1.2.840.1135<IP_ADDRESS>
[ hashName]  SHA256
[ publicKeyName]  RSA

PublicKeyAlgorithm

[ oid]  1.2.840.1135<IP_ADDRESS>
[ publicKeyName]  RSA
[ publicKeyLength]  2048

TrustStatus

ErrorStatus

[ value]  0

InfoStatus

[ value]  10C
[ CERT_TRUST_HAS_NAME_MATCH_ISSUER]  true
[ CERT_TRUST_IS_SELF_SIGNED]  true
[ CERT_TRUST_HAS_PREFERRED_ISSUER]  true

ApplicationUsage

Usage

[ oid]  <IP_ADDRESS>.<IP_ADDRESS>.2
[ name]  Client Authentication

Usage

[ oid]  <IP_ADDRESS>.<IP_ADDRESS>.3
[ name]  Code Signing

Usage

[ oid]  <IP_ADDRESS>.<IP_ADDRESS>.4
[ name]  Secure Email

Usage

[ oid]  <IP_ADDRESS>.<IP_ADDRESS>.1
[ name]  Server Authentication

Usage

[ oid]  <IP_ADDRESS>.<IP_ADDRESS>.8
[ name]  Time Stamping

IssuanceUsage

[ any]  true

RevocationInfo

RevocationResult

[ value]  0

EventAuxInfo

[ ProcessName]  w3wp.exe

CorrelationAuxInfo

[ TaskId]  {55A15864-664B-47AC-ABBA-BF86A029943C}
[ SeqNumber]  10

Result The revocation function was unable to check revocation because the revocation server was offline.

[ value]  80092013

Sent: Monday, July 18, 2022 10:39 AM
Subject: RE: [Yvand/AzureCP] Timeout People Picker (AzureCP used) - reprocued with SP2019 and SE (not the case if we remove AzureCP) (Issue #155)
Dear Yvan
I collected and attached the extract of SP logs … below some extra info that could help:
Started the test (typing on People Search – Grant permission – See screen shot below) : 10:04:57 (or 58) – today CEST time
Correlation ID: 392852a0-1ee7-e074-fabb-16443b6a3573
The red error message appeared 30s after (10:04:28) – should be mapped to the below log entry
10:05:28;w3wp.exe (0x41E4);0x2E2C;AzureCP;Lookup;1337;Unexpected
[AzureCP] Queries on Azure AD tenant 'novartis.onMicrosoft.com' exceeded timeout of 4000 ms and were cancelled.
Although we see the mention of 4000ms for the timeout – my feeling --> timeout is ~30s
I am also wondering is this could be related to an issue I already reported to the MS support a couple of months ago (and I was not relly fixed) --> Certificate validation (taking 30s --> a lot of similar record in the Event log and SP log)
See one of the last record (~30s timeout)
Leaving Monitored Scope: ([AzureCP] Total time spent to query Azure AD tenant(s)) Execution Time=28932.614979467; CPU Milliseconds=0; SQL Query Count=0; Parent=SPClaimProvider.FillSearch()
I am also wondering is this could be related to an issue I already reported to the MS support a couple of months ago (and I was not relly fixed) --> Certificat validation (taking 30s --> a lot of similar record in the Event log and SP log)
Below an extract from event log (10:05:27 --> to be mapped with the SP log and unexpected event)
A certificate validation operation took 29618.3802 milliseconds and has exceeded the execution time threshold.  If this continues to occur, it may represent a configuration issue.  Please see http://go.microsoft.com/fwlink/?LinkId=246987 for more details.
@.***
@.***
A bundle of thanks in advance
Best Regards
From: Yvan Duhamel<EMAIL_ADDRESS>Sent: Friday, July 15, 2022 4:09 PM
To: Yvand/AzureCP<EMAIL_ADDRESS>Well, can you open the issue in the browser and format the log in a way where it can actually be read? Because as is I just give up trying to make sense of it.
Also, the first "Leaving Monitored Scope" is missing the time taken information
—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/Yvand/AzureCP/issues/155*issuecomment-1185583651__;Iw!!N3hqHg43uw!vlbK-pd0YtskKubMyH0BYqr8RO196CSS8rO10ieG_KZMAhZ0j-A--5n7Lh24EIk6-gAol91LUpk3fDR9Kg8eNmNjgnozKvU$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A2AIWUJBLZXAOOKDYHJEVGTVUFWHDANCNFSM53HVN4QA__;!!N3hqHg43uw!vlbK-pd0YtskKubMyH0BYqr8RO196CSS8rO10ieG_KZMAhZ0j-A--5n7Lh24EIk6-gAol91LUpk3fDR9Kg8eNmNj3X5KfOk$.
You are receiving this because you were mentioned.Message ID<EMAIL_ADDRESS>

Hi -
For the 3 Invoke-WebRequest --> perfectly working when I use my account (on the SP server and also from my laptop)
I also tried to run that using the IIS Pool Account (behind the Web Application) and I did figure out that the last  call (graph.microsoft.com) is failing ... no idea why ... do you have some clues ?
Best Regards

It is interesting that the app pool account fails to query graph.microsoft.com, it reproduces the AzureCP error.
Could it be a particular proxy setting that prevents CRL validation for this account on the SharePoint servers?

Yes, it is compatible with the CRL verification error: It times out the 1st time, then it gives up and next requests work fine.
It is not a good idea to use an interactive account as an app pool account as it can have side effects (besides the potential security issues).
It seems to me that the app pool accounts is just denied by a firewall the access to the CRL endpoints set in the graph.microsoft.com certificate. Any chance that it gets the permission to access them?

Agree 100% (reg the interactive account used with the pool). As I mentioned: idea is just to test it (temporary setup)
Could you please let me know the required permission for the app pool account and I will try to implement that.
By the way, the test server hosting this farm is not behind the FW (only FW: Windows built-in one).
Final setup: will be more complex (as the farm will be behind a Real FW + WAF)

I can only imagine a proxy/firewall configuration issue to explain why a specific AD account is not able to validate the CRL of graph.microsoft.com certificate (assuming it is the actual issue in CAPI2 logs).
If you already excluded both, sorry I'm out of idea...

I did try to add the below list of possible CRL to the local hosts file (not helpfull)
<IP_ADDRESS> crl3.digicert.com
<IP_ADDRESS> crl4.digicert.com
<IP_ADDRESS> ocsp.digicert.com
<IP_ADDRESS> www.d-trust.net
<IP_ADDRESS> root-c3-ca2-2009.ocsp.d-trust.net
<IP_ADDRESS> crl.microsoft.com
<IP_ADDRESS> oneocsp.microsoft.com
<IP_ADDRESS> ocsp.msocsp.com
Any idea how to get the full list of the CRL end points used by graph.microsoft.com ?

Update:
temporary tested (on the server hosting the POC SP farm) to use my Server Admin account (behind the pool)
perfectly working (no more timeout)
Using the dedicated pool account --> leading to the timeout issue
Any idea what is the direction to take for the troubleshooting ?

Update 2:
I did run the below PS script using the pool account --> super fast (no timeout) and I got the expected result (for 1 given account) ...
Still wondering why we have issue (timeout) when this is done through AzureCP ...
Start-Transcript -Path "C:\temp\troubleshooting_azurecp.log" -Append
#Invoke-WebRequest -Uri "https://login.windows.net" -UseBasicParsing
#Invoke-WebRequest -Uri "https://login.microsoftonline.com" -UseBasicParsing
#Invoke-WebRequest -Uri "https://graph.microsoft.com" -UseBasicParsing
get-date
$clientId = "989xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$clientSecret = "............................"
$tenantName = "company_xyz.onmicrosoft.com"
$headers = @{ "Content-Type" = "application/x-www-form-urlencoded" }
$body = "grant_type=client_credentials&client_id=$clientId&client_secret=$clientSecret&resource=https%3A//graph.microsoft.com/"
$response = Invoke-RestMethod "https://login.microsoftonline.com/$tenantName/oauth2/token" -Method "POST" -Headers $headers -Body $body
$response | ConvertTo-Json
get-date
get-date
<# My unitary test to troubleshoot AzureCP timeout issue using pool account #>
$Headers = @{
"Authorization" = "Bearer $($response.Access_Token)"
"Content-type"  = "application/json"
}
<EMAIL_ADDRESS>$apiUri = "https://graph.microsoft.com/v1.0/users/$usr"
$response2 = Invoke-RestMethod -Headers $Headers -Uri $apiUri -Method GET
$response2
get-date
Stop-Transcript

To be sure I understand correctly:

Running Invoke-WebRequest -Uri "https://graph.microsoft.com/" -UseBasicParsing as the app pool reproduces the issue
Running Invoke-RestMethod -Uri "https://graph.microsoft.com/"  as the app pool does not reproduce the issue
Is this correct?

Hi - Not really
I did implement a scheduled task (executed with the POOL ACCOUNT) in order to simply:

get the token
make a simple GET call to graph (get basic input related to a user)

PS script is below ...
The scheduled task was running since yesterday EOB (every 15mn) with the POOL ACCOUNT and it is perfectly working (getting the input related to the user) ... no timeout - super fast (1 - 2s max for the end to end process including the token acquisition ...)
This test is showing that the POOL ACCOUNT is able to interact with Graph without facing the timeout issue and/or the CRL check problem ... What do you think ?
P.S: provided below an example of the output + extract of the get-date (basically - it is taking ~1s ...)
*** script used with the scheduled task ***
Start-Transcript -Path "C:\temp\troubleshooting_azurecp.log" -Append
#Invoke-WebRequest -Uri "https://login.windows.net/" -UseBasicParsing
#Invoke-WebRequest -Uri "https://login.microsoftonline.com/" -UseBasicParsing
#Invoke-WebRequest -Uri "https://graph.microsoft.com/" -UseBasicParsing
get-date
$clientId = "989xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$clientSecret = "............................"
$tenantName = "company_xyz.onmicrosoft.com"
$headers = @{ "Content-Type" = "application/x-www-form-urlencoded" }
$body = "grant_type=client_credentials&client_id=$clientId&client_secret=$clientSecret&resource=https%3A//graph.microsoft.com/"
$response = Invoke-RestMethod "https://login.microsoftonline.com/$tenantName/oauth2/token" -Method "POST" -Headers $headers -Body $body
$response | ConvertTo-Json
get-date
get-date
<# My unitary test to troubleshoot AzureCP timeout issue using pool account #>
$Headers = @{
"Authorization" = "Bearer $($response.Access_Token)"
"Content-type" = "application/json"
}
<EMAIL_ADDRESS>$apiUri = "https://graph.microsoft.com/v1.0/users/$usr"
$response2 = Invoke-RestMethod -Headers $Headers -Uri $apiUri -Method GET
$response2
get-date
Stop-Transcript
*** last output + response time for the last runs ***
Windows PowerShell transcript start
Start time:<PHONE_NUMBER>1330
Username: DOM\POOL_DOM_ACCOUNT
RunAs User: DOM\POOL_DOM_ACCOUNT
Machine: HOSTNAME (Microsoft Windows NT 10.0.14393.0)
Host Application: C:\windows\System32\WindowsPowerShell\v1.0\powershell.EXE c:\temp\xxxxx.ps1
Process ID: 21260
PSVersion: 5.1.14393.5127
PSEdition: Desktop
PSCompatibleVersions: 1.0, 2.0, 3.0, 4.0, 5.0, 5.1.14393.5127
BuildVersion: 10.0.14393.5127
CLRVersion: 4.0.30319.42000
WSManStackVersion: 3.0
PSRemotingProtocolVersion: 2.3
SerializationVersion: <IP_ADDRESS>

Transcript started, output file is C:\temp\xxxx.log
Tuesday, July 26, 2022 9:13:31 AM
{
"token_type":  "Bearer",
"expires_in":  "3599",
"ext_expires_in":  "3599",
"expires_on":  "1658823211",
"not_before":  "1658819311",
"resource":  "https://graph.microsoft.com/",
"access_token":  "eyJ0eXAiOiJKV1QiLCJub25jZSI6IlpjVkhKN3VtLWQ3cFN6WGYyWVRxdUV5ejV1TUJnVFg0RjM3VE0yc1FIR2siLCJhbGciOiJSUzI1NiIsIng1dCI6IjJaUXBKM1VwYmpBWVhZR2FYRUpsOGxWMFRPSSIsImtpZCI6IjJaUXBKM1VwYmpBWVhZR2FYRUpsOGxWMFRPSSJ9.eyJhdWQiOiJodHRwczovL2dyYXBoLm1pY3Jvc29mdC5jb20vIiwiaXNzIjoiaHR0cHM6Ly9zdHMud2luZG93cy5uZXQvZjM1YTY5NzQtNjA3Zi00N2Q0LTgyZDctZmYzMWQ3ZGM1M2E1LyIsImlhdCI6MTY1ODgxOTMxMSwibmJmIjoxNjU4ODE5MzExLCJleHAiOjE2NTg4MjMyMTEsImFpbyI6IkUyWmdZSkFON3VrSmxUM0hmM2o1bTI2RjAvN0xBQT09IiwiYXBwX2Rpc3BsYXluYW1lIjoiU1NEVFVfUklUTTM2OTQ3NDBfMjk0NzciLCJhcHBpZCI6Ijk4OWNjZjU5LTMxODMtNGJiOS1hYzIxLTkwMWQ3NTJiYzA2YSIsImFwcGlkYWNyIjoiMSIsImlkcCI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0L2YzNWE2OTc0LTYwN2YtNDdkNC04MmQ3LWZmMzFkN2RjNTNhNS8iLCJpZHR5cCI6ImFwcCIsIm9pZCI6ImM2Njk2MzNlLWU0YWUtNDBmYS04MmNkLTdkZTQwYWQyZTQyNiIsInJoIjoiMC5BUkVBZEdsYTgzOWcxRWVDMV84eDE5eFRwUU1BQUFBQUFBQUF3QUFBQUFBQUFBQVJBQUEuIiwicm9sZXMiOlsiR3JvdXAuUmVhZC5BbGwiLCJVc2VyLlJlYWQuQWxsIl0sInN1YiI6ImM2Njk2MzNlLWU0YWUtNDBmYS04MmNkLTdkZTQwYWQyZTQyNiIsInRlbmFudF9yZWdpb.........................LjAiLCJ3aWRzIjpbIjA5OTdhMWQwLTBkMWQtNGFjYi1iNDA4LWQ1Y2E3MzEyMWU5MCJdLCJ4bXNfdGNkdCI6MTM5NzU5MTk0MX0.ifcxJMD1nt0fENJFAwIuNI6yWBfIaprWoU-hc8Ya7_aLJAzCjOIkf8pQ8XVhJewzI8ZHoRpmcN_fGdfjgsF0ZQLqhm7U9cTZMG1nE3wSy8o6ise2LCjunR2jM-oGl0-xdChFtVjC3uxA2cvsnfi4BUFfP5v_9GPECNG7Y4LDwP2cYjbLPRZqwS8oJC0VXMcaVXfM6ySJWB1qr178begm51INAoooCg16aMddmXVd5DCMkfeAFnspnNFzxiDR2t9v1h46IA9CjNNI66mYV018xFi7fi61sCiNa0jgHbBbTht2Eo077MUnWYEQXqkKVs-3JfennOo-OJ752HZplC5qow"
}
Tuesday, July 26, 2022 9:13:32 AM
Tuesday, July 26, 2022 9:13:32 AM
@odata.context    : https://graph.microsoft.com/v1.0/$metadata#users/$entity
businessPhones    : {+xxxxx}
displayName       : xxx_family name, xxx_first_name
givenName         : xxx_first_name
jobTitle          : xxx_job_title
mail<EMAIL_ADDRESS>mobilePhone       : +xxxxx
officeLocation    : xxxxx
preferredLanguage :
surname           : xxxx
userPrincipalName<EMAIL_ADDRESS>id                : 76781xxxxxx
Tuesday, July 26, 2022 9:13:32 AM

Windows PowerShell transcript end
End time:<PHONE_NUMBER>1332

Tuesday, July 26, 2022 7:13:30 AM
Tuesday, July 26, 2022 7:13:31 AM
Tuesday, July 26, 2022 7:13:31 AM
Tuesday, July 26, 2022 7:13:31 AM
Tuesday, July 26, 2022 7:28:30 AM
Tuesday, July 26, 2022 7:28:30 AM
Tuesday, July 26, 2022 7:28:30 AM
Tuesday, July 26, 2022 7:28:31 AM
Tuesday, July 26, 2022 7:43:30 AM
Tuesday, July 26, 2022 7:43:30 AM
Tuesday, July 26, 2022 7:43:30 AM
Tuesday, July 26, 2022 7:43:31 AM
Tuesday, July 26, 2022 7:58:30 AM
Tuesday, July 26, 2022 7:58:30 AM
Tuesday, July 26, 2022 7:58:30 AM
Tuesday, July 26, 2022 7:58:31 AM
Tuesday, July 26, 2022 8:13:30 AM
Tuesday, July 26, 2022 8:13:30 AM
Tuesday, July 26, 2022 8:13:30 AM
Tuesday, July 26, 2022 8:13:31 AM
Tuesday, July 26, 2022 8:28:30 AM
Tuesday, July 26, 2022 8:28:30 AM
Tuesday, July 26, 2022 8:28:30 AM
Tuesday, July 26, 2022 8:28:31 AM
Tuesday, July 26, 2022 8:43:30 AM
Tuesday, July 26, 2022 8:43:31 AM
Tuesday, July 26, 2022 8:43:31 AM
Tuesday, July 26, 2022 8:43:31 AM
Tuesday, July 26, 2022 8:58:31 AM
Tuesday, July 26, 2022 8:58:32 AM
Tuesday, July 26, 2022 8:58:32 AM
Tuesday, July 26, 2022 8:58:32 AM
Tuesday, July 26, 2022 9:13:31 AM
Tuesday, July 26, 2022 9:13:32 AM
Tuesday, July 26, 2022 9:13:32 AM
Tuesday, July 26, 2022 9:13:32 AM

Update 2 (for Tuesday 26th of July)
Hi Yvan
I am not sure if the simple unitarian test using
Invoke-WebRequest -Uri "https://graph.microsoft.com" -UseBasicParsing
is REALLY RELEVANT for the below reasons:

this last invoke is not working with my admin account (this last was used as pool account and leading to the expected result - no timeout)
there is a redirection when you simply try https://graph.microsoft.com (did try it on the server - see output below)
adding after the https://graph.microsoft.com/v1.0 will be enough to have an expected result (200 http code ... no more error) with the POOL account : SEE BELOW

different outputs
*** related to redirection as raised in point 2 ***
This site can’t be reacheddeveloper.microsoft.com took too long to respond.
Try:
Checking the connection
Checking the proxy and the firewall
Running Windows Network Diagnostics
ERR_CONNECTION_TIMED_OUT
*** related to point 3 ***
PS C:\windows\system32> Invoke-WebRequest -Uri "https://graph.microsoft.com/"
Invoke-WebRequest : Unable to connect to the remote server
At line:1 char:1

Invoke-WebRequest -Uri "https://graph.microsoft.com/"

  + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
  + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

PS C:\windows\system32> Invoke-WebRequest -Uri "https://graph.microsoft.com/v1.0"
StatusCode        : 200
StatusDescription : OK
Content           : {"@odata.context":"https://graph.microsoft.com/v1.0/$metadata","value":[{"name":"invitations","kind":"EntitySet",
"url":"invitations"},{"name":"users","kind":"EntitySet","url":"users"},{"name":"applica...
RawContent        : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Strict-Transport-Security: max-age=31536000
request-id: f18262bf-ca68-4902-a59e-a91ba3fd5874
client-request-id: f18262bf-ca68-4902-a59e-a91ba3fd5874
x-m...
Forms             : {}
Headers           : {[Transfer-Encoding, chunked], [Strict-Transport-Security, max-age=31536000], [request-id,
f18262bf-ca68-4902-a59e-a91ba3fd5874], [client-request-id, f18262bf-ca68-4902-a59e-a91ba3fd5874]...}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : mshtml.HTMLDocumentClass
RawContentLength  : 4435

I found this script (credit) that can download the certificate chain of a specific site locally:
$webRequest = [Net.WebRequest]::Create("https://graph.microsoft.com/")
$webRequest.AllowAutoRedirect = $false
$webRequest.GetResponse()
$cert = $webRequest.ServicePoint.Certificate
$chain = New-Object -TypeName System.Security.Cryptography.X509Certificates.X509Chain
$chain.build($cert)
$chain.ChainElements.Certificate | % {set-content -value $($_.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)) -encoding byte -Path "$pwd\$($_.Thumbprint).cer"}

I'm not sure it can be really useful here, but at least you can see the CRL endpoints that need to be reached.
Did you consider to run Fiddler as a proxy of the site?

Hi Yvan
Adding all the certificate chain did not help at all ...
Tried with fiddler but I am not able to capture the calls (graph) or the crl tries ...
Maybe you need to guide me what to do exactly with Fiddler ...
Thank you in advance for you patience ;-)

Hi Yvan
Good News: Setting the Proxy on the server helped to fix the issue (after collecting trace with netsh - discovered many retransmission for the <IP_ADDRESS> -->www.microsoft.com --> it should be used for the CRL Distribution Points (check the details behind of the certificate used with graph.microsfot.com/v1.0).
Now: We need to find a solution as the Proxy could no be used for our final env (located behind a WAF and a FW).
Do you see an option to make AzureCP bypass the CRL check ? Actually, using PS (graph call), it seems that there is no CRL check ...
It will be great if we could have this adjustment ...
Best Regards

1 extra piece of details ... in our setup - servers could easily communicate over https with graph.micorsoft.com and also login.microsoft.com ... see below
But for CRL, it seems that we need www.microsoft.com over 80 (not always acceptable by the security experts !)
see the behavior from 1 of our servers !
PS C:\windows\system32> tnc -ComputerName www.microsoft.com -port 80
WARNING: TCP connect to www.microsoft.com:80 failed
WARNING: Ping to www.microsoft.com failed -- Status: TimedOut
ComputerName           : www.microsoft.com
RemoteAddress          : <IP_ADDRESS>
RemotePort             : 80
InterfaceAlias         : Ethernet0 2
SourceAddress          : <IP_ADDRESS>
PingSucceeded          : False
PingReplyDetails (RTT) : 0 ms
TcpTestSucceeded       : False
PS C:\windows\system32> tnc -ComputerName graph.microsoft.com -port 443
ComputerName     : graph.microsoft.com
RemoteAddress    : <IP_ADDRESS>
RemotePort       : 443
InterfaceAlias   : Ethernet0 2
SourceAddress    : <IP_ADDRESS>
TcpTestSucceeded : True

Hi Yvan
back to you to know if you had the chance to look at my last update.
Still wondering if you could Skip/Disable the Online CRL checking for AzureCP* ...
It is the case when PS is used (to make some graph call - we are extensively using that with servers protected by FWs).
*/ and/or offer to option to enable/disable the CRL check through the GUI as it is the case for Edge
https://www.ssl.com/blogs/how-do-browsers-handle-revoked-ssl-tls-certificates/#ftoc-heading-3
Also: I did check Chrome and FireFox (on a server without any Proxy ... only access to graph and MS login end points ...)
Super fast response on Chrome (https://graph.microsoft.com/v1.0) --> less than 100ms (Network tracing)
Did empty the CRL cache before the test and checked after the call --> empty
C:\Users\my_srv_account> certutil -urlcache CRL
...
...
...
http://ctldl.windowsupdate.com/msdownload/update/v3/static/trustedr/en/pinrulesstl.cab
...
WinHttp Cache entries: 6
...
PS C:\Users\my_srv_account> certutil -urlcache CRL
...
...
...
...
CertUtil: -URLCache command completed successfully.

Just to clarify, it is not AzureCP that does CRL check, or not even IIS, it is the crypto API of Windows.
It is possible to deactivate it on a per-user basis but I did that a very long time ago and I don't know how to do it now.
But you should not do it: It is a security mechanism to guarantee that a certificate was not compromised, it should never be deactivated in a production environment.

Great … Will prepare the full set of logs (on SE & 2019) and upload the files
—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/Yvand/AzureCP/issues/155*issuecomment-1185588765__;Iw!!N3hqHg43uw!ogd9MoFmZPFHFtgHW0jhJ5Y-zwzGVKx0TJw9AgLwBKEFuwNcbM1wzFgsfxltyBXJ5K7hPKz2cL0kgU5atjEb55JIwTeYajM$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A2AIWUKOGFZ4MLQIWFOYDRDVUFW4NANCNFSM53HVN4QA__;!!N3hqHg43uw!ogd9MoFmZPFHFtgHW0jhJ5Y-zwzGVKx0TJw9AgLwBKEFuwNcbM1wzFgsfxltyBXJ5K7hPKz2cL0kgU5atjEb55JIRfruXsc$.
You are receiving this because you were mentioned.Message ID<EMAIL_ADDRESS>

Here we go (will do tomorrow more tests and collect more logs) …
Timestamp                    TID       Area                               Category                        Level                 Message
18:53:42           0x3480 SharePoint Foundation               Config Cache                              Medium Refreshed SPPersistedObject with Name AzureCPConfig, Id 0e9f8fb6-b314-4ccc-866d-dec0be76c237, Parent 91f66da3-a131-4a61-b3d5-2dda25631a9e, Version 1765658. Last update info: [User: DOMAIN\AD_ACCOUNT Process:w3wp (14224) Machine:Server-name Time:July 07, 2022 17:09:00.0000 Stack trace (Thread [17] CorrelationId [1eb64ea0-0113-e0a9-7ac6-ef27af952086]): ]
18:53:42           0x3480 AzureCP                       Core                               Medium [AzureCP] Configuration 'AzureCPConfig' changed to version 1765658, refreshing local copy
18:53:42           0x3480 AzureCP                       Core                               Verbose            [AzureCP] Refreshing local copy of configuration 'AzureCPConfig'
18:54:12           0x308C SharePoint Foundation               Config Cache                              Medium Refreshed SPPersistedObject with Name AzureCPConfig, Id 0e9f8fb6-b314-4ccc-866d-dec0be76c237, Parent 91f66da3-a131-4a61-b3d5-2dda25631a9e, Version 1765658. Last update info: [User: DOMAIN\AD_ACCOUNT Process:w3wp (14224) Machine:Server-name Time:July 07, 2022 17:09:00.0000 Stack trace (Thread [17] CorrelationId [1eb64ea0-0113-e0a9-7ac6-ef27af952086]): ]
18:54:12           0x308C AzureCP                       Core                               Medium [AzureCP] Configuration 'AzureCPConfig' changed to version 1765658, refreshing local copy
18:54:12           0x308C AzureCP                       Core                               Verbose            [AzureCP] Refreshing local copy of configuration 'AzureCPConfig'
18:54:51           0x24AC AzureCP                       Core                               Verbose            [AzureCP] Getting new access token for tenant 'XYZXYZ.onMicrosoft.com' on cloud instance 'AzurePublic' using client ID 989ccXYZXYZXYZ and a client secret.
18:54:55           0x4500 AzureCP                       Core                               Unexpected            [AzureCP] Could not get access token before timeout of 4000 ms for tenant 'XYZXYZ.onMicrosoft.com' on cloud instance 'AzurePublic'
18:54:55           0x308C AzureCP                       Core                               Verbose            [AzureCP] Getting new access token for tenant 'XYZXYZ.onMicrosoft.com' on cloud instance 'AzurePublic' using client ID 989ccXYZXYZXYZ and a client secret.
18:54:59           0x4614 AzureCP                       Core                               Unexpected            [AzureCP] Could not get access token before timeout of 4000 ms for tenant 'XYZXYZ.onMicrosoft.com' on cloud instance 'AzurePublic'
18:54:59           0x2EF8 AzureCP                       Core                               Verbose            [AzureCP] Getting new access token for tenant 'XYZXYZ.onMicrosoft.com' on cloud instance 'AzurePublic' using client ID 989ccXYZXYZXYZ and a client secret.
18:55:03           0x3E3C AzureCP                       Core                               Unexpected            [AzureCP] Could not get access token before timeout of 4000 ms for tenant 'XYZXYZ.onMicrosoft.com' on cloud instance 'AzurePublic'
18:55:03           0x37F0 AzureCP                       Core                               Verbose            [AzureCP] Getting new access token for tenant 'XYZXYZ.onMicrosoft.com' on cloud instance 'AzurePublic' using client ID 989ccXYZXYZXYZ and a client secret.
18:55:07           0x3FC0 AzureCP                       Core                               Unexpected            [AzureCP] Could not get access token before timeout of 4000 ms for tenant 'XYZXYZ.onMicrosoft.com' on cloud instance 'AzurePublic'
18:55:07           0x4530 AzureCP                       Core                               Verbose            [AzureCP] Getting new access token for tenant 'XYZXYZ.onMicrosoft.com' on cloud instance 'AzurePublic' using client ID 989ccXYZXYZXYZ and a client secret.
18:55:11           0x4544 AzureCP                       Core                               Unexpected            [AzureCP] Could not get access token before timeout of 4000 ms for tenant 'XYZXYZ.onMicrosoft.com' on cloud instance 'AzurePublic'
Could you filter the logs on Message (not Product/Area) contains AzureCP?
Doing this will show the monitored scope specific to AzureCP, and help to see why this happens
—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/Yvand/AzureCP/issues/155*issuecomment-1183400290__;Iw!!N3hqHg43uw!okpj5Fi22ovSLpizB-c_MwjBCfpRuxrezaeQrieOqpeZrEn3beKsO9-jaz0cF5ZZt0SRLvLw3M2pcndUEvtE6tVvEsYIlg0$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A2AIWUIHTZFW7WNBAUK3DA3VT3RO3ANCNFSM53HVN4QA__;!!N3hqHg43uw!okpj5Fi22ovSLpizB-c_MwjBCfpRuxrezaeQrieOqpeZrEn3beKsO9-jaz0cF5ZZt0SRLvLw3M2pcndUEvtE6tVvDV3DK9Q$.
You are receiving this because you were mentioned.Message ID<EMAIL_ADDRESS>

So AzureCP is not able to get an access token.
Can you try the scripts documented in https://azurecp.yvand.net/docs/help/troubleshooting/#test-connectivity-with-azure-ad and confirm if you can get the access token using PowerShell?


How can I reduce GPU memory consumption？

Checklist

I have searched related issues but cannot get the expected help.
I have read related documents and don't know what to do.

Describe the question you meet
I started training on 2080Ti GPU, but encoutered OOM error, how can I reduce the memory consumption to solve it?
The training command is
 mim train diffengine stable_diffusion_v15_dreambooth_lora_dog.py \
  --cfg-options train_dataloader.dataset.class_image_config.num_images=50 \
  train_dataloader.batch_size=1 \
  train_dataloader.dataset.class_image_config.recreate_class_images=False \
  train_cfg.max_iters=1000 \
  optim_wrapper.type='AmpOptimWrapper' \
  optim_wrapper.dtype='float16' \
  optim_wrapper.optimizer.lr=1e-4 \
  model.lora_config.rank=8

The error message is
RuntimeError: class `IterBasedTrainLoop` in mmengine/runner/loops.py: class `HFDreamBoothDataset` in diffengine/datasets/hf_dreambooth_d│$
atasets.py: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 10.75 GiB total capacity; 9.07 GiB already allocated; 248.50 MiB f│
ree; 9.20 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentati│
on.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

@wuwenbin970731 You can try using model.gradient_checkpointing=True. In my experiments, it reduced memory usage from 8473MiB to 5989MiB.
mim train diffengine stable_diffusion_v15_dreambooth_lora_dog.py \
  --cfg-options train_dataloader.dataset.class_image_config.num_images=50 \
  train_dataloader.batch_size=1 \
  train_dataloader.dataset.class_image_config.recreate_class_images=False \
  train_cfg.max_iters=1000 \
  optim_wrapper.type='AmpOptimWrapper' \
  optim_wrapper.dtype='float16' \
  optim_wrapper.optimizer.lr=1e-4 \
  model.lora_config.rank=8 model.gradient_checkpointing=True

Thanks for your reply. I set model.gradient_checkpointing=True, but still encounter OOM error.
The OOM error is invoked at HFDreamBoothDataset, I am not familiar with SD, why it would use GPU in Dataset? Could I avoid using it?

Thanks, it works. I will learn more about stable diffusion later.
Thanks for your great work.


Add missing As imports.

Fixes build on master.

It doesn't fix build for me, still get the following:
| => ./scripts/build.sh --verbose
*** Building wasm binaries in core/executor/wasm
    Finished release [optimized] target(s) in 0.18s
*** Building wasm binaries in node/runtime/wasm
   Compiling srml-timestamp v0.1.0 (/Users/jonatan/Projects/substrate/srml/timestamp)
   Compiling srml-session v0.1.0 (/Users/jonatan/Projects/substrate/srml/session)
error[E0599]: no function or associated item named `sa` found for type `<T as system::Trait>::BlockNumber` in the current scope
  --> /Users/jonatan/Projects/substrate/srml/session/src/lib.rs:91:74
   |
91 |         pub SessionLength get(length) config(session_length): T::BlockNumber = T::BlockNumber::sa(1000);
   |                                                                                ^^^^^^^^^^^^^^^^^^ function or associated item not found in `<T as system::Trait>::BlockNumber`
   |
   = help: items from traits can only be used if the trait is in scope
help: the following trait is implemented but not in scope, perhaps add a `use` for it:
   |
44 | use primitives::traits::As;
   |

error: aborting due to previous error

For more information about this error, try `rustc --explain E0599`.
error: Could not compile `srml-session`.

To learn more, run the command again with --verbose.


As arguments to a function, constrained generic function types with a generic return type behave oddly.

TypeScript Version:  3.1.0-dev.201xxxxx

Search Terms:
generic argument constraint function generic return type
Code
const Function1 = <R, F extends (...args: any[]) => R>(f: F): R => {
    return f()
}

const Function2 = <F extends (...args: any[]) => any>(f: F): ReturnType<F> => {
    return f()
}

const Function3 = () => 5

const Example1 = Function1(Function3) // typeof Example1 === {}
const Example2 = Function2(Function3) // typeof Example2 === number

Expected behavior:
typeof Example1 === number
Actual behavior:
typeof Example1 === {}
I am fairly new to TypeScript, so perhaps I am missing some crucial information on the correct usage of  generics.
Playground Link
Related Issues:

Constraints aren't inference sites for other type parameters.
As an aside, what you should probably write is
const Function4 = <R>(f: (...args: any[]) => R): R => {
    return f()
}

Ahhh, okay. Thank you for the quick response. Where could I find more information regarding this topic?


NPoS Challenge Mode

Builds on #8160. Only 741b488 and subsequent are distinct to this PR.

Updates the PJR check to return a counterexample if one exists
Adds functions to cheaply check counterexamples

This is in support of off-chain PJR challenges: if a miner discovers
that an accepted election solution does not satisfy PJR, it will be
eligible for substantial rewards. This helps ensure that validator
elections have an absolute quality floor, so even if someone manages
to censor well-behaved solutions to give themselves unfair representation,
we can catch them in the act and penalize them.

You probably intended to change this to merge into your other branch?

I'll follow whatever the convention is; my intent was to rebase once #8160 is merged.

The code doesn't iterate over all candidates, except to the degree that it searches the candidate list for the one of interest. However, it does require the full list of supports, all candidate, all voters. The chain of requirements goes like this:

To validate a PJR challenge fundamentally requires calculating its pre_score
pre_score requires calculating the slack for each voter
slack requires an accurate list of the voter's edges: all the candidates it voted for, and by how much
Therefore we need, bare minimum, &[Voter<AccountId>].
Voter is a struct with private fields and a complicated internal structure. It would be hard for users to accurately construct it on their own even if we made the fields public or provided a constructor.
We can construct the list of voters simply by reusing fn prepare_pjr_input, which requires the full list of supports, candidates, voters.

A reasonable argument could be, "but why not extract a subset of prepare_pjr_input which can handle constructing the inputs relevant to a single candidate?" My answer to that argument is that it's more trouble than it's worth. This module assumes that the lists of supports, all candidates, all voters are available to modules who care about the election. Making those inputs a prerequisite for counterexample proofs seems to be a small additional burden.

My answer to that argument is that it's more trouble than it's worth. This module assumes that the lists of supports, all candidates, all voters are available to modules who care about the election. Making those inputs a prerequisite for counterexample proofs seems to be a small additional burden.

We will surely need to run the pjr-counter-example check on-chain and per proof that we receive. I suppose this could be due to you not being yet very familiar with the internals of frame, but something that runs on-chain and can be triggered by the user is def. worth being optimized.
All in all, we could also meet in the middle and merge this as-is and update the API once the challenge phase is in. But note that that is not imminent and we first need to finish the unsigned + signed phase, so I see it happening in the next ~3 months or so, based on how fast we move.

but something that runs on-chain and can be triggered by the user is def. worth being optimized.

Could we apply an economic incentive instead? We're already planning on incentivizing users to submit counterexamples by providing some reward to them. Couldn't we also apply some penalty in the even that the user submits an invalid counterexample?

Reduced to unique commits; should be easier to review now.

bot merge


Document overrides and overridesas

You probably already have this on your todo list, but I think documenting these, especially for how to use them for unit tests would be very good to have, as all of the tests in the examples so far don't seem to make use of this.

I've added how to use them in the testing cookbook: https://riverpod.dev/docs/cookbooks/testing


Only compile tests when testing

We need to add #[cfg(test)] to our tests.
Might need another issue to ensure we're doing idiomatic Rust testing with how tests are laid out and their module names.

See https://doc.rust-lang.org/stable/book/testing.html .


[ Interface ][ Stats ]. Prevent tab swiping here

You still can click on bottom tab. But you dont need to navigate from here using swipe - its unclear

Tab swiping removed due inconsistency.


Suspected high concurrency caused the operation to be cancelled

The operation was canceled.

Suspected high concurrency caused the operation to be cancelled

The error message is not from ffuf but from OS maybe? Closing this issue as it seems stale.


Optimizations, linting, handling lief exceptions

Refactoring, optimizing, linting

I have refactored to make the code easier to maintain and understand, as well as to reduce unnecessary resource use .
Where possible and practical, I have extracted sections of very long functions out to make the code more modular and easier to test (we really must improve our coverage)
I have removed all but "missing-module-docstring" and "logging-fstring-interpolation" from our pylint disable list. The refactoring resolved almost everything else.
I increased the max line length to 99 since 80 was decreasing readability rather than increasing it.
What the refactoring did not resolve, or where there is a good reason for ignoring the pylint warning, I have addressed with pylint disable flags.

Minor bugfix
I improved the logic for checking dll characteristics so that we report all missing characteristics rather than only the first one found.
Lief issues
This PR also addresses remaining issues with the lief upgrade from 0.12.3 -> 0.14.0 (and upgrades to 0.14.1). These issues pertain to  lief no longer raising exceptions but returning an instance of a lief exception object so that we are unable to know whether there has been a failure unless we check type. We also won't be able to rely on simple if exists statements since these will evaluate to True when lief has actually returned an exception object.
My attempt to handle this change consists of:

checking type where a lief exception object is going to result in an obvious Attribute error. E.g. we get binary.a and then later expect to be able to do binary.a.b.
Removing lief.lief_error instance objects from our metadata at the end of the scan.

Given how extensively we have previously been able to rely on checking for truthiness and on lief exceptions being raised, it is probable there may be additional changes that need to be made that I have not anticipated.

poetry run blint sbom -i /mnt/work/sandbox/apk_tests -o /mnt/work/sandbox/apk_tests/bom-bumble.json --deep
ERROR    Caught <class 'AttributeError'>: 'AndroidIdent' object has no attribute 'is_core' while parsing
                    /tmp/blint_android_appxy6reend/lib/arm64-v8a/libsentry-android.so.
                    Traceback (most recent call last):
                      File "/mnt/work/owasp-depscan/blint/blint/binary.py", line 662, in parse
                        metadata = add_elf_metadata(exe_file, metadata, parsed_obj)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      File "/mnt/work/owasp-depscan/blint/blint/binary.py", line 724, in add_elf_metadata
                        metadata["notes"] = parse_notes(parsed_obj)
                                            ^^^^^^^^^^^^^^^^^^^^^^^
                      File "/mnt/work/owasp-depscan/blint/blint/binary.py", line 67, in parse_notes
                        data.extend(extract_note_data(idx, note) for idx, note in enumerate(notes))
                      File "/mnt/work/owasp-depscan/blint/blint/binary.py", line 67, in <genexpr>
                        data.extend(extract_note_data(idx, note) for idx, note in enumerate(notes))
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      File "/mnt/work/owasp-depscan/blint/blint/binary.py", line 89, in extract_note_data
                        type_str = note.type_core if note.is_core else note.type
                                                     ^^^^^^^^^^^^
                    AttributeError: 'AndroidIdent' object has no attribute 'is_core'


Is there support for Upload and subscriptions?

no because those are not built-in to GraphQL, they are external features. There's no middleware engine either. I think you have to manually extend the lib to add those features.


Fix file extension checks for tar compressed files

Addresses #217

I am closing this. I mistook some other error for this one. The existing code achieves the same as the change I made.


Blogs Ideas

Data Analysis:
Bika's new career - gems not fully explored - easy /
Statistics: Hamza's background - rare content (End to end projects) / Complex
Data Science:
ba7r / Wide market (choose topics wisely)

[ ] - [ ] gtsummary in python using rpy2

Data viz: Bika's Passion /

[ ] Power BI

Roadmaps:
Tools Guide:

[ ] Kaggle book summary

Data Analysis:

Bika's` new career - gems not fully explored - easy /

Statistics:

Hamza's background - rare content (End to end projects) / Complex

Data Science:

ba7r / Wide market (choose topics wisely)

Data viz:

Bika's Passion /

Roadmaps:
Tools Guide:


Clang flag

See #331

This change is

Not ready for review, but welcome - if you know alternative for macro _GNU_SOURCE on MacOSX.

Reviewed 10 of 12 files at r1, 4 of 4 files at r2.
Review status: all files reviewed at latest revision, 13 unresolved discussions, some commit checks broke.

a discussion (no related file):
There is a merge conflict from an earlier PR that added some stats to the PR result.

main.go, line 328 at r2 (raw file):

func init() {
	transpileCommand.Var(&clangFlags, "clang-flag", "input C code and specific flags for clang")

"Pass arguments to clang. You may provide multiple -clang-flag items."

main_test.go, line 338 at r2 (raw file):

	for pos, tc := range tcs {
		t.Run(fmt.Sprintf("Test %d", pos), func(t *testing.T) {
			var args = ProgramArgs{}

We must always use DefaultProgramArgs() and override all the setting you need to keep it consistent.

main_test.go, line 372 at r2 (raw file):

func TestTrigraph(t *testing.T) {
	var args = ProgramArgs{}

We must always use DefaultProgramArgs() and override all the setting you need to keep it consistent.

main_test.go, line 405 at r2 (raw file):

func TestExternalInclude(t *testing.T) {
	var args = ProgramArgs{}

We must always use DefaultProgramArgs() and override all the setting you need to keep it consistent.

tests/multi2/head.h, line 5 at r2 (raw file):

#include <stdio.h>

void say_four(){

These should be just prototypes, the whole header file should be:
#include <stdio.h>

#ifndef HEADER
#define HEADER

void say_four();
void say_two();

#endif /* HEADER */

tests/multi2/main.c, line 1 at r2 (raw file):

#include"head.h"

If you are going to use the header file for forward-declaring functions you should place the actual functions after the main.

tests/multi2/main.c, line 3 at r2 (raw file):

#include"head.h"

#ifndef HEADER

*.c files should not contains these conditional pragmas because C files are not used in #include. Please remove the #if and #define lines.

tests/multi3/1.c, line 4 at r2 (raw file):

#ifndef SQLITE
#define SQLITE

Remove these lines.

tests/multi3/1.c, line 13 at r2 (raw file):

#include <stdio.h>

int main(){

I don't understand what these files are testing that is different from multi2?

tests/multi3/1.h, line 4 at r2 (raw file):

#define SQLITE

void sql(){

void sql();

tests/multi4/include/head.h, line 3 at r2 (raw file):

#include <stdio.h>

void print(){

Add the comment above:
// multi4 tests that functions can be declared in the header.

tests/trigraph/main.c, line 3 at r2 (raw file):

#include <stdio.h>
int main(){
        printf("??=");

Add the comment above:
// Trigraph tests require the `-trigraph` clang option, so it cannot be amount other general tests.

This also raised the question. You should have the same file in the tests directory to make sure the default behaviour is also the same.

Comments from Reviewable

Review status: 12 of 14 files reviewed at latest revision, 13 unresolved discussions.

a discussion (no related file):
Previously, elliotchance (Elliot Chance) wrote…
There is a merge conflict from an earlier PR that added some stats to the PR result.

Ok.

main.go, line 328 at r2 (raw file):
Previously, elliotchance (Elliot Chance) wrote…
"Pass arguments to clang. You may provide multiple -clang-flag items."

Done.

main_test.go, line 338 at r2 (raw file):
Previously, elliotchance (Elliot Chance) wrote…
We must always use DefaultProgramArgs() and override all the setting you need to keep it consistent.

Done.

main_test.go, line 372 at r2 (raw file):
Previously, elliotchance (Elliot Chance) wrote…
We must always use DefaultProgramArgs() and override all the setting you need to keep it consistent.

Done.

main_test.go, line 405 at r2 (raw file):
Previously, elliotchance (Elliot Chance) wrote…
We must always use DefaultProgramArgs() and override all the setting you need to keep it consistent.

Done.

tests/multi2/head.h, line 5 at r2 (raw file):
Previously, elliotchance (Elliot Chance) wrote…
These should be just prototypes, the whole header file should be:
#include <stdio.h>

#ifndef HEADER
#define HEADER

void say_four();
void say_two();

#endif /* HEADER */

Done.

tests/multi3/1.c, line 4 at r2 (raw file):
Previously, elliotchance (Elliot Chance) wrote…
Remove these lines.

No. case multi3 is prototype of SQLITE.

tests/multi3/1.c, line 13 at r2 (raw file):
Previously, elliotchance (Elliot Chance) wrote…
I don't understand what these files are testing that is different from multi2?

Main different is - using two *.c files - like in SQLITE, it is happen because if you run clang -E 1.c 2.c and you have some #define LIB in both files, then clang don't  defined for both files.

tests/multi3/1.h, line 4 at r2 (raw file):
Previously, elliotchance (Elliot Chance) wrote…
void sql();

No. It is really void sql(){};. It is not a mistake.

tests/multi4/include/head.h, line 3 at r2 (raw file):

// multi4 tests that functions can be declared in the header.
Done.

tests/trigraph/main.c, line 3 at r2 (raw file):
Previously, elliotchance (Elliot Chance) wrote…
Add the comment above:
// Trigraph tests require the `-trigraph` clang option, so it cannot be amount other general tests.

This also raised the question. You should have the same file in the tests directory to make sure the default behaviour is also the same.

Done.

Comments from Reviewable

:+1: Ready for review.
Now, compile sqlite3 is platform depend and need more expertise for understood - How to compile any group of source code.

Review status: 3 of 19 files reviewed at latest revision, 8 unresolved discussions.

tests/multi2/main.c, line 1 at r2 (raw file):
Previously, elliotchance (Elliot Chance) wrote…
If you are going to use the header file for forward-declaring functions you should place the actual functions after the main.

If you want, then I will also that case.

tests/multi2/main.c, line 3 at r2 (raw file):
Previously, elliotchance (Elliot Chance) wrote…
*.c files should not contains these conditional pragmas because C files are not used in #include. Please remove the #if and #define lines.

I will add a new case with without #if and ...

Comments from Reviewable

Reviewed 15 of 16 files at r3, 1 of 1 files at r4.
Review status: all files reviewed at latest revision, 5 unresolved discussions.

a discussion (no related file):
I'm sorry for the slow reply.
After reviewing the changes I see what's trying to be achieved but it it hard to understand with generic names like "multi2".  To make it clear for the next person to maintain we need to be clear about what each of the tests so and how they differ from each other.
We can reduce all the multi files into a single folder that contains all of these cases with clear documentation:
multi/header.h:
#include <stdio>

// This header file is included by multiple C files. Make sure the preprocessor
// is working correctly by not declaring duplicates.
#ifndef HEADER
#define HEADER

// Headers usually do not contain whole functions, but we want to make sure this
// is still OK to do.
void say_four() {
    printf("4");
}

// Forward-declared prototypes that are defined in one of our other C files.
void say_two();   // main1.c
void say_three(); // main2.c

#endif /* HEADER */

multi/main1.c:
#include <stdio>
#include "header.h"

int main() {
    say_two();
    say_three();
    say_four();

    return 0;
}

// The body for the definition (declared in the header). Notice this is declared
// after using the forward declaration above.
void say_two() {
    printf("2");
}

multi/main2.c:
#include <stdio>
#include "header.h"

void say_three() {
    printf("3");
}

You will still need to keep the trigraphs.c separate.
I will dismiss all of the discussions on multi files, please create a single multi folder with the examples above.

ast/empty_decl.go, line 21 at r3 (raw file):

		Addr:       ParseAddress(groups["address"]),
		Pos:        NewPositionFromString(groups["position"]),
		Position2:  groups["position2"],

This is a position so it should be parsed like the line above.

transpiler/transpiler.go, line 379 at r4 (raw file):

	case *ast.EmptyDecl:
		// Fix that...

Fix this?

travis/test.sh, line 100 at r4 (raw file):

# sqlite3.c
echo "Transpiling sqlite3.c..."
./c2go transpile -o=$SQLITE_TEMP_FOLDER/sqlite3.go $SQLITE_TEMP_FOLDER/$SQLITE3_FILE/sqlite3.c >> $OUTFILE 2>&1

These two transpile steps need to be combined now:
echo "Transpiling sqlite3 source files..."

./c2go transpile $SQLITE_TEMP_FOLDER/$SQLITE3_FILE/sqlite3.c $SQLITE_TEMP_FOLDER/$SQLITE3_FILE/shell.c >> $OUTFILE 2>&1

SQLITE_WARNINGS=`cat $SQLITE_TEMP_FOLDER/sqlite3.go | grep "// Warning" | wc -l`

Replace $SQLITE_WARNING_SQLITE3 + $SQLITE_WARNING_SHELL below with $SQLITE_WARNINGS.

travis/test.sh, line 117 at r4 (raw file):

# SQLITE
echo "----------------------"

I dont understand what all this code below here is for?

Comments from Reviewable

Review status: all files reviewed at latest revision, 5 unresolved discussions.

a discussion (no related file):
Previously, elliotchance (Elliot Chance) wrote…
I'm sorry for the slow reply.
After reviewing the changes I see what's trying to be achieved but it it hard to understand with generic names like "multi2".  To make it clear for the next person to maintain we need to be clear about what each of the tests so and how they differ from each other.
We can reduce all the multi files into a single folder that contains all of these cases with clear documentation:
multi/header.h:
#include <stdio>

// This header file is included by multiple C files. Make sure the preprocessor
// is working correctly by not declaring duplicates.
#ifndef HEADER
#define HEADER

// Headers usually do not contain whole functions, but we want to make sure this
// is still OK to do.
void say_four() {
    printf("4");
}

// Forward-declared prototypes that are defined in one of our other C files.
void say_two();   // main1.c
void say_three(); // main2.c

#endif /* HEADER */

multi/main1.c:
#include <stdio>
#include "header.h"

int main() {
    say_two();
    say_three();
    say_four();

    return 0;
}

// The body for the definition (declared in the header). Notice this is declared
// after using the forward declaration above.
void say_two() {
    printf("2");
}

multi/main2.c:
#include <stdio>
#include "header.h"

void say_three() {
    printf("3");
}

You will still need to keep the trigraphs.c separate.
I will dismiss all of the discussions on multi files, please create a single multi folder with the examples above.

Done.

ast/empty_decl.go, line 21 at r3 (raw file):
Previously, elliotchance (Elliot Chance) wrote…
This is a position so it should be parsed like the line above.

Ok, so I grep little bit:
grep -rnw -e Position2 | grep -v test
parm_var_decl.go:44:		Position2:    strings.TrimSpace(groups["position2"]),
var_decl.go:42:		Position2:    strings.TrimSpace(groups["position2"]),
indirect_field_decl.go:28:		Position2:  strings.TrimSpace(groups["position2"]),
typedef_decl.go:39:		Position2:    strings.TrimSpace(groups["position2"]),
record_decl.go:43:		Position2:  strings.TrimSpace(groups["position2"]),
function_decl.go:44:		Position2:    strings.TrimSpace(groups["position2"]),
empty_decl.go:21:		Position2:  groups["position2"],
enum_constant_decl.go:26:		Position2:  groups["position2"],
enum_decl.go:24:		Position2:  NewPositionFromString(groups["position2"]),
go_stmt.go:21:		Position2:  groups["position2"],
field_decl.go:30:		Position2:  strings.TrimSpace(groups["position2"]),

May I change on other ast elements?

transpiler/transpiler.go, line 379 at r4 (raw file):
Previously, elliotchance (Elliot Chance) wrote…
Fix this?

Add warning for future

travis/test.sh, line 100 at r4 (raw file):
Previously, elliotchance (Elliot Chance) wrote…
These two transpile steps need to be combined now:
echo "Transpiling sqlite3 source files..."

./c2go transpile $SQLITE_TEMP_FOLDER/$SQLITE3_FILE/sqlite3.c $SQLITE_TEMP_FOLDER/$SQLITE3_FILE/shell.c >> $OUTFILE 2>&1

SQLITE_WARNINGS=`cat $SQLITE_TEMP_FOLDER/sqlite3.go | grep "// Warning" | wc -l`

Replace $SQLITE_WARNING_SQLITE3 + $SQLITE_WARNING_SHELL below with $SQLITE_WARNINGS.

Done, but not direct like in your snippet. Because:
./c2go transpile $SQLITE_TEMP_FOLDER/$SQLITE3_FILE/sqlite3.c $SQLITE_TEMP_FOLDER/$SQLITE3_FILE/shell.c >> $OUTFILE 2>&1

Not acceptable for clang:
panic: clang failed: exit status 1:

/usr/include/pthread.h:743:12: warning: declaration of built-in function '__sigsetjmp' requires inclusion of the header <setjmp.h> [-Wbuiltin-requires-header]
extern int __sigsetjmp (struct __jmp_buf_tag *__env, int __savemask) __attribute__ ((__nothrow__));
           ^
/tmp/SQLITE/sqlite-amalgamation-3190300/sqlite3.c:26938:9: warning: equality comparison with extraneous parentheses [-Wparentheses-equality]
  if( (p==0) ) return 7;
       ~^~~

.......

It is happen, because clang work with multifiles like we expected. clang transpile input files separatly, so if in first file we have #ifdef ..., then definition is not continue to next files.
So, we have to continues definition by hand. In our case for sqlite that also dependence at platform(linux, darwin).
In detail:
Step 1 : file shell.c include sqlite.h - Ok.
Step 2 : file sqlite3.c include body of file sqlite.h inside! But defenition SQLITE from step 1 is not continie to here.
So, If we stay like that, then after combine of clang results from both steps, then we will have body of file sqlite.h twice.
For solving that, at step 2 we include line at begin of source sqlite3.c - #include "sqlite.h". So, defenition SQLITE inside sqlite3.c will work and don't included.
Now, after combine of clang result we will have twice including of file sqlite.h, but for preprocessor - it is ok and preprocessor not gived for transpiling twice includes.

travis/test.sh, line 117 at r4 (raw file):
Previously, elliotchance (Elliot Chance) wrote…
I dont understand what all this code below here is for?

See upper.

Comments from Reviewable

Reviewed 30 of 30 files at r5.
Review status: all files reviewed at latest revision, 4 unresolved discussions.

main_test.go, line 132 at r5 (raw file):

				args = append(
					args,
					//	"-race",

Please leave this turned on. I know it makes the tests slower but its important to catch race conditions if they occur because they can be almost impossible to debug.

main_test.go, line 315 at r5 (raw file):

		{
			[]string{
				"./tests/multi/case1/four.c",

As I said with my original comment there is no need to have multiple "cases" this is just confusing and makes the build slow. They are all testing the same thing.
You only need three files for a single multi file test. All of the cases can do into those file (as I had documented).

ast/empty_decl.go, line 21 at r3 (raw file):
Previously, Konstantin8105 (Konstantin) wrote…
Ok, so I grep little bit:
grep -rnw -e Position2 | grep -v test
parm_var_decl.go:44:		Position2:    strings.TrimSpace(groups["position2"]),
var_decl.go:42:		Position2:    strings.TrimSpace(groups["position2"]),
indirect_field_decl.go:28:		Position2:  strings.TrimSpace(groups["position2"]),
typedef_decl.go:39:		Position2:    strings.TrimSpace(groups["position2"]),
record_decl.go:43:		Position2:  strings.TrimSpace(groups["position2"]),
function_decl.go:44:		Position2:    strings.TrimSpace(groups["position2"]),
empty_decl.go:21:		Position2:  groups["position2"],
enum_constant_decl.go:26:		Position2:  groups["position2"],
enum_decl.go:24:		Position2:  NewPositionFromString(groups["position2"]),
go_stmt.go:21:		Position2:  groups["position2"],
field_decl.go:30:		Position2:  strings.TrimSpace(groups["position2"]),

May I change on other ast elements?

All of the other cases can go into another PR.

travis/test.sh, line 117 at r4 (raw file):
Previously, Konstantin8105 (Konstantin) wrote…
See upper.

Remove all the code below. We only want to deal with transpiring the amalgamated files to get the warnings right now.

Comments from Reviewable

Review status: all files reviewed at latest revision, 3 unresolved discussions.

main_test.go, line 132 at r5 (raw file):
Previously, elliotchance (Elliot Chance) wrote…
 I know it makes the tests slower but its important to catch race conditions if they occur because they can be almost impossible to debug.

Done.

main_test.go, line 315 at r5 (raw file):
Previously, elliotchance (Elliot Chance) wrote…
As I said with my original comment there is no need to have multiple "cases" this is just confusing and makes the build slow. They are all testing the same thing.
You only need three files for a single multi file test. All of the cases can do into those file (as I had documented).

Done.

ast/empty_decl.go, line 21 at r3 (raw file):
Previously, elliotchance (Elliot Chance) wrote…
All of the other cases can go into another PR.

Ok.

travis/test.sh, line 117 at r4 (raw file):
Previously, elliotchance (Elliot Chance) wrote…
Remove all the code below. We only want to deal with transpiring the amalgamated files to get the warnings right now.

Done.

Comments from Reviewable

Reviewed 6 of 35 files at r5, 18 of 23 files at r6.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


fix(api): add manual log for error

:tada: This PR is included in version 2.42.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Use a bug tracker tool

Use a defect tracking tool to manage software defects in your project

Fixed in 4fe977b


acceptance prerequisite: data_ingress_v1: 1 - data fixtures

What is this
We're going to want to run acceptance tests that check that the transform pipelines are doing what they're supposed to be doing. Which can lead to source control issues (you don't want gigs of test data in your repo).
This task is to create the functionality to create data test fixture so we can handle this in a sustinable way.
What to do
This is a skeleton acceptance test.
Feature: Data Ingress v1
  As a data engineer I want a configurable pipeline that..
  - can confirm correct configuration and inputs
  - can send sending notifications in the event of issues
  - can successfully transform input into rexpected outputs
  - will call the required apis to pass along these outputs
  - will notify publishing that a resource is ready for publishing

  # TODO - this will grow in time to encapsulate the happy path for the above
  Scenario: Pipeline runs without errors
    Given a temporary source directory of files
        | file                   |  fixture                           |
        | pipeline-config.json   |  pipeline_config_basic_valid.json  |
        | data.sdmx              |  data_sdmx_valid_1.sdmx            |
    And v1_data_ingress starts using the temporary source directory

What we want is:
1.0 Fixture Setup
1.1 A zip of data, just use 1 random (small) sdmx fie file for developing
store it in ./fixtures/data-fixtures.zip and call it sdmx_data_sdmx_valid_1.sdmx (we'll renamed or replace it in later tasks).
1.2 Create a pipeline config fixture
A fixture like the above but just return a json config. Json is light weight but given we've established a compression mechanism it way as well also go in the zip.
1.3 Create ./features/environment.py
Add a before_all statement to it.
Your before_all function should unzip ./fixtures/data.zip into the directory ./fixtures/data if it does not yet exist.
2.0 Feature skeleton

add the yaml snippet on this task as /features/data_ingress_v1.yml
install behave
run behave - that should give you the step functions with NotImplementedError in them
put this step in a file /features/steps/data.py
use the dictionary (denoated by the table, all those '|'s in the yaml) to copy the fixtures requested (guidance on reading tables below) into a temporary directory.
start the function v1_data_ingress() using the Path to this temporary directory - this.

Note: to get keys and values out of a table do something like this - just remember its "file" and "fixture" not "key" and "value" here.
Accpeptance Criteria

[x] Acceptance test data is stored as a zip
[x] The skeleton acceptance test will unzip if the unzipped version is not on the machine in question
[x] the uncopmpressed data assets can be accessed via fixtures
[ ] The skeleton acceptance test can populate the temporart directory using the specified fixtures
[ ] We can create a temp directory with the files in it.

done


SharedState - rename schemaName (and possibly registerSchema)

These name are a bit misleading, the name relates to the set of related states not to the schema itself, as several set of states can share the same schema definition

Possible solution:

schemaName -> collectionName
registerSchema -> defineCollection


Added missing config settings

Added missing settings documented via https://github.com/mattermost/mattermost-plugin-msteams-sync/blob/main/docs/plugin_setup.md

@jasonblais - Azure setup is documented via https://github.com/mattermost/docs/pull/6659.


ValueError in MICE

I tried the following:
from fancyimpute import MICE
imputer = MICE()
imputed = imputer.complete(dummied)

The code crashed with the following stack trace:
imputed = imputer.complete(dummied)
[MICE] Completing matrix with shape (244796, 723)
Traceback (most recent call last):
  File "<ipython-input-50-db9415235e63>", line 1, in <module>
    imputed = imputer.complete(dummied)
  File "build/bdist.macosx-10.5-x86_64/egg/fancyimpute/mice.py", line 364, in complete
    imputed_arrays, missing_mask = self.multiple_imputations(X)
  File "build/bdist.macosx-10.5-x86_64/egg/fancyimpute/mice.py", line 314, in multiple_imputations
    self._check_missing_value_mask(missing_mask)
  File "build/bdist.macosx-10.5-x86_64/egg/fancyimpute/solver.py", line 54, in _check_missing_value_mask
    if not missing.any():
  File "/Users/zeeshan.sayyed/anaconda/lib/python2.7/site-packages/pandas/core/generic.py", line 887, in __nonzero__
    .format(self.__class__.__name__))
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

When I applied SoftImpute to the same matrix, it seems to be working.
Thanks

Is dummied a pandas dataframe? It should be a numpy array. I'll add a conversion.

Aah. My bad. Yes. It's a dataframe. I'll convert it. Thanks.
On Apr 28, 2016 4:03 PM, "Sergey Feldman"<EMAIL_ADDRESS>wrote:

Is dummied a pandas dataframe? It should be a numpy array. I'll add a
conversion.
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/hammerlab/fancyimpute/issues/10#issuecomment-215588017

OK I added a commit that will do np.asarray to the input.

Thanks.
On Apr 28, 2016 4:12 PM, "Sergey Feldman"<EMAIL_ADDRESS>wrote:

OK I added a commit that will do np.asarray to the input.
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/hammerlab/fancyimpute/issues/10#issuecomment-215589395

Hello, @sergeyf. I tried to pass a pandas dataframe as input value, the same as @zeeshansayyed did, but got an error. My dataframe consists  of int and float values.
Here's the error
ValueError                                Traceback (most recent call last)
<ipython-input-55-aeccbccb11e0> in <module>()
     14
     15 mice = MICE()
---> 16 full_df_filled = MICE.complete(mice, mice_input)
     17
     18

/home/velika12/anaconda2/envs/main/lib/python2.7/site-packages/fancyimpute/mice.pyc in complete(self, X)
    335         # average the imputed values for each feature
    336         average_imputated_values = imputed_arrays.mean(axis=0)
--> 337         X_completed[missing_mask] = average_imputated_values
    338         return X_completed

/home/velika12/anaconda2/envs/main/lib/python2.7/site-packages/pandas/core/frame.pyc in __setitem__(self, key, value)
   2324
   2325         if isinstance(key, (Series, np.ndarray, list, Index)):
-> 2326             self._setitem_array(key, value)
   2327         elif isinstance(key, DataFrame):
   2328             self._setitem_frame(key, value)

/home/velika12/anaconda2/envs/main/lib/python2.7/site-packages/pandas/core/frame.pyc in _setitem_array(self, key, value)
   2344             indexer = key.nonzero()[0]
   2345             self._check_setitem_copy()
-> 2346             self.loc._setitem_with_indexer(indexer, value)
   2347         else:
   2348             if isinstance(value, DataFrame):

/home/velika12/anaconda2/envs/main/lib/python2.7/site-packages/pandas/core/indexing.pyc in _setitem_with_indexer(self, indexer, value)
    577
    578                     if len(labels) != len(value):
--> 579                         raise ValueError('Must have equal len keys and value '
    580                                          'when setting with an iterable')
    581

ValueError: Must have equal len keys and value when setting with an iterable

If I convert the dataframe to a numpy array, everything works. So it seems like converting input only in your multiple_imputations function is not enough.

Have you tried passing in a numpy array?
On Jun 21, 2017 2:33 AM, "Darya"<EMAIL_ADDRESS>wrote:

Hello, @sergeyf https://github.com/sergeyf. I tried to pass a pandas
dataframe as input value, the same as @zeeshansayyed
https://github.com/zeeshansayyed did, but got an error. My dataframe
consists of int and float values.
Here's the error
ValueError                                Traceback (most recent call last)
 in ()
14
15 mice = MICE()
---> 16 full_df_filled = MICE.complete(mice, mice_input)
17
18
/home/velika12/anaconda2/envs/main/lib/python2.7/site-packages/fancyimpute/mice.pyc in complete(self, X)
335         # average the imputed values for each feature
336         average_imputated_values = imputed_arrays.mean(axis=0)
--> 337         X_completed[missing_mask] = average_imputated_values
338         return X_completed
/home/velika12/anaconda2/envs/main/lib/python2.7/site-packages/pandas/core/frame.pyc in setitem(self, key, value)
2324
2325         if isinstance(key, (Series, np.ndarray, list, Index)):
-> 2326             self._setitem_array(key, value)
2327         elif isinstance(key, DataFrame):
2328             self._setitem_frame(key, value)
/home/velika12/anaconda2/envs/main/lib/python2.7/site-packages/pandas/core/frame.pyc in _setitem_array(self, key, value)
2344             indexer = key.nonzero()[0]
2345             self._check_setitem_copy()
-> 2346             self.loc._setitem_with_indexer(indexer, value)
2347         else:
2348             if isinstance(value, DataFrame):
/home/velika12/anaconda2/envs/main/lib/python2.7/site-packages/pandas/core/indexing.pyc in _setitem_with_indexer(self, indexer, value)
577
578                     if len(labels) != len(value):
--> 579                         raise ValueError('Must have equal len keys and value '
580                                          'when setting with an iterable')
581
ValueError: Must have equal len keys and value when setting with an iterable
If I convert the dataframe to a numpy array, everything works. So it seems
like converting input only in your multiple_imputations function is not
enough.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/hammerlab/fancyimpute/issues/10#issuecomment-310024552,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABya7B91nDqkY4uiPQo1h8PLplFI4r-Zks5sGONggaJpZM4ISZxX
.

Yes, passing in a numpy array works. But I thought you added support of pandas dataframe as input before closing this issue, so I decided to report a bug here.


fix jcli path for windows

fix #286

The shasum check is failing. Would you mind looking into that and updating the shasum to the correct one? Thanks.


generate nuspecs in IntermediateOutputPath

this is so they go in obj/Configuration/ instead of obj/
closes #2865

Pinging @matthid because I'm on vacation

@johlrich  can you please take a look at the broken test in appveyor
(travis is expected to be red)

Looks like I didn't test with the IntermediateOutputPath variable enough. That one includes the configuration and the target framework in the path. I switched back to $(BaseIntermediateOutputPath)$(Configuration)/ and will keep an eye on the appveyor result.

Mhm still red?

Maybe the test is no good?

Hmmmmm the test passes locally after switching. The test also fails here without the changes complaining about no description, and when looking at the in the scenario temp folder I see the same situation as the issue: an extra blank nuspec at obj root. Not sure why it's passing now.
Strange that I'm getting the opposite results as appveyor. Not really sure of a good way to troubleshoot atm. @forki any ideas?
My only thought was to somehow capture the scenario temp folder from the appveyor build to take a closer look in both scenarios, but not sure how to do that tbh.

Maybe different dotnet cli versions?
Am 29.10.2017 15:35 schrieb "Jonathan Ohlrich"<EMAIL_ADDRESS>
Hmmmmm the test passes locally after switching. The test also fails here
without the changes complaining about no description, and when looking at
the in the scenario temp folder I see the same situation as the issue: an
extra blank nuspec at obj root. Not sure why it's passing now.
Strange that I'm getting the opposite results as appveyor. Not really sure
of a good way to troubleshoot atm. @forki https://github.com/forki any
ideas?
My only thought was to somehow capture the scenario temp folder from the
appveyor build to take a closer look in both scenarios, but not sure how to
do that tbh.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fsprojects/Paket/pull/2871#issuecomment-340266789,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNAciZT8ZLldpDj6X_iokUlmvKWY1ks5sxI02gaJpZM4QJ4Ux
.

@forki I think that's it. I isolated that one test and then ran it with DOTNET_EXE_PATH set to both the preview one that the build downloads (2.1.0-preview1-007002) and the one I have installed (2.0.2). The preview one fails, the install one passes.
Going to change to 2.0.2 so appveyor gets kicked off again with it to prove that out there. If that works then we'll have to see how you want to handle that. 🤞

More weirdness, the full build is currently churning through the integration tests on my machine with 2.0.2 but appveyor can't seem to get past restoring Paket.Core.preview3.

Yes changing the dotnet cli version is not the solution here
Am 29.10.2017 20:10 schrieb "Jonathan Ohlrich"<EMAIL_ADDRESS>
More weirdness, the full build is currently churning through the
integration tests on my machine with 2.0.2 but appveyor can't seem to get
past restoring Paket.Core.preview3.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fsprojects/Paket/pull/2871#issuecomment-340286320,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNIOhbWNJ4kf9Gxhx16VYH_hMwyu4ks5sxM2-gaJpZM4QJ4Ux
.

Ok, it is now checking for existence in the new location and if it couldn't find it, falling back to the old location. Build finally green and worked with both cli versions here.
Not super familiar with MSBuild so let me know if there was a simpler way to do that since Exists doesn't seem to do wildcards


ButtonGroup items should have key prop

Hello,
In button-group.tsx component, a key prop should be specified, otherwise, you get a warning : each child in a list should have a unique "key" prop.
As a solution, the interface ButtonGroupItem could take a key, which would be passed to the loop inside ButtonGroup as a key.
Thanks :)

Thanks @appliquant , not sure how I missed this! Fixed in v0.7.1


[Visual Studio Code - Project Manager] Sort by Recent

Extension – Visual Studio Code - Project Manager
Author: @MarkusLanger

Describe the feature and the current behavior/state.
The VSCode project manager extension supports sorting the list of projects by most recently opened. This companion extension should also support that option. Ideally the recently opened state would be shared with the VSCode extension, but if that isn't possible maintaining a separate state is acceptable.
Who will benefit with this feature?
Devs who frequently switch between a small number of VSCode projects but have a large total number of projects.
Any Other info.

Hi @diminutivesloop,
Thanks for the idea. I think it only makes sense if the recently opened state is synchronised with the VSCode extension. If the projects are changed directly in the VSCode, my extension does not notice this and therefore the state is not consistent.
I do not currently see this possiblity.
Also the core code of the VSCode extension is not public. See https://github.com/alefragnani/vscode-project-manager/issues/498


.srt file gets italics even though .ttml has none

[X] I've verified and I assure that I'm running youtube-dl 2018.05.26
[X] At least skimmed through the README, most notably the FAQ and BUGS sections
[X] Searched the bugtracker for similar issues including closed ones
[X] Bug report (encountered problems with youtube-dl)

$ youtube-dl -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.05.26
[debug] Python version 2.7.10 (CPython) - Darwin-16.7.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.0, ffprobe 4.0, rtmpdump 2.4
[debug] Proxy map: {}

When converting videos (geo-restricted, so small example enclosed) with .ttml subtitles using --write-sub --sub-lang en,no --convert-subtitles srt:
<?xml version="1.0" encoding="UTF-8"?><tt xml:lang="no" xmlns="http://www.w3.org/ns/ttml" xmlns:tts="http://www.w3.org/ns/ttml#styling">

    <styling>


datasource/vmware: allow building on arm systems

When built on ARM, the datasource will permanently report that it is offline.

Fixes https://github.com/coreos/bugs/issues/830.

lgtm


⚠ AGEG website (prod) has degraded performance

In aed3736, AGEG website (prod) (https://www.ageg.ca/) experienced degraded performance:

HTTP code: 200
Response time: 1227 ms

Resolved: AGEG website (prod) performance has improved in f8f2a7b.


getRegisteredImage() not implemented?

I don't see getRegisteredImage() among the listed methods. Has that not been implemented?

Looks like that's what you're trying to do with coordinateMapperRGBDepth, but not working yet, I take it?

It was implemented, however, it was removed due to low performace


Improve video input pipeline (using TFRecord files)

I am building a video input pipeline for DeepMind's Kinetics dataset using TFRecord files. Since the dataset is large (200k videos) my TFRecord files store the frames as compressed JPG images; otherwise it would require too much space on disk. Each tf.train.Example has the following structure:
Example {
  'num_frames': tf.int64,
  'label': tf.int64,
  'frames/0001': tf.string,
  'frames/0002': tf.string,
  ...
}

Where all the frames store a JPG image as compressed bytes. Using tf.data.TFRecordDataset and tf.image.decode_jpg I am able to load the images and decode from JPG into tf.uint8 tensors (full code can be found here):
def decode(serialized_example):

    # Prepare feature list; read encoded JPG images as bytes
    features = dict()
    features["class_label"] = tf.FixedLenFeature((), tf.int64)
    for i in range(64):
        features["frames/{:04d}".format(i)] = tf.FixedLenFeature((), tf.string)

    # Parse into tensors
    parsed_features = tf.parse_single_example(serialized_example, features)

    # Decode the encoded JPG images
    images = []
    for i in range(64):
        images.append(tf.image.decode_jpeg(parsed_features["frames/{:04d}".format(i)]))

    # Pack the frames into one big tensor of shape (N,H,W,3)
    images = tf.stack(images)
    label  = tf.cast(parsed_features['class_label'], tf.int64)

    return images, label

Two things currently seem impossible with the current features of TFRecord files:

There seems to be no way to take a random sample of frames. The code example now takes the first 64 frames from the TFRecord, but what is often preferred is taking a random sample of consecutive frames. In one of my failed attempts I have tried to accomplish this along the lines of:

num_frames = tf.cast(parsed_features['num_frames'], tf.int64)
offset = tf.random_uniform(shape=(), minval=0, maxval=label, dtype=tf.int64)

The number of frames in the video example seems impossible to access in TensorFlow. It can be obtained using  tf.train.Example.FromString as given here, but that does not help me in this case. If this was possible I could just load all the video frames into a tensor (at increased cost...) and than use tf.random_crop to sample a random number of frames from the video.

My overall question is whether the input pipeline for videos using TFRecord files can be improved? This needs to consider speed of reading data and compression options to limit file size for enormous  datasets. It would be convenient to directly use mp4 streams with TFRecord files, however decoding this is problably much slower than decoding JPG images (EDIT: this pull request is related: https://github.com/tensorflow/tensorflow/pull/13242)
Note that there are many ways to setup the data pipeline for videos. I have described some of them in this post on StackOverflow and motivated why I chose for TFRecord files. This post also describes the problem described here, so it may be informative: https://stackoverflow.com/questions/48101576/tensorflow-read-video-frames-from-tfrecords-file
Have I written custom code: N/A
OS Platform and Distribution: N/A
TensorFlow installed from: N/A
TensorFlow version: N/A
Bazel version: N/A
CUDA/cuDNN version: N/A
GPU model and memory: N/A
Exact command to reproduce: N/A

@mrry @saeta can you comment or redirect? Thanks!

I posted an answer on Stack Overflow. What you're trying do do seems possible, but it's not super easy (or efficient), and I suggested a workaround in the answer. I dare say the experience for working with video could be improved, and we'd welcome feature requests and/or contributions in this regard.

Nagging Awaiting Response: It has been 14 days with no activityand the awaiting response label was assigned. Is this still an issue?

@mrry Thank you for the helpful comment on StackOverflow. For now, your suggestion seems to be the way to go.

Thanks for letting us know! I'll close this issue for now, but please feel free to reopen it if you have any more requests for improving video processing with tf.data.

What if I wanna pick a random number of images from a random class, where images are not a segment of video but from random frame indices? @tomrunia @mrry