ManagedIdentityCredential token cache not working?

I recently upgraded Azure.Identity to version 1.8.2, hoping I can get rid of my token caching implementation and rely on the internal caching. however, from looking at Application Insights logs, it looks like there is no caching going on, and a token is acquired before every access to the database. Here is a screenshot for example. I highlighted requests for a token and the corresponding database calls.

I am using a ChainedTokenCredential like this:
        private static TokenCredential GetCredential()
        {
            return new ChainedTokenCredential(
                new ManagedIdentityCredential(),
                new VisualStudioCredential(),
                new EnvironmentCredential()
            );
        }

and then get a token like this:
        public SqlConnection GetConnection()
        {
            SqlConnection conn = new SqlConnection(Config.ConnectionString);
            if (!Config.IsDevEnvironment)  //in dev we use an on-prem db
            {
                TokenCredential credential = GetCredential();

                AccessToken token = credential.GetToken(new TokenRequestContext(new[] { "https://database.windows.net/.default" }), CancellationToken.None);

                conn.AccessToken = token.Token;
            }
            return conn;
        }

Am I doing something wrong?

Thank you for your feedback.  Tagging and routing to the team member best able to assist.

Hi @tsahi
It doesn't appear that you are doing anything wrong.
Would you mind providing the logging output after reproducing this with logging enabled? This should show some MSAL logging which would provide some insight.

How can I upload a file confidentially? Or alternatively, if you can narrow down the data items you need from a request operation (in terms or Application Insights) - telemetry type, fields.

Thanks for sharing

I'm interested in any of the MSAL log lines that mention cache.

Azure-Identity trace.xlsx
these are all the lines from a CSV export of the logs that contain the word "cache", without empty columns and columns containing resource information.
There was a line which repeated under the title "=== Request Data ===" which specified "LongRunningOboCacheKey set: False".
thanks,

Hi @tsahi
Unfortunately, strictly filtering lines that mention cache loses some context. I'm also interested in all the related lines. For example, here is an excerpt from my logging showing the token served from cache:
[Informational] Azure-Identity: False MSAL <IP_ADDRESS> MSAL.NetCore .NET 5.0.17 Microsoft Windows 10.0.22621 [2023-05-18 15:23:56Z - <...>] Looking for scopes for the authority in the cache which intersect with https://storage.azure.com/.default
[Informational] Azure-Identity: False MSAL <IP_ADDRESS> MSAL.NetCore .NET 5.0.17 Microsoft Windows 10.0.22621 [2023-05-18 15:23:56Z - <...>] Intersecting scope entries count - 1
[Informational] Azure-Identity: False MSAL <IP_ADDRESS> MSAL.NetCore .NET 5.0.17 Microsoft Windows 10.0.22621 [2023-05-18 15:23:56Z - <...>] Matching entries after filtering by user - 1
[Informational] Azure-Identity: False MSAL <IP_ADDRESS> MSAL.NetCore .NET 5.0.17 Microsoft Windows 10.0.22621 [2023-05-18 15:23:56Z - <...>] [SaveTokenResponseAsync] Saving Id Token and Account in cache ...
[Informational] Azure-Identity: False MSAL <IP_ADDRESS> MSAL.NetCore .NET 5.0.17 Microsoft Windows 10.0.22621 [2023-05-18 15:23:56Z - <...>] [SaveTokenResponseAsync] Saving RT in cache...
[Informational] Azure-Identity: False MSAL <IP_ADDRESS> MSAL.NetCore .NET 5.0.17 Microsoft Windows 10.0.22621 [2023-05-18 15:23:56Z - <...>] IsLegacyAdalCacheEnabled: yes
[Informational] Azure-Identity: False MSAL <IP_ADDRESS> MSAL.NetCore .NET 5.0.17 Microsoft Windows 10.0.22621 [2023-05-18 15:23:56Z - <...>] Not writing FRT in ADAL legacy cache.
[Informational] Azure-Identity: False MSAL <IP_ADDRESS> MSAL.NetCore .NET 5.0.17 Microsoft Windows 10.0.22621 [2023-05-18 15:23:56Z - <...>]
  === Token Acquisition finished successfully:

Azure-Identity: [Informational] ManagedIdentityCredential.GetToken succeeded. Scopes: [ https://database.windows.net/.default ] ParentRequestId:  ExpiresOn: 2023-05-15T06:30:33.7524323+00:00
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - cd9bc186-da17-4076-826c-7338d168e74c] Fetched access token from host login.microsoftonline.com.
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - cd9bc186-da17-4076-826c-7338d168e74c]  AT expiration time: 5/15/2023 6:30:33 AM +00:00, scopes: https://database.windows.net/.default. source: IdentityProvider
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - cd9bc186-da17-4076-826c-7338d168e74c]
	=== Token Acquisition finished successfully:"
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - cd9bc186-da17-4076-826c-7338d168e74c] Intersecting scope entries count - 0
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - cd9bc186-da17-4076-826c-7338d168e74c] Looking for scopes for the authority in the cache which intersect with https://database.windows.net/.default
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - cd9bc186-da17-4076-826c-7338d168e74c] [SaveTokenResponseAsync] Saving AT in cache and removing overlapping ATs...
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - cd9bc186-da17-4076-826c-7338d168e74c] Cannot determine home account id - or id token or no client info and no subject
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - cd9bc186-da17-4076-826c-7338d168e74c] [SaveTokenResponseAsync] ID Token not present in response.
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - cd9bc186-da17-4076-826c-7338d168e74c] Saving token response to cache..
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - cd9bc186-da17-4076-826c-7338d168e74c] Checking client info returned from the server..
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - cd9bc186-da17-4076-826c-7338d168e74c] [Instance Discovery] Skipping Instance discovery because it is disabled.
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - cd9bc186-da17-4076-826c-7338d168e74c] [Region discovery] Not using a regional authority.
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - cd9bc186-da17-4076-826c-7338d168e74c] === Token Acquisition (ClientCredentialRequest) started:
	 Scopes: https://database.windows.net/.default
	Authority Host: login.microsoftonline.com"
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - cd9bc186-da17-4076-826c-7338d168e74c]
=== Request Data ===
Authority Provided? - True
Scopes - https://database.windows.net/.default
Extra Query Params Keys (space separated) -
ApiId - AcquireTokenForClient
IsConfidentialClient - True
SendX5C - False
LoginHint ? False
IsBrokerConfigured - False
HomeAccountId - False
CorrelationId - cd9bc186-da17-4076-826c-7338d168e74c
UserAssertion set: False
LongRunningOboCacheKey set: False
Region configured:"
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - cd9bc186-da17-4076-826c-7338d168e74c] === AcquireTokenForClientParameters ===
SendX5C: False
ForceRefresh: False"
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - cd9bc186-da17-4076-826c-7338d168e74c] MSAL MSAL.Desktop with assembly version '<IP_ADDRESS>'. CorrelationId(cd9bc186-da17-4076-826c-7338d168e74c)
Azure-Identity: [Informational] ManagedIdentityCredential.GetToken invoked. Scopes: [ https://database.windows.net/.default ] ParentRequestId:
Azure-Identity: [Informational] ManagedIdentityCredential.GetToken succeeded. Scopes: [ https://database.windows.net/.default ] ParentRequestId:  ExpiresOn: 2023-05-15T06:30:33.5961808+00:00
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - a1feece4-b720-4a50-baf5-6218fc7c8501] Fetched access token from host login.microsoftonline.com.
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - a1feece4-b720-4a50-baf5-6218fc7c8501]  AT expiration time: 5/15/2023 6:30:33 AM +00:00, scopes: https://database.windows.net/.default. source: IdentityProvider
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - a1feece4-b720-4a50-baf5-6218fc7c8501]
	=== Token Acquisition finished successfully:"
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - a1feece4-b720-4a50-baf5-6218fc7c8501] Intersecting scope entries count - 0
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - a1feece4-b720-4a50-baf5-6218fc7c8501] Looking for scopes for the authority in the cache which intersect with https://database.windows.net/.default
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - a1feece4-b720-4a50-baf5-6218fc7c8501] [SaveTokenResponseAsync] Saving AT in cache and removing overlapping ATs...
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - a1feece4-b720-4a50-baf5-6218fc7c8501] Cannot determine home account id - or id token or no client info and no subject
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - a1feece4-b720-4a50-baf5-6218fc7c8501] [SaveTokenResponseAsync] ID Token not present in response.
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - a1feece4-b720-4a50-baf5-6218fc7c8501] Saving token response to cache..
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - a1feece4-b720-4a50-baf5-6218fc7c8501] Checking client info returned from the server..
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - a1feece4-b720-4a50-baf5-6218fc7c8501] [Instance Discovery] Skipping Instance discovery because it is disabled.
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - a1feece4-b720-4a50-baf5-6218fc7c8501] [Region discovery] Not using a regional authority.
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - a1feece4-b720-4a50-baf5-6218fc7c8501] === Token Acquisition (ClientCredentialRequest) started:
	 Scopes: https://database.windows.net/.default
	Authority Host: login.microsoftonline.com"
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - a1feece4-b720-4a50-baf5-6218fc7c8501]
=== Request Data ===
Authority Provided? - True
Scopes - https://database.windows.net/.default
Extra Query Params Keys (space separated) -
ApiId - AcquireTokenForClient
IsConfidentialClient - True
SendX5C - False
LoginHint ? False
IsBrokerConfigured - False
HomeAccountId - False
CorrelationId - a1feece4-b720-4a50-baf5-6218fc7c8501
UserAssertion set: False
LongRunningOboCacheKey set: False
Region configured:"
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - a1feece4-b720-4a50-baf5-6218fc7c8501] === AcquireTokenForClientParameters ===
SendX5C: False
ForceRefresh: False"
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - a1feece4-b720-4a50-baf5-6218fc7c8501] MSAL MSAL.Desktop with assembly version '<IP_ADDRESS>'. CorrelationId(a1feece4-b720-4a50-baf5-6218fc7c8501)
Azure-Identity: [Informational] ManagedIdentityCredential.GetToken invoked. Scopes: [ https://database.windows.net/.default ] ParentRequestId:
Azure-Identity: [Informational] ManagedIdentityCredential.GetToken succeeded. Scopes: [ https://database.windows.net/.default ] ParentRequestId:  ExpiresOn: 2023-05-15T06:30:34.4399289+00:00
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - df092eed-4bb5-4feb-9140-246433eee883] Fetched access token from host login.microsoftonline.com.
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - df092eed-4bb5-4feb-9140-246433eee883]  AT expiration time: 5/15/2023 6:30:34 AM +00:00, scopes: https://database.windows.net/.default. source: IdentityProvider
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - df092eed-4bb5-4feb-9140-246433eee883]
	=== Token Acquisition finished successfully:"
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - df092eed-4bb5-4feb-9140-246433eee883] Intersecting scope entries count - 0
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - df092eed-4bb5-4feb-9140-246433eee883] Looking for scopes for the authority in the cache which intersect with https://database.windows.net/.default
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - df092eed-4bb5-4feb-9140-246433eee883] [SaveTokenResponseAsync] Saving AT in cache and removing overlapping ATs...
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - df092eed-4bb5-4feb-9140-246433eee883] Cannot determine home account id - or id token or no client info and no subject
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - df092eed-4bb5-4feb-9140-246433eee883] [SaveTokenResponseAsync] ID Token not present in response.
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - df092eed-4bb5-4feb-9140-246433eee883] Saving token response to cache..
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - df092eed-4bb5-4feb-9140-246433eee883] Checking client info returned from the server..
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - df092eed-4bb5-4feb-9140-246433eee883] [Instance Discovery] Skipping Instance discovery because it is disabled.
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - df092eed-4bb5-4feb-9140-246433eee883] [Region discovery] Not using a regional authority.
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - df092eed-4bb5-4feb-9140-246433eee883] === Token Acquisition (ClientCredentialRequest) started:
	 Scopes: https://database.windows.net/.default
	Authority Host: login.microsoftonline.com"
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - df092eed-4bb5-4feb-9140-246433eee883]
=== Request Data ===
Authority Provided? - True
Scopes - https://database.windows.net/.default
Extra Query Params Keys (space separated) -
ApiId - AcquireTokenForClient
IsConfidentialClient - True
SendX5C - False
LoginHint ? False
IsBrokerConfigured - False
HomeAccountId - False
CorrelationId - df092eed-4bb5-4feb-9140-246433eee883
UserAssertion set: False
LongRunningOboCacheKey set: False
Region configured:"
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - df092eed-4bb5-4feb-9140-246433eee883] === AcquireTokenForClientParameters ===
SendX5C: False
ForceRefresh: False"
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:38Z - df092eed-4bb5-4feb-9140-246433eee883] MSAL MSAL.Desktop with assembly version '<IP_ADDRESS>'. CorrelationId(df092eed-4bb5-4feb-9140-246433eee883)
Azure-Identity: [Informational] ManagedIdentityCredential.GetToken invoked. Scopes: [ https://database.windows.net/.default ] ParentRequestId:
Azure-Identity: [Informational] ManagedIdentityCredential.GetToken succeeded. Scopes: [ https://database.windows.net/.default ] ParentRequestId:  ExpiresOn: 2023-05-15T06:30:34.2992724+00:00
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:36Z - f23f9e31-ee95-43f7-8c90-4d4f6d6660fe] Fetched access token from host login.microsoftonline.com.
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:36Z - f23f9e31-ee95-43f7-8c90-4d4f6d6660fe]  AT expiration time: 5/15/2023 6:30:34 AM +00:00, scopes: https://database.windows.net/.default. source: IdentityProvider
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:36Z - f23f9e31-ee95-43f7-8c90-4d4f6d6660fe]
	=== Token Acquisition finished successfully:"
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:36Z - f23f9e31-ee95-43f7-8c90-4d4f6d6660fe] Intersecting scope entries count - 0
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:36Z - f23f9e31-ee95-43f7-8c90-4d4f6d6660fe] Looking for scopes for the authority in the cache which intersect with https://database.windows.net/.default
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:36Z - f23f9e31-ee95-43f7-8c90-4d4f6d6660fe] [SaveTokenResponseAsync] Saving AT in cache and removing overlapping ATs...
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:36Z - f23f9e31-ee95-43f7-8c90-4d4f6d6660fe] Cannot determine home account id - or id token or no client info and no subject
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:36Z - f23f9e31-ee95-43f7-8c90-4d4f6d6660fe] [SaveTokenResponseAsync] ID Token not present in response.
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:36Z - f23f9e31-ee95-43f7-8c90-4d4f6d6660fe] Saving token response to cache..
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:36Z - f23f9e31-ee95-43f7-8c90-4d4f6d6660fe] Checking client info returned from the server..
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:36Z - f23f9e31-ee95-43f7-8c90-4d4f6d6660fe] [Instance Discovery] Skipping Instance discovery because it is disabled.
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:36Z - f23f9e31-ee95-43f7-8c90-4d4f6d6660fe] [Region discovery] Not using a regional authority.
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:36Z - f23f9e31-ee95-43f7-8c90-4d4f6d6660fe] === Token Acquisition (ClientCredentialRequest) started:
	 Scopes: https://database.windows.net/.default
	Authority Host: login.microsoftonline.com"
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:36Z - f23f9e31-ee95-43f7-8c90-4d4f6d6660fe]
=== Request Data ===
Authority Provided? - True
Scopes - https://database.windows.net/.default
Extra Query Params Keys (space separated) -
ApiId - AcquireTokenForClient
IsConfidentialClient - True
SendX5C - False
LoginHint ? False
IsBrokerConfigured - False
HomeAccountId - False
CorrelationId - f23f9e31-ee95-43f7-8c90-4d4f6d6660fe
UserAssertion set: False
LongRunningOboCacheKey set: False
Region configured:"
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:36Z - f23f9e31-ee95-43f7-8c90-4d4f6d6660fe] === AcquireTokenForClientParameters ===
SendX5C: False
ForceRefresh: False"
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:36Z - f23f9e31-ee95-43f7-8c90-4d4f6d6660fe] MSAL MSAL.Desktop with assembly version '<IP_ADDRESS>'. CorrelationId(f23f9e31-ee95-43f7-8c90-4d4f6d6660fe)
Azure-Identity: [Informational] ManagedIdentityCredential.GetToken invoked. Scopes: [ https://database.windows.net/.default ] ParentRequestId:
Azure-Identity: [Informational] ManagedIdentityCredential.GetToken succeeded. Scopes: [ https://database.windows.net/.default ] ParentRequestId:  ExpiresOn: 2023-05-15T06:30:34.5805022+00:00
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:35Z - b54091cd-0d38-4a58-8a24-7d729ff511b4] Fetched access token from host login.microsoftonline.com.
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:35Z - b54091cd-0d38-4a58-8a24-7d729ff511b4]  AT expiration time: 5/15/2023 6:30:34 AM +00:00, scopes: https://database.windows.net/.default. source: IdentityProvider
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:35Z - b54091cd-0d38-4a58-8a24-7d729ff511b4]
	=== Token Acquisition finished successfully:"
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:35Z - b54091cd-0d38-4a58-8a24-7d729ff511b4] Intersecting scope entries count - 0
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:35Z - b54091cd-0d38-4a58-8a24-7d729ff511b4] Looking for scopes for the authority in the cache which intersect with https://database.windows.net/.default
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:35Z - b54091cd-0d38-4a58-8a24-7d729ff511b4] [SaveTokenResponseAsync] Saving AT in cache and removing overlapping ATs...
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:35Z - b54091cd-0d38-4a58-8a24-7d729ff511b4] Cannot determine home account id - or id token or no client info and no subject
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:35Z - b54091cd-0d38-4a58-8a24-7d729ff511b4] [SaveTokenResponseAsync] ID Token not present in response.
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:35Z - b54091cd-0d38-4a58-8a24-7d729ff511b4] Saving token response to cache..
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:35Z - b54091cd-0d38-4a58-8a24-7d729ff511b4] Checking client info returned from the server..
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:34Z - b54091cd-0d38-4a58-8a24-7d729ff511b4] Authority validation - is known env? True.
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:34Z - b54091cd-0d38-4a58-8a24-7d729ff511b4] Authority validation enabled? False.
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:34Z - b54091cd-0d38-4a58-8a24-7d729ff511b4] [Instance Discovery] Skipping Instance discovery because it is disabled.
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:34Z - b54091cd-0d38-4a58-8a24-7d729ff511b4] [Region discovery] Not using a regional authority.
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:34Z - b54091cd-0d38-4a58-8a24-7d729ff511b4] === Token Acquisition (ClientCredentialRequest) started:
	 Scopes: https://database.windows.net/.default
	Authority Host: login.microsoftonline.com"
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:34Z - b54091cd-0d38-4a58-8a24-7d729ff511b4]
=== Request Data ===
Authority Provided? - True
Scopes - https://database.windows.net/.default
Extra Query Params Keys (space separated) -
ApiId - AcquireTokenForClient
IsConfidentialClient - True
SendX5C - False
LoginHint ? False
IsBrokerConfigured - False
HomeAccountId - False
CorrelationId - b54091cd-0d38-4a58-8a24-7d729ff511b4
UserAssertion set: False
LongRunningOboCacheKey set: False
Region configured:"
"Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:34Z - b54091cd-0d38-4a58-8a24-7d729ff511b4] === AcquireTokenForClientParameters ===
SendX5C: False
ForceRefresh: False"
Azure-Identity: [Informational] False MSAL <IP_ADDRESS> MSAL.Desktop 4.8 or later Windows Server 2016 Datacenter [2023-05-14 09:10:34Z - b54091cd-0d38-4a58-8a24-7d729ff511b4] MSAL MSAL.Desktop with assembly version '<IP_ADDRESS>'. CorrelationId(b54091cd-0d38-4a58-8a24-7d729ff511b4)
Azure-Identity: [Informational] ManagedIdentityCredential.GetToken invoked. Scopes: [ https://database.windows.net/.default ] ParentRequestId:

HTH

Hi @christothes , I added the logs above.

Hi @tsahi - Looking back at your code example - does GetConnection run each time before the token requests you are seeing in your logs? If so, the problem is that you are creating a new instance of the credential on each call, and the cache is being recreated each time. Could you confirm whether or not the problem reproduces if you ensure that the instance of ManagedIdentityCredential is cached and re-used for each request?

I added the entire ChainedTokenCredential to a System.Runtime.Caching.MemoryCache , and now it looks like this:

Notice that now there are no calls to localhost:41298/msi/token after each call to GetToken. Does that make sense?
The MemoryCache object is held on a static variable for now. I plan to inject it as a singleton in the future.

If there are no additional calls to the msi endpoint, then the token is being cached. You can validate with the logging, as it will contain something like this:
[Informational] Azure-Identity: False MSAL <IP_ADDRESS> MSAL.NetCore .NET 5.0.17 Microsoft Windows 10.0.22621 [2023-05-31 21:40:59Z - tenantGuid]  AT expiration time: 5/31/2023 11:10:56 PM +00:00, scopes: https://storage.azure.com/user_impersonation https://storage.azure.com/.default. source: Cache


Add amenity=coworking_space as non-searchable preset

In 23a37183762adb2b1fe044b9bee3b50d5095eff3 amenity=coworking_space was changed into office=coworking.
I'd propose to add a non-searchable preset for amenity=coworking_space so it doesn't show up as a blank Amenity in the Inspector.

Good idea, thank you!


BuyLimit with "ConditionType"

i use bittrex API 1.1 and 2.0 for trading client (Delphi).
how use method BuyLimit with "ConditionType" and "Target" parameters?
this sample dont work? :
https://bittrex.com/api/v2.0/auth/market/tradebuy?&nonce=1502481723&MarketName=BTC-BCC&OrderType=LIMIT&Quantity=0.0005&Rate=0.10271362101&TimeInEffect=IMMEDIATE_OR_CANCEL&ConditionType=GREATER_THAN&Target=0.10271362101&__RequestVerificationToken=********

@trojanhost the 2.0 API is undocumented, and prone to change.
I personally wouldn't trade using that API.
The API docs for the official API (v1.1) are over at https://bittrex.com/home/api
You should refer to this endpoint: /market/buylimit
And here is my implementation.
Hope this helps.

By the way, for further questions on your personal project, you should head over to the bittrex slack group ! There's an api-developers channel
bittrex.slack.com


⚠ ClamAV has degraded performance

In 81161c1, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 87 ms

Resolved: ClamAV performance has improved in 23222d5.


vswitch instead of bridge

Hey,
I'm using freebsd with vimage as a kernel extension. With that feature enabled all my Jails and my primary network interface are attached to vswitch1 and not to bridge0 .
So now my virtual machine does not have ip connectivity anymore, because the tap interface is always attach to bridge0. I could not figure out how to configure iohyve to use vswitch1 as a default ? Any chance to change it ?

As it stands right now, iohyve only attaches tap interfaces to bridge0. I am currently working on adding support for more than one bridge interface in iohyve.
I would definitely not be opposed to letting that be a bridge or a vswitch.
Can you go through what you did to set up the vswitch? I would be interested in dropping proposed VALE support for VIMAGE if it's stable enough.
This way users could use either bridge or vswitch interfaces, or even both if they wanted.

@sharky1337 as a temporary solution you can change line 495 of /usr/local/lib/iohyve/ioh-guest from ifconfig bridge0 addm $tap to ifconfig vswitch1 addm $tap. After that either restart the host or manually add the tap interfaces to vswitch1 with ifconfig vswitch1 addm tap{n}, where {n} is the tap interface number.
@pr1ntf I believe "vswitch1" is an arbitrary name for a bridge device. A bridge name can be changed by using this command ifconfig bridge0 name vswitch0 or even ifconfig bridge0 name TacoBell. The name seems to be truly arbitrary but is limited to a 15 character length. You can change the name of any interface with ifconfig with this same syntax.
I do not believe integrating VIMAGE into iohyve is possible because it is not implemented in bhyve. At least with the method of utilizing VIMAGE through an epair device. While I have not used VIMAGE in a jail myself, I did take a look at the code in iocage and I think I roughly figured it out. It seems that an epair pair interface is created, then the 'a' portion gets attached to host and the 'b' side gets attached in the jail. Both are renamed. It then seems that both are associated to a jail ID and use a 'vnet' directive in the ifconfig command: "ifconfig $epair_b vnet ioc-${2}". The last part being where the magic happens.

Yes, epair is still where the magic happens, I had assumed something had changed in the latest updates where you can add tap interfaces, thus the question to the user still remains:

Can you go through what you did to set up the vswitch?

If it's just a normal TacoBell bridge, not it's own network stack, then VALE will still be the way to go.

If there are no more questions or comments on this, I'll close out the issue. Having support for having more than just bridge0 is planned, and should be available soon, pending bringing in patches from #170 and #183.

Hello pr1ntf ,
sorry for the late reply, I was on pto.
I used this tutorial for compiling the kernel.
https://jsherz.com/freebsd/jails/kernel/vnet/2015/11/17/custom-kernel-jails-vnet.html
For settings up a  jail I'm using ezjail .
This is my example configuration.
`freebsd% cat samba
To specify the start up order of your ezjails, use these lines to
create a Jail dependency tree. See rcorder(8) for more details.

PROVIDE: standard_ezjail
REQUIRE:
BEFORE:

export jail_samba_hostname="samba"
export jail_samba_rootdir="/usr/jails/samba"
export jail_samba_exec_start="/bin/sh /etc/rc"
export jail_samba_exec_prestart0="ifconfig epair4 create up"
export jail_samba_exec_prestart1="ifconfig vswitch1 addm epair4a"
export jail_samba_exec_poststart0="ifconfig epair4b vnet samba"
export jail_samba_exec_poststart1="jexec samba /sbin/dhclient epair4b "
export jail_samba_exec_poststart2="jexec samba /sbin/ifconfig epair4b inet6 <> prefixlen 64"
export jail_samba_mount_enable="YES"
export jail_samba_devfs_enable="YES"
export jail_samba_devfs_ruleset="devfsrules_jail"
export jail_samba_procfs_enable="YES"
export jail_samba_fdescfs_enable="YES"
export jail_samba_image=""
export jail_samba_imagetype="zfs"
export jail_samba_attachparams=""
export jail_samba_attachblocking=""
export jail_samba_forceblocking=""
export jail_samba_zfs_datasets=""
export jail_samba_cpuset=""
export jail_samba_fib=""
export jail_samba_parentzfs="zroot/ezjail"
export jail_samba_parameters="enforce_statfs=1 allow.mount=1 allow.mount.zfs=1 allow.mount.procfs=1 allow.mount.devfs=1 allow.raw_sockets=1 allow.sysvipc=1"
export jail_samba_post_start_script=""
export jail_samba_retention_policy=""
export jail_samba_exec_poststop0="ifconfig epair4a destroy"
``

Thanks for the info!


AuthUI.signOut(Activity) deprecation

Just a question to ask why AuthUI.signOut(Activity) is deprecated.  In the context of this library, is there a technical reason that we're being asked to use FragmentActivity?  I only use Activity, so I'm concerned that this deprecated signOut method might be removed in the future.

@pushbit You use plain old Activity??? What's your min sdk version? I'm curious: why aren't you using AppCompatActivity?
Yeah, the plan was to remove those old methods in #609. As for why they were deprecated, we've noticed memory leaks when using the old methods. In addition, their logic was much harder to understand because we were trying to manage our own GoogleApiClient instead of using enableAutoManage which abstracts that logic away from us.

I use minSdkVersion 21.  I don't use FragmentActivity or descendants because I haven't had a need for their backported functionality and I trust the framework classes more.  Perhaps it's much less of an issue today, but over the years I've seen too many bizarre issues and bugs with the support library versions of classes.
Thanks for pointing out the GoogleApiClient auto manage thing.  I can understand the benefit of letting them deal with the lifecycle details and I see that unfortunately they only provide that for FragmentActivity, for some reason.  So I understand if you want to follow their lead, but it would be a pity for those of us who would prefer to stick with framework classes.

@pushbit thanks for explaining.  I think you'll find no downside of moving to AppCompatActivity and a lot of upside.  It's the general recommendation for Android apps that target multiple SDK versions.
The reason that enableAutoManage requires a FragmentActivity is because it attaches an invisible Fragment to the activity to track the lifecycle and start/stop the service connection to Google Play services when appropriate.  Since auto-management also requires starting Activities for result and receiving the results (in some cases) there's no other way to do this that will be reliable.
For the forseeable future FirebaseUI will rely heavily on the support libraries, I hope you can find a way to make them work for you too.
I am going to close this issue since moving back to Framework classes is not on our roadmap.


Test: IDE/crashes/040... is recently fixed.

@swift-ci please test

Build failed
Jenkins build - Swift Test OS X Platform
Git Commit - 21cc5e4f9c6a2567524a0b870f73ce230ac8c392
Test requested by - @gparker42

Build failed
Jenkins build - Swift Test Linux Platform
Git Commit - 21cc5e4f9c6a2567524a0b870f73ce230ac8c392
Test requested by - @gparker42

I don't think this was fixed before, but it should be fixed by https://github.com/apple/swift/pull/6076

Thanks. Looks like the test was failing to crash on some platforms or build configurations, which made CI unhappy.


Move OCRSearch from Cineast to vitrivr-engine

Task Description
Cineast sported OCR search, which must be migrated to vitrivr-engine. The search functionality relies on labels provided with the V3C collection. Hence, no extraction is needed (for now).

[ ] Re-implement OCR search: Create an Analyser and Retriever implementation that allows for OCR search.

Dependencies

Issue #6 must be tackled (in parts) so that the OCR features existing in Cineast can be transferred.

Boundary Conditions

The implementation can be based on what is done in Cineast's OCRSearch feature.
Extraction not needed at this point.

What is the status here? Given that we would want to use this for VBS, it should be available as soon as possible...


Codecov and build system tweaks

Tweaking our codecov config a bit as well as trying to make our CI more stable. Attempting to rid ourselves of “Error 65”.

All green, finally. Merging 😄


Use consistent language for formula titles

Addresses the wording inconsistencies mentioned in #231 among other places.
Branched off of sort because the changes are dependent on the new Sort that has two variants.

Those seem more consistent to me - I've made the changes


mwc-wallet (mwc713) need to reuse connection to the node

Initial research shown that our client/server CAN reuse the connection and it improve latency a lot.
Practically wallet does reconnect for every http call.

Also add cache on Node Client level

Testing:  There are no feature to test. You should be able to see about 2 times less waiting on mwc71 regular commands like 'sync', 'info', 'txs'  e.t.c.
Performance for 'check' or recover form seed will be almost the same.

wallet713> outputs
Wallet Outputs - Account 'default' - Block Height: 398162
Output Commitment                                                   MMR Index  Block Height  Locked Until  Status   Coinbase?  # Confirms  Value        Tx
092204375b8b7862bb16dd43f07a18f6195de97f33c26c4471182ebf0a87db8cc8  None       351584        0             Unspent  false      46579       3.992000000  1
wallet713> info
Verified the regular commands like 'sync', 'info', 'txs', hence closing this ticket.
____ Wallet Summary Info - Account 'default' as of height 398163 ____
Confirmed Total                  | 3.992000000
Awaiting Confirmation (< 10)     | 0.000000000
Awaiting Finalization            | 0.000000000

Locked by previous transaction
0.000000000

Currently Spendable
3.992000000

wallet713> sync
Your wallet data successfully synchronized with a node
wallet713>

Verified the commands in MWC wallet, hence closing this ticket.


Support creating production droplets from create.py

Useful for quickly creating production droplets for testing purposes. Takes GitHub username as an argument just like how we create dev droplets.
Except for the 1 commit that adds the creating production droplet functionality rest of the commits are prep commits or cleanup commits,
Related discussion https://chat.zulip.org/#narrow/stream/9-issues/topic/outgoing.20webhook.20should.20catch.20407.20error.20more.20gracefully/near/1131419

I have tested this manually by creating both dev and production droplets. And it works fine. Ready for a review.

This is great, merged, thanks @hackerkid!


Light/Dark mode toggle updated.

Fixes:  #1516
Description
The toggle icon of dark and light mode is been updated and functioning appropriately.

Issue : #1516
Type of PR

[x] Feature enhancement

Screenshots / videos (if applicable)

Checklist:

[x] I have made this change from my own.
[x] My code follows the style guidelines of this project.
[x] I have performed a self-review of my own code.
[x] My changes generate no new warnings.
[x] I have tested the changes thoroughly before submitting this pull request.
[x] I have provided relevant issue numbers and screenshots after making the changes.

@anuragverma108 kindly review this and merge.

Conflicting files
assets/css/style.css
Please see the conflicts highlighted in the image above. Resolve the conflicts in the mentioned files. Ensure you pull all the latest changes before pushing your feature/bug fixes to the main branch. After resolving the conflicts, please open a new PR for review.
Thank you!


NameTable question - can each name have a dif B & D date in the index?"

You ask: "De-normalized Birth and death dates should go in the PersonTable. A person can have multiple names, can each name have a dif B & D date in the index?"
In my testing the answer is no. Changing either the date or the primary flag for a birth or death Fact changes the index date for all names/alt names.
The is also true with the issue related to wrong dates in the index. This occur when a person has 2 birth or death Facts with different dates and the primary birth or death fact gets deleted. The remaining birth fact becomes primary but the index is not updated with the new date,. While the dates shown in the index are no longer correct, they remain consistent across all names for a person.

I have no id a why this was implemented this way. I sure hope it gets
changed .
So if a person has ten alt names the b-date has to be updated in all ten
when one birth fact is changed to primary or the date changed. weird
On Wed, Mar 27, 2024, 5:54 PM optout2024 @.***> wrote:

You ask: "De-normalized Birth and death dates should go in the
PersonTable. A person can have multiple names, can each name have a dif B &
D date in the index?"
In my testing the answer is no. Changing either the date or the primary
flag for a birth or death Fact changes the index date for all names/alt
names.
The is also true with the issue related to wrong dates in the index. This
occur when a person has 2 birth or death Facts with different dates and the
primary birth or death fact gets deleted. The remaining birth fact becomes
primary but the index is not updated with the new date,. While the dates
shown in the index are no longer correct, they remain consistent across all
names for a person.
—
Reply to this email directly, view it on GitHub
https://github.com/ricko2001/RootsMagic_Database_Design/issues/4, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AE3RD6BEVWQ2PXC2QW4FTGTY2NS3TAVCNFSM6AAAAABFL4KSE6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGIYTEMJTHE4DGMI
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Thanks for testing this


Track reading progress

Right now, Poketo is geared around keeping up-to-date with series. We track reading progress by the latest chapter release timestamp: if you've read a chapter that was released on August 20th, 2018, you've presumably read all the chapters released up until that point.
This assumption works fine when you're only tracking new series, but I find many of the series I read on Poketo are ones that I'm actively reading from the beginning. Doing this works, but its not great for a few reasons:

New chapters of these series get marked as "unread" which is annoying since that notification sits around for a while and I'm not even caught up enough to care about the new chapter.
When reading these series, I have to manually remember what the last chapter I read was. There's a few times where I've hopped around to find my place, which means accidentally re-reading chapters or, at worst, risks spoiling a series.

To solve this, lets build a system to track reading progress in a way that's distinct from release time. It'll make Poketo a lot more useful for reading series — not just following updates.
Related issues: #21, poketo/poketo-service#4

Fixed in #26


fix(detail page): Show More/Less will be made visible based on the content height.

Show More/Less will be made visible based on the content height.

@nimishamukherjee snapshot fabric8-ui is deployed and available for testing at https://fabric8-ui-fabric8-ui-pr-1743-fabric8-planner.int.rdu2c.fabric8.io

@mindreeper2420 fixed - https://github.com/fabric8io/fabric8-planner/pull/1743/commits/1056eefb4389943a3e2d2c463eef1f1d3a17ccbd

@nimishamukherjee snapshot fabric8-ui is deployed and available for testing at https://fabric8-ui-fabric8-ui-pr-1743-fabric8-planner.int.rdu2c.fabric8.io

@nimishamukherjee snapshot fabric8-ui is deployed and available for testing at https://fabric8-ui-fabric8-ui-pr-1743-fabric8-planner.int.rdu2c.fabric8.io

@nimishamukherjee snapshot fabric8-ui is deployed and available for testing at https://fabric8-ui-fabric8-ui-pr-1743-fabric8-planner.int.rdu2c.fabric8.io

[test]

@nimishamukherjee snapshot fabric8-ui is deployed and available for testing at https://fabric8-ui-fabric8-ui-pr-1743-fabric8-planner.int.rdu2c.fabric8.io

@nimishamukherjee snapshot fabric8-ui is deployed and available for testing at https://fabric8-ui-fabric8-ui-pr-1743-fabric8-planner.int.rdu2c.fabric8.io

@nimishamukherjee snapshot fabric8-ui is deployed and available for testing at https://fabric8-ui-fabric8-ui-pr-1743-fabric8-planner.int.rdu2c.fabric8.io


Focal Loss

Hi @rjanvier , I saw there was an implementation src/loss/focal.py of focal loss and I would like to apply it as loss function during traning on my dataset (imbalanced). Could you please advise how to do so? Thanks

Hi, you can try moving from CrossEntropy to FocalLoss by changing these lines to:
criterion:
     _target_: src.loss.FocalLoss

In my experiments, this did not change much for the benchmarked datasets. The code for the Focal loss and Lovasz lossare remnant of past experiments, I cannot guarantee that they will work well with the multi-stage loss formulation (triggered by the multi_stage_loss_lambdas parameter). If it causes any issue, you can turn it off by setting multi_stage_loss_lambdas: null in the corresponding config.


unable to exec /etc/s6-overlay/scripts/startup: Permission denied

Hello,
I encounter a problem when I launch the image with an error message :
s6-rc-oneshot-run: fatal: unable to exec /etc/s6-overlay/scripts/startup: Permission denied
I launched it with docker-compose with this simple configuration :
services:
  speedtest-tracker:
    build: ./speedtest-tracker

The repo is freshly cloned in speedtest-tracker directory.
Architecture : aarch64
Can you help me ?
Thanks

Hey @benjamin-feron I'd avoid the main branch atm and use a release. I'm aware of a permission issue when creating the SQLite DB during startup if you don't specify an external db.

Hey,
Thank you for your response
I've done a git checkout v0.1.0-alpha.6 in the repo and the same issue persits.
networks:
  databases:
    external: true

services:
  speedtest-tracker:
    build: ./speedtest-tracker
    restart: unless-stopped
    networks:
      - databases
    environment:
      DB_CONNECTION: mysql
      DB_HOST: mariadb
      DB_PORT: 3306
      DB_DATABASE: speedtest-tracker_test
      DB_USERNAME: speedtest-tracker_test
      DB_PASSWORD: speedtest-tracker_test

s6-rc-oneshot-run: fatal: unable to exec /etc/s6-overlay/scripts/startup: Permission denied
The database exists with privileges on speedtest-tracker_test user.

That is correct the current alpha release is no multi-arch. The code sitting in main has basic support for multi-arch but has not been tested yet. Looking to get that done over the weekend.
#32

Allright !
thank you

Hi, do you have some news about the multi-arch compatibility ?

I'm obviously behind on it, got some client work to wrap up before I circle back around to the issues backlog here. You might be able to pull in main and build the image locally (not from the releases) and give it a test.
My rpi's are offline right now, waiting on 1 more poe hat to come in before I put them back in the rack or I'd test it sooner.

Closing as the initial issue was resolved in #51 and multi-arch will be addressed and is logged in another issue


Google Styled Maps Support

Does the ember-google-map add-on have support for Google Styled Maps? Google's API that controls the map styles. https://developers.google.com/maps/documentation/javascript/styling

I believe (but didn't test) that you can define this style array object in a property of your view (I think it's better to leave this in the view than in the controller), let's say mapStyles. Then use this property when calling the component from your template:
{{google-map ... gopt_styles=view.mapStyles}}

Lemme know if this does not work (and re-open the issue then).

FYI I added a wiki section for the styling.


Unused shortcodes should use INFO rather than WARN debug level

If a shortcode is unused then it shouldn't be logged as a warning. An example:

Feel free to use your own hugo serve or npm script command that suits your liking


WIP: Add register_object_store to SessionContext

Related to #22
With this function added I can now (with the proper environment set):
ctx = datafusion.SessionContext()
ctx.register_object_store("s3://{bucket}")
ctx.register_parquet("table_name", "s3://{bucket}/{dataset}/created_year=2022/created_month=9/created_day=19/", [], True, ".parquet")

df = ctx.sql("""
        SELECT
            first,
            second,
            sum(third)
        FROM table_name
        group by
            first, second, third
        LIMIT 20000
        """
        )
df.show()
<data prints>

I'm currently unable to test the gcp integration, may need some help there.

It would be nice to figure out a cleaner abstraction for configuration for the different providers in the object_store crate, unfortunately their APIs are very different for client configuration. Maybe a 3 varant enum w/ fields for the exposed config? Could potentially just represent that on the python side as **storage_kwargs.

@matthewmturner I'm not quite sure what changes you are suggesting do you mind elaborating? The PR basically does the equivalent of the cookbook example, except it generates the clients using default credential providers (like ::from_env) and just dispatches based on the URI scheme, this seems like the most straightforward way to do things

@wseaton I thought the function signature would be similar to the rust api.  That being said, im not sure how close the existing python function signatures match their equivalent on the rust side so maybe this isnt a design goal of the python bindings.
I also think that it would be good to allow users to create ObjectStore with more than just the default credential providers so that s3 API providers, such as MinIO, could be used.

im not sure how close the existing python function signatures match their equivalent on the rust side so maybe this isnt a design goal of the python bindings

Yeah, the builder pattern doesn't really map well into Python space.

I also think that it would be good to allow users to create ObjectStore with more than just the default credential providers so that s3 API providers, such as MinIO, could be used.

MinIO and Ceph (which I need as well) should be supported by setting AWS_ENDPOINT, but I agree it'd be nice to let users generate this config via the python APIs instead of purely environment driven

Agree on the builder pattern not mapping as well to python.  I was focusing more on the signature for register_object_store.  But of course with what im suggesting it would require figuring out how to create the object store.
If I recall correctly some of the read options and other configs that are used with session context also use builder pattern and its possible some of those are already exposed in our python bindings somewhere and maybe their implementation could be used as inspiration for how to approach creating object store.

@wseaton have you seen https://github.com/roeap/object-store-python (cc: @roeap) ? Looks like not not all services are fully supported yet, but this project can be used to construct an object_store and then registered to a SessionContext through a simple binding exposed from this package. You wouldn't have to parse urls again to create the right object store.

@turbo1912 oh wow! thanks for the heads up, I had just refactored to remove the parse redundancy to externalize some of the config and am catching up on this. It looks like object-store-python is pretty far along

The new api from the last commit looks like this:
from datafusion.store import AmazonS3

s3 = AmazonS3(region="us-west-2", bucket_name="my_bucket", endpoint="https://myendpoint.url")
ctx.register_object_store("s3", "rh-stage-ssa", s3)

@matthewmturner is this closer to what you were thinking?

@wseaton yes! exactly what i had in mind. a nit would be to have it datafusion.object_store instead to be consistent with name.  i confess i havent looked at implementation but the api is what i had in mind.  are there feature flags for the object store functionality?

are there feature flags for the object store functionality

@matthewmturner it'd be pretty easy to do cargo features on our side as each provider is gated behind an object_store crate feature, but I don't think maturin currently supports passthrough of cargo features to pip extras and vice-versa (did a little bit of digging last night), open to ideas on that though!

ah thats unfortunate.  i suppose the alternative would be a totally separate object_store module but im not even sure that would solve it.
do you have an idea how large this makes the datafusion python package?  last i checked it was quite large because it was datafusion + pyarrow (which is large) and now were adding the full object_store crate.

Not sure how scientific this is (just output from ls), but:
Before:
Release:
-rwxr-xr-x. 1 weaton weaton  27M Sep 23 17:21 _internal.abi3.so

Debug:
-rwxr-xr-x. 1 weaton weaton 553M Sep 23 17:19 _internal.abi3.so

After:
Release build:
-rwxr-xr-x. 1 weaton weaton  32M Sep 23 17:17 _internal.abi3.so

Debug build:
-rwxr-xr-x. 1 weaton weaton 624M Sep 23 17:18 _internal.abi3.so

I was referring more to the size after installing the wheel. for example, right now when i pip install datafusion it comes out to 141mb (datafusion itself is only 25mb and the remainder is pyarrow / numpy). so im curious how big datafusion will be now with full featured object store bundled with it.

@matthewmturner honestly not quite sure how to emulate those numbers locally, when I build with maturin build --release I get a 8MB -> 11MB bump in wheel size compared to master from this branch. Wheel is a compressed binary format so I would think the symbol comparison from above would be more apt for comparison (the 27MB quoted for above master is pretty close to the 25MB you mention).
So anywhere from 5-20% bigger would be my guess. Including object storage drivers seems like a good idea and is basically standard for other tools like spark who bundle them with their distributions WRT hadoop (which is in the hundreds of megabytes)

In addition to unifying the API, I also made bucket_name an optional kwarg, because for most providers it can be elided from the user's PoV and just inferred from the connection.

I think we also need a Local store.

I think it's okay, but it depends on Andy.

@wseaton Hi Will, I think you have a cargo fmt check error, so it can't be merged.

@francis-du should be good now, lmk how you want to handle the lock conflict

I usually rebase locally with master branch.

@francis-du @andygrove is there a chance this can get in w/ the 13.0.0 release?

I would love to release a new version of the Python bindings soon but could use some help. I am not familiar with releasing Python packages. Here is the issue to track releasing the next version (0.7.0 because the version numbers for the Python bindings are independent of the DataFusion version)
https://github.com/apache/arrow-datafusion-python/issues/7

Hi Will. There is a good practice to share with you.
If you don't like waiting for maintainers to approval your commit,you can try adding GPG validation. your commit will trigger the CI directly. Like this:

FYI: https://docs.github.com/en/authentication/managing-commit-signature-verification/adding-a-gpg-key-to-your-github-account

@francis-du ah great, I didn't realize it was commit verification that disabled the CI. I'll be sure to set that up for future contributions.

This PR looks pretty straightforward, amazing work @wseaton 🎊 On the question of testing, another relatively simple approach might be using LocalFileSystem from object store (and actually implement a backend for it) for testing. It is generally impossible to test every possible path / variation across all object store implementations, but having a simple/single test that an object store is I think way better than having none.

@wseaton We just merged https://github.com/apache/arrow-datafusion-python/pull/63 to enable auditing for license headers, in preparation for releasing the Python bindings. Could you please merge this into this PR (or rebase). Thanks,

@andygrove done. I can also work on @isidentical's suggestions for testing the LocalFileSystem, but also happy to do it in a follow up MR if a release is imminent.


openapi-gen removed from kubernetes/code-generator

What this PR does / why we need it:
docker-builder container is broken because openapi-gen was removed from kubernetes/code generator in this commit: https://github.com/kubernetes/code-generator/commit/384879d8d5e96ba89dec26f061cd4a0c533f9de4
So, build openapi-gen from kubernetes/kube-openapi.  Unfortunately, that repo is not tagged by k8s release, so use current latest commit.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
Special notes for your reviewer:
Release note:

NONE

@mhenriks thanks! :+1:  I guess, first checking out the branch and then  going into the directory would have worked too.

@rmohr - yeah, I considered checking out the entire kubernetes/code-generator tree then switching to the appropriate release branch once to build all the generators.  But that won't work long term.  Only up to 1.12.  Hopefully they start tagging the kube-openapi repo, though


Anno 2070 not responding d3d11

Please describe your issue as accurately as possible.
If you use Windows, please check the following page: https://github.com/doitsujin/dxvk/wiki/Windows
Software information
Name of the game, settings used etc.
System information

GPU: Rx580
Driver: 31.0.23013.5003
Wine version: Windows
DXVK version: 2.4

Apitrace file(s)

Put a link here

For instructions on how to use apitrace, see: https://github.com/doitsujin/dxvk/wiki/Using-Apitrace
Log files
Please attach Proton or Wine logs as a text file:

When using Proton, set the Steam launch options for your game to PROTON_LOG=1 %command% and attach the corresponding steam-xxxxx.log file i
d3d11.log
dxgi.log
n your home directory.
When using regular Wine, use wine game.exe > game.log 2>&1 and attach the resulting game.log file.

I am not using Nvidia. I am on AMD and I checked in my system tray. I am not running any overlay or AMD adrenaline software

Update to latest graphics driver and try again. I can see you are using a older one for your GPU.

Download drives from AMD directly 31.0.21916.2 same issue. does not seem to be driver related.


config: Document 'rbind' and 'bind' mount options extensions

My last attempt to get (recursive) bind mounts documented was in #530, which was rejected based on “MS_REC makes no sense what-so-ever outside of a bind mount”.  That doesn't make sense to me, with mount(8) exposing MS_REC | MS_SHARED as --make-rshared, MS_REC | MS_SLAVE as --make-rslave, etc., but here's a reroll that uses rbind and bind instead, with a goal of specifying the (recursive) bind syntax so runtime-tools can rely on that instead of blindly assuming runtimes support an unspecified bind syntax.

I think we should link to the kernel documents instead.

I dont think there's a kernel doc for this. Since we are converting the list of known strings values (i.e. rbind,noatime,slave etc to a list of mount flags), maybe having a table of required mappings to refer to here is useful.
For example taken from https://github.com/opencontainers/runc/blob/653207bc29a6d2d62b5d4f55b596467cb715a128/libcontainer/specconv/spec_linux.go#L630

flags
syscall

bind
syscall.MS_BIND

rbind
syscall.MS_BIND | syscall.MS_REC

On Wed, Apr 26, 2017 at 08:28:36AM -0700, Daniel, Dao Quang Minh wrote:

rbind | syscall.MS_BIND | syscall.MS_REC

Now that we're becoming very clear about the mapping between ‘options’
and MS_* constants, do maintainers want to revisit a separate string
for MS_REC (like the ‘recursive’ I floated in #530)?  Or are we
forging ahead with r* options for common MS_REC & MS_* pairings?
We could just link to mount(2) or 1 to define the MS_* values
instead of including a table.  ‘rlimit’ would no longer be supported,
which has a backwards-compat cost, but we'd be able to get out of the
business of defining a wrapper around the kernel API (and based on
#780 that's a business we want to get out of).

@wking NO
When I said rec makes no sense outside of binds, that was because you used binds in your examples and I try to simplify things for you because you take everything so literally.
rec is never used by itself and the format and options are spec'd based on fstab style that are common between /etc/fstab and mount(8).
I like @dqminh suggestion on the table if you feel like you must write more docs on this. Just follow the list that is already in the code and don't make up anything, document as is.

I like @dqminh suggestion on the table…

That's pretty much what I have now, with the list here.  It currently only covers bind and rbind.  Do I need to convert that list to a table?  Do I need to fill it out with the other entries from runC?
I expect we do need to fill it out somewhat, but don't think that has to happen in this PR.  This PR can establish the pattern (table or list, wording pattern for each entry, etc.) and then once we get something landed I'll file a followup adding the other entries that aren't covered by the mount(8) references.

I'm not a fan of the current PR contents and would rather not specify rbind and bind without the others.  They should be no different than any of the others.

I'm not a fan of the current PR contents…

Is there anything other than which values are explicitly listed in the spec you would like to see changed?

… and would rather not specify rbind and bind without the others. They should be no different than any of the others.

Of the options listed in runC's flags, bind and rbind are the only ones not covered by mount(8) filesystem-independent options 's (source here).  RunC differs from the mount page in defining defaults as a no-op, while the man page defines it as rw (clear MS_RDONLY), suid (clear MS_NOSUID), dev (clear MS_NODEV), exec (clear MS_NOEXEC), auto (clear MS_NOAUTO), nouser (clear MS_USER), and async (clear MS_SYNCHRONOUS).  The man page also defines silent and loud (set and clear MS_SILENT), which are not supported in runC (at least as of 1.0.0-rc3).  The other filesystem-indepentent options from the man page look like they're backed by data and not mountflags in the mount(2) call.
So if this PR lands as it stands in bd28d9bd, I expect runC will need to fix default and add silent and loud.  If we want runC to be compliant as it stands, we'll have to drop the filesystem-independent mount(8) reference and inline a list with all of the MS_*-backed entries and then say something like “any unrecognized options are passed to mount(2) as data”.  Or is there a third approach I'm missing that still allows compliance testing?  And which approach sounds best to you?

Rerolled in bd28d9b→9073ddf to target mount(2) and replace the list with a table as discussed in yesterday's meeting.  Since I was clarifying the mountflags and data values, I went ahead and clearly tied the other properties to mount(2) as well.  This should be a no-op as far as spec-consumers are concerned unless they were using the unspecified bind type without putting (r)bind in options.
There are still a number of differences vs. runC where runC diverges from mount(8)'s filesystem-independent options:

The effect of defaults, as described earlier.  We could follow runC and make defaults a no-op, but I don't see a point to that (who would put defaults in their options knowing it is a no-op?).
Missing support: silent, loud (as described earlier), (no)iversion, (no)lazytime.  And the mount(8) filesystem-specific options which are represented by MS_* flags: (no)acl.  And the mount(8) double-dash options which are represented by MS_* flags: move.  We could follow runC and not define options for the missing MS_* settings, but that just breaks consumers who try to use them (since runtimes would have to pass lazytime, etc. through in data instead of setting MS_LAZYTIME).

I'm happy to submit patches to runC to correct those issues.  But if runC patches are not acceptable and maintainers won't accept this PR unless I strictly match the current runC implementation, please say so again and I'll spin them off into follow-up PRs.
More details in the commit message for those who are interested.

I've filed opencontainers/runC#1460, opencontainers/runC#1461, opencontainers/runC#1462, and opencontainers/runC#1463 addressing the missing options from mount(8).
I have not filed a runC PR for MS_MOVE yet, because the current spec does not claim support for it.  We'll see how the missing-option PRs play out, and I may file a MS_MOVE PR if they go over well.

Can we re-open this now that opencontainers/runc#1460 has landed?


Default ADDON_TREE_CACHING to cacheKeyForTree.

The proposed public API for this is cacheKeyForTree, with this  change when the symbol is removed (i.e. the feature is stable) a new version of ember-engines will not need to be released (and users will not accidentally fall back to uncached builds).
Specific scenarios:

In older versions of ember-cli where ember-cli/lib/experiments is not present, we will not add the caching method at all.

In versions of ember-cli with ember-cli/lib/experiments AND ADDON_TREE_CACHING symbol in the experiements file: that symbol will be used to name the method.

In versions of ember-cli with a ember-cli/lib/experiments file that does not contain the ADDON_TREE_CACHING symbol in the experiments file (i.e. it has been promoted to a full public API) the method will be named  cacheKeyForTree.

Thanks for this and the breakdown of different scenarios!


Obtaining word-level timestamps

Question
I am using the lexicon-based beam search decoder with KenLM language model to obtain a transcription from the output from Wav2Vec 2.0. The implementation is based on Python bindings. I would like to additionally get timestamps for every word. Is there a way to obtain this?
Additional Context
For example, the ctcdecode library returns timesteps which is described as "The timestep at which the nth output character has peak probability. Can be used as alignment between the audio and the transcript." Is it possible to get this using flashlight? Perhaps word-level timestamps could be computed based on this output. Thanks!

You can compute timestamps based on decoder output. Here https://github.com/flashlight/flashlight/blob/master/bindings/python/example/decoder_example.py#L207 results.tokens contains tokens predictions for each frame, so you can recompute timestamp depends on your model stride and fft options.

@tlikhomanenko I checked the output of results.tokens and it seems like it is shorter than the total number of frames. For example, for one input, the total number of frames is 998 and results.tokens contains 312 characters. So I suppose we would somehow need the frame number for which each character is predicted.

No, you're right actually. The original results.tokens has the same length as the number of frames. There was some extra processing in the decoder in fairseq which made it shorter. Thanks! :)

@tlikhomanenko do you have any additional information on how to recompute the timestamp depends on your model stride and fft options, to get that into seconds?

@Nithin-Holla could you please explain where I can find the values for number of frames, segment_start and segment_duration? I can't find them in the decoder, or anywhere else for that matter and I don't know where to look anymore. I would greatly appreciate some pointers!

I figured out how to get the word-level timestamps from the fairseq W2lDecoder subclasses' outputs. In examples/speech_recognition/w2l_decoder.py, I added the following code:

in W2lDecoder.__init__(): self.symbols = tgt_dict.symbols # symbols (usually chars) understood by the ASR model, that are predicted in the emission matrix.
W2lKenLMDecoder already had get_timesteps() method implemented, but I extended the functionality to all other decoders by adding the get_timesteps() function to the parents class W2lDecoder and also created a get_symbols() method:

def get_timesteps(self, token_idxs: List[int]) -> List[int]:
        """Returns frame numbers corresponding to every non-blank token.

        Parameters
        ----------
        token_idxs : List[int]
            IDs of decoded tokens (including blank tokens), i.e. list of tokens spanning all frames of the emission matrix.

        Returns
        -------
        List[int]
            Frame numbers corresponding to every non-blank token.
        """
        timesteps = []
        for i, token_idx in enumerate(token_idxs):
            if token_idx == self.blank:
                continue
            if i == 0 or token_idx != token_idxs[i-1]:
                timesteps.append(i)

        return timesteps

    def get_symbols(self, token_idxs: List[int]) -> List[int]:
        """Returns characters corresponding to every non-blank token.

        Parameters
        ----------
        token_idxs : List[int]
            IDs of non-blank tokens.

        Returns
        -------
        List[int]
            Character corresponding to every non-blank token.
        """
        chars = []
        for token_idx in token_idxs:
            chars.append(self.symbols[token_idx])

        return chars

I slightly modified the code of the decode() method of each decoder to ensure the character symbols, timesteps corresponding to each symbol and list of words is returned in the hypos object.

The new decode() method for W2lViterbiDecoder:

def decode(self, emissions):
        B, T, N = emissions.size()
        hypos = []
        if self.asg_transitions is None:
            transitions = torch.FloatTensor(N, N).zero_()
        else:
            transitions = torch.FloatTensor(self.asg_transitions).view(N, N)
        viterbi_path = torch.IntTensor(B, T)
        workspace = torch.ByteTensor(CpuViterbiPath.get_workspace_size(B, T, N))
        CpuViterbiPath.compute(
            B,
            T,
            N,
            get_data_ptr_as_bytes(emissions),
            get_data_ptr_as_bytes(transitions),
            get_data_ptr_as_bytes(viterbi_path),
            get_data_ptr_as_bytes(workspace),
        )

        for b in range(B):
            hypos.append(
                [
                    {
                        "tokens": self.get_tokens(viterbi_path[b].tolist()).tolist(), # non-blank token idxs.
                        "symbols": self.get_symbols(self.get_tokens(viterbi_path[b].tolist()).tolist()), # characters (symbols) corresponding to non-blank token idxs.
                        "score": 0,
                        "timesteps": self.get_timesteps(viterbi_path[b].tolist()), # frame numbers of non-blank tokens.
                        "words": post_process(self.tgt_dict.string(self.get_tokens(viterbi_path[b].tolist()).int().cpu()), 'letter').split(' ') # the transcript as a list of words.
                    }
                ]
            )

        return hypos

The new decode() method for W2lKenLMDecoder:

def decode(self, emissions):
        B, T, N = emissions.size()
        hypos = []
        for b in range(B):
            emissions_ptr = emissions.data_ptr() + 4 * b * emissions.stride(0)
            results = self.decoder.decode(emissions_ptr, T, N)

            nbest_results = results[: self.nbest]
            hypos.append(
                [
                    {
                        "tokens": self.get_tokens(result.tokens).tolist(), # non-blank token idxs.
                        "symbols": self.get_symbols(self.get_tokens(result.tokens)), # characters (symbols) corresponding to non-blank token idxs.
                        "score": result.score,
                        "timesteps": self.get_timesteps(result.tokens), # frame numbers of non-blank tokens.
                        "words": [self.word_dict.get_entry(x) for x in result.words if x >= 0] # the transcript as a list of words.
                    }
                    for result in nbest_results
                ]
            )
        return hypos

The new decode() method for W2lFairseqLMDecoder:

def decode(self, emissions):
        B, T, N = emissions.size()
        hypos = []

        def idx_to_word(idx):
            if self.unit_lm:
                return self.idx_to_wrd[idx]
            else:
                return self.word_dict[idx]

        def make_hypo(result):
            hypo = {
                        "tokens": self.get_tokens(result.tokens).tolist(),  # non-blank token idxs.
                        "score": result.score
                    }
            if self.lexicon:
                hypo["words"] = [idx_to_word(x) for x in result.words if x >= 0] # the transcript as a list of words.
            hypo["symbols"] = self.get_symbols(hypo["tokens"]) # characters (symbols) corresponding to non-blank token idxs.
            hypo["timesteps"] = self.get_timesteps(result.tokens) # frame numbers of non-blank tokens.

            return hypo

        for b in range(B):
            emissions_ptr = emissions.data_ptr() + 4 * b * emissions.stride(0)
            results = self.decoder.decode(emissions_ptr, T, N)

            nbest_results = results[: self.nbest]
            hypos.append([make_hypo(result) for result in nbest_results])
            self.lm.empty_cache()

        return hypos

I then postprocess the results in my own custom script to get the word-level time alignments (in seconds) for each hypothesis:

For results of beam search-based decoders (W2lKenLMDecoder, W2lFairseqLMDecoder) I use the following function to process the result of decode():

def beam_search_decode_fairseq(hypos, emission_mx, audio_lens, num_hyps, time_aligns):
"""Process the results of a W2lDecoder object from fairseq.

    Args:
        hypos (Union[List[Dict], List[List[Dict]]]):
            List of results for each audio file returned by a W2lDecoder object. If the number of hypotheses to return (W2lDecoder.nbest) is 1, hypos will be a list of just the best hypotheses dicts.
             If W2lDecoder.nbest > 1, hypos will be a list of lists, where for each audio file there will be N best hypotheses dicts.
        emission_mx (torch.tensor(B,T,N)):
            The batched emission matrix outputted by the w2v2 acoustic model trained in fairseq.
        audio_lens (List[int]):
            The lengths of the original audio files in the batch, measured in number of samples.
        num_hyps (int):
            The number of best hypotheses to return per audio file.
        time_aligns (bool):
            Flag used to specify whether to calculate word-level time alignment in seconds for each hypothesis.

    Returns:
        transcripts (Union[List[Dict], List[List[Dict]]]):
            List of processed results for each audio file. If W2lDecoder.nbest = 1, transcripts will be a list of just the best hypotheses dicts.
             If W2lDecoder.nbest > 1, transcripts will be a list of lists, where for each audio file there will be N best hypotheses dicts.
            A hypothesis dict has the following fields:
                'pred_txt': (str) the transcript hypothesis itself.
                'timestamps_word': (List[Dict]) List of word Dict objects, one for each word in the transcript, with the following fields:
                    'word': the word itself.
                    'start_time': the start time of the word in seconds in the corresponding audio file.
                    'end_time': the end time of the word in seconds in the corresponding audio file.
    """
    transcripts = []
    for i in range(emission_mx.size(dim=0)):
        # if the batch_size is > 1, use the maximum original audio length in the batch, as all other audio files are padded to the max length during preprocessing.
        audio_len = audio_lens[i] if emission_mx.size(dim=0) == 1 else max(audio_lens)
        if num_hyps > 1:
            all_results = []
            for hyp in hypos[i]:
                hyp_dict = dict()
                if hyp['words']:
                    # 'words' field is not empty if using a lexicon.
                    transcript = ' '.join(hyp['words']).lower()
                else:
                    # 'words' field is [] if lexicon-free decoding, convert from non-blank symbols to words instead.
                    tokens_str = ''.join(hyp['symbols'])
                    transcript = ' '.join(tokens_str.split('|')).strip().lower()
                hyp_dict['pred_txt'] = transcript
                if time_aligns:
                    word_times = get_word_time_alignments_fairseq(audio_len, emission_mx.size(dim=1), 16000, hyp['symbols'], hyp['timesteps'])
                    timestamps_word = normalize_timestamp_output_w2v2(hyp_dict['pred_txt'].split(' '), word_times)
                    hyp_dict['timestamps_word'] = timestamps_word
                # add a hypothesis dict
                all_results.append(hyp_dict)

            transcripts.append(all_results)
        else:
            hyp_dict = dict()
            # append the decoded phrase (as a list of words) from the prediction of the first beam [0] (most likely transcript).
            if hypos[i][0]['words']:
                # 'words' field is not empty if using a lexicon.
                transcript = ' '.join(hypos[i][0]['words']).lower()
            else:
                # 'words' field is [] if lexicon-free decoding, convert from non-blank symbols to words instead.
                tokens_str = ''.join(hypos[i][0]['symbols'])
                transcript = ' '.join(tokens_str.split('|')).strip().lower()
            hyp_dict['pred_txt'] = transcript
            if time_aligns:
                word_times = get_word_time_alignments_fairseq(audio_len, emission_mx.size(dim=1), 16000, hypos[i][0]['symbols'], hypos[i][0]['timesteps'])
                timestamps_word = normalize_timestamp_output_w2v2(hyp_dict['pred_txt'].split(' '), word_times)
                hyp_dict['timestamps_word'] = timestamps_word
            # add a hypothesis dict
            transcripts.append(hyp_dict)

    return transcripts

For Viterbi decoder (W2lViterbiDecoder) I use the following code to process the result of decode():

transcripts = []
        for i in range(emission_mx.size(dim=0)):
            # if the batch_size is > 1, use the maximum original audio length in the batch, as all other audio files are padded to the max length during preprocessing.
            audio_len = audio_lens[i] if emission_mx.size(dim=0) == 1 else max(audio_lens)
            hyp_dict = dict()
            # append the decoded phrase (as a list of words) from the prediction of the first beam [0] (most likely transcript).
            transcript = ' '.join(hypos[i][0]['words']).lower()
            hyp_dict['pred_txt'] = transcript
            if self.time_aligns:
                word_times = get_word_time_alignments_fairseq(audio_len, emission_mx.size(dim=1), 16000, hypos[i][0]['symbols'], hypos[i][0]['timesteps'])
                timestamps_word = normalize_timestamp_output_w2v2(hyp_dict['pred_txt'].split(' '), word_times)
                hyp_dict['timestamps_word'] = timestamps_word
            # add a hypothesis dict
            transcripts.append(hyp_dict)

        return transcripts

Most importantly, get_word_time_alignments_fairseq() is where I calculate the word-level time alignments:
def get_word_time_alignments_fairseq(audio_len, num_frames, sample_rate, symbols, timesteps):
    """Get word time alignments information for a hypothesis transcript input by converting from timesteps to seconds.
    Args:
        audio_len (int):
            The length of audio file in number of samples.
        num_frames (int):
            The number of frames in the ASR acoustic model emission matrix.
        sample_rate (int):
            The sample rate of the loaded audio file.
        symbols (List[str]):
            Decoded list of characters corresponding to the non-blank tokens returned by the decoder.
        timesteps (List[int]):
            Frame numbers corresponding to the non-blank tokens/symbols.

    Returns:
        word_times (List[Tuple[float, float]]):
            List of tuples of start_time and stop_time in seconds for word in the transcript.
    """
    # list of times in seconds in the corresponding audio file for the the non-blank tokens/symbols.
    timestamps = []
    # get the timestep in seconds corresponding to each non-blank token.
    for frame_num in timesteps:
        timestamp = frame_num * (audio_len / (num_frames * sample_rate))
        timestamps.append(timestamp)

    # NOTE: algorithm only works if the first and last symbols are '|', so add them in if that's not the case.
    frame_offset = 0
    if symbols[0] != '|':
        symbols.insert(0, '|')
        # if adding a symbol at index 0, all symbols will have their frame idx increased by 1, so an offset of -1 is created.
        frame_offset = -1
    if symbols[-1] != '|':
        symbols.append('|')

    word_boundary_idxs = [] # tuples of word start and stop indices.
    # get the indices of all word-boundary tokens (|).
    wb_tokens_idxs = [i for i in range(len(symbols)) if symbols[i] == '|']

    # create tuples for each word that contains the indices of its start symbol and end symbol.
    tup = [] # initialise the first tuple of word start character and word end character indices.
    # loop through the indices of the '|' tokens and find the indices of the word-boundary symbols/characters that are the start and end characters of each word.
    for wb_tokens_idx in wb_tokens_idxs:
        try:
            if symbols[wb_tokens_idx-1] != '|' and tup:
                # there is a start index in tuple, but no end index yet.
                # end index has been found.
                if wb_tokens_idx-1 == tup[0]:
                    # word is composed of only one character, add the index of this '|' token as the end character index for the word.
                    tup.append(wb_tokens_idx)
                else:
                    # word is composed of more than one character.
                    tup.append(wb_tokens_idx-1) # add an end character index for the word.
                # add the tuple as complete word to the list of word start and end index tuples.
                word_boundary_idxs.append(tup)
                tup = [] # reset the tuple.
                # continue onto the next if statement as this '|' token may be the boundary between two words.
            if symbols[wb_tokens_idx+1] != '|':
                # start character of new word reached.
                tup.append(wb_tokens_idx+1) # add a start character index for the word.
        except IndexError:
            continue

    # create tuples of start and stop times for each word
    word_times = [(timestamps[start_idx + frame_offset], timestamps[end_idx + frame_offset]) for start_idx, end_idx in word_boundary_idxs]

    return word_times

And normalize_timestamp_output_w2v2() is just a utility function to create a Dict containing time alignment information for each word:
def normalize_timestamp_output_w2v2(words, word_time_tuples):
    """Get word Dict objects with time information for each word in the hypothesis transcript.

    Args:
        words (List[str]):
            List of words in the transcript.
        word_time_tuples (List[Tuple[float,float]]):
            List of tuples of start_time and stop_time in seconds for word in the transcript.

    Returns:
        values (List[Dict]):
            List of dict objects where each dict has the following fields:
                'word': (str) the word itself.
                'start_time': (float) the start time in seconds of the word in the corresponding audio file.
                'end_time': (float) the end time in seconds of the word in the corresponding audio file.
    """
    values = []
    for word, (word_start, word_end) in zip(words, word_time_tuples):
        vals_dict = dict()
        vals_dict['word'] = word
        vals_dict['start_time'] = word_start
        vals_dict['end_time'] = word_end
        values.append(vals_dict)

    return values

The formula I use to calculate the time in seconds in the corresponding audio for each non-blank symbol in the transcript is the following:
timestamp = frame_num * (audio_len / (num_frames * sample_rate))
where:

frame_num = the timestep of the symbol, as returned in the 'timesteps' field of Wl2Decoder.decode() outputs.
audio_len = the number of samples in the loaded audio file corresponding to the transcript (if using batched w2v2 acoustic model inference, will be zero padded to the length of the longest loaded audio file in the batch).
num_frames = the number of frames in the emission matrix returned by the w2v2 acoustic model inference for that audio file (if using batched inference, the number of frames for each audio file will be the same as in this case all loaded audio files are padded to the length of the longest audio file in the batch).
sample_rate = sample rate of loaded audio files (usually 16000 Hz).

I figured out how to get the word-level timestamps from the fairseq W2lDecoder subclasses' outputs and have detailed my approach in https://github.com/facebookresearch/fairseq/pull/3627#issuecomment-1859183176


Can't open or create new file

when i call
reacto in terminal its just opened
can't open or create anything

you can look here https://ibb.co/bNyQvd

Hi @husafwan,
Could you give me more information about how to reproduce the issue, please?
I'm from a macOS environment so only a more detailed issue could guide me towards an answer for you.
Let me know. :)

can you give me instruction to install it in right way ?

Please give me more information about everything: what have you tried so far, what OS are you on...
I can't help you if I don't have any precise explanation.

.AppImage working with me but .deb not working with me
there is no more information tell what you want to know ?

Could you try to open the developer tools and see if there is any error message?
It's usually Ctrl+Alt+I.


DocumentIndex: support for Qdrant

Implement a Document Index backed by Qdrant.
References:

Implementation manual: https://github.com/docarray/docarray/blob/feat-rewrite-v2/docs/tutorials/add_doc_index.md
Document Index design doc (slightly outdated, but the main ideas remain): https://lightning-scent-57a.notion.site/Document-Stores-v2-design-doc-f11d6fe6ecee43f49ef88e0f1bf80b7f
Original PR adding Document Index base class: https://github.com/docarray/docarray/pull/1124

Hey @JohannesMessner , I would like to move on with the issue . It would be my pleasure if you assign this issue to me

Hey @JohannesMessner , I would like to move on with the issue . It would be my pleasure if you assign this issue to me

Hi @Arnav131003, thanks for your interest!
Unfortunately we already have confirmation from Kacper, who works ar Qdrant, that he will take this issue himself. I just couldn't officially assign him.

Oh ! , Sure no problem .

Yeah, I'm already making some efforts to implement it. @JohannesMessner Unfortunately, I cannot assign myself. Could you please do that?

Done! Looking forward to the PR! And please don't forget about our implementation manual, I know it can be a bit hidden :) https://github.com/docarray/docarray/blob/feat-rewrite-v2/docs/tutorials/add_doc_index.md

Hi, @JohannesMessner I'll open a draft PR today in the afternoon ;) It's still in progress, and I want to provide something already partially functional. I'm sorry, but there has been a lot going on recently.


https://dotnet.microsoft.com/download/linux-package-manager/rhel/sdk-current Operating System: Linux RHEL - x64

Problem encountered on https://dotnet.microsoft.com/download/linux-package-manager/rhel/sdk-current
Operating System: Linux RHEL - x64
Provide details about the problem you are experiencing. Include your operating system version, exact error message, code sample, and anything else that is relevant.

@deywit can you provide more information? What problem are you facing?

3 weeks no response, closing. Feel free to reopen if there is more info available. Thanks!


Websockets unable to set up

Hey there,
hope you guys can help me out i've been trying to figure this our for weeks now
There's a lot of mixed info about how to include websockets and the client SDK in a sails app and I managed to get the dependencies loading up correctly but when executing:
io.socket.get('/user', function serverResponded (body, JWR) {

      // JWR ==> "JSON WebSocket Response"
      console.log('Sails responded with: ', body);
      console.log('with headers: ', JWR.headers);
      console.log('and with status code: ', JWR.statusCode);

      // first argument `body` === `JWR.body`
      // (just for convenience, and to maintain familiar usage, a la `JQuery.get()`)
    });

I console log the socket with req.isSocket and it returns true but then my inspect element gives me an error I can't do anything with:
Sails responded with:  Not implemented in core yet
VM12409:6 with headers:  Object {Access-Control-Allow-Origin: "", Access-Control-Allow-Credentials: "", Access-Control-Allow-Methods: "", Access-Control-Allow-Headers: "", access-control-allow-origin: ""…}
VM12409:7 and with status code:  501
not implemented in core yet?!?!

anyone?

@Master244 You'll probably have better luck in Stack Overflow's SailsJS channel, this is more for reporting deeper issues. While not discrediting the possibility of this being broken, there are plenty of people using Socket.io and some of the error messages can be pretty bad.

@Master244 is right; this forum is for reporting bugs in the Sails core.  That being said, that error message usually means that you're trying to render a view over sockets, which is not supported.  I'd check that you haven't overridden the find action in UserController to send back a view.


CI(Fastlane)

Add Notorisation to Fastlane
Create documentation for setting up Fastlane
Which other actions should we add to Fastlane

I went with Xcode cloud so closing this


[WIP] First pass at better error message handling

This is a first pass at improving things, Here's what I'm thinking right now:

I saw the JNI crate using the "thiserror" to help with error handling, so I added it here to make it easier to display errors that we aren't specifically handling

Most of the errors from Rust should be handled by doing more than just printing an error, so I'm setting up a mapping between Rust errors and Exceptions that'll be thrown. I think that interpreting the errors into simple exceptions should be handled on the Rust side, partly to reduce the amount of data that needs to be copied over.

I've also updated the rust-JNI library to 0.21, which improves safety especially related to reference lifetimes.

I could use feedback on what I have so far, and especially the Rust syntax that I'm using.


[AIRFLOW-3431] Document how to report security vulnerabilities.

Make sure you have checked all steps below.
Jira

[x] My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-XXX
In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.

Description

[x] Here are some details about my PR, including screenshots of any UI changes:

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[x] In case of new functionality, my PR adds documentation that describes how to use it.

When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

[x] Passes flake8

This file is more like doc for airflow security feature. Should we put this into contribute file?

Oh i didn't know Airflow and found a vulnerability of look on the docs, not in the contribution docs - and this is where other Apache projects have this (Kafka, Zookeeper)


map_from_entries: empty subquery returns null map

Describe the bug
I couldn't create a sample as I don't know how to how to create an empty array of rows. But a query like below return a null map when the subquery return no rows. I expected it to return an empty map instead.
Also, couldn't we simplify the syntax by removing the explicit array() ?
Error message/log
No response
To Reproduce
select map_from_entries(array(select row(foo.id, foo.name) from foo where foo.id=unrelated.id))

Expected behavior
Return an empty map instead of null
How did you deploy RisingWave?
No response
The version of RisingWave
version | PostgreSQL 13.14.0-RisingWave-2.0.4 (2d757981a1f747c057f79bfb4a19fffbde563ece)
Additional context
No response

It seems to return empty map, instead of null:
dev=> select * from t;
┌───┐
│ x │
├───┤
└───┘
(0 rows)

dev=> select map_from_entries( array( select row(x,x) from t ));
┌──────────────────┐
│ map_from_entries │
├──────────────────┤
│ {}               │
└──────────────────┘
(1 row)


Add open/close sounds to NPCTalk and Currently

Opening and closing these windows is eerily silent at present.

uwu


Change contact merge fixup from future to queueable

Critical Changes
Changes
Issues Closed
New Metadata
Deleted Metadata

@force2b @RadSFDeveloper - figured I'd tag you both on the actual work item PR, it should be short and sweet to review. No rush, since I do not believe we'd be able to QA and merge this sprint.
@force2b - we are checking for customizable rollups for the contact rollup fixup but not for the account rollup fixup. Should we add a work item for that?

**lurch: attach W-030823

Tracking W-030823

@RadSFDeveloper - thanks for the feedback, after our discussion I did change that inner class name so that it would not be ambiguous!
@force2b - did you have a chance to look this one over yet?

@force2b - I created the issue / work item for the account rollup call. Is there anything else that needs to be changed in this PR or is this ready for QA?

Thanks @force2b - I believe both of those scenarios are covered in the tests Sean added in TDTM_TEST2.
contactMergeShouldNotUseLegacyRollupWhenCRLPEnabled
contactMergeShouldUseLegacyRollupWhenCRLPNotEnabled

Thanks @jjbennett. Though, looking at the contactMergeShouldNotUseLegacyRollupWhenCRLPEnabled test, I'm not really sure that is properly setup. Normally customizable rollup metadata has to be mocked and injected to be used in a test. If I had to guess, this code relies on the CRLP metadata existing in the org that the test is running in.

@force2b - I don't think we need a full integration test, we just need to know that the correct rollup process was called.


Does not work in Moodle core h5p activity

There are 2 ways to create/upload an H5P activity in Moodle:
method a) using the 3rd-party plugin provided by the H5P team (the black icon),
method b) using the core Moodle H5P activity (the blue icon).
In the current version of Moodle (4.2.x), h5p-ndla-timeline works OK using method a).
If using method b), it's possible to create/upload an h5p-ndla-timeline activity, but impossible to display it. Or rather there is a weird way of doing it. If you manage to click on the right of the line which is supposed to hold that content, it will open full screen and you will be able to view the content! But when you close/escape that full screen, it disappears again.
PS.- Interestingly I notice that there is exactly the same problem in LUMI... So not a bug special to Moodle core H5P.

I assume that it's a combination of the creators of NDLA Timeline not using the embedType of iframe for the content type (something that is bound to cause other trouble, too) but moodle enforcing displaying H5P content in iframe mode.

@otacke That's a likely explanation. Is it the same for LUMI, then?

@rezeau What about Lumi? Does it show the same behavior?

@otacke Yes, exactly the same behaviour (and same hack to manage to display it fullscreen).

@otacke Thanks for your work, I will test your PR #492 ASAP and report here.