WhenAnyCharacteristicDiscovered observable does raise the onCompleted action on subscription

Please check all of the platforms you are having the issue on (if platform is not listed, it is not supported)

[x] iOS
[x] Android
[ ] tvOS
[ ] macOS
[ ] .NET Standard
UWP IS NOT SUPPORTED DURING THE BETA PERIOD - ONLY PRs ARE WELCOME FOR NOW

Version of OS(s) listed above with issue
iOS 11.4.1
Android 8.1.0
Version of Library
6.2.0
Expected Behaviour
I expect the onComplete action to be called after all services and characteristics have been discovered
Actual Behavior
The onComplete action never get's called. I now that all characteristics get discovered by looking at my debug output but the onComplete action just doesn't get called.
Here's my code I use for connecting and getting the characteristics:
device.Connect();

// get characteristics
var characteristics = new List<IGattCharacteristic>();
var subscription = device..WhenAnyCharacteristicDiscovered().Subscribe(characteristic =>
{
    // onNext
    Log?.Trace("Discovered the characteristic {0}", characteristic.Uuid);
    characteristics.Add(characteristic);
}, () =>
{
    // onCompleted
    Log?.Trace("Discovered all {0} characteristics", characteristics.Count);
});

Am I missing something? Is this behavior intended?
Imho the onCompleted should be raised after all characteristics have been discovered. And I know for certain that discovery works because my debug output lists all characteristics my device has in the onNext action
Thanks!

"When" type functions/observables were not meant to complete.  They are meant to fire across reconnections automatically without the need to hook into connection state yourself.
For characteristics, this is particularly important since the characteristic state is lost across (re)connections

You'll notice I do things like this for:
IDevice.GetKnownCharacteristics  & IDevice.WhenKnownCharacteristicsDiscovered

Ok, understood so far.
Thank you very much for your quick response!
So instead of doing the following:

Connect to device
Get all characteristics from device
Read single characteristic
Disconnect

I should rather

Connect to device
use the "ReadCharacteristic" extension to read the value from the device using the service and characteristic uuid. I should not "care" about getting all characteristics first as the library will handle this for me
Disconnect

Right?
Sorry for my uninformed questions, I'm still pretty new to all the reactive stuff.

You should hook the "When" characteristic methods first, then do Connect.  I noticed you are "pinning" the characteristics in a collection - I would avoid using that list because it is invalid on a disconnect.
CrossBleAdapter.Current.ScanForDevice(uuid or name) (if you know the device is in range or has to be in range)
or
CrossBleAdapter.Current. GetKnownDevice(uuid) if you know the device

both methods yield a device
After that

IDevice.WhenKnownCharacteristicDiscovered(serviceUuid, characteristicUuidS).Subscribe(ch =>
{
});
I would suggest taking a look at http://www.introtorx.com/ to get a good handle on the reactive stuff before deep diving through this library.  You can also use the samples and unit tests in this project for help.
Best of luck

Allright, I think I got it!
Thank you for your support and thanks for the great library!
Best thing out there for cross platform bluetooth development!


feat!: migrate kintsugi to session + collator_selection

Signed-off-by: Gregory Hill<EMAIL_ADDRESS>
Removes the LiquidationVaultFixMigration as Kintsugi has already been upgraded
Reorders Kintsugi runtime numbering
Adds Session + CollatorSelection pallets
Adds migration for vanilla Aura

Is there a way we can test the correctness of this migration & switch before we upgrade kintsugi?


Remote manifest URI flipped for --sidecar

When testing the latest remote manifest reference it appears the logic is flipped here:
           if args.sidecar {
                manifest.set_embedded_manifest_with_remote_ref(remote);
            } else {
                manifest.set_remote_manifest(remote);
            }

I believe if args.sidecar is true, then it should be manifest.set_remote_manifest(remote). When I run c2patool <path/to/input/file> -m manifest.json --remote <some/uri> -f -o <path/to/output/file> I get a new <path/to/output/file>.c2pa, but I thought the remote reference would be embedded in the output file instead. I was able to confirm if I use --sidecar I then get the remote manifest reference in my output file.

I opened a PR with the fix, if I misunderstood please do let me know.

After having a walkthrough, I realize I misunderstood what it means to embed and use remote vs just remote. Closing as this is bogus.


Add note to modify/3 docs about avoiding type mods in Postgres.

Hello, I was investigating why there was no modify/2 function that allowed options to be updated without type update as well. I found this discussion and thought I'd submit a doc update as @josevalim suggested there.
I am not sure if this documentation is still desired by the core team, but I would've found it helpful myself, so thought I'd submit.
Thank you for ecto_sql! Love the work yall do :D

:green_heart: :blue_heart: :purple_heart: :yellow_heart: :heart:


Fix undeclared class issue on iOS

First of all, thank you very much for taking the time to do a PR.
Could you please rename the changed files to STStereoPlugin.h and STStereoPlugin.m for consistency? Thank you in advance.

@faku99 I met some error when build on iOS like below. So I've changed name. If I changed name I could fix the issue. Is there any other workaround?
Error output from Xcode build:
↳
** BUILD FAILED **
Xcode's output:
↳
=== BUILD TARGET Runner OF PROJECT Runner WITH CONFIGURATION Debug ===
/Users/gentlejo/Projects/myapp/ios/Runner/GeneratedPluginRegistrant.m:25:4: error: unknown receiver 'STStereoPlugin'; did you mean 'StereoPlugin'?
[STStereoPlugin registerWithRegistrar:[registry registrarForPlugin:@"STStereoPlugin"]];
^~~~~~~~~~~~~~
StereoPlugin
In module 'stereo' imported from /Users/gentlejo/Projects/myapp/ios/Runner/GeneratedPluginRegistrant.m:13:
/Users/gentlejo/Projects/myapp/build/ios/Debug-iphonesimulator/stereo/stereo.framework/Headers/StereoPlugin.h:5:12: note: 'StereoPlugin' declared here
@interface StereoPlugin : NSObject
^
1 error generated.

@faku99 Have you check this?

I just did. I don't get the compilation error you're referring to in your PR message.
Did you update Flutter to the latest version? It might fix things up.

Would this fix #41?


reproduce the deadlock

@vyzo this adds a Deadlocker client program to the transactor project.  It's very braindead - it just constantly creates new artefacts in a background thread, and awaits an infinite flatMap chain of requests for the current block.
I've been testing it against a local server, but I think my connection to the local dynamo is messed up; I'm getting a bunch of these:
INFO: Unable to execute HTTP request: Unrecognized SSL message, plaintext connection?
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
	at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
	at sun.security.ssl.InputRecord.read(InputRecord.java:527)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)

...

But anyway, it will reliably lock up for me, and interestingly, it always does so after either block #507 or 508 - no idea why, or if it will behave the same on your machine.  When it does fail, the background thread keeps successfully creating blocks, but the request for the current block doesn't retry; it just hangs.
I've been testing with the assembled jar using java -cp ./transactor/target/scala-2.11/transactor-assembly-0.0.1.jar io.mediachain.transactor.Deadlocker <IP_ADDRESS>:54321
adding -Dorg.slf4j.simpleLogger.defaultLogLevel=debug results in mountains of logs.
I'll try to make this nicer in the morning, but please let me know if you find anything 😄

@yusefnapora how can I run the server to try this out against? We should put the invocation in the readme

@yusefnapora you need to specify "http://localhost:8000" (with http prefix) as the endpoint to avoid the dynamo error message.
@parkan you run JournalServer with a properties file that specify its config.

closing in favor of #98


[BUG]: [Child] Admin | Sync Logs constantly failing

Description
We are configuring CoE for the first time and the "[Child] Admin | Sync Logs" flow always fail.
Component

Solution [CenterofExcellenceAuditLogs_1_3_2_managed]
Version [e.g. 1.3.2]
Flow [Child] Admin | Sync Logs

Steps to reproduce the behavior:
Start the "Admin | Sync Audit Logs" and wait for it to fail. It turns out it is caused by its child flow.
Screenshots
The "Admin | Sync Audit Logs" result

Few clusters of the "[Child] Admin | Sync Logs" always fail.

Parse JSON step output:
[   {     "message": "Invalid type. Expected String but got Null.",     "lineNumber": 0,     "linePosition": 0,     "path": "[54].ResultStatus",     "schemaId": "#/items/properties/ResultStatus",     "errorType": "type",     "childErrors": []   },   {     "message": "Invalid type. Expected String but got Null.",     "lineNumber": 0,     "linePosition": 0,     "path": "[54].ClientIP",     "schemaId": "#/items/properties/ClientIP",     "errorType": "type",     "childErrors": []   },
The rest of the flow:

Any idea what is wrong?
Thank you

Could you share what the GetContentDetails output looks like before the failure? This is what is parsed afterwards. Please make sure to remove sensitive information before pasting the output here.

Could you share what the GetContentDetails output looks like before the failure? This is what is parsed afterwards. Please make sure to remove sensitive information before pasting the output here.

I was getting this same error on solution version 1.3.2.  It appeared to only happen if the audit log detail contained an operation that used "ClientIPs" instead of "ClientIP" in the results schema.  When I looked at that particular JSON, yes it shows ClientIP as null but ClientIPs was an array.
It was only on a specific log operation that was not related to PowerApps/Automate activities.  I have not updated the Audit Log solution to the new version yet (still working on the Core Components).
I hope this helps.
Thanks

I was getting this same error on solution version 1.3.2.  It appeared to only happen if the audit log detail contained an operation that used "ClientIPs" instead of "ClientIP" in the results schema.  When I looked at that particular JSON, yes it shows ClientIP as null but ClientIPs was an array.
It was only on a specific log operation that was not related to PowerApps/Automate activities.  I have not updated the Audit Log solution to the new version yet (still working on the Core Components).
I hope this helps.
Thanks

Hi, thanks for your feedback. I am uploading output information...I changed, emails, IP and tenantID
Outputs.txt

I was getting this same error on solution version 1.3.2. It appeared to only happen if the audit log detail contained an operation that used "ClientIPs" instead of "ClientIP" in the results schema. When I looked at that particular JSON, yes it shows ClientIP as null but ClientIPs was an array.
It was only on a specific log operation that was not related to PowerApps/Automate activities. I have not updated the Audit Log solution to the new version yet (still working on the Core Components).
I hope this helps.
Thanks

I searched for ClientIPs but I found none. Only ClientIP.
Thank you
J.

Hi, thanks for your feedback. I am uploading output information...I changed, emails, IP and tenantID
Outputs.txt

I was getting this same error on solution version 1.3.2. It appeared to only happen if the audit log detail contained an operation that used "ClientIPs" instead of "ClientIP" in the results schema. When I looked at that particular JSON, yes it shows ClientIP as null but ClientIPs was an array.
It was only on a specific log operation that was not related to PowerApps/Automate activities. I have not updated the Audit Log solution to the new version yet (still working on the Core Components).
I hope this helps.
Thanks

I searched for ClientIPs but I found none. Only ClientIP.
Thank you
J.

Thank you both, that is really helpful! Looks like the Audit Log event schema is different in some tenants or for certain events that we can't repo in our development tenant. We are actually not using the ClientIP/ClientIPs data, so I think I can update the Parse JSON to ignore this field entirely. I will get back to you in a few hours, once I have tested this.

Thank you both, that is really helpful! Looks like the Audit Log event schema is different in some tenants or for certain events that we can't repo in our development tenant. We are actually not using the ClientIP/ClientIPs data, so I think I can update the Parse JSON to ignore this field entirely. I will get back to you in a few hours, once I have tested this.

Would you be able to help me test something, as I can't repo this in my development tenant.
Can you edit the [Child] Admin | Sync Logs and navigate to the Parse JSON action

and replace the Schema that's currently there with this one:
{
    "type": "array",
    "items": {
        "type": "object",
        "properties": {
            "CreationTime": {
                "type": "string"
            },
            "Id": {
                "type": "string"
            },
            "Operation": {
                "type": "string"
            },
            "OrganizationId": {
                "type": "string"
            },
            "RecordType": {
                "type": "integer"
            },
            "ResultStatus": {
                "type": "string"
            },
            "UserKey": {
                "type": "string"
            },
            "UserType": {
                "type": "integer"
            },
            "Version": {
                "type": "integer"
            },
            "Workload": {
                "type": "string"
            },
            "AppName": {
                "type": "string"
            },
            "ObjectId": {
                "type": "string"
            },
            "UserId": {
                "type": "string"
            },
            "CrmOrganizationUniqueName": {
                "type": "string"
            },
            "Fields": {
                "type": "array"
            },
            "InstanceUrl": {
                "type": "string"
            },
            "ItemType": {
                "type": "string"
            },
            "ItemUrl": {
                "type": "string"
            },
            "UserAgent": {
                "type": "string"
            },
            "CorrelationId": {
                "type": "string"
            },
            "EntityId": {
                "type": "string"
            },
            "EntityName": {
                "type": "string"
            },
            "Message": {
                "type": "string"
            },
            "PrimaryFieldValue": {
                "type": "string"
            },
            "Query": {
                "type": "string"
            },
            "QueryResults": {
                "type": "string"
            },
            "ServiceContextId": {
                "type": "string"
            },
            "ServiceContextIdType": {
                "type": "string"
            },
            "ServiceName": {
                "type": "string"
            },
            "SystemUserId": {
                "type": "string"
            },
            "UserUpn": {
                "type": "string"
            },
            "AdditionalInfo": {
                "type": "string"
            },
            "Name": {
                "type": "string"
            }
        }
    }
}

I removed the ClientIP/ClientIPs attribute from the JSON as we are not using the data so can safely ignore it. Please save the flow and wait for the next run to see if that solves the issue and completes successfully for you!
Thank you

Would you be able to help me test something, as I can't repo this in my development tenant.
Can you edit the [Child] Admin | Sync Logs and navigate to the Parse JSON action

and replace the Schema that's currently there with this one:
{
    "type": "array",
    "items": {
        "type": "object",
        "properties": {
            "CreationTime": {
                "type": "string"
            },
            "Id": {
                "type": "string"
            },
            "Operation": {
                "type": "string"
            },
            "OrganizationId": {
                "type": "string"
            },
            "RecordType": {
                "type": "integer"
            },
            "ResultStatus": {
                "type": "string"
            },
            "UserKey": {
                "type": "string"
            },
            "UserType": {
                "type": "integer"
            },
            "Version": {
                "type": "integer"
            },
            "Workload": {
                "type": "string"
            },
            "AppName": {
                "type": "string"
            },
            "ObjectId": {
                "type": "string"
            },
            "UserId": {
                "type": "string"
            },
            "CrmOrganizationUniqueName": {
                "type": "string"
            },
            "Fields": {
                "type": "array"
            },
            "InstanceUrl": {
                "type": "string"
            },
            "ItemType": {
                "type": "string"
            },
            "ItemUrl": {
                "type": "string"
            },
            "UserAgent": {
                "type": "string"
            },
            "CorrelationId": {
                "type": "string"
            },
            "EntityId": {
                "type": "string"
            },
            "EntityName": {
                "type": "string"
            },
            "Message": {
                "type": "string"
            },
            "PrimaryFieldValue": {
                "type": "string"
            },
            "Query": {
                "type": "string"
            },
            "QueryResults": {
                "type": "string"
            },
            "ServiceContextId": {
                "type": "string"
            },
            "ServiceContextIdType": {
                "type": "string"
            },
            "ServiceName": {
                "type": "string"
            },
            "SystemUserId": {
                "type": "string"
            },
            "UserUpn": {
                "type": "string"
            },
            "AdditionalInfo": {
                "type": "string"
            },
            "Name": {
                "type": "string"
            }
        }
    }
}

I removed the ClientIP/ClientIPs attribute from the JSON as we are not using the data so can safely ignore it. Please save the flow and wait for the next run to see if that solves the issue and completes successfully for you!
Thank you

Hi Manuela,
I used your JSON but it failed again. So I checked the JSON output of the failed cluster. I found "ResultStatus":null
{"Activity":"ConnectFromExternalApplication","Id":"9faa99d4-2c66-42f0-8e04-4dacca1f4507","RecordType":20,"CreationTime":"2021-01-07T06:59:42","Operation":"ConnectFromExternalApplication","OrganizationId":"00000000-0000-0000-0000-00000000","UserType":0,"UserKey":"john.doe@contoso.com","Workload":"AS","ResultStatus":null,"UserId":"john.doe@contoso.com","ClientIP":null,"CustomData":""}
So, I removed ResultStatus from JSON as well. We triggered the flow 3times within 15mins and it finished successfully.
We have a question now. What is the impact of removed elements? Can it cause any issues later?
Thanks
Jakub

Hi Manuela,
I used your JSON but it failed again. So I checked the JSON output of the failed cluster. I found "ResultStatus":null
{"Activity":"ConnectFromExternalApplication","Id":"9faa99d4-2c66-42f0-8e04-4dacca1f4507","RecordType":20,"CreationTime":"2021-01-07T06:59:42","Operation":"ConnectFromExternalApplication","OrganizationId":"00000000-0000-0000-0000-00000000","UserType":0,"UserKey":"john.doe@contoso.com","Workload":"AS","ResultStatus":null,"UserId":"john.doe@contoso.com","ClientIP":null,"CustomData":""}
So, I removed ResultStatus from JSON as well. We triggered the flow 3times within 15mins and it finished successfully.
We have a question now. What is the impact of removed elements? Can it cause any issues later?
Thanks
Jakub

Hello,
thank you, that's great news - this won't have an impact later on, the only fields we use to update the Audit Log table are these ones:
 "CreationTime": {
                "type": "string"
            },
            "Id": {
                "type": "string"
            },
            "Operation": {
                "type": "string"
            },
            "Workload": {
                "type": "string"
            },
            "AppName": {
                "type": "string"
            },
            "ObjectId": {
                "type": "string"
            },
            "UserId": {
                "type": "string"
            },
            "UserKey": {
                "type": "string"
            },

The ResultStatus field is not one we use.
The initial schema was put together based on a sample of an Audit Log event, under the assumption that the schema for Audit Log event is consistent. As that's not the case, I'll work on updating the Parse JSON schema to only require fields we use later on and remove all optional fields we don't.
Thank you for your help in testing this!

Hello,
thank you, that's great news - this won't have an impact later on, the only fields we use to update the Audit Log table are these ones:
 "CreationTime": {
                "type": "string"
            },
            "Id": {
                "type": "string"
            },
            "Operation": {
                "type": "string"
            },
            "Workload": {
                "type": "string"
            },
            "AppName": {
                "type": "string"
            },
            "ObjectId": {
                "type": "string"
            },
            "UserId": {
                "type": "string"
            },
            "UserKey": {
                "type": "string"
            },

The ResultStatus field is not one we use.
The initial schema was put together based on a sample of an Audit Log event, under the assumption that the schema for Audit Log event is consistent. As that's not the case, I'll work on updating the Parse JSON schema to only require fields we use later on and remove all optional fields we don't.
Thank you for your help in testing this!

Hello Manuela,
the flow executes successfully since last week. It is solved.
Thank you for your help,
Jakub

Hello Manuela,
the flow executes successfully since last week. It is solved.
Thank you for your help,
Jakub

Great news, thanks for letting me know! I will make the same update in the next version then to ensure future customers don't experience the same issue. Thank you for your help in testing this!

Great news, thanks for letting me know! I will make the same update in the next version then to ensure future customers don't experience the same issue. Thank you for your help in testing this!

It seems that chaning the "type":"string" to "type":"[string,null]" in the schema to make the fields nullable also resolves the issue

Hi @manuelap-msft,
I'm still facing this issue in the v1.3.5. Parse JSON Schema isn't updated as you mentioned. Once done, it works well.

When you updated, did you check to remove unmanaged layers from the flow?
This is described in installing updates.
Unfortunately, if unmanaged layers exist you wont see the changes from the updated install.

I well check to remove unmanaged layers. I just cross-checked again and I'm still see the full json schema instead of the one proposed by Manuela.

re-opening as it seems the fix may not have gone out.

validated on another install that this change is live


Defining code style for Introspection features

Multiple asserts use introspection, among others the isEqualToComparingFieldByField() method.
This method expect that the fields be public, or accessible through a getter. First of all, the documentation should warn the user that only the JavaBeans conventions is supported.
A common convention (especially among Android programmers) is to prefix member variables with a lowercase m. For instance, a field which would be named "foo" in JavaBeans would be named "mFoo". In both cases, the getter is named getFoo().
Problem is that the Introspection class only uses the JavaBean introspector which doesn't recognizes the second convention.
It would be a great benefit to allow the use of a custom inspector, or adding support to other field naming conventions.

In 2.0, private fields are used in field by field comparison (this can be disabled but it's the default behaviour in 2.0).
This should meet your need unless I'm mistaken, if so please close the issue.
2.0 is expected to be released in few weeks.

Indeed this will meet my needs, thanks a lot


Update to 2024.6.1

Recently imagecodecs version 2024.6.1 was released. This is notable as it includes support for NumPy 2 ( https://github.com/cgohlke/imagecodecs/issues/100 )
However it looks like the bot ran into some issues performing this update ( https://github.com/regro/cf-scripts/issues/2691 ). So this may need to be done manually

@conda-forge-admin , please update version


Rename 'Object' to 'CoreObject'

The name Object is too generic and could clash with other libraries.

Here's a good one to ensure we follow through with.  I think that eliminating the name Object is one of the last barriers to a good 1.0 release.


IOS Post-clone NOT Working

Hi there I tried to build flutter with app center the android build done well but the ios build still throw error when strat the post clone script as the below
Task         : Shell Script
Description  : Run a shell script using bash
Version      : 2.1.3
Author       : Microsoft Corporation
Help         : More Information
[command]/bin/bash /Users/runner/runners/2.164.6/work/1/s/ios/appcenter-post-clone.sh
(node:931) Warning: Use Cipheriv for counter mode of aes-256-ctr
(node:931) Warning: Use Cipheriv for counter mode of aes-256-ctr
(node:931) Warning: Use Cipheriv for counter mode of aes-256-ctr
(node:931) Warning: Use Cipheriv for counter mode of aes-256-ctr
(node:931) Warning: Use Cipheriv for counter mode of aes-256-ctr
(node:931) Warning: Use Cipheriv for counter mode of aes-256-ctr
(node:931) Warning: Use Cipheriv for counter mode of aes-256-ctr
(node:931) Warning: Use Cipheriv for counter mode of aes-256-ctr
/Users/runner/runners/2.164.6/work/1/s/ios/appcenter-post-clone.sh: line 3: cd: ..
: No such file or directory
fatal: destination path 'flutter' already exists and is not an empty directory.
/Users/runner/runners/2.164.6/work/1/s/ios/appcenter-post-clone.sh: line 6: flutter: command not found
/Users/runner/runners/2.164.6/work/1/s/ios/appcenter-post-clone.sh: line 7: flutter: command not found
Installed flutter to /Users/runner/runners/2.164.6/work/1/s/ios/flutter
/Users/runner/runners/2.164.6/work/1/s/ios/appcenter-post-clone.sh: line 9: flutter: command not found
##[error]/bin/bash failed with return code: 127
##[error]Bash failed with error: /bin/bash failed with return code: 127
##[section]Finishing: Post Clone Script
##[section]Starting: Xcode build (not signed)
please help me
thanks alot

Hi. Any progress ? I have same issue right now.


Move fluentbit deployment to fleet and helm

Deploy fluent bit through helm charts and fleet. Also split up the configs for kube and external ingestation, since some clusters will only need the kube logs shipped.
This currently uses the prom_cluster label to distinguish which cluster is sending which logs. These will also be used in further dashboards being made available. In the future this label might change, prom_cluster might be a weird term since it's not Prometheus sending this data. But by using this label we have a correlation with the metric labels that can be used in searching for information.
Tested and confirmed working in ayekan after shaking the Opensearch nodes back into working action.

@jhoblitt I might need you're input on how to best handle https://github.com/lsst-it/k8s-cookbook/pull/504/files#diff-0a6c1c007a9df1475a5e46c866ee3107054596aef7719ad66928773f3bf43bcdR44 here. This creates the services that is a LoadBalancer type since jamming all the possible log TCP traffic through the ingress sounds like a bad idea and it also allows for the UDP traffic. For other clusters this can be dynamically assigned, but that allows for misalignment of the load balancer IP and the DNS name used.
I'm think and overlay should probably to the heavy lifting here and an IP from the load balancer is hard assigned.

I haven't looked at this yet but, FYI, I started testing fluent-operator on kueyen several weeks ago. https://github.com/lsst-it/k8s-cookbook/compare/master...IT-5176/fluent-client

@jhoblitt I might need you're input on how to best handle https://github.com/lsst-it/k8s-cookbook/pull/504/files#diff-0a6c1c007a9df1475a5e46c866ee3107054596aef7719ad66928773f3bf43bcdR44 here. This creates the services that is a LoadBalancer type since jamming all the possible log TCP traffic through the ingress sounds like a bad idea and it also allows for the UDP traffic. For other clusters this can be dynamically assigned, but that allows for misalignment of the load balancer IP and the DNS name used.
I'm think and overlay should probably to the heavy lifting here and an IP from the load balancer is hard assigned.

The way we have been handling this is setting an annotation in an overlay that sets the specific IP address.

Rebased on current master.


'heebo.css' requests always return html root

Summary
According to the browser network logs, requests to assets/styles/heebo.css are invalid. Accessing the resource directly shows that the index html is returned instead of any css. A base_href envvar is in use, but it seems reproducible without it.
This is a namespaced install.
Other resource seem simarly affected, from the browser console log:
Refused to apply style from 'https://<snip>/infra/argo/assets/styles/heebo.css' because its MIME type ('text/html') is not a supported stylesheet MIME type, and strict MIME checking is enabled.

DevTools failed to load SourceMap: Could not parse content for https://<snip>/infra/argo/ReactToastify.css.map: Unexpected token < in JSON at position 0

DevTools failed to load SourceMap: Could not parse content for https://<snip>/infra/argo/swagger-ui.css.map: Unexpected token < in JSON at position 0

Diagnostics
What Kubernetes provider are you using?

AKS

What version of Argo Workflows are you running?

2.11.8

hugh@dev:~$ k get deployment -n argo argo-server -o yaml | grep image:
        image: argoproj/argocli@sha256:9a268191b1a877004bef3d47a8bf9527af78e03fea1b872afd55ce3fda82fae6
hugh@dev:~$ kubectl get pods --field-selector=status.phase=Running -n argo -o wide
NAME                                  READY   STATUS    RESTARTS   AGE     IP            NODE                           NOMINATED NODE   READINESS GATES
argo-server-fcb4cd55c-vzghg           1/1     Running   0          9m19s   <IP_ADDRESS>   aks-main-26567389-vmss000001   <none>           <none>
minio-7d94dd75db-95n7g                1/1     Running   0          14d     <IP_ADDRESS>   aks-main-26567389-vmss000000   <none>           <none>
workflow-controller-d6479fd97-jrltb   1/1     Running   0          9m18s   <IP_ADDRESS>   aks-main-26567389-vmss000001   <none>           <none>
hugh@dev:~$ kubectl run --rm -it -n argo --image=alpine:3.12 shell -- sh
If you don\'t see a command prompt, try pressing enter.
/ # apk add curl
<snip>
OK: 7 MiB in 18 packages
/ # curl <IP_ADDRESS>:2746
<!DOCTYPE html>

    <meta charset="UTF-8">
    <title>Argo</title>
    <base href="/infra/argo/">
    <link href="assets/styles/heebo.css" rel="stylesheet">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <meta name="robots" content="noindex">
    <link rel="icon" type="image/png" href="assets/favicon/favicon-32x32.png" sizes="32x32">
    <link rel="icon" type="image/png" href="assets/favicon/favicon-16x16.png" sizes="16x16">

/ # curl <IP_ADDRESS>:2746/assets/styles/heebo.css
<!DOCTYPE html>

    <meta charset="UTF-8">
    <title>Argo</title>
    <base href="/infra/argo/">
    <link href="assets/styles/heebo.css" rel="stylesheet">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <meta name="robots" content="noindex">
    <link rel="icon" type="image/png" href="assets/favicon/favicon-32x32.png" sizes="32x32">
    <link rel="icon" type="image/png" href="assets/favicon/favicon-16x16.png" sizes="16x16">

/ # curl <IP_ADDRESS>:2746/infra/argo/assets/styles/heebo.css
<!DOCTYPE html>

    <meta charset="UTF-8">
    <title>Argo</title>
    <base href="/infra/argo/">
    <link href="assets/styles/heebo.css" rel="stylesheet">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <meta name="robots" content="noindex">
    <link rel="icon" type="image/png" href="assets/favicon/favicon-32x32.png" sizes="32x32">
    <link rel="icon" type="image/png" href="assets/favicon/favicon-16x16.png" sizes="16x16">

/ # Session ended, resume using 'kubectl attach shell -c shell -i -t' command when the pod is running
pod "shell" deleted
hugh@dev:~$

Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

thank you for the super-quick analysis/resolution!


[FEATURE] Update the slider section and put their description

Detailed description
View link https://www.figma.com/file/rmCKKIsjSWCH0q6lObDQC8/Dev-Protocol-design?node-id=0%3A1
Please put their title and description as well
HiDE
HiDE is a decentralized blogging platform from Japan. In Japan, blogging projects like Steem It did not last, but they used the Dev Protocol to monetize blog posts and create a sustainable reward distribution model for users. They are a newly founded startup, but have made $60,000 in revenue in six months with Stakes.social and are about to start giving back to their blog users.
Sindre
He is one of the most influential independent developers in the OSS fields. He has developed over 1000 OSS projects and lives as a full-time Open Sourcer. One of his famous OSS project, Chalk is downloaded 90 million times a week, but he only receives $360 a month in donations on Patreon. He has been involved with Dev Protocol as an MVP user since 2019, and has given back about $2.1M in revenue earned there to GitHub's most active developers this year.
Context

Possible implementation
The slider should be automatic and the arrow button with their name is clickable
Additional information
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Closed because of this changes #47

Sorry wrong, I thought this was the same issue as our current issue.

A component can be made in vue if you prefer, then I'll add it to the page using partial hydration

Nice @ewatch, we really needed this one tonight but we are new to it xD. I'll assign this to you @ewatch. Feel free to join us on discord for continuous conversation


Dev 2.0.0 cleanup+indentation refactor

The indentation of some functions can be refactored to make for cleaner code.

already committed to https://github.com/rdmorganiser/rdmo/pull/624


Extend functionality of DB

Excellent!

Only problem with serialize/unserialize is that it's quite hard to debug. Maybe I'll switch to Flywheel again, which I've used before at the CQRS workshop (https://github.com/ibuildingsnl/cqrs-workshop-code/blob/master/src/Twitsup/ReadModel/UserProfileRepository.php) - https://github.com/jamesmoss/flywheel


Extract Content from CNCF sources

User story

As a Data Engineer
I need to extract the content from CNCF sources
So that i can create a dataset with the gathered data.

Acceptance criteria

[ ] Data can be extracted

[ ] Data in Markdown Format from CNCF sources can be extracted
[ ] Data in pdf-Format from CNCF sources can be extracted
[ ] Data in yml/yaml-Format from CNCF sources can be extracted

[ ] Implemented strategy for failing requests

[ ] Failed requests are retried if the cause is temporary
[ ] Data is dropped if retry is not possible or retries exceeded

[ ] The data can be extracted in chunks (Not all data must be processed at once)

DoD general criteria

Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release

Files are uploaded in Huggingface. Separated by each category in CNCF.
https://huggingface.co/datasets/anosh-rezaei/CNCF_projects

reestimated to one as it is already nearly finished

Extracted files uploaded to huggingface, yml files and non-English files excluded.
https://huggingface.co/datasets/anosh-rezaei/CNCF_exclude_yml/tree/main


Error when attempting to install ape-safe via pip install and ape plugin install [APE-1367]

Environment information

ape and plugin versions:

(venv) user@Users-Laptop safe % ape --version
0.6.15
(venv) user@Users-Laptop safe % ape plugins list
No plugins installed. Use '--all' to see available plugins.

Python Version: 3.10.12
OS: OSX

What went wrong?
Error when attempting to install ape-safe via pip install and ape plugin install
(venv) user@Users-Laptop safe % ape plugins install safe
INFO: Installing safe...
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [86 lines of output]

      Error compiling Cython file:
      ------------------------------------------------------------
      ...
          cdef readonly object func
          cdef readonly tuple args
          cdef readonly dict keywords
          cdef public object __doc__
          cdef public object __name__
          cdef public object __module__
                             ^
      ------------------------------------------------------------

      cytoolz/functoolz.pxd:18:23: '__module__' redeclared

      Error compiling Cython file:
      ------------------------------------------------------------
      ...
          cdef readonly object func
          cdef readonly tuple args
          cdef readonly dict keywords
          cdef public object __doc__
          cdef public object __name__
          cdef public object __module__
               ^
      ------------------------------------------------------------

      cytoolz/functoolz.pxd:18:9: Compiler crash in AnalyseDeclarationsTransform

      File 'ModuleNode.py', line 203, in analyse_declarations: ModuleNode(functoolz.pxd:1:0,
          full_module_name = 'cytoolz.functoolz',
          is_pxd = True)
      File 'Nodes.py', line 393, in analyse_declarations: StatListNode(functoolz.pxd:1:0)
      File 'Nodes.py', line 5497, in analyse_declarations: CClassDefNode(functoolz.pxd:10:5,
          as_name = 'curry',
          class_name = 'curry',
          in_pxd = True,
          module_name = '',
          punycode_class_name = 'curry',
          visibility = 'private')
      File 'Nodes.py', line 393, in analyse_declarations: StatListNode(functoolz.pxd:11:4)
      File 'Nodes.py', line 1528, in analyse_declarations: CVarDefNode(functoolz.pxd:18:9,
          in_pxd = True,
          modifiers = [...]/0,
          visibility = 'public')

      Compiler crash traceback from this point on:
        File "/private/var/folders/3_/z8rn91w566l816hsprr2k4w80000gn/T/pip-build-env-e05_0hgb/overlay/lib/python3.11/site-packages/Cython/Compiler/Nodes.py", line 1528, in analyse_declarations
          self.entry = dest_scope.declare_var(
                       ^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/3_/z8rn91w566l816hsprr2k4w80000gn/T/pip-build-env-e05_0hgb/overlay/lib/python3.11/site-packages/Cython/Compiler/Symtab.py", line 2389, in declare_var
          entry = self.declare(name, cname, type, pos, visibility)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/3_/z8rn91w566l816hsprr2k4w80000gn/T/pip-build-env-e05_0hgb/overlay/lib/python3.11/site-packages/Cython/Compiler/Symtab.py", line 550, in declare
          entries[name].already_declared_here()
        File "/private/var/folders/3_/z8rn91w566l816hsprr2k4w80000gn/T/pip-build-env-e05_0hgb/overlay/lib/python3.11/site-packages/Cython/Compiler/Symtab.py", line 262, in already_declared_here
          error(self.pos, "Previous declaration is here")
        File "/private/var/folders/3_/z8rn91w566l816hsprr2k4w80000gn/T/pip-build-env-e05_0hgb/overlay/lib/python3.11/site-packages/Cython/Compiler/Errors.py", line 178, in error
          raise InternalError(message)
      InternalError: Internal compiler error: Previous declaration is here
      [1/5] Cythonizing cytoolz/dicttoolz.pyx
      [2/5] Cythonizing cytoolz/functoolz.pyx
      Traceback (most recent call last):
        File "/Users/user/.local/pipx/shared/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/user/.local/pipx/shared/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/user/.local/pipx/shared/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/3_/z8rn91w566l816hsprr2k4w80000gn/T/pip-build-env-e05_0hgb/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/3_/z8rn91w566l816hsprr2k4w80000gn/T/pip-build-env-e05_0hgb/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/3_/z8rn91w566l816hsprr2k4w80000gn/T/pip-build-env-e05_0hgb/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 507, in run_setup
          super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
        File "/private/var/folders/3_/z8rn91w566l816hsprr2k4w80000gn/T/pip-build-env-e05_0hgb/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 85, in <module>
        File "/private/var/folders/3_/z8rn91w566l816hsprr2k4w80000gn/T/pip-build-env-e05_0hgb/overlay/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize
          cythonize_one(*args)
        File "/private/var/folders/3_/z8rn91w566l816hsprr2k4w80000gn/T/pip-build-env-e05_0hgb/overlay/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one
          raise CompileError(None, pyx_file)
      Cython.Compiler.Errors.CompileError: cytoolz/functoolz.pyx
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

ERROR: Failed to install plugin 'safe.
How can it be fixed?
Fill this in if you have ideas on how the bug could be fixed.

Solved via direct install from GitHub and not via the released versions


Candidate selection behavior change with multiple measures

We currently have the pipeline selecting candidates that we don't like, for instance with Provider_29
Need to have Esteemer or somewhere else use either tweaks to the algorithm's weights or additional parameters to prevent measures from becoming selected candidates where the data is insufficient for feedback.

@mackgalante  Is this issue already addressed ? Can you give more information.

Apologies for leaving this open, was related to the root causes that were corrected in the pull request for #172

Closing now


Intrinsic functions of cloudformationm yaml do not convert correctly

That's true, they are not loaded according to the Node convention
I'll check it out in a couple of days


Improve contractions involving scalar-like ITensors or blocks

This bumps NDTensors to v0.1.20, which has optimizations for contractions involving scalar-like tensors (https://github.com/ITensor/NDTensors.jl/pull/57). This particularly improves the efficiency of block sparse contractions that involve many blocks of size 1 (which is common for MPO tensors of local Hamiltonians).

Here are results for the 1D S=1 Heisenberg model with chi = 800, 4 BLAS threads, with and without splitblocks, using NDTensor master:
julia> main(800; use_splitblocks = false)
After sweep 1 energy=-138.940086143531 maxlinkdim=800 time=51.740
 53.246273 seconds (23.64 M allocations: 170.844 GiB, 2.88% gc time)

julia> main(800; use_splitblocks = true)
After sweep 1 energy=-138.940086143531 maxlinkdim=800 time=41.392
 42.461887 seconds (21.85 M allocations: 107.442 GiB, 2.99% gc time)

while using NDTensors before the scalar contraction optimizations:
julia> main(800; use_splitblocks = false)
After sweep 1 energy=-138.940086143531 maxlinkdim=800 time=52.738
 54.213462 seconds (29.86 M allocations: 171.229 GiB, 2.95% gc time)

julia> main(800; use_splitblocks = true)
After sweep 1 energy=-138.940086143531 maxlinkdim=800 time=49.531
 51.053270 seconds (44.21 M allocations: 152.692 GiB, 2.99% gc time)

so there is not much change without using splitblocks, but a nearly 20% speedup using splitblocks.


Eliminate the need to reflect the project structure in imports.

Removed relative paths in imports and changed cmake configuration so that header files are still found and compiled when compiling. I'm using clangd, so I have another configuration file so that the development environment doesn't give an error about the header file not being found. I can also commit it if needed.

I think the same procedure as in commits 0d478d61c807a57deaf4829dd077ddec7b0182a5 and f416705792fe274a6c33b7ff41836fbe0bae360c should be done with the other two files in the languages folder. this will make it possible not to worry in the future about the order of inclusion of header files.


fn fill2d_16x2: Prepare for constification

fn fill2d_16x2 is difficult to make const directly since it takes &'static mut args that are referenced in the return value.  I'm going to switch to a different API that can make this work, but these are some initial steps to prepare for constification before that.

Huh, so dst and masks_444 were pointing to the same thing? Does that mean that the code was UB before since we were aliasing mutable references? Either way glad you fixed that ^__^

Yeah, I think so.  And originally it was okay since they were raw pointers.  I'm not sure why the borrow checker didn't stop that (me making them references), though, maybe because they were statics.


Slow loading time

Hi, I'm just testing this app out by cloning and running it on my device. The APK is 19.8 Mb and takes about 8 seconds to show the main view.
I tried running with --release flag and the APK is 19.7 Mb (0.1 Mb difference) and loading time does not improve.
Anything that I missed out?
Also, just some suggestion in the readme file.
git clone --depth 1 git@github.com:NativeScript/nativescript-sdk-examples-ng.git
should change to:
git clone --depth 1 https://github.com/NativeScript/nativescript-sdk-examples-ng.git

Hey @davistan - the size of the app is due to the large amount of pages and images in the app (~170 pages).
About the loading time - yes you are correct and we are putting a lot of efforts focused on improving drastically the loading time of this application - especially on Android. Our target for this app is to load between 1 and 2 seconds . We already have a POC internally that optimizes this and we are in a process of enabling it.
We have several efforts running together - both inside the core framework and in the Angular SDKs that will help us achieve this in the next several weeks - our target date is with the 2.5 release.
Please keep us posted on any other improvements you would like to see.

Cool! I saw the NS 2.4 release notes for the V8 symbols removal and to gauge the performance difference, hence I tried on this project.
Anyway, good to know that's in the pipeline and will wait for updates from you guys.
Keep it up!

Hi @valentinstoychev,
At the moment I'm working on a small app for personal use which is sort of an evaluation of NS. App has 2 views, it sends, reads and receives smses, has a listview, uses several third-party and own plugins. Current apk file is 23.7MB, there might be some problem with it https://github.com/NativeScript/nativescript-cli/issues/2269, but so we can skip this one for now.
When apk file is opened and then the tns_modules folder, it becomes quite obvious that AngularJS related modules are the largest one. The start up time (splashscreen to main view) was about 4-5 seconds, but after adding a couple of plugins it became around 6, which is a lot for such simple application.
Thus I'm having following thoughts and would really appreciate if you could validate them:

everything that is possible to do with Angular should be possible to implement via core API and still use Typescript. Yes, Angular provides some nice built-in things, however in the context of NS (and mobile development) those might not be necessary, moreover, could be harmful: ngFor is not virtualized, personally ran into ListView problems while using DatePipe transformation (quick scrolling wasn't smooth) and the solution was to preprocess all the data and use ListView to just output. Some stackoverflow posts suggest that not using Angular quite nicely improved the start-up time.

potentially more overhead while using Angular at run-time which could introduce occasional minor lags. I do get those with my current implementation, however I will yet need to re-implement another version of it without Angular to compare, so this one is more of a guess, yet common sense says that the less JS engine needs to handle the better things are :)

output package file size, start-up time and the first "just in time compilation". First one is quite obvious: the smaller - the better. Start-up time is a big one in here. I expect my app to start under three seconds, ideally under two, but waiting six seconds on Nexus 5 a bit of a pain. Also I noticed that the first time I trigger, say navigation, it takes longer to perform the action. Then occasionally when navigation is triggered instead of going to page B and showing the activity indicator app freezes for around a second and then shows the Page B with the data already loaded.

UX over developer comfort: I'd rather give-up Angular and write a reasonable amount of extra boilerplate code than sacrifice a) run-time performance b) start-up time c) app package size

What do you think of these in the light of expected performance optimizations coming in 2.5.0? Should I switch to just core API + TS or am I over-exaggerating the Angular's impact and doing a lot of guess work? :)
Thank you!

Hello, I've tested my project and I am experiencing the same issues, startup really slow about 6 seconds, refreshing flex box layout items with ngfor takes about 4 seconds with simple array populated with 10 items... And the most terrible is navigating to another page and then navigate back it takes 5..6 seconds each navigation transition  and sometimes it freezes...
I am using NS 2.5.0 Angular and typescript testing on Android target 5.1.1 galaxy tab 4, any suggestions to improve performance?

@felipewom Same here.... Or no, even worse.... I've just installed an app of my own... 10 to 13 seconds... So, I guessed I'd do the LazyNinjas.... Both on release mode.... Both over 10 seconds.... my phone is not slow.... So, I guess I'm doing something wrong here....? Any thoughts?

@valentinstoychev Hi Valentin!... You've mentioned somewhere here above that your team was working on an improvement of the loading time... And after that, your colleague @sis0k0  Stanimira wrote an article about Lazy Loading, which I follow in my sources..... How are your thoughts about the loading times I mentioned here above?? What do I do wrong??

@Dutchboy I'm a bit late to the party but let me share a few things:

If i start a project with the angular template i get 10~11 secs load time consistently.
If i install the android-snapshot plugin it goes down to 6~7 secs load time.
If i install the nativescript-dev-webpack package it goes down to 4~5 secs (Please note that this and the previous one are incompatible with each other).

Please note that:

4~5 secs is the best i have achieved even after uglifying the app (with --uglify flag).
This is the starter angular template, doesn't make sense that an empty app takes 10~11 secs right after creating the app.
I feel like it's damn slow after comparing it to React Native 2~3 secs max load time (which already comes with React included, my only complain is that i cannot make it work with typescript).

下载完成后，执行tns run ios报这个错误
Unable to apply changes on device: 695247EC-9384-4827-8C47-D8FD81125BEF. Error is: nativescript-googlemaps has Podfile and you don't have Cocoapods installed or it is not configured correctly. Please verify Cocoapods can work on your machine..
tns install 报如下错误
Error: ENOENT: no such file or directory, open '/Users/weiweibing/Desktop/nativescript-sdk-examples-ng/node_modules/node-pre-gyp/package.json'
Error: ENOENT: no such file or directory, open '/Users/weiweibing/Desktop/nativescript-sdk-examples-ng/node_modules/node-pre-gyp/package.json'
请问如何解决


doc: Improve the contribution documentation

What does this PR do?
Improves the Contribution doc
Related ticket
Fixes #846
Type of change

[X] Chore
[ ] Bug fix
[ ] New feature
[ ] Documentation update

Check list

[ ] Unit tests for any smart contract change
[ ] Contracts and functions are documented

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 48.59%. Comparing base (ef284ae) to head (868f5d2).

Additional details and impacted files
@@           Coverage Diff           @@
##              dev     #847   +/-   ##
=======================================
  Coverage   48.59%   48.59%
=======================================
  Files          18       18
  Lines         391      391
  Branches       63       63
=======================================
  Hits          190      190
  Misses        198      198
  Partials        3        3

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


range select with shift key pressed

@emp3ror, I see you have added a very good feature, kindly add the specs (tests) for this feature as well. The specs are located in test/spec/selectMultipleSpec.js. We use Jasmine for tests.


Chained cipher performance on Denverton

My Rangeley CPU finally died on me last week and so I upgraded to a Denverton.
While I was already aware on Rangeley that offloading symmetric ciphers with buffer sizes any less than 4k resulted in significantly poorer performance than just using the CPU instructions and indeed prepared to see the same thing with Denverton, I was quite perplexed to see the results below.
QAT is working fine, as the results from the RSA speed test show. However, even with large buffers, the speed for chained ciphers on Denverton is way slower than not offloading.
Is this normal?
RSA results: more or less what I expected
$ openssl speed -engine qat -elapsed -async_jobs 72 rsa2048

                  sign    verify    sign/s verify/s
rsa 2048 bits 0.000105s 0.000013s   9486.0  74558.1

$ openssl speed -elapsed -async_jobs 72 rsa2048

                  sign    verify    sign/s verify/s
rsa 2048 bits 0.002705s 0.000078s    369.7  12815.9

AES-CBC: not at all what I expected. The results are the same even when the number of async jobs are upped to 72
$ openssl speed -engine qat -elapsed -async_jobs 32 -multi 2 -evp aes-128-cbc-hmac-sha1

evp             180743.19k   470527.89k   782870.78k   971217.58k   906024.28k   893157.38k

$ openssl speed -elapsed -async_jobs 32 -multi 2 -evp aes-128-cbc-hmac-sha1

evp             318426.91k   674715.48k   897365.59k  1002786.47k  1050793.30k  1052153.17k

Hi dl12345
Thank-you for your query/issue.  We shall be actively investigating this issue and shall get back to you with our findings shortly.
Regards,
paulturx

I'm curious as to whether you could reproduce these result...?


Venafi Issuer Read caBundle from Configmap or Secret

Is your feature request related to a problem? Please describe.
I would like to be able to specify the caBundle from the contents of a secret or configmap resource.
In my Issuer / ClusterIssuer resource you have a caBundle field that must be specified as a base64 encoded string, eg.
spec:
  venafi:
    tpp:
      caBundle: <B64_ENCODED_STRING>
      credentialsRef:
        name: tpp-token
      url: https://my-server.com/vedsdk/
    zone: Certificates\public

This becomes a configuration management issue when you have many issuers in a cluster using the same TPP endpoint, as you now have many places to copy the same CA information.
If using the cert-manager trust project, it would be a good practice to define the caBundle once and distribute that to all relevant namespaces in a cluster. However Trust outputs to either a Configmap or Secret resource.
Between the Issuer and the trust Bundle there is a disconnect.
Describe the solution you'd like
I could image for the Venafi issuer, you could have something like:
spec:
  venafi:
    tpp:
      caBundleRef: # A new key could be either or with the existing `caBundle`
        type: configMap  # configMap | secret | bundle
        name:  tpp-trust-bundle
        key: ca-certificates.crt
      credentialsRef:
        name: tpp-token
      url: https://my-server.com/vedsdk/
    zone: Certificates\public

Describe alternatives you've considered
The alternative is really to copy an paste the output into the Issuer configuration. I'm not sure there are any others.
Additional context
Most consumers of the Venafi integration would generally use few or only 1 actual TPP endpoint with a single CA source. By enabling the reference of a configMap or Secret resource, then this means users can effectively distribute their TPP CA via  trust bundle.
Environment details (remove if not applicable):
N/A
/kind feature

I've seen that in v1.10.0 release there is a small enhancement for the Vault Issuer: https://github.com/cert-manager/cert-manager/pull/5387 adding caBundleSecretRef.
This is very similar except it just covers the Vault issuer and only from a secret, no configmap support.

/remove-lifecycle stale

/remove-lifecycle stale

/assign sankalp-at-gh

Hey @sankalp-at-gh I didn't think I could but apparently I can. I've assigned you now.

Would be great if we can have this feature!

Design Details
Add Secret, ConfigMap, ClusterTrustBundle optional fields to the spec which can be used to refer a key to find the caBundle. Below is the example for venafi issuer.
This method is compatible with existing API's
spec:
  venafi:
    tpp:
      caBundle: <B64_ENCODED_STRING>
      caBundleSecretRef:
        name: <>
        key: <>
      caBundleConfigMapRef:
        name: <>
        key: <>
      clusterTrustBundleRef:
        name: <>
        key: <>
      credentialsRef:
        name: tpp-token
      url: https://my-server.com/vedsdk/
    zone: Certificates\public

@AppalaKarthik I think that I would remove the clusterTrustBundleRef from the above. The trust Bundle is a cluster level resource, so may not be accessible from an Issuer without extensive RBAC. It also isn't inteded to be consumed directly I don't think, instead you would make reference to the ConfigMap that it creates in each namespace selected and use the ca.crt from that configmap. That configmap will update when the Bundle is updated.
So I propose that you would have:
spec:
  venafi:
    tpp:
      caBundle: <B64_ENCODED_STRING>
      caBundleSecretRef:
        name: <>
        key: <>
      caBundleConfigMapRef:
        name: <>
        key: <>
      credentialsRef:
        name: tpp-token
      url: https://my-server.com/vedsdk/
    zone: Certificates\public

@hawksight  Makes sense to me. Became little over ambitious there. Like you said configmap covers it anyways

@AppalaKarthik I think that I would remove the clusterTrustBundleRef from the above. The trust Bundle is a cluster level resource, so may not be accessible from an Issuer without extensive RBAC. It also isn't inteded to be consumed directly I don't think, instead you would make reference to the ConfigMap that it creates in each namespace selected and use the ca.crt from that configmap. That configmap will update when the Bundle is updated.
So I propose that you would have:
spec:
  venafi:
    tpp:
      caBundle: <B64_ENCODED_STRING>
      caBundleSecretRef:
        name: <>
        key: <>
      caBundleConfigMapRef:
        name: <>
        key: <>
      credentialsRef:
        name: tpp-token
      url: https://my-server.com/vedsdk/
    zone: Certificates\public

@hawksight , seems there is a bit confusion here. The idea in @AppalaKarthik's approval was to support the kubernetes native  ClusterTrustBundles and not Trust Manager in cert-manager project, which indeed creates configmaps in target namespaces. Please let me know if not.
As far as access for ClusterTrustBundles is concerned, I see below mentioned in https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#ctb-common, from which I assume there would not be a need for extensive RBAC(I might be wrong too, needs more testing). And seems it doesn't create any configmaps in namespaces.
ClusterTrustBundle objects should be considered world-readable within the cluster. If your cluster uses RBAC authorization, all ServiceAccounts have a default grant that allows them to get, list, and watch all ClusterTrustBundle objects. If you use your own authorization mechanism and you have enabled ClusterTrustBundles in your cluster, you should set up an equivalent rule to make these objects public within the cluster, so that they work as intended.

Please let us know if support for ClusterTrustBundles can be slated for discussion at a later point and if we could proceed with proposal for just Secrets and ConfigMaps. We initially discussed about supporting ClusterTrustBundles when @irbekrm brought it up in one of the standups.
Ref: https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/3257-trust-anchor-sets

Hey @sankalp-at-gh - you are correct that I had defaulted to trust-manager and not realised about the new Cluster Trust Bundle resource. Thank you for pointing that out and the detail provided. I don't know much about this feature yet so I'll be sure to have a good look into it.
Just wanted to say I am not a maintainer, but I am very interested in this issue.
From an initial look, I think it makes sense to have this as an option in the Issuer configuration in the long term. However it is an alpha feature in 1.27 and therefore needs enables by users. It might also need some additional CI things to fully test adding this.

Please let us know if support for ClusterTrustBundles can be slated for discussion at a later point and if we could proceed with proposal for just Secrets and ConfigMaps.

Yes I agree with this, lets move forward with Secret and ConfigMap support first.
Cluster Trust Bundles probably needs some more discussion but I do think it's on people's minds that cert-manager would like to integrate with this feature.

Looking at what's needed to do this, in context of the sibling issue (https://github.com/cert-manager/cert-manager/issues/6117), I am not convinced this is a "good first issue". It will require modification to some internal controller mechanics and extension of RBAC. 😊

Looking at what's needed to do this, in context of the sibling issue (#6117), I am not convinced this is a "good first issue". It will require modification to some internal controller mechanics and extension of RBAC. 😊

agreed !
Even though I'm familiar with making such changes, just by the amount of time I needed to make these changes makes me feel this is a bit more for a "good first issue" too.
Please expect a PR soon as I'm almost done with the changes 😊

Hello @erikgb,@hawksight, opened up above pull today. Might need some help in taking it forward for a review.
Thanks

/remove-lifecycle stale

/remove-lifecycle stale

Hi @hawksight / @inteon , I have opened up PR #7036 which only supports secretref and includes all suggested fixes in #6454 .
Could you please review the PR and if it looks good, could you help me with how to go about updating documentation. Thanks !!

Hey @sankalp-at-gh - It looks really good to me. Thank your for taking the time to submit this.
Regarding the documentation, we have the cert-manager website here:

https://github.com/cert-manager/website

The page that comes to mind for me would be here:

https://cert-manager.io/docs/configuration/venafi/#creating-a-venafi-trust-protection-platform-issuer

And adding even a commented out example of using the caBundleSecretRef:
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: tpp-issuer
  namespace: <NAMESPACE YOU WANT TO ISSUE CERTIFICATES IN>
spec:
  venafi:
    zone: \VED\Policy\devops\cert-manager # Set this to the Venafi policy folder you want to use
    tpp:
      url: https://tpp.venafi.example/vedsdk # Change this to the URL of your TPP instance
      caBundle: <base64 encoded string of caBundle PEM file, or empty to use system root CAs>
      # Use only caBundle above or the caBundleSecretRef below
      # caBundleSecretRef:
      #  name: custom-tpp-ca
      #  key: ca.crt
      credentialsRef:
        name: tpp-secret

When you raise a PR for the docs update, please raise it against the release-next branch, and not master :)

Perfect, I'll keep the doc updates going while this PR is reviewed 🙂
Thank you @hawksight !

@hawksight @sankalp-at-gh @SimeonPoot
A pre-release is now available containing the fix for this issue. Please test and feedback if you have time.

https://github.com/cert-manager/cert-manager/releases/tag/v1.16.0-alpha.0

@hawksight @sankalp-at-gh @SimeonPoot
A beta-release is now available which contains the fix for this issue. Please test and feedback if you have time.

https://github.com/cert-manager/cert-manager/releases/tag/v1.16.0-beta.0


Add options to use the Intel inflater or deflater when reading or

writing BAM files. Use of this option requires that you provide
the GKL dependency (https://github.com/Intel-HLS/GKL) on the classpath.

Addresses #109

Coverage decreased (-0.05%) to 62.881% when pulling 39009f491469d2e1a04b245d08fb9431d0521665 on tomwhite:intel-inflater-and-deflater into 223cb8e7bb488cab7338613a367b6e4b5d63b65f on HadoopGenomics:master.

Coverage decreased (-0.08%) to 62.856% when pulling aeecd8ea7e576c79a94dffc3516ca2a0e94ad8a2 on tomwhite:intel-inflater-and-deflater into 71e83a7c7971fc7584c1a54a122d0fe5b6a069f7 on HadoopGenomics:master.


URL regex does not fully match every URL

For example:

I would like it to return both google.co and google.com. Sadly, it may be impossible considering the way regular expressions work. Thus, it would be amazing to match the longest string (pywhat google.com/help should return google.com/help). It is crucial for implementing URL subcategories properly(#51). Btw, URL regex is too long, I do not think that valid TLDs should be checked, so it may be shortened.

I do not think that valid TLDs should be checked

The reason we do this is because of false positives by the way :)

sad

Reopening this since pywhat does not match something like https://www.google.com fully.

Hey, @amadejpapez, do you have any ideas about this one?

Hey, @amadejpapez, do you have any ideas about this one?

Hm will check this later today and see if I get any.


router: retry on connect failure

Commit Message: retry on connect failure
Additional Description: see bellow
Risk Level: High
Testing: unit test and build locally
Docs Changes: No
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]
More context
Description:
When retrying on connection failure all requests having a body bigger than the connection buffer limit will not be retried.
Give the configuration bellow all requests bigger than 32768 bytes will not be retried. If not overridden per route it will come from the listener config per_connection_buffer_limit_bytes and default to 1MiB
The following happens:

the router receive the body and proxy it to the upstream requests
the buffer limit get reached at the router level
the router give up on retry but keep on forwarding to the upstream requests
the buffer of the upstream connection will get full and notify to stop readying on the downstream connection.

If the upstream requests fail the requests will not be retried.
Proposed solution:
The router can a callback for when the upstream connection get a connection via the addUpstreamCallbacks.
If the router buffer get full before any upstream connection got establish keep on buffering above the limit and call the onDecoderFilterAboveWriteBufferHighWatermark() callback to ensure we stop on reading from the downstream. This allows to keep on retrying will keeping the buffer at a value close to the desired limit.
Once it get a new connection enable drain the buffer which enable read from downstream and drop retry, shadow, and buffering.
For all other retry the behaviour is unchanged
Example:
config:
static_resources:
  listeners:
  - name: test_connect_failure
    address:
      socket_address:
        protocol: TCP
        address: <IP_ADDRESS>
        port_value: 8100
    per_connection_buffer_limit_bytes: 32768
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: AUTO
          stat_prefix: web1
          route_config:
            virtual_hosts:
            - name: backend
              domains:
              - "*"
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: local_service_fail
                  timeout: 0s
                  retry_policy:
                    retry_on: "connect-failure"
                    num_retries: 3
                    retry_back_off:
                      base_interval: 0.050s
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
  - name: local_service_fail
    connect_timeout: 1s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    upstream_connection_options:
      tcp_keepalive:
        keepalive_time: 300
    common_lb_config:
      healthy_panic_threshold:
        value: 40
    load_assignment:
      cluster_name: local_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: <IP_ADDRESS>
                # port doesn't match anything
                port_value: 8085

with Envoy docker image: "envoyproxy/envoy:v1.29.0":
only relevant logs:
[2024-02-13 09:25:46.619][21][debug][connection] [./source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"0"] current connecting state: false
[2024-02-13 09:25:46.619][21][debug][router] [source/common/router/router.cc:514] [Tags: "ConnectionId":"0","StreamId":"742379432996652670"] cluster 'local_service_fail' match for URL '/'
[2024-02-13 09:25:46.619][21][debug][router] [source/common/router/router.cc:731] [Tags: "ConnectionId":"0","StreamId":"742379432996652670"] router decoding headers:
':authority', 'localhost:8100'
':path', '/error'
':method', 'POST'
':scheme', 'http'
'user-agent', 'Go-http-client/1.1'
'content-length', '92768'
'accept-encoding', 'gzip'
'x-forwarded-proto', 'http'
'x-request-id', '00db4f6a-b6cb-4fe3-b4d4-34bad1e002f2'

[2024-02-13 09:25:46.619][21][debug][connection] [./source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"1"] current connecting state: true
[2024-02-13 09:25:46.619][21][debug][connection] [source/common/network/connection_impl.cc:1009] [Tags: "ConnectionId":"1"] connecting to <IP_ADDRESS>:8085
[2024-02-13 09:25:46.619][21][debug][connection] [source/common/network/connection_impl.cc:1028] [Tags: "ConnectionId":"1"] connection in progress
[2024-02-13 09:25:46.619][21][trace][connection] [source/common/network/connection_impl.cc:654] [Tags: "ConnectionId":"0"] read ready. dispatch_buffered_data=0
[2024-02-13 09:25:46.619][21][trace][connection] [source/common/network/raw_buffer_socket.cc:25] [Tags: "ConnectionId":"0"] read returns: 32768
[2024-02-13 09:25:46.619][21][debug][router] [source/common/router/router.cc:856] The request payload has at least 65412 bytes data which exceeds buffer limit 32768. Give up on the retry/shadow.
[2024-02-13 09:25:46.619][21][debug][http] [source/common/http/filter_manager.cc:541] [Tags: "ConnectionId":"0","StreamId":"742379432996652670"] request data too large watermark exceeded
[2024-02-13 09:25:46.619][21][debug][http] [source/common/http/conn_manager_impl.cc:1922] [Tags: "ConnectionId":"0","StreamId":"742379432996652670"] Read-disabling downstream stream due to filter callbacks.
[2024-02-13 09:25:46.619][21][trace][connection] [source/common/network/connection_impl.cc:401] [Tags: "ConnectionId":"0"] readDisable: disable=true disable_count=0 state=0 buffer_length=0
[2024-02-13 09:25:46.619][21][debug][connection] [source/common/network/connection_impl.cc:757] [Tags: "ConnectionId":"1"] delayed connect error: 111
[2024-02-13 09:25:46.619][21][debug][connection] [source/common/network/connection_impl.cc:278] [Tags: "ConnectionId":"1"] closing socket: 0
[2024-02-13 09:25:46.619][21][trace][connection] [source/common/network/connection_impl.cc:469] [Tags: "ConnectionId":"1"] raising connection event 0
[2024-02-13 09:25:46.619][21][debug][router] [source/common/router/router.cc:1331] [Tags: "ConnectionId":"0","StreamId":"742379432996652670"] upstream reset: reset reason: remote connection failure, transport failure reason: delayed connect error: 111
[2024-02-13 09:25:46.619][21][debug][http] [source/common/http/conn_manager_impl.cc:1912] [Tags: "ConnectionId":"0","StreamId":"742379432996652670"] Read-enabling downstream stream due to filter callbacks.
[2024-02-13 09:25:46.619][21][trace][connection] [source/common/network/connection_impl.cc:401] [Tags: "ConnectionId":"0"] readDisable: disable=false disable_count=1 state=0 buffer_length=0
[2024-02-13 09:25:46.619][21][debug][http] [source/common/http/filter_manager.cc:1065] [Tags: "ConnectionId":"0","StreamId":"742379432996652670"] Sending local reply with details upstream_reset_before_response_started{remote_connection_failure,delayed_connect_error:_111}
[2024-02-13 09:25:46.619][21][debug][http] [source/common/http/conn_manager_impl.cc:1869] [Tags: "ConnectionId":"0","StreamId":"742379432996652670"] encoding headers via codec (end_stream=false):
':status', '503'
'content-length', '152'
'content-type', 'text/plain'
'date', 'Tue, 13 Feb 2024 09:25:46 GMT'
'server', 'envoy'
'connection', 'close'

Test

unit test
integration test
tested in one AZ of a staging environment, saw the UF response flags going away

/retest

also opened an issue: https://github.com/envoyproxy/envoy/issues/32464

@fl0Lec thank you for the change. Can you describe the problem that you are trying to solve, please? I.e. is there an open issue you are fixing?
I think retries on TCP connection failures should already working. Are they not?
@alyssawilk for more comments.
/wait-any

+1 - I think it would help if you could write an integration test which simulates these conditions, fails without your change and succeeds with your change.  Mind giving that a shot?
/wait

Make sense, I forgot to add the link to the issue I created: https://github.com/envoyproxy/envoy/issues/32464
This add more context on the problem that I am trying to solve here.
I will look into adding an integration test

Ah, with the attached issue that makes much more sense, thanks so much.  So to summarize if we'd be willing to buffer X bytes in an upstream send buffer before applying flow control we should be willing to buffer X bytes (larger than the default of Y) in the retry buffer when we don't haven a upstream connection?   As long as we cancel the retry once we connect so we don't buffer the extra bytes for a long duration (which it looks like you're doing) I think that's sensible.   Definitely lets get e2e tests, ideally a bit more commenting, and runtime guard as well as it's a pretty high risk change.

I added per your suggestion @alyssawilk:

runtime guard (defaulting to false as it is a pretty risky change)
renamed some variable for clarity
2 integration tests:

1 with the new behaviour (retry even when buffer limit exceeded)
1 with the old behaviour (return 5xx)

I also update the PR description to give more context.
This PR has also currently been tested in part of our staging environment. I will like a graph in the PR comment soon

/retest

adding in @KBaichoo for first pass

hm, can you look into test failures? may need a main merge?
/wait

hm, can you look into test failures? may need a main merge?
Rebase on merge, still need to figure out the new test failures
/wait

tests looks better now. Only one test failed not for arm64, will retest it
/retest

/retest

@alyssawilk maintainer on-call ping

To clarify the goal of this PR:
The intent is to ensure we retry the connection-failure until we get at least one connection with an upstream.
While the router is waiting to get a connection with an upstream, we allow buffering up to cluster connection buffer limit plus 1 Mib.
Once a connection is acquired, we will use the defaultretry_per_shadow_buffer_limit_ (min between the downstream connection buffer limit and route buffer limit).
Why do we need plus 1 MiB
We rely on the connection filter to pause reads on the downstream connection. Therefore, the router must be able to buffer more than the upstream buffer limit, as it will not pause read until the buffer is full (reached high watermark).

/retest

To clarify the goal of this PR:
The intent is to ensure we retry the connection-failure until we get at least one connection with an upstream. While the router is waiting to get a connection with an upstream, we allow buffering up to cluster connection buffer limit plus 1 Mib. Once a connection is acquired, we will use the defaultretry_per_shadow_buffer_limit_ (min between the downstream connection buffer limit and route buffer limit).
Why do we need plus 1 MiB We rely on the connection filter to pause reads on the downstream connection. Therefore, the router must be able to buffer more than the upstream buffer limit, as it will not pause read until the buffer is full (reached high watermark).

So in general, the router will buffer request data without sending a 4xx until it starts sending data upstream. After data has been sent upstream, the router will not be able to retry requests unless the request body fits in the shadow/retry buffer.
All increasing the router buffer does is increase the amount of data the router will buffer in this manner before pushing back downstream.  This increases the risk of OOM so without a concrete value add I don't think it's worth tackling.  If you want larger buffers, you can configure larger buffers.  I don't think it's worth the additional complexity to already complicated code to reduce latency by buffering more if and only if we are waiting on upstream connections.
Your e2e test passes just fine with a clean envoy checkout (after removing the reloadable flag set) so I think enovy handles so I think that shows envoy already does retries for large requests on connect failure?
bazel test --config=remote //test/integration:integration_test --test_timeout=120 --test_arg=--gtest_filter=IpVersionsAndHttp1Parser/IntegrationTest.EnsureConnectionRetry/IPv4BalsaParser

/assign @botengyao

Fundamentally, the router used to be the point at which data was shipped upstream. So it would buffer the data in its own buffers until an upstream was selected, then send data upstream.
I think a better fix would be to (optionally?) defer sending data upstream until the connection is established, trusting the HCM/watermark chain to keep the buffer small until we know we can safely proxy. If we had a connect failure we can in that one case retry using the HCM buffer instead of shadow/retry.

This all sounds more than reasonable to me. I would just have two concerns with this approach:

waiting to get a callback from the upstream to inform the router that it has a connection might decrease Envoy performance. From what I understand now, once the upstream gets a connection, it can immediately start sending data. With this approach, it would have to fail for the data to be sent by the router and processed by the filters
when relying on the HCM buffer plus watermark, the router will exceed the buffer watermark, which will pause reading on the downstream listener. Until then, everything is fine.
But once it's started sending the data upstream, right now, the router copies the data, which makes sense when a none connection-failure retry is enabled (not when only connection-failure is enabled; maybe an optimization in the future).
So the buffer will stay above the watermark. We can not really drain it because other filters might be using it. So what would be the path there?

I also have a question regarding the "HCM buffer instead of shadow/retry". From my understanding, they are the same: https://github.com/DataDog/envoy/blob/3763185bf4ce0fa6a3667fcd11921aa3edcdaf45/source/common/router/router.cc#L934.
Please bear with me has this is my first contribution in this repo.

but it adds a lot of complexity, and opens up an OOM vector for everyone who doesn't want a per-connection limit applied on a per-stream basis when upstream connections are slow to establish.

I agree that the workaround that I implemented introduces some memory risk and is not ideal. But I believe the discussion we are having now can bring great benefits.

Aside, @botengyao should the fix here be returning StopIterationAndWatermark rather than sendLocalReply? Seems if we've never sent data upstream, we should just delay until we have one.
https://github.com/envoyproxy/envoy/commit/3c234770d2b1660fa6f3fdb1f9c3916933ec7b84#diff-e3584a650478334f7373c0df344c85d499b5123f5237ce2bbabaad5ee53b899eR910

I don't think so, since retry & shadow is disabled, and upstream_request is created during decoding header, the empty here means no more upstream_requests should happen, so we can local reply.

/wait

Aside, @botengyao should the fix here be returning StopIterationAndWatermark rather than sendLocalReply? Seems if we've never sent data upstream, we should just delay until we have one.
3c23477#diff-e3584a650478334f7373c0df344c85d499b5123f5237ce2bbabaad5ee53b899eR910

I don't think so, since retry & shadow is disabled, and upstream_request is created during decoding header, the empty here means no more upstream_requests should happen, so we can local reply.

From my understanding empty means that we are waiting for a retry to happens (we are in the backoff wait).
I believe this issue is linked to it https://github.com/envoyproxy/envoy/issues/15211.
Ideally we would wait for the retry to happened instead of returning a 507

Sorry was OOO,
I would also rely on the buffer reaching the watermark to pause the read. So:

getting connection from downstream
trying to get a connection with upstream
filter buffer get full (we go over the watermark which pause the downstream read)
we get a connection

At that point to go bellow the watermark, we have the following options:

draining the buffer: no possible as other filter might also use it.
increasing the limit to be right above the currently buffered data. Probably the option for now
do we have other option ?

WDYT @botengyao

/remove-wait

Aside, @botengyao should the fix here be returning StopIterationAndWatermark rather than sendLocalReply? Seems if we've never sent data upstream, we should just delay until we have one.
3c23477#diff-e3584a650478334f7373c0df344c85d499b5123f5237ce2bbabaad5ee53b899eR910
I don't think so, since retry & shadow is disabled, and upstream_request is created during decoding header, the empty here means no more upstream_requests should happen, so we can local reply.
From my understanding empty means that we are waiting for a retry to happens (we are in the backoff wait).
I believe this issue is linked to it https://github.com/envoyproxy/envoy/issues/15211.
Ideally we would wait for the retry to happened instead of returning a 507

Aside, @botengyao should the fix here be returning StopIterationAndWatermark rather than sendLocalReply? Seems if we've never sent data upstream, we should just delay until we have one.
3c23477#diff-e3584a650478334f7373c0df344c85d499b5123f5237ce2bbabaad5ee53b899eR910

I don't think so, since retry & shadow is disabled, and upstream_request is created during decoding header, the empty here means no more upstream_requests should happen, so we can local reply.

From my understanding empty means that we are waiting for a retry to happens (we are in the backoff wait). I believe this issue is linked to it #15211. Ideally we would wait for the retry to happened instead of returning a 507

This is not the same issue I believe and let's go back to this PR ;-)

So, the goal of this PR is going to allow retry if the downstream decoder buffer is full when waiting for the upstream connection (with a hard limit to avoid super large requests as well). This is basically is to add the flow control for the retry buffer, but the key point here is we don't drain the downstream decoder buffer that is used for retry even some retry conditions are no longer held.
This PR discussed 5 approaches from my understanding.

Explicitly call the onAboveWriteBufferHighWaterMark and read enable it when the upstream connection established, as commit #1. However, we should never let the data be more than the downstream decoder buffer limit, since we will never drain it, and the data will continue to disable the read if we call addDecodedData(data, true).

Using the upstream flow control by adjusting/incresing the downstream filter buffer limit to upstream_buffer_limit + 1Mb or whatever.

Discussed above like, leverage the downstream decoder buffer flow control introduced by addDecodeData(data, true) to let the retry buffer exceed the buffer limit once. Since we cannot drain the buffer, to re-read enable the downstream read we need to change the buffer limit to current_data.length(). If it is too large we also need to drop. This delayed the buffer adjustment from approach 2.

As Allyssa mentioned, in high level another approach is do not buffer data before upstream connection is established with the HCM buffer and watermark chain. Alyssa, are you suggesting the network buffer/codec buffer from HCM, or the decoder filter chain buffer? If it is the later then it is similar as approach 3 in high level.

Maybe a third buffer to only buffer the connection failure retry, and drain it when upstream connection is established. The cost should be small with only buffer slice movement, cc @KBaichoo.

Approach 3 is what I think to solve the problem, but still with the buffer limit change. Otherwise 4/5 deserves exploration and further tests as @fl0Lec brought up. Happy to hear other thoughts @alyssawilk @KBaichoo.

/retest

@fl0Lec it seems ci is not running due to conflicts (altho not clear what conflict)
could you merge main please
/wait

/retest

@fl0Lec it seems ci is not running due to conflicts (altho not clear what conflict)
could you merge main please

Done, all tests passed 👍

Ping @botengyao

/wait

/retest

@botengyao friendly ping.

friendly ping @botengyao :)

/wait


gha: Cache cargo registry/index and target directories between builds

Approximately 50% reduction in build time, more on the device flag it happens to cache. Could extend to cache every device flag, but that's unnecessary for now.
bors r+


ExpandableStringEnum impl (analogous to what we did in .NET)

A base class impl for hand-written expandable enums, which:

unifies and simplifies the creation of such enums,
takes care of shared method impls (equals(), toString(), hash, etc)
and enables true enum constant-like identity semantics, i.e. a value instance is globally unique and always the same object. (Our previous approach generated the same "value" as a new class instance each time it was accessed/parsed)

This is achieved by maintaining a static, thread safe, concurrent hash map.
Compute's and ResourceMgrs hand-written expandable enums have so far been converted to this approach in this PR.
Also included is a thread-safety unit stress test verifying that multiple threads reading and writing values on various expandable enums subclassing ExpandableStringEnum in parallel, work reliably.
See PowerState as an example of how to define such an enum.  The fromString() on the concrete enum is a convenience method, so that constants can be defined without referencing .class of the enum class. If this convenience fromString() is not provided, then ExpandableStringEnum's fromString() is the fallback, but it requires the .class of the subclassing enum to be passed.
This is analogous, but not identical, to what we enabled on the .NET side with a base class by the same name. The difference is only due to the difference between how Java and .NET handle generic parameters. In .NET, a specific generic param passed to the generic type is part of that type's definition and a unique set of static fields is initialized for each "sublassing" enum. So each gets its own Dictionary. In Java though, because of type erasure, the generic type is still only one global type, so its statics are initialized only once. That's why a single Map needs to be used for all enum constants (hence, indexing by constant name is not enough -- the name is prefixed by the fully qualified type name).
That's also why the .class needs to be provided to the ExpandableStringEnum at runtime it order for it to know how to instantiate a new instance of the subclassing enum (via .fromString()).
Since this does not change any naming, does not remove anything, but only adds methods and a new type to the inheritance hierarchy of the existing expandable enums, I consider this a back-compat preserving enhancement.

Can one of the admins verify this patch?

Yes, we should move at least the ExpandableStringEnum into the runtime as soon as we feel it's stable enough and before the next release, so that no user code gets to take a dependency on it in its current package.


Add 32 Bit Support

this will not be added as we can NOT make an archiso config for 32 bit devices (SteamOS does not support 32 bit devices)

this may help https://gist.github.com/bmwalters/58120221f1c8a08e264e0bc1d46c67f2


Updates Internal release with 7.139.1 hotfix changes

Adds an experiment override for Control group, and accounts for the DuckPlayer Feature flag to be active when replying to the FE requests.

Task/Issue URL: https://app.asana.com/0/1207252092703676/1208450923559111/f
Description:

Internal release updates with hotfix
Account for FF when sending user values to the Frontend

Steps to test this PR:

Go to Settings > All Debug Settings > Override DuckPlayer Experiment (Control)

Force-close / Restart the app

Go to SERP, and then confirm you can this page:

Go to SERP, search for a video and tap it

Confirm you see the old overlay

@Bunn These are the changes you've reviewed and approved in. https://github.com/duckduckgo/iOS/pull/3421
There is one difference though, and that is that I added a Feature flag check here too (better safe than sorry): https://github.com/duckduckgo/iOS/pull/3426/files#diff-0ea71c12b12d7dc638044e8d81230ceb6a54488ed33a57512a533e9c4bd329e8R191


Update concepts-security.md

Updated auditing doc link to the one for flexible server. It was using auditing doc under single server.

#label:"aq-pr-triaged"

#sign off


New way of handling dependencies

Completely remove sub-directories as I found them weird especially when updating an existing lib.
Now a CMake script in deps/ folder will handle all dependencies and installation too.
This open FastEngine to become a complete multi-platform CMake config package with easier usage.
also bump :
fmt -> 10.2.1
freetype -> 2.13.2
SDL2 -> 2.28.5
SDL2_image -> 2.8.2
glslang -> 14.0.0
cmake version -> 3.23
Compiled dependencies, will now be in Release mode no matter what mode FastEngine is in. (configurable in future)
This PR also add artifacts build inside github actions (macos, linus and windows 32/64), I can now create release tag with complete binary installation that you can download for your platform.
With separated dependencies build, I can now remove the (not sure if working correctly) filter in the codeql action.
This also remove the fge_install tools as it is not relevant anymore.
Vulkan headers is now downloaded with others dependencies.

flows summarize

Hello, I am a PR summary agent on flows.network.
It could take a few minutes for me to analyze this PR. Relax, grab a cup of coffee and check back later. Thanks!


ccond and TopicsProvider

I hava some confusions, delete this issue if I make it wrong.
1 code: https://github.com/surgemq/surgemq/blob/master/service/buffer.go#L121-L123
is:
this.pcond.L.Lock()
this.ccond.Broadcast()
this.pcond.L.Unlock()
or just:
this.ccond.L.Lock()
this.ccond.Broadcast()
this.ccond.L.Unlock()
?
2 https://github.com/surgemq/surgemq/blob/master/topics/memtopics.go#L30-L48
why there are two same lines:
var _ TopicsProvider = (*memTopics)(nil)
Appreciate!

On buffer.go, I think what's there is correct. I need to look back and document the logic.
On memtopics.go, it's a dup line. Thanks for finding it. Though that line is really ignored during compilation. I will fix though.

@cloudzhou Regarding your first question: The variable naming is a little bit demanding to be honest but pcond is the PRODUCER condition. So what the code correctly does is it gains exclusive PRODUCER access and then notifies the CONSUMERS about new data.