[ZEPPELIN-5523] Activate interpreter tests

What is this PR for?
This PR adds influxdb, ksql and scalding to our ci job.
What type of PR is it?

Improvement

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-5523

How should this be tested?

CI

Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

I will merge this on Wednesday if no further comments are received.

LGTM


Update BLS test suite to BLS standard draft v2 format

Make sure that BLS -Verify APIs would only return True or False, no exceptions.
Use eth2spec.utils.bls wrapper instead of py_ecc for test generator (I'm worried about that it's possible py_ecc throws Exceptions)
Add assertions in the test generator
Add some special test cases for the -Verify APIs
Clean up the test format documents

@djrtwo

Do we have test cases that are throwing?

The len(pubkey) == 0 case I added. In py_ecc I suggested we throw errors for debugging-friendly, but that leads to inconsistency, my bad! (/cc @CarlBeek)
In the BLS standard spec:

The -Verify APIs and Aggregate API have this precondition where n is the length of pubkeys or the length of signatures: "n >= 1, otherwise return INVALID"
The Aggregate API should return "signature, an octet string encoding a aggregated signature that combines all inputs; or INVALID".

However, we don't have such INVALID as the output from Aggregate, what we do is throwing Exception.

The -Verify APIs should return "result, either VALID or INVALID".

For the preconditions, what py_ecc do is throwing Exception (similar to Aggregate)
If the preconditions pass, then the verification would return True or False.

I added len(pubkey) == 0 test cases for the -Verify APIs, but it's a bit tricky to determine what the output of Aggregate([]) should be... maybe None? Do you think we have to add this test case?

Aggregate([]) should be... maybe None

is this not defined in the ietf spec?

is this not defined in the ietf spec?

The result is defined as "INVALID", and I can't find the formal definition of INVALID from the spec.
Just looked around other implementations:

milagro rust: it returns custom error
herumi: it seems to be that it returns default signature (0x00...): https://github.com/herumi/bls/blob/78a130231b67490fdde4ef5c454685c04698804b/test/bls_test.hpp#L432-L433 ...?  /cc @herumi 👋

We only use Aggregate(sigs) in validator guide (get_aggregate_signature) in the spec. We can update the description from "Collect attestations seen via gossip ..." to "Collect attestations seen via gossip ... , where len(attestations) > 0".

I think that it is better Aggregate([]) sets zero because

Verify always returns false for zero signature except for the secret key is zero. It is mathematically exact.
Many languages (including Python, Haskell) return zero for sum([]).
If blsAggregateSignature is changed to return true/false from now, It's afraid that many existing applications (including non-Ethereum application) remain not to check the results.

Specifically, the spec has a precondition that n>=1 so I think there is some lenience in implementation here.
Personally, I feel that the type of Aggregate should either be a Maybe Signature or return the point at infinity, but I don't think that that is what the spec defines. As was mentioned, they return INVALID for empty aggregates which we have been mapping to False everywhere else. (eg. spec defines Verify to return {VALID, INVALID} which in py_ecc we define as Verify -> bool.)

Verify always returns false for zero signature except for the secret key is zero. It is mathematically exact.

This assumes that the point at infinity is mapped to 0, which is not necessarily the case. The encoding of a point uses  the second most significant bit in addition to all zeros to encode infinity.

This assumes that the point at infinity is mapped to 0, which is not necessarily the case. The encoding of a point uses the second most significant bit in addition to all zeros to encode infinity.

Yes, "zero signature" means inifinity.  The array of zero bytes is my library's internal representaion, which is converted to c00...0 by serializeToHexStr().

@herumi @CarlBeek thanks for the reply!
I understand the standard v.s. usability tradeoff that you both mentioned.
We have made explicit handling in Eth2 spec to avoid Aggregate([]) as possible. (#1037) Having said that, if somehow, unfortunately, Aggregate([]) happens during consensus validation, it is considered as an invalid case.
So I think it's correct that py_ecc & milagro-rust throw Error/Exception and handle the error in client implementation. As for herumi-bls, for safety, maybe the Eth2 clients can add a wrapper to throw an exception or handle 0xc0...00 result especially after calling Aggregate? (currently, it only happens in the validator guide spec)
/cc @prestonvanloon
As for the test vector, I'm inclined to return None as we use None for the invalid case for the state transition test vector.

Okay, I'll modify Go/Rust/WebAssembly binding of my library to throw an exception.


seeing zombie processes after running many uproxy pairs on Docker

Sometimes when I've run a bunch of Zork pairs via Docker I can't stop one of the containers. Huh? The tell-tale sign is that I have a bunch of zombie (defunct) processes. I don't remember seeing this with Firefox.
Turns out zombie processes are a common problem on Docker and one of the most popular base images specifically addresses it:
http://phusion.github.io/baseimage-docker/

I've played with baseimage-docker a bit, it does seem to fix this issue. In-progress branch:
https://github.com/uProxy/uproxy-docker/tree/trevj-baseimage-docker

Oops, forgot to close this.


Validation job - System.ArgumentException: Messages cannot be larger than 65536 bytes

Error when validating https://www.nuget.org/packages/Aspose.Imaging/3.8.0 (due to large release notes field). @skofman1 @xavierdecoster should we truncate release notes for the validation process (we don't currently use the value so might as well just skip the field being added)?
Stacktrace:
Validation.Runner.exe Error: 0 : [FAILED]: System.ArgumentException: Messages cannot be larger than 65536 bytes.
   at Microsoft.WindowsAzure.Storage.Queue.CloudQueueMessage.GetMessageContentForTransfer(Boolean shouldEncodeMessage, QueueRequestOptions options) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\Common\Queue\CloudQueueMessage.Common.cs:line 285
   at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.AddMessageImpl(CloudQueueMessage message, Nullable`1 timeToLive, Nullable`1 initialVisibilityDelay, QueueRequestOptions options) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Queue\CloudQueue.cs:line 2567
   at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.BeginAddMessage(CloudQueueMessage message, Nullable`1 timeToLive, Nullable`1 initialVisibilityDelay, QueueRequestOptions options, OperationContext operationContext, AsyncCallback callback, Object state) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Queue\CloudQueue.cs:line 1356
   at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.BeginAddMessage(CloudQueueMessage message, AsyncCallback callback, Object state) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Queue\CloudQueue.cs:line 1333
   at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.TaskFromVoidApm[T1](Func`4 beginMethod, Action`1 endMethod, T1 arg1, CancellationToken cancellationToken) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 174
   at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.AddMessageAsync(CloudQueueMessage message, CancellationToken cancellationToken) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Queue\CloudQueue.cs:line 1394
   at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.AddMessageAsync(CloudQueueMessage message) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Queue\CloudQueue.cs:line 1382
   at NuGet.Jobs.Validation.Common.PackageValidationQueue.<EnqueueAsync>d__5.MoveNext() in D:\BuildAgent\work\ab85ff06568d51d2\src\Validation.Common\PackageValidationQueue.cs:line 46
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NuGet.Jobs.Validation.Common.PackageValidationService.<StartValidationProcessAsync>d__5.MoveNext() in D:\BuildAgent\work\ab85ff06568d51d2\src\Validation.Common\PackageValidationService.cs:line 60
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at NuGet.Jobs.Validation.Runner.Job.<RunOrchestrateAsync>d__10.MoveNext() in D:\BuildAgent\work\ab85ff06568d51d2\src\Validation.Runner\Job.cs:line 255
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NuGet.Jobs.Validation.Runner.Job.<Run>d__8.MoveNext() in D:\BuildAgent\work\ab85ff06568d51d2\src\Validation.Runner\Job.cs:line 112
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NuGet.Jobs.JobRunner.<JobLoop>d__6.MoveNext() in D:\BuildAgent\work\ab85ff06568d51d2\src\NuGet.Jobs.Common\JobRunner.cs:line 158
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NuGet.Jobs.JobRunner.<Run>d__2.MoveNext() in D:\BuildAgent\work\ab85ff06568d51d2\src\NuGet.Jobs.Common\JobRunner.cs:line 66

Can you please provide more details about the job? What's the scenario? What do we push to the queue?

THe package metadata is enqueued to our various validators like virus scan. Virus scan doesn't really need release notes to do its thig

We should also remove description, copyright, LicenseUrl, ProjectUrl, Summary, tags  (that are limited to 4000 chars in the DB).
I'm basing this on the documentation here: https://github.com/NuGet/NuGetGallery/blob/master/src/NuGetGallery.Core/Entities/Package.cs#L37

Sounds good! Truncate it to a max length (e.g. 1000 chars) or simply not add the data?

I think either way will keep us safely away from 65536 bytes :sunglasses:

Opened a PR. Decided to keep the data in our audits, but truncate it for processing in the queue.


Don't use native element types directly (label: universal)

Bug, feature request, or proposal:
Bug
What is the expected behavior?
material2 components should be compilable by a server side renderer such as angular-universal
What is the current behavior?
Compile fails because of direct use of native elements such as HTMLElement.
What are the steps to reproduce?
Follow the readme for my test repo at https://github.com/SimplyComplexCo/material-universal-test. Once overlay is functional adding providers for more material2 components will mostly also fail.
What is the use-case or motivation for changing an existing behavior?
Angular Universal support, I understand that #308 is creating a consistent test to keep this functionality in mind.
This issue is to spur the parallel change of making the current functionality work with Angular Universal.
Which versions of Angular, Material, OS, browsers are affected?
Angular Material master

Instead of HTMLElement and ElementRef should be used such as
foo(element: ElementRef) {
  let nativeElement: any = element.nativeElement;
  // Do thing with native element
}

This is going to be addressed as part of #308


ImVoxelNet SUN RGB-D without the D

Hello,
I am looking to implement ImVoxelNet on my own custom dataset. (Fantastic work by the way!)
My task is monocular 3D object detection in which the camera (a standard RGB camera) is mounted at an angle (unlike autonomous cars). Because of the change of angle, objects have pitch and roll to account for in addition to the standard yaw angle. Pitch and roll are not taken into account in dataset formats like KITTI. In this way, I would like to try the indoor dataset SUN-RGBD. However, since I do not have a depth camera, I would like to know if there is a way I can "switch off" the D dependence of ImVoxelNet on SUN-RGBD, and if you think this approach will be feasible for my task. Thank you for your time!

Hi @stevengnow ,
Actually ImVoxelNet is a RGB detector either multi-view or single view. So, it does not work with depth or point clouds. And for this reason you should not remove depth :)

Hello,
Thanks for this reply. Yes since i am doing monocular detection (and single view), so I do not have depth or point cloud data. However, my input data is simply RGB. Do you recommend a way to infer the depth then so I can make it a pseudo-RGBD input for ImVoxelNet?
Please let me know if my question makes any sense. Thank you for your time!

I do not quite understand the question. In case of SUN RGB-D  dataset ImVoxelNet is already a monocular detector. So you do not need depth or pseudo depth, only a single image (and may be camera calibration matrix).

Oh I see. For some reason I was under the impression that the SUNRGBD dataset holds depth information as well that is used in ImVoxelNet. So in my case with my custom dataset, I can just send in the RGB images, calibration stuff, and ground truth labels in SUNRGBD format right?

Yes, sure.


Setup docker for testing on recent Ubuntu distros

Note: use system pybind11 for Bionic as it's available

Resolved by #263


Getting error on - iOS [UserSessionService] no push registration id. Aborting...

After upgrading to version 4.0.0 we are getting below on iOS
"infobip-mobile-messaging-react-native-plugin": "4.0.0",
[UserSessionService] no push registration id. Aborting...
In Infobip Portal, the device is not getting registered now, earlier it was working fine.

Also received this error in iOS. Android is working fine.
[MobileMessaging] 🛠 [DynamicBaseUrlHTTPSessionManager] Response received
url: https://mobile.infobip.com/mobile/3/version?platformType=APNS
status code: 403
headers: [AnyHashable("X-Request-Id"):<PHONE_NUMBER>149245875, AnyHashable("Content-Length"): 4, AnyHashable("Access-Control-Allow-Methods"): GET, POST, PUT, PATCH, DELETE, COPY, HEAD, OPTIONS, LINK, UNLINK, PURGE, AnyHashable("Date"): Mon, 31 Jan 2022 15:18:10 GMT, AnyHashable("Strict-Transport-Security"): max-age=31536000; includeSubDomains; preload;, AnyHashable("Content-Type"): application/json;charset=UTF-8, AnyHashable("Server"): SMS API, AnyHashable("Access-Control-Allow-Origin"): *, AnyHashable("Access-Control-Allow-Headers"): Authorization, Content-Type, X-Enrollment-Token, Cache-Control, X-Requested-With, AnyHashable("Vary"): Origin, Access-Control-Request-Method, Access-Control-Request-Headers, AnyHashable("X-Account-Id"): 119107, 119107]
data: "null"
2022-01-31 20:48:11:133 [MobileMessaging] ‼ [VersionManager] An error occurred while trying to validate library version: Error Domain=MobileMessaging.AFError Code=3 "Response status code was unacceptable: 403."

Also, getting below error when calling saveUser method. Any idea about this? mobileMessaging.saveUser( user, user => { console.log("Infobip SaveUser Success: ", user); //alertBox("saveUser", "Inserted data on Success"); }, user => { console.log("Infobip SaveUser Error: ", user); //alertBox("saveUser", "Inserted data error"); }, );
Error: { "description": "The application instance is not registered on the server yet. The registration will be perfomed eventually. Subscribe to the notification MMRegistrationUpdated to be notified as the registration succeeded.", "domain": "com.mobile-messaging", "code": 1 }

This is essentially the same case that you have reported in the first message.

Also received this error in iOS. Android is working fine.
[MobileMessaging] 🛠 [DynamicBaseUrlHTTPSessionManager] Response received
url: https://mobile.infobip.com/mobile/3/version?platformType=APNS
status code: 403
headers: [AnyHashable("X-Request-Id"):<PHONE_NUMBER>149245875, AnyHashable("Content-Length"): 4, AnyHashable("Access-Control-Allow-Methods"): GET, POST, PUT, PATCH, DELETE, COPY, HEAD, OPTIONS, LINK, UNLINK, PURGE, AnyHashable("Date"): Mon, 31 Jan 2022 15:18:10 GMT, AnyHashable("Strict-Transport-Security"): max-age=31536000; includeSubDomains; preload;, AnyHashable("Content-Type"): application/json;charset=UTF-8, AnyHashable("Server"): SMS API, AnyHashable("Access-Control-Allow-Origin"): *, AnyHashable("Access-Control-Allow-Headers"): Authorization, Content-Type, X-Enrollment-Token, Cache-Control, X-Requested-With, AnyHashable("Vary"): Origin, Access-Control-Request-Method, Access-Control-Request-Headers, AnyHashable("X-Account-Id"): 119107, 119107]
data: "null"
2022-01-31 20:48:11:133 [MobileMessaging] ‼ [VersionManager] An error occurred while trying to validate library version: Error Domain=MobileMessaging.AFError Code=3 "Response status code was unacceptable: 403."

you can ignore this error. The VersionManager that triggers this error would be disabled in your Release build. Version Manager just checks whether or not you are using the most up-to-date version of Mobile Messaging.

Thanks @kas-kad for the Info.
Now the issue is i am not able to register users in such case and the Push Notification triggered via the designated method is not working. How can i resolve this.
Again, this is happening only on iOS.

@agupta-avenir I'm not sure I understand the problem. Could you maybe describe in details what is your use-case.

i am not able to register users

what do you do to register users?

Push Notification triggered via the designated method

how do you trigger push notification?

@kas-kad  After Login, we send some essential parameter(Login Id) in saveUser call and from Infobip portal/Backend we use that Primary Login Id to send push Notification. Now since the saveUser call is failing, the Push Notification are not getting sent.

@agupta-avenir you need to wait until registration finishes successfully and call "saveUser" afterwards. I suggest you to subscribe for event/notification called "registrationUpdated" and perform "saveUser" within the event handling code, more details in doc: https://github.com/infobip/mobile-messaging-react-native-plugin/wiki/Library-events.

@kas-kad  Already have this "registrationUpdated" event. It's not getting called at all.

@kas-kad  One more thing
In Appdelegate.m we have this line commented out - //[MobileMessagingPluginApplicationDelegate install];
and our Push Notifications are working. What could be the impact of this?
Should i enable it?

@agupta-avenir
According to our documentation, this line must not be commented out. What was the reason you chose not to follow our guide?

Thanks @kas-kad , no idea why it was disabled. Now enabled it.

@agupta-avenir please doublecheck if you have all the steps done with respect to this guide: https://github.com/infobip/mobile-messaging-react-native-plugin/blob/master/README.md

Thanks @kas-kad. All other step are good.
Also, we are facing issue in updating the Infobip version to 4.1.6 as on Android we are able to run the App. Can you help on that.
`ERROR Invariant Violation: "appName" has not been registered. This can happen if:
Metro (the local dev server) is run from the wrong folder. Check if Metro is running, stop it and restart it in the current project.
A module failed to load due to an error and AppRegistry.registerComponent wasn't called.`

Hi @agupta-avenir
About Android issue looks like smth related to local environment.
Coukld you please check what RN and node version do you use?
Requirements for 4.1.6:
node (v10.17.0 or higher)
React Native (v0.66.3)

These are the version's currently set, so we need to update to RN version 0.66.3 for v4.1.6 but we tried to update to 4.1.0, 41.2, 4.1.3 got same issue so it's some related to something else. Can you give us minimum requirement for 4.1.3 as that version is having an event we are interested in.
Node: v17.0.1
React Native: 0.65.1

Hi @agupta-avenir,
We didn't manage to respond to you in time. Did you manage to update your app with latest RN versions? Is this issue still ongoing?
BR,
Tereza

closing the issue because we didn't receive any response for a long time


gekko instances (load/workers/loadCandles/child) never killed when using Chrome, consuming memory

I'm submitting a ...
[x ] bug report
[ ] question about the decisions made in the repository
Action taken (what you did)

In Chrome start live gekko (market watcher)
close Chrome and open again.
click on live market watcher, now loadCandles/child get instantiated, with ps -aux | grep gekko you can see it.
close Chrome and open again and click market watcher. Again new instance of is created.
now leave Chrome. This intances willl run forever on a server and will cause OutOfMemory.

With Firefox I can not reproduce it, the instances of loadCandles are not created? with ps -aux | grep gekko I can not see any of them.
And also not tried with IE.
Expected result (what you hoped would happen)
When closing chrome all  loadCandles/child processes should be killed.
Actual result (unexpected outcome)
Intances of oad/workers/loadCandles/child will run forever on a server and will cause OutOfMemory.
Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, etc)
example with one UI, one market watcher and 3 loadCandle children.
ps -aux | grep gekko
node gekko.js --config sample-config.js --ui
evgeny   29230  0.1  0.3<PHONE_NUMBER>8 pts/4    Sl+  14:32   0:00 /home/evgeny/.nvm/versions/node/v10.9.0/bin/node /home/evgeny/workspace/angular/gekko-master/core/workers/pipeline/child
evgeny   29252 60.8  0.5<PHONE_NUMBER>4 pts/4    Rl+  14:32   5:42 /home/evgeny/.nvm/versions/node/v10.9.0/bin/node /home/evgeny/workspace/angular/gekko-master/core/workers/loadCandles/child
evgeny   29374 59.9  0.5<PHONE_NUMBER>2 pts/4    Rl+  14:33   5:00 /home/evgeny/.nvm/versions/node/v10.9.0/bin/node /home/evgeny/workspace/angular/gekko-master/core/workers/loadCandles/child
evgeny   29467 59.3  0.5<PHONE_NUMBER>0 pts/4    Sl+  14:34   4:18 /home/evgeny/.nvm/versions/node/v10.9.0/bin/node /home/evgeny/workspace/angular/gekko-master/core/workers/loadCandles/child

Nice find. And chrome should not have much to do with it. Will test

Thanks for creating the PR :)


NIFI-13027 - Warn users for small files processing in PutIceberg

Summary
NIFI-13027 - Warn users for small files processing in PutIceberg
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[ ] Apache NiFi Jira issue created

Pull Request Tracking

[ ] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[ ] Pull Request based on current revision of the main branch
[ ] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[ ] Build completed using mvn clean install -P contrib-check

[ ] JDK 21

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

@exceptionfactory The problem with doing that in the flow with logmessage is that you're putting the onus on the user to do that.  And the point is they might not know.
@pvillard31 I think this makes sense but to my response to exceptionfactory the whole idea here is to convey information that might not be obvious to the end user.  This seems like it could be more thoughtfully done by skipping new properties altogether.  Is there a somewhat established "datasets on average going to Iceberg should be X bytes" big?  If there is something like that then just have the processor keep a bit of state about this and after a bit of time and set of files go by periodically kick out such guidance.

From discussions with Iceberg experts, a good size is 10MB. Problem is that we've seen users using PutIceberg without any kind of Merge operation before (imagine ConsumeKafka (not record based) to PutIceberg) creating thousands of small (hundreds of bytes) files and associated snapshots. This has basically killed the performances and made the query engine on top of the Iceberg tables pretty much unusable. While I do agree that the approach with a specific property sets a precedent that is not great, I don't really see another good approach to make sure the flow designer is aware of this as soon as the designer tries to run the processor to make some tests. I definitely don't think we should go in the direction of adding the Merge capabilities into the processor itself (that is going to be a mess and extremely complicated). The rules engine would be great but you can't be sure that the rule would be set and enforced by users before someone starts designing a flow where Iceberg is a destination. The reason for this change in this case is really because creating so many snapshot files has basically killed the downstream system.
Another approach that I can think of: if the processor is processing a flow file that is below the agreed size, then we would yield the processor, and the default yield duration would be set to something like 1 minute. But we would still want to emit a bulletin to explain the behavior or something...

@pvillard31 I was meaning to say it is good to warn the user about usage not making sense with that processor.  But it does not need to be configurable what the size is - we already have too many dials/knobs.  Instead the processor should keep some simple state about the rate/avg size of files seen and if over say a 1-5 minute interval the data is on average 'too small' then log a warn/bulletin.  The bulletin could certainly advise the user that data to this processor should be batched together and that a component such as MergeContent or MergeRecord can be helpful.

My concern with changing the routing approach is that it'd be a significant breaking change for existing flows.
Having a logic that is not managed through properties also means that if a user does want small files (no idea why...) that would not be an option. But yeah I guess having a built-in rate limiter is probably the best option if we don't want to add properties.

@pvillard31 Can you please rebase and advise whether you are good removing the configurability of the size?
I think the warning is fair game but the configurable size just seems unnecessary.  I'm fine leaving it so the only thing really needing done is a rebase - but I'm asking if you'd consider dropping that.
Thanks

Just following up on this, glad to take another look after the rebase, but I still disagree with adding a property to enable warning, since the warning is not really actionable.

Hey @joewitt & @exceptionfactory - will close for now. I won't have the opportunity to get back to this before quite some time and I believe @mark-bathori may be able to come up with a PR that follows your recommendations.


python312Packages.django-soft-delete: 1.0.14 -> 1.0.16

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python312Packages.django-soft-delete is: Soft delete models, managers, queryset for Django
meta.homepage for python312Packages.django-soft-delete is: https://github.com/san4ezy/django_softdelete
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.0.15 with grep in /nix/store/wgjm0w8b37nv13pwd42n4s0d7a1a5415-python3.12-django-soft-delete-1.0.15
found 1.0.15 in filename of file in /nix/store/wgjm0w8b37nv13pwd42n4s0d7a1a5415-python3.12-django-soft-delete-1.0.15

Rebuild report (if merged into master) (click to expand)

4 total rebuild path(s)

4 package rebuild(s)

First fifty rebuilds by attrpath

paperless-ngx
python311Packages.django-soft-delete
python312Packages.django-soft-delete

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/wgjm0w8b37nv13pwd42n4s0d7a1a5415-python3.12-django-soft-delete-1.0.15 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python312Packages.django-soft-delete https://github.com/r-ryantm/nixpkgs/archive/dce46fdd7833148aa3549b4e2376d9279b17ec63.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/dce46fdd7833148aa3549b4e2376d9279b17ec63#python312Packages.django-soft-delete

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/wgjm0w8b37nv13pwd42n4s0d7a1a5415-python3.12-django-soft-delete-1.0.15
ls -la /nix/store/wgjm0w8b37nv13pwd42n4s0d7a1a5415-python3.12-django-soft-delete-1.0.15/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review

x86_64-linux

:white_check_mark: 5 packages built:

paperless-ngx
python311Packages.django-soft-delete
python311Packages.django-soft-delete.dist
python312Packages.django-soft-delete
python312Packages.django-soft-delete.dist

Maintainer pings

[!TIP]
As a maintainer, if your package is located under pkgs/by-name/*, you can comment @NixOS/nixpkgs-merge-bot merge to automatically merge this update using the nixpkgs-merge-bot.

Add a :+1: reaction to pull requests you find important.

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python312Packages.django-soft-delete is: Soft delete models, managers, queryset for Django
meta.homepage for python312Packages.django-soft-delete is: https://github.com/san4ezy/django_softdelete
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.0.16 with grep in /nix/store/2b44n863j2qhscrk7ic2rv65g43dvj56-python3.12-django-soft-delete-1.0.16
found 1.0.16 in filename of file in /nix/store/2b44n863j2qhscrk7ic2rv65g43dvj56-python3.12-django-soft-delete-1.0.16

Rebuild report (if merged into master) (click to expand)

4 total rebuild path(s)

4 package rebuild(s)

First fifty rebuilds by attrpath

paperless-ngx
python311Packages.django-soft-delete
python312Packages.django-soft-delete

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/2b44n863j2qhscrk7ic2rv65g43dvj56-python3.12-django-soft-delete-1.0.16 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python312Packages.django-soft-delete https://github.com/r-ryantm/nixpkgs/archive/23ddc7b6e5e19e785b5936d98e7a966d0707868e.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/23ddc7b6e5e19e785b5936d98e7a966d0707868e#python312Packages.django-soft-delete

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/2b44n863j2qhscrk7ic2rv65g43dvj56-python3.12-django-soft-delete-1.0.16
ls -la /nix/store/2b44n863j2qhscrk7ic2rv65g43dvj56-python3.12-django-soft-delete-1.0.16/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review

x86_64-linux

:white_check_mark: 5 packages built:

paperless-ngx
python311Packages.django-soft-delete
python311Packages.django-soft-delete.dist
python312Packages.django-soft-delete
python312Packages.django-soft-delete.dist

Maintainer pings

[!TIP]
As a maintainer, if your package is located under pkgs/by-name/*, you can comment @NixOS/nixpkgs-merge-bot merge to automatically merge this update using the nixpkgs-merge-bot.

Add a :+1: reaction to pull requests you find important.


error on repo.first.subject in example

…/vendor/cache/ruby/2.3.0/gems/rdf-2.2.11/lib/rdf/model/dataset.rb:67:in each_statement': undefined method each' for nil:NilClass (NoMethodError)     from …/vendor/cache/ruby/2.3.0/gems/rdf-2.2.11/lib/rdf/mixin/queryable.rb:172:in first'
from test-store.rb:24:in <main>'
fwiw repo.first_subject works

This particular error is in RDF.rb, not rdf-do. The error likely results from an issue in rdf-do originally, but it's shown up in other gems, and RDF.rb should provide a more meaningful error.
This gem has languished and does not pass specs, so it needs more attention.
From the original email:

Though imho this info could feature more prominently in the initial README at https://github.com/ruby-rdf/rdf in the tutorial section in between "Using RDF.rb and Spira to process RDF data from the British Ordnance Survey" and "Getting started with RDF and SPARQL using 4store and RDF.rb"?

Specifically the Using code from https://github.com/ruby-rdf/rdf-do:

require 'rdf'
require 'rdf/ntriples'
require 'data_objects'
require 'do_sqlite3'
require 'rdf/do'

repo = RDF::DataObjects::Repository.new uri: 'sqlite3:test.db'
# repo = RDF::DataObjects::Repository.new uri: 'postgres://postgres@server/database'
# heroku_repo = RDF::DataObjects::Repository.new uri: ENV['DATABASE_URL']
repo.load('http://datagraph.org/jhacker/foaf.nt')

# How many statements did we have?
repo.count
#=> 10

# Get the URI of the first subject
jhacker = repo.first.subject
#=> #<RDF::URI(http://datagraph.org/jhacker/foaf)>

# Delete everything to do with it
jhacker_statements = repo.query(subject: jhacker)
repo.delete *jhacker_statements
repo.count
#=> 7

# with Postgres, we could have done this, but SQLite gives us a locking error:
# repo.delete(*repo.query(subject: jhacker))

# Changed our mind--bring it back
repo.insert *jhacker_statements
repo.count
#=> 10

Having said that I'm getting an error on repo.first.subject

  …/vendor/cache/ruby/2.3.0/gems/rdf-2.2.11/lib/rdf/model/dataset.rb:67:in `each_statement': undefined method `each' for nil:NilClass (NoMethodError)
      from …/vendor/cache/ruby/2.3.0/gems/rdf-2.2.11/lib/rdf/mixin/queryable.rb:172:in `first'
      from test-store.rb:24:in `<main>'

Is there anything I can do to help bring rdf-do up to spec Gregg?

As you might have seen, specs are not passing with essentially the same error you reported. Also, we've never configured Postgress on Travis.
The Statement#to_hash deprecation messages are indicative of calls to an interface that automatically invokes that deprecated interface, due to a **options argument and that the statement is the last argument. Probably best for me or @no-reply to track this down, as it's part of a general update.
As you can tell, this gem has languished, due to general lack of use. Generally, for a simple and more performant option, I use rdf-mongo. I also think that a file-backed (or other persistent store) version of the base RDF::Repository (or similar) would provide great performance and satisfy what you're interested.
That said, we do need to get rdf-do working, so whatever you would like to contribute, in terms of a PR, would be helpful.
In the mean time, I'll track down the DEPRECATION issue, which may solve a bunch of problems.

It seems that the problem is because RDF::DataObjects::Repository implements #each, and RDF::Dataset implements #each_statement. The header of RDF::Enumerable says that implementation should implement #each, but when we introduced RDF::Dataset, it implemented #each_statement, instead of #each.
See ruby-rdf/rdf#365.

BTW, fixing this allows all but one spec to pass in Sqlite3, that being repository count:
  1) RDF::DataObjects::Repository The SQLite adapter behaves like an RDF::Repository as dataset behaves like an RDF::Dataset when enumerating statements behaves like an RDF::Enumerable when enumerating terms #each_term has correct number of terms
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected: 98
            got: 101

I've seen these before in other repos (e.g., rdf-aggregate-repo), and should look there to see how the issue was resolved.

Do you want me to have a go at fixing it, or are you on top of it? I know you that

Probably best for me or @no-reply to track this down, as it's part of a general update.

but if you want me to take a look at it I will.
I'd need to know how to run the specs for rdf-do locally, am unsure how to do that. I'll tell you why I like using the Sqlite back-end rather than a simple file or Mongo. The reason is that when I'm using Rails in dev mode Sqlite is already automatically being used. That means I point th triplestore repo at the dev Sqlite db and have the triples table quad in the same db as the resource table that I use for tracking local resources. Makes sense, no?
Anyway, thanks for following so quickly – as has been my experience with Ruby-RDF in the past :+1:

I'm giving @no-reply a bit more time to respond to ruby-rdf/rdf#365 (PR ruby-rdf/rdf#366) before I merge it, as he did the RDF::Dataset work. You could either work against that branch, or wait for the merge and then look at the failing spec. It would also be great to get Postgres support working, particularly on Travis, for which there are probably some existing recipes.
Note that DataObjects hasn't been updated in a while, and you'll get numerous warnings when running on a Ruby 2.4 code base, but it functions okay. Unfortunately, the release hasn't had much attention. It may be that it's time to deprecate rdf-do, and simply use the sqlite3 gem directly; same could be done for other database backends, but I don't see evidence for Postgres use with RDF.rb.

I went ahead and merged #366.
I'm open minded about how to handle bringing this gem up to date and willing to throw some cycles at it if we can get a sense of what would be most valuable to the community.

The update to RDF.rb has been pushed, so this gem should work reasonably, other than deprecation messages issued by data_objects (which is way out of date) and the above mentioned error. @igravious if you'd like to track this down, and potentially see about Postgres going on Travis, that would be great.


Enforcing SD-only playback in Chrome desktop?

Have you read the FAQ and checked for duplicate open issues?
Yes
What version of Shaka Player are you using?
2.5.10 (Shaka Demo Player)
Please ask your question
I'm unclear about how/if Widevine restricts HD playback in software CDM environments.
 My (perhaps incorrect) expectation is that Shaka will only play the SD streams b/c Chrome Mac is a software CDM environment. Instead, it plays the 1080 stream.
My manifest is at https://jluther.s3.amazonaws.com/enc/labels.mpd, license server is https://cwip-shaka-proxy.appspot.com/no_auth
Am I doing something wrong?

If the license server sent 1080p keys, Shaka will be fine with playing 1080p content.  Policies like "no HD for software CDM devices" can only be enforced by the license server by deciding which keys to return in the license.
In this case, it would seem that the license server returned HD keys.

Got it, thanks. I was assuming that the server would only return the SD and AUDIO keys based on the DRM labels I used in the packager.


missing condition "core.fieldvalueisnot" fix

For some reason the "core.fieldvalueisnot" was left blank

@schonert Oops :no_mouth: shame on me. Copy/paste error. Looks like the same issue is on the sync JS. I'll merge this and add the same fix to the sync JS.
Thanks! :+1:


The code snippet is misleading and inaccurate

The code snippet is misleading and inaccurate. There is no need to specify a Database in a Use statement to query sys.databases. Furthermore, for those unfamiliar with TSQL, the reference to a Database called "AdventureWorks2012" is likely to confuse; especially if their SQL Server instance does not contain a Database by that name.
Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 0ef0bcf2-f4b6-0bc0-a568-36f425aa4c24
Version Independent ID: faec9433-5f72-8ddc-28bf-d0821701a634
Content: View list of databases on SQL Server - SQL Server
Content Source: docs/relational-databases/databases/view-a-list-of-databases-on-an-instance-of-sql-server.md
Product: sql
Technology: supportability
GitHub Login: @WilliamDAssafMSFT
Microsoft Alias: wiassaf

@nvingoe -- Neil, thank you for your clear feedback.
@WilliamDAssafMSFT -- William, please look into this issue.

Agreed, @nvingoe! Thanks for taking the time to provide this feedback, this improves documentation for everyone. A doc update is in flight to remove the confusingly-placed USE statement.


Flow trigger service enhancement

The PR consists of following fix/enhancement around flow trigger service:

Cancel non-recoverable trigger instances after web server restarts:
the following situation would cause trigger instances unrecoverable:
1.1. project A with flow A associated with flow trigger A is uploaded
1.2. flow trigger A starts to run
1.3. project A with flow B without any flow trigger is uploaded
1.4. web server restarts
In this case, flow trigger instance of flow trigger A will be equipped with latest project, thus failing to find the flow trigger since new project doesn't contain flow trigger at all.

fix edge cases in getIncompleteTriggerInstances

shutdown executor service of TriggerInstanceProcessor when shutting down FlowTriggerService.

@kunkun-tang thanks!


Version in docs incorrect

The docs on github pages refer to an incorrect version of GridKit (0.1.0.beta).
It should refer to the most recent release the docs are reflecting.
This is a problem related to the version in version.py.
This version is updated when the publish to pypi workflow is ran in the CICD. The line that replaces the version number to that of the tag is:
run: sed -i -r "s|\"[[:alnum:]]+\.[[:alnum:]]+\.[[:alnum:]]+.+|\"${GITHUB_REF/refs\/tags\//}\"|" ./gridkit/version.py; cat ./gridkit/version.py
We need to run a similar line in documentation.yaml such that the version number in the docs accurately represents the current version.

It does look like the sed command is successfully executed after the previous doc build, but it is not correct in the build docs somehow. From https://github.com/tmillenaar/GridKit/actions/runs/6067143264/job/16458462991 :

The version has been fixed in the recent release where the CICD deployment process was overhauled

The problem was related to the command pip install ./[doc]. I needed to call pip install -e ./[doc].
In the former case the package was installed on the system and that was used. I then updated the local file but not the recently installed files. With the -e flag I now make sure not to install the package on the system and use the files in the local folder instead (they will be linked) so now the modified files are used when building the docs.


Update project.html

Added my dev profile .
fixes #260

Added my dev profile . fixes #260

You are not following account.

I am following now.


sourceUrl - disable tests for command files used as comparisons

I added some command files in the Comment_sourceUrl/data folder to use in tests. These need to be disabled because they are static data.  Otherwise, they can generate errors when running automated tests.

I made this change.  Close this issue.


Remove unsupported versions from docs

We recently remved support for Tenzir versions older than Tenzir v4.9 from app.tenzir.com. This also removes them from the documentation side to keep the version dropdown lean.

This PR is good for your GitHub stats. ⭐

Just a matter of time until I overtake you here 😅


Unable to send mkv file (regardless of size)

Describe the bug
Observed result: as long as the file with the suffix of mkv cannot be sent.
Other files are fine
flutter: Sending aaa（复件）.mkv
flutter: *** Request ***
flutter: uri: https://<IP_ADDRESS>:53317/api/localsend/v2/upload?sessionId=66e22282-f2da-4e5e-9c54-30aa1a62d067&fileId=e91f17ed-4af6-4fd7-9e56-caa4ab35c634&token=7c86403f-7d4c-46b1-affb-76b30b3fcd21
flutter: data:
flutter: Instance of '_FileStream'
flutter:
[ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: PlatformException(channel-error, Unable to establish connection on channel., null, null)
#0      WakelockApi.toggle (package:wakelock_platform_interface/messages.dart:89)
<asynchronous suspension>
#1      MethodChannelWakelock.toggle (package:wakelock_platform_interface/method_channel_wakelock.dart:20)
<asynchronous suspension>

To reproduce
Desktop => Smartphone
send file.mkv
Expected behavior
I don't get it, is there anything special about the mkv file?

Other files are sent fine, and mkv should be fine too.
Screenshots

Desktop (please complete the following information)

OS: flatpak
Version: 1.10.0

Smartphone (please complete the following information)

Device: iphone14ProMax
OS: 16.2
Version: 1.10.0

Additional context
No response

Have you tried to rename the file name?

Yes, pay attention to the pictures in the screenshot column.

Same


awscli from pip instead of Ubuntu packages?

Howdy, I was wondering if you would be ok with a PR that switched awscli from being installed by the Ubuntu package to being installed by pip.
The Ubuntu package is quite a bit out of date (1.2.9 vs 1.8.13) and doesn't support things like Lambda that I happen to need.
What do you think? Can we switch to pip?
Thanks.

Sounds good to me!  Should be an easy change.


Line Break

My code in HTML page
My Link

See https://github.com/carhartl/jquery-cookie/issues/379#issuecomment-159687492


[Onboarding]: Complete GitHub Skill - First Week on GitHub

Select the onboarding task
First Week on Github
Comments
No response

issues wite 3&4


Add ST for global network sets.

Description

Add a basic end-to-end ST test for network sets.  There are thorough FV tests that test a lot more packet flows in Felix's FV suite but I thought it was important to have at least one test that tested everything together.
Todos

[x] Tests
[x] Documentation
[x] Release note

Release Note

None required

htmlproofer failure looks unrelated.


Create 'delete-project' command

Command: ./proton_cli delete-project <admin-key> <proj-name>
Reason: We want to be able to delete old projects

Probably want to do #57 first


Pull in new volume config + mount script(s)

@diranged

Scripts are now in the repo. Just have to point calling scripts to them... @siminm


[feature] Portus Garbage Collection (tag removal) should support an exclusion filter

Description
While setting up and using the garbage collection, I notice that while you can set a inclusion tag filter, there is no way to easily exclude a set of tags/images.
I think we need a way to exclude images or tags from the GC process.

Description
While setting up and using the garbage collection, I notice that while you can set a inclusion tag filter, there is no way to easily exclude a set of tags/images.
I think we need a way to exclude images or tags from the GC process.

Hi @ArcticSnowman Idid not say anything yet, because I had nothing to say.
Today, I think you might be interested in reading this :
https://github.com/SUSE/Portus/issues/2281#issuecomment-592591559
I support your request,  agree with it, and want to add :

that this exclusion filter, should enable the user to define the set of exclusion using :

a single regular expressions : directly specified as value of a config option, like PORTUS_DELETE_EXCLUSION_FILTER="phab-*/*hashicorp*:1.0.*"

a list of regular expressions : one regular expression per line in a file which path is specified with PORTUS_DELETE_EXCLUSION_FILTER_FILE=/path/to/file/gc-exclusion.filter
explicit arbitrary list of image IDs (GUN like for notary, so service netname / netID / service ID + namespace + image name + tag) : one image ID (GUN) per line in a file which path is specified with PORTUS_DELETE_EXCLUSION_FILTER_FILE=/path/to/file/gc-exclusion.filter
So all in all PORTUS_DELETE_EXCLUSION_FILTER_FILE may have , per line either a full image ID (GUN), considered as a special case of regular expression,  or any regular expression that geenrates any sub dictionnary of the "GUN language", like "phab-*/*hashicorp*:1.0.*". Though I'm a pacifist.

@ArcticSnowman If I may, and If it is okay that I ask, related to https://github.com/SUSE/Portus/issues/807#issuecomment-511461401 :

I have my reasons, for not focusing only on harbor , starting with I think it is very important that harbor is not left alone on the market, to rule us all
But can I ask your reasons for being willing to use Portus, instead of harbor,  though it is obvious you know about harbor ?

And btw, the content trust / The Update Framework issue is thrilling.

@Jean-Baptiste-Lasselle -  When we first built out our internal docker registry, 3+ years ago, Portus was the best out there at the time.
We did look at habor about a year ago, mainly to look at the inclusion of Clair.   We did not feel it was any better that Portus at the time and it pulled in lots of new tech that, in an enterprise environment, we did not want to support.  And then we found support had been added to Portus.
I made reference to Harbor in #807, mainly to show that other registry tools have the support.

@Jean-Baptiste-Lasselle - When we first built out our internal docker registry, 3+ years ago, Portus was the best out there at the time.
We did look at habor about a year ago, mainly to look at the inclusion of Clair. We did not feel it was any better that Portus at the time and it pulled in lots of new tech that, in an enterprise environment, we did not want to support. And then we found support had been added to Portus.
I made reference to Harbor in #807, mainly to show that other registry tools have the support.

HI @ArcticSnowman thank you so much for your answer, and sorry I did not answer sooner : I think what we are discussing here, is probably the most exciting challenge with Portus, and probably most important in the container world, though it looks like we are lost in the comments of a forgotten project. I will answer you on #807 regarding notary.  About Clair scanner, I could set it up,and get actual vulnerabilities identified, displayed in Portus, with CVE-MITRE actual references. See https://github.com/SUSE/Portus/issues/2241#issuecomment-593370789
I am also working on releasing a complete fully tested automation with docker compose, to finally provide people a secured, coherent working first setup of  a private docker registry / Portus.
Will happen in the next weeks, It's just that I got involved into a project aside, So I work a little bit everyday, but it takes time.
tty on #807

Hi again @ArcticSnowman , After a good night of sleep, I today have to add something about this issue ( #807 is another subject) :

you need a way  to exclude tags from Garbage collection

I join to you for this feature request : because it is logical, it makes sense,

and moreover, what you describe is a feature that extends an existing one, which you probably will be interested in :  the keep_latest feature.

you will find a complete description and study of the keep_latest feature in this issue : https://github.com/SUSE/Portus/issues/2241

and in that issue, I want to point especially this comment and answer to it by @robgiovanardi :

SYNC feature, is closely related to garbage collection, and both are carried out by background.
Say your registry stores its images into a minio database for example (or any S3 storage) : to delete any set of tags, without using keep_latest, all you have to do, is to delete those tags from the Database (S3 storage), and then restart portus and its background, with SYNC. because what SYNC will do, is progressively update what is displayed in portus WebUI,  until it is in sync with the riegsitry : so until all deleted images no longer appear in Portus WebUI. You will read here that @diranged did that to delete more than 80 terabytes of docker images. The only problem he had, is to configure properly the SYNC feature.
and for all other use cases, keep_latest feature is enough.

The most interesting thing that happened, is that thanks to @diranged noticing that, we proved that no opensuse docker image contains a version of portus including that keep_latest feature. Except one image , the opensus/portus:2.5 on dockerhub :

the problem is that this image is really a mess, so I had to completely rebuild it, to finally be able to use the keep_latest feature.
There are even funnier details about that little story, and a complete analysis proving how and why that happened in the opensuse team, here : https://github.com/SUSE/Portus/issues/2241#issuecomment-586972528

But what you probably want is to try keep_latest, so here you go :

with a debian based complete rebuild from Portus Source code Docker image I recently released : https://github.com/pokusio/portus-build-from-source/releases/0.0.1
And if you care about having opensuse instead of Debian, underneath your Portus, I kind of repaired the official opensuse/portus:2.5 image : https://github.com/pokusio/opensuzie-oci-library/releases/tag/0.0.2

@ArcticSnowman I just want to share the big friendly laugh I had at home, when I discovered that you work tax collection, it is very comforting to know they they rely on us, and maybe they 'lll have to remember that someday :rofl:

Hi @Jean-Baptiste-Lasselle  -  We have tried the keep_latest, however we found a problem with it.  The way our pipeline creates image we generate a fair number of tags for every checking in change.  Some of those changes might never get all the way to production because of testing failures.   This can easily  push the current production release tag off the end of the keep_latest.
I feel that having an exclude_tags regexp would compliment that keep_latest. Allow us to preserve specifically names tags while allowing the GC to remove all the 'extras' that we generated during development and testing.

Hi @Jean-Baptiste-Lasselle - We have tried the keep_latest, however we found a problem with it. The way our pipeline creates image we generate a fair number of tags for every checked in change. Some of those changes might never get all the way to production because of testing failures. This can easily push the current production release tag off the end of the keep_latest.
I feel that having an exclude_tags regexp would compliment that keep_latest. Allow us to preserve specifically names tags while allowing the GC to remove all the 'extras' that we generated during development and testing.

Hi @ArcticSnowman , how very interesting case indeed. Just delivering you my thoughts as I write, I actually never considered the case, and how about you use several docker registry, one for integration tests, one for staging, one for production etc... ?  Apart from pure curiosity, I am asking because there might be very interesting reason, like business or regulations constraints which prevent you from using that pattern.


Encryption and decryption

What would you like to have changed?

I would like to request support for encrypting and decrypting common archive formats such as ZIP, RAR, and others in this project. This feature would include the ability to compress files with password protection and to extract encrypted archives.
Why is this feature a useful, necessary, and/or important addition to this project?

Many users handle sensitive data that requires secure compression and storage. Adding support for encryption would make this project more versatile and useful in real-world scenarios where data security is a priority. Additionally, decrypting encrypted archives is a common use case that many users encounter. Currently, the absence of this feature limits the usability of the project for these scenarios.
What alternatives are there, or what are you doing in the meantime to work around the lack of this feature?

In the absence of built-in support, I’ve had to rely on external tools such as zip and unrar command-line utilities or other libraries. However, integrating this functionality directly into this project would streamline workflows and provide a more cohesive experience for users.
Please link to any relevant issues, pull requests, or other discussions.

I couldn't find any existing issues or discussions directly related to this feature request, but I believe it would align well with the goals of this project. If there are plans or ongoing work in this area, I would love to know more about them and always ready to help build this project.

Thanks for the feature request.
7Zip and RAR formats already support decryption. (Specify a Password in their structs.) They don't support encryption because they don't support creating archives. 7Zip because there's no pure Go implementation yet (that I know of) and RAR because it's a proprietary format.
I don't know of any pure Go implementation for encrypted ZIP files.
Unfortunately as I don't have time to implement this myself, we will have to wait for a contribution / someone to do it.


Why do u use Rsocket

Hi,i am reading your code, why do you use Rsocket instead of other protocols like gRPC or socket, websocket... What is your use case which you use Rsocket, thank you so much

instead of other protocols like gRPC or socket

httpx supports RSocket, gRPC, WebSocket all, and they are different protocols.


#12 Updates Main ReadME with necessary sections

It creates an outline for further updates of the readme like additions of sections like contributors, dependencies etc to guide through the process of running the project.

I think we can work on this same branch throughout the project for updating readMe. Is it okay if we don't delete this branch even after it has been merged to the main until the end of the project?

Yes, we do not delete this branch after the merge, we can continue to use it to update README.md

Good job!


How to set options for track_clicks, track_opens, transactional

Hello!
I have been trying for hours now to set options for track_clicks, track_opens and transactional but no matter what I try it doesn't have any effect.
Searching around it seems like at least there used to be a way to do this but I can't get it to work.
Is there a way to do this?
Thanks!

Hey @lefffen1! Do you know what version you are using? If you are using 2.x you should have something like this.
$sparky->transmissions->post([
    'options' => [
        'open_tracking': true,
        'click_tracking': true,
    ],
    'content' => [
        'from' => [
            'name' => 'SparkPost Team',
            'email' =><EMAIL_ADDRESS>        ],
        'subject' => 'First Mailing From PHP',
        'html' => '
Congratulations, {{name}}!

You just sent your very first mailing!
',
        'text' => 'Congratulations, {{name}}!! You just sent your very first mailing!',
    ],
    'substitution_data' => ['name' => 'YOUR_FIRST_NAME'],
    'recipients' => [
        [
            'address' => [
                'name' => 'YOUR_NAME',
                'email' => 'YOUR_EMAIL',
            ],
        ],
    ]
]);
?>

Hi aymg, thanks for getting back to me!
I am using version 2.1.0
I thought I had tried that but apparently I hadn't. The problem was that I used the wrong names for the options, 'track_clicks' instead of 'click_tracking' and 'track_opens' for 'open_tracking'.. Don't know where I got those from.
Thank you so much for solving this for me!

No problem 😄


Feature request: Additional Console Outputs for more efficient logging and debugging

Heya, i have the common issue that for example when using LLAVA 34b on a small-ish GPU with CPU offloading it sometimes gets stuck.
I can't really trace the issue anywhere, is it the BLAST Batch Processing, is it a OOM error, what is it?
key clip.vision.image_grid_pinpoints not found in file
key clip.vision.mm_patch_merge_type not found in file
key clip.vision.image_crop_resolution not found in file

Is the most i can get out of a log but that's about it... no metrics, no nothing :/

It's possible this might be an OOM problem...
If you're still hitting this, please make sure to upgrade to the latest version, and share a bit more details about your setup.  What is your GPU size?  Server logs may help diagnose it as well.


Two ScyllaDb updates

Motivation
ScyllaDb remains the system we use for the Testnet though this may change. Here we update the configuration.
Proposal
Two things are done:

The replication_factor is changed from 1 to 3 as per the recommendation of the manual.
The version is bumped from 0.12.0 to 0.13.0

Test Plan
The CI.
Release Plan
Not relevant.
Links

reviewer checklist

Yes, that PR is closed. It does not do what I intended to.


Fix twemoji failure

Checklist

[x] I have run nox and all the pipelines have passed.
[x] I have made unittests according to the code I have added/modified/deleted.

also another thought, could a backport of the latest unicodedata work for this (e.g. https://pypi.org/project/unicodedata2/)?

also another thought, could a backport of the latest unicodedata work for this (e.g. https://pypi.org/project/unicodedata2/)?

It would, but this way is something that I have been planning for a while as we now actually verify that the emojis are correct, instead of just assuming people pass the correct values. This can also help people have some sort of emoji check for commands, as it will fail to parse it if its an invalid emoji.

also another thought, could a backport of the latest unicodedata work for this (e.g. https://pypi.org/project/unicodedata2/)?

It would, but this way is something that I have been planning for a while as we now actually verify that the emojis are correct, instead of just assuming people pass the correct values. This can also help people have some sort of emoji check for commands, as it will fail to parse it if its an invalid emoji.

wasn't that why UnicodeEmoji.parse had validation in the first place?

wasn't that why UnicodeEmoji.parse had validation in the first place?

That validation didn't check if it was a valid unicode emoji, let alone a valid unicode emoji. It just checked if all the characters were valid unicode

import asyncio

import unicodedata
from hikari import emojis
from hikari.impl import rest as rest_impl

CHANNEL_ID: int =
MESSAGE_ID: int =
FOO = "\N{WAVING WHITE FLAG}"
TOKEN: str =

async def main():
    for i, codepoint in enumerate(FOO, start=1):
        print(unicodedata.name(codepoint))
        print(emojis.is_valid_discord_emoji(FOO))

    print(emojis.is_valid_discord_emoji(FOO))
    async with rest_impl.RESTApp().acquire(TOKEN, token_type="Bot") as cli:
        await cli.add_reaction(CHANNEL_ID, MESSAGE_ID, FOO)

asyncio.run(main())

Output: ```py
WAVING WHITE FLAG
False
False
To be honest this kind of feels like overcomplication that goes against the whole "light" aspect of this lib. In the few cases I can think of where people would want to make sure they're using/getting "valid discord emojis" (right now just unicode/Emoji 13.0 emojis) they'll end getting an error down the line from discord anyway and I don't really see any good way to keep up to date with the unicode version that discord is on as is (they barely publicise or announce when they update, reverse engineering the client to check which emojis it has declared in a way that's forwards compatible would be more effort than it's worth and just manually following and checking each client update seems like too much effort with using the raw client funnily enough likely not directly matching up with what the REST API allows and disses out). Plus accounting for any special casing Discord may introduce or remove for emojis in the future also seems like a difficult task if it's even possible (right now this likely isn't compatible with how discord kept backwards compatibility for their weird use of variation selector 16 in places as a random off the top of my head example). I've not actually seen any lists that keep up to date with the unicode version discord's doing properly with even the list we currently rely on being manually updated and lagging behind by something between 3 months to half a year so...
Anyway, the example I sent above uses an emoji that's supposedly "invalid" but is accepted (and returned) by the REST API.

Anyway, why does unicode_names need to be removed when it works perfectly fine on python 3.9 and there are backports which can make it work fine with 3.8

Anyway, the example I sent above uses an emoji that's supposedly "invalid" but is accepted (and returned) by the REST API.

This is actually cause the client follows a separate emoji set from the server, which seems to normalize them using https://github.com/davfsa/discord_emojis/blob/master/discord_emojis/utils.py#L24 (will push an new commit here in a bit)
I don't see how having this as an external package defeats the point of "light" to the lib. This is the same thing as having a backport of unicodedata, although it actually verifies for the user that the unicode emoji they are using is valid and not just a normal string.
As per the update problem you mentioned, unicode emojis don't update regularly, nor does Discord. We can always keep an eye out and ask emzi to update it if he missed an update.
And just wanted to add, the emoji you sent in the example is valid by REST, but it will never return it. I believe this update should cover all edge cases

I don't see how having this as an external package defeats the point of "light" to the lib. This is the same thing as having a backport of unicodedata, although it actually verifies for the user that the unicode emoji they are using is valid and not just a normal string.

why does this emoji validation have to be a part of hikari's core that brings in another dependency rather than a separate utility lib for the niche amount of ppl who can't rely on Discord to tell them which emojis are valid for some unexplained reason

This is the same thing as having a backport of unicodedata

Last time i checked version specific dependencies are a thing in setuptools though

why does this emoji validation have to be a part of hikari's core that brings in another dependency rather than a separate utility lib for the niche amount of ppl who can't rely on Discord to tell them which emojis are valid for some unexplained reason

If we are going to have an UnicodeEmoji object with validation, I would rather have proper validation. The reason for this is not only for Discord, it is also for .url and supporting re-uploading it. This way we can ensure that all emojis passed into that object will be able to be re-uploaded as an image, and not result in a nasty error.

also last time I checked python version specific dependencies are a thing in setuptools though

That is not what I meant, what I mean it is that it is another dependency that will not actually provide what we want.

I'm going to change this pr into a draft for emoji validation. I would like to merge the other open prs and continue development without blocking everything for this pr.


[BUG]System.Net.Sockets.SocketException in netcoreapp TargetFramework

Describe the bug
Simple console application targeting net48 works without problem, retargeting to netcoreapp3.0 cause SocketException
Exception or Stack Trace
No connection could be made because the target machine actively refused it
To Reproduce
Change TargetFramework from net48  to necoreapp3.0
Code Snippet
class Program
{
static void Main(string[] args)
{
var serviceBusConnectionString = "yourConnectionString"
var queue = "yourqueue";
var messageReceiver = new MessageReceiver(serviceBusConnectionString, queue);
messageReceiver.ServiceBusConnection.TransportType = Microsoft.Azure.ServiceBus.TransportType.AmqpWebSockets;
var message = messageReceiver.ReceiveAsync().Result;
    }
}

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Setup (please complete the following information):

OS: [Windows 10]
IDE : [Visual Studio 2019]
Version 4.0.0

Additional context
Add any other context about the problem here.
Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[x] Bug Description Added
[x] Repro Steps Added
[x] Setup information Added

@AndreiDegtiarev, thank you for letting us know of this issue. Aparna is trying to reproduce this problem. We will get back with our findings.

@AndreiDegtiarev, I just installed netcore3.0 and tried a sample app and it works pretty well. I would think its a machine problem. Maybe try it on another machine and see if its working.
Also, consider updating to latest visualstudio + .netcore SDK to see if it helps.

I don’t think that it is machine problem. I tried it already on different machines and I have the latest versions of Visual Studio and SDK. I think the problem is how .nectcore and .netframework interact with proxy server (otherwise I wouldn’t use TransportType.AmqpWebSockets). Please take more time to investigate this problem. Also I’m not alone with it, see for example discussion here: https://github.com/stewartadam/aspnet-servicebus-hostedservice/issues/1

We do not do any special handling for proxy in our code. Like you mentioned, this might be a .netcore issue.

@stephentoub @karelz, looks like a possible issue in .NET Core 3.0. Is it best to move this issue over to a .NET Core repo?

Based on the info above I believe @AndreiDegtiarev is using https://www.nuget.org/packages/Microsoft.Azure.ServiceBus/ version 4.0.0. That package only has a .NET Standard 2.0 asset in it.

Yes, I use Microsoft.Azure.ServiceBus version 4.0.0 and the problem also occurs if i target netcoreapp2.1

@AndreiDegtiarev , how are you configuring the proxy settings?

What's the full stack from the exception?
What proxy is being used / how is it being configured?

Is it best to move this issue over to a .NET Core repo?

This library is a black box... is there a repro that can be written directly against whatever .NET API is being used under the covers here?  Obviously there's a Socket at some point, but is this library talking directly to Socket, is it using HttpClient, something else?

There is a callstack in this issue https://github.com/stewartadam/aspnet-servicebus-hostedservice/issues/1#issuecomment-537644345
Source code for the library is here: https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/servicebus/Microsoft.Azure.ServiceBus/src

@stephentoub do you think this could be related to the replacement of HttpClient that happened in 2.1? Is there still a way to opt into the older one?

I work in quite big enterprise company and I have little info about proxy settings. From intranet following description from Symantec is referenced https://support.symantec.com/us/en/article.tech244697.html
Retarget to netcoreapp2.0 doesn’t help
My callstack looks different:
at System.Net.Http.ConnectHelper.d__1.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Threading.Tasks.ValueTask1.get_Result()    at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable1.ConfiguredValueTaskAwaiter.GetResult()
at System.Net.Http.HttpConnectionPool.d__52.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at System.Threading.Tasks.ValueTask1.get_Result()    at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable1.ConfiguredValueTaskAwaiter.GetResult()
at System.Net.Http.HttpConnectionPool.d__53.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at System.Threading.Tasks.ValueTask1.get_Result()    at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable1.ConfiguredValueTaskAwaiter.GetResult()
at System.Net.Http.HttpConnectionPool.d__45.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at System.Threading.Tasks.ValueTask1.get_Result()    at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable1.ConfiguredValueTaskAwaiter.GetResult()
at System.Net.Http.HttpConnectionPool.d__47.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()    at System.Net.Http.RedirectHandler.<SendAsync>d__4.MoveNext()    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at System.Net.WebSockets.WebSocketHandle.d__24.MoveNext()

So to summarize:

Simple app does not work (on .NET Core 2.0 nor 3.0), but works fine on .NET Framework 4.8

Questions:

Is it related to the environment / proxy setup / etc.? In other words, are you able to reproduce the same problem on vanilla Azure VM, not connected to domain, VPN, or anything?
Are you able to try to remove the Azure SDK layer out of the picture? (i.e. use HttpClient directly against the same endpoints)

BTW: You can use GitHub code formatting on code and on exceptions for better readability.

Source code for the library

@petermarcu, that doesn't appear to be the library that's actually making the connection; it looks like this library sits on top of Microsoft.Azure.Amqp, which does have multiple assets in its nuget package, with different libraries for .NET Framework vs .NET Standard, and more:

In the future for things like this it'd be helpful if the team that owns the library being used could start the investigation and rule out the possibility that the difference is due to an issue with the intermediate library rather than something lower down in the core platform.

Retarget to netcoreapp2.0 doesn’t help

That means that two completely different WebSocket implementations, one based on WinHTTP and one based on SocketsHttpHandler, are unable to connect to the target site. They both, however, follow the same approach to resolving proxies, so more and more I think it's an environment-related issue.
@AndreiDegtiarev, would you be willing to take a trace and share it with us?  You can do so with a tool like PerfView, in which case when you do the collection you'd add *Microsoft-System-Net-Http into the "additional providers" text box.  Or you can do so by copying/pasting https://github.com/dotnet/corefx/blob/master/src/Common/tests/System/Diagnostics/Tracing/ConsoleEventListener.cs into your app, and then at the beginning of your main method, adding a line:
using var listener = new System.Diagnostics.Tracing.ConsoleEventListener("Http");

after which when you run the app you should see a lot of diagnostics output to the console. Either way, you can then email me the results (stoub at microsoft) or share it in whatever other manner you feel most comfortable.

In perusing the Microsoft.Azure.Amqp library, I believe I see the problem.  This code:
https://github.com/Azure/azure-sdk-for-net/blob/765ff59083a3dd611fbb9430be4d9437cbbd613b/sdk/servicebus/Microsoft.Azure.ServiceBus/src/ServiceBusConnection.cs#L314-L318
is going to result in WebRequest.DefaultWebProxy being passed down into the underlying implementation.  On .NET Framework, that implementation uses WebRequests under the covers, and thus understands WebRequest's DefaultWebProxy.  On .NET Core, WebRequests are considered legacy and aren't used by ClientWebSocket, and thus doesn't understand the sentinel value that is DefaultWebProxy.
cc: @davidsh

Is there a ".NET Standard" value for proxy that can be used that will be respected by both .NET Core and .NET Framework?

Is there a ".NET Standard" value for proxy that can be used that will be respected by both .NET Core and .NET Framework?

No.
We recommend callers use HttpClient and not HttpWebRequest.  Not sure that is possible with the layered stack of libraries being used.
With HttpClient, the default properties of HttpClientHandler are for 'UseProxy=true' and 'Proxy=null'. And that combination means to use the 'system default' proxy.  This differs from HttpWebRequest where there is a single property related to proxies and the .Proxy property is non-null by default. That non-null value is the 'system default' proxy. Using null for .Proxy with HttpWebRequest means not to use any proxy.

@davidsh, what should the Amqp library be doing then, given that it may run on different underlying stacks?

@davidsh, what should the Amqp library be doing then, given that it may run on different underlying stacks?

By 'different stacks' do you mean different versions of .NET Framework or .NET Core?
If so, if Amqp uses HttpClient instead of HttpWebRequest I think the platform differences would work better regarding proxy support.

@davidsh, it's just using ClientWebSocket.

@davidsh, it's just using ClientWebSocket.

Thanks for clarifying. Yes,
ClientWebSocketOptions.Proxy is the only proxy-related property for ClientWebSocket.
WebRequest.DefaultWebProxy is considered a legacy type/property. In .NET Core, it resides in the System.Net.Requests library. And we do not have any dependency from System.Net.WebSockets.ClientWebSocket to System.Net.Requests. We would not want to add such as thing because it would create a circular dependency.  System.Net.Requests depends on System.Net.Http already. And System.Net.WebSockets.ClientWebSocket depends on System.Net.Http.
Currently in .NET Core, System.Net.Http and thus SocketsHttpHandler, et. al. do not observe WebRequest.DefaultWebProxy at all.
In .NET Core 3.0, we added a new HttpClient.DefaultProxy static property (https://github.com/dotnet/corefx/pull/37333) where the intent was to support similar 'global settings' scenarios as WebRequest.DefaultWebProxy.
For .NET Core 3.0, a solution could be to set ClientWebSocketOptions.Proxy = HttpClient.DefaultProxy. That would tell the a particular clientwebsocket to use the 'system default' proxy. But I don't think we have a general solution for all versions of .NET Core.

@davidsh, the Amqp package doesn't have a netcoreapp asset.
What I hear you saying is there's no way on .NET Core prior to 3.0 to tell ClientWebSocket to use a default proxy.  Is that correct?
And on .NET Core 3.0, if you're targeting netcoreapp3.0, it could be done by passing in HttpClient.DefaultProxy.  Correct?
It seems then like the fix is for WebRequest.DefaultWebProxy to return HttpClient.DefaultProxy (and fix System.Net.Requests as-is needed).  That way, any code using WebRequest.DefaultWebProxy today will "just work".  And the Amqp package will be broken on .NET Core for default proxies until we deploy such a fix, so we should consider servicing it to release/3.0.

Also seems like we should augment ClientWebSocketOptions with better support for proxies, so we don't hit this same issue in the future should the underlying implementation change again.

From a design perspective, ClientWebSocketOptions was designed in the timeframe before HttpClient. So, it used the semantics of a single .Proxy property. ClientWebSocketOptions.Proxy sort of became broken when we moved to .NET Core.
I believe that on .NET Framework, the default value of ClientWebSocketOptions.Proxy is a non-null value where that value is the current value of WebRequest.DefaultWebProxy. And that value represents to use the 'system default' proxy (which may or may not actually result in using a proxy...because it depends on system settings).  But on .NET Core, we ended up I think where ClientWebSocketOptions.Proxy has a default value of null. And that means don't use any proxy by default.
So, we should investigate this and determine how to reconcile .NET Core moving forward.
However, I don't think we need to add any more properties to ClientWebSocketOptions to get "better support for proxies". Having a single property, .Proxy, is fine as long as a 'system default' proxy can be represented by a object. And that would also support the scenarios where credentials need to be sent to a 'system default' proxy. Because that would inherit the IWebProxy.GetCredentials() of any proxy set into the property.

And on .NET Core 3.0, if you're targeting netcoreapp3.0, it could be done by passing in HttpClient.DefaultProxy. Correct?

And specifically, yes, to this question.

@davidsh , I see that PR https://github.com/dotnet/corefx/pull/41692 issue is marked with 5.0 milestone.
Meanwhile, for this issue, in .net core 3.0, would you suggest that we expose IWebProxy as a property on ServiceBusConnection which customers can then set it to HttpClient.DefaultProxy so that we need not wait for 5.0 release?

@davidsh , I see that PR dotnet/corefx#41692 issue is marked with 5.0 milestone.

Changes are made first to master branch (5.0). I suspect that after that change is validated, then @stephentoub will create a release/3.0 PR for evaluation for servicing release of .NET Core 3.0.

@AndreiDegtiarev The fix would be released in .net-core 3.0.2 and 3.1 version

@AndreiDegtiarev, we are closing this issue for now as there is now visibility into in what version a fix will be available. Please feel free to request to open it again if you have a question.


#11087 Refactor row insertion mechanism to respect partitioned selection

https://user-images.githubusercontent.com/35821147/114702276-8b880080-9d2c-11eb-915a-0882b15dda8b.mp4

New algorithm along with refactored addNewRow makes this PR "dangerous". We need to test it in different databases with different data types.
Including MongoDB, CouchDB, Cassandra and with complex data types in PostgreSQL.


Laravel 5.2 Package Shift

This pull request includes the changes for upgrading your Laravel Package to Laravel 5.2. Feel free to commit any additional changes to the laravel-package-5.2-shift branch.
Before merging, you should:

Checkout the laravel-package-5.2-shift branch
Review all pull request comments for additional changes
Run composer update (if the pre-commands fail, add --no-scripts)
Thoroughly test your code

This Shift is still in beta. If there were changes you feel could have been automated, please leave a comment on this pull request to help improve Laravel Shift.

ℹ Laravel 5.2 updated the database session driver. If you are using the database driver, you need to add the user_id (nullable integer), ip_address (nullable string), and user_agent (text) columns to your session database table.


about network

Hello.
Can I know why you choose the two teacher networks,Can I replace a teacher network with a network whose parameters are updated independently using SGD?
Thank you !

Hi @ruizes
I apologise for the late reply, I was in a travel.
1). The two teachers based structure can improve the reliance of the pseudo label and further improve the generalisation when they meet the hard unlabelled samples.
2). Yes, you can.
Thanks for your interest.
Cheers,
Yuyuan

Thank you for your reply. Have a great trip!


causal_ts: fix issue #12498

Issue Number: #12498
Signed-off-by: pingyu<EMAIL_ADDRESS>
What is changed and how it works?

Issue Number: Close #12498
What's Changed:

Fix issue #12498

By logging timestamp of regions, we found that the observing of change from follower to leader by on_role_change would be later than the real role change in raft state and adjacent write commands.
When this happen, the adjacent write commands would get unexpected smaller timestamp than the peer transferred from.
The following screenshot shows such a case.

The "leader transfer" indicates where the leader transfer happen. The log with "pre_propose" is the adjacent write command. "[is_leader=false]" indicates that the change of peer role have not been observed yet. "on_role_change" is the observing of peer role change.
We can see that the ts of "pre_propose" is smaller than ts of "on_role_change", which is just the violation of causality correctness.
In this PR, we observe region role change to "Candidate" other than "Leader", to avoid the late of flush.
Related changes

PR to update pingcap/docs/pingcap/docs-cn:
No.
Need to cherry-pick to the release branch
No.

Check List
Tests

Manual test (add detailed scripts or steps below)
E2E testing, with "shuffle_leader_scheduler".

Side effects

No.

Release note
Fix issue #12498.

/cc @BusyJay @Connor1996

Test case should be added.

OK~
I will add test cases in another PR coming soon.

/merge

/run-all-tests

/run-test

/run-test

/run-test

/run-test

/run-test

/run-test

/run-test

/run-test

/run-test


Fix the NetworkUnavailable workaround

The upstream change in 1.18 to disable the implicit local API endpoint meant that all calls to kubectl need the explicit config. Apparently, we missed one in the poke_network workaround for cloud-providers which don't properly manage the flag (e.g., GCP).
Fixes lp:1907088

LGTM


No info on failures

PHPunit now distincts between errors and failures. When we have failures the suite exits with status != 0 as expected but we dont get any clue on the problem.
 ./vendor/bin/paratest

Running phpunit in 5 processes with /home/jean/projects/compufacil/Backend/vendor/bin/phpunit

Configuration read from /home/jean/projects/compufacil/Backend/phpunit.xml.dist

.........................................S.............SS....   61 / 1534 (  3%)
.....S............SSS.........................S..............  122 / 1534 (  7%)
........................................SSSSSSSSS............  183 / 1534 ( 11%)
..........S..................................................  244 / 1534 ( 15%)
.............................................SSSSSS..........  305 / 1544 ( 19%)
.............................................................  366 / 1545 ( 23%)
.............................................................  427 / 1545 ( 27%)
...............SS............................................  488 / 1545 ( 31%)
.............................................SSS.........S...  549 / 1545 ( 35%)
.............................................................  610 / 1545 ( 39%)
.............................................................  671 / 1547 ( 43%)
.............................................................  732 / 1547 ( 47%)
..........................S......S...........................  793 / 1554 ( 51%)
.........S..............=.....................................  854 / 1556 ( 54%)
.............................................................  915 / 1556 ( 58%)
......SSS.........S..........SSSSSSSSSS......................  976 / 1556 ( 62%)
............................................................. 1037 / 1557 ( 66%)
......S...................................................... 1098 / 1562 ( 70%)
............................................................. 1159 / 1576 ( 73%)
............................................................. 1220 / 1613 ( 75%)
............................................................. 1281 / 1616 ( 79%)
............................................................. 1342 / 1621 ( 82%)
............................................................. 1403 / 1621 ( 86%)
.....S..............................SSSSS.................... 1464 / 1621 ( 90%)
...........................S.........SSSSSSSSS............... 1525 / 1621 ( 94%)
..............................................S.............. 1586 / 1623 ( 97%)
.................S...................

Time: 1.08 minutes, Memory: 48.00MB

FAILURES!
Tests: 1623, Assertions: 3900, Failures: 1, Errors: 0.

What version of Paratest and PHPUnit are you running? I couldn't reproduce this one on the latest version of both. do you know what the specific failure was caused by? Was it simply a failed assertion or was it something else, like an E_ERROR raised.

Do anyone know some workaround around this? Its really hard to chase the warning down if you have really huge test-set that you are notable run in PHP Unit due to memoory requirements.

I'm pretty sure this was fixed in later versions of paratest and phpunit. But if you're not able to upgrade (I highly recommend you do if you can), you can solve this by adding a simple test listener class and add it in phpunit.xml. on the test listener you can throw an exception on test warning, this way you will get a stack trace.
Test listeners can also be used to debug unknown failure issues, by putting a log statement for test start and test end... If there is any start without an end, you know that test crashed.

I am on version 1.1.0 of paratest and 6.5.7 phpunit. Do you suggest to move on 7?

Thanks a lot for that listener hint, trying it now :)

1.1.0 and 6.5.7 should be fine

Here is a listener I use if anyone else got into same trouble as I did.
<?php

use PHPUnit\Framework\AssertionFailedError;
use PHPUnit\Framework\Test;
use PHPUnit\Framework\TestListener;
use PHPUnit\Framework\TestSuite;
use PHPUnit\Framework\Warning;

/**
 * @see https://github.com/paratestphp/paratest/issues/282
 */
class MakeWarningErrorAgainTestListener implements TestListener
{

    /**
     * @inheritdoc
     */
    public function addError(Test $test, Exception $e, $time)
    {
    }

    /**
     * @inheritdoc
     */
    public function addWarning(Test $test, Warning $e, $time)
    {
        throw new LogicException($e->getMessage(), $e->getCode(), $e);
    }

    /**
     * @inheritdoc
     */
    public function addFailure(Test $test, AssertionFailedError $e, $time)
    {
    }

    /**
     * @inheritdoc
     */
    public function addIncompleteTest(Test $test, Exception $e, $time)
    {

    }

    /**
     * @inheritdoc
     */
    public function addRiskyTest(Test $test, Exception $e, $time)
    {

    }

    /**
     * @inheritdoc
     */
    public function addSkippedTest(Test $test, Exception $e, $time)
    {

    }

    /**
     * @inheritdoc
     */
    public function startTestSuite(TestSuite $suite)
    {

    }

    /**
     * @inheritdoc
     */
    public function endTestSuite(TestSuite $suite)
    {
    }

    /**
     * @inheritdoc
     */
    public function startTest(Test $test)
    {
    }

    /**
     * @inheritdoc
     */
    public function endTest(Test $test, $time)
    {
    }
}

I am still having this issue with the latest paratest and phpunit 7.3
Time: 26.25 seconds, Memory: 90.00MB
FAILURES!
Tests: 14851, Assertions: 44497, Failures: 2, Errors: 0.```

PHPUnit gives the warnings:

There were 2 warnings:

Warning
The data provider specified for zzzTest:xxx is invalid.
...
Warning
The data provider specified for zzzTest::yyy is invalid.
...

^ Same issue

^ Same issue here

As a temporary solution. you can try to use the Paratest Logger https://github.com/paslandau/paratest/blob/wrapper-runner-log/docs/logging.md

We are willing to sponsor support for Warnings and Risky tests.

@willemstuursma Thank you for your generous offer. I gave this issue the label "most-upvotes" now (which basically means "often requested for"). I'll take a look at this once I have some more spare time.

@andreasschroth we can send you some Mollie swag or stuff from your Amazon wishlist instead, please contact me at<EMAIL_ADDRESS>

We are also having problems with errors in the setUp() method. They are ignored by paratest and work with PHPUnit.
We are also willing to sponsor fixing this.

Same issue here, not seeing what failed and why is a hindrance. Using WrapperRunner here, and I'm calling paratest on a Docker container application via SSH.
Example: made empty test method that does not run assertions. PHPUnit complains about a risky test, but Paratest says there was an error but does not tell me it was a risky test.
Will check if those listeners and loggers are an OK solution in the meantime.
Let me know if you need any help with this, I think I can get a bit of company time to assist here and there.

The PR #402 might fixed the problem with warnings. I'm not sure about the risky tests, IMHO, they must be implemented in the same way as I did with warnings

The workaround from above did not properly work for risky tests. We were getting weird framework-related object instantiation errors without any valid stacktrace.
I researched it a little bit and the problem seems to be in phpunit itself (partly), as it's JUnit logs are different for risky tests.
https://github.com/sebastianbergmann/phpunit/issues/3937

No because of https://github.com/sebastianbergmann/phpunit/issues/3937

Good news: https://github.com/sebastianbergmann/phpunit/pull/4438 has been merged, and together with https://github.com/paratestphp/paratest/pull/523 now risky tests are reported correctly in ParaTest too:
There were 2 risky tests:

1) ParaTest\Tests\fixtures\failing_tests\UnitTestWithMethodAnnotationsTest::testRisky
This test did not perform any assertions

./paratest/test/fixtures/failing_tests/UnitTestWithMethodAnnotationsTest.php:68

2) ParaTest\Tests\fixtures\failing_tests\UnitTestWithErrorTest::testRisky
This test did not perform any assertions

./paratest/test/fixtures/failing_tests/UnitTestWithMethodAnnotationsTest.php:68

I still have risky tests (mostly no assertion tests) reported as errors
PHPUnit 9.4.2 && ParaTest 5.0.4

That's correct, because until the end they cannot be distinguished by errors due to a JUnit XML schema.
By now risky tests are reported correctly in ParaTest too I meant that now they are verbose and you can debug them.

so no way to ignore them in paratest report them as warnings instead ?

No, because this is what ParaTest gets from PHPUnit: https://github.com/sebastianbergmann/phpunit/blob/3866b2eeeed21b1b099c4bc0b7a1690ac6fd5baa/tests/end-to-end/loggers/log-junit.phpt#L40-L45

No, because this is what ParaTest gets from PHPUnit: https://github.com/sebastianbergmann/phpunit/blob/3866b2eeeed21b1b099c4bc0b7a1690ac6fd5baa/tests/end-to-end/loggers/log-junit.phpt#L40-L45

I see, well the quick solution for me is to address those risky tests I guess, thanks :+1:


Remove proxy.config.http.server_tcp_init_cwnd

This configuration option only has meaning on Solaris, a platform that we currently don’t support. As such, it’s actually confusing that it’s there to begin with, since it doesn’t do anything on any of the platforms we do support.
I’d like to remove this option, and the code around it.

Blocks #5457


Enable habana flash attention for llama2 ft

Add Habana flash attention support to llama2-70b finetuning

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Drop this PR and created a separate one


save problem

how can I save the output as png or jpg?

Use:
dot.format = format
dot.render(filename)

where

dot is graph representation return using make_dot
format is format from graphviz (see https://graphviz.readthedocs.io/en/stable/manual.html#formats)
filename specifies path, where to save graph

I tried to do that and the image I get is nonetype:
            disp = model(imgL,imgR)

            dot = make_dot(disp)
            print(dot)

            dot.render("test.png")

@kamalravi it gives
----> 7 dot.format("png")
      8 dot.render("danet_model")
TypeError: 'str' object is not callable

I think a full example for saving might be helpful, especially for those not immediately familiar with graphviz.
dot = torchviz.make_dot(y.mean(), params=dict(model.named_parameters()))
dot.format = 'png'
dot.render('torchviz-sample')

And the image is saved under the current working directory as torchviz-sample.png.