Voting - Official deactivation of the push notification functionality

Hello everyone,
When I decided to create this plugin, there was still no way to run Dart code while the application was terminated/ killed. Therefore, to make use of all available resources of the plugin, even when the application was finished, i had to create this plugin with local notifications and push notifications, to be able to natively create the notification while the app was terminated.
Since then, two situations have occurred that made me rethink the need for the AwesomeNotifications plugin to work with push messages:
1 - It is now possible to run Dart codes while the application is closed, using the firebase onBackgroundMessage method. This also eliminates the need to use iOS Extensions and eliminates many of the problems that our developers have faced on the apple platform.
2 - Many of the developers are using the firebase_messaging and awesome_notifications plugins concurrently, and both of then are pushing notifications at same time. This creates several problems, due to the two plugins accessing the push feature, which is a global feature of the application.
Therefore, I am suggesting to permanently remove the push notification functionality from the AwesomeNotifications plugin and leave this role to the firebase_messaging plugin.
I would like your opinion if I should remove this feature or not and leave this plugin only for local notifications.
To vote, please access the link bellow:
 SurveyMonkey - Voting - Official deactivation of the push notification functionality

I previously used firebase_messaging and flutter_local_notifications before starting with awesome_notifications. I don't realize yet which different implications has this decision.
But this is my point of view:
With firebase_messaging and flutter_local_notifications (what I previously used) I had to configure firebase_messaging, test multiple bodies (of push notifications) for both OS (Android and iOS), handle the notifications in different scenarios and run whatever with flutter_local_notifications. 👎🏼
Now with awesome_notifications, I don't care about firebase_messaging, how to handler the incoming notifications, or the push notifications' body. 👍🏼

using the firebase onBackgroundMessage method. This also eliminates the need to use iOS Extensions

Now I can use ringtones for notification calls, display images in the notifications.. thanks to the AwesomeContentExtension, that's right? Do we still have the same display options without those iOS Extensions?

Many of the developers are using the firebase_messaging and awesome_notifications plugins concurrently

What sense has this? I'm currently developing a messaging application and I don't need to set up or work with firebase_messaging anymore (at least directly I mean).

Everything is ordered and working fine 😄
So.. why change?

I previously used firebase_messaging and flutter_local_notifications before starting with awesome_notifications. I don't realize yet which different implications has this decision.
But this is my point of view:
With firebase_messaging and flutter_local_notifications (what I previously used) I had to configure firebase_messaging, test multiple bodies (of push notifications) for both OS (Android and iOS), handle the notifications in different scenarios and run whatever with flutter_local_notifications. 👎🏼
Now with awesome_notifications, I don't care about firebase_messaging, how to handler the incoming notifications, or the push notifications' body. 👍🏼

using the firebase onBackgroundMessage method. This also eliminates the need to use iOS Extensions

Now I can use ringtones for notification calls, display images in the notifications.. thanks to the AwesomeContentExtension, that's right? Do we still have the same display options without those iOS Extensions?

Many of the developers are using the firebase_messaging and awesome_notifications plugins concurrently

What sense has this? I'm currently developing a messaging application and I don't need to set up or work with firebase_messaging anymore (at least directly I mean).

Everything is ordered and working fine 😄
So.. why change?

Many of the developers are using the firebase_messaging and awesome_notifications plugins concurrently

What sense has this? I'm currently developing a messaging application and I don't need to set up or work with firebase_messaging anymore (at least directly I mean).

That was what I thought, but in practice many developers discovered this plugin after had used and implemented many functions based on firebase_messaging with flutter_local_notification.
So, thinking in not lose job already done, they keep firebase_messaging with awesome_notifications, and both of then try concurrently access the notification data. This creates many undesired issues and time to figure out what's going on.

Many of the developers are using the firebase_messaging and awesome_notifications plugins concurrently

What sense has this? I'm currently developing a messaging application and I don't need to set up or work with firebase_messaging anymore (at least directly I mean).

That was what I thought, but in practice many developers discovered this plugin after had used and implemented many functions based on firebase_messaging with flutter_local_notification.
So, thinking in not lose job already done, they keep firebase_messaging with awesome_notifications, and both of then try concurrently access the notification data. This creates many undesired issues and time to figure out what's going on.

Hello @rafaelsetragni , I believe what you did is really great as all developers believes that too. What I think makes people confused, is that the way this plugin integrate with FCM. I for example, followed the instructions to install FCM, like creating Application.kt, write those lines in build.gradle .. etc. the question makes me and other confused, should we remove all things related to firebase_messaging? as I believe also your plugin will help me do what I need more than the firebase_messaging.
what I mean, its just people who come from firebase_messaging messaging there is no integration instructions help 'em in the read me file.
so my suggestions is, keep it up just help us with the README file.
Thank you Rafael.

Hello @rafaelsetragni , I believe what you did is really great as all developers believes that too. What I think makes people confused, is that the way this plugin integrate with FCM. I for example, followed the instructions to install FCM, like creating Application.kt, write those lines in build.gradle .. etc. the question makes me and other confused, should we remove all things related to firebase_messaging? as I believe also your plugin will help me do what I need more than the firebase_messaging.
what I mean, its just people who come from firebase_messaging messaging there is no integration instructions help 'em in the read me file.
so my suggestions is, keep it up just help us with the README file.
Thank you Rafael.

Agree with @ralmarri12.
I think people confused for immigrate firebase_messaging official library to awosome_notification.
It was a problem for me before.
So, In library's docs can provide these steps for immigrate, it would be very helpful to many people.
Also, one of awesome_notification's user, I can help to write immigrate docs with my pleasure.
Thx to @rafaelsetragni  for make good library :D

Many of the developers are using the firebase_messaging and awesome_notifications plugins concurrently

What sense has this? I'm currently developing a messaging application and I don't need to set up or work with firebase_messaging anymore (at least directly I mean).

That was what I thought. But in practice many developers discovered this plugin after had used and implemented many functions based on firebase_messaging with flutter_local_notification.
So, thinking in not lose job already done, they keep firebase_messaging with awesome_notifications, and both of then try concurrently access the notification data. This creates many undesired issues and time to figure out what's going on.

Yes. Exactly. I (and many developers) used firebase_messaging and flutter_local_notifications before starting with awesome_notifications. I required action buttons, so that started to use awesome_notifications.

Hello everyone,
When I decided to create this plugin, there was still no way to run Dart code while the application was terminated/ killed. Therefore, to make use of all available resources of the plugin, even when the application was finished, i had to create this plugin with local notifications and push notifications, to be able to natively create the notification while the app was terminated.
Since then, two situations have occurred that made me rethink the need for the AwesomeNotifications plugin to work with push messages:
1 - It is now possible to run Dart codes while the application is closed, using the firebase onBackgroundMessage method. This also eliminates the need to use iOS Extensions and eliminates many of the problems that our developers have faced on the apple platform.
2 - Many of the developers are using the firebase_messaging and awesome_notifications plugins concurrently, and both of then are pushing notifications at same time. This creates several problems, due to the two plugins accessing the push feature, which is a global feature of the application.
Therefore, I am suggesting to permanently remove the push notification functionality from the AwesomeNotifications plugin and leave this role to the firebase_messaging plugin.
I would like your opinion if I should remove this feature or not and leave this plugin only for local notifications.
To vote, please access the link bellow:
 SurveyMonkey - Voting - Official deactivation of the push notification functionality

I just discovered this plugin and it's awesome! Before I had to write my own implementation of notifications that I need when a new fcm notification was recived, but I don't know how to write iOS implementation, and this plugin does all of that I need. Please don't remove the fcm functionality, it's so handy.
Btw, why don't create a callback or something that replace the official plugin to the end user? So the end user don't need to have firebase_messaging.

The survey reach the free limit. I gonna post the results soon.

Why remove it? You can't make a setting to enable it? It's so useful the connection with fcm

As you can see, the decision was very tight!
I saw and avaliated all the interactions on voting about both sides, and took every single one in consideration, specially with the comments.
Many people enjoyed the fact that this plugin already do both things and makes everything more easy. This helped many initial developers to implement the notifications localy and for push.
But, many other developers preffer to remove the push functionality, not only because they already used firebase_messaging plugin, but because they understood that those other projects that only use awesome_notification can be easily modified to use firebase apart.
And, for last, remove the push functionality will eliminate a lot of problems that we are facing, such as:

iOS content extensions
pod files issues in release mode
Issues with 3rd projects that are using firebase_messaging with awesome_notification, untill they realise that there is a problem.
Problems with native libraries out of sync against they last version, resulting in a double global effort from flutter community.

So, I will remove the push functionality, but also I will leave a special notice in the README file about this decision, in addition to maintaining the push topics with all instructions for integrating with the firebase_messaging plugin (and even others later on).
I apologize to everyone who believed in this library for the inconvenience that you will have to go through to adapt to the new structure. As this plugin is still in development, I need to take advantage of these learnings to modify when necessary and make a better final product, which serves the entire community, giving more freedom of creation and more power in the hands of the developers.
I promisse to make my best to do this transition smooth as possible to everyone, and also i belive this is the best solution for the future.
Thank you so much for all the support and i hope soon will have a better plugin to help every one.

@rafaelsetragni there will be a easy way to (re)implement the push notifications feature?
For example the plugin can expose a static method that automatically parse the data coming from the official fcm plugin and shows a notification.
What I mean is if the plugin will be ready for the connection, but will be the final developer to "connect" the callbacks from fcm to the method of this plugin

What I mean is if the plugin will be ready for the connection, but will be the final developer to "connect" the callbacks from fcm to the method of this plugin

Exactly! And i will leave all instructions to make it easy to do.
And also, you gonna end with more development freedom to handle the push notification.

@rafaelsetragni
When can we expect new updated package in live ?

Maybe in next week. I need to compile all current demands plus the push deactivation itself.

To be honest i started using awesome_notification without installing firebase_messaging. Until i realized i had to subscribe and unsubscribe my devices to topics. So i installed both without any issue until now. But if you think it is a cleaner solution i am with you. Luckily i am not in production yet. Thanks for this awesome plugin.

Maybe in next week. I need to compile all current demands plus the push deactivation itself.

@rafaelsetragni
Any update on this ?

I have an app in production and I would like to use this plugin... There are any updates?

I gonna release soon a final .5 version, with push notifications yet activated. After that, i will remove definetely the push feature and release a .6 version

@rafaelsetragni
Any update on this ? We are waiting for this feature to remove push notifications and use only for local notifications.
And also, will you work these changes with Flutter 1.x.x or 2.0 new version ?
I think many developers will not migrate immediately to flutter 2.0 version.

@rafaelsetragni
In version 0.0.5+8, you have mentioned "FCM service with comments removed for iOS."
Which means completely removed FCM from this package both for Android and iOS ? Or still need to do some changes to complete that functionality ?

No, it's still working. I only removed the comment tag from the FCM classes, which I used to test whether the fcm could be removed or not, but I forgot to delete it later.
FCM will be completely removed in version 0.0.6.

Okay. Thank you. Please release v 0.0.6 very soon. We are waiting for this. Then only we can submit our app to production.

This plugin is still in development. Only plug-ins above 1.0.0 are considered stable.
If you intend to use it in production, check if it is working correctly on Android 10 or higher, as I was not able to run the emulator on my local machine. I am receiving BSOD all the time.

Okay. Sure.

Hello,
I've been using local notif plugin, so I'm a new user :)
on my side I'm testing your plugin on : android10 real device, and also in emulator with android11.
Looks good so far. But I'm just testing local notifications.
I'm also using firebase messaging plugin for background, and am eager to test your next release for local notif in background msg.
I just wonder how it will work, when receiving a background msg from FCM in android, and call an awesome notif with actions, will that create two notifications in android ?? (maybe a stupid question and I should read docs..^^)
Keep the good work, your package looks really promising

I gonna develop a example to show how to do that integration and also document it on our Readme file. Unfortunately, i don't start to code this part yet, so i dont know yet how to do it in details.

How to completely disable this package for iOS only ?
It is working perfectly for android for me. For Android, i have checked the Platform.isAndroid, then only i intialized this package for Android. Below is the code.
 if (Platform.isAndroid) {
      initializeAwesomeNotifications();
    }

But it is initializing for iOS also and ios push notifications are not working and getting below error.
"receiveNotification: notification discarted"
I am aware of opened issue regarding discart notification.
Is it anyway to disable awesome notifications for iOS completely and use it for android for now ?
Mainly i am using awesome notifications for action buttons. Because of push notificatin issues, i am not able to move forward.

@rafaelsetragni
Please help me on this to work my app without this awesome package for iOS only.
I want to use this package for Android as it is working perfectly.  So i want to remove for only iOS. Since month, you are not updating any code for this.

In fact, im updating it in a different branch, and im doing it for free, please be patient.
Now i need to test it in android and iOS. Everything ia already coded for 0.0.6 version.

Okay. Thank you.
When can we expect 0.0.6 version in live ?

@rafaelsetragni what is the status of 0.0.6? I discovered this plugin a few days ago and I want to take a look and try to contribute but I don't know what branch you are working on, what has been done, or what the next steps are


Make queue less specific

We have a whole pile of data we could get out of there, why limit it to only mail-queue-count?

A -l flag will be added to statdin 3.3r2 listing all available counters
system-cpu-usage
system-cpu-temperature
system-mem-usage
...

module fixed in https://github.com/halonsecurity/halonctl/commit/4fcec243fc8b7cd4f4ea94fdf56aeca245f13fe6
-l got fixed in https://github.com/halonsecurity/halonctl/commit/fdfcc59bf18cf3837bf7cde1312214ce0551a2e7


Added docs for how to use data-URIs

Fixes #217

I expanded this content somewhat and included it in issue#238/PR#243. Thanks for bringing this up and explaining it!


Bump dcos-test-utils

High-level description
Fixes invalid bump form https://github.com/dcos/dcos/pull/5841
Corresponding DC/OS tickets (required)

DCOS-54793

Related tickets (optional)

DCOS-ID JIRA title / short description.

Checklist for component/package updates:
If you are changing components or packages in DC/OS (e.g. you are bumping the sha or ref of anything underneath packages), then in addition to the above please also include:

[ ] Change log from the last version integrated (this should be a link to commits for easy verification and review): example
[ ] Included a test which will fail if code is reverted but test is not. If there is no test please explain.
[ ] Test Results: [link to CI job test results for component]
[ ] Code Coverage (if available): [link to code coverage report]

https://github.com/dcos/dcos/pull/5841

@mesosphere-mergebot request-review @orsenthil @cprovencher @pcrews @ArmandGrillet

@ArmandGrillet I it updated to 5fb66a9f9be41ca50eea2069d6f72ba686db4cd2

@mesosphere-mergebot label Ready For Review

@mesosphere-mergebot request-review @drewvanstone @br-lewis @makkes @ArmandGrillet  @orsenthil

@mesosphere-mergebot changelog-not-required minor dcos-test-utils bump


⚠ Bofrost.de has degraded performance

In 584bfcf, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1345 ms

Resolved: Bofrost.de performance has improved in cf7eb23 after 48 minutes.


Must unwrap page instances in Swift

Fixes issue #150

Is there more coming?

Oh... looks like the code is OK?

That's it! The user has to force unwrap the page instance optionals. No changes in library code needed.

Ah, a better fix is to annotate your initializers to indicate they cannot fail.

Change your initializers to have nonnull annotation. For example,  - (nonnull instancetype)initWithContents:...

I will modify the PR accordingly.

Don't merge this yet. For some reason the initializers are still marked as failable. Debugging now.

All good, merge away!

👍


Code doesn't work without internet

Silently fails, no error or warning
Run yarn test produces empty out.svg
To test, turn off wifi on compy

Need to get better error reporting but issue categorized wrong


feat: add basic functionality for icicle chart to display static data

🏆 Enhancements
adds functionality for the icicle chart to have data passed in and display (static, no interactions
yet)

do you have any screenshots?

Yes, I have some some screenshots! This is what it looks like currently in Storybook with the intention that it is not the final look. Since this PR was already getting a little big, I wanted to make it in a separate one:

Additionally, the README for this package has a screenshot of a prototype that is slightly outdated (we're still going through design, but have made some changes to spacing between icicles, etc.)


[09-25 00:51:48] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:51:48] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 14433 for upload handle: 9 current stream handle: 9068541
[09-25 00:51:48] postReadCallback(): Wrote 14433 bytes to Kinesis Video. Upload stream handle: 9
[09-25 00:51:48] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:51:48] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 0 for upload handle: 9 current stream handle: 9068541
[09-25 00:51:48] postReadCallback(): Pausing CURL read for upload handle: 9
[09-25 00:52:24] curlCompleteSync(): curl perform failed for url https://s-xxxxxxxxx.kinesisvideo.us-west-2.amazonaws.com/putMedia with result Timeout was reached: Operation too slow. Less than 30 bytes/sec transferred the last 30 seconds
[09-25 00:52:24] curlCompleteSync(): HTTP Error 0 : Response: (null)
[09-25 00:52:24] terminateCurlSession(): Force stopping the curl connection
[09-25 00:52:24] putStreamCurlHandler(): Stream with streamHandle 9068541 has exited without triggering end-of-stream.  Upload handle 10. Duration available in 100ns: 0. Bytes available: 0
[09-25 00:52:25] defaultStreamDataAvailableCallback(): Reported streamDataAvailable callback for stream handle 9068541. Upload handle 10. Duration available in 100ns: 0. Bytes available: 0
[09-25 00:52:25] stepStateMachine(): PIC Stream State Machine - Current state: 0x76cdf5b0, Next state: 0x80
[09-25 00:52:25] writeHeaderCallback(): RequestId: e203afaa-9c1e-47e6-b5e4-ee2312e34675
[09-25 00:52:25] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:52:25] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 10 current stream handle: 9068541
[09-25 00:52:25] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 10
[09-25 00:52:25] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:52:25] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 10 current stream handle: 9068541
[09-25 00:52:25] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 10
[09-25 00:52:25] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:52:25] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 10 current stream handle: 9068541
[09-25 00:52:25] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 10
[09-25 00:52:25] postWriteCallback(): postBodyStreamingWriteFunc (curl callback) invoked
[09-25 00:52:25] postWriteCallback(): Curl post body write function for stream with handle: b0ee5507ef4d31b1f778e95f3b9b65c67ed26fec7b86a4531e2d8da17ad1260e and upload handle: 10 returned: {"EventType":"BUFFERING","FragmentTimecode":1600994737365,"FragmentNumber":"91343852333201150301627800168691665192074428761"}
[09-25 00:52:25] postWriteCallback(): Processed ACK OK.
[09-25 00:52:25] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:52:25] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 10 current stream handle: 9068541
[09-25 00:52:25] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 10
[09-25 00:52:25] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:52:25] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 10 current stream handle: 9068541
[09-25 00:52:25] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 10
[09-25 00:52:25] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:52:25] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 10 current stream handle: 9068541
[09-25 00:52:25] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 10
[09-25 00:52:25] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:52:25] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 10 current stream handle: 9068541
[09-25 00:52:25] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 10
[09-25 00:52:25] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:52:25] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 10 current stream handle: 9068541
[09-25 00:52:25] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 10
[09-25 00:52:25] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:52:25] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 10 current stream handle: 9068541
[09-25 00:52:25] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 10
[09-25 00:52:25] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:52:25] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 10 current stream handle: 9068541
[09-25 00:52:25] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 10
[09-25 00:52:25] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:52:25] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 10 current stream handle: 9068541
[09-25 00:52:25] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 10
[09-25 00:52:25] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:52:26] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 10 current stream handle: 9068541
[09-25 00:52:26] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 10
[09-25 00:52:26] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:52:26] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 10 current stream handle: 9068541
[09-25 00:52:26] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 10
[09-25 00:52:26] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:52:26] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 10 current stream handle: 9068541
[09-25 00:52:26] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 10
[09-25 00:52:26] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:52:26] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 10 current stream handle: 9068541
[09-25 00:52:26] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 10
[09-25 00:52:26] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:52:26] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 14433 for upload handle: 10 current stream handle: 9068541
[09-25 00:52:26] postReadCallback(): Wrote 14433 bytes to Kinesis Video. Upload stream handle: 10
[09-25 00:52:26] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:52:26] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 0 for upload handle: 10 current stream handle: 9068541
[09-25 00:52:26] postReadCallback(): Pausing CURL read for upload handle: 10
[09-25 00:53:00] curlCompleteSync(): curl perform failed for url https://s-xxxxxxxxx.kinesisvideo.us-west-2.amazonaws.com/putMedia with result Timeout was reached: Operation too slow. Less than 30 bytes/sec transferred the last 30 seconds
[09-25 00:53:00] curlCompleteSync(): HTTP Error 0 : Response: (null)
[09-25 00:53:00] terminateCurlSession(): Force stopping the curl connection
[09-25 00:53:00] putStreamCurlHandler(): Stream with streamHandle 9068541 has exited without triggering end-of-stream.  Upload handle 11. Duration available in 100ns: 0. Bytes available: 0
[09-25 00:53:01] defaultStreamDataAvailableCallback(): Reported streamDataAvailable callback for stream handle 9068541. Upload handle 11. Duration available in 100ns: 0. Bytes available: 0
[09-25 00:53:01] stepStateMachine(): PIC Stream State Machine - Current state: 0x76cdf5b0, Next state: 0x80
[09-25 00:53:01] writeHeaderCallback(): RequestId: e5c55f34-c85e-baf0-b222-1e9ac5561090
[09-25 00:53:01] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:53:01] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 11 current stream handle: 9068541
[09-25 00:53:01] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 11
[09-25 00:53:01] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:53:01] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 11 current stream handle: 9068541
[09-25 00:53:01] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 11
[09-25 00:53:01] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:53:01] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 11 current stream handle: 9068541
[09-25 00:53:01] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 11
[09-25 00:53:01] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:53:01] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 11 current stream handle: 9068541
[09-25 00:53:01] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 11
[09-25 00:53:01] postWriteCallback(): postBodyStreamingWriteFunc (curl callback) invoked
[09-25 00:53:01] postWriteCallback(): Curl post body write function for stream with handle: b0ee5507ef4d31b1f778e95f3b9b65c67ed26fec7b86a4531e2d8da17ad1260e and upload handle: 11 returned: {"EventType":"BUFFERING","FragmentTimecode":1600994737365,"FragmentNumber":"91343852333201155253387957310212774485145176370"}
[09-25 00:53:01] postWriteCallback(): Processed ACK OK.
[09-25 00:53:01] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:53:01] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 11 current stream handle: 9068541
[09-25 00:53:01] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 11
[09-25 00:53:01] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:53:01] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 11 current stream handle: 9068541
[09-25 00:53:01] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 11
[09-25 00:53:01] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:53:01] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 11 current stream handle: 9068541
[09-25 00:53:01] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 11
[09-25 00:53:01] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:53:01] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 11 current stream handle: 9068541
[09-25 00:53:01] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 11
[09-25 00:53:01] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:53:01] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 11 current stream handle: 9068541
[09-25 00:53:01] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 11
[09-25 00:53:01] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:53:01] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 11 current stream handle: 9068541
[09-25 00:53:01] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 11
[09-25 00:53:01] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:53:01] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 11 current stream handle: 9068541
[09-25 00:53:01] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 11
[09-25 00:53:01] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:53:02] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 11 current stream handle: 9068541
[09-25 00:53:02] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 11
[09-25 00:53:02] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:53:02] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 11 current stream handle: 9068541
[09-25 00:53:02] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 11
[09-25 00:53:02] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:53:02] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 11 current stream handle: 9068541
[09-25 00:53:02] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 11
[09-25 00:53:02] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:53:02] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 16372 for upload handle: 11 current stream handle: 9068541
[09-25 00:53:02] postReadCallback(): Wrote 16372 bytes to Kinesis Video. Upload stream handle: 11
[09-25 00:53:02] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:53:02] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 14433 for upload handle: 11 current stream handle: 9068541
[09-25 00:53:02] postReadCallback(): Wrote 14433 bytes to Kinesis Video. Upload stream handle: 11
[09-25 00:53:02] postReadCallback(): postBodyStreamingReadFunc (curl callback) invoked
[09-25 00:53:02] postReadCallback(): Get Stream data returned: buffer size: 16372 written bytes: 0 for upload handle: 11 current stream handle: 9068541
[09-25 00:53:02] postReadCallback(): Pausing CURL read for upload handle: 11

Confirming. From the log indeed it shows that your applications stops/pauses producing frames causing the timeout. My suggestion above is applicable - best is to update the SDK. Alternatively, you could recreate the stream with v2.x when a new “session” starts

thank you!
We don't want to recreate the stream when a new event starts, since recreate the stream would clear endpoint cache, and cause more get data endpoint API calls. We are close to the API TPS limit.

is there a way we could achieve the same goal without clearing endpoint cache?

Updating to the latest SDK will allow automatic handling of the intermittent producer with caching endpoint. Using older sdk you could stopSync the stream and reset the stream before streaming with caching APIs. It should work fine

Good to know, thank you!


Update test utils for transformers 4.40

What does this PR do?

Updates test_utils and test_framework_agnostic according to transformers 4.40 tests.
Add the possibility of decoder_start_token_id to be a list when token_idx is not None.
Raises an AssertionError if using inputs_embeds and token_idx since input_ids will be empty and this code path crashes.
Raises an ValueError if the user requests scores when using static_shapes, no scores are computed in this case.

Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ ] Did you make sure to update the documentation with your changes?
[ ] Did you write any new necessary tests?

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@malkomes, please re-base this PR with the main. I will check it as soon you are done.

Please, note that this PR does upgrade the tests to 4.40 but it doesn't fix all the issues. Specifically, I'm expecting the following tests to fail:

Test
Comment

GenerationIntegrationTests::test_beam_search_low_memory
RuntimeError: Incompatible input shapes, broadcast not possible. Tensor1 Size: 42 1 12 1 Tensor2

GenerationIntegrationTests::test_custom_stopping_criteria
AssertionError: Lists differ: [1, 22] != [1, 20]

GenerationIntegrationTests::test_eos_token_id_int_and_list_beam_search
AssertionError: False is not true

GenerationIntegrationTests::test_eos_token_id_int_and_list_contrastive_search
AssertionError: False is not true

GenerationIntegrationTests::test_eos_token_id_int_and_list_greedy_search
AssertionError: False is not true

GenerationIntegrationTests::test_generate_vision2text_conditioning
RuntimeError: Incompatible input shapes, broadcast not possible. Tensor1 Size: 32 5 2 Tensor2 Size: 32 6 2

GenerationIntegrationTests::test_max_new_tokens_encoder_decoder
RuntimeError: Graph compile failed. synStatus=synStatus 26 [Generic failure].

GenerationIntegrationTests::test_stop_sequence_stopping_criteria
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

Since this PR is already long and I'm not 100% sure what's the best fix for some of these issues, I propose and I would be happy to further investigate them independently in different PRs.

@malkomes I am checking the issues. Can you share which tests have you done!

Not sure if I understand your ask. I can give you a list of what we have in main vs branch, is that what you want?
Here's a short summary. test_utils in main collects 48 tests where 22 failed, 17 passed, 9 skipped.
This branch has 59 tests, of which 8 failed, 26 passed, 9 skipped, 16 xfailed. I've marked fails the tests which raise NotImplementedError or have a feature that is not supported yet.
I share more details on the GS-166 ticket.

Recent changes to main branch might have made this PR obsolete.

@malkomes thanks for your work and sorry for being late in reviewing this PR.

Can you start this PR again?
If it is obsolete, can you close this one and open a new one targeted for the main?
Please ping me as soon as you start and I will make sure to review and update with your progress. Thanks

sure, @yafshar . I'm back from PTO and the huggingface:main has changed a little bit. I'm going to go with the second option of closing this one and pinging you on the next PR, I think that's going to be cleaner. Thanks


WIP: Usbmanager spec.sh

improve speed by minimizing the amount of execves in order to make developing faster
add usb devices
installer: generate usb model manifest
add parent group to model manifest (currently disabled)
ignore NICs but allow modems

putting to WIP as I have to do some more testing

I see lots of commits including fome rrverts. Does it make sense to squash them to one ir a few?

I understand that it makes it sometimes more time-intensive to read as it is more code to read, but recently I bisected several issues and both times it was a small commit (that would have nice being squashed) and it helped to revert only a small commit instead of a big one.

If we want to work with the (currently broken) zcli then the trailing "M" needs to be dropped.


Add Support for Purview GraphQL API Queries (Currently in Purview Preview Status)

Describe the feature
The GraphQL API (in Preview) allows for querying of entities and related entities, as shown in the following tutorial:
https://learn.microsoft.com/en-us/purview/tutorial-graphql-api
Having a GraphQL client and related methods for executing the API calls in https://{{endpoint}}/datamap/api/graphql
would allow for much richer retrieval of entity sets based on related entities and their attributes.
Detailed Example
Please see the tutorial at https://learn.microsoft.com/en-us/purview/tutorial-graphql-api
for examples of functionality provided by GraphQL.
Issues that this feature solves
None known
Suggested Implementation
Similarly to the pyapacheatlas.core.discovery.purview.PurviewDiscoveryClient, a ...purview.GraphQLClient could then implement the required query method for executing GraphQL queries against Purview
Additional context
See https://graphql.org/learn/ for basics of GraphQL.

@nigelmeakins thank you for this awesome suggestion!  I've implemented a bare bones support for it but would love your feedback on how to make it more useful for the budding user of GraphQL.  Please see PurviewClient.graphql.query to execute a search query.

Great, thanks very much. I'll get back to you on suggested extensions once I've taken it for a spin.


How Do I Contact Robinhood Support Number

How can I recover my Robinhood account? If you need to recover your Robinhood account, you can contact their customer service hotline at +1-(707)-852-8230 or +1-(707)-852-8230 USA for assistance in the recovery process.
How to contact Robinhood Manager? You can access live chat by clicking the "Contact Us" ☎+1-(707)-852-8230 button in the help center and then selecting "Chat." Phone: Robinhood also offers phone support for urgent issues. You can call Robinhood at ☎+1-(707)-852-8230 for Quick response & Support. There are in brief ways to talk to someone on Robinhood.
Does Robinhood have a live chat? Yes, Robinhood does offer a live chat option for customer support. To access the live chat feature:
Go to Robinhood's Help Center on the platform.
Look for the "Contact Us" ☎+1-(707)-852-8230 option.If you are trying to reach Robinhood Support +1-(707)-852-8230 (24/7), the only Robinhood phone number is ☎+1-1-(707)-852-8230 (24/7) or +1-(707)-852-8230 USA for Quick response & Support. You can also access live chat by clicking the "Contact Us" button in the help center and then selecting "Chat."
Choose the chat option to speak with a live person.
You can also contact Robinhood support at ☎+1-(707)-852-8230 (Talk with Human) & (☎+1-(707)-852-8230 (Contact-USA)).
Follow the prompts to describe your issue and get assistance.

Closing this issue as spam.


Multiple customers

Atm the program only works with one customer. We need a customer class and we need to modify the program so every customer can have its own cart.

#113 fixed this


HTTPS websites not working when chaining VPN

After the script says "Chain is connected", I noticed that when I do a cURL request to a website not using HTTPS it works but it fails when the website is using HTTPS (the cURL request keeps loading indefinitely without displaying anything).
This problem does not happen when I don't chain VPNs and test only with either the first or second VPN (which means it is not the website blocking the VPN IP address). The chain was tested with 2 VPNs only.
I can also reproduce this HTTPS problem without the script if I start the first VPN on my real Linux computer and the second VPN in an LXC container or a Linux virtual machine using VirtualBox.
Do you have an idea where the problem comes from?
I reported a similar issue here: https://superuser.com/questions/1521812/use-a-proxy-or-another-vpn-before-connecting-to-a-vpn

[I copy here my answer in S.O.]
It might be an issue of MTU. Try to lower the MTU in your OpenVPN client configuration (see OpenVPN config for the correct syntax). Usually I use 1400, but you need to try to see the higher value that works for you. Nested VPNs needs smaller values

@Enrico204 thanks, adding mssfix 1400 to the nested OpenVPN client config file fixed the issue!

Hi, 2 years have passed but I've encountered the same problem for the same reason. Https calls are not working when I connect through 2 VPNs. I was wondering if "DNS_PROBE_FINISHED_NXDOMAIN" is the same error you got. Changing the MTU to 1400 did not work.

DNS_PROBE_FINISHED_NXDOMAIN is a problem with DNS, it has nothing to do with the issue above. You should check the DNS settings


vs2022 cant build

1.txt

I don't have a windows machine I can test this on, I can only recommend you use mingw or clang for now: scons use_mingw=yes or scons use_llvm=yes

It appears to be fault of a new update to the Visual C++ compiler, version 19.38 is now violating the C++ standard that it once followed.

make no sense to me , the same code i copyed to another test project works fine.

i move the #include "SimpleGrammar.hpp" to the bottom it worked.
so i assume its the your code mess with lexy

wrong pic  this one

This only started being reported as a problem in Visual C++ 19.38, every case for which it has occurred has only been on that version of the compiler, it does not appear in GCC or Clang and neither does it appear before Visual C++ 19.38, so this is a compiler regression violating the C++ standard that was previously acceptable. Also why is there a red error squiggle following the SimpleGrammar.hpp? As well it should be corrected to #include "v2script/SimpleGrammar.hpp" honestly. But there should be no reason for what is currently in the repo to not compile, there especially should be no reason for changing where SimpleGrammar.hpp's include is should fix this issue either.

For now #33 fixes this issue, should later investigate why this is happening and a make a less hacky fix.


Flaky functest - [test_id:1779] fixing race in the Expecter

What this PR does / why we need it:
Having multiple expect.BExp one after the other without an expect.BSnd
between them is raceful.
expect.BExp is waiting for the desired expression in the current buffer
and flushes the buffer once the expression is found.
For example -
&expect.BSnd{S: "a\nb"},
&expect.BExp{R: "a"},
&expect.BExp{R: "b"}

In the good scenario we will get -
Sent: "a\nb"
Match for RE: "a" found: ["a"] Buffer:
$a
Match for RE: "b" found: ["b"] Buffer:
b

In the bad scenario we will get -
Sent: "a\nb"
Match for RE: "a" found: ["a"] Buffer:
$a
b

The second match (looking for 'b') will never be met since 'b' was already
in the buffer when the first match was found, therefore it was flushed by
the first expect.BExp.
To avoid this we need to make sure the expect.BExp is only expecting text
that was added to the buffer after the previous expectation was met.
Possible race in [test_id:1779] that is fixed by this PR -
Sent: "cat /etc/resolv.conf\n"
Match for RE: "search example.com" found: ["search example.com"] Buffer:
$ cat /etc/resolv.conf
search example.com
Match for RE: "\\$" found: ["$"] Buffer:
nameserver <IP_ADDRESS>
nameserver <IP_ADDRESS>
$
Sent: "cat /etc/resolv.conf\n"
Match for RE: "nameserver <IP_ADDRESS>" found: ["nameserver <IP_ADDRESS>"] Buffer:  cat /etc/resolv.conf
search example.com
nameserver <IP_ADDRESS>
nameserver <IP_ADDRESS>
$

Since the '$' is part of the last buffer, the next Exp expecting for it won't ever be met.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #3602
Special notes for your reviewer:
Release note:

NONE

/cc @oshoval

i would suggest to add the example from the log itself
{9 cat /etc/resolv.conf\r\nsearch example.com\r\nnameserver <IP_ADDRESS>\r\nnameserver <IP_ADDRESS>\r\n$  [nameserver <IP_ADDRESS>]}
it shows here as well that the prompt was already grabbed by this EXP
Thanks
beside that /lgtm

i would suggest to add the example from the log itself
{9 cat /etc/resolv.conf\r\nsearch example.com\r\nnameserver <IP_ADDRESS>\r\nnameserver <IP_ADDRESS>\r\n$ [nameserver <IP_ADDRESS>]}
it shows here as well that the prompt was already grabbed by this EXP
Thanks
beside that lgtm

Updated the PR message.

i would suggest to change please
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
#3602

Special notes for your reviewer:

to
Fixes #3602
and to preformat the sections of the first example
For example -
&expect.BSnd{S: "a\nb"},
&expect.BExp{R: "a"},
&expect.BExp{R: "b"}

In the good scenario we will get -
Sent: "a\nb"
Match for RE: "a" found: ["a"] Buffer:
$a
Match for RE: "b" found: ["b"] Buffer:
b

In the bad scenario we will get -
Sent: "a\nb"
Match for RE: "a" found: ["a"] Buffer:
$a
b

Thanks
/lgtm

Nice catch!
It's really tricky to get goexpect sequences right...
/lgtm
/approve
/retest

/restart

/retest

/cherry-pick release-0.30


HOSTEDCP-1046: Allow HCP Specification to Support ICSP & IDMS

What this PR does / why we need it:
Allow HCP Specification to Support ICSP & IDMS. IDMS is replacing ICSP in OCP 4.13+. hcp.Spec.ImageContentSources was updated in OCPBUGS-11939 to replace ICSP with IDMS; however, HCP needs to support both ICSP & IDMS. Note, OpenShift clusters can only have either ICSP CRs or IDMS CRs; clusters cannot utilize both CR types at the same time.
Which issue(s) this PR fixes:
Fixes HOSTEDCP-1046
Checklist

[x] Subject and description added to both, commit and PR.
[x] Relevant issues have been referenced.
[ ] This change includes docs.

/jira refresh

/test e2e-aws

so just to be clear so I understand:
Currently we set the old path on 4.13: that would still work as is and have the desired behavior of laying down the proper mirror config
Then on 4.14 since "both cannot be done at once" we would simply move to updating our hostedcluster struct to only have the new path
And it will handle downstream removing the old field in the HCP and adding the new field in the HCP?

so just to be clear so I understand: Currently we set the old path on 4.13: that would still work as is and have the desired behavior of laying down the proper mirror config
Then on 4.14 since "both cannot be done at once" we would simply move to updating our hostedcluster struct to only have the new path
And it will handle downstream removing the old field in the HCP and adding the new field in the HCP?

You can still use ICSPs in 4.14. I don't think they are being deprecated until 4.17. You cannot have CR instances of both ICSP and IDMS in your cluster though.

/test e2e-aws

/test e2e-aws

/approve

/cherry-pick release-4.13

/lgtm

/test e2e-aws

/test e2e-aws

/test e2e-aws

/retest


Iterate account nft pages without using successor

NFTs are iterated in reverse order, starting from the max page, working towards the min page.
fetchSuccessorKey will not work if the input key does not exist, and clio's cache is not full. The alternative approach to this is to insert dummy entries into the successor table, but this is quite tedious. Instead, since the max page is guaranteed to exist if an account has any NFTs, we start from the max page and iterate backwards.

This change is

I changed the marker to be a pair of values, so that way we can jump directly to the nft page we need to continue iterating from

I mean, doing what you suggest still makes it a linear algorithm so it's not terrible. Originally, CJ had this backwards by page and backwards by token, but I asked him to change the latter to forwards for code readability. Do you think a consistently backwards approach is better than this one?


DDNS add and remove items.

It's still the same after I restore the default settings.

@jow- Amazing efficiency. 😋


Feat 49 upgrade to aind-data-schema 1.0

close #49
Upgrades the pins on this repo to support aind-data-schema version 1.0.0.
Also upgrades Pydantic pin to >= 2.7, which caused some issues with the usage of construct_model. This led to converting all Upgraders to intake a dict, rather than a model. Functionality has not otherwise changed.

Tested in the ephys pipeline and it works :)


Long standing py_lru_cache bug.

Hello and first of all thanks for making this library.
I have a question/suggestion and I was wondering if noone ever ran into this issue.
There is a longstanding bug in py_lru_cache that has not been fixed by now:
https://github.com/stucchio/Python-LRU-cache/pull/18
We went around by pinning to: 'py_lru_cache @ git+https://github.com/kirill578/Python-LRU-cache.git'

Thanks. I've seen that a couple of times but couldn't reproduce it reliably. I'll bump after a release of the lib.

I've decided to just move away from that package since it seems unmaintained.

I've decided to just move away from that package since it seems unmaintained.


DAP seems not work with 2 gpus

Environment: docker, cudatoolkit11.3, A100, triton, inference data do not has a batch dimension.
No matter how long the sequence, 908M memory used on each gpu, and 100% gpu util. But seems never finished.

Hi, have you tried to terminate this process(e.g. using Ctrl C) to see the operation blocked?
And would you please show us your run.sh? If you can paste your sequence as well, we would try to reproduce this bug.
Meanwhile, triton needs cuda11.4, it seems unreasonable to run it with cudatoolkit11.3.
DAP doesn't need a batch dimension.

Thx for replay. Here is the ctrl+c message:

Here is my run.sh:
complex_id=CM012-1
python inference_with_features_pkl.py
--output_dir ./output/
--output_prefix $complex_id
--model_name model_1_multimer_v3
--model_preset multimer
--gpus 2
--inplace
--chunk_size 10
--param_path /data/ab01/gongchaohui/B30/20230301_alphafold-tools/alphafold/data/params/params_model_1_multimer_v3.npz
/data/ab03/gongchaohui/B30/20221205_EpitopePipeline/output/180-rerun/$complex_id/save/$complex_id/features.pkl
I'm using a features.pkl which contain all input features to alphafold, I test it to single gpu and work well. Also old kernel(not using triton) test, still has the same problem. I'll try cuda 11.4 and report the test result later.

Hi, I found it strange that there is colossalai init information in your log.
I am not sure if there is something wrong in your inference_with_features_pkl.py. I guess you use colossalai launch before in your script.

# Copyright 2021 AlQuraishi Laboratory
# Copyright 2021 DeepMind Technologies Limited
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import argparse
import os
import random
import sys
import time
import tempfile
import contextlib

import numpy as np
import torch
import torch.multiprocessing as mp
import pickle
from fastfold.model.hub import AlphaFold

import fastfold
import fastfold.relax.relax as relax
from fastfold.common import protein, residue_constants
from fastfold.config import model_config
from fastfold.model.fastnn import set_chunk_size
from fastfold.model.nn.triangular_multiplicative_update import set_fused_triangle_multiplication
from fastfold.data import feature_pipeline

from fastfold.utils.inject_fastnn import inject_fastnn
from fastfold.utils.import_weights import import_jax_weights_
from fastfold.utils.tensor_utils import tensor_tree_map

if int(torch.__version__.split(".")[0]) >= 1 and int(torch.__version__.split(".")[1]) > 11:
    torch.backends.cuda.matmul.allow_tf32 = True

def seed_torch(seed=1029):
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True
    torch.use_deterministic_algorithms(True)

@contextlib.contextmanager
def temp_fasta_file(fasta_str: str):
    with tempfile.NamedTemporaryFile('w', suffix='.fasta') as fasta_file:
        fasta_file.write(fasta_str)
        fasta_file.seek(0)
        yield fasta_file.name

def add_data_args(parser: argparse.ArgumentParser):
    parser.add_argument('--chunk_size', type=int, default=None)
    parser.add_argument('--inplace', default=False, action='store_true')

def inference_model(rank, world_size, result_q, batch, args):
    os.environ['RANK'] = str(rank)
    os.environ['LOCAL_RANK'] = str(rank)
    os.environ['WORLD_SIZE'] = str(world_size)
    # init distributed for Dynamic Axial Parallelism
    fastfold.distributed.init_dap()
    torch.cuda.set_device(rank)
    config = model_config(args.model_name)
    if args.chunk_size:
        config.globals.chunk_size = args.chunk_size

    if "v3" in args.param_path:
        set_fused_triangle_multiplication()

    config.globals.inplace = args.inplace
    config.globals.is_multimer = args.model_preset == 'multimer'
    model = AlphaFold(config)
    import_jax_weights_(model, args.param_path, version=args.model_name)

    model = inject_fastnn(model)
    model = model.eval()
    model = model.cuda()

    set_chunk_size(model.globals.chunk_size)

    with torch.no_grad():
        batch = {k: torch.as_tensor(v).cuda() for k, v in batch.items()}

        t = time.perf_counter()
        out = model(batch)
        print(f"Inference time: {time.perf_counter() - t}")

    out = tensor_tree_map(lambda x: np.array(x.cpu()), out)

    result_q.put(out)

    torch.distributed.barrier()
    torch.cuda.synchronize()

def main(args):
    if args.model_preset == "multimer":
        inference_multimer_model(args)
    else:
        inference_monomer_model(args)

def inference_multimer_model(args):
    print("running in multimer mode...")
    config = model_config(args.model_name)
    output_dir_base = args.output_dir
    output_prefix = args.output_prefix
    random_seed = args.data_random_seed
    if random_seed is None:
        random_seed = random.randrange(sys.maxsize)
    # seed_torch(seed=1029)

    feature_processor = feature_pipeline.FeaturePipeline(
        config.data
    )

    if not os.path.exists(output_dir_base):
        os.makedirs(output_dir_base)

    # 读取输入特征文件 features.pkl
    with open(args.input_feature_path, 'rb') as f:
        feature_dict = pickle.load(f)
    # 处理序列长度特征
    num_res = feature_dict['aatype'].shape[0]
    feature_dict['seq_length'] = np.asarray(
        [num_res] * num_res,
        dtype=np.int32
    )
    processed_feature_dict = feature_processor.process_features(
        feature_dict, mode='predict', is_multimer=True,
    )

    batch = processed_feature_dict

    manager = mp.Manager()
    result_q = manager.Queue()
    torch.multiprocessing.spawn(inference_model, nprocs=args.gpus, args=(args.gpus, result_q, batch, args))

    out = result_q.get()
    out['asym_id'] = feature_dict['asym_id']
    if args.save_prediction_result:
        # Save the prediction result .pkl
        print(f'Saving prediction result to {args.output_dir}..')
        prediction_result_path = os.path.join(args.output_dir,
            f'{output_prefix}_{args.model_name}.pkl')
        with open(prediction_result_path, 'wb') as f:
            pickle.dump(out, f)

    # Toss out the recycling dimensions --- we don't need them anymore
    batch = tensor_tree_map(lambda x: np.array(x[..., -1].cpu()), batch)

    plddt = out["plddt"]
    mean_plddt = np.mean(plddt)

    plddt_b_factors = np.repeat(plddt[..., None], residue_constants.atom_type_num, axis=-1)

    unrelaxed_protein = protein.from_prediction(features=batch,
                                                result=out,
                                                b_factors=plddt_b_factors)

    # Save the unrelaxed PDB.
    unrelaxed_output_path = os.path.join(args.output_dir,
        f'{output_prefix}_{args.model_name}_unrelaxed.pdb')
    with open(unrelaxed_output_path, 'w') as f:
        f.write(protein.to_pdb(unrelaxed_protein))

    if(args.relaxation):
        print('Relaxing...')
        amber_relaxer = relax.AmberRelaxation(
            use_gpu=True,
            **config.relax,
        )

        # Relax the prediction.
        t = time.perf_counter()
        relaxed_pdb_str, _, _ = amber_relaxer.process(prot=unrelaxed_protein)
        print(f"Relaxation time: {time.perf_counter() - t}")

        # Save the relaxed PDB.
        relaxed_output_path = os.path.join(args.output_dir,
            f'{output_prefix}_{args.model_name}_relaxed.pdb')
        with open(relaxed_output_path, 'w') as f:
            f.write(relaxed_pdb_str)

def inference_monomer_model(args):
    print("running in monomer mode...")
    config = model_config(args.model_name)

    output_dir_base = args.output_dir
    output_prefix = args.output_prefix
    random_seed = args.data_random_seed
    if random_seed is None:
        random_seed = random.randrange(sys.maxsize)
    # seed_torch(seed=1029)

    # 读取输入特征文件 features.pkl
    with open(args.input_feature_path, 'rb') as f:
        feature_dict = pickle.load(f)
    feature_dict['seq_length'] = np.array([feature_dict['seq_length']])
    feature_processor = feature_pipeline.FeaturePipeline(config.data)
    processed_feature_dict = feature_processor.process_features(
        feature_dict,
        mode='predict',
    )

    batch = processed_feature_dict

    manager = mp.Manager()
    result_q = manager.Queue()
    torch.multiprocessing.spawn(inference_model, nprocs=args.gpus, args=(args.gpus, result_q, batch, args))

    out = result_q.get()

    # Toss out the recycling dimensions --- we don't need them anymore
    batch = tensor_tree_map(lambda x: np.array(x[..., -1].cpu()), batch)

    plddt = out["plddt"]
    mean_plddt = np.mean(plddt)

    plddt_b_factors = np.repeat(plddt[..., None], residue_constants.atom_type_num, axis=-1)

    unrelaxed_protein = protein.from_prediction(features=batch,
                                                result=out,
                                                b_factors=plddt_b_factors)

    # Save the unrelaxed PDB.
    unrelaxed_output_path = os.path.join(args.output_dir,
                                            f'{output_prefix}_{args.model_name}_unrelaxed.pdb')
    with open(unrelaxed_output_path, 'w') as f:
        f.write(protein.to_pdb(unrelaxed_protein))

    if(args.relaxation):
        amber_relaxer = relax.AmberRelaxation(
            use_gpu=True,
            **config.relax,
        )

        # Relax the prediction.
        t = time.perf_counter()
        relaxed_pdb_str, _, _ = amber_relaxer.process(prot=unrelaxed_protein)
        print(f"Relaxation time: {time.perf_counter() - t}")

        # Save the relaxed PDB.
        relaxed_output_path = os.path.join(args.output_dir,
                                            f'{output_prefix}_{args.model_name}_relaxed.pdb')
        with open(relaxed_output_path, 'w') as f:
            f.write(relaxed_pdb_str)

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "input_feature_path",
        type=str,
    )
    parser.add_argument(
        "--output_dir",
        type=str,
        default=os.getcwd(),
        help="""Name of the directory in which to output the prediction""",
    )
    parser.add_argument(
        "--output_prefix",
        type=str,
        help="""Prefix for output files."""
    )
    parser.add_argument("--model_name",
                        type=str,
                        default="model_1",
                        help="""Name of a model config. Choose one of model_{1-5} or
             model_{1-5}_ptm or model_{1-5}_multimer, as defined on the AlphaFold GitHub.""")
    parser.add_argument("--param_path",
                        type=str,
                        default=None,
                        help="""Path to model parameters. If None, parameters are selected
             automatically according to the model name from
             ./data/params""")
    parser.add_argument(
        "--relaxation", action="store_true", default=False,
    )
    parser.add_argument("--gpus",
                        type=int,
                        default=1,
                        help="""Number of GPUs with which to run inference""")
    parser.add_argument('--save_prediction_result',
                        type=bool,
                        default=True)
    parser.add_argument('--data_random_seed', type=str, default=None)
    parser.add_argument(
        "--model_preset",
        type=str,
        default="monomer",
        choices=["monomer", "multimer"],
        help="Choose preset model configuration - the monomer model, the monomer model with "
        "extra ensembling, monomer model with pTM head, or multimer model",
    )
    add_data_args(parser)
    args = parser.parse_args()

    main(args)

This is my script, modified from inference.py. I thought the log from init_dap.

update cuda to 11.6 and pytorch to 1.13, still has the problem. Seems using more memory at the initial state.

After some debugging, I found the process stop at this line code.

Hi, I can't reproduce your problem.
Can you provide more information of your environment, such as colossalai version and your communication backend?

After lengthy troubleshooting and investigation, I finally discovered that the issue may have been caused by enabling IOMMU. However, I wasn't sure if disabling IOMMU would affect the use of Docker. Later, I found that setting the environment variable NCCl_P2P_DISABLE=1 could resolve this issue. But I am unsure if this environmental variable setting would cause other issues for distributed training.


Implement controller rumble support to ui/Gamepad

This PR adds a controller rumble method to the ui/Gamepad class.
At this time, it only implements the native/SDL2 backend. HTML5 should be simple as well, although browser support for rumble is lacking as of writing this.
Closes #869

I've tested this as working!

many of the whitespacing/formatting issues should be fixed up now @player-03 and the changes should be easier to parse through and read!

Any updates on this?

im not op (but I paid him for this mini series of PRs!) but were there any particular blockers to this gamepad rumble PR?

Yeah, it needs to go into the 8.3-dev branch not develop. I'll try to test and see about merging it before the end of the weekend.

I'm still not convinced the operator overload is the correct implementation, but I'm glad to have someone else look at it.


Router parsing of fragment hash causes information to be lost or misinterpreted

The general code setup:
Backbone.Router.extend({
routes: {
'*query': 'onQueryRoute'
},
initialize: function() {
Backbone.history.start();
},
onQueryRoute: function(query) {
// "query" is impossible to reliably parse as a query string
// due to full-string url-decode on line 1306 of backbone.js
// inside of _extractParameters().
}
});
Here's the scenario. Imagine I have a single query parameter with an arbitrary value that happens to contain "1&bar=2".
So I construct my fragment string like so:
var value = encodeURIComponent("1&bar=2");
router.navigate("foo=" + value);
When I refresh the page and my onQueryRoute event handler gets called, I will receive this string: "foo=1&bar=2", which is obviously impossible to parse now because it can be interpreted as two
query parameters when it should actually only be one. This is happening because on line 1306 inside of _extractParamters(), you are url-decoding the whole fragment hash without splitting it first.
In my particular use-case, the query parameters are not fixed; there can be an arbitrary number of them so I can't easily set a route regex that automatically parses the individual values. The structure of my URL is like this: key1[0]=value1&key2[0]=value2&key1[1]=...&foo=bar&baz=blah
The execute() preprocessor method is useless here because it gets called after _extractParameters() has mangled the hash string by url-decoding it whole without splitting it first by default. My current workaround is to override the method with this:
_extractParameters: function(route, fragment) {
return Backbone.Router.prototype._extractParameters
.call(this, route, encodeURIComponent(fragment));
}
I basically url-encode it again before passing it back to the original implementation so that when I get it in my callback, I get the original version that I can proceed to parse normally. This is not very efficient and I'd like to see this fixed properly in the source.
Thoughts?

This is not working for me with Backbone 1.3.3.
I'm using Backbone.history.navigate("?q=" + encodeURIComponent(searchQuery)) to change the URL when submitting a search form.
In my case, searchQuery could be something like "max müller", so I encode it with encodeURIComponent into "max+m%C3%BCller". But within Backbones navigate function, my searchQuery is beeing decoded back to "max müller" and returned like that.
So at the end the URL gets changed into "myroot.com/?q=max müller", which is not a valid URL. Instead it should be like this: "myroot.com/?q=max+müller".
Am I missing something? This should be the standard behavior or not? Is there a way to achieve it beside modifying Backbones core?


Update plugin.xml

Hi there,
I have edited some of configuration. Would it be updated soon?

Hi,
it will be updated in the next version (hopefully after 2 weeks)


#686 Add C++ compiler check & update build instructions

/usr/bin/g++ not being installed causes a couple of issues:

env['CXX'] will be None, and the version check will throw an exception
third_party g++ has /usr/bin/g++ hard-coded (e.g. in ./src/NATIVE-REL-x86_64/third_party/bin/g++)

Also, on a clean CentOS 6 install, the gtest RPM will not build because of a missing dependency.
I updated the documentation to add the default development tools, and also put a compiler check into the SConstruct file.
As long as all these requirements are met, libpng is detected and there are no problems building; I tested on both CentOS 6.8 and CentOS 6.10.

I signed it!

Does this PR fixes #1103 since #686 could not reproduced any more ?

There need to be some documentation updates for #1103, and a re-test for a debug build.
I think #686 involves SCons caching results about what libraries have been found.  I doubt that this pull request fixes it, but, it's worth re-testing.

I was unable to reproduce the bug mentioned in #686 as well. That issue may have been fixed elsewhere.
This branch does address most, if not all, of the concerns raised in #1103, though. Maybe we should re-name this branch to reference that issue?

I don't believe caching was a factor here.  The build script had no issues detecting libpng whether or not that build step was cached.
I'd be happy to rename the branch -- it does seem like there's a lot of overlap with #1103.

I'll leave the decision to rename the branch up to @tst-ccamp. Even if it doesn't happen I think we can sort out what happened.


Problem with Ip Camera

I trying to get the stream from an ip camera. So when i pass the follwing url http://<IP_ADDRESS>:6021/cgi-bin/cmd/encoder?GET_STREAM it ask fot user and password. So, when i entry the credentials it crashes showing the following exception
StatusCode: 200, ReasonPhrase: 'OK', Version: 1, Content: Windows.Web.Http.HttpStreamContent, Headers:
{
Connection: Keep-Alive
Cache-Control: no-cache
Pragma: no-cache
}{
Content-Type: multipart/x-mixed-replace;boundary=----------------boundary
}
Exception: 'System.InvalidOperationException' en mscorlib.ni.dll
Do you know how it could works with these device?

Can you share the callstack of the exception and the HRESULT property value if available?

It appears this message {"Problema de seguridad. (Exception from HRESULT: 0x800C000E)"}

StatusCode: 200, ReasonPhrase: 'OK', Version: 1, Content: Windows.Web.Http.HttpStreamContent, Headers:
{
Connection: Keep-Alive
Cache-Control: no-cache
Pragma: no-cache
}{
Content-Type: multipart/x-mixed-replace;boundary=----------------boundary
}


karma watch tends to fail with large streamed bundles

i currently use gulp to produce two large JS bundles from TypeScript with browserify and factor-bundle.  One bundle has all the main code and the other has all the specs.
The problem is when I use watch mode and update the bundles, karma detects two file change events on both files.  I notice karma picks up the first change where the file is empty.  This is confirmed by doing a readFileSync around here.  Additionally, the syntax error message from the test execution also confirms this (i.e. Unexpected end of input).
Setting autoWatchBatchDelay to a higher number doesn't mitigate this problem.  There is usually a 0.5 second delay between the first change event (empty file) and the second one (file generated).  I'm not exactly sure what's going on there (maybe there's a race condition somewhere?)  I remember adding a console.log(self.files) somewhere in the aforementioned section of code and since the output is very long, it takes time to flush to the terminal and the problem stops occurring (however waiting a long time is not a very productive activity!)
By enabling awaitWriteFinish in chokidar, the problem is mitigated.  Maybe this is the solution?
https://github.com/karma-runner/karma/compare/master...khoomeister:ck-awaitwritefinish#diff-3c250c1426c199f7eccf17c774f0adfaR84
It'd be great to find a permanent fix for this as this is vital to our TDD workflow.

I don't think we should enable this by default, as it has some considerable drawbacks, as mentioned in  https://github.com/paulmillr/chokidar. But it would be good to expose as an option to be configurable.

I did a bit of digging into this as I hit the same problem with ClojureScript compilation. As best as I can tell, what's happening is:

Chokidar sets up a watcher for file change events
A file is changed by compilation. The file changes are streamed in. Chokidar's docs mention that a 'change' event can be emitted before a file has finished being changed.
Chokidar picks up the change and calls the fileList.changeFile callback
In the changeFile callback, in part of the promise chain, return self._preprocess(file) is called.
The preprocessor runs. If there is no preprocessor, an identity transform runs
The processed file is saved somewhere for serving
The rest of the file changes are saved
No further events are called
When the file is requested, the partially written version is served


[Shadowbane Inquisition] dark mode fix. again

Submission Checklist
Required

[x] The pull request title clearly contains the name of the sheet I am editing.
[x] The pull request title clearly states the type of change I am submitting (New Sheet/New Feature/Bugfix/etc.).
[x] The pull request makes changes to files in only one sub-folder.
[x] The pull request does not contain changes to any json files in the translations folder (translation.json is permitted)

Changes / Description

fixed darkmode by adding !importants tags everywhere

Character Sheet Info Roll20 Internal Use only.


update SelectorField to remove special typing for $contains queries

What type of PR is this? (check all applicable)

[ ] Refactor
[ ] Feature
[x] Bug Fix
[ ] Optimization
[ ] Documentation Update

Description
Remove special typing for $contains in SelectorField type alias
Related Tickets & Documents

Related Issue #331
Closes #331

Added/updated tests?

[ ] Yes
[x] No, and this is why: please replace this line with details on why tests
have not been included
[ ] I need help with writing tests

Is this change backwards compatible?

[x] Yes
[ ] No, and this is why: please replace this line with details on why?

Does it require updates to Tigris docs?

[ ] Yes, and here is the link: please create an issue in tigris-docs repo
and replace this text as tigrisdata/tigris-docs#123
[x] No

Checklist

[x] npm run build - builds successfully
[x] npm run test - tests passing
[x] npm run lint - no lint errors

[optional] Are there any post deployment tasks we need to perform?

Codecov Report
Patch coverage: 100.00% and project coverage change: -0.02 :warning:

Comparison is base (d5d0794) 90.21% compared to head (410b86b) 90.20%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #332      +/-   ##
==========================================
- Coverage   90.21%   90.20%   -0.02%
==========================================
  Files          29       29
  Lines        5591     5585       -6
  Branches      652      652
==========================================
- Hits         5044     5038       -6
  Misses        546      546
  Partials        1        1

Impacted Files
Coverage Δ

src/types.ts
91.33% <100.00%> (-0.07%)
:arrow_down:

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Chore: Use tool-versions instead of node-version

This project was using the .node-version file to specify the version of node to be used. This seems to be deprecated, in favour of using .tool-versions. (https://github.com/asdf-vm/asdf-nodejs?tab=readme-ov-file#nvmrc-and-node-version-support)
Although the old file should still be supported, it did not work for me locally even after doing the workaround. We therefore decided to implement the new .tool-versions file. It has already been created in a different PR. This PR is just to update the github actions with this change, and to delete the .node-version file.

@manuelpuchta FYI

Nice, thanks @Deeds67


[FEATURE] Add memory management mechanisms to regression trees and streamline generic code in tree-based estimators

Summary
This PR primarily addresses #128, i.e., adds model size constraining capabilities to Hoeffding Tree Regressor (HTR) algorithms. The proposed memory control mechanism is based on the depth of the learning nodes. Contrary to the last hypothesis mentioned in issue #128, and after discussing this matter with the development team, using the nodes' depth proved to be a simpler yet effective approach. Thank you, @jacobmontiel and professor Bernhard Pfahringer, for your advice on this new functionality. :-)
Nonetheless, to effectively perform the needed changes, some aspects of the current implementation had to be firstly addressed. Also, during the development, some bugs were found and also fixed.

The complete list of changes is presented following:

Addresses #128. For this end, now the FoundNode class (optionally) carries information about the depth of its encapsulated tree node
Make trees' constants internal and private
Remove unused imports
Standardize the Multi-target regression nodes to follow the same pattern used by their single-target counterparts
Streamline Active/Inactive learning node creation in HTs. This action enables us to use fewer lines of code and leverage base classification/regression memory management mechanisms across different tree algorithms
Rename test files and methods to comply with skmultiflow's standards
Add a specialized method to reactivate learning nodes in HTRs
Remove redundant methods from ExtremelyFastDecisionTreeClassifier and LabelCombinationHoeffdingTreeClassifier
Add coverage tests for tracking memory management in classification and regression Hoeffding Trees
Fix an instance normalization bug present in HTR/HATR. Currently, the instances are not normalized when the nominal_attributes parameter is None.
Make the random behavior of the local bagging approach used in HAT/HATR's learning nodes be controlled by a random_state parameter. This change is intended to increase reproducibility and controlled variability of results across repeated executions of the algorithms
Fix pointer problem regarding the perceptron' weights inherited from previous learning nodes. In other words, force new learning nodes to perform a deep copy of their ancestors' weights so that they do not access the same region in memory. In some cases a weight normalization step has to be performed to enforce convergence.
Enforce PEP8 convention for the modified scripts (here I'm using a custom E501 setup, as discussed in #221)

Checklist
Implementation

[x] Implementation is correct (it performs its intended function).
[x] Code complies with PEP-8 and is consistent with the framework.
[x] Code is properly documented.
[x] PR description covers ALL the changes performed.
[x] Files changed (update, add, delete) are in the PR's scope (no extra files are included).

Tests

[x] New functionality is tested.
[x] Tests are created for the new functionality or existing tests are updated accordingly.
[x] ALL tests pass with no errors.
[x] CI/CD pipelines run with no errors.
[x] Test Coverage is maintained (coverage may drop by no more than 0.2%).

I did some tests with the HTRs and generated some plots to assess the correct functioning of the new feature. In all the cases, I used instances of the RegressionGeneration for the tests and kept most of the trees' parameters constant and with their default values.

Next, I present the obtained results:

HTR model constrained to 10 MB (memory checks every 200 instances)

HTR model constrained to 40 MB (memory checks every 200 instances)

HTR model constrained to 40 MB (memory checks every 1000 instances)

HTR model constrained to 50 MB (memory checks every 10K instances)

iSOUP-Tree model constrained to 5 MB (3 targets, memory checks every 200 instances)

SST-HT model constrained to 5 MB (3 targets, memory check intervals every 200 instances)

Thanks for the PRM @smastelini

I have left some comments in the code.

Hi Jacob, thank you for your revision. I will explain my reasoning for both points.

* I didn't follow the change about the deep copy in learning nodes. Can you direct me to that change?

Nice catch! Let me explain my reasoning. When a split occurs and leaf_prediction='perceptron', the new nodes are initialized with the weights of their parent nodes. This action is performed to avoid training the linear predictors from scratch. The weights are instances of numpy.ndarray. In the previous implementation, the child nodes simply assigned their ancestor's weights to their own perceptron_weight variables, i.e., they made a shared reference to a region in the memory. As a result, different nodes ended up updating the same set of weights. In the automatic tests, this action resulted in smaller errors, but I believe that this occurs because RegressorGenerator basically creates linear regression problems. To avoid that, I forced the nodes to create copies of their ancestor's weights, and thus, just learn from the samples lying within the subspace they are located within.

* I saw that there are now multiple tests for memory management. Do we really need to replicate it? I would expect that inherited methods are tested once. I like the idea of checking that they work, but we must also consider the overhead in testing.

This is another valid point. I agree with you. The only part of the code that is not replicated is the _new_learning_node method. This specific portion is responsible for yielding the active and inactive nodes. My main reason for creating multiple tests was to cover _new_learning_node. Do you have any suggestions? Since this method in question is pretty much straightforward, we could let parts of it without coverage (basically, the portion referring to the inactive learning nodes).

To give more context about the bug fixes mentioned in the main PR's description, I am going to detail their aspects next. These fixes resulted in changes in the automatic tests and the observed prediction errors of the trees.

1. Lack of a deep copy operation on the perceptron weights (affects iSOUP-Tree and StackedSingleTargetHoeffdingTreeRegressor)
When creating new perceptron nodes, HoeffdingTreeRegressor initializes the new leaf learners with the same weights of their ancestor:
https://github.com/scikit-multiflow/scikit-multiflow/blob/563187dc850f51075e19e70817a39706d5ea4ac5/src/skmultiflow/trees/nodes/active_learning_node_perceptron.py#L36-L39
The same action does not effectively happen in the multi-target regression tree algorithms. Take as an example, iSOUP-Tree:
https://github.com/scikit-multiflow/scikit-multiflow/blob/563187dc850f51075e19e70817a39706d5ea4ac5/src/skmultiflow/trees/nodes/active_learning_node_perceptron_multi_target.py#L34
Without forcing a deep copy of the weights, multiple learning nodes are effectively updating the same weight matrix. In this PR, I enforced a deep copy operation in the multi-target regression learning nodes. Moreover, their initialization parameters were modified to follow the same pattern used in their single-target counterparts.

2. Instance normalization bug (affects HoeffdingTreeRegressor and HoeffdingAdaptiveTreeRegressor)
Instances are normalized before being processed by the perceptron models at the leaves. Nevertheless, categorical attributes are not changed, given their constrained and discrete nature. This action is implemented in HoeffdingTreeRegressor's method normalize_sample:
https://github.com/scikit-multiflow/scikit-multiflow/blob/563187dc850f51075e19e70817a39706d5ea4ac5/src/skmultiflow/trees/hoeffding_tree_regressor.py#L234-L242
Nonetheless, the if statement does not correctly handle the situation where all the input features are numeric, i.e., _nominal_attributes=None. In this PR I have modified the previous if to the following form:
if (self._nominal_attributes is None or (self._nominal_attributes is not None and
                                         i not in self._nominal_attributes)) and \
        self.samples_seen > 1:

Moreover, a new weight normalization step was added to ensure convergence, in the same form as performed in the original MOA's implementation of iSOUP-Tree:
# Normalize perceptron weights
self.perceptron_weight = self.perceptron_weight / np.sum(self.perceptron_weight)

Without it, the observed prediction errors became much higher than the current ones in the automatic tests.

Thanks @smastelini for the further discussion and explanation of the 2 main bugs addressed here. It is clearer now.
Regarding the tests, I saw the running time of the Azure pipelines and on average it does not seem to add too much overhead. However, we could consider mixing the memory test with some of the existing tests, just to avoid re-processing the stream again. What do you think?

Regarding the tests, I saw the running time of the Azure pipelines and on average it does not seem to add too much overhead. However, we could consider mixing the memory test with some of the existing tests, just to avoid re-processing the stream again. What do you think?

Hi @jacobmontiel, thank you for the suggestions. I have streamlined the memory tests. In fact, I have even found some redundancy among the current checks. Hence, I removed the specific memory tests in favor of enhanced coverage tests (I actually updated the coverage-suffixed ones). Some of the existing classification tests already covered parts of the memory management mechanism. Nonetheless, they previously did not check for the final size of the trained models. Now, all the memory management-based checks follow the same standard, regardless of the tree algorithm.
I also reduced the number of needed stream instances for checking the memory management mechanisms. The tests now run faster than before and keep the same coverage.
Since most of the HTRs already have a specific test to cover categorical attributes, I was able to remove two "reference" data files (regression_data.npz and multi_target_regression_data.npz) without any impacts on the existing coverage. What do you think of that approach?


Extend Docker build to allow for a folder called prerun_hook_files

If the folder exists in cwd, will be copied alongside the prerun_hook.sh script. Files inside will then be available to the script (from /app/prerun_hook_files).
References issue #4417
By putting certificates in this folder, I can use the Java keytool to add them to the store in the prerun_hook.sh script. These certificates are then successfully used when connecting to a Mongo database using ssl.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Closed this as it was against master, noticed that it is supposed to be against alpha.

Ok, not really sure about the alpha branch, all my testing has been against master, so reopening this pull request here.

Feel free to ignore this, as it probably makes more sense to do this (getting certs in) via building an entirely new Docker image based on the prisma one. I'm a bit new to Docker, so didn't see the obvious solution at first.


Dragging over and dropping to drop targets don't work anymore

It seems like because the code for figuring out the matching targets was commented out: https://github.com/yahoo/react-dnd-touch-backend/blob/master/src/Touch.js#L259
And browsers don't fire touchmove/touchend events on a node if the touch was not originated on the node.
Rolling back to 0.2.7 solves the issue.

cc @DetweilerRyan

Also seeing this issue

Yeap, if dropTarget overlapped by children its not work;

I'm not sure I understand, is this issue that dragging and dropping to any drop target, period, won't work? Basically making the current version of the touch backend unusable? Or is it a more specific case. Thanks in advance

Currently, version 0.3.0 only works for mouse events and not touch events.

Gotcha, thank you very much for the quick response! Very much appreciate this library, since I don't need nested children will try 0.2.7

Yes, I recommend using 0.2.7.

Seems to work with 0.3.2

0.3.2 works for me as well 👍

Seems to be working :smile:

i see the same issue in the latest version 0.3.21 rolling back all the way to 0.2.7 solved the issue. The hover only triggered when the mouse left the preview when dragging quickly.


Consider using March Hare on JRuby

While Hutch works well on JRuby today with recent Bunny versions, one would get several times better throughput if March Hare was used when running on JRuby.
Worth considering, given that MH and Bunny API's are really close.

Hi,
Related to https://github.com/ruby-amqp/bunny/pull/328#issuecomment-127007716, would you need any help implementing jruby driver using March Hare ?

@teodor-pripoae feel free to look into it, that would be appreciated!

Fixed in master.


record debug code for fix block_attn

把调试block_attn精度问题的代码放在这个pr中，方便后续调试需要

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.