Store only hash tables in memory

As far as I understand, the underlying algorithm doesn't require storing the dataset in memory, only the generated hash tables.
Given the favorable computational complexity, falconn is very interesting to use with large datasets, but then storing whole datasets in memory becomes either infeasible or very expensive.
All in all, I'm not sure where the requirement of storing the whole dataset comes from – is it an inherent limtation of the method or was it just not implemented yet (and if so, are there plans to?)

That's a good point. If you only want to know the candidate set and not the nearest neighbors among the candidates, FALCONN does not require access to the data set. Currently we do not explicitly support this feature, but if you only use the get_candidates_with_duplicates() and get_unique_candidates() methods, FALCONN should not try to access the point set.
In the future, we will make the distinction between candidate generation and nearest neighbor computation more clear.

Just to confirm, is it the case that FALCONN never copies the data even when used via python bindings? In my experience, the memory consumption was essentially doubling at the start of indexing, and then increasing a bit further when hash tables were constructed – and changing the number of bits per hash and the number of hash tables seemed to suggest this is what was happening.

The C++ library never copies data. For the Python bindings, the story is a bit more complicated. There are two levels of wrapping going on in the Python bindings:
A: The inner wrapper is using swig. The swig numpy bindings should not make a copy. But I'm not 100% certain about this. For instance, when the input to the swig wrapper is not a flat memory array internally, maybe the swig numpy wrapper creates a copy. Is your data a flat memory array internally, i.e, a simply NumPy array?
B: The outer wrapper is just a thin Python wrapper around the swig bindings. Its goal is to improve usability. The thin wrapper stores a reference to the dataset to avoid cases where the garbage collector removes the dataset and the user assumes FALCONN would still work (this simply leads to a segmentation fault).
We haven't really optimized the Python bindings for memory efficiency so far, partly because we assumed that people would use the C++ bindings for memory-critical settings.
Would the C++ library be an option for you? If you have ideas for how to prevent memory copies in the Python bindings, do let us know!

Thanks for your reply, we'll investigate using c++ directly vs some basic python binding that doesn't keep the copied data.
That opens the question about saving/loading hash tables, since the workflow would be to create the hash tables on a huge instance (or create them sequentially) and then use the generated hash tables on a smaller instance. But, as I understand, that is already tracked in https://github.com/FALCONN-LIB/FALCONN/issues/46

I'm not exactly sure how you would use the generated hash tables on a smaller instance. But #46 is indeed the ideal solution for loading and storing hash tables.

@funtusov Could you control the memory usage of FALCONN in the end?

@ludwigschmidt not really, we eventually went with an in-house LSH implementation

OK, thanks for the update. We'll investigate this when we update the Python wrapper in #69 .


change conf files so that prop names start with pekko instead of akka

ensure code is updated to use right prop names
update the docs

I think @mdedetrich is doing this as part of #21 solution

I will handle this after https://github.com/apache/incubator-pekko-http/pull/34 is merged since I had to do some conf file workarounds.


[firebase_analytics] Tutorial  "Add Firebase to your Flutter app" did not work

Describe the bug
https://firebase.google.com/docs/flutter/setup does not work for Firebase Analytics.
Steps to Reproduce

Go through Flutter Firebase tutorial with Firebase Analytics: https://firebase.google.com/docs/flutter/setup. Using https://github.com/FirebaseExtended/flutterfire/tree/master/packages/firebase_analytics/example

Ensure GoogleService-Info.plist is in XCode:

I added "-FIRDebugEnabled" to scheme.
Wait 24 hours
Firebase Analytics does not show anything:

Expected behavior
Firebase Analytics show some data from the app.

Target Platform:
iOS
Target OS version/browser:
iOS 13.2 (Simulator)
Devices:
iOS Simulator iPhone 11 Pro Max
and
iPhone SE (not simulator)
Logs
Here is logs from the application startup in XCode when I ran it for the simulator:
2019-11-07 08:49:44.125876-0500 Runner[88829:6937609]  - <AppMeasurement>[I-ACS036002] Analytics screen reporting is enabled. Call +[FIRAnalytics setScreenName:setScreenClass:] to set the screen name or override the default screen class name. To disable screen reporting, set the flag FirebaseScreenReportingEnabled to NO (boolean) in the Info.plist
2019-11-07 08:49:44.260443-0500 Runner[88829:6937320] Configuring the default Firebase app...
2019-11-07 08:49:44.308298-0500 Runner[88829:6937320] Configured the default Firebase app __FIRAPP_DEFAULT.
2019-11-07 08:49:44.363260-0500 Runner[88829:6937642] flutter: Observatory listening on http://<IP_ADDRESS>:62660/uT7_mQy6iVw=/
2019-11-07 08:49:44.941378-0500 Runner[88829:6937609] 6.11.0 - [Firebase/Core][I-COR000004] App with name __FIRAPP_DEFAULT does not exist.
2019-11-07 08:49:44.941701-0500 Runner[88829:6937609] 6.11.0 - [Firebase/Core][I-COR000001] Configuring the default app.
2019-11-07 08:49:44.942312-0500 Runner[88829:6937609] 6.11.0 - [Firebase/Core][I-COR000033] Data Collection flag is not set.
2019-11-07 08:49:44.942488-0500 Runner[88829:6937609] 6.11.0 - [Firebase/InstanceID][I-IID023000] Info is not found in Keychain. OSStatus: -25300. Keychain query: {
    acct = "com.appspot.sztm.HelloFirebaseSztm";
    class = genp;
    gena = "com.google.iid";
    "m_Limit" = "m_LimitAll";
    "r_Attributes" = 1;
    "r_Data" = 1;
    svce = "com.google.iid.checkin";
}
2019-11-07 08:49:44.942640-0500 Runner[88829:6937609] 6.11.0 - [Firebase/InstanceID][I-IID023000] Info is not found in Keychain. OSStatus: -25300. Keychain query: {
    atag = {length = 18, bytes = 0x636f6d2e676f6f676c652e6969642d7c4b7c};
    class = keys;
    "r_Ref" = 1;
    type = 42;
}
2019-11-07 08:49:44.942776-0500 Runner[88829:6937609] 6.11.0 - [Firebase/InstanceID][I-IID023000] Info is not found in Keychain. OSStatus: -25300. Keychain query: {
    atag = {length = 18, bytes = 0x636f6d2e676f6f676c652e6969642d7c507c};
    class = keys;
    "r_Ref" = 1;
    type = 42;
}
2019-11-07 08:49:44.947071-0500 Runner[88829:6937609] 6.11.0 - [Firebase/InstanceID][I-IID009000] No keypair info is found with tag com.google.iid-|K|
2019-11-07 08:49:44.947249-0500 Runner[88829:6937609] 6.11.0 - [Firebase/InstanceID][I-IID009006] There's no legacy keypair so no need to do migration.
2019-11-07 08:49:44.947376-0500 Runner[88829:6937609] 6.11.0 - [Firebase/InstanceID][I-IID023000] Info is not found in Keychain. OSStatus: -25300. Keychain query: {
    acct = "com.appspot.sztm.HelloFirebaseSztm";
    class = genp;
    gena = "com.google.iid-tokens";
    "m_Limit" = "m_LimitAll";
    "r_Attributes" = 1;
    "r_Data" = 1;
    svce = "947567047323:*";
}
2019-11-07 08:49:44.947527-0500 Runner[88829:6937609] 6.11.0 - [Firebase/Analytics][I-ACS024000] Debug mode is on
2019-11-07 08:49:44.947649-0500 Runner[88829:6937609] 6.11.0 - [Firebase/Analytics][I-ACS005000] The AdSupport Framework is not currently linked. Some features will not function properly. Learn more at http://goo.gl/9vSsPb
2019-11-07 08:49:45.261748-0500 Runner[88829:6937611] 6.11.0 - [GULReachability][I-REA902003] Monitoring the network status
2019-11-07 08:49:45.262162-0500 Runner[88829:6937611] 6.11.0 - [Firebase/Analytics][I-ACS023007] Analytics v.60103000 started
2019-11-07 08:49:45.264925-0500 Runner[88829:6937611] 6.11.0 - [Firebase/Analytics][I-ACS023008] To disable debug logging set the following application argument: -noFIRAnalyticsDebugEnabled (see http://goo.gl/RfcP7r)
2019-11-07 08:49:45.265908-0500 Runner[88829:6937611] 6.11.0 - [Firebase/Analytics][I-ACS023009] Debug logging enabled
2019-11-07 08:49:45.277997-0500 Runner[88829:6937644] 6.11.0 - [Firebase/Analytics][I-ACS031003] setScreenName:screenClass: must be called after a view controller has appeared
2019-11-07 08:49:45.280275-0500 Runner[88829:6937644] 6.11.0 - [GULReachability][I-REA902004] Network status has changed. Code:2, status:Connected
2019-11-07 08:49:45.311344-0500 Runner[88829:6937644] 6.11.0 - [Firebase/Analytics][I-ACS031010] Tracking view controller. Class, ID: FlutterViewController,<PHONE_NUMBER>934354282
2019-11-07 08:49:45.311614-0500 Runner[88829:6937644] 6.11.0 - [Firebase/Analytics][I-ACS031006] View controller already tracked. Class, ID: FlutterViewController,<PHONE_NUMBER>934354282
2019-11-07 08:49:45.632050-0500 Runner[88829:6937644] 6.11.0 - [GoogleUtilities/AppDelegateSwizzler][I-SWZ001008] Successfully created App Delegate Proxy automatically. To disable the proxy, set the flag GoogleUtilitiesAppDelegateProxyEnabled to NO (Boolean) in the Info.plist
2019-11-07 08:49:46.473780-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS029017] Configuration was not found in database. The configuration will be fetched from the network when necessary
2019-11-07 08:49:46.474840-0500 Runner[88829:6937644] 6.11.0 - [Firebase/Analytics][I-ACS023016] Analytics is ready to receive events
2019-11-07 08:49:46.619744-0500 Runner[88829:6937644] 6.11.0 - [Firebase/Analytics][I-ACS032003] iAd framework is not linked. Search Ad Attribution Reporter is disabled.
2019-11-07 08:49:46.623121-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS002002] Measurement timer scheduled to fire in approx. (s): -76.61791610717773
2019-11-07 08:49:46.623653-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023028] Upload task scheduled to be executed in approx. (s): -76.61791610717773
2019-11-07 08:49:46.623898-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS033003] Scheduling user engagement timer
2019-11-07 08:49:46.624197-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS002002] Engagement timer scheduled to fire in approx. (s): 3600
2019-11-07 08:49:46.625797-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023012] Analytics enabled
2019-11-07 08:49:46.626540-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023051] Logging event: origin, name, params: auto, screen_view (_vs), {
    ga_event_origin (_o) = auto;
    ga_screen_class (_sc) = FlutterViewController;
    ga_screen_id (_si) =<PHONE_NUMBER>934354282;
}
2019-11-07 08:49:46.627417-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023073] Debug mode is enabled. Marking event as debug and real-time. Event name, parameters: screen_view (_vs), {
    ga_debug (_dbg) = 1;
    ga_event_origin (_o) = auto;
    ga_realtime (_r) = 1;
    ga_screen_class (_sc) = FlutterViewController;
    ga_screen_id (_si) =<PHONE_NUMBER>934354282;
}
2019-11-07 08:49:47.086213-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023072] Event logged. Event name, event params: screen_view (_vs), {
    ga_debug (_dbg) = 1;
    ga_event_origin (_o) = auto;
    ga_realtime (_r) = 1;
    ga_screen_class (_sc) = FlutterViewController;
    ga_screen_id (_si) =<PHONE_NUMBER>934354282;
}
2019-11-07 08:49:47.103606-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023027] Do not schedule an upload task. Task already exists. Will be executed in seconds: -77.08841514587402
2019-11-07 08:49:47.108694-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS002001] Measurement timer fired
2019-11-07 08:49:47.113948-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS002003] Measurement timer canceled
2019-11-07 08:49:47.114452-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023033] Starting data upload
2019-11-07 08:49:47.115125-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023037] No data to upload
2019-11-07 08:49:47.115603-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023126] Fetching configuration from the server
2019-11-07 08:49:47.241090-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS900001] Downloading data. Host: https://app-measurement.com/config/app/1:947567047323:ios:c8456dabdaf3d0d3944c3e?platform=ios&app_instance_id=160A9A2BD9C24151A452C298D95C4A3F&gmp_version=60103
2019-11-07 08:49:47.349510-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS901006] Received SSL challenge for host. Host: https://app-measurement.com/config/app/1:947567047323:ios:c8456dabdaf3d0d3944c3e?platform=ios&app_instance_id=160A9A2BD9C24151A452C298D95C4A3F&gmp_version=60103
2019-11-07 08:49:47.577356-0500 Runner[88829:6937611] 6.11.0 - [Firebase/Analytics][I-ACS023127] Fetched configuration. Status code: 200
2019-11-07 08:49:47.594505-0500 Runner[88829:6937611] 6.11.0 - [Firebase/Analytics][I-ACS023133] Successful fetched configuration from the server. Got network response. Code, size: 200, 236
<ERROR> MeasurementConfig decoder failed: varint overflow
2019-11-07 08:49:47.622003-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023135] Configuration data is invalid.
2019-11-07 08:49:47.633837-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS034010] Transaction failed
2019-11-07 08:49:47.643274-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023033] Starting data upload
2019-11-07 08:49:47.645430-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023105] Event is not subject to real-time event count daily limit. Marking an event as real-time. Event name, parameters: user_engagement (_e), {
    engagement_time_msec (_et) = 1;
    ga_debug (_dbg) = 1;
    ga_event_origin (_o) = auto;
    ga_realtime (_r) = 1;
    ga_screen (_sn) = /;
    ga_screen_class (_sc) = Flutter;
    ga_screen_id (_si) = 1;
}
2019-11-07 08:49:47.646820-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023105] Event is not subject to real-time event count daily limit. Marking an event as real-time. Event name, parameters: screen_view (_vs), {
    ga_debug (_dbg) = 1;
    ga_event_origin (_o) = auto;
    ga_realtime (_r) = 1;
    ga_screen_class (_sc) = FlutterViewController;
    ga_screen_id (_si) = 1;
}
2019-11-07 08:49:47.647596-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023080] Setting user property. Name, value: lifetime_user_engagement (_lte), 12
2019-11-07 08:49:47.648459-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023087] User property set. Name, value: lifetime_user_engagement (_lte), 12
2019-11-07 08:49:47.649067-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023080] Setting user property. Name, value: session_user_engagement (_se), 12
2019-11-07 08:49:47.649684-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023087] User property set. Name, value: session_user_engagement (_se), 12
2019-11-07 08:49:47.649989-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS012018] Saving bundle. size (bytes): 421
2019-11-07 08:49:47.650241-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023116] Bundle added to the upload queue. BundleID, timestamp (ms): 13,<PHONE_NUMBER>297
2019-11-07 08:49:47.861894-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023038] Uploading events. Elapsed time since last successful upload (s): 41228.05698013306
2019-11-07 08:49:47.871812-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS023039] Measurement data sent to network. Timestamp (ms), data:<PHONE_NUMBER>851, <APMPBMeasurementBatch: 0x60000084e630>
2019-11-07 08:49:47.882457-0500 Runner[88829:6937616] 6.11.0 - [Firebase/Analytics][I-ACS900000] Uploading data. Host: https://app-measurement.com/a
2019-11-07 08:49:48.211802-0500 Runner[88829:6937611] 6.11.0 - [Firebase/Analytics][I-ACS901006] Received SSL challenge for host. Host: https://app-measurement.com/a
2019-11-07 08:49:48.265827-0500 Runner[88829:6937610] 6.11.0 - [Firebase/Analytics][I-ACS023044] Successful upload. Got network response. Code, size: 204, -1
2019-11-07 08:49:48.268232-0500 Runner[88829:6937610] 6.11.0 - [Firebase/Analytics][I-ACS002002] Measurement timer scheduled to fire in approx. (s): -1.970462083816528
2019-11-07 08:49:48.268810-0500 Runner[88829:6937610] 6.11.0 - [Firebase/Analytics][I-ACS023028] Upload task scheduled to be executed in approx. (s): -1.970462083816528
2019-11-07 08:49:48.393171-0500 Runner[88829:6937644] 6.11.0 - [Firebase/Analytics][I-ACS023024] No data to upload. Upload task will not be scheduled
2019-11-07 08:49:48.398344-0500 Runner[88829:6937644] 6.11.0 - [Firebase/Analytics][I-ACS002003] Measurement timer canceled

~/Documents/hello_firebase_sztm $ flutter analyze
Analyzing hello_firebase_sztm...
No issues found! (ran in 4.2s)

<EMAIL_ADDRESS>~/Documents/hello_firebase_sztm
~/Documents/hello_firebase_sztm $ flutter doctor -v
[✓] Flutter (Channel stable, v1.9.1+hotfix.6, on Mac OS X 10.14.6 18G87, locale en-JP)
    • Flutter version 1.9.1+hotfix.6 at /Users/suztomo/local/flutter
    • Framework revision 68587a0916 (8 weeks ago), 2019-09-13 19:46:58 -0700
    • Engine revision b863200c37
    • Dart version 2.5.0

[✗] Android toolchain - develop for Android devices
    ✗ Unable to locate Android SDK.
      Install Android Studio from: https://developer.android.com/studio/index.html
      On first launch it will assist you in installing the Android SDK components.
      (or visit https://flutter.dev/setup/#android-setup for detailed instructions).
      If the Android SDK has been installed to a custom location, set ANDROID_HOME to that location.
      You may also want to add it to your PATH environment variable.

[✓] Xcode - develop for iOS and macOS (Xcode 11.2)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 11.2, Build version 11B52
    • CocoaPods version 1.8.4

[!] Android Studio (not installed)
    • Android Studio not found; download from https://developer.android.com/studio/index.html
      (or visit https://flutter.dev/setup/#android-setup for detailed instructions).

[!] IntelliJ IDEA Community Edition (version 2019.1.4)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
    • For information about installing plugins, see
      https://flutter.dev/intellij-setup/#installing-the-plugins

[✓] VS Code (version 1.39.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.6.0

[✓] Connected device (1 available)
    • iPhone 11 Pro Max • 16B6A8C8-4CDC-4AB6-BBA7-88FAFC3562A3 • ios • com.apple.CoreSimulator.SimRuntime.iOS-13-2 (simulator)

! Doctor found issues in 3 categories.

Other information
Cloud Firestore worked fine with my Flutter development environment. This means GoogleService-Info.plist has been added to XCode correctly. I'm feeling this is specific to Flutter Firebase Analytics.

Today I tried the tutorial and now it works.

It does not work for me, I tried on Android with emulator as well as on a physical Samsung S-9-XL and did not get the DebugView to show any connected devices. I followed all the guides that I could find online and also tried other flutter sample apps that were supposed to show working FB analytics on flutter but nothing.

@Easycatchup Try the Firebase Analytics tutorial with Swift using the same Firebase account. In my case it didn’t work either. Therefore the problem was not in this Flutter library, but somewhere in the backend.

@suztomo Were you able to fix it? Sounds like it is somehow linked to the Firebase account or to the Firebase settings?

Did Swift work?
I just waited few months and the problem was gone. I didn’t fix anything.

I don't have the developer environment for swift setup.
I checked on the Firebase slack channels though and it seems like there's multiple people having the issue in the last days of having nothing show up on the DebugView.

Then the problem does not seem to related to this Flutter library.


Added tagFilterButton to plugin list

Added tagFilterButton to plugin list

Whoa, whoa, whoa. You've made a pull request from your gh-pages to Leaflet's master branch!
Instead of that, please make a PR from your gh-pages to Leaflet's gh-pages.

Sory, this is my mistake


Service contract pool values and DKG contribution margin made public

DKG contribution margin, DKG fee pool, and request subsidy fee pool can be public without hurting anything. Having them public makes the system more transparent and helps with debugging (see #1736).

@pdyraga _baseCallbackGas has to be updated to reflect the contract size change, just tested locally it should be 10226. Plus an update for "should correctly evaluate entry fee estimate" test
Do you want me to push the change ? I have it ready

Yes @ngrinkevich, please do it! I have another PR in the queue that depends on this one.

Looks good to me, thank you @ngrinkevich :+1:
Leaving the final decision and merge honours to you.


Consider switching "alias" annotation to "json"

The rationale is twofold:

It is almost certain that if a type is to be shared across multiple ingress methods, such as HTTP, gRPC, and GraphQL, users will want to be able to use different aliases for different protocols. The choice of alias doesn't allow for that.
It will simplify interacting with HTTP ingress endpoints from Go code such as integration tests, because their will only need to be a single annotation, json:"xyz".

Schema should change to:
data Foo {
  field String alias http bar
}

verb bar()
  ingress http GET /bar

Go annotations should change to:
json:"bar"

Kotlin annotation should change to something like this, but dig into whether we should just use @JsonProperty? Is this part of JAX-RS or some random thing from Jackson?
@JsonProperty("bar")

Edit: there doesn't seem to be any standard at all. Gson uses SerializedName, Jackson uses @JsonProperty - let's just use @Json.

Yeah after a bit of googling and chatting with my friend Chad, it seems like there's no standard for this. There are standards for serializing, but they leave out the part about this being for JSON on http. So @Json sounds good to me as well 👍


dynamic api

similar to pages but for REST endpoints
the method api.getResourceConfig accepts a req and returns an object with a payload and a getPayload string to call an endpoint to override the payload
It should deal with caches, roles, redirections, and pagination.

Starting with the JSON driver


Added cert-manager and https cert automatic issuing

Add installation of cert-manager
/fix https://github.com/chainstack/graph-deployment/issues/11

WDYT if we install certmanager using helm provider like it is done here?
https://github.com/chainstack/infrastructure/tree/master/tf-modules/helm/certmanager
I don't want to increase the complexity of the repository by nesting directories. Also the number of manual steps users need to go through is undesirable IMHO.
I understand that you don't want to duplicate certmanager installation in each tf module.

I decided not to add that to terraform cause:

It could be useful for baremetal clusters
It doesn't depends on any terraform resource outputs.

I also don't like idea adding more nested structures, but I'm going to merge on helmfile in future, so I think it would be simplified in 3rd milestone.
P.S. Also this helmfile is going to contain kube-dashboard and prometheus+grafana installation


Broken templates

Reported from a student:
in the first course (java example), to make templates working i had to add
spring.freemarker.template-loader-path: classpath:/templates
spring.freemarker.suffix: .ftl
to application.properties otherwise i got 404 in the browser.

FYI @salaboy

This is due to the IDE. mvn clean and build should solve the issue.


Multischema initializes required contents as optional

Description
This bug was introduced in ngx-formly/core v 5.10.19. When a multischema contains items that are required, these items are initially rendered as optional. The multischema contents' true required status is only rendered once something in the multischema has been interacted with. This also affects overall form validity.
Minimal Reproduction
https://stackblitz.com/edit/ngx-formly-multischema-required-as-optional?file=app%2Fapp.component.ts
Your Environment

Angular version: 10.2.5
Formly version: 5.10.19

This issue has been fixed and released as part of v5.10.20 release.
Please let us know, in case you are still encountering a similar issue/problem.
Thank you!


Add a good table component to display list of jx resources

We could use react-table as the latest version supports svelte.

Decided to go with ag-grid, it's pretty slick: https://github.com/ag-grid/ag-grid

fixed by #6


Partial model coverage upon download - raytracing

Scenario: With new buffering code our ask for HRRR download area  extends outside the coverage of the model. Currently we proceed and don't error out. If we let the code proceed then ray-tracing would start to have bias.
For example if more than a 5 degree buffer is requested in northern CA along the coast then we might not have enough HRRR data. https://rapidrefresh.noaa.gov/RAPv4HRRRv3-OpsTrim.jpg
The buffering code currently is very conservative. Ideally we implement it based on actual orbit and look direction. Aka buffering does not need to be in all directions only the direction of the ray.

I think this is closed with a number of buffering related PRs, most recently #564


[Question]: Flutter Appbar disappears when using Share

What is your question?
Flutter Appbar disappears when using Share
I have the same :  https://stackoverflow.com/questions/74950793/in-flutter-share-a-link-using-share-plus-then-page-moves-to-status-bar-and-stat
Checklist before submitting a question

[X] I Google'd a solution and I couldn't find it
[X] I searched on StackOverflow for a solution and I couldn't find it
[X] I read the README.md file of the plugin
[X] I am using the latest version of the plugin
[X] All dependencies are up to date with flutter pub upgrade
[X] I did a flutter clean
[X] I tried running the example project

Could you share the code on how your screen looks like?

class FeesPage extends StatelessWidget {
const FeesPage({super.key});
Future share() async {
await Share.share("Example share", subject: "Test subject");
}
@override
Widget build(BuildContext context) {
return MaterialApp(
home: Scaffold(
appBar: AppBar(
title: const Text('Plugin example app'),
),
body: Center(
child: Column(
crossAxisAlignment: CrossAxisAlignment.center,
mainAxisAlignment: MainAxisAlignment.center,
children: [
ElevatedButton(
child: Text('Share text and link'),
onPressed: share,
),
ElevatedButton(
child: Text('Share local file'),
onPressed: share,
),
],
),
),
),
);
}
}

`class FeesPage extends StatelessWidget {
const FeesPage({super.key});
Future share() async {
await Share.share("Example share", subject: "Test subject");
}
@override
Widget build(BuildContext context) {
return MaterialApp(
home: Scaffold(
appBar: AppBar(
title: const Text('Plugin example app'),
),
body: Center(
child: Column(
crossAxisAlignment: CrossAxisAlignment.center,
mainAxisAlignment: MainAxisAlignment.center,
children: [
ElevatedButton(
child: Text('Share text and link'),
onPressed: share,
),
ElevatedButton(
child: Text('Share local file'),
onPressed: share,
),
],
),
),
),
);
}
}
`

  const FeesPage({super.key});

  Future<void> share() async {
    await Share.share("Example share", subject: "Test subject");
  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        appBar: AppBar(
          title: const Text('Plugin example app'),
        ),
        body: Center(
          child: Column(
            crossAxisAlignment: CrossAxisAlignment.center,
            mainAxisAlignment: MainAxisAlignment.center,
            children: <Widget>[
              ElevatedButton(
                child: Text('Share text and link'),
                onPressed: share,
              ),
              ElevatedButton(
                child: Text('Share local file'),
                onPressed: share,
              ),
            ],
          ),
        ),
      ),
    );
  }
}

Any solution for this annoying issue ?

Any solution for this annoying issue ?
SafeArea doesn't work.

I have been facing same issue on Samsung Galaxy A32 (Android 13). I think Android version is not reason of this issue. Only samesung device. On other device realme C35 (Android-13) it's fine.

my error gone away automatically it is not a bug of Share Plugin it is only on some devices. if you test it out with a production level build you will not see the error and make sure developer mode off when running. you can close the issue now @vbuberen

my error gone away automatically it is not a bug of Share Plugin it is only on some devices. if you test it out with a production level build you will not see the error and make sure developer mode off when running.

Thanks for your feedback.
In fact, on Android when Share.share is used no UI manipulation is done by the plugin as in Android native part standard Intent is created for sharing without any changes to window, UI, etc. Thus, there is nothing to fix in the plugin as the issue is caused by the system itself on some devices where people experience this problem.


Empty element adds residual divergence

Noticed by Xianbo, a screen/slit (empty element) modifies the beam in col 5.
See effect of oe3 (empty).
If suppressed, OK.
If col6 copied, OK.
otherwise NOT OK.
#
# Python script to run shadow3. Created automatically with ShadowTools.make_python_script_from_list().
#
import Shadow
import numpy

# write (1) or not (0) SHADOW files start.xx end.xx star.xx
iwrite = 0

#
# initialize shadow3 source (oe0) and beam
#
beam = Shadow.Beam()
oe0 = Shadow.Source()
oe1 = Shadow.OE()
oe2 = Shadow.OE()
oe3 = Shadow.OE()
oe4 = Shadow.OE()

#
# Define variables. See meaning of variables in:
#  https://raw.githubusercontent.com/srio/shadow3/master/docs/source.nml
#  https://raw.githubusercontent.com/srio/shadow3/master/docs/oe.nml
#

oe0.FDISTR = 3
oe0.F_PHOT = 0
oe0.HDIV1 = 0.0
oe0.HDIV2 = 0.0
oe0.ISTAR1 = 1001
oe0.NPOINT = 100000
oe0.PH1 = 25000.0
oe0.SIGDIX = 1e-10
oe0.SIGDIZ = 3.7e-06
oe0.SIGMAX = 0.0
oe0.SIGMAZ = 0.0046
oe0.VDIV1 = 0.0
oe0.VDIV2 = 0.0

oe1.DUMMY = 0.1
oe1.FCYL = 1
oe1.FHIT_C = 1
oe1.FMIRR = 2
oe1.RLEN1 = 200.0
oe1.RLEN2 = 200.0
oe1.RWIDX1 = 10.0
oe1.RWIDX2 = 10.0
oe1.T_IMAGE = 24700.0
oe1.T_INCIDENCE = 89.856761
oe1.T_REFLECTION = 89.856761
oe1.T_SOURCE = 35300.0

oe2.DUMMY = 0.1
oe2.F_REFRAC = 2
oe2.F_SCREEN = 1
oe2.I_SLIT = numpy.array([1, 0, 0, 0, 0, 0, 0, 0, 0, 0])
oe2.N_SCREEN = 1
oe2.RX_SLIT = numpy.array([1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])
oe2.RZ_SLIT = numpy.array([0.002, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])
oe2.T_IMAGE = 118695.0
oe2.T_INCIDENCE = 0.0
oe2.T_REFLECTION = 180.0
oe2.T_SOURCE = 0.0

oe3.DUMMY = 0.1
oe3.F_REFRAC = 2
oe3.F_SCREEN = 1
oe3.N_SCREEN = 1
oe3.T_IMAGE = 0.0
oe3.T_INCIDENCE = 0.0
oe3.T_REFLECTION = 180.0
oe3.T_SOURCE = 0.0

oe4.DUMMY = 0.1
oe4.FCYL = 1
oe4.FHIT_C = 1
oe4.FMIRR = 2
oe4.F_ANGLE = 1
oe4.F_DEFAULT = 0
oe4.RLEN1 = 150.0
oe4.RLEN2 = 150.0
oe4.RWIDX1 = 1.0
oe4.RWIDX2 = 1.0
oe4.SIMAG = 305.0
oe4.SSOUR = 119695.0
oe4.THETA = 89.856761
oe4.T_IMAGE = 305.0
oe4.T_INCIDENCE = 89.856761
oe4.T_REFLECTION = 89.856761
oe4.T_SOURCE = 1000.0

#Run SHADOW to create the source

if iwrite:
    oe0.write("start.00")

beam.genSource(oe0)

if iwrite:
    oe0.write("end.00")
    beam.write("begin.dat")

#
#run optical element 1
#
print("    Running optical element: %d"%(1))
if iwrite:
    oe1.write("start.01")
beam.traceOE(oe1,1)
if iwrite:
    oe1.write("end.01")
    beam.write("star.01")

#
#run optical element 2
#
print("    Running optical element: %d"%(2))
if iwrite:
    oe2.write("start.02")
beam.traceOE(oe2,2)
if iwrite:
    oe2.write("end.02")
    beam.write("star.02")

#
#run optical element 3
#
print("    Running optical element: %d"%(3))
if iwrite:
    oe3.write("start.03")
ray_before = beam.rays.copy()

beam.traceOE(oe3,3)

beam.rays[:,5]  = ray_before[:,5]
ray_after = beam.rays.copy()
for i in range(6):
    print(">>>>>>>>>>>>>>>>>>>>>",i,(ray_after-ray_before)[:,i])
if iwrite:
    oe3.write("end.03")
    beam.write("star.03")

#
#run optical element 4
#
print("    Running optical element: %d"%(4))
if iwrite:
    oe4.write("start.04")
beam.traceOE(oe4,4)
if iwrite:
    oe4.write("end.04")
    beam.write("star.04")

Shadow.ShadowTools.plotxy(beam,1,3,nbins=101,nolost=1,title="Real space")
# Shadow.ShadowTools.plotxy(beam,1,4,nbins=101,nolost=1,title="Phase space X")
# Shadow.ShadowTools.plotxy(beam,3,6,nbins=101,nolost=1,title="Phase space Z")

The problem is in PI:
If I declare (as I was doing in shadow_globaldefinitions) (skr=8):
real(kind=skr), parameter :: pi     = 3.141592653589793238462643

and then I print pi I get: 3.14159274 )you get the same in T_REFLECTION in end.xx of your cleaning slit)
Even if declared as double precision, and manipulating it in double precision, fortran reads the number as single precision. Funny, no? Did you know it?
I must do:
real(kind=skr), parameter :: pi     = 3.141592653589793238462643D0

(and also added D0 to all parameter declarations) and everything works well.
This is probably the cause of the shift of the center of spot when using CRLs.


centos install nvm is fail.

Operating system and version: centos 6.4

nvm debug output:
[root@aspire ~]# curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.11/install.sh | bash
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
100 12819  100 12819    0     0  12872      0 --:--:-- --:--:-- --:--:-- 57484
=> Downloading nvm from git to '/root/.nvm'
=> Unknown option: -c
usage: git [--version] [--exec-path[=GIT_EXEC_PATH]] [--html-path]
[-p|--paginate|--no-pager] [--no-replace-objects]
[--bare] [--git-dir=GIT_DIR] [--work-tree=GIT_WORK_TREE]
[--help] COMMAND [ARGS]
Failed to clone nvm repo. Please report this!

nvm ls output:

How did you install nvm? (e.g. install script in readme, Homebrew):

What steps did you perform?

What happened?

What did you expect to happen?

Is there anything in any of your profile files (.bashrc, .bash_profile, .zshrc, etc) that modifies the PATH?

If you are having installation issues, or getting "N/A", what does curl -I --compressed -v https://nodejs.org/dist/ print out?

What git version do you have? nvm requires v1.7.10 or higher.

I'm have the same issue.
$ curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.11/install.sh | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12819  100 12819    0     0  72114      0 --:--:-- --:--:-- --:--:--  782k
=> Downloading nvm from git to '/home/staging/.nvm'
=> Unknown option: -c
usage: git [--version] [--exec-path[=GIT_EXEC_PATH]] [--html-path]
           [-p|--paginate|--no-pager] [--no-replace-objects]
           [--bare] [--git-dir=GIT_DIR] [--work-tree=GIT_WORK_TREE]
           [--help] COMMAND [ARGS]
Failed to clone nvm repo. Please report this!

Using git version 1.7.1

@shanedenham 1 < 10, so you’ll need to upgrade git.

Thanks, @ljharb! I hadn't read your original comment carefully enough. I upgraded git and NVM installed without any problem.

for people coming here from Google. You can find how to update git to the latest version on Centos: https://stackoverflow.com/a/27674776/31278

yes,update git can fix this .....


Set hyperlink to selected text

I implemented this feature in dfb3a395d79fc3f4d3d8d3660d10ced5046b9b00 and 34dd2ca3e3974f49d760dd9733c29fd1606842ab.
I Added insertLink function and Added LinkButton&UnLinkButton to the Demo Editor (https://baechoofarm.github.io/sketchbox/?path=/story/sketchbox-editor--basic).
You can select texts by drag and click LinkButton to set Link Element. And Click UnLinkButton to reset element.


KeyBindingHandler handles repeats when original key is not handled

https://github.com/ppy/osu-framework/blob/1f7f73735e1dae2add77f6e6ad417a322acb0b1b/osu.Framework/Input/Bindings/KeyBindingContainer.cs#L118-L119
Consumption should be based on whether the original (non-repeat) event was handled or not, rather than whether any binding matches it. As seen in https://github.com/ppy/osu/issues/14428#issuecomment-918865134.
Should probably be addressed after https://github.com/ppy/osu-framework/pull/2601 is merged, as a lot of the logic around this has changed.

This might get a bit complex to handle, as it probably should be making the decision on a per-binding level (ie. rather than returning early in the Handle function, it would need to do a lookup for the currently pressed keys in the incoming event, match and ensure it should actually be blocked).
A better path forward may be to just remove/obsolete SendRepeats alongside the UIEvent change, as it shouldn't be required with the new event arguments.

Closing as resolved.


Please relax pydantic dependency

For the webauthn extra, there is a dependency on pydantic 1.x.
Pydantic has now released 2.0 with a lot of improvements, which I need in a project that also uses django-two-factor-auth.
I have analyzed the situation:

py_webauthn (probably wrongly) allows all Pydantic versions, it ignores semantic versioning
For pydantic 2.0, it looks like this is ok, as no features used by webauthn have broken in Pydantic 2.0
django-two-factor-auth only imports an exception to catch it, which is also still in place

So, in order to allow newer Pydantic version, I propose one of the following:
Remove the pydantic dependency
django-two-factor-auth itself does not need it; the exception class can be re-imported from webauthn instead
Relax the dependency to >=1.9.0,<3.0
Allow pydantic 2.0 explicitly instead

I would be fine with removing the dependency.

I followed @Natureshadow 's suggestion since it's the official path for the symbol until at least <3.0. Re-importing from webuthn.entrails.that.might.change seems dangerous even if it lets us drop the direct dependency on pydantic.


Adding validation for CR cleanup

What this PR does / why we need it:
Adds validation for Backup and Restore CR cleanup
Which issue(s) this PR fixes (optional)
Closes #PB-5016
Special notes for your reviewer:
Aetos Dashboard:
Schedule Validation : https://aetos.pwx.purestorage.com/resultSet/testSetID/466285
Manual Backup CR Validation : https://aetos.pwx.purestorage.com/resultSet/testSetID/466274

Updated Aetos Dashboard: https://aetos.pwx.purestorage.com/resultSet/testSetID/468768

Updated Aetos: https://aetos.pwx.purestorage.com/resultSet/testSetID/469802

Ran full pipeline with this change: https://aetos.pwx.purestorage.com/resultSet/testSetID/470325
Failure is not related to the PR


Add support for dimming color and fade customizations

Allows apps to customize the color/opacity of the dimmed background when using SVProgressHUDMaskTypeBlack
Allows apps to customize the fade-in and fade-out animation speed (without calling dismissWithDuration: everywhere)
https://github.com/SVProgressHUD/SVProgressHUD/issues/564

Updated based on your feedback.
The dismissWithDuration was added because everything was being done in instance methods, but we need to fall back on the duration default. Previously this default was a class constant. By moving to an instance variable, had to refactor the code a bit to leverage the change without breaking other functions.

Almost done, added some stuff.


Python virtual environment not used by Studio, completely different Python injected instead

Somewhat similar to #9
I have a Docker image with a virtual environment and dependencies installed by pip:
# Dockerfile
RUN python -m venv /opt/venv \
    && . /opt/venv/bin/activate \
    && pip install --upgrade pip wheel \
    && pip install --requirement requirements.txt \
    && pip cache purge \
    && python -m ipykernel install

The image has a kernelspec in the right place:
# in the image terminal
$ cat /usr/local/share/jupyter/kernels/python3/kernel.json

{
 "argv": [
  "/opt/venv/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3 (ipykernel)",
 "language": "python",
 "metadata": {
  "debugger": true
 }
}

However, in Studio the virtual environment is not used, instead I get:
# in Studio with image kernel running
import sys
sys.exectuable
# '/opt/.sagemakerinternal/conda/bin/python'

which is some Python which isn't even in the image:
# in image terminal
ls -lh /opt/.sagemakerinternal/conda/bin/python3
# lrwxrwxrwx 1 nobody nogroup 9 Oct 12 17:38 /opt/.sagemakerinternal/conda/bin/python3 -> python3.7
ls -lh /opt/.sagemakerinternal/conda/bin/python3.7
# -rwxr-xr-x 1 nobody nogroup 13M Oct 12 17:38 /opt/.sagemakerinternal/conda/bin/python3.7

I was expecting Studio to look through the usual kernelspec locations and find the one matching the config for the attached image (which is set to a kernel called "python3").
What do I have to do for Studio to use the virtual environment as a kernel?

Seems that SageMaker Studio does not look for kernel specs in the system locations listed here:
https://jupyter-client.readthedocs.io/en/5.2.3/kernels.html#kernel-specs
That is, it does not find the kernel I installed in /usr/local/share/jupyter/kernels.
Instead it looks for the kernel spec in {sys.prefix}/share/jupyter/kernels so
python -m ipykernel install --sys-prefix   # good, works in SageMaker Studio

instead of
python -m ipykernel install  # bad, does not work in SageMaker Studio

solved my problem.

Thank you for posting this, I ran into a similar issue trying to get rocker R images (rocker/binder specifically) to work in Studio.  It turns out that the binder image installs the kernel to the python venv path: https://github.com/rocker-org/rocker-versioned2/blob/master/scripts/install_binder.sh#L29.  Simply adding another R -e "IRkernel::installspec()" to my Dockerfile fixed the kernel not being found error.
In case anyone else finds this, this Dockerfile worked for me (also addressed the seemingly very limited GID range allowed, which I did not find documented anywhere):
FROM rocker/binder:4.1.2

USER root

RUN usermod -g users rstudio
RUN usermod -l sagemaker-user rstudio

USER sagemaker-user

RUN pip3 install --user boto3 sagemaker
RUN R -e "IRkernel::installspec()"


无法安装dotnet

bili_task_base.sh: line 10: dotnet: command not found which dotnet:  开始安装dotnet ------尝试使用apk安装------ ERROR: unable to select packages:   bizCode (no such package):     required by: world[bizCode]   bizMsg (no such package):     required by: world[bizMsg]   dotnet6-sdk (no such package):     required by: world[dotnet6-sdk]   lxml (no such package):     required by: world[lxml] bili_task_base.sh: line 20: dotnet: command not found 安装成功
其实这些依赖都安装好了，不知道为什么不能扫码登录，其他脚本也都是同样报错。

不要重复提issue


Fix exception on setNotification method call for devices that do not …

…set CCCD_ID. (#185)

I'll try if this gets rid of some of my crashes which I get out in the wild in production


Run memory benchmarks in release mode

We currently run memory benchmarks in --profile mode. This allows us to use the VM service to synchronize the framework with memory measurements, but it also adds overhead from the VM service and timeline skewing the results. We should be able to run these benchmarks in release mode. However, we'd need a replacement for the VM service. We perform the following operations via the VM service:

wait for the first frame to render before measuring the initial memory usage (start_total_kb)

this can be replaced by a simple print statement and read out of the logcat

run a driver test that exercises the app in order to get a "cruising" memory usage (end_total_kb)

this can be replaced by an ad hoc test that drives the app from within the app process

/cc @rmacnak-google @eseidelGoogle

Well I have this done for the scroll benchmark, but the numbers aren't much less flaky. Here's the raw data from five back-to-back runs:
"start_total_kb": 57430,
"end_total_kb": 185386,
"diff_total_kb": 127956

"start_total_kb": 92919,
"end_total_kb": 181339,
"diff_total_kb": 88420

"start_total_kb": 93804,
"end_total_kb": 179534,
"diff_total_kb": 85730

"start_total_kb": 95652,
"end_total_kb": 181096,
"diff_total_kb": 85444

"start_total_kb": 75448,
"end_total_kb": 176476,
"diff_total_kb": 101028

Two options I can think of:

make the memory benchmark process much more long-lived (right now this test takes a few seconds at most)
run the test multiple times and use the median each time, rather than running it just once

@rmacnak-google @a-siva Any advice as to what would be a good way to get less noisy and more useful data regarding memory usage of Flutter apps?

I am wondering if the number you are getting depends on whether a GC happens or not before the memory usage is measured.

Quite possibly. Is there some way I can force a GC in a release build?

I've done what this bug wanted. It hasn't done all that much to get us less noisy data though. I'll keep an eye on the numbers and figure out what to do next.


TUF expiry contains microseconds

From TUF specification:

The expected format of the combined date and time string is "YYYY-MM-DDTHH:MM:SSZ". Time is always in UTC, and the "Z" time zone designator is attached to indicate a zero UTC offset. An example date-time string is "1985-10-21T01:21:00Z".

To me this says that the expiry string should not contain microseconds. Current sigstore metadata contains microseconds: https://github.com/sigstore/root-signing/blob/main/repository/repository/2.root.json#L27
I'm not sure if defining expiry this strictly in the spec is useful but the definition seems clear and in python-tuf we currently implement the spec strictly so fail to deserialize this metadata.
The related python-tuf issue (we'll have to decide if we should support msecs or not): https://github.com/theupdateframework/python-tuf/issues/1858

Oooh nice! Thank you for finding this -- I can set to truncate at seconds

@jku When you truncated to seconds, did you end up getting this error in python-tuf?
ValueError: ('Could not deserialize key data. The data may be in an incorrect format, it may be encrypted with an unsupported algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', [_OpenSSLErrorWithText(code=151584876, lib=9, reason=108, reason_text=b'error:0909006C:PEM routines:get_name:no start line')])

I'm just wondering whether that's what caused you to open https://github.com/theupdateframework/python-tuf/issues/1866 or whether there's some other issue.

I mean yes I found both issues while I was doing a manual interop testing round.
Current state to my knowledge is:

metadata with ecdsa keys created by go-tuf is non-compliant: https://github.com/theupdateframework/go-tuf/issues/223
python-tuf does not have workarounds to support this non-compliant format: https://github.com/theupdateframework/python-tuf/issues/1859.  I'm not completely opposed to adding support if it's possible to handle in an obviously correct way

Ah thanks @jku, that makes sense. I missed the key format issues.

Going to make a separate issue about the ECDSA key format -- this will be tough because root keys will have different key IDs if we immediately switching the encodings.


Crash when calling RegionPtr.Slice()

This was sent to me, I checked and they do have the latest version of the plugin installed (<IP_ADDRESS>):
This text file was created because [paint.net](http://paint.net/) crashed. Please e-mail this to [crashlog5<EMAIL_ADDRESS>so we can diagnose and fix the problem.

Application version: [paint.net](http://paint.net/) 5.0.13 (Stable 5.13.8830.42291)
Time of crash: 11/05/2024 22:44:14
Application uptime: 00:00:17.9771986
Application state: Running
Working set: 383,920 KiB
Handles and threads: 1561 handles, 97 threads, 1003 gdi, 329 user
Install type: Classic
Application directory: C:\Program Files\[paint.net](http://paint.net/)
Current directory: C:\Program Files\[paint.net](http://paint.net/)
Install directory: C:\Program Files\[paint.net](http://paint.net/)
OS Version: Windows 10 Home 10.0.19045.0 x64
.NET version: .NET 7.0.16 x64
Processor: "Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz" @ ~3600MHz (4C/8T, SSE, SSE2, SSE3, SSSE3, SSE4_1, SSE4_2, AVX, AVX2)
Memory: 16,335 MB physical (10,355 MB free), 133,947,977 MB virtual free, 18,767 MB pagefile (10,603 MB free)
Video card: NVIDIA GeForce GTX 1660 (fl:12.1, v:10DE, d:2184, r:A1), Microsoft Basic Render Driver (fl:12.1, v:1414, d:8C, r:0)
Hardware acceleration: True (default: True) (rendering: 🚀 Performance (NVIDIA GeForce GTX 1660))
Remote session: no
Pointers: none
UI animations: True
UI DPI: 96 dpi (1.00x scale)
UI theme: VisualStyleCategory=Aero, EffectiveTheme=Aero, DWM=yes, AeroColorScheme=Light, ThemeFileName=Aero.msstyles
Updates: True, 11/05/2024
Locale: pdnr.c: en-GB, hklm: en-GB, hkcu: en-GB, cc: en-GB, cuic: en-GB
Flags:

Exception details:
System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'bounds (768,0,27,256) is not valid (0,0,794,644)')
   at PaintDotNet.RegionPtr`1.SliceError(RectInt32 bounds) in D:\src\pdn\src_5_0_x\Primitives\RegionPtr`1.cs:line 470
   at PaintDotNet.Imaging.BitmapBase`1.LockImpl(RectInt32 rect, BitmapLockOptions lockOptions) in D:\src\pdn\src_5_0_x\Windows.Core\Imaging\BitmapBase`1.cs:line 123
   at PaintDotNet.Imaging.BitmapBase`1.Lock(RectInt32 rect, BitmapLockOptions lockOptions) in D:\src\pdn\src_5_0_x\Windows.Core\Imaging\BitmapBase`1.cs:line 92
   at PSFilterLoad.PSApi.Imaging.Internal.WICBitmapSurface`1.Lock(Rectangle bounds, SurfaceLockMode mode) in D:\Dev_projects\PSFilterPdn\src\PSApi\Imaging\Internal\WICBitmapSurface.cs:line 134
   at PSFilterLoad.PSApi.LoadPsFilter.FillInputBuffer(FilterRecord* filterRecord) in D:\Dev_projects\PSFilterPdn\src\PSApi\LoadPsFilter.cs:line 1788
   at PSFilterLoad.PSApi.LoadPsFilter.AdvanceStateProc() in D:\Dev_projects\PSFilterPdn\src\PSApi\LoadPsFilter.cs:line 1355

I am not able to reproduce that crash with any of the filters I tried.

I'm guessing you're either being given an out-of-bounds rectangle, or the rectangle becomes out-of-bounds when applying the padding values. This is in AdvanceStatePro -- I looked at the other methods in this call stack and none of them are doing any clipping or bounds checking, they just forward it along to the next method.
I'm not sure what you should do here in this case, whether you should clip or throw.

Yeah, I was thinking that the filter was passing me bad data. But without having a copy of the filter in question, there is no way for me to debug the issue.


Support for deployment with Docker Swarm

Would be a nice feature to have support and documentation for deploying a cluster with Docker Swarm.

for testing or production ?

What exactly do you mean for testing?
For myself, I'd use the swarm setup for both testing and production environments.

Noticed there are some compose files under /tests/compose that I think arrange a Pulsar cluster, but use test images. Can this be adapted to use production images?

I've managed to get a cluster working with Docker Swarm by adapting the instructions from the bare metal cluster deployment outlined here in the official documentation.
There are 4 stacks defined with compose files, and are brought up in the natural order:

pulsar-zookeeper.yml
pulsar-metadata.yml
pulsar-bookies.yml
pulsar-brokers.yml

The source code can be found in the upload zip: pulsar-docker-swarm.zip
I'd appreciate any feedback, questions, or comments.

I'm hoping this can be developed further and soon be put up as official documentation. A few goals of mine with this setup in the future are:

Add health checks
Automate with Ansible
Blue/green deployment

pulsar-metadata.yml:
Instead of hardcode each node; use swarm service name (note changes to pulsar-brokers.yml below):
--web-service-url http://broker:8080
--broker-service-url pulsar://broker:6650
pulsar-bookies.yml:
Instead of repeating bookie, use replicas instead
bookie:
image: apachepulsar/pulsar
command: >
/bin/bash -c
"bin/apply-config-from-env.py conf/bookkeeper.conf
&& bin/pulsar bookie"
networks:
- pulsar
environment:
zkServers: zk1:2181,zk2:2181,zk3:2181
deploy:
replicas: 3
placement:
max_replicas_per_node: 1
constraints: [node.role == manager]
pulsar-brokers.yml:
Should expose 6650/8080 for ingestion
broker:
image: apachepulsar/pulsar
command: >
/bin/bash -c
"bin/apply-config-from-env.py conf/broker.conf
&& bin/pulsar broker"
networks:
- pulsar
ports:
- 6650:6650
- 8080:8080
environment:
zookeeperServers: zk1:2181,zk2:2181,zk3:2181
configurationStoreServers: zk1:2181,zk2:2181,zk3:2181
clusterName: pulsar-cluster-1
brokerServicePort: "6650"
webServicePort: "8080"
deploy:
replicas: 3
placement:
max_replicas_per_node: 1
constraints: [node.role == manager]

Closed as stale. Please open a new issue if it's still relevant in maintained versions.
We have the docker image so this is possible. But two years later there's little interest on implementing this feature among the community. If an optional improvement doesn't have a concrete proposal and no contributor works on it, it's valueless to keep it open.

通过调整官方文档中概述的裸机集群部分的说明，我已经设计方法让集群与 Docker Swarm 一起工作。
使用compose 文件确定了4个堆垛，并按自然顺序排列：

pulsar-zookeeper.yml
pulsar-metadata.yml
pulsar-bookies.yml
pulsar-brokers.yml

源代码可以在上传的zip中找到：pulsar-docker-swarm.zip
如果有任何反应、问题或意见，我将不胜感激。

I used version 2.11.0 of pulsar and encountered an issue at this time. When deploying meta.yml, I was unable to successfully create it.


Ability to Scroll in Menu ?

@ scripthookvdotnet-Team
Suggestion: It may be a useful addition to make own menus scollable, e.g: you have 50 entries for your menu but you only want to appear 10 at the same time, cause your otherwise your screen is too small to see all the items.
It says that the value is read only...
thanks in advance

#211
Not scroll, but i think you can rewrite it to add scroll :wink:

Thank you i'll give it a try later :+1: .....  u may know a solution for the problem with the MenuNumericScroller?

Look how the Value is formed: https://github.com/crosire/scripthookvdotnet/blob/dev/source/MenuItem.hpp#L202

well , i guess that was kinda too easy.... Thank you :smile:

the scrolling is working now as well, again thanks for your help man :+1:

I have just modified the menu class to autoscroll, will put in a commit when I figure out how to draw up and down arrows to indicate there are more things in the menu than shown on screen

https://github.com/crosire/scripthookvdotnet/pull/247

pretty nice :+1:    i personally changed the footer to show Itemselected/Numberoficons automatically   so you a better overview over your items, may this would be an useful addition (maybe not by changing the footer but show it somewhere else) ?

@GrandTheftMods I thought about having a system like that, but decided against it because there wasnt an obvious place to put it without it being too obtrusive, which is why i use the up and down arrows. Using the footer for item count/max display would cause an issue with people who prefer not to use the footer in their menu

yep thats the problem i was facing too, i decided to use the numbers but i also kinda like the idea of using the arrows now i have to decide...

@GrandTheftMods If your scrolling menu inherits the GTA.Menu then once the next release is made all GTA.Menus will have this scrolling ability

@zorg93  so i can have both. nice :smile:

i looked up the arrowthing  wouldn't it be more smooth to scroll item by item and not that kind of side thing ?  and it would also be nice to have the option too choose how many items are shown at one time ?

I'm just testing altering how many items can be displayed now, as for your first point, I have no idea what you are asking

if i scoll down it jumps too the end of the men, it would be smoother that way : the selected item is the last one which can be seen -> you press down and on the the top the upmost item disappears and on the bottem the niext item is added

if i scroll down it jumts to the end of the menu ....

Wtf my keyboard

if i scroll down it jumps to the end of the menu, it would be smoother that way : the selected item is the last one which can be seen -> you press down and on  the upmost item disappears and on the bottem the next item is added

Still dont know what you're asking

okay another try ^^
public class ScrollMenu : Menu
{
private List _allItems;
private int _itemsPerPage;
private int Itemstoskip = 0;
    public ScrollMenu(string headerCaption, IMenuItem[] items, int itemsPerPage)
        : base(headerCaption, items.Take(itemsPerPage).ToArray())
    {
        this._allItems = new List<IMenuItem>();
        this._allItems.AddRange(items);
        foreach (IMenuItem item in this._allItems)
        {
            item.Parent = this;
        }
        this._itemsPerPage = itemsPerPage;
        this.ShowMenu();

    }

    public override void OnChangeSelection(bool down)
    {
        if (down)
        {
            if (this.SelectedIndex >= _itemsPerPage - 1)
            {
                bool skip = false;

                if (this.Itemstoskip + this._itemsPerPage < this._allItems.Count)
                {
                    this.Itemstoskip += 1;
                }
                else
                {
                    this.Itemstoskip = 0;
                    skip = true;
                }
                this.ShowMenu();

                if (skip) { this.SelectedIndex = 0; skip = false; }
                else { this.SelectedIndex = _itemsPerPage - 1; }
            }
            else
            {
                this.SelectedIndex++;
            }
        }
        else if (down == false)
        {

            if (this.SelectedIndex <= 0)
            {
                bool skip = false;

                if (this.Itemstoskip - this._itemsPerPage > -this._itemsPerPage)
                {
                    this.Itemstoskip -= 1;
                }
                else
                {
                    this.Itemstoskip = this._allItems.Count - _itemsPerPage;
                    skip = true;
                }

                this.ShowMenu();

                if (skip) { this.SelectedIndex = _itemsPerPage - 1; skip = false; }
                else { this.SelectedIndex = 0; }
            }
            else
            {
                this.SelectedIndex--;
            }
        }
    }

    private void ShowMenu()
    {
        int index = 1;
        foreach (IMenuItem i in this._allItems)
        {
            i.Description = index + "/" + this._allItems.Count.ToString();
            index += 1;
        }

        if (_allItems.Count < _itemsPerPage)
        {
            _itemsPerPage = _allItems.Count;
        }
        this.Items = this._allItems.Skip(this.Itemstoskip).Take(this._itemsPerPage).ToList();
        this.Initialize();
    }
}

may its a bit sloppy and not c++  but anyways i hope its understandable......
based on code of IncoCode

So you wait til you're selecting the item at the bottom of the list to start scrolling where as i start scrolling a items above that. or was that not the point you were getting at

i think my code and the new menu doesnt fit together so some strange things are happening ....   anyways i'll just wait for your final version and we'll see i guess!  thanks for taking the time


question: migration guide from Immutable.js for Map

Hello, since immutable.js does not appear to be maintained.
https://github.com/immutable-js/immutable-js/issues/1689
we wanted to migrate to something else.
We just use Map's   .get .set  .delete .map .filter  in web and react native apps.
With that light use,  it is hard to justify 58kb minimized file size that immutable js brings
(they were thinking of making imports splittable in the 2015 roadmap, and this is when we started using that library,  but that did not materialize )
With unmutable's map replacement (that I show below), our bundle size shows 9.8 kb minimized for unmutable (so we are saving about 50kb , based on our usage pattern)
I looked at the documentation for unmutable ( https://unmutable.blueflag.codes/api )
It seems that there are no 'replacement classes' so to to speak for immutable.
Is that undestanding correct? (I can see that it is possible to compose functionality from individual functions available in unmutable)
thank you
ps. this is how we are thinking of doing it at the moment (and then add types later...)
import get from 'unmutable/get';
import map from 'unmutable/map';
import filter from 'unmutable/filter';
import set from 'unmutable/set';
import deleteF from 'unmutable/delete';
import forEach from 'unmutable/forEach';

export class CommnIMap {
  constructor (data) {
    this._dataO= (data!=null)? data : {};
  }

  get (key) {
    return get(key)(this._dataO);
  }

  set (key, val) {
    return new CommnIMap(set(key, val)(this._dataO));
  }

  delete (key) {
    return new CommnIMap (deleteF(key)(this._dataO));
  }

  // map(mapper: (value: any, key: string|number, original: any) => newValue) =>
  map (mapperFn) {
    return new CommnIMap( map(mapperFn)(this._dataO));
  }

  filter ( fnValKey) {
    return new CommnIMap( filter(fnValKey)(this._dataO) );
  }

  forEach (fnValKey) {
    forEach(fnValKey)(this._dataO);
  }
}

Glad to hear it!


[Feature][Connector-V2][Clickhouse] Add clickhouse connector time zone key,default system time zone

Purpose of this pull request

Check list

[ ] Code changed are covered with tests, or it does not need tests for reason:
[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[ ] If you are contributing the connector code, please check that the following files are updated:

Update change log that in connector document. For more details you can refer to connector-v2
Update plugin-mapping.properties and add new connector information in it
Update the pom file of seatunnel-dist

[ ] Update the release-note.

[Bug] [zeta] clickhouse to Doris，8 hours difference in datetime type #4935


[contracts] Raid tx error

To reproduce:

Realm A w/ defender army
Realm B attacks realm A
Realm B army tries to raid

Tx went through on 2nd raid attempt
0: Error in the called contract (contract address: 0x03256bd286aff3d2edc85513b69865526ee90f297ac27bbb726e381d854437a0, class hash: 0x05400e90f7e0ae78bd02c77cd75527280470e2fe19c54970dd79dc37a9d3645c, selector: 0x015d40a3d6ca2ac30f4031e42be28da9b056fef9bb7357ac5e85627ee876e5ad):
Error at pc=0:4573:
Cairo traceback (most recent call last):
Unknown location (pc=0:67)
Unknown location (pc=0:1835)
Unknown location (pc=0:2478)
Unknown location (pc=0:3255)
Unknown location (pc=0:3809)
Unknown location (pc=0:3795)

1: Error in the called contract (contract address: 0x01c3310b831c666aeeefbe9ec95fe3140cb32ecb1a279b7898782b0e450c79a8, class hash: 0x03ffe895aab1e77ab9772da474ac6a0ed785c9226973c9d8b623962682c96e1d, selector: 0x018f505bd039d3fbc20655d8359bcda825bd8fac64a17ead4b26d26917199428):
Execution failed. Failure reason: "resource locked for entity 39".

    at ee.waitForTransactionWithCheck (index.ts:72:56)
    at async ee.executeAndCheckTransaction (index.ts:58:27)
    at async ee.battle_leave_and_pillage (index.ts:593:26)
    at async Object.battle_leave_and_pillage (createSystemCalls.ts:260:5)
    at async BattleManager.pillageStructure (BattleManager.ts:271:7)
    at async handleRaid (BattleActions.tsx:83:5)```

Seems like tx fails on successful raids, and not on failures

We've discussed this with @credence0x, it might already be closed by #1385


How to add relaton of postId to uploaded file

Hi,
I am trying to find a way to attach the postId to the uploaded file (a video) so then each video will belong to his specific post..  I am using CollectionFS and the amazon s3 package, the upload and preview all works well but can't make the relation between the post and video.
Really appreciate any help

You could add a property to the fileObj on upload..
  var fileObj = new FS.File(file);
  fileObj.userId = {userId: "1234"};
  Images.insert(fileObj,...);

This meteor pad example makes use of it
http://meteorpad.com/pad/zzpdePANFghRbGGmD/File Upload with Submit


Is there a way to set the cc_lang_pref in _youTubeReady()?

Expected behaviour
I want to be able to change the prefered cc language. Is this possible through the options object? In any other way that does not include messing with the source code?
Actual behaviour
English as permanently set as the cc language
Environment

Browser: Google Chrome
Version: ?
Operating System: Mac OS
Version: ?

Players affected:

[ ] HTML5 Video
[ ] HTML5 Audio
[x] YouTube
[ ] Vimeo

Steps to reproduce
Play a YouTube video with captions

I've added this in v3 but do you have a video with captions I could test with by any chance?

I've added this to v3 (it pulls from the same language as the caption settings) so assuming it's fixed? Feel free to re-open if that's not the case 👍


Segmentation Fault When Using NaiveGraphlab Engine.

Hi,
I am trying to implement label propagation using the naive Graphlab engine. It works fine when I run it on one core and one process. However, i get segmentation fault when i try to run it on more than one core.
My first attempt was this:
struct label_counter {
  // A map that counts the number of time each label is used.
  std::map<std::string, int> label_count;

  explicit label_counter(std::string label) {
    if (label != Unknown)
      label_count[label] = 1;
  }

  explicit label_counter() {}

  label_counter& operator+=(const label_counter& other) {
    for ( std::map<std::string, int>::const_iterator iter = other.label_count.begin();
              iter != other.label_count.end(); ++iter ) {
            label_count[iter->first] += iter->second;
    }

    return *this;
  }
};

using G = Graph<CCData,Empty>;

struct LabelPropagation : public GraphlabVertexProgram<G, label_counter> {
  bool do_scatter;
  std::string changelabel;

  LabelPropagation(Vertex& v) {
    do_scatter = false;
  }

  bool gather_edges(const Vertex& v) const { return true; }

  Gather gather(const Vertex& v, Edge& e) const {
    return label_counter(v->label);
  }

  void apply(Vertex& v, const Gather& total) {
    int maxCount = 0;
    std::string maxLabel = v->label;

    // Figure out which label of the vertex's neighbors' labels is most common
    for ( std::map<std::string, int>::const_iterator iter = total.label_count.begin();
          iter != total.label_count.end(); ++iter ) {
        if (iter->second > maxCount) {
          maxCount = iter->second;
          maxLabel = iter->first;
        }
    }

    // All of neighbours are unknown, or it doesn't have nieghbour
    if (maxLabel == Unknown) {
      //LOG(INFO) << "vertex id:  unknown ";// << v->id << " Label is unknown.";
      do_scatter = false;
      v->activate();
    } else if (maxLabel.compare(v->label) != 0) {
        v->label = maxLabel;
      //  LOG(INFO) << "vertex id: found"; // << v->id << " Label is: ." << v->label;
        changelabel = v->label;
        do_scatter = true;
    } else {
      // LOG(INFO) << "vertex id: "; // << v->id << " Label is: ." << v->label << " lebel didn't change";
      do_scatter = false;
    }
  }

  bool scatter_edges(const Vertex& v) const { return do_scatter; }

  Gather scatter(const Edge& e, Vertex& target) const {
    // I think this is not needed.
    return label_counter(changelabel);

  }
};

Which causes segmentation fault when used on more than one node. I used freeze option and attached gdb, I see the below stack:
#9  0x00000000005b933e in _ZNSt8_Rb_treeIiSt4pairIKiiESt10_Select1stIS2_ESt4lessIiESaIS2_EE14_M_create_nodeIJRKS2_EEEPSt13_Rb_tree_nodeIS2_EDpOT_ ()
#10 0x00000000005b57a1 in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_clone_node(std::_Rb_tree_node<std::pair<int const, int> > const*) ()
#11 0x00000000005b0480 in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_copy(std::_Rb_tree_node<std::pair<int const, int> > const*, std::_Rb_tree_node<std::pair<int const, int> >*) ()
#12 0x00000000005b04bf in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_copy(std::_Rb_tree_node<std::pair<int const, int> > const*, std::_Rb_tree_node<std::pair<int const, int> >*) ()
#13 0x00000000005a9c78 in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_Rb_tree(std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > > const&) ()
#14 0x00000000005a3a55 in std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > >::map(std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > const&) ()
#15 0x000000000059d2e7 in label_counter::label_counter(label_counter const&) ()
#16 0x000000000059d30e in void NaiveGraphlabEngine<Grappa::Graph<CCData, Grappa::Empty>, LabelPropagation>::run_sync<CCData, Grappa::Empty>(GlobalAddress<Grappa::Graph<CCData, Grappa::Empty> >)::{lambda(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&)#3}::operator()(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&) const::{lambda(Grappa::Graph<CCData, Grappa::Empty>::Edge&)#1}::operator()(Grappa::Graph<CCData, Grappa::Empty>::Edge) const::{lambda(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&)#1}::operator()(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&) const ()
#17 0x00000000005a9d2d in decltype ({parm#2}(*(({parm#1}.pointer)()))) Grappa::impl::call<(Grappa::SyncMode)1, &(Grappa::impl::local_gce), Grappa::impl::Vertex<CCData, Grappa::Empty, false>, void, void NaiveGraphlabEngine<Grappa::Graph<CCData, Grappa::Empty>, LabelPropagation>::run_sync<CCData, Grappa::Empty>(GlobalAddress<Grappa::Graph<CCData, Grappa::Empty> >)::{lambda(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&)#3}::operator()(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&) const::{lambda(Grappa::Graph<CCData, Grappa::Empty>::Edge&)#1}::operator()(Grappa::Graph<CCData, Grappa::Empty>::Edge) const::{lambda(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&)#1}>(GlobalAddress<Grappa::impl::Vertex<CCData, Grappa::Empty, false> >, void NaiveGraphlabEngine<Grappa::Graph<CCData, Grappa::Empty>, LabelPropagation>::run_sync<CCData, Grappa::Empty>(GlobalAddress<Grappa::Graph<CCData, Grappa::Empty> >)::{lambda(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&)#3}::operator()(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&) const::{lambda(Grappa::Graph<CCData, Grappa::Empty>::Edge&)#1}::operator()(Grappa::Graph<CCData, Grappa::Empty>::Edge) const::{lambda(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&)#1}, void (Grappa::impl::Vertex<CCData, Grappa::Empty, false>::*)({lambda(Grappa::impl::Vertex<CCData, Grappa::Empty, false>&)#1}&) const)::{lambda()#1}::operator()() const ()
#18 0x00000000005b5827 in _ZZN6Grappa4impl11SpecializerILNS_8SyncModeE1EXadL_ZNS0_9local_gceEEEZNS0_4callILS2_1EXadL_ZNS0_9local_gceEEENS0_6VertexI6CCDataNS_5EmptyELb0EEEvZZZN19NaiveGraphlabEngineINS_5GraphIS5_S6_EE16LabelPropagationE8run_syncIS5_S6_EEv13GlobalAddressINS9_IT_T0_EEEENKUlRS7_E1_clESJ_ENKUlRNSA_4EdgeEE_clESM_EUlSJ_E_EEDTclfp0_decldtfp_7pointerEEESE_IT1_ET3_MSS_KFT2_RSQ_EEUlvE_E4callEsSX_MSX_KFvvEENKUlvE_clEv ()
#19 0x00000000005eba59 in _ZN6Grappa7MessageIZNS_4impl11SpecializerILNS_8SyncModeE1EXadL_ZNS1_9local_gceEEEZNS1_4callILS3_1EXadL_ZNS1_9local_gceEEENS1_6VertexI6CCDa---Type <return> to continue, or q <return> to quit---bt
taNS_5EmptyELb0EEEvZZZN19NaiveGraphlabEngineINS_5GraphIS6_S7_EE16LabelPropagationE8run_syncIS6_S7_EEv13GlobalAddressINSA_IT_T0_EEEENKUlRS8_E1_clESK_ENKUlRNSB_4EdgeEE_clESN_EUlSK_E_EEDTclfp0_decldtfp_7pointerEEESF_IT1_ET3_MST_KFT2_RSR_EEUlvE_E4callEsSY_MSY_KFvvEEUlvE_E20deserialize_and_callEPc ()
#20 0x000000000061bb9b in Grappa::impl::MessageBase::deserialize_and_call(char*) ()
#21 0x00000000006151df in Grappa::impl::RDMAAggregator::deaggregate_buffer(char*, unsigned long) ()
#22 0x0000000000616cda in Grappa::impl::RDMAAggregator::receive_buffer(Grappa::impl::RDMABuffer*) ()
#23 0x0000000000615e73 in Grappa::impl::RDMAAggregator::receive_worker() ()
#24 0x0000000000613697 in Grappa::impl::RDMAAggregator::activate()::{lambda()#2}::operator()() const ()
#25 0x000000000061a618 in void Grappa::impl::worker_heapfunctor_proxy<Grappa::impl::RDMAAggregator::activate()::{lambda()#2}>(Grappa::Worker*,

Then, I thought maybe that is because I am using map and allocate the memory locally, I changed the code to:
struct label_counter {
  // A map that counts the number of time each label is used.
  GlobalAddress<int> label_count;

  explicit label_counter(int label) {
    label_count = global_alloc<int>(totalGroup);
    // Init the array
    for (int i =0 ; i < totalGroup; i++) {
       delegate::write((label_count + i ), 0);
    }

    if (label != Unknown)
      delegate::write(label_count + label,  1);
  }

  explicit label_counter() {}

  label_counter& operator+=(const label_counter& other) {
    for (int i = 0; i < totalGroup; i++) {
      int currentValue = delegate::read(other.label_count + i);
      delegate::write((label_count + i), currentValue + 1);
    }

    return *this;
  }
};

using G = Graph<CCData,Empty>;

struct LabelPropagation : public GraphlabVertexProgram<G, label_counter> {
  bool do_scatter;
  int changelabel;

  LabelPropagation(Vertex& v) {
    do_scatter = false;
  }

  bool gather_edges(const Vertex& v) const { return true; }

  Gather gather(const Vertex& v, Edge& e) const {
    return label_counter(v->label);
  }

  void apply(Vertex& v, const Gather& total) {
    int maxCount = 0;
    int maxLabel = v->label;

    for (int i = 0; i < totalGroup; i++) {
        int labelValue = delegate::read(total.label_count + i);
        // LOG(INFO) << "label value: is " << labelValue;
        if (labelValue > maxCount) {
          maxCount = labelValue;
          maxLabel = i;
        }
    }

    // All of neighbours are unknown, or it doesn't have nieghbour
    if (maxLabel == Unknown) {
      //LOG(INFO) << "vertex id:  unknown ";// << v->id << " Label is unknown.";
      do_scatter = false;
      v->activate();
    } else if (maxLabel != v->label) {
        v->label = maxLabel;
        //LOG(INFO) << "vertex id: found"; // << v->id << " Label is: ." << v->label;
        changelabel = v->label;
        do_scatter = true;
    } else {
       //LOG(INFO) << "vertex id: "; // << v->id << " Label is: ." << v->label << " lebel didn't change";
      do_scatter = false;
    }
  }

  bool scatter_edges(const Vertex& v) const { return do_scatter; }

  Gather scatter(const Edge& e, Vertex& target) const {
    // I think this is not needed.
    return label_counter(changelabel);

  }
};

Which still works fine on one node, but now I am getting this error when I run it on more than one node:

0: I0511 17:11:36.921659 25537 lp1.cpp:207] Finished loading graph
0: I0511 17:11:36.921665 25537 lp1.cpp:209] "construction_time": 19.1613
0: I0511 17:11:36.921675 25537 lp1.cpp:214] start laoding seed file
0: I0511 17:11:40.701381 25537 lp1.cpp:166] Loaded Seed file. Count: 1005593
0: I0511 17:12:12.902834 25537 lp1.cpp:224] Init the labels
0: I0511 17:12:14.383291 25537 lp1.cpp:244] Init is complete
2: F0511 17:12:14.533294 25282 TaskingScheduler.hpp:154] Check failed: in_no_switch_region_ == false (1 vs. 0) Trying to context switch in no-switch region
1: F0511 17:12:14.534068 25538 TaskingScheduler.hpp:154] Check failed: in_no_switch_region_ == false (1 vs. 0) Trying to context switch in no-switch region
3: F0511 17:12:14.533303 25283 TaskingScheduler.hpp:154] Check failed: in_no_switch_region_ == false (1 vs. 0) Trying to context switch in no-switch region
1: *** Check failure stack trace: ***
1: *** Aborted at<PHONE_NUMBER> (unix time) try "date -d @1431389534" if you are using GNU date ***

I feel there is something big in the design of this that I am missing. Appreciate your help.

Okay, so you were correct that the problem you had initially was that the map was allocating memory locally, and we don't deep-copy so you were dereferencing pointers to memory that's only valid on another node.
This leads to the problem you had with your second version. I can't remember exactly how I implemented the graphlab engine, but my assumption was that vertex programs wouldn't do their own communication (delegates) in the gather or scatter phases (i.e. these are already doing delegate calls to move data for the scatter/gather). So what it's telling you is that you're trying to do a delegate (in this case your delegate::write inside the label_counter constructor) from inside another delegate (or message) (inside the graphlab engine). I think you should be alright doing delegates in apply (but I'd have to check the code).
Is there any way to bound the number of labels in a given label_counter? To fix your gather/scatter issues, the easiest solution would be to find a way to make the label_counter class a fixed size and have it hold the labels inside it (i.e. set a maximum number of labels, stored statically as part of the object). This won't be super efficient, but might get you off the ground. The other issue is that strings are bad too, because they also just hold a pointer to other memory with the content of the string. It'd be better if you could reduce the string labels to something like a single char or an int. For instance, something like:
struct label_counter {
  pair<int,int> label_counts[8];
  // ...
}

The ideal solution would be to hack NaiveGraphlabEngine to serialize the contents of label_counter automatically (using boost::serialize). This is significantly more advanced. If you feel up to it, I can walk you through it in more detail — I just don't have the time to implement and debug it myself.
Alternatively, you could give up on the NaiveGraphlabEngine and just implement this manually. The only "magic" that the NaiveGraphlabEngine does is copy the vertex/edge/reduce data automatically for the gather and scatter phases, so if you're going to need to move the data yourself anyway, then you may as well do it all yourself.
One other note: you should not be using global_alloc in label_counter as you are here:
label_count = global_alloc<int>(totalGroup);

The global_alloc method is for allocating data that is spread out over multiple nodes — it will be strided across nodes every 64 bytes. What you want is to use locale_alloc to allocate memory on the node where that vertex lives, and then wrap that in a GlobalAddress so you can reference it from other nodes. This creates what we call a "2D" global address (a pointer and a node). So instead, what I would do is something like:
int* local_ptr = locale_alloc<int>(totalGroup);
label_count = make_global(local_ptr);

Hi Brandon,
Thanks a lot for the help. I changed the label_counter to a static array, and it no longer give me any error. However, I got to another error. Again, when I try it on one core it works fine, but when I try it on more cores, my algorithm won't converge. It stays with constants number of active nodes until it reaches maximum number of iterations.
It seems that I have some kind of deadlock.

// Graph label data
struct CCData : public GraphlabVertexData<CCData> {
  int label;
  VertexID id;
};

struct label_counter {
  int label_count[number_of_groups];

  explicit label_counter(int label) : label_count() {

    CHECK(label < number_of_groups) << "Label should be less than supported number of groups.";
    if (label != Unknown)
      label_count[label] = 1;
  }

  explicit label_counter() {
    // Init the array
    for (int i =0 ; i < number_of_groups; i++) {
       label_count[i] = 0;
    }
  }

  label_counter& operator+=(const label_counter& other) {
    for (int i = 0; i < number_of_groups; i++) {
      label_count[i] += other.label_count[i];
    }

    return *this;
  }
};

using G = Graph<CCData,Empty>;

struct LabelPropagation : public GraphlabVertexProgram<G, label_counter> {
  bool do_scatter;
  int changelabel;

  LabelPropagation(Vertex& v) {
    do_scatter = false;
  }

  bool gather_edges(const Vertex& v) const { return true; }

  Gather gather(const Vertex& v, Edge& e) const {
    return label_counter(v->label);
  }

  void apply(Vertex& v, const Gather& total) {
    int maxCount = 0;
    int maxLabel = v->label;

    for (int i = 0; i < number_of_groups; i++) {
        if (total.label_count[i] > maxCount) {
          maxCount = total.label_count[i];
          maxLabel = i;
        }
    }

    // All of neighbours are unknown, or it doesn't have nieghbour
    if (maxLabel == Unknown) {
      //LOG(INFO) << "vertex id:  unknown " << v->id << " Label is unknown.";
      do_scatter = false;
      v->activate();
    } else if (maxLabel != v->label) {
        v->label = maxLabel;
        //LOG(INFO) << "vertex id: found"; // << v->id << " Label is: ." << v->label;
        changelabel = v->label;
        do_scatter = true;
    } else {
       //LOG(INFO) << "vertex id: " << v->id << " Label is: ." << v->label << " lebel didn't change";
      do_scatter = false;
    }
  }

  bool scatter_edges(const Vertex& v) const { return do_scatter; }

  Gather scatter(const Edge& e, Vertex& target) const {
    return label_counter(changelabel);
  }
};

Here is how I initialize my labels. I used to use parallel loops, but I changed it to sequential thinking it might help with this problem.

      GRAPPA_TIME_REGION(total_time) {
        LOG(INFO) << "Init the labels";
        LOG(INFO) << "Size of the graph is: " << g->nv;
        int totalUnlabeled = 0;
        for (int64_t i = 0; i < g->nv ; i++) {
          if (seeds.find(i) == seeds.end()) {
            totalUnlabeled++;
             delegate::call(g->vs+i, [i](G::Vertex& v) {
                 v.data.label = -1;
                 v.data.id = i;
                 v.data.activate();
             });
          } else {
              auto label = seeds[i];
              CHECK(label < number_of_groups);
              CHECK(label >= 0);
              delegate::call(g-> vs+i, [label,i] (G::Vertex& v){
                v.data.label = label;
                v.data.id = i;
                v.data.deactivate();
              });
          }
        }
        LOG(INFO) << "Total unlabeled is:" << totalUnlabeled;
        LOG(INFO) << "Init is complete";
        NaiveGraphlabEngine<G,LabelPropagation>::run_sync(g);
      }
    }

This might be because of a hack I had to add in order to make gather work even though we only have outgoing edges in our graph representation, and it may have something to do with how I implemented delta caching, too. Double-checking the code for NaiveGraphlabEngine (around line 151), I see that gather is only called in the first iteration and then it's just apply and scatter after that. This is because in my implementation I'm counting on being able to use "delta caching" to avoid doing the full gather every iteration (because without in-edges it requires traversing all the edges every time). Instead, what you can do is return a "delta" value from scatter that represents all the changes you made to any of the label counts during apply. This is slightly more explicit than how Graphlab implements delta caching, but returning the delta from the scatter makes it easiest for my NaiveGraphlabEngine to capture that and update the right fields.
It looks like what you're doing is returning a single int (changelabel), which doesn't seem like enough to represent the changes to all the counts. I haven't given it a ton of thought, but it seems like this delta-caching approach should be fine for your label propagation application.

I have to give some thought on how I can do a delta caching. Not sure how I can fit it in my current design.
On another note, I thought we are assuming that we only support undirected graphs. When I create my graph I make sure that for each edge I have both in and out edges i.e. (x y) and (y x) are both guaranteed to be defined. That should give us the ability to gather in each iterations. Am I right?
Thanks again :)

I make sure that for each edge I have both in and out edges i.e. (x y) and (y x) are both guaranteed to be defined. That should give us the ability to gather in each iterations. Am I right?

You should be able to do it, but you'll have to do things with scatter still. My NaiveGraphEngine doesn't know that you've made your graph undirected by duplicating edges. In fact, if you have data on the edges it's going to be a problem because the two different directions may have different data. So, similarly, it doesn't know that it can treat the out-edges as in-edges. You should be able to just merge the label_counters (with +=) in the scatter step if you stick the data in the Edge in the right way.

Another thing that I cannot still understand is, why does this work on one Core? Usually after 5-6 iteration it converges. Shouldn't we have the same problem on one core?

That's a great question. I don't know. I can't think of what might be going wrong in that case.

At first I wasn't sure what you mean by delta caching, but after reading your code and comparing it with what I do here. It seems that I am actually using delta caching. In the scatter, I create a label_counter object which has 1 for the label that we just found and zero for all other. When we do scatter, we do a post_delta of this object, so the value on the cache of the neighbor vertex would be what ever it is plus one for the new label. I think that is why it works fine on one core. If I wan't using the delta caching the algorithm wouldn't converge even on one node.
I noticed that you are using async calls in the _do_scatter function, and the loop that calls it, is also async. I am wondering wouldn't this cause problem because we might actually look at vertex before it is updated.

What vertex update are you talking about? The forall that does apply will finish completely before scatters are done. Is there something you think might be happening out of order in the scatter phase? It's certainly possible, but I don't see any race conditions — all of the scatters are supposed to happen in parallel, and your scatter function doesn't seem to have any problem with this. Something like that could explain the behavior, but it's one of many, unfortunately. I don't have an answer after skimming it.

I might be wrong, but:
static void _do_scatter(const VertexProg& prog_copy, Edge& e,
                  Gather (VertexProg::*f)(const Edge&, Vertex&) const) {
    auto e_id = e.id;
    auto e_data = e.data;
    call<async>(e.ga, [=](Vertex& ve){
      auto local_e_data = e_data;
      Edge e = { e_id, g->vs+e_id, local_e_data };
      auto gather_delta = prog_copy.scatter(e, ve);
      prog(ve).post_delta(gather_delta);
    });
  }

and then
    forall(g, [=](Vertex& v){
        if (v->active_minor_step) {
          v->active_minor_step = false;
          auto prog_copy = prog(v);
          // scatter
          forall<async>(adj(g,v), [=](Edge& e){
            _do_scatter(prog_copy, e, &VertexProg::scatter);
          });
        }
      });

They are both async, it means that we don't finish scatter to neighbors of one node before going to the next node.

right, all of the scatters of all the vertices will be in parallel (by design! that's the whole point of the graphlab gather-apply-scatter model). The only potential gotcha is if you're reading and updating the same state on a vertex in the scatter phase. In this case you're only reading changelabel and updating the delta cache. All the delta cache updates are still atomic so there aren't any lost updates. Then we wait for all the scatters to finish before moving on to the next iteration of the while loop and next apply stage.

(we fixed this offline a while ago, so closing this issue)


IDEA: Before cache persist and cache fetch callbacks

When working on my project, there were a few API calls that would be in an error state, but the API creator responded with an HTTP 200 instead of a 4xx or 5xx.
It would be nice if we could provide custom callbacks before persisting or before using a cache to transform the response -- in my case either changing the status code/being able to tell httpdisk that it should be an error code based on the contents of the message.
Similarly, when popping a value out of the cache, it would be nice to be able to mutate the response or tell httpdisk that it's a :miss on custom logic.
What do you think?

It's amazing the kinds of problems we have to deal with in the wild... Buggy servers are quite common, producing either weird transient errors or permanent anomalies. There are a few ways to handle this sort of issue:

A before_write callback as proposed above. The callback could manually set the status code to 500 or 999. We could also cancel the write if the callback returns false, to facilitate retries. If the response is JSON it would require parsing each body twice to detect errors, unfortunately. That would be a deal breaker for some crawls.
Custom options when setting up the retry middleware. Same performance implications as (1), though.
After the user parses the response, call uncache on httpdisk. This has a few advantages - parsing only occurs in one place, and the body only gets parsed one time. Calling uncache is a challenge, though, since you have to carefully replicate the original request to generate the correct cache key. Clunky.

Have you seen sinew? It's built on top of httpdisk. It's not well publicized yet either. Sinew 4 is right around the corner, with some important features that will make this kind of thing much cleaner 😄

I like approach 3, especially if there was an after_write callback and an easy to use object/method that would allow you to call uncache for a given FaradayResponse object
Either way, I’m fine for now.
sinew looks neat and would have made my project go faster. I’ll keep it in mind for next time.


Need support baud rate greater than 38400 on posix systems.

pyserial does this:
https://github.com/pyserial/pyserial/blob/master/serial/serialposix.py

pyserial does this:
https://github.com/pyserial/pyserial/blob/master/serial/serialposix.py

I'll see what I can do. The thing that I'm trying to avoid is for end users having to do stuff like:
when defined(linux):
  port.baudRate = 38400
else:
  port.baudRate = 9600

Awesome, I’d be happy to see a posix only patch for now, I can look at the Windows side at a later point.

On 1 Aug 2018, at 11:22, Jeff Ciesielski<EMAIL_ADDRESS>wrote:
FYI: I recently added extended baudrates to the stdlib: nim-lang/Nim#8322.
I'd very much like to see support for this added (I'm even happy to add it on the posix side) as 38400 is dreadfully slow
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.


Bump version to v1.14.2-pre1

Bump version to v1.14.2-pre1. Stopped adding pre1 to core version.

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE                                    FILE SIZE
 ++++++++++++++ GROWING                      ++++++++++++++
   +13%     +11 src/cpp/common/version_cc.cc     +11   +13%
       +28%     +11 grpc::Version[abi:cxx11]         +11   +28%

 -+-+-+-+-+-+-+ MIXED                        +-+-+-+-+-+-+-
  -0.0%     -11 [None]                          +109  +0.0%

  [ = ]       0 TOTAL                           +120  +0.0%

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE                                    FILE SIZE
 ++++++++++++++ GROWING                      ++++++++++++++
   +13%     +11 src/cpp/common/version_cc.cc     +11   +13%
       +28%     +11 grpc::Version[abi:cxx11]         +11   +28%

 -+-+-+-+-+-+-+ MIXED                        +-+-+-+-+-+-+-
  -0.0%     -11 [None]                          +109  +0.0%

  [ = ]       0 TOTAL                           +120  +0.0%

[trickle] No significant performance differences

Known Tsan C failure https://github.com/grpc/grpc/issues/16352
Python bazel failure seems related to infra.
Known MacOS failures on this branch.


[FEATURE REQUEST] Saving Eddy-outputs in output folder for use in EddyQC

In order to use FSL's EddyQC with data preprocessed using qsiprep, I need to save the working directory locally using -w WORK_DIR.
Is there an option to save the necessary eddy files (e.g. index.txt, acqparams.txt, etc.) to a folder in a subject's output directory?

Hi @mattcieslak, I just tried qsiprep for the first time, it is a great tool! Thank you for providing it to the community.  I would be interested to try the qsiprep_group_report functionality. How do you launch this command?
Is it possible to use it via a singularity image of qsiprep? Sorry if it is a silly question!

Hi @mattcieslak, thanks again for this great pipeline!
I am struggling a little bit to produce the eddy files, and wondered what I might be doing wrong? We have a nipype file that sets remove_unnecessary_outputs to false, and placed in the root directory, but still cannot produce the eddy files. Do you have any thoughts? Is there an additional argument we should be using to run qsiprep? The --config flag does not seem to be recognised.

Hi @CaitlinLloyd, the nippy.cfg file with remove_unnecessary_outputs = false retains all the unnecessary outputs in the working directory where qsiprep is running. But depending on your computing architecture (eg HPC cluster with job sumibbsion) this may get automatically deleted when the qsiprep job finishes. I had to string together commands to copy the entire working directory from the scratch space on our cluster back to my local directory, after which I could copy out the eddy output files. This working directory is 5-10GB depending on the size of your dataset - which I guess is why qsiprep by default doesn't retain all this stuff. So you may have to watch your own storage quota if running a lot of them.

I should have added - once you've saved the working directory, you'll find the eddy output in:
<workdir_name>/qsiprep_wf/single_subject_<subjID>_wf/dwi_preproc_<scan_ID>_wf/hmc_sdc_wf/eddy
In addition to that entire folder I copy these additional files from the working directory to run eddy_quad:
<workdir_name>/qsiprep_wf/single_subject_<subjID>_wf/dwi_preproc_<scan_ID>_wf/hmc_sdc_wf/gather_inputs/eddy*.txt <workdir_name>/qsiprep_wf/single_subject_<subjID>_wf/dwi_preproc_<scan_ID>_wf/hmc_sdc_wf/pre_eddy_b0_ref_wf/enhance_and_mask_b0/topup_imain_corrected_avg_mask.nii.gz <workdir_name>/qsiprep_wf/single_subject_<subjID>_wf/dwi_preproc_<scan_ID>_wf/hmc_sdc_wf/topup_to_eddy_reg/topup_reg_image_flirt.mat <workdir_name>/qsiprep_wf/single_subject_<subjID>_wf/dwi_preproc_<scan_ID>_wf/hmc_sdc_wf/topup/fieldmap_HZ.nii.gz
as well as the bvals from qsiprep output and the slspec file from a common location.

Thank you for this! I am still not seeing anything other than these files in the eddy subfolder within qsiprep workingdir:
_inputs.pklz
_node.pklz
_report
command.txt
eddy_corrected.eddy_command_txt
eddy_corrected.eddy_post_eddy_shell_PE_translation_parameters
eddy_corrected.eddy_values_of_all_input_parameters
As far as I can tell nothing is being removed automatically. I think maybe we have the cfg file set up incorrectly or not visible to the container somehow? Would you mind sharing an example of the nipype.cfg file, just so I can rule that out?

Are you running this via docker or singularity? I wonder if the nipype config isn't visible inside the container

docker, here is command in case you see anything odd!
nohup /usr/bin/docker service create --constraint 'node.hostname==gpu' --replicas 1 --reserve-cpu 8 --reserve-memory 16g --mode replicated --restart-condition none --name=qsi_id -un$qpath --mount type=bind,source=$indir,destination=/input,readonly=true --mount type=bind,source=$outdir,destination=/output --mount type=bind,source=$workdir,destination=/workdir --mount type=bind,source=/DATA /bin/freesurferlicense.txt,destination=/opt/freesurfer/license.txt,readonly=true --mount type=bind,source=/DATA/testing/nipype.cfg,destination=/home/root/.nipype/nipype.cfg pennbbl/qsiprep /input /output participant -w /workdir $addarg > $log 2>&1 &

I've actually never seen a docker service being used for qsiprep. What happens if you mount the config file at /root/.nipype/nipype.cfg?

Sadly that does not work either :/ we have tried a few different locations without too much luck

It looks like you can set where nipype will look for the config file with the NIPYPE_CONFIG_DIR environment variable. If you set this in your container and mount the config file at ${NIPYPE_CONFIG_DIR}/nipype.cfg then it should work regardless of which user you're running qsiprep as

It must be that your qsiprep job is not seeing the nipype config file. Here's what the eddy sub-folder in the working directory contained on one of my recent runs:
_0x13da34fc6dfdee0ae211ccc7f4259587.json
command.txt
eddy_corrected.eddy_cnr_maps.nii.gz
eddy_corrected.eddy_command_txt
eddy_corrected.eddy_mbs_first_order_fields.nii.gz
eddy_corrected.eddy_movement_over_time
eddy_corrected.eddy_movement_rms
eddy_corrected.eddy_outlier_free_data.nii.gz
eddy_corrected.eddy_outlier_map
eddy_corrected.eddy_outlier_n_sqr_stdev_map
eddy_corrected.eddy_outlier_n_stdev_map
eddy_corrected.eddy_outlier_report
eddy_corrected.eddy_parameters
eddy_corrected.eddy_post_eddy_shell_alignment_parameters
eddy_corrected.eddy_post_eddy_shell_PE_translation_parameters
eddy_corrected.eddy_residuals.nii.gz
eddy_corrected.eddy_restricted_movement_rms
eddy_corrected.eddy_rotated_bvecs
eddy_corrected.eddy_values_of_all_input_parameters
eddy_corrected.nii.gz
_inputs.pklz
_node.pklz
_report
result_eddy.pklz
I am running eddy with just about everything - cuda, slice-to-vol correction with mporder=8, cnr maps and residuals on, movement-by-susceptability correction, so that adds to the plethora of files. Your eddy job may not be generating all of these.


Started to remove Convict.

Merging the Settings type with the Configuration type

will quite possibly find the odd problem caused by this - but is hard to detect without just merging and getting on with using. (for instance there may be deployment in AWS issues)