Issue with registerable.register_user function change

I've been working on upgrading from Flask Security 3.0.0 to Flask Security Too 3.4.2.
There was change in this commit (https://github.com/Flask-Middleware/flask-security/commit/3c5fd22bf45c21d769544344d985df67ffbf2b09#diff-2e9556678333fc9e7dfe58473f411bb4) that changed how the register_user function works. Instead of a dict-like object, it now takes in the register form, which is fine.
However I'm encountering a problem with it since my RegisterForm includes some custom fields that belong to the user model, but can't be processed correctly since it's a SQLAlchemy 1 to many field, much like roles. So that will break the form.
My current solution is to change the name of the field to something else that doesn't match the model field. So I can handle the processing after calling the register function.
Granted in the old version where register_user accepted a dict was not much better, since I had to manually remove this field from to_dict()
Example (psuedocode):
User:
   # normal fields
   hobbies # which is a 1 to many relationship

Hobby:
  id
  name

RegisterForm(FlaskForm, RegisterFormMixin, UniqueEmailFormMixin, NewPasswordFormMixin):
  hobbies = MultiSelectField()

Interesting - do you have any thoughts on a better solution?

At the moment I do not have a better solutions in mind. But here are some other options I considered.
One option is in my register form, I guess I could create a custom WTForm field that returned my Hobbies as Hobby models.
Since I use SQL Alchemy, for this particular use case, I think I could use the SQL Alchemy Association Proxy (https://docs.sqlalchemy.org/en/13/orm/extensions/associationproxy.html)
Another option would be to add a "excluded_fields" parameter to register_user, which would allow me to specify "hobbies" as a field to ignore when creating the user.
A fourth option I considered is going back to allowing a dict. One of the benefits is that it allows me to pre-process the fields before passing them into the registration form. It allows me to bypass the form. I have also have use case where I register some user accounts in bulk.

Ultimately, I ended up creating my own version of the register_user function that accepted a dict, so sort of reverted to the previous functionality. I'm not sure that's the best solution in general, but for my very specific use case that was the best option for me.

This is very old and submitter has a working solution. I was thinking about this and it would seem another way to solve this is is to simply override the to_dict method (since in this case the app already has a custom RegisterForm and derives from RegisterFormMixin).
In that method the form args could be removed or actually manipulated so that the _datastore.create_user actually can do the entire action.


How can we achieve a  expandable list view 4 level deeper(nested) using json.

How can we achieve a 4 level expandable list view.
Our proposed solution
Can we add one more method "renderSubMenu"  like RenderHeader and RenderContent in Collapsable class for rendering the list view under list view.
What we want to achieve.
Want to implement a expandable table or list view. we have a json with 4 level expandable, want to display that json on list view using ur library collapsable.
What have done so far
Have achieved the 2 expandable level but not able to achieve on further. means on click on second level not able to load the 3 expandable level.
Here attached a sample image

Please help me..
Thanks in Advance.

@mohitgaur2  Did you try nesting multiple <Collapsible> instead of using the renderSubmenu?

Closing this as inactive as there's been no activity for over 30 days. If the problem re-occurs on the latest version of react-native-collapsible then ping me to reopen or create a new issue.

Nesting collapsibles is not working for me, the inner <Collapsible/> is disappearing when it gets toggled. Any help would be appreciated.


Add option to disable webcam feed in avatar-animator

I think that there is no need to show the webcam feed in avatar-animator other than for debug proposes. If possible, I would add an option to enable this and default to disabled.

I think for transparency and understanding that your camera video is captured it's good to show the feed by default but allow to disable it.

Done in #22


sonobuoy run --wait hangs on k3s environment on raspberry pis

What steps did you take and what happened:
I am trying to run tests on my k8s infrastructure running on raspberry pis using sonobuoy as described on the how to get started.

What did you expect to happen:
have k8s tests run on the edge.
Environment:

Sonobuoy version: v0.19.0
Kubernetes version: (use kubectl version): v1.19.2
Kubernetes installer & version: v.1.18.9+k3s1
OS (e.g. from /etc/os-release): Raspberry Pi OS

hi @CloudGrimm, thanks for reporting the issue! we currently support arm64, so I would expect Sonobuoy to work on Raspberry Pi 4. Is your hardware arm64?

Yes it is, a raspberry pi 4 model B

@CloudGrimm does docker pull sonobuoy/sonobuoy:v0.19.0 work in your setup?


Vue tries to observe enumerable but non-configurable properties

Vue.js version
1.0.16
Reproduction Link
https://jsfiddle.net/5sH6A/
Steps to reproduce
Add a non-configurable, but enumerable property to component data/props.
What is expected?
Vue.js should not be redefining properties that are not configurable in defineReactive as by virtue of being non-configurable, they cannot change anyway so observing them is pointless (and also throws an error).
This bug currently makes it impossible to add Three.js objects to Vue components without causing this error. However a current workaround is to set convertAllProperties = true, as this triggers checks to ensure that non-configurable properties are not redefined.
What is actually happening?
Uncaught TypeError: Cannot redefine property: <property name>

FYI the fiddle is blank?

Thanks @Mat-Moo, updated the link

It seems it's better to skip observing the entire object? I saw your other comment about a non-reactive object being converted when passed down as a prop. Maybe we need a way to explicitly declare non-reactive properties on a vm instance.

I agree that that would be a good feature, but think it should be separate to this fix, which fixes objects that you want to be reactive with some non-configurable properties.
I think the way to solve the 'non-reactive object' suggestion would be to never observe objects that are frozen (e.g., with Object.freeze), and can be tested with Object.isFrozen.

Maybe we need a way to explicitly declare non-reactive properties on a vm instance.

Yes please.

@saul makes sense, let's work off #2394 to fix this.
On the other hand, Vue already skips observing objects that are not extensible. But the user may have objects that should not be observed but should be mutable. So maybe a new option like nonReactive?
export default {
  // reactive data
  data () {
    return { a: 1 }
  },
  nonReactive () {
    return {
      vector: new THREE.Vector3( 1, 0, 0 )
    }
  }
})

Basically, this simply attaches the property to the instance without converting it - in addition, it should probably add a private marker to the object (if it's extensible) so that when the object is passed down as a prop, it doesn't get converted again.

For clarity should it be nonReactiveData so that it becomes an opposite to data? Maybe there could then be reactiveData (pseudo name for data)?

@yyx990803 Please consider to add nonReactive option. It will make working with external graphic lib much easier.


feat(github): add support for github enterprise

Add support for github enterprise with a new key api-url.
This feature needs to be checked properly since I don't have access to the github of an entreprise.
I just checked that it works correctly with github.com.
Closes #1812.

Looks good otherwise. We probably don't need to validate that it is a proper URL.

The aim isn't validate the URL.
It's to support both: http://api.github.com/ and http://api.github.com and to avoid guessing if you need to put (or not) the final /

It's to support both: http://api.github.com/ and http://api.github.com and to avoid guessing if you need to put (or not) the final /

Oh, that's not what I meant. It was just a general comment that we probably don't want to add extra validation code.
We should definitely force a / at the end there ;)

We should definitely force a / at the end there ;)

Okay. I'll add an exception.

Oh, that's not what I meant. It was just a general comment that we probably don't want to add extra validation code.

Ha my bad but I think curl does that verification for us no? (See the get method) If it isn't a valid URL an exception is thrown, I think.

Okay. I'll add an exception.

What do you mean? This is already in the code

I think curl does that verification for us no?

Probably, yeah

What do you mean? This is already in the code

Currently, if there is no /. I add it, but since we should force having a / at the end then I should add an exception no?
Damn I think I misunderstood everything... By 'we' you wanted to say 'polybar/ the code'...
Sorry

Yeah, I meant the code should add the /, just as it is now ;)

Updated to take the review into account.


Command PhaseScriptExecution failed with a nonzero exit code

I am facing this issue with my Flutter project where my Android version is working without any issue.  The --tree-shake-icons flag will be ignored.

Command PhaseScriptExecution failed with a nonzero exit code
`

Please help me. I am trying to solve this error for past one month and no luck so far :(

@Leoudayan
Can you provide flutter doctor -v, complete flutter run --verbose logs ? The log will be huge, so put that in a text file and share it here.
Also, provide below:

ios/Flutter/Generated.xcconfig

ios/Runner.xcodeproj/project.pbxproj

Hi Darshan,
Thanks for your reply. I have added the required files.
Please find the required files below,

ios/Flutter/Generated.xcconfig

// This is a generated file; do not edit or check into version control.
FLUTTER_ROOT=/Users/mr22726/Flutter/flutter/flutter
FLUTTER_APPLICATION_PATH=/Users/mr22726/Flutter/block_indy
COCOAPODS_PARALLEL_CODE_SIGN=true
FLUTTER_TARGET=lib/main.dart
FLUTTER_BUILD_DIR=build
SYMROOT=${SOURCE_ROOT}/../build/ios
FLUTTER_BUILD_NAME=1.2.0
FLUTTER_BUILD_NUMBER=4
EXCLUDED_ARCHS[sdk=iphonesimulator*]=arm64 i386
DART_OBFUSCATION=false
TRACK_WIDGET_CREATION=false
TREE_SHAKE_ICONS=true
PACKAGE_CONFIG=/Users/mr22726/Flutter/block_indy/.dart_tool/package_config.json

ios/Runner.xcodeproj/project.pbxproj

project-pbxproj.pdf

flutter --verbose
flutter run --verbose.pdf

@Leoudayan
From your verbose log, it seems to be failing due to below error:

[ +73 ms] No supported devices connected.
[ +89 ms] The following devices were found, but are not supported by this
project:
 Chrome (web) • chrome • web-javascript • Google Chrome
91.0.4472.106
 If you would like your app to run on web, consider running `flutter
create .` to generate projects for these platforms.

Did you try running as suggested in the error ?  If you would like your app to run on web, consider running flutter
create . to generate projects for these platforms.
Also, please share your ios/podfile as well.

Hi Darshan,
thanks for your reply. The problem only exist in the ios Xcode platform and Android code is working fine in the emulator.
Here is the ios/podfile.

Uncomment this line to define a global platform for your project
platform :ios, '9.0'
CocoaPods analytics sends network stats synchronously affecting flutter build latency.
ENV['COCOAPODS_DISABLE_STATS'] = 'true'
project 'Runner', {
'Debug' => :debug,
'Profile' => :release,
'Release' => :release,
}
def flutter_root
generated_xcode_build_settings_path = File.expand_path(File.join('..', 'Flutter', 'Generated.xcconfig'), FILE)
unless File.exist?(generated_xcode_build_settings_path)
raise "#{generated_xcode_build_settings_path} must exist. If you're running pod install manually, make sure flutter pub get is executed first"
end
File.foreach(generated_xcode_build_settings_path) do |line|
matches = line.match(/FLUTTER_ROOT=(.*)/)
return matches[1].strip if matches
end
raise "FLUTTER_ROOT not found in #{generated_xcode_build_settings_path}. Try deleting Generated.xcconfig, then run flutter pub get"
end
require File.expand_path(File.join('packages', 'flutter_tools', 'bin', 'podhelper'), flutter_root)
flutter_ios_podfile_setup
target 'Runner' do
use_frameworks!
use_modular_headers!
flutter_install_all_ios_pods File.dirname(File.realpath(FILE))
end
post_install do |installer|
installer.pods_project.targets.each do |target|
flutter_additional_ios_build_settings(target)
target.build_configurations.each do |config|
config.build_settings['IPHONEOS_DEPLOYMENT_TARGET'] = '9.0'
end
end
end

@Leoudayan
In your podfile, try adding config.build_settings['EXCLUDED_ARCHS[sdk=iphonesimulator*]'] = 'arm64 i386'
and in project.pbxproj, try adding `EXCLUDED_ARCHS = "arm64 i386";

should i add the confit.build settings next to the config build?

post_install do |installer|
installer.pods_project.targets.each do |target|
flutter_additional_ios_build_settings(target)
target.build_configurations.each do |config|
config.build_settings['IPHONEOS_DEPLOYMENT_TARGET'] = '9.0'
config.build_settings['EXCLUDED_ARCHS[sdk=iphonesimulator*]'] = 'arm64 i386'
end
end
end

Also, where exactly should I add 'EXCLUDED_ARCHS = "arm64 i386"; in project.pbxproj file?

@darshankawar i get this error when i try to run it within the Android Studio with Iphone 12 pro max simulator. Really getting upset with this error.

Launching lib/main.dart on iPhone 12 Pro Max in debug mode...
Running pod install...
Running Xcode build...
Xcode build done.                                           25.4s
Failed to build iOS app
Error output from Xcode build:
↳
** BUILD FAILED **
Xcode's output:
↳
/Users/mr22726/.pub-cache/hosted/pub.dartlang.org/flutter_web_browser-0.14.0/ios/Classes/FlutterWebBrowserPlugin.m:24:19: warning: unused variable 'controlColorArg' [-Wunused-variable]
NSString *controlColorArg = call.arguments[@"ios_control_color"];
^
1 warning generated.
Command PhaseScriptExecution failed with a nonzero exit code
note: Using new build system
note: Building targets in parallel
note: Planning build
note: Constructing build description

Could not build the application for the simulator.
Error launching application on iPhone 12 Pro Max.

I got the same issues

Launching lib/main.dart on iPhone 11 in debug mode...
lib/main.dart:1
Xcode build done.                                           14.4s
Failed to build iOS app
Error output from Xcode build:
↳
** BUILD FAILED **
Xcode's output:
↳
Command PhaseScriptExecution failed with a nonzero exit code
note: Using new build system
note: Building targets in parallel
note: Planning build
note: Constructing build description
Could not build the application for the simulator.
Error launching application on iPhone 11.
Exited (sigterm)

@Leoudayan
That error seems to be pointing to 3rd party plugin, flutter_web_browser. If you remove it, do you still get similar error for another plugin or your original issue is now resolved ?

@darshankawar no, that plugin is just giving a warning. It use to work with Both Android and iOS. But, now it only works with Android. I remember doing some upgrade and doing with pod file. Then only this error started to pop-up.
I repeat, my Android build is working. But, my iOS is giving the above errors. Do you have any solution?

@darshankawar I completely removed the flutter_web_browser plugin and i am getting the following errors now,

Launching lib/main.dart on iPhone 12 Pro Max in debug mode...
Running Xcode build...
Xcode build done.                                           14.3s
Failed to build iOS app
Error output from Xcode build:
↳
** BUILD FAILED **
Xcode's output:
↳
Google-Mobile-Ads-SDK-gpzghpwcukkoufeoomexzqqzagwh
Invalid depfile: /Users/mr22726/Flutter/block_indy/.dart_tool/flutter_build/0f770f28578bd0dafb7f69284c44eed7/kernel_snapshot.d
Invalid depfile: /Users/mr22726/Flutter/block_indy/.dart_tool/flutter_build/0f770f28578bd0dafb7f69284c44eed7/kernel_snapshot.d
Invalid depfile: /Users/mr22726/Flutter/block_indy/.dart_tool/flutter_build/0f770f28578bd0dafb7f69284c44eed7/kernel_snapshot.d
Invalid depfile: /Users/mr22726/Flutter/block_indy/.dart_tool/flutter_build/0f770f28578bd0dafb7f69284c44eed7/kernel_snapshot.d
Command PhaseScriptExecution failed with a nonzero exit code
Google-Mobile-Ads-SDK-gpzghpwcukkoufeoomexzqqzagwh
note: Using new build system
note: Building targets in parallel
note: Planning build
note: Constructing build description

Could not build the application for the simulator.
Error launching application on iPhone 12 Pro Max.

I SOLVED THE PROBLEM!
I did 'Flutter Upgrade' and then 'Flutter Clean'.
Now everything works :D Thanks for the help!

@Leoudayan thanks a lot
I have faced the same issue and didn't know how to fix it.
For the past 5 days been trying but no solution worked.
Thanks.

I got the same issues

Launching lib/main.dart on iPhone 11 in debug mode...
lib/main.dart:1
Xcode build done.                                           14.4s
Failed to build iOS app
Error output from Xcode build:
↳
** BUILD FAILED **
Xcode's output:
↳
Command PhaseScriptExecution failed with a nonzero exit code
note: Using new build system
note: Building targets in parallel
note: Planning build
note: Constructing build description
Could not build the application for the simulator.
Error launching application on iPhone 11.
Exited (sigterm)

[flutter] flutter doctor -v

[✓] Flutter (Channel master, 2.3.0-17.0.pre.326, on macOS 11.4 20F71 darwin-x64, locale zh-Hans-CN)
• Flutter version 2.3.0-17.0.pre.326 at /Applications/flutter
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision 3954642 (7 days ago), 2021-06-11 00:19:02 -0400
• Engine revision 6b02e7bdd3
• Dart version 2.14.0 (build 2.14.0-199.0.dev)
• Pub download mirror https://pub.flutter-io.cn
• Flutter download mirror https://storage.flutter-io.cn

[✗] Android toolchain - develop for Android devices
✗ Unable to locate Android SDK.
Install Android Studio from: https://developer.android.com/studio/index.html
On first launch it will assist you in installing the Android SDK components.
(or visit https://flutter.dev/docs/get-started/install/macos#android-setup for detailed instructions).
If the Android SDK has been installed to a custom location, please use
flutter config --android-sdk to update to that location.

[✓] Xcode - develop for iOS and macOS
• Xcode at /Applications/Xcode.app/Contents/Developer
• Xcode 12.4, Build version 12D4e
• CocoaPods version 1.10.1

[✓] Chrome - develop for the web
• Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[!] Android Studio (not installed)
• Android Studio not found; download from https://developer.android.com/studio/index.html
(or visit https://flutter.dev/docs/get-started/install/macos#android-setup for detailed instructions).

[✓] VS Code (version 1.51.1)
• VS Code at /Users/wh-js/Downloads/Visual Studio Code.app/Contents
• Flutter extension version 3.17.0

[✓] Connected device (2 available)
• iPhone 11 (mobile) • 26BEF922-E1E2-4147-AFBB-4F9FE75E5138 • ios • com.apple.CoreSimulator.SimRuntime.iOS-14-4 (simulator)
• Chrome (web) • chrome • web-javascript • Google Chrome 83.0.4103.61

! Doctor found issues in 2 categories.
exit code 0

I have solved the problem!
open Runner.xcworkspace in xcode and then selected project Runner -> Info -> Debug -> change the Runner None.


Proposal: Split plugin into to segments

Hi,
I would like to leave an proposal that may could help some users, too.
Currently, this plugin requires an chrome instance availible on the grafana container. Typically, the chrome binary isn't part of the grafana container. For this scenario, a custom image is required.
An alternative solution would be to separate the plugin and the renderer process. similar what grafana does with the grafana-image-render.
The benefit would be that no custom image would be required. In Kubernetes, it would be possible do a separate scaling for the rendering process, including auto-scaling based on request.
In short: Instead exec chrome, do an http request to an external web server (separate container) which does the exec.
What did you think about?

Thanks for the proposal.
Well, I agree that building a custom image for k8s is not very "convenient". Splitting the plugin would require a major rewrite and will have to manage more things like auth between Grafana and this external service.
Moreover, the plugin delegates most of the heavy work to grafana-image-renderer which creates a PNG for each panel in the dashboard. The plugin itself calls chrome at most twice: once to get dashboard model and once to generate report. So, I am not quite sure if scaling is a major issue here.
Nevertheless it is an interesting idea to pursue in the future if there is a real need for it.

Would it be possible that the plugin itself ships the chromium binary?

That should be possible but we need it test it though. If it works, that would be a great enhancement!

Hey @mahendrapaipuri
today, I had another implementation idea
As I know, the developer protocol of chrome supports network.
https://github.com/chromedp/chromedp/blob/ebf842c7bc28db77d0bf4d757f5948d769d0866f/allocate.go#L513-L528
instead using a local ExecAllocation,
https://github.com/mahendrapaipuri/grafana-dashboard-reporter-app/blob/88d0c230b07ce19f896a6960935ca1a55e7cd959/pkg/plugin/report.go#L323
the NewRemoteAllocator can be used to connect against a persistence running chrome instance. This could also safe resource, since it would not be longer required to setup a new chrome instance each time.

Hello @jkroepke Thanks a lot for your active involvement. Really appreciate it.
Indeed it is a very good idea. We can use a remote instance "out-of-the-box" for the report generation and I dont really any big blockers here.
But, we use chromium to get the "current" dashboard model here. As you can see, in this invocation we set user cookie in the chromium to get the dashboard. The rationale is that using user cookie will ensure that the correct RBAC is respected. The blocker here is that we cannot have multiple user cookies in the same instance of chromium.
One solution is to use UserDataDir option of chromedp and use a different profile for each request so that cookies will be isolated. But again, we need to test it before.
What do you think?

One solution is to use UserDataDir option of chromedp and use a different profile for each request so that cookies will be isolated. But again, we need to test it before.

I did not expect that this is working. UserDataDir is a start-up option for chrome and I did not expect that this can be changed after chrome is started.
For an initial POC, I would set the concurrency to 1 (via mutex) that only one PDF can be generated at the same time.

I did not expect that this is working. UserDataDir is a start-up option for chrome and I did not expect that this can be changed after chrome is started.

We dont have to change UserDataDir, we have set set a different profile for each user using chromedp.Flag("profile-directory", username). Each user will have their own profile-directory and they all will be saved in UserDataDir. Does it make sense?

For an initial POC, I would set the concurrency to 1 (via mutex) that only one PDF can be generated at the same time. and then set the cookie each time.

This can be a solution as well in the worst case.

Instead playing with cookies/ cookie storage, what about set the cookie header directly?
Like here:
https://github.com/mahendrapaipuri/grafana-dashboard-reporter-app/blob/88d0c230b07ce19f896a6960935ca1a55e7cd959/pkg/plugin/client.go#L210-L211
in theory, cookies are just HTTP headers.

Instead playing with cookies/ cookie storage, what about set the cookie header directly?

Well, when Grafana is hosted on subdomain, browser will send all cookies that are set on the domain to the server. And the fact that operators can configure cookie name will make retrieving correct cookie complicated if we want to set cookie header. That is why we are setting all the cookies in the request.

The mechanic (just using the cookie header) is already used, I assume, that any mention issues are already covered?
https://github.com/mahendrapaipuri/grafana-dashboard-reporter-app/blob/88d0c230b07ce19f896a6960935ca1a55e7cd959/pkg/plugin/client.go#L98

The mechanic (just using the cookie header) is already used, I assume, that any mention issues are already covered?

I remember testing this strategy as well but didnt work. Probably I didnt look too deep on why it didnt work.
In any case, if we would like to use one single chrome instance, we must use different profile for each user to avoid any accidental leak of sensitive info in either case setting cookie header or cookies themselves. Don't you agree?

I do a local test:

Details
// Command headers is a chromedp example demonstrating how to add extra HTTP
// headers to browser requests.
package main

import (
	"context"
	"fmt"
	"log"
	"os"

	"github.com/chromedp/cdproto/network"
	"github.com/chromedp/cdproto/storage"
	"github.com/chromedp/chromedp"
)

func main() {
	dir, err := os.MkdirTemp("", "chromedp-example")
	if err != nil {
		log.Fatal(err)
	}
	defer os.RemoveAll(dir)

	opts := append(chromedp.DefaultExecAllocatorOptions[:],
		chromedp.DisableGPU,
		chromedp.UserDataDir(dir),
	)

	ctx, cancel := chromedp.NewExecAllocator(context.Background(), opts...)
	defer cancel()

	// new browser, first tab
	ctx1, cancel1 := chromedp.NewContext(ctx, chromedp.WithDebugf(log.Printf))
	defer cancel1()

	var (
		res1 string
		res2 string
		res3 string
	)

	// ensure the first tab is created
	if err := chromedp.Run(ctx1, setheaders(
		"https://httpbin.org/cookies",
		map[string]interface{}{
			"Cookie": "name=1value; name2=1value2; name3=1value3",
		},
		&res1,
	)); err != nil {
		log.Fatal(err)
	}

	// same browser, second tab
	ctx2, _ := chromedp.NewContext(ctx1)

	// ensure the second tab is created
	if err := chromedp.Run(ctx2, setheaders(
		"https://httpbin.org/cookies",
		map[string]interface{}{
			"Cookie": "name=2value; name2=2value2; name3=2value3",
		},
		&res2,
	)); err != nil {
		log.Fatal(err)
	}

	// same browser, thrid tab
	ctx3, _ := chromedp.NewContext(ctx1)

	// ensure the second tab is created
	if err := chromedp.Run(ctx3, setheaders(
		"https://httpbin.org/cookies",
		map[string]interface{}{},
		&res3,
	)); err != nil {
		log.Fatal(err)
	}

	fmt.Printf("%+v\n", res1)
	fmt.Printf("%+v\n", res2)
	fmt.Printf("%+v\n", res3)
}

// setheaders returns a task list that sets the passed headers.
func setheaders(host string, headers map[string]interface{}, res *string) chromedp.Tasks {
	return chromedp.Tasks{
		network.Enable(),
		network.SetExtraHTTPHeaders(network.Headers(headers)),
		chromedp.Navigate(host),
		chromedp.Text(`body > pre`, res),
		chromedp.ActionFunc(func(ctx context.Context) error {
			cookies, err := storage.GetCookies().Do(ctx)
			if err != nil {
				return err
			}

			log.Printf("chrome cookie: %+v", cookies)

			return nil
		}),
		network.ClearBrowserCookies(),
		chromedp.Stop(),
	}
}

In that test, I'm do multiple REST calls within one browser. I set cookie via network.SetExtraHTTPHeaders.
I could observe that the header is only used inside a browser tab. The Tab3 did not have have access to the cookies from Tab1.
Cookies are not persistent stored inside chrome. storage.GetCookies() reports 0 cookies after that.
Based on that given test, I do not have the feeling that dedicated browser instance with separate profiles are not necessary. Browser contexts (aka tabs) are sufficient to separat requests from each other, since no cookies are persistent. However we can also enforce ClearCookies action after all.

Thanks a lot for gist @jkroepke
Well, we can give a another go using cookie header instead setting cookies. If we going to use the chrome instance in a separate container (in k8s context), we need to look on how to secure the access to this instance.

remote debugging protocol doesn't support authentication and TLS.
the most secure way would running the dedicated chrome container inside them Kubernetes pod. Grafana and Chrome still share the same network namespace and chrome would be reachable via localhost or unix socket.

I suppose Grafana Image Renderer when running in remote mode uses sidecar architecture as well. There was a severe CVE reported last year and Grafana team proposed a patch by using a shared secret between Grafana and remotely running Grafana image renderer. This is what bothers me and I am pretty sure attack vectors like this will be possible when running chrome instance in a separate side car container with any auth between them.

The Grafana image renderer is running as dedicated pod (not as sidecar) and is exposing all network capabilities.
My idea was to run the chrome for grafana-dashboard-reporter-app as sidecar which does not require to expose the network capabilities. The chrome can listen on localhost only.
And unlike grafana-image-renderer, grafana-dashboard-reporter-app does not permanently store any credentials. If a remote attacker gets access to an chrome instance, he just finds an empty chrome instance.

The Grafana image renderer is running as dedicated pod (not as sidecar) and is exposing all network capabilities.

Ah, I wasnt aware of that. Thanks for detail!

The chrome can listen on localhost only.

That should be reassuring. That should prevent a lot of attack vectors already.

If a remote attacker gets access to an chrome instance, he just finds an empty chrome instance.

I am not quite sure. As it is a remote debugging protocol, I guess (not entirely sure), the attacker can introspect the current open tabs and potentially retrieve secrets. But as you said, if chrome instance will be on localhost with respect to Grafana main container, the chances of attacker getting hold of chrome instance is very low.

One more thing: In Kubernetes it's possible to setup an volume to share file between containers.
Render a local html file
https://github.com/mahendrapaipuri/grafana-dashboard-reporter-app/blob/88d0c230b07ce19f896a6960935ca1a55e7cd959/pkg/plugin/report.go#L336
would still work and is not a blocker here.

If we are moving towards remote chrome instance approach, we can do all the processing in-memory by converting PNG to base64 encoded strings. That will avoid all sorts of file system access which can pose problems in k8s context.

temporary file system access does create any issues. End users just have to create an empyDir Volume and kubernetes will take care off all correct permissions.
The step you mention is an general improvement, and not strictly necessary for the first milestone.


python 3.4 compatibility?

Hi :)
for what python version is your code?
It seems 3.4 is not supported, I get:
entry = (*data, ts,)
                ^

SyntaxError: can use starred expression only as assignment target
Searching for this error shows that this is something that was fixed in 3.5, but not in 3.4.
What would be the 3.4 equivalent of it?
edit:
And a question: How exactly and where is the data stored? From your description it sounds like that somewhere a file with the orderbook information is stored. But where? And is it necessary?
I would like to know where, so I can delete it after closing websocket connection. Or is this not necessary?

The lib is developed for 3.5+, yes.
The equivalent for the above entry would be:
entry = tuple(data + [ts])
A file should only be created if you use the BtfxWssRaw class - this subclass dumps all data as is onto the disk.
If you use the BtfxWss class, data should be stored in the class's attributes: tickers, books, raw_books, _trades, and candles.
There are, however, a crap ton of bugs left to fix - I apologize for that, I just didnt have the time the past few months to upload a proper fix for them.


feature: new ui options title and title_pos

Did you check the docs?

[X] I have read all the lazy.nvim docs

Is your feature request related to a problem? Please describe.
After nvim v0.9.0 release, there were introduced new options title and title_pos for floating windows. It would be great to have the same options for lazy window.
Describe the solution you'd like
The ui config have new ui options title and title_pos
Describe alternatives you've considered
There is no alternative
Additional context
No response

I don't really see the point of this. Closing

The point is to be able to unify the appearance of all floating windows. Each plugin offers its own look and feel, and they often don't play well together.

Just added this.


Release v0.8.0

Release 0.8.0:

Dropping testing for Python 3.6. Moving to Python 3.7 to 3.10
Adding upper pins to requirements file (edgetest)
General housekeeping

#119
#120
#126

Adding edgetest

#125

@ak-gupta Sorry need another quick review.


frama-c: 21.0 (Scandium) → 21.1 (Scandium)

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg build framac

I have just discovered that Frama-C 21+ in nixpkgs has WP plugin disabled because nixpkg's why3 is too old. Configure process didn't fail but simply disabled WP upon discovering why3's version.
I've addressed this in #94258


When I use apex on 2080ti, I get the following error, how can I solve it?

RuntimeError: expected scalar type Float but found Half (data at /usr/local/lib/python3.5/dist-packages/torch/include/ATen/core/TensorMethods.h:1386)
frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7f9f2dbdd441 in /usr/local/lib/python3.5/dist-packages/torch/lib/libc10.so)
frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7f9f2dbdcd7a in /usr/local/lib/python3.5/dist-packages/torch/lib/libc10.so)
frame #2: float* at::Tensor::data() const + 0xcf (0x7f9f1c69fa2f in /home/yyz/bigdisk/CenterNet-master0/src/lib/models/networks/DCNv2/_ext.cpython-35m-x86_64-linux-gnu.so)
frame #3: dcn_v2_cuda_forward(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, int, int, int, int, int, int, int, int, int) + 0xbc0 (0x7f9f1c6a4b50 in /home/yyz/bigdisk/CenterNet-master0/src/lib/models/networks/DCNv2/_ext.cpython-35m-x86_64-linux-gnu.so)
frame #4: dcn_v2_forward(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, int, int, int, int, int, int, int, int, int) + 0x8b (0x7f9f1c689c0b in /home/yyz/bigdisk/CenterNet-master0/src/lib/models/networks/DCNv2/_ext.cpython-35m-x86_64-linux-gnu.so)
frame #5:  + 0x1f91c (0x7f9f1c69591c in /home/yyz/bigdisk/CenterNet-master0/src/lib/models/networks/DCNv2/_ext.cpython-35m-x86_64-linux-gnu.so)
frame #6:  + 0x1f99e (0x7f9f1c69599e in /home/yyz/bigdisk/CenterNet-master0/src/lib/models/networks/DCNv2/_ext.cpython-35m-x86_64-linux-gnu.so)
frame #7:  + 0x1cdc0 (0x7f9f1c692dc0 in /home/yyz/bigdisk/CenterNet-master0/src/lib/models/networks/DCNv2/_ext.cpython-35m-x86_64-linux-gnu.so)

frame #11: python3() [0x4e3423]
frame #14: THPFunction_apply(_object*, _object*) + 0x6b1 (0x7f9f2e3bf491 in /usr/local/lib/python3.5/dist-packages/torch/lib/libtorch_python.so)
frame #18: python3() [0x4e3537]
frame #22: python3() [0x4e3423]
frame #24: python3() [0x4f08be]
frame #26: python3() [0x55fbf6]
frame #30: python3() [0x4e3537]
frame #34: python3() [0x4e3423]
frame #36: python3() [0x4f08be]
frame #38: python3() [0x55fbf6]
frame #42: python3() [0x4e3537]
frame #46: python3() [0x4e3423]
frame #48: python3() [0x4f08be]
frame #50: python3() [0x55fbf6]
frame #54: python3() [0x4e3537]
frame #58: python3() [0x4e3423]
frame #60: python3() [0x4f08be]
frame #62: python3() [0x55fbf6]

Same error with you, have you worked out ?

This adds apex support with level O1. But I got the following error when running it.
RuntimeError: Function _DCNv2Backward returned an invalid gradient at index 1 - expected type torch.cuda.HalfTensor but got torch.cuda.FloatTensor

The problem is solved. Besides following the code of https://github.com/lbin/DCNv2, I add the following three lines before return in the _backward function of _DCNv2 in dcn_v2.py:
grad_input = grad_input.half()
grad_offset = grad_offset.half()
grad_mask = grad_mask.half()

The problem is solved. Besides following the code of https://github.com/lbin/DCNv2, I add the following three lines before return in the _backward function of _DCNv2 in dcn_v2.py:
grad_input = grad_input.half()
grad_offset = grad_offset.half()
grad_mask = grad_mask.half()

Did you modified other code besides dcn_v2.py? I modified those code but still get the same error as @yeyuanzheng177 .

me too,some one has any update?

In dcn_v2.py the "_backend.dcn_v2_forward" and "_backend.dcn_v2_backward" only expect float32 input.
So if you use mix-precision(Apex/amp)，you should convert float16 to float32, and convert the final output from float32 to float16.
You can consult

In dcn_v2.py the "_backend.dcn_v2_forward" and "_backend.dcn_v2_backward" only expect float32 input.
So if you use mix-precision(Apex/amp)，you should convert float16 to float32, and convert the final output from float32 to float16.
You can consult https://github.com/jasonkena/yolact/tree/amp/external/DCNv2
@steven22tom Thank you for your hint, It works!

In dcn_v2.py the "_backend.dcn_v2_forward" and "_backend.dcn_v2_backward" only expect float32 input.
So if you use mix-precision(Apex/amp)，you should convert float16 to float32, and convert the final output from float32 to float16.
You can consult https://github.com/jasonkena/yolact/tree/amp/external/DCNv2
@steven22tom Thank you for your hint, It works!


Update to 1.12.1

This is a brand-new feedstock, so the recipe was crafted from scratch. Intel needs this version (1.12.1) as soon as possible.

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_ab0h705n87/clone/recipe/meta.yaml:28: no_git_on_windows: git should not be used as a dependency on Windows.
WARNING: /tmp/abs_ab0h705n87/clone/recipe/meta.yaml:35: build_tools_must_be_in_build: The build tool cmake is not in the build section.
ERROR: /tmp/abs_ab0h705n87/clone/recipe/meta.yaml:52: missing_imports_or_run_test_py: Python packages require imports or a python test file in tests.
ERROR: /tmp/abs_ab0h705n87/clone/recipe/meta.yaml:57: incorrect_license: The recipe's about/license key is not an SPDX compliant license or license exception, closest match: Apache-2.0
WARNING: /tmp/abs_ab0h705n87/clone/recipe/meta.yaml:68: license_file_overspecified: Using license_file and license_url is overspecified.
Errors were found

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_adfnfus9tz/clone/recipe/meta.yaml:52: missing_imports_or_run_test_py: Python packages require imports or a python test file in tests.
Errors were found

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_f26jcbahv1/clone/recipe/meta.yaml:53: missing_imports_or_run_test_py: Python packages require imports or a python test file in tests.
Errors were found

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_2emxqah2o0/clone/recipe/meta.yaml:55: missing_imports_or_run_test_py: Python packages require imports or a python test file in tests.
Errors were found

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_bcmnqrw4wd/clone/recipe/meta.yaml:57: missing_imports_or_run_test_py: Python packages require imports or a python test file in tests.
Errors were found

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_ea73w6vypi/clone/recipe/meta.yaml:59: missing_imports_or_run_test_py: Python packages require imports or a python test file in tests.
Errors were found

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_693cpy4gqj/clone/recipe/meta.yaml:58: missing_imports_or_run_test_py: Python packages require imports or a python test file in tests.
Errors were found

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_ec6e82grn1/clone/recipe/meta.yaml:58: missing_imports_or_run_test_py: Python packages require imports or a python test file in tests.
Errors were found

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_26a5b53lml/clone/recipe/meta.yaml:58: missing_imports_or_run_test_py: Python packages require imports or a python test file in tests.
Errors were found


Broken Links - 16 Links return 404 Error codes found by CodeAuditor

CC:  @adamcogan @christianmorfordwaitessw @bradystroud @tiagov8 @SebastienBoissiere @AndreasLengkeek

There are 16 broken links on SSW Rules that are found by CodeAuditor. See more details here
Please action to fix these links by editing the source contents
Once fixed, re-run the migration to update new contents on production site

Figure: Scan Result from CodeAuditor that shows 16 broken links
AB#58972

Hi @tombui99
This highlighted error from Code Auditor is a bug:

CA caught a line of code example and reported it as a broken link...

Hi @tombui99
This error is a false positive:

The link is working indeed: https://www.nuget.org/packages/SSW.SqlVerify.EF/

Hi @tombui99
FYI - I did not find this link:

Hi @tombui99
This error has been explained in this new issue: https://github.com/SSWConsulting/SSW.Rules/issues/298

It should be fixed after we define how to migrate these PDF files stored in SharePoint

Hi @tombui99

There are 16 broken links on SSW Rules that are found by CodeAuditor. See more details here
Please action to fix these links by editing the source contents

Done - I've fixed all reported broken links in SharePoint rules (except for the ones mentioned on comments above)

Once fixed, re-run the migration to update new contents on production site

Please action (let me know if other broken links appear)

Replaced by #869


About 'next_state'

Hi, Thank you for open-source your code, I have a question: when the agent makes an action according to the current state, how can we get the next state? I don't seem to have found the relevant explanation and method in your paper and code。

Thanks for your concern of this project :), and sorry for replying late. The next state, which is quite important in this project, can be obtained as the return values of the method named 'step(action)' in the line 332 of the directory 'XRouting/rl/env/dynamic_rerouting_env.py'. If you have any question, please feel free to write here. Thanks again!

Hi, Thank you for open-source your code, I have a question: when the agent makes an action according to the current state, how can we get the next state? I don't seem to have found the relevant explanation and method in your paper and code。
Thanks for your concern of this project :), and sorry for replying late. The next state, which is quite important in this project, can be obtained as the return values of the method named 'step(action)' in the line 332 of the directory 'XRouting/rl/env/dynamic_rerouting_env.py'. If you have any question, please feel free to write here. Thanks again!


Added horizontal scrolling in table widget

Pull Request Template
Use this template to quickly create a well written pull request. Delete all quotes before creating the pull request.

Description
Adding display type as table to allow horizontal scrolling in table.
Fixes #1419
Type of change

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Manually tested the fix

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes

@vicky-primathon, we have an issue with this fix. For some reason, the table now does not reflow to fill the full width of the table widget.

Please check it out.


Add pyproject.toml for Custom Node Registry

We are working with dr.lt.data and comfyanon to build a global registry for custom nodes (similar to PyPI). Eventually, the registry will be used as a backend for the UI-manager. All nodes go through a verification process before being published to users.
The main benefits are that authors can

publish nodes by version and users can safely update nodes knowing ahead of time if their workflows will break or not
automate testing against new commits in the comfy repo and existing workflows through our CI/CD dashboard

Action Required:

[ ] Go to the registry. Login and create a publisher id (everything after the @ sign on your registry profile).
[ ] Add the publisher id into the pyproject.toml file.
[ ] Merge the separate Github Actions PR, then merge this PR.

If you want to publish the node manually, install the cli by running pip install comfy-cli, then run comfy node publish
Check out our docs if you want to know more about the registry. Otherwise, feel free to message me on discord at haohao_81202 or join our server if you have any questions!

hey @yiwangsimple, wanted to send a small nudge and ask if you had any questions!

@yiwangsimple, wondering if there's any way we can support you on this PR!


Support Rails 7.0

This PR adds rails 7.0 to the matrix test and fixed a compatibility error.
I've confirmed that jruby fails for all ruby versions.
It is for bson incompatibility for jruby 9.3+: https://github.com/mongodb/bson-ruby/pull/252

Do you want to get rid of jruby?
Do people actually use jruby?

Hmm, I have know idea on jruby.
How about specifying jruby 9.2?
It passes most of the matrix test cases: https://github.com/a2ikm/mongomapper/pull/11
Even if so, we need to exclude the pair of rails 7.0 and jruby 9.2.
It is because raiils 7.0 requires ruby >= 2.7.0, but jruby 9.2 is compatible with ruby 2.5.

I've added a commit to specify jruby 9.2.

Note that jruby 9.3 is compatible with ruby 2.6 but doesn't satisfy rails 7.0.
https://www.jruby.org/2021/09/22/jruby-9-3-0-0.html

bson 4.14.0 supports jruby 9.3, so I removed jruby version specifier.
@smtlaissezfaire
Could you review again?
I will make another pull request to support Ruby 3.1 later.

On the other hand, I think we can consider discontinuing support for JRuby for some reasons:

It took a few weeks to release bson 4.14.0 to support JRuby 9.3
The development of JRuby is still active, but its compatibility is a bit outdated

The latest JRuby 9.3 is compatible with ruby 2.6 whose EOL is this March

Its delay can make our maintenance hard.

[x] passed CI
[x] updated CHANGELOG

Let me merge 🙌


fix(release): auto release

Issue
If we want to release miranum-connect, an authorised person must confirm the release.
Fix
With autoReleaseAfterClose, this step will be skipped.
Checklist

Code is easy to understand
No obvious errors or bugs
CI/CD Pipelines did run successfully
Tests were implemented
Documentation was created

Codecov Report
Patch has no changes to coverable lines.
:loudspeaker: Thoughts on this report? Let us know!.


Errors when installing against a case-sensitive SQL Server 2016 instance

Do you want to request a feature or report a bug?
Bug
What is the current behavior?
Install-CISL throws errors about parameters which were not declared when run against a case sensitive SQL Server 2016 instance.
I was able to fix this by:
find: @currenttrimmedRGsPerc  replace with: @currentTrimmedRGsPerc
find: @currenttrimmedRGs  replace with @currentTrimmedRGs
I just did this across StoredProcs and saved all the files it changed (it was a couple), and after that was able to install A-OK.
Edit: I am no good at markdown but I think I finally got the capitalization to show up :)

Hi Kendra,
thank you very much! I will update the code this week or feel free to update the source code as well !
Best regards,
Niko

This issue has been fixed & checked into the upcoming 1.5.0 release.
Thank you for reporting!


More autocompletion

Right now it doesn't appear that we autocomplete for @ arguments on components or named blocks.

Our fundamental restriction right now is that @glimmer/syntax doesn't have an error-tolerant parsing option and will only provide an AST for a 100% valid template. Because of that, we can autocomplete for @args and named blocks, but only if you go out of your way to ensure your surrounding template is syntactically valid.

For named blocks especially, that's pretty much useless, as the only way to get completions is to make it self-closing, go back and trigger a suggestion, then un-self-close the block and add the closing tag.
Both for Glint itself and for other tools like template-lint it would be super helpful for @glimmer/syntax to have an error-tolerant mode that gave a best-effort parse tree. With that, we could better support these and a bunch of other completion scenarios with little to no change on this end.
I know others are interested in that, and I'm banging the drum—yesterday in #st-typescript for instance 😄. For now there are still a ton of other things to do in Glint proper, but if we reach a point of readiness here that this completion pain makes it to the top of the priority list and no one else has been able to make progress with the parser in the meantime, I may take a look.


Kerchunk enhancements for Fast NODD Grib Aggregations

Project Description
This project improves on previous GSOC work to provide faster, easier access to public weather forecast data via widely used open source python libraries.
Motivation
Weather data and weather forecasts in particular are essential information for individuals, businesses and government. Extreme weather events are becoming more common with climate change. The electric utility industry in particular needs weather forecasts to make choices that help reduce emissions.
Details
A prototype has demonstrated the ability to build large aggregations from NODD grib forecasts in a fraction of the time using the idx files. The intern would work with the mentors to generalize the Camus Energy implementation and move it into the open source Kerchunk library. Some Camus code has already moved into Kerchunk but we believe there is more value to share with the community that will help realize the potential of the Google-NOAA NODD program.
(AWS and Azure participate in NODD as well and the techniques are equally useful)
Technical background

grib2 is a files format used primarily for climate/weather modelling. It contains a number of "messages", where each message contains a coordinate grid definition, a set of metadata attributes, and a image or cube of data
zarr is a file format for multi-dimensional array data designed for easy parallel access with remote/cloud data: it is "cloud native".
xarray is a data analysis package geared to multi-dimensional data, where labels can be assigned to each dimension and then indexed/regridded. Xarray can read datasets from zarr, grib2 and various other formats, but only zarr works well for distributed or parallel workloads
fsspec is a very popular python package for accessing bytes in a large number of storage systems in a uniform way, as if they were similar to the local filesystem
kerchunk is a library to find the data buffers within a few different scientific/archival data formats, and save these as "references". Using fsspec, the original data (from potentially a large number of input HDF, grib, or other files) can be viewed as a single zarr dataset, and loaded with xarray. This gets you the benefits of cloud native data access even for formats that were not designed for it.

Expected Outcomes

Better community tools for working with grib files, specifically better leverage on the incredible NODD archive.
A deep technical experience for the intern with referenceable job skills
More community use & support for the NODD - Cloud data partnership
More community use & support for these open source projects

In addition to IOOS & NOAA, these tools are already widely used in the ESIP and PanGeo community. We will also be working with the ESIG community to share this work and expand the impact.
Skills required
Python, Cloud Storage (S3, GCS), Git, multi dimensional arrays. Prior experience with Xarray, Zarr, Kerchunk, Fsspec or Rust would be very helpful.
Mentor(s)
David Stuebe (@emfdavid), Martin Durant (@martindurant)
Expected Project Size
175
What is the difficulty of the project?
Expert

Som extra detail on some of the technologies referenced in this proposal.

grib2 is a files format used primarily for climate/weather modelling. It contains a number of "messages", where each message contains a coordinate grid definition, a set of metadata attributes, and a image or cube of data
zarr is a file format for multi-dimensional array data designed for easy parallel access with remote/cloud data: it is "cloud native".
xarray is a data analysis package geared to multi-dimensional data, where labels can be assigned to each dimension and then indexed/regridded. Xarray can read datasets from zarr, grib2 and various other formats, but only zarr works well for distributed or parallel workloads
fsspec is a very popular python package for accessing bytes in a large number of storage systems in a uniform way, as if they were similar to the local filesystem
kerchunk is a library to find the data buffers within a few different scientific/archival data formats, and save these as "references". Using fsspec, the original data (from potentially a large number of input HDF, grib, or other files) can be viewed as a single zarr dataset, and loaded with xarray. This gets you the benefits of cloud native data access even for formats that were not designed for it.

I'm looking forward to this project. How do I get started?

I'm looking forward to contribute in this project. How do I get started? And what communication method do you use for kerchunk?

Hi Anurag
We communicate primarily through github issues, PRs and comments.
There will be a pangeo presentation
https://discourse.pangeo.io/t/pangeo-showcase-optimizations-for-kerchunk-aggregation-and-zarr-i-o-at-scale-for-machine-learning/4074
today at 1200EST introducing this work. If you can't make it on short
notice, there should be a recording.
Best
David
On Wed, Mar 6, 2024 at 8:05 AM Anurag Nayak @.***>
wrote:

I'm looking forward to contribute in this project. How do I get started?
And what communication method do you use for kerchunk?
—
Reply to this email directly, view it on GitHub
https://github.com/ioos/gsoc/issues/42#issuecomment-1980833637, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AUDN2W5EVMNQ24TOCW3QNADYW4ICFAVCNFSM6AAAAABCWALCLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBQHAZTGNRTG4
.
You are receiving this because you were mentioned.Message ID:
@.***>

grib2 -> kerchunk -> fsspec -> xarray -> zarr files.
This is the supposed flow of the whole environment ?, where grib2 files are stored in a cloud or a local filesystem.  I've downloaded some sample grib files from the internet to work with kerchunk.

grib2 -> kerchunk -> fsspec -> xarray -> zarr files.
This is the supposed flow of the whole environment ?, where grib2 files are stored in a cloud or a local filesystem. I've downloaded some sample grib files from the internet to work with kerchunk.

Your order is not quite right.

kerchunk scans grib files, producing JSON or parquet references
fsspec can read and interpret these references as a filesystem
zarr opens such an fsspec filesystem
xarray includes zarr as one of its backend

The last three points can be achieved in a single line:
import xarray as xr
xr.open_dataset("path/to/references", engine="kerchunk", ...)

Here is a high level view of how we use these tools operationally.

So it is grib2 -> kerchunk -> json references -> fsspec -> zarr files -> xarray.
So this project only involves, kerchunk to create indexes for the references of the grib files during scanning  or only the indexes for the grib files during the scanning ?
And the indexes will be stored in cloud database for machine learning applications and the application will build the aggregation as the data is used.
P.S:- I'm new to this whole field and I'd love to learn about it. I used kerchunk to produce the references locally using some sample grib files. Also it would be very helpful if you could provide some resources to get started with moving the V2 aggregation into kerchunk.

So it is grib2 -> kerchunk -> json references -> fsspec -> zarr files -> xarray.

We are not producing zarr files, only ephemeral zarr datasets - that is, an object you can use as any other zarr dataset, but without any real zarr store. Not copying/translating the original grib data is a big feature of kerchunk.
I'm afraid I didn't understand what you asked in the second paragraph.

I meant to ask for this project, we're to add the indexing process for grib files in V2 NODD Aggregations to kerchunk.
It'd to very helpful if you could provide me some resources to understand this whole field to know it better and contribute to kerchunk, as I'm totally new to this domain.

As far as the kerchunk stack is concerned, you can go through tutorials:

https://fsspec.github.io/kerchunk/tutorial.html
https://projectpythia.org/kerchunk-cookbook/README.html

What else would you like to know? We don't expect you to know the byte-wise details within the grib format.

@emfdavid The role of zarr references made by scan_grib in one-to-one mapping of the index files to the grib files is to help in making the index table from the grib tree which is made using the references, right?

The role of zarr references made by scan_grib in one-to-one mapping of the index files to the grib files is to help in making the index table from the grib tree which is made using the references, right?

The scan grib message reads the actual grib file message and parses it to get the metadata. This is expensive and slow (both the IO and the processing time!).  By making a mapping with the index file, we can get the same information from the small text files.

chunks are referred to the groups made from the grib messages?

Yes, each zarr chunk is a reference to a range of bytes in a grib file that can be decoded to get the array of values. Some of the chunks contain only a single value, like the timestamp, so these are stored by value rather than by reference.
Have you been able to try out any of the Kerchunk tutorials? You can also try running the notebook liked from this issue. The setup instructions are a bit terse, but if you let me know where you have trouble I can help.

Is there any template for writing out the GSOC proposal. What should be the proposal about?

I am still new to this process too, but I think this github issue is the proposal. We will follow up with more questions as the selection process gets going. This is an open discussion of the project where applicants can ask clarifying questions and start to learn about the project.

@emfdavid I've been able to read some sample grib files with kerchunk which I've collected over the Internet and produced the json references for the same on a Jupyter notebook locally. I also went through the Pangeo Presentation for the enhancements.

@martindurant This is a medium size project, right? Can you give a brief idea of what should the proposal consist of? I'm writing a proposal for this project using the IOOS's template?

Essentially the template, together with our description and discussion here should be plenty. If you are setting up specific milestones, we can help with that. But don't be hung up on them - what you would produce by the end of the project doesn't need to match closely with what you propose, so long as the work is useful in some way. We, the mentors, don't care what format you write, so stick to Google's guidelines.
@emfdavid did you have any tangible milestones beyond the "expected outcomes" you think worth mentioning?

@martindurant I'm getting this error NoCredentialsError: Unable to locate credentials while I'm trying to run this notebook https://nbviewer.org/gist/peterm790/92eb1df3d58ba41d3411f8a840be2452. I don't think any credentials were involved in this notebook.

@martindurant I'm getting this error NoCredentialsError: Unable to locate credentials while I'm trying to run this notebook https://nbviewer.org/gist/peterm790/92eb1df3d58ba41d3411f8a840be2452. I don't think any credentials were involved in this notebook.

Cell [2] says:
fs_write = fsspec.filesystem('s3', anon=False)
#fs_write = fsspec.filesystem('') #uncomment this if you intend to write jsons to a local folder

If you don't have AWS credentials or otherwise an s3 bucket to write to, you need to use the second variant, not the first.

I figure it out, the error was here
reference_jsons = sorted(['s3://'+f for f in reference_jsons])
reference_jsons = sorted([f for f in reference_jsons]) # it should be without the s3 protocol as I'm working in a local filesystem

@martindurant On opening the references as a dataset, I'm getting this error
ds = xr.open_dataset("/home/anurag/kerchunk_jupyter/jp_grib_references", engine="kerchunk", chunks={'valid_time':1})

FileNotFoundError: [Errno 2] No such file or directory: '/home/anurag/kerchunk_jupyter/jp_grib_references/.zmetadata'

I also tried doing this,
mzz = MultiZarrToZarr(reference_jsons,
                        concat_dims = ['valid_time'],
                        identical_dims=['latitude', 'longitude', 'heightAboveGround', 'step'])
d = mzz.translate()
ds = xr.open_dataset(d, engine="kerchunk", chunks={'valid_time':1})

ValueError: unrecognized chunk manager dask - must be one of: []

the first error: did you write your references to a JSON file? You are passing what looks like a directory, so fsspec is trying to treat it like a parquet reference set.
the second: this looks like you have something wrong with your environment. Does it work without the chunks= (which would not use dask)?

For the first, I wrote the references as json files. I'm trying to open the local reference jsons as a single dataset to view it in a jupyter notebook.
I have to convert the references into a single unit using MultiZarrToZarr and use translate function on it before open it as a single dataset.
I tried doing this,
mzz = MultiZarrToZarr(reference_jsons,
                        concat_dims = ['valid_time'],
                        identical_dims=['latitude', 'longitude', 'heightAboveGround', 'step'])
d = mzz.translate()
fs = fsspec.filesystem("file", fo=d)
m = fs.get_mapper("")
ds = xr.open_dataset(m, engine="kerchunk" ,chunks={'valid_time':1})

I'm getting this error,
GroupNotFoundError: group not found at path ''

For the first, I wrote the references as json files.

Please name your files ".json"

fs = fsspec.filesystem("file", fo=d)

This is wrong. You want filesystems of type "reference", or use engine="kerchunk" in xr.open_dataset.

@martindurant I was able to produce and view the datasets in the jupyter notebook, with your help. Thank you for that. Along with this, the project primarily involves with grib2.py module. What kind of milestones should I mention in the proposal?

@emfdavid , I'l leave that one to you: what does minimal useful outcome look like, do you think?

@martindurant I'll mention the project's requirement and the points mentioned in the Pangeo community  as the milestones. How does that sound?

@emfdavid @martindurant I've sent the proposal. Kindly review it and suggest any changes.

Thank you for your application @Anu-Ra-g
I am sorry I did not respond last week. Thank you @martindurant for helping in my absence.
I don't see the applications posted yet. Do mentors only get access after the application period ends tomorrow?
As Martin said, the project outcomes will evolve.
I progress toward using the idx files for grib aggregation would be a great milestone.
If that is in your submitted application, great. If not, please don't worry.

I'm wondering in case the index file is not present, how should we create the mapping?

To build the mapping definitely requires both the idx file and the grib file to be present and correct.
To do aggregation of many files using the mapping, we can try the idx file if it is present and fall back to reading whole grib files if needed.
That is more of an operational concern though. I think our initial milestones would be limited to providing some functions that operate on idx files and grib files. Then we can provide suggestions and examples of how to use them in practice.

All, I believe an Org administrator needs to assign mentors to applications individually in the GSoC portal.  I just did this for @Anu-Ra-g 's application, I believe.  Not sure if you get a notification or not @emfdavid and @martindurant - but please log in to confirm you can see it now.

Yes - thank you @mwengren I can see the Contributor Proposal from Anurag now - thank you.


isAlive has been depreacted and removed in favor of is_alive in Python 3.9

 Issue by tirkarthi
Sun Apr 26 06:30:32 2020
Originally opened as https://github.com/codelucas/newspaper/issues/804

isAlive has been deprecated alias for is_alive and has been removed in Python 3.9 . Python 3.9 PR : https://github.com/python/cpython/pull/15225
https://github.com/codelucas/newspaper/blob/4c9cde0749275c82827d76c064413a479f3daaef/newspaper/utils.py#L144

fixed


Address #157 and #158 install controller-gen and upgrade to 0.10.0

Fixes #157
Fixes #158
I am unsure if this upgrade to controller-gen 0.10.0 is safe.
I can confirm I was able to compile on Linux and Mac after this change, have not confirmed I can run correctly yet.

Please fix merge conflicts.
Looks like it also conflicts with https://github.com/konveyor/tackle2-hub/pull/176.

Closing this in favor of https://github.com/konveyor/tackle2-hub/pull/176
That address #157, still need to address #158 with a follow up PR to bump version of controller_gen


Fix User objects not saving, Add support for Parse Files, Anonymous Login

Hi Phil,
I am using your sdk to develop an application for a hackathon. I ran over a few issues that I fixed and I thought it woud be a good idea to contribute back to this project. Here are my changes:

Fixed changes on User object not syncing to server due to missing session token when accessing API
Added the ability to login anonymously by providing "AUTH_DATA" POST data with a unique UUID. Can be extended to allow for facebook and twitter auth in the future
Added the ability to create a Parse file with a url and name and download the file to temporary storage
Fixed bug with ParseFile initialized to null _file value.

Let me know if these need work

helps with #36

Hi klanmiko, are you sure parse files works ?
Because I tested with a new "picture" column with png file.
And when I print the payload, I have this sample:
"picture":"{\"__type\": \"Pointer\", \"className\": \"File\", \"objectId\": \"null\"}"}

Amazing work.
I'm happy to merge and will test tonight to ensure the file upload is working correctly.
Thanks!

Thanks @phillwiggins! Thanks for making this library.
@ScanEat yes it works. The reason you see that is because the parse_encoder.dart encodes the object as a pointer and not a file. I think it is done this way for a specific reason...? to send to the server? Otherwise the print statements for the ParseUser and ParseFile, ParseGeo should be before that

Ok @klanmiko but can you show us a code sample to retrieve the picture.url from database ?

Hi Guys,
Have you checked the ParseFile upload is working?
Do you have any sample code to send me?

@klanmiko You added support for anon login, which is great! 😊 However, I wonder how to check if the currentUser is an anonymous user? Do you have a suggestion?
Thanks in advance!

@phranck you can check if
var user = await ParseUser.currentUser();
if (user != null && user['userData'] != null && user['userData']['anonymous'] != null) {
  // do anonymous user stuff
}


Default configuration doesn't produce graphs similar to the image anymore

Grokking is clearly present at 25% training data, though!

At 50%

At 25%

@averageflamethrowerguy did you ever figure out what's going on here (my graphs look like yours with default HPs)

I can also confirm that this code does not reproduce graph. I think author needs to put a note that this doesn't work.

I am getting similar results as @averageflamethrowerguy when running the code provided in the repo.
It still resembles "grokking"-like behaviour (validation curves that follow training), but not quite as extreme as in the picture on the main page.
I wonder whether:
a) the original image was created with different hyperparam settings or another architecture?
or
b) the difference is due to other factors (package updates, hardware, etc.)
@danielmamay, curious to hear your thoughts on this?

@bhairavmehta95 Thanks for the link, I'll read the paper. However, I am looking to reproduce claims of grokking behaviour on LLMs specifically which I believe the repo you shared does not seem to provide


Replace randint() with user-chosen method.

This is a very small change to the Python implementation of the BaseNoise-class. I chose the parameter-method over the subclass-method, both suggested by @caseman . I chose this one because I think the parameter-solution basically covers the other one, too (create subclass of BaseNoise, add custom randint-method, use it to call BaseNoise-constructor). And it can work even without subclassing BaseNoise.
If there are any further changes I should make or if I missed anything, please let me know.

Can we call the argument/instance var randint_function instead of randint_substitute? Other than that it looks good.

I renamed it. Hoping to make the use a bit more convenient I also added this: https://github.com/caseman/noise/pull/10/files#diff-9a0810b4e11652c8b44090f9e7388e8eR102


WIP: Replacing PowerShell Runspaces with Register-EventObject for Asynchronous handling

I coded up something that looks to be working to get some early feedback from you on my first stab at this. I think this makes things soooo much simpler and the debugging experience is grand.
I'll start working on the tests for this so we can try and have all the tests passing again before the PR is accepted.
Issue #93

@tylerl0706 - Sigh sorry about all those commits. I thought I had it but one more check in and all the history came back. If you've had experience with this before and have a suggestion let me know.
Other than that if it's really a big problem let me know and I can just wipe my fork, re-fork and drop the files in as a single change. That should do it.

@Tiberriver256 don't worry about the history, we can squash and merge :)

I'm SO excited for this PR 🎉
looks like AppVeyor isn't as happy. Can you tell me how you're running the tests locally? I can try to get AppVeyor happy again.

I haven't gotten to rewriting the tests yet. I wanted to make sure I was getting the logic right. I am experiencing some oddness passing errors up from the c# listener back (even w/ try catch blocks everywher) to PowerShell, if the network address is already taken for example, PowerShell just crashes.
I did find an older blog with a something of a genius solution of creating a c# class as just a pure event bridge and thinking of plugging that in so the listener and everything is created and handled in PowerShell the only thing c# is the handler.
http://www.nivot.org/post/2009/10/09/PowerShell20AsynchronousCallbacksFromNET
Going to give it a go and see if it's a bit friendlier with the errors.

Hey @Tiberriver256 loving the changes! I stumbled across a small bug. Using the route path as the root for a static route causes the route to be parsed incorrectly.
New-PolarisStaticRoute -FolderPath ./static -RoutePath /
When I go to http://localhost/index.html it parses the route as 'index' and returns a 404.

I did find an older blog with a something of a genius solution of creating a c# class as just a pure event bridge and thinking of plugging that in so the listener and everything is created and handled in PowerShell the only thing c# is the handler.

That's exactly what I did in that example, right? https://github.com/PowerShell/Polaris/issues/93#issuecomment-378662831

I'm a little confused by the relationship between CallbackEventBridge and AsyncHelper. I'm not sure how they interact with each other.
In the example I did before, I didn't need a CallbackEventBridge and could just Register-ObjectEvent from the class that is similar to AsyncHelper.

@tylerl0706 - This would remove the need for the AsyncHelper (Sorry forgot to remove those files). It's really close to the example you had given except the loop and all of the .Net objects we're using are managed outside of the c# class in PowerShell.
The class is more generic being just a bridge and could really be used on any project wanting to work with async and await inside of PowerShell.

Agreed 👍 this looks like the right direction

@TylerSiegrist - thanks for catching that. I'll make sure we have a test case for that and fixed in this PR.
If you want it in faster feel free to check out the contributing guidelines and submit a PR if you have a fix.

We finally have a passing build for this!! Had to do quite a bit of work on the tests but I think I came to a reasonable solution. Sorry to slog you through another giant code review @tylerl0706 but I'm looking forward to hearing what you think.

Hey ran into one more issue. In the current build of Polaris I'm able to load a snapin within a route's scriptblock. In this build I can load it but don't have access to the cmdlets, and when I do a Get-PSSnapin it doesn't appear to be loaded. Trying to load it again throws:

System.ArgumentException: An item with the same key has already been added.

Not sure if it's all snap-ins or just the one I'm working with, Citrix.PVS,Snapin.
Is there perhaps a way to expose modules & snapins to the route scriptblocks without needing to load them every time?

@TylerSiegrist - The scope changed a bit now all of the routes are executed in the main scope.
Try adding the PSSnapin outside your routes at the beginning of your script and see if the cmdlets are available inside your routes.
Still a bit odd though, I would think that adding a PSSnapin inside or outside of a scriptblock shouldn't make a difference. Could you provide some sample code?

@Tiberriver256 It seems to be working now that I pulled from your repository again, perhaps I was just a version or two behind. I'm seeing another error in the repository where I'm getting a 500 back unless I load polaris.psd1 twice. Just running your example.ps1 and browsing to http://localhost:8082/public/index.html returns a 500 with the following response:

System.Management.Automation.RuntimeException: Unable to find type [PolarisResponse].
at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)At line:33 char:41

            $response.ContentType = [PolarisResponse]::GetContent ...

@TylerSiegrist -- Hmm... okay that sounds more like something with your environment then if the base example isn't working for you as we definitely have tests that should cover those passing here on this PR in both Linux and Windows. Could you open a new issue to track / discuss with your $PSVersionTable and I can help troubleshoot?

I'm excited to review this!! 🎉🎉🎉🎉
Give me a couple days since it's quite the change :)

ok... here we go...

@tylerl0706 - Round 3 has arrived. See how you like it! :)

"linux case sensitivity... so sensitive" lol

Looking forward to it!!

No problem man! I appreciate the awesome reviews. This thing is looking pretty sweet.

Keep em coming 😀

IT'S TIME 🎉🎉🎉🎉

Awesome work @Tiberriver256 😃

ok this should be the last rewrites for a bit at least I'll get this signed and on the Gallery in the next couple days :)


MCS-1048: Loading Addressables Indicator Shown

Jira: https://nextcentury.atlassian.net/browse/MCS-1048
This implements a spinning loading icon that appears when addressables are being loaded. I imported a free package of different loading icons for this.
It currently uses a loading animation that looks like gears turning, but feel free to open the MCS-Starter-Scene, find the LoadingAddressables object under the Canvas, and swap out the "Cog 3" object for another prefab found in the "Animated Loading Icons" folder if you find another one you like better.

We may also want to move the above-mentioned folder, just let me know!
I tried to implement a progress bar, but unfortunately this isn't possible with the AddressablesUtil script, since addressable loading is done synchronously instead of asynchronously in that script.

@alexcraneunity We should probably delete all of the logos and animations we arent using if we agree to use the cogs. We can always add that them back later if we need them.

@alexcraneunity We should probably delete all of the logos and animations we arent using if we agree to use the cogs. We can always add that them back later if we need them.

That was my thinking as well. I left them in in case anyone wants to try out other animations, but I'll remove the extras before merging.

@ThomasSchellenbergNextCentury  I moved the "loading addressables" text higher. Let me know if this is enough

!

Before I merge this, just checking: Is the great icon the one we want to use moving forward? Here are some other examples as well:

@alexcraneunity That positioning looks good -- thank you! I don't mind those other icons, but I think that I like the gears the best!

Overall, I really like this, however for me the canvas items flicked a bit as the scene loads and ended up on the video.

@kdrumm  Can you explain a bit more on what you mean by flicked? The icon animation should pop up in between every JSON scene as it loads in addressable objects.
@brianpippin Sounds good. I'll delete all the others. Should I move the "Animated Loading Icons" folder into the UnityAssetStore folder (though it's a free asset)?

@kdrumm Can you explain a bit more on what you mean by flicked? The icon animation should pop up in between every JSON scene as it loads in addressable objects.

Sorry, i meant 'flickered'.  It would show up and then disappear and then show up again while loading a single scene.  It would also appear when the scene was loaded and would end up on the visual video when run from the MCS python side.  It may be just be something on my end if no one else had this issue.

@kdrumm Can you explain a bit more on what you mean by flicked? The icon animation should pop up in between every JSON scene as it loads in addressable objects.

Sorry, i meant 'flickered'. It would show up and then disappear and then show up again while loading a single scene. It would also appear when the scene was loaded and would end up on the visual video when run from the MCS python side. It may be just be something on my end if no one else had this issue.

Ah that makes sense. That is expected behavior since it shows up every time addressables are being loaded, though I'm not sure what the "visual video" is.
Another thing worth noting is that this currently errors our on test 93, with this stack trace:
Error Message = System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object   at MCSMain.AssignProperties (UnityEngine.GameObject gameObject, MCSConfigGameObject objectConfig, MCSConfigObjectDefinition objectDefinition) [0x00001] in C:\CACI-MCS\ai2thor\unity\Assets\Scripts\MCSMain.cs:1050    at MCSMain.CreateCustomGameObject (MCSConfigGameObject objectConfig, MCSConfigObjectDefinition objectDefinition) [0x00089] in C:\CACI-MCS\ai2thor\unity\Assets\Scripts\MCSMain.cs:1572    at MCSMain.CreateGameObject (MCSConfigGameObject objectConfig) [0x00021] in C:\CACI-MCS\ai2thor\unity\Assets\Scripts\MCSMain.cs:1620    at MCSMain.InitializeGameObject (MCSConfigGameObject objectConfig) [0x00001] in C:\CACI-MCS\ai2thor\unity\Assets\Scripts\MCSMain.cs:1716    at System.Collections.Generic.List1[T].ForEach (System.Action1[T] action) [0x00024] in <eae584ce26bc40229c1b1aa476bfa589>:0    at MCSMain.ChangeCurrentScene (MCSConfigScene scene) [0x00133] in C:\CACI-MCS\ai2thor\unity\Assets\Scripts\MCSMain.cs:289    at MCSController.Initialize (ServerAction action) [0x000e3] in C:\CACI-MCS\ai2thor\unity\Assets\Scripts\MCSController.cs:282    at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)   at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in <eae584ce26bc40229c1b1aa476bfa589>:0
Is this a known current issue, and does anyone else run into that problem?

Another thing worth noting is that this currently errors our on test 93, with this stack trace:
Error Message = System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object at MCSMain.AssignProperties (UnityEngine.GameObject gameObject, MCSConfigGameObject objectConfig, MCSConfigObjectDefinition objectDefinition) [0x00001] in C:\CACI-MCS\ai2thor\unity\Assets\Scripts\MCSMain.cs:1050 at MCSMain.CreateCustomGameObject (MCSConfigGameObject objectConfig, MCSConfigObjectDefinition objectDefinition) [0x00089] in C:\CACI-MCS\ai2thor\unity\Assets\Scripts\MCSMain.cs:1572 at MCSMain.CreateGameObject (MCSConfigGameObject objectConfig) [0x00021] in C:\CACI-MCS\ai2thor\unity\Assets\Scripts\MCSMain.cs:1620 at MCSMain.InitializeGameObject (MCSConfigGameObject objectConfig) [0x00001] in C:\CACI-MCS\ai2thor\unity\Assets\Scripts\MCSMain.cs:1716 at System.Collections.Generic.List1[T].ForEach (System.Action1[T] action) [0x00024] in <eae584ce26bc40229c1b1aa476bfa589>:0 at MCSMain.ChangeCurrentScene (MCSConfigScene scene) [0x00133] in C:\CACI-MCS\ai2thor\unity\Assets\Scripts\MCSMain.cs:289 at MCSController.Initialize (ServerAction action) [0x000e3] in C:\CACI-MCS\ai2thor\unity\Assets\Scripts\MCSController.cs:282 at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&) at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in <eae584ce26bc40229c1b1aa476bfa589>:0
Is this a known current issue, and does anyone else run into that problem?

After rebasing on dev and rebuilding addressables, it no longer hands on test 93. However, I still do get this error above as well as failed integration tests.
Specifically, it fails these tests:
FAILED INTEGRATION TESTS:

    031.stacking_blocks_bottom_to_top (level1): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    031.stacking_blocks_bottom_to_top (level2): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    031.stacking_blocks_bottom_to_top (oracle): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    032.stacking_blocks_top_to_top (level1): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    032.stacking_blocks_top_to_top (level2): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    032.stacking_blocks_top_to_top (oracle): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    033.stacking_blocks_bottom_to_bottom (level1): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    033.stacking_blocks_bottom_to_bottom (level2): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    033.stacking_blocks_bottom_to_bottom (oracle): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    034.stacking_blocks_top_to_bottom (level1): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    034.stacking_blocks_top_to_bottom (level2): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    034.stacking_blocks_top_to_bottom (oracle): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    035.stacking_blocks_side_to_top_v1 (level1): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    035.stacking_blocks_side_to_top_v1 (level2): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    035.stacking_blocks_side_to_top_v1 (oracle): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    036.stacking_blocks_side_to_top_v2 (level1): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    036.stacking_blocks_side_to_top_v2 (level2): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    036.stacking_blocks_side_to_top_v2 (oracle): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    037.stacking_blocks_bottom_to_side_v1 (level1): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    037.stacking_blocks_bottom_to_side_v1 (level2): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    037.stacking_blocks_bottom_to_side_v1 (oracle): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    038.stacking_blocks_bottom_to_side_v2 (level1): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    038.stacking_blocks_bottom_to_side_v2 (level2): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    038.stacking_blocks_bottom_to_side_v2 (oracle): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    039.stacking_blocks_many (level1): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    039.stacking_blocks_many (level2): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    039.stacking_blocks_many (oracle): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    040.stacking_blocks_too_far (level1): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    040.stacking_blocks_too_far (level2): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    040.stacking_blocks_too_far (oracle): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    093.drop_rectangular_objects (level1): Step 3 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    093.drop_rectangular_objects (level2): Step 3 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    093.drop_rectangular_objects (oracle): Step 3 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    095.torque_object (level1): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    095.torque_object (level2): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    095.torque_object (oracle): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    098.rotate_object (level1): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    098.rotate_object (level2): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    098.rotate_object (oracle): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    100.rotate_stacking_objects (level1): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    100.rotate_stacking_objects (level2): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    100.rotate_stacking_objects (oracle): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    102.move_object (level1): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    102.move_object (level2): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    102.move_object (oracle): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    104.move_stacking_objects.actions (level1): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    104.move_stacking_objects.actions (level2): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    104.move_stacking_objects.actions (oracle): Step 1 failed:

        return_status: NOT_OBJECT != SUCCESSFUL

    110.tool_cannot_walk_into (oracle): Step 0 failed:

        objects tool texture_color_list: ['black'] != ['grey', 'black']

    111.tool_push (oracle): Step 0 failed:

        objects tool texture_color_list: ['black'] != ['grey', 'black']

    112.tool_move (oracle): Step 0 failed:

        objects tool texture_color_list: ['black'] != ['grey', 'black']

    113.tool_torque (oracle): Step 0 failed:

        objects tool texture_color_list: ['black'] != ['grey', 'black']

    114.tool_rotate (oracle): Step 0 failed:

        objects tool texture_color_list: ['black'] != ['grey', 'black']

    115.tool_push_into_soccer_ball (oracle): Step 0 failed:

        objects tool texture_color_list: ['black'] != ['grey', 'black']

    116.tool_move_into_soccer_ball (oracle): Step 0 failed:

        objects tool texture_color_list: ['black'] != ['grey', 'black']

@ThomasSchellenbergNextCentury I finished addressing each of those points and testing. My MCS in my VM was actually not up to date, so I had to install pydantic and then update the local repo.
I made sure my branch of AI2Thor was rebased on the latest commit on the development branch.
I also pulled over the latest UnityAssetStore folder, but do I need the UnityAssetStoreNonAddressables folder? If so, where does it go?
Then, I rebuilt addressables, built an executable of the application, and ran the server to test it.
Unfortunately, my application seems to crash on startup at this point, so I'm working on resolving that and seeing what I might be missing. Any ideas?
Here is the returned error:
RUNNING SCENE: 001.empty_room_movement.scene.json

Traceback (most recent call last):
  File "/home/alex/MCS/integration_tests/run_tests.py", line 21, in <module>
    start_handmade_tests(
  File "/home/alex/MCS/integration_tests/run_handmade_tests.py", line 349, in start_handmade_tests
    successful, status = run_single_scene(
  File "/home/alex/MCS/integration_tests/run_handmade_tests.py", line 224, in run_single_scene
    step_metadata = controller.start_scene(scene_data)
  File "/home/alex/venv/lib/python3.9/site-packages/typeguard/__init__.py", line 1033, in wrapper
    retval = func(*args, **kwargs)
  File "/home/alex/MCS/machine_common_sense/controller.py", line 192, in start_scene
    (pre_restrict_output, output) = self._output_handler.handle_output(
  File "/home/alex/MCS/machine_common_sense/controller_output_handler.py", line 266, in handle_output
    unrestricted = self._get_step_metadata(
  File "/home/alex/MCS/machine_common_sense/controller_output_handler.py", line 329, in _get_step_metadata
    self._scene_event.structural_object_list)
  File "/home/alex/MCS/machine_common_sense/controller_output_handler.py", line 151, in structural_object_list
    return self._get_objects('structuralObjects')
  File "/home/alex/MCS/machine_common_sense/controller_output_handler.py", line 134, in _get_objects
    [
  File "/home/alex/MCS/machine_common_sense/controller_output_handler.py", line 135, in <listcomp>
    self.retrieve_object_output(
  File "/home/alex/MCS/machine_common_sense/controller_output_handler.py", line 241, in retrieve_object_output
    associated_with_agent=object_metadata['associatedWithAgent'],
KeyError: 'associatedWithAgent'

@alexcraneunity I think something weird might have happened when you rebased, since you're still out-of-sync with the latest development. For example, in AgentManager, the ObjectMetadata class has an associatedWithAgent property on development, but not in your branch. Here's the code from development: https://github.com/NextCenturyCorporation/ai2thor/blob/development/unity/Assets/Scripts/AgentManager.cs#L1177

@ThomasSchellenbergNextCentury I think you were right. I rebased again and it no longer crashes on startup. I let it run all the way through the integration tests just to be sure, and sure enough it was able to execute all of them and never errored or crashed. However, it did fail these tests:

Is this a cause for concern?

I believe I had errors similar to that when I had part of @jaudick fixes for agent collisions.  Although those were fixed and merged.  Its possible that the jenkins build issues yesterday caused you to have a similar problem, but I am assuming you are using your own aithor build with your own built addressables.  If you haven't merged ai2thor in a couple days, that may be necessary.

Thank you @kdrumm. I pulled MCS again on my VM since I hadn't for a few days, and ran it again with no failed tests! Are we alright to go ahead and merge this then?

Maybe we could keep the LoadingAddressables object ONLY on the MCS-Starter-Scene?  I'm not sure if there are negative implications of this though.

I've looked at the code and I don't understand why it does this, but the 'loading addressables...' dialog comes back for the first frame (after I type in Pass). If I am the only one who gets this, we can assume its a local issue. If it is common, I think it is a show-stopper, unfortunately. Otherwise, I really like this.

Wanted to confirm I'm seeing this too on the first frame.

I have now removed the loading icon for every scene except the starter scene. This should resolve the first frame issue listed above. @brianpippin @kdrumm @rartiss55


'Namespace' object has no attribute 'key'

After i run the file in the terminal It started to show the error
'Namespace' object has no attribute 'key'
How can i override this?

Hi there
I've fixed the issue. I did not test in an environment which didn't have encryption libraries installed.
Please pull the script and try it now.

Thanks for the feedback, I was able to override this by installing
"Pycryptodome" since pycrypto would show a c++ error.
The encryption isn't working still but at least the embedding if now
possible.
Thanks
On Mar 23, 2017 17:02, "Adrian-George Bostan"<EMAIL_ADDRESS>wrote:

Hi there
I've fixed the issue. I did not test in an environment which didn't have
encryption libraries installed.
Please pull the script and try it now.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/adrg/lsbsteg/issues/1#issuecomment-288728470, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AGlj9edOQNBwBbziwmscK80r49Lub_-Mks5ront0gaJpZM4Ml95-
.


Fix/add tabindex to language switcher

🔗 Linked issue

❓ Type of change

[ ] 📖 Documentation (updates to the documentation, readme or JSdoc annotations)
[x] 🐞 Bug fix (a non-breaking change that fixes an issue)
[ ] 👌 Enhancement (improving an existing functionality like performance)
[ ] ✨ New feature (a non-breaking change that adds functionality)
[ ] 🧹 Chore (updates to the build process or auxiliary tools and libraries)
[ ] ⚠ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description
This PR adds a tabindex to the language switcher component to improve accessibility. Added the change in the changelog accordingly.
📝 Checklist

[ ] I have linked an issue or discussion.
[ ] It's submitted to the main branch
[ ] When resolving a specific issue, it's referenced in the PR's title (e.g. fix #xxx[,#xxx], where "xxx" is the issue number)
[x] I have updated the documentation accordingly.
[ ] All tests are passing
[ ] New/updated tests are included
[ ] I have updated the "upcoming" section inside docs/changelog.md explaining the changes I contributed

If adding a new feature, the PR's description includes:

[ ] A convincing reason for adding this feature (to avoid wasting your time, it's best to open a suggestion issue first and wait for approval before working on it)

Other information:

I have moved the tabindex attribute to an inner container, so only the language gets selected without the separator
Before

After