[3.x] Allow exporting custom resources from/to any scripting language (GDScript, VisualScript, C#, NativeScript, PluginScript)

This PR adds the ability to export custom resources and arrays of custom resources of any language's type from an exported property defined in any language.
Closes godotengine/godot-proposals#18, at least, in Godot 3.2.
This builds on top of the commit in #44796, so that would have to be merged first. This now includes the PR content of #44796 as of the 3.x rebase. For a detailed breakdown of the features commit-by-commit, see the 4.0 version at #48201.
This is a continuation of work started by me and @JFonS and would replace his 3.2 PR which deals exclusively with GDScript (#41264).
It includes editor changes to support script classes as valid resource exports.
Because the definition of PropertyInfo objects is manual in VisualScript, NativeScript, and PluginScript, the feature works out-of-the-box for those languages.
GDScript support was already implemented by JFonS's PR, but I've incorporated those changes and more.
C# support is added as well. The [Export] attribute now supports the following forms...
// Infer hint and hint string (no changes)
[Export]

// Manually define hint and hint string. Only applies if valid hint cannot be calculated (no changes).
[Export(PropertyHint.None, "")]

// Applies custom resource name if applicable to the member's data type (new!).
// Only necessary when exporting a Resource or List<Resource> for a non-C# resource type.
[Export("ClassName")]

Demo Project:
Demo_GP22_3_2_Resources.zip
Screenshots:
GDScript usage, exported properties contracted:

GDScript usage, exported properties expanded:

Note I also have export var gd_data: GDData syntax supported as well (so you can infer the export type from the GDScript type hint).
C# usage doing the same thing:

This qualifies Godot as a tier one engine for data intensive games. Congrats @willnationsdev, @JFonS and all devs involved.

This qualifies Godot as a tier one engine for data intensive games. Congrats @willnationsdev, @JFonS and all devs involved.

@vitorbalbio I will say that, for simulation-heavy games that also require a lot of data, this alone won't really be enough. This is merely a big usability improvement for general-purpose games that won't data-driven behavior. To get a significant runtime performance improvement (i.e. processing over large amounts of data), you'd have to resort to writing module code with a custom CPU cache-efficient code with your own Server implementations, etc.
@AndreaCatania is doing great work in this domain as he is writing a module that introduces an ECS framework that can run alongside the Godot Engine runtime and cooperates with the scripting API. He has seen major performance improvements for any simulation logic he moves over to his module's systems, so if you need a game like that, keep an eye on his development. He has multiple customizable system pipelines defined with editor tools. Time will tell if the developers ever decide to actually integrate his ECS module with the core engine (they absolutely should though).

@vitorbalbio I will say that, for simulation-heavy games that also require a lot of data, this alone won't really be enough. This is merely a big usability improvement for general-purpose games that won't data-driven behavior. To get a significant runtime performance improvement (i.e. processing over large amounts of data), you'd have to resort to writing module code with a custom CPU cache-efficient code with your own Server implementations, etc.
@AndreaCatania is doing great work in this domain as he is writing a module that introduces an ECS framework that can run alongside the Godot Engine runtime and cooperates with the scripting API. He has seen major performance improvements for any simulation logic he moves over to his module's systems, so if you need a game like that, keep an eye on his development. He has multiple customizable system pipelines defined with editor tools. Time will tell if the developers ever decide to actually integrate his ECS module with the core engine (they absolutely should though).

@vitorbalbio I will say that, for simulation-heavy games that also require a lot of data, this alone won't really be enough. This is merely a big usability improvement for general-purpose games that want data-driven behavior. To get a significant runtime performance improvement (i.e. processing over large amounts of data), you'd have to resort to writing module code with custom CPU cache-efficient code with your own Server implementations, etc.
@AndreaCatania is doing great work in this domain as he is writing a module that introduces an ECS framework that can run alongside the Godot Engine runtime and cooperates with the scripting API. He has seen major performance improvements for any simulation logic he moves over to his module's systems, so if you need a game like that, keep an eye on his development. He has multiple customizable system pipelines defined with editor tools and script code. Time will tell if the developers ever decide to actually integrate his ECS module with the core engine (they absolutely should though).

I mean more like RPGs and Tactical games that you need to define lots of data/status for large amount of items, objects and so on. The same use cases where Scripted Objects of Unity shines.

@vitorbalbio I will say that, for simulation-heavy games that also require a lot of data, this alone won't really be enough. This is merely a big usability improvement for general-purpose games that want data-driven behavior. To get a significant runtime performance improvement (i.e. processing over large amounts of data), you'd have to resort to writing module code with custom CPU cache-efficient code with your own Server implementations, etc.
@AndreaCatania is doing great work in this domain as he is writing a module that introduces an ECS framework that can run alongside the Godot Engine runtime and cooperates with the scripting API. He has seen major performance improvements for any simulation logic he moves over to his module's systems, so if you need a game like that, keep an eye on his development. He has multiple customizable system pipelines defined with editor tools and script code. Time will tell if the developers ever decide to actually integrate his ECS module with the core engine (they absolutely should though).

I mean more like RPGs and Tactical games that you need to define lots of data/status for large amount of items, objects and so on. The same use cases where Scripted Objects of Unity shines.

@vitorbalbio Ah, yeah. I really would like to see a proper spreadsheet class added to the engine/Editor to simplify working with tabular data with full static typing and all the custom editors from the Editor integrated out-of-the-box. godotengine/godot-proposals#13 That would make it much easier to just create large amounts of data for projects like those.

@vitorbalbio Ah, yeah. I really would like to see a proper spreadsheet class added to the engine/Editor to simplify working with tabular data with full static typing and all the custom editors from the Editor integrated out-of-the-box. godotengine/godot-proposals#13 That would make it much easier to just create large amounts of data for projects like those.

Fixed a bug where type hints for GDScript inner classes were mistakenly not parsing to the end of the line anymore.

Fixed a bug where type hints for GDScript inner classes were mistakenly not parsing to the end of the line anymore.

Note that someone on Reddit reported a bug with this where referencing an inner class within the same file using a relative symbol name doesn't work (even though I fixed it so the fully-qualified name in other scripts works).

Note that someone on Reddit reported a bug with this where referencing an inner class within the same file using a relative symbol name doesn't work (even though I fixed it so the fully-qualified name in other scripts works).

Fixed a bug where script classes that were part of a nested res://addons/* directory weren't showing up in the CreateDialog unless they were part of an enabled plugin. I've now changed the CreateDialog code to explicitly check for the presence of a plugin.cfg file within the directory, and if there isn't one, it will continue to display the script class in the tree of types.

Fixed a bug where script classes that were part of a nested res://addons/* directory weren't showing up in the CreateDialog unless they were part of an enabled plugin. I've now changed the CreateDialog code to explicitly check for the presence of a plugin.cfg file within the directory, and if there isn't one, it will continue to display the script class in the tree of types.

Fixed a bug where, once you had assigned a script class to a node, there was no way to remove the script class from the node because the Change Type operation would preserve the script property value even if the node you were "changing type" to was explicitly the instance base type.

Fixed a bug where, once you had assigned a script class to a node, there was no way to remove the script class from the node because the Change Type operation would preserve the script property value even if the node you were "changing type" to was explicitly the instance base type.

Added a warning message when explicitly removing a built-in script. Built-in scripts' inheritance cannot be determined because they are never compiled, ergo their get_base_script() method won't produce a result (at least, in GDScript). Added a WARN_PRINT_ONCE so that users are not alarmed/confused as to why the entire script was removed rather than removing just down to the script class like it should (mimicking the custom type feature).

Added a warning message when explicitly removing a built-in script. Built-in scripts' inheritance cannot be determined because they are never compiled, ergo their get_base_script() method won't produce a result (at least, in GDScript). Added a WARN_PRINT_ONCE so that users are not alarmed/confused as to why the entire script was removed rather than removing just down to the script class like it should (mimicking the custom type feature).

what is the development status of this PR?

what is the development status of this PR?

I'm not working in 3.2 so I'll let some else review.

I'm not working in 3.2 so I'll let some else review.

@DmitriySalnikov It's status is that no one has thus far reported any issues that I have not fixed, so it's probably in a beta state or something. We need more people to use it / review it to confirm if there are any unforeseen problems.

@DmitriySalnikov It's status is that no one has thus far reported any issues that I have not fixed, so it's probably in a beta state or something. We need more people to use it / review it to confirm if there are any unforeseen problems.

Some days ago I tried https://github.com/willnationsdev/godot/commit/9ff013e7c0d66ea8d6a0a659ea96a866b08e78a2 with a small game that used custom resources for enemies and turrets. Everything was working as expected. Unfortunately I didn't push the branch that I used to test the new features.
https://github.com/godotengine/godot/issues/41442 was still an issue for me, but I don't know if it's related to this PR.

Some days ago I tried https://github.com/willnationsdev/godot/commit/9ff013e7c0d66ea8d6a0a659ea96a866b08e78a2 with a small game that used custom resources for enemies and turrets. Everything was working as expected. Unfortunately I didn't push the branch that I used to test the new features.
https://github.com/godotengine/godot/issues/41442 was still an issue for me, but I don't know if it's related to this PR.

So I have been working with this for a bit and love it, is it going to be merged for the next RC?

No, but we can merge it for the first 3.2.5 beta if I can get some contributors to review it :)

would really like to see this merged.. is great news!

Working on a game that could really use this, hopefully it can get reviewed soon

I'm really glad I found this because I was already spending time trying to implement a different fix for the same problem. Your solution is much more generic and elegant. Finding it now prevented me from spending a lot more time on my approach :sweat_smile:.
@willnationsdev Feedback:
This works great! I haven't found any issues with it so far however I do have a question (which may actually be an issue or request in disguise). Is there a way to set a default value for these custom resources?
I am working with my own simple experiment using your branch to test the boundaries, and to me it seemed natural to try to set a default value. I tried two main things.
Note that the static typing in these examples is more verbose/redundant than it needs to be, and is not really part of the problem.
But I only get one of the lint errors I am going to mention if I annotate the property as : GdData.
Exporting an instance
This makes the most sense to me logically because you are setting the variable to a new instance of the type, which is what I believe is expected.
However this shows as [empty] in the inspector and is not editable.
export(GdData) var gdData: GdData = preload("res://GdData.gd").new()

Exporting the class itself
This makes less sense, but the linting error it gives is interesting:

The assigned expression's type (GdData) doesn't match the variable's type (GdData)

I suspect despite being stringified the same, the former is actually "GdData class" and the latter is "GdData instance" but you would know better.
export(GdData) var gdData: GdData = preload("res://GdData.gd")

Closing
I guess it kind of doesn't make sense to statically set a default export variable to a new instance because that might imply the instance should be shared across all of them... but I can't really think of any other ways to set a default and was wondering if you could suggest anything.
Thank you very much for your time on this, it will make working on games the way I am used to much more sane! <3

@rosshadden

This works great!

That's good to hear. In particular though, I haven't heard much feedback from folks about the way the script classes now work in the SceneTreeDock. Could you experiment with adding scripts, removing scripts, changing types, and doing all the same with anonymous scripts that derive script classes too? Get a feel for how the new relationships work and tell me what you think?
The first commit, associated with a different PR, changes the way script classes work so that the SceneTreeDock treats them more like built-in C++ classes, and no one has really commented on that so far, despite probably being the most controversial aspect of the changes (and which are totally subject to change).

However this shows as [empty] in the inspector and is not editable.

I would've expected your "exporting an instance" example to work. I would've also expected this to work:
export var gdData := GdData.new()

If not though, then that means it's a bug and needs to be fixed. That other bug you mention sounds like it could be the culprit though.

I suspect despite being stringified the same, the former is actually "GdData class" and the latter is "GdData instance" but you would know better.

This is correct. We could probably make the error message in that case a little bit more clear.

Finding it now prevented me from spending a lot more time on my approach
Thank you very much for your time on this

Yeah, I spent a lot of time on this approach myself (these two commits are, like, a year's collective research/development/debugging in my off time, lol). Glad it's finally helping people.

@rosshadden Another thing you could try (although this might be not really what you want) is to define a setter for the variable so that whenever someone attempts to assign it to null, it instead gets assigned a new instance of the class. That will then make the default initialization always populate it with a ready-to-edit object in the Inspector. Though...I haven't actually tried that and don't know if that could cause any problems.

@willnationsdev I played with your changes a lot more, in particular some of the things you mentioned like using script classes in the SceneTreeDock. For example I created a "New Resource" and chose a custom class as the resource type. After doing so all normal operations with the resulting .tres file work as I expected. The type stored in the tres file is "Resource" and not the specific class I chose, but this isn't untrue as they all extend Resource anyway.
I actually haven't encountered any issues with this so far other than the one I mentioned above where you cannot set a default (again, seems to be a problem in 3.2 in general, not introduced by your changes). I did try your other suggestions for setting defaults as well as a few more and none of them worked. Nor did creating a setter.
I also experimented with nesting resources which worked as well. I haven't tried nesting a resource as a property inside itself, but that's rather silly. I feel like at this point if there are problems they won't be caught with the workflows I am using this for, and thus I give it my :+1:. Hopefully this can gain traction and others can review as well so we can catch any problems with it and fix them.
BTW the approach I mentioned I was working on before I found this PR was adding a "Make instances unique" menu option at the resource level which would basically do what clicking "Make Unique" does per resource instance (the OBJ_MENU_MAKE_UNIQUE resource menu option). The benefit being you could set this in one place, on a resource, instead of having to click it every time you use a certain resource. In my experience most of the time you want a resource to be unique you want it to be unique everywhere.
This PR does essentially exactly this in effect, because the end result of both approaches is that the properties of the resource are stored in the scene file in question.

Basing on @akien-mga comment it seems that this pull request is frozen until some good soul (i.e. contributor well-versed in the way of Godot) reviews it (and yes, there is a bit of code to be reviewed ;)). That said, I am guessing in order to push it a little closer to RC wagon some testing would be helpful I guess?

It's not going to be merged for 3.2.4 RCs, but yes testing would be very useful. This will likely be reviewed and merged for 3.2.5 betas, but any issues found prior to that and fixed in the PR would be quite useful.

Thank you for this feature!
I was testing it with a project and I think I found a bug.
Specifically with C# when using a [Tool] script trying to access to a property of a custom resource, running the script in editor causes a godot crash.
Here is the repo of the project I tested.
I don't tested this project with GDScript, but I tested this behaviour (or the same case) with GDScript and it works well
Steps to reproduce:

Build project solution.
Open ErrorTest.tscn in root folder.
Test it by changing some script variables of the parent in the editor and clicking "update" checkbox (The custom resource is called HeightTerrain).
Open MapUtils.cs in Scripts folder on script editor.
Uncomment line 25 then build project solution.
Run the scene and see the console print -> should work
Click "update" checkbox -> should crash

Apparently the crash occurs because the class object is not defined in the editor before running the scene, if we change the color property of the custom class HeightTerrain and change the print of line 25 with heightTerrain.Color it will print a random value like (0,0,0,13e-20). When running the scene it works good.
I don't know if this is my problem so I share it here, I anticipate to apologize if the error was something that I missed.

Would be great to get this merged soon for 3.4, and it should likely also be forward-ported to the master branch. Most core refactoring to script languages has happened in master already so if this issue hasn't been solved in the process, it still needs to be.
Would be great to have in-depth review by @vnen and @neikeq, and @godotengine/gdnative stakeholders.
Would also be good to figure out the crash reported in https://github.com/godotengine/godot/pull/44879#issuecomment-803851774 before we merge.

@willnationsdev Do you expect to have a version of this PR available for the master branch? It would make sense to make this change there first, even if it's not implemented for GDNative there yet. It might also be easier to merge since it would be changing fewer things at once.

Yeah, I could try and make time for that. I wanted it to be in master in the first place, hehe.

@willnationsdev
Curious if this would fix the bug I am seeing, or if I should file a new bug.
If I create a custom resource from C#, the script on it seems to lack a proper resource_path, meaning it gets saved to disk wrong, can't load later, and overall has issues. I assume this is because the script resource was not loaded through the ResourceLoader's cache.
In other words:
var cust = new CustomResource();
GD.Print($"script path = { ((Script)cust.GetScript())?.ResourcePath }");

prints: script path =

Possibly. I'm actually not really sure. Does GDScript print something more reasonable in its equivalent? If so, it could be related. I don't know too much about the inner workings of resource loading and don't have time to investigate atm. Perhaps someone else can chime in?

Some explanation on Resources loading/saving just in case it relates to this PR:
ResourceLoader tries to cache the resources it loads, including scripts. This is basically done by using resource_path as a kind of "cache key." If you call load() with the same path twice, you get one shared reference.
Usually scripts are loaded from the FileSystem, and are subsequently caching using their file path as their resource_path. Scene and resource files use this caching as well when loading an ext_resource, like a script.
When creating a custom resource in C# using its constructor, the script appears to be set as a non-cached version. That means when it comes time to save, the custom resource's script gets saved as a sub_resource instead of an ext_resource.
What gets saved:
[sub_resource type="CSharpScript" id=1]

What should have been saved:
[ext_resource path="res://CustomResource.cs" type="Script" id=2]

A subresource in a scene or resource file should contain below it everything needed to create it. An external resource in a scene or resource file simply loads or gets the cached file for the given path.
As you can see above, the sub_resource version contains nothing. The script cannot be loaded, and even if it could be, it would not be the cached, shared version that the rest of the system would be using. Manually edit the scene file to replace the sub_resource with an ext_resource and your custom resource will magically start working again!

That certainly sounds like a solution the problem then if the custom resource works after fixing that issue. The next step would just be figuring out why it saves it as a sub resource instead of as an external resource. I previously didn't know the function of resource_path, but I knew the resources were cached, so that makes a lot of sense now.
If you have time to figure out the details further, by all means. I'm neck-deep in work stuff atm. XD

Hmm... looks like it's happening with all objects that have a custom C# script. All those scripts are technically detached from the cached one. I'm going to file it as a separate bug.

Have used this PR casually in my project for awhile now, nothing directly to add except that for my project it's working great! (non C# project).
Really awesome work, makes using custom resources so much nicer!

@phelioz, thanks for the confirmation that it is working well for you in a non-C# project.

I just pushed a change that includes the XML documentation for all changes to the scripting API introduced by this PR. I also amended the ScriptServer.get_global_class_map() implementation so that Dictionary for each class contains not only keys for "language", "path", and "base", but also "native" and "icon_path". This way, if you need to avoid making repeated queries to the ScriptServer's methods across the scripting API, you can just grab the entire Dictionary of data and use it locally.

@cgbeutler Lol, I finally got around to testing out your C# bug. Figured it out. It is indeed a user error. All you had to do was add the [Tool] attribute to your HeightTerrain C# script, in addition to the [Global] attribute ([Global("HeightTerrain")] is unnecessary). Now that your custom type is also capable of being loaded at editor time, since it's a tool script, your logic now ends up working fine.

I backported some of the cleanliness updates that I made during the 4.0 port and improved the degree to which the 3.x and 4.0 versions match 1-to-1 (to make comparing the PRs easier). I also identified and fixed a bug where the EditorPropertyResource's menu options wouldn't handle file extensions in cases where multiple classes are provided and one of those classes is a script class.
With this, and identifying the C# issue was a user problem, this PR should be pretty ready to merge whenever it can be properly reviewed.

@willnationsdev I think you're mixing me up with something else. I have no HeightTerrain class... Also, I didn't forget tool.
Regardless, my thing is a separate bug.

@cgbeutler Ah, you're right, my bad. It was @tomasjm who reported the C# crash bug. (For context, tomasjm, you just needed to add [Tool] to your HeightTerrain script)

@cgbeutler Ah, you're right, my bad. It was @tomasjm who reported the C# crash bug. (For context, tomasjm, you just needed to add [Tool] to your HeightTerrain script)

omg, thank you, my bad! I had not reviewed the code with the freshest mind to find that haha, hope this PR gets merged, nice work!

I've been using this patch almost since it was released and haven't had any problems so far.
However, recently I updated to the latest version and loading custom resources in the editor is now broken. Exporting custom resources works fine, it is displayed correctly in the inspector, I can click the "New CustomResource" button and it correctly creates a new resource, but the load function fails with a warning "The selected resource (Resource) does not match any type expected for this property (GDData)."
At the moment when I press "Load" and the file-open-dialog shows, the console spams about 30 times "Cannot get class 'GDData'.".
Tested with the most recent gdres-all-3.2 branch and the demo project.

@mphe what gets saved to the tscn file for the ones that won't load? For me, In the past, it has specifically been the script resource that isn't linked properly. (See #48828)

@cgbeutler Nothing gets saved. The property remains unchanged and keeps the value it had before.

@mphe

Have you been regularly updating to the latest version of the gdres-all-3.2 branch thinking that it only applies to the 3.2 tag, cause it actually has been attempting to keep track with latest versions of the 3.x branch of the Godot repo. There is a preserved 3.2 branch with gdres-all-3.2-backup if that is what you need.
Just to be sure, are you somehow using the sample project I originally published with this that had GDData as a resource type?  Or does your project have independent stuff so that an error message like that makes no sense? If the latter, then maybe I somehow over-committed a change that kept info about a demo project's resource type? Don't see how that could have happened though.
Which language is your custom resource written in and in which language are you exporting a reference to that resource type? For example, is it a C# class that owns a GDScript resource? Are there certain combinations that do not create problems?

I'm following the 3.x branch and update every 1-3 months or after official releases. Hence, I can't say exactly when this bug appeared, but I think the last time it definitely worked was at the 3.3.2 release. Just for clarity, when updating I create a clean branch from upstream 3.x and merge this gdres-all-3.2 branch (among others).
After I noticed this problem in my project, I cloned and built your gdres-all-3.2 branch directly and tested it with the demo project from your top post. Since the problem persists, it is unlikely to be related to my project and godot version.
I'm only using GDScript. Not tested with C#.

@mphe Gotcha. Thanks. I'll look into it tonight.

@mphe
So, I investigated as much as I could, but I couldn't seem to replicate any kind of error similar to the one you described.
Based on the error message, it sounds like there is a property typed for GDData that is being assigned a basic Resource (or at least, that's what the GDScriptParser appears to think is happening?), so that would indicate to me that there is some GDData property info configured somewhere, but the Editor doesn't know what that is, so the assigned GDData instance is interpreted as just a basic Resource, hence the error - or something like that.
However, I did a fresh update from upstream/3.x, rebased the gdres-all-3.2 branch, and did not encounter any issues with the demo projects' various scenes. I even tried creating new resources, saving them to files, and loading them from files (since you said you clicked 'load', opened the dialog, and got spammed with errors). I saw no errors of that sort when I was messing around.
Not really sure what your issue is since I can't seem to reproduce it.
I did notice there was some sort of bug with GDScript's parsing of the class_name statement when an existing global GDScript is renamed or moved, even via the Filesystem Dock, but restarting the editor appears to fix that issue (probably just a cache issue in the ScriptServer code).
I also noticed that the newfound EditorResourcePicker breaks the display of the proper class name for the currently selected resource in its dropdown picker, so I made a fix for that and re-pushed. The fix is also present on the 4.0 version of this PR.

The problem also persists with the new version.
Maybe it is somehow related to a regular, non-Mono build of Godot? Unfortunately, I'm failing to get the Mono version compiling, so I can't test it right now.

@mphe Alrighty, take 2, and this time I seem to have reproduced the issue. Not sure why I wasn't triggering the errors before, but now they are happening. It's strange though. If you create a built-in, inline instance in the Inspector it works fine. And if you create a standalone custom resource in the Inspector and save it, it's fine. Now, clicking the Load button at the top of the Inspector for a custom resource works fine (that's what I tested earlier), but clicking the Load option in the dropdown beside a specific resource does not work, nor does dragging and dropping a resource.
These errors are likely because of the migration to the EditorResourcePicker which had other associated bugs I had to fix and this is just more of the same. I will look into it and get a fix uploaded ASAP.

@mphe There were indeed some errors related to the EditorResourcePicker that have now been fixed, but there were also two other errors in very different places for which there are now meaningful changes/fixes.

The EditorData class has a cached lookup table of path-to-name mappings for script classes so that, while you are in the Editor, the engine can look up the name of a global script class based on its filepath. Part of the problem is that this map is only updated accurately if the typename also has a corresponding icon. It would be easy enough to correct it so that it works for other global scripts too, however, this is information that could also be useful to runtime instances. For example, any apps/games/editors/modding tools that iterate over resources, get the scripts of said resources, and check if the script is a global script class with a typename. Thus, I have moved the reverse lookup map to the ScriptServer and replaced all references to the EditorData one (of which there were only a few) for the ScriptServer one.
Because all core ResourceFormatLoaders evaluate every type, even types that are global script classes, I made a slight change to the ResourceLoader so that only ResourceFormatLoaders WITH a script evaluate the loaded resource type as given. Otherwise, it converts the stated type to the base C++ type and passes it along. This way, loaders defined in the C++ code only have to worry about C++ resource types and it is only the user-defined loaders that need to account for scripted types.

I will make an update to the 4.0 version with similar changes soon.

@willnationsdev Great! Everything seems to work again. Thanks for fixing!

but clicking the Load option in the dropdown beside a specific resource does not work, nor does dragging and dropping a resource.

That's what I meant. Didn't think of the other load button.

Is below code possible with this PR ?
class Order extends Resource:
	func _init():
		pass

class Wait extends Order:
	export(float) var seconds

class Wander extends Order:
	export(float) var seconds = 0.0

class Towards extends Order:
	export(NodePath) var object

class Interact extends Order:
	export(Resource) var behavior

export(Array, Order) var orders

Just showing classes which are inherited from Order class.

@willnationsdev Back in February I brought up that defaults were not being respected in the inspector and you said "seems to be a problem in 3.2 in general, not introduced by your changes". I just discovered that this seems to have indeed fixed upstream and that bug is no longer present.
When I set a default in code, the default is present as expected in the inspector!
export(Whatever) var whatever = preload("res://path/to/whatever.tres")

FYI, am trying to rebase this, but I found that, for some reason, the Make Unique operation is not working successfully (don't think it ever was(?)), so now I'm trying to fix that before I push an update.

Any idea when this will be merged? This is a life changer!

Any idea when this will be merged? This is a life changer!

The 3.x version of this pull request will be merged after 3.4 is released to avoid introducing regressions in 3.4.

If you get a spare moment, could you test if this branch also fixes https://github.com/godotengine/godot/issues/52729 ?  I'm guessing it's a separate issue, but just in case.

@cgbeutler Sure, I'll give it a shot when I have free time. Kinda busy, but I'll try to find some time.

@willnationsdev it's looking like an unrelated issue. Ignore my previous request.

3.4 has been released now, it seems like all that's missing before this PR can be merged is fixing the doctool checks and maybe a rebase

What's stopping this from getting merged?

What's stopping this from getting merged?

Lack of reviews. None of the maintainers for the area have been able to test and code review it yet.

Additionally, @willnationsdev needs to implement my changes but I guess he couldn't yet find the time.

I think it's totally needed for 3.5. It's been ages since first pull request with fixes for this issue, and at least 3 such requests was archived because authors tired of updating it for months. If this will happen here too, it will be most regrettable. And this fix really needed.

It's been ages since first pull request with fixes for this issue, and at least 3 such requests was archived because authors tired of updating it for months.

It's the nature of doing something in the core systems, especially something big. This is a huge PR with a lot of consequences beyond just the ability to pick a custom resource in the inspector. It needs thorough review from the core team, and the core team has been pretty busy these last couple of years.

@pycbouh sorry for mentioning you directly, but you recently commented and maybe have a quick comment on this:
In #48201 @willnationsdev and I are talking about me adopting the two PRs. I was think about getting one of the two ready and merged in a new PR and based on the comments there (back-)port to the other version. Would that make sense to you?

@dploeger Feel free to open a new PR based on Will's work. You could also ask him for a write permission on his fork and that way you could push directly to the same branch on his repo and update the PR this way. If not, opening a new one is fine, we'll close one as superseded.
You can keep Will's credit by using the co-authored-by tag in the commit messages.

@pycbouh I'll do that, thanks. What about my idea to only target one version first and after that's accepted port to the other one?

If you make it for master first, that's a preferred way to contribute any time.

Gotcha.

@dploeger Rebased this branch too, if it proves to be relevant at all.

This PR will be stalled until #48201 is merged, so I don't need to do things twice.

With 3.5 being almost out of the door, I hope the maintainers can comment if there is a chance for this to make it into a 3.x branch.
It would be such a welcome addition in order to round off a stable Godot version while 4.0 is being ironed out - even for existing projects, I'd pick it up immediately since without it I tend to have to write editor extensions just to be able to easily handle custom resources.
Thanks for the hard work.

I'm still waiting for a review on the 4.0.0 PR. I will see, whether it makes sense to backport it then.

That is my most wanted feature for 3.5.
I think Resources are way under considered and underrated, and this would bring custom resources to a new level.
It will definitely boost the usage and creation of custom resources.
I'll make a video explaining it and why it's so powerful as soon as it's available.
Will it be ready for 3.5 ?
Can we do something to speedup the integration and make it available faster ?

Will it be ready for 3.5 ?

see the comment right above yours.
3.5 seems kind of unlikely at this rate, but there's still a chance I guess.

Can we do something to speedup the integration and make it available faster ?

https://github.com/godotengine/godot/pull/48201 this needs to get finalized and merged first.

Are there any updates on whether this will start to be backported again now that most of the 4.0 version is merged?

Hey all, if anyone's interested I have this PR rebased and cherry-picked onto my personal Godot 3.x branch, and I'm automatically building my custom Godot version using Github Actions here: https://github.com/Atlinx/Godot3CustomBuilds. Feel free to download a build if you'd like.

Hey all, if anyone's interested I have this PR rebased and cherry-picked onto my personal Godot 3.x branch, and I'm automatically building my custom Godot version using Github Actions here: Atlinx/Godot3CustomBuilds. These builds have both GDScript and C# support -- feel free to download a build if you'd like.

Linking to it directly for people who would want to backport it to their own forks: https://github.com/Atlinx/godot/commit/02d1f70ee5ec1b7bb08bc6dddac7fe39b4727e1e

At the current stage, I don't expect this PR to be merged in 3.x, as it's a very significant rework that had a lot of implications in the master branch for 4.0, went through multiple iterations and several months of time and many bugfix PRs. We don't have the resources to do the same effort on the 3.x branch.
So I'll close this not to give false expectations on the likelihood of this feature being included in 3.6 or later. Anyone who absolutely needs the feature can try the rebased version linked by Atlinx in their custom builds.


Implementation of SRL in Javascript

I'm currently building something quite complex with Regular Exp in JS and would love to have this implemented in JS.

👍

In the future, I'll be moving this project to SimpleRegex/PHP, to have a better overview of everything to come.
If someone wants to contribute an official JS version of this, please let me know. I'll then create a repository at SimpleRegex/JavaScript and give you permission on it.

@KarimGeiger I'd love to help contribute to the JS one. :)

@chriscorwin awesome! I've created the project and gave you permissions. If you need any further help, let me know.

@KarimGeiger Can I request that the project names not be simply "PHP" and "JavaScript", but rather something like "simple-regex-php" and "simple-regex-javascript"  ?
When I go to clone it, the idea of cloning into a directory named "JavaScript" hits a funny-sounding bell for me.

Yeah, that's a valid point... I guess it makes sense to call it SRL-[Language]

@KarimGeiger @chriscorwin I would also love to contribute in SRL Javascript.

@KarimGeiger @chriscorwin I wanna know how is about the progress of SRL Javascript, and if I can help to make contribute.

@jsenjoy looks like there isn't any progress made yet :( I've added you to the team and gave you access to the repository. Feel free to join us. I'd love to see a JS implementation for https://simple-regex.com as well. Would make it easier than to always call an API.
If you'd like me to add you to the Slack team as well, just leave me a note.

@KarimGeiger Yeah, plz add me to the Slack.

The JavaScript repository now contains a working port of SRL. Great work @jsenjoy!


feat(terminal): add support for redirects and allow specific commands

⚠ Blocked by updating WebContainer, we also need to update the docs ⚠
This PR adds support for setting allowRedirects and allowCommands when defining the terminal panels. You can define the options for all terminals in the lesson, or separately per terminal.
I also removed defining the panels with a tuple where the second argument is a config object, because the syntax was bad... This means that you can't do this anymore
terminal:
  - [terminal, { name: 'Terminal Name' }]

But instead do
terminal:
  - type: terminal
    name: 'Terminal Name'

or providing the name as the second argument is still supported.
terminal:
  - [terminal, 'Terminal Name']

With the new options, it looks like this
terminal:
  panels:
    - terminal
    - output
  allowRedirects: true
  allowCommands:
    - ls
    - echo

terminal:
  panels:
    - type: terminal
      allowRedirects: true
      allowCommands:
        - ls
    - type: terminal
      allowRedirects: false
      allowCommands:
        - echo
    - output

Oh could we keep the [terminal, 'Terminal Name'] shorthand for only specifying the type and the title? I quite like that syntax and it's also supported by the previews field and it's nice to support it for both.
Note that this syntax is optional, we don't force anyone to use it but for those that enjoys it (like me) it's pretty cool to support it.

I'm not a big fan of having 100 different syntaxes for the same thing. But I can add it back if we want to keep it though. Wasn't aware it was supported by the previews as well, because in that case it kinda makes sense I guess.

Added it back, but removed the [type, object] syntax though. So the second argument can only be the name, not the config object anymore.


fix(secret): metadata not fully passed to super in child classes (#1665)

Backport
This will backport the following commits from k8s-25/main to k8s-23/main:

fix(secret): metadata not fully passed to super in child classes (#1665)

Questions ?
Please refer to the Backport tool documentation

⚪ Backport skipped
The pull request was not backported as there were no branches to backport to. If this is a mistake, please apply the desired version labels or run the backport tool manually.
Manual backport
To create the backport manually run:
backport --pr 1666

Questions ?
Please refer to the Backport tool documentation


Use O_EXCL to avoid interference?

See http://www.ibiblio.org/pub/Linux/docs/HOWTO/other-formats/html_single/SCSI-Generic-HOWTO.html and specifically 7.1 open().
Reading that, we might also be fine with opening with RDONLY
No idea how this works with nvme though.

What is the status of this? This relevant? Can you give me a little more information what might be the benefit of this?

The benefit/problem statement is to handle multiple concurrent runs of the library/tools. I'd imagine a way to test this is to run hundreds of instances of the tools against one drive and ensure that no protocol errors happen. Either by synchronizing around something like O_EXCL or something else


Auto-exit brightness set mode after x seconds.

When I'm in the brightness adjustment mode, if I don't manually exit out, it will stay in that mode indefinitely, and not update the numbers (I think). I adjusted the brightness earlier today, and started noticing after ~2 hours that wow, the AQI has been super steady all day... and then noticed the brightness bar, turned it off, and the numbers updated.  :-)
Suggest adding a timer that auto-exits the brightness set mode after some number of seconds (10, 30, 60?) without any any button presses.

Would it exit saving the current value? I'm thinking that's the best. This can also happen during setting the correction, but I guess it is blinking at you in that case.

 And 2 minutes seems like an eternity! But after 2 minutes, we are pretty sure somebody forgot.

That all sounds good... saving current value, and 2min. Thanks for the quick fix!


PDF Preview options breaking preview in outlook

Microsoft PowerToys version
0.63
Installation method
PowerToys auto-update
Running as admin

[X] Yes

Area(s) with issue?
PDF Preview, PDF Thumbnail
Steps to reproduce
Installed Outlook "Microsoft 365 apps for enterprise" latest version
When PDF preview/thumbnail preview are enabled in power toys settings, the PDF preview in outlook does not work.
It reads "this file cannot be previewed because there is no previewer installed for it."
Upon checking attachment preview settings in outlook, I confirm attachment previews are on, and for some strange reason there are 2 PDF preview handlers on the list of handlers.
I also confirm adobe acrobat reader is installed on the machine, and is working properly in outlook if I turn off the power toys PDF preview options.
✔ Expected Behavior
I want to be able to have PDF previews in explorer as well as in outlook
❌ Actual Behavior
I have to choose one of the above options
Other Software
Windows 11 Enterprise, 64 bit
version 22H2
build 22623.741

I have found the same behaviour in Outlook 2016. After disabling PowerToys PDF Preview, voilà the Adobe Reader previews kicked in and I can see PDF previews in Outlook again. I even have a 32bit Outlook and 64 bit Adobe Reader, and yet they work together without any trouble (I believe the x64 Adobe Reader has both 32 bit and 64 bit dlls?)

Is this still an issue?
Similar issues: #16875 #17221 #15851 #17018 #17020 #18451 #16993
/needsinfo


Automate releases

It shouldn't be too hard to automate builds using the new CI setup. This has already been set up in https://github.com/spacemeshos/gpu-post/.
needs #2205

I think builds should be triggered when we create a tag (not on every merge to develop).
A build should push an image to Dockerhub (we should separate this part from the tag-and-build command in the makefile).
It should also build binaries for all platforms and attach them as artifacts to a draft release (that we can then manually release by just clicking a button).
We can use this tool for the job, or at least steal some of their source code... I like their automatic release notes--at least as a basis for manual editing.

Yeah, there are many tools that auto generate changelogs, doesn't matter to me which one we use.

Done in #2223

Done in #2223


Add flow_framework namespace specs

Description
Part of Adding missing API specs #168
Issues Resolved
#508
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@dblock Hi, I have addressed all the comments and made the necessary changes. Regarding the suggestion about:
file: 'namespaces/flow_framework.yaml',
location: 'Operation: DELETE /_plugins/_flow_framework/workflow/{workflow_id}',
message: "The 404 response must be a reference object to<EMAIL_ADDRESS>
I believe that using the error name directly like this:
          oneOf:
            - $ref: '../schemas/flow_framework._errors.yaml#/components/schemas/BadRequestError'
            - $ref: '../schemas/flow_framework._errors.yaml#/components/schemas/ConflictError'
            - $ref: '../schemas/flow_framework._errors.yaml#/components/schemas/MissingParameterError'
            - $ref: '../schemas/flow_framework._errors.yaml#/components/schemas/ParameterConflictError'
            - $ref: '../schemas/flow_framework._errors.yaml#/components/schemas/MaxWorkflowsLimitError'
            - $ref: '../schemas/flow_framework._errors.yaml#/components/schemas/WorkflowSaveError'

makes the it clearer in this particular case. Please let me know your thoughts on this approach.

Please let me know your thoughts on this approach.

Makes sense to me. I am still figuring the best way to do these things too! @nhtruong might have other ideas.

Please let me know your thoughts on this approach.

Makes sense to me. I am still figuring the best way to do these things too! @nhtruong might have other ideas.
Iterate until https://github.com/opensearch-project/opensearch-api-specification/actions/runs/10713232032/job/29722019367?pr=553 is green either way?

sure, will do

@dblock Hi, I just fix the lint spec errors, hope everything goes green this time !

@dblock Hi, got all lint spec passed locally, hope everything goes well this time!

👏


getnotes

Many MIDI files will encode note off as note on with velocity zero. I therefore changed the getnotes code in order to adapt to this.

Good find, we definitely need this.


Misleading context availability on 'on.workflow_call.inputs.<inputs_id>.default'

Code of Conduct

[X] I have read and agree to the GitHub Docs project's Code of Conduct

What article on docs.github.com is affected?
https://docs.github.com/en/actions/learn-github-actions/contexts#context-availability
What part(s) of the article would you like to see updated?
On https://docs.github.com/en/actions/learn-github-actions/contexts#context-availability , is stated that we can use the inputs context when evaluating default values of another input.

Workflow key
Context
Special functions

on.workflow_call.inputs.<inputs_id>.default
github, inputs, vars
None

We tried to define it here https://github.com/n8n-io/n8n/blob/master/.github/workflows/e2e-reusable.yml#L28
      cache-key:
        description: 'Cache key for modules and build artifacts.'
        required: false
        default: ${{ github.sha }}-${{ inputs.run-env }}-e2e-modules
        type: string

But it ignores the ${{ inputs.run-env }} part, as you can see here: https://github.com/n8n-io/n8n/actions/runs/6765793705/job/18385947360#step:1:40
    run-env: browsers:node18.12.0-chrome107
    cache-key: b11c4de0392289ff1ac4c4293ff92ba3bdeecbc0--e2e-modules

Just need to understand if it should work or not.
Additional information

Are you able to reliably reproduce the problem? How often does it occur? How many users are affected?

Yes, I was able to reproduce it on each and every run of my jobs. All users in Github Actions.

How do I claim what's mind I don't how to use this site help me out

@cmwilson21 this is still a very valid issue, can you please reopen and get someone to reply?

@TWiStErRob 👋 Thanks for the ping here! I'll get one of our Actions SMEs to review this issue

Here is a permalink to the relevant version of the file https://github.com/n8n-io/n8n/blob/721a36637c42a396149156ab651333ba4d2f16b8/.github/workflows/e2e-reusable.yml#L28
I can confirm it's not intended to work the way you are describing. When evaluating workflow_call.inputs.*.default, the inputs context would contain workflow dispatch inputs, not inputs from the workflow being called.

@ericsciple Thank you for reviewing this! ✨
I'll go ahead and close out this issue 💛


Error: undefined constant StopWords

Thank you for your great work!
I've tried to use your library and got an error:
In lib/cadmium_tfidf/src/cadmium/tfidf/tfidf.cr:5:13

 5 | include StopWords
             ^
Error: undefined constant StopWords

Oops. Should be an easy fix, but for now you should be able to fix it by requiring cadmium_util before you require cadmium_tfidf. Be sure to add cadmiumcr/cadmium_util as a dependency in your shard.yml first.
https://github.com/cadmiumcr/utilities

thanks mate!


Improve performance

Reference tests:

Case 1: Desktop browser on a fast connection on the US east coast
Case 2: Mobile device on a 3G connection in South Africa

From those, the main problems are:

Size of the JS: even on a fast connection it’s ~3.6 seconds before anything is displayed (https://www.webpagetest.org/video/compare.php?tests=210721_AiDcJC_ed69d396a91d993156e59cd530aeba32-r%3A1-c%3A0-s%3A1%2C210721_AiDcJC_ed69d396a91d993156e59cd530aeba32-r%3A2-c%3A0-s%3A1%2C210721_AiDcJC_ed69d396a91d993156e59cd530aeba32-r%3A3-c%3A0-s%3A1&thumbSize=200&ival=100&end=visual)

Size of the images: once the JS has downloaded and executed, it’s then time to download 3.7MB worth of images (https://www.webpagetest.org/result/210721_AiDcP3_dd9fffea321e334c1d8b949fa048c98b/1/performance_optimization/#image_compression_step1). They’re not optimised, nor served at an appropriate size (the mobile device does not need a 2644x1388 image).

No browser caching. The 2nd and 3rd WPT runs should be faster as it shouldn’t need to re-download everything, but they’re not. (This makes the cache busting filenames where are in place (e.g. the random characters in https://prereview.org/sc-morning-post.f7e3a52a.jpg) pointless.)

Re-ran case 1 after enabling HTTP caching. 'Cache static content' drops from an A to an F, and 'Effective use of CDN' from a pass to fail. This is expected as the current TTL is very short, but it also shows that WebPageTest doesn't pick up when caching should be enabled and isn't (it definitely should not have been an A initially.)

Re-ran case 1 after improving the images. 'Compress Images' goes from a C to an A, and the page weight has dropped from around 5 MB to 1.9 MB.

I ran case 1 before and after the code splitting change. I can see a definite improvement in load time and data transfer.


Mocking Isar during unit test throws 'Future<List>' is not a subtype of type 'Query'

I want to test some logic in my data layer. To achieve that I tried to mock isar for some parts of testing. But when I tried to mock isar.collection.where().findAll(). I ran into this error.
type 'Future<List<Checkout>>' is not a subtype of type 'Query<Checkout>'
MockQueryBuilder.buildInternal
QueryExecute.build
QueryExecute.findAll
LocalCheckout.getAllCheckout
main.<fn>.<fn>
main.<fn>.<fn>

this is my test file. I am using mocktail to mock isar.
class MockIsarCollection<T> extends Mock implements IsarCollection<T> {}

class MockQueryBuilder<R, S> extends Mock
    implements QueryBuilder<R, S, QWhere> {}

void main(){
   late Isar isar;
   late LocalCheckout localCheckout;
   late IsarCollection<Checkout> checkout;
   late QueryBuilder<Checkout, Checkout, QWhere> checkoutBuilder;
   const Checkout tCheckout = Checkout(
    totalPrice: 100,
    quantity: 2,
    variantId: 2,
    variantName: 'variant',
    gst: 0,
    receiptId: 'receipt',
    cartId: 2,
    hawkerId: 2,
    hawkerName: 'hawker singh',
    offerId: 2,
    offerName: 'offerName',
    offerPercent: 0,
    paymentMode: 'cash',
    sellingPrice: 100,
    finalSellingPrice: 100,
  );

  setUp(() {
    isar = MockIsar();
    checkout = MockIsarCollection<Checkout>();
    checkoutBuilder = MockQueryBuilder<Checkout, Checkout>();
    localCheckout = LocalCheckout(isar: isar);
  });

   test('should get all the entries from checkout collection', () async {
      final list = <Checkout>[tCheckout, tCheckout.copyWith(brandId: 99)];
      when(() => isar.checkouts).thenAnswer((_) => checkout);
      when(() => checkout.where()).thenReturn(checkoutBuilder);
      when(() => checkoutBuilder.findAll()).thenAnswer((_) async => list);

      final result = await localCheckout.getAllCheckout();

      expect(result, list);
    });
}

LocalCheckout.dart
/// class to handle local checkout data
/// performs CRUD for checkout data
class LocalCheckout {
  /// default constructor
  LocalCheckout({required Isar isar}) : _isar = isar;

  final Isar _isar;

  /// get sales from local storage
  Future<List<Checkout>> getAllCheckout() async {
    try {
      final checkouts = await _isar.checkouts.where().findAll();
      return checkouts;
    } catch (e) {
      rethrow;
    }
  }
}

I also tried to mock when(() => checkouts.where().findAll()).thenAnswer((_) async => list);. But it still gives error.
Version

Platform: iOS, Android, Mac, Windows, Linux, Web
Flutter version: 3.0.1
Isar version: 2.5.0

I found the solution. Thanks for the new update https://github.com/isar/isar#unit-tests

@dewbambs would it be possible for you to share how you managed to have unit-tests working?
I seem to be unable to make my tests work with await Isar.initializeIsarCore(download: true) .

In newer version testing is not working. I have raised an issue here and if you want to maintain testable code then try using isar version 3.0.0-dev.0.


Reviews are not dismissed after rebasing

Hi, this issue can be considered as a re-opening of #95.
After conducting further tests using action from your PR branch #116, we've identified that no-owner action still takes priority in some use cases:
Examples:

UC: Approved reviews are dismissed when changes don’t have a code owner and no-owner-action setting is set to dismiss-none, changes are force-pushed. Sample execution can be found here
Changes are force-pushed(cause: resetting branch to previous commit), but log message mentions about no-owner-action.

Current Log message

Expected Log message
Commit '{commit}' doesn't exist in the history. It may be because it was overwritten by force push or because it's outside of checkout depth.
"force-push-action" is set to "dismiss-all", so the following reviews are dismissed...

UC: Approved reviews are NOT dismissed when changeset without a code owner are force-pushed, both no-owner-action and force-push-action set to dismiss-none. Sample execution can be found here

Current Log message

Expected Log message
Commit '{commit}' doesn't exist in the history. It may be because it was overwritten by force push or because it's outside of checkout depth.
"force-push-action" is set to "dismiss-all", so the following reviews are dismissed...

Hi, I checked the examples and it seems fine. The force-push didn't affect (alter) the commit that is "approved" (each approval is tied to a specific commit sha). So the action was able to get the changed files without a problem, and it was resolved based on no-owner-action config.
Does it make sense to you to have it like this?

Hi @Balvajs, yes it make sense. I will close the issue now.


pandora box surfacing

Proposed changes
@StWiemann found out that upgrade from 0.8.2 generate some issues. for some reason when inserting new data in a old table, instanceId column is not added, generating issues. (this could be actually due to some change in cratedb, we haven't handled properly)
this pr surface the issue in bc testing (at least it should). moreover, this pr, by fixing a configuration issue in the backward test script, also evidenced an issue with a PR just merged (#564):

while the pr forces new data passed with not service path to get assigned / as service path, old data with None service path are not updated. thus queries to check old data with the new logic, of course fail. (this the reason why some lines are now commented - otherwise the test would bomb before detecting the issue exposed above.

The pr is not yet a fix. It's just to reproduced in a controlled environment the issue.
Types of changes
What types of changes does your code introduce to the project?
Put an x in the boxes that apply

[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Maintenance (update of libraries or other dependences)

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[x] I have read the CONTRIBUTING doc
[x] I have signed the CLA
[x] I have added tests that prove my fix is effective or that my feature works
[ ] I have run all the existing tests locally (not just those related to my feature) and there are no errors
[ ] After the last push to the PR branch, I have run the lint script locally and there are no changes to the code base
[ ] I have updated the RELEASE NOTES
[ ] I have added necessary documentation (if appropriate)
[ ] Any dependent changes have been merged and published in downstream modules

Further comments
N/A

actually the change from dynamic to strict was introduced my me here:
https://github.com/orchestracities/ngsi-timeseries-api/commit/57f721f74610028bd658eb7b2f45bfbe6d415957
the reason why this is now an issue, is that ìnstanceId` it's not tracked in metadata, thus, we don't know if it is missing or not, when adding new data.

@chicco785 cool, thanks for this! We should keep a lid on the box :-)
Let's merge this PR when we've got a fix for #638. Here's what I have in mind

Fix #638. This should include tests to check instanceid is handled correctly (metadata, etc.) but it can leave out backward compat tests.
Rebase this PR on the fix. Tweak backward compat tests if needed. At this point the currently failing tests should pass.

Thoughts?

sure, this pr should be use as a base for the fix, since it reproduces the issue.


when choosing current image and other neighbor image to train dff, why set the probability of choosing current frame and other neighbor frame the same?

when choosing current image and other neighbor image to train dff, why set the probability of choosing current frame and other neighbor frame the same? As written in image.py(224)

He sets the test method to test the mIOU with different time intervals. But when training the network, it needs to learn many time intervals' impact to the nwtwork to improve the network's generalization ability. I hope this can help you.

I follow the training strategy of the original paper, @littletomatodonkey has given a good explanation.

Thanks. I got it.


Suggestion: Rename / Extend Condition Trait Methods.

Current State
At the moment, the Condition trait provides the following methods:

and_then
or_else

These methods are effectively analogs to the boolean and and or operators. For example, the Update scheduled systems zoom_camera and translate_camera in the following code snippet will only run if in GameState::OpponentTurn OR GameState::PlayerTurn:
        // Add Update scheduled systems to the main application.
        app.add_systems(
            Update,
            (zoom_camera, translate_camera)
                .run_if(in_state(AppState::InGame))
                .run_if(in_state(AssetsState::Loaded))
                .run_if(in_state(BootState::NotInBoot))
                .run_if(in_state(GameState::OpponentTurn).or_else(in_state(GameState::PlayerTurn))),
        );

The readability can sometimes be weird (in my opinion) with this naming convention:
app.add_systems(
    // `my_system` will only get run if the resource `A` exists & resource 'B' exists.
    my_system.run_if(resource_exists::<A>.and_then(resource_exists::<B>)),
);

Proposed Change
Definitely not a necessary change, but I personally would be much happier renaming and_then and or_else to:

and_if
or_if

Previous examples with updated naming convention:
        // Add Update scheduled systems to the main application.
        app.add_systems(
            Update,
            (zoom_camera, translate_camera)
                .run_if(in_state(AppState::InGame))
                .run_if(in_state(AssetsState::Loaded))
                .run_if(in_state(BootState::NotInBoot))
                .run_if(in_state(GameState::OpponentTurn).or_if(in_state(GameState::PlayerTurn))),
        );

app.add_systems(
    // `my_system` will only get run if the resource `A` exists & resource 'B' exists.
    my_system.run_if(resource_exists::<A>.and_if(resource_exists::<B>)),
);

Additional Discussion
As I'm writing this, I think it's also worth noting that Bevy's not() function that operates on conditions is also not included as a method for the Condition trait (It exists, but not as a member of the trait). For consistency and completeness, would it make sense to bundle all boolean operator analogs under this trait?

and_if
nand_if
or_if
nor_if
xor_if
xnor_if
not

Is there a specific reason you opted for the _if suffix? Why not does rename the methods to and and or?
Furthermore, I'm not sure I agree with not being added as a method. Usually I prefer to ready not before the function, same as using ! on a boolean. For example, which one reads nicer?
// Current system
my_system.run_if(not(resource_exists::<A>))

// Proposed system
my_system.run_if(resource_exists::<A>.not())

I prefer the first because I won't accidentally miss not, which may not be the case with the second.
Overall, though, I like the idea behind these suggestions. Condition could definitely use a bit more love. :)

No good reason for the suffix. I guess my brain thought that if the suffixes _then and _else were being dropped another suffix ought to take their place, but I actually prefer your suggestion of just dropping them completely if that's possible / reasonable.
I also agree with your point that the current implementation of not is the most readable, and honestly I would prefer using it the way it is too. My reasoning for suggesting otherwise was that if the full suite of logical operators were to be added for conditions, they ought to be under a single common interface (under the Condition trait).
But now that I think about it, the fact that not is the only unary operator might be justification enough to treat him as separate and leave as is.
Thanks for your input!

Awesome! We can make sure to add a link to not from Condition so others can find it.

Awesome! We can make sure to add a link to not from Condition so others can find it.

Fantastic - may I be assigned to this? I would love to take a crack at it sometime this week.

We don't really do official assignments, but feel free to open a pull request and we'll happily review it!

Closing as implemented in #13784


Terraform 0.14 upgrade

what

Upgrade to support Terraform 0.14 and bring up to current Cloud Posse standard

why

Support Terraform 0.14

/test all

@osterman @Nuru looks like this PR needs codeowners TLC


remove "preview" since this is GA now

the method being referred to here is available in the latest official SDK, so I am removing the "preview" from this topic

@wickste : Thanks for your contribution! The author, @nberdy, has been notified to review your proposed change.

Thanks @wickste ! #sign-off


Suggestions for more engine backend

Hello @yishn
First off, you're really awesome for building this - thanks a ton man :+1:  :smile: !! Your code style is also a delight to read :)
Btw,  I wanted to point out that there are two more open source backends that you can easily integrate within the package as a whole for a great experience.
https://github.com/ujh/iomrascalai
https://github.com/brilee/MuGo
These two do provide GTP and are fairly cross platform with liberal licenses.

Thank you for your praise! I'd like to keep Sabaki as minimal as possible, but you're free to add instructions to the documentation as to how to integrate the engines you mentioned.


Index isomorphism

Not sure if this has been suggested before or even if it is already implemented (neither as far as I can tell), but I think a useful function could take an array as a parameter, and return an isomorphism that goes between a value and its index in that array.
Obviously there would need to be some logic dictating what to do if the value isn't in the array or if the index is invalid.
If this is a good idea I would be happy to have a go at creating a PR for it
Thanks

Hmm... Do you mean something like this?

Yep, pretty much exactly that, that's almost identical to what I've been using, but I figured it seemed useful enough that it could be included

Thanks for the detailed response, I hadn't thought of composing find and is to produce the same result as the contains lens, because it can be so easily produced from existing lenses, it's probably not worth adding it as its own lens.
My use case for the index isomorphism was to show a list of items, and have elsewhere a form that involves selecting using the index of the list item.  It doesn't really require the use of lenses but I thought it would be more fun this way.  I expect that most of the time, it is better to store the index as state instead of the item itself as indexOf might be quite slow for large arrays, which I think would limit its usefulness.


[SDP-1015]: Add SMS preview & editing before sending a new disbursement

Adds the ability to trigger disbursements for multiple organizations. We want to have a custom SMS per organization.

As a user, I want to see (preview) the invitation SMS before sending a disbursement to ensure the content is correct.
As a user, I want the ability to edit the invitation SMS before sending a disbursement to customize the message depending on the campaign or organization.

Something went wrong with PR preview build please check

stellar-disbursement-platform-frontend-preview is available here:http://sdp-frontend-pr66.previews.kube001.services.stellar-ops.com/

stellar-disbursement-platform-frontend-preview is available here:http://sdp-frontend-pr66.previews.kube001.services.stellar-ops.com/

Something went wrong with PR preview build please check

Something went wrong with PR preview build please check

stellar-disbursement-platform-frontend-preview is available here:http://sdp-frontend-pr66.previews.kube001.services.stellar-ops.com/

can you add PR description as well.

stellar-disbursement-platform-frontend-preview is available here:http://sdp-frontend-pr66.previews.kube001.services.stellar-ops.com/

stellar-disbursement-platform-frontend-preview is available here:http://sdp-frontend-pr66.previews.kube001.services.stellar-ops.com/


Add Traffic Ops 2.0 service to get ATS configs

This service acts as a 'microservice' and serves ATS config files, by querying the Traffic Ops API.
This currently only creates storage.config, as a proof-of-concept. But it has the framework to easily add the rest of the configs. If we decided not to do the microservice thing, the config file logic easily be adapted to a library within a larger service. In fact, it would probably be good to make it a package anyway.

To clarify, this pulls from the existing Traffic Ops 1.2 API. That's why it's still using the mojo cookie, etc. Once the 2.0 API is stable enough and has all the features this needs, I'll change this to use it. It should be pretty simple to adapt.


Contributor dataset: DopaminergicNeuronDifferentiation

Dataset/group this task is for:
Oliver Stegle's group
DopaminergicNeuronDifferentiation
Latest sheet: https://docs.google.com/spreadsheets/d/1Eblt1hHBAwk84BFiuTzbANGcJUYdtj5k/edit#gid=56358489
Folder: https://drive.google.com/drive/folders/1kRNGIsBsIviLHEPv1ipkLG0lrM8TLLbL
https://contribute.data.humancellatlas.org/submissions/detail?id=60a253c5901b6d17e5f3a4f0
Wrangler responsible for this dataset/lab:

[x] Primary: Ami

[x] Secondary: Ray

Since this is a contributor dataset, there might be some contributor and/or publication metadata missing, I am yet to send it back to them to check it all looks ok to them. Want to get secondary review first.
Description of the task:

[x] curate metadata
[x] get ontologies
[x] graph validator - returns orphan node error message for organoid protocols - this is expected. No organoid data, need to ask about this.
[x] validate in ingest staging
[x] secondary review
[ ] Check with Anna - is it as expected, missing anything? - Anna did not reply to multiple emails.

@Wkt8 @ESapenaVentura can either of you take on this for secondary review?

Stalled. Waiting for Anna/Jeongbin to respond. I emailed them to remind them today.

Thanks @rays22 , I have made the changes.
The organoid data protocols were orphaned because I had asked the authors whether they should be included (I couldn't see any organoid samples in the info they sent me). They have been terrible at getting back so I have removed those protocols.

I have asked Tony about the gdpr question

Tony says it is fine to upload the raw data. So will start on that.

the data is transferring a to hca-util area

I am having problems with the data download from ENA. It looks to be an issue on the ENA server side. I have message Eugene to ask about this.

Uploading the fastq to an hca-util upload area from local folder on EC2.

Alegria is downloading the fastq from ENA (about 800-900 files) and then uploading to an hca-util upload area. The project submission is here: https://contribute.data.humancellatlas.org/submissions/detail?uuid=15142b86-b5b3-49cb-bad0-cb3eb8ba0a79&project=72c636f3-d51f-4e5d-9cf8-9b91427a9e0c the metadata is valid except that the fastq files need to be uploaded to ingest.

Alegria is still downloading the fastqs from ENA, but I also noticed some discrepancies with the accession numbers and the fastq files.
Discrepancy 1:
Cell Suspension ID: SAME6833352 with BioSamples ID: SAME6833352
In the spreadsheet, this entity is linked to:
ERR4699951_1.fastq.gz
ERR4699952_1.fastq.gz
ERR4699953_1.fastq.gz
ERR4699954_1.fastq.gz
However, on the ENA Browser (https://www.ebi.ac.uk/ena/browser/view/PRJEB38269)
Those four fastq files should be linked to: SAMEA6833353 instead.
Discrepancy 2
Additionally, the project on the ENA Browser contains 532 sample accessions, for a total of 1064 fastq files, but there are only 968 metadata sequence file entities in the spreadsheet.
As such, we weren't sure if we should:

add the additional metadata file entities (to bring to a total of 1064 fastq files) or
if there was a specific reason those specific files were left out (e.g. the aforementioned GDPR)

Due to this and the time constraints in downloading the files we have decided to wait for release 8 for export.

Files are now in this hca-util upload area: s3://hca-util-upload-area/1268551e-f2d0-43eb-9511-968e46901e72/ as mentioned in https://app.zenhub.com/workspaces/operations-5fa2d8f2df78bb000f7fb2b5/issues/ebi-ait/hca-ebi-dev-team/432

@amnonkhen unfortunately I need to review this dataset and potentially make edits before it gets exported, so can we please move the milestone from July to August as I won't be back in time to make the updates?

Working on getting the missing fastq files. Removing the July milestone as it won't be done by then, changing it to the August milestone.

Emailed Oli about the missing samples on 02/08/2021. Some samples in the ENA study are missing from the dataset files they provided us with and some samples in the files are missing from ENA.

Decided to go ahead and upload this dataset to the September release milestone using the fastqs available in ENA given no response from the authors. Currently re-uploading ~80 fastq files which were displayed as invalid in ingest prod.

@ami-day I've removed the extra files for this submission: https://contribute.data.humancellatlas.org/submissions/detail?uuid=d3dc95e5-7154-4d2a-b684-81a852cfb9d9

Thanks @aaclan-ebi , it's now exporting :)

Successfully exported and just needs the import form which ami is completing now.

Curating to SCEA format. Assigning it E-HCAD-42.

Moving this to stalled as I'm not sure if this experimental design is suitable for scea. Have messaged them and waiting for a reply.

Pre-converted the files and uploaded them here: https://drive.google.com/drive/folders/14m-j87nBFQUi3yCrIDhTrQ54KHoCLCe2

E-HCAD-58
Generated MAGET-TAB files
Currently validating

In review by SCEA team


Add a script for testing push notifications

Between pem and houston, we should be able to write a script (or lane for fastlane) to send test notifications on the sandbox APNS queue.
I'm picturing this as a starting point that can be adjusted for each project to match the expected payload from real push notifications. Since we're working our way toward a shared fastfile (see #9), it probably makes sense to do this in one or more scripts, rather than having project-specific lanes.

Closing this as we use other 3rd party tools for testing push.


[TIR] Allow string/buffer arguments to Schedule cache_read/write

Previously, the argument needed to be an integer specifying the index into the read/write regions of a block.  Now, the argument can be a string specifying the name of the buffer, or the Buffer object itself. This is a follow-up from https://github.com/apache/tvm/pull/11624.
cc @Hzfengsy @junrushao1994

Thanks for the UX improvement


Allow multiple views to be debounced via @OnClick

The code below handles the debouncing of multiple view but each view is given a separate OnClickListener to handle the debounce. This solves of the problem of clicking the same view multiple times but does not solve of the problem of debouncing multiple clicks of multiple views.
@OnClick({ R.id.door1, R.id.door2, R.id.door3 })
public void pickDoor(DoorView door) {
  if (door.hasPrizeBehind()) {
    Toast.makeText(this, "You win!", LENGTH_SHORT).show();
  } else {
    Toast.makeText(this, "Try again", LENGTH_SHORT).show();
  }
}

This may exist already (though I didn't see it in the documentation), but it would be nice to have the functionality to debounce clicks to multiple views by handling all click events in a single DebouncingOnClickListener. Thoughts?

The DebouncingClickListener uses a static for debouncing clicks which means that it should already be doing this.

Per its Javadoc:

A click on one button disables all buttons for that frame.

Thank you for the clarification. I misunderstood how the DebouncingClickListener operates.


Thunk - negative testing

What about adding API for negative testing for verifying no dispatches have been made.
For example - this use case: make sure that in case the reddit-server returns an error, action POSTS_UPDATED hasn't been dispatched.

What about expect(dispatches.length).toBe(0) ?


macos vm are using cocoapods version 1.15.0 but this version is buggy and got fixed in 1.15.2

Description
Hi Team,
Using this cocoapods 1.15.0 version, customers are not able to install the pods!
Is it possible can we get the cocopods version upgrade to the latest version 1.15.2 ?
https://github.com/CocoaPods/CocoaPods/issues/12226
Platforms affected

[ ] Azure DevOps
[ ] GitHub Actions - Standard Runners
[ ] GitHub Actions - Larger Runners

Runner images affected

[ ] Ubuntu 20.04
[ ] Ubuntu 22.04
[ ] macOS 11
[X] macOS 12
[X] macOS 13
[ ] macOS 13 Arm64
[ ] macOS 14
[ ] macOS 14 Arm64
[ ] Windows Server 2019
[ ] Windows Server 2022

Image version and build link
https://github.com/actions/runner-images/blob/macOS-12/20240202.1/images/macos/macos-12-Readme.md
Also, seems macos 13 also having the same:
https://github.com/actions/runner-images/releases/tag/macos-13%2F20240204.1
Is it regression?
macos 12 Previous version (20240127.1)
Expected behavior
should able to run cocoapods install
Actual behavior
cocoapods install failed with error: File exists @ syserr_fail2_in
Repro steps
User cocopods with xcode build

Hello @ankasani!
Thank you for reporting this!
The new cocoapods version will be available in the next images rollout. For now please install it in runtime:
sudo gem install cocoapods -v 1.15.2

@MaksimZhukov , Thanks!

@MaksimZhukov , Just want to know, when will be the next image rollout will happen ?

@MaksimZhukov , When can we expect the fix for this?

+1 for an upgrade to 1.15.2. Our react native project no longer builds on this agent


unknown option "--dport"

Hi !
I'm getting this error when running portforward.sh:
# docker exec -it pia /portforward.sh
Forwarded port for IP <IP_ADDRESS> is: 32233
Detecting target VPN interface...tun0
Accepting input traffic through tun0 to port 32233...iptables v1.8.3 (legacy): unknown option "--dport"
Try `iptables -h' or 'iptables --help' for more information.
[ERROR] Undefined error, with status 2)

You should use the -p option in your iptables command, and probably run it twice (one for TCP and one for UDP).

Actually, the iptables command calls the $PORT variable (https://github.com/qdm12/private-internet-access-docker/blob/master/portforward.sh#L32), but the port is defined in $port (https://github.com/qdm12/private-internet-access-docker/blob/master/portforward.sh#L22)...
Did this script ever worked ?

Hi, it worked for me a few weeks ago as I recall, but I'll look into it today. Thanks for your help!

Just an idea, you could add portforwarding as an env option. The resulted port returned by PIA can be printed to a file. That port number can be used to update in Deluge with a simple command, and Transmission too. Maybe Qbittorrent as well.


Hardware transcoding not working

This is a new device Intel i7 gen 11 Tiger Lake. I run Proxmox and Handbrake is installed in Docker in LXC Ubuntu server 21.04 container. Intel graphics drivers are installed and are working - Jellyfin in a Docker on same LXC container works with HW transcoding.  I have Handbrake installed and working on another with Pentium Gold CPU and QSV works.
Nothing I can do gets QSV working on new device and I've tried the suggestions here: https://github.com/jlesage/docker-handbrake.
Any help appreciated.
This is the Portainer stack:
version: '3'
services:
handbrake:
image: jlesage/handbrake
ports:
- "5800:5800"
volumes:
- "/home/john/appdata/handbrake/config:/config:rw"
- "/home/john/appdata/handbrake/storage:/storage:ro"
- "/home/john/appdata/handbrake/watch:/watch:rw"
- "/home/john/appdata/handbrake/output:/output:rw"
environment:
- PUID=1000
- PGID=1000
- TZ=Pacific/Auckland
- AUTOMATED_CONVERSION_FORMAT=MKV
- AUTOMATED_CONVERSION_PRESET=Hardware/H.265 QSV 1080p
- DISPLAY_WIDTH=1920
- DISPLAY_HEIGHT=1080
- HANDBRAKE_DEBUG=1
devices:
- /dev/dri/renderD128:/dev/dri/renderD128
- /dev/dri/card0:/dev/dri/card0
restart: unless-stopped
This is what I think is the most relevant part of the log:
[12:29:25] sync: expecting 133452 video frames
[12:29:25] hb_display_init: using VA driver 'iHD'
libva info: VA-API version 1.12.0
libva info: User environment variable requested driver 'iHD'
libva info: Trying to open /opt/intel/mediasdk/lib64/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_12
libva info: va_openDriver() returns 0
[12:29:25] encqsvInit: MFXVideoENCODE_Init failed (-15)
[12:29:25] encqsvInit: using encode-only (LowPower) path
[12:29:25] encqsvInit: H.265/HEVC (null) profile @ level (null)
[12:29:25] encqsvInit: TargetUsage 7 AsyncDepth 6
[12:29:25] encqsvInit: GopRefDist 3 GopPicSize 25 NumRefFrame 1
[12:29:25] encqsvInit: BFramesMax 2 BRefType off
[12:29:25] encqsvInit: RateControlMethod ICQ ICQQuality 22
[12:29:25] Failure to initialise thread 'Quick Sync Video encoder (Intel Media SDK)'
[12:29:25] aac-decoder done: 0 frames, 0 decoder errors
[12:29:25] h264_qsv-decoder done: 0 frames, 0 decoder errors
[12:29:25] sync: got 0 frames, 133452 expected
[12:29:25] Finished work at: Sun Mar  6 12:29:25 2022

Kia Ora John
I think your issue, and your earlier issue (both of which I also have) are linked, and are covered here:
https://github.com/jlesage/docker-handbrake/issues/204
I know that Jocelyn is working on a fix for the next release, but as you will see he's hit some difficulties.
Ngā mihi
Pete

From Handbrake (https://github.com/HandBrake/HandBrake/issues/4201), a fairly simple workaround from @heldiho53.
Adding lowpower=0 to VideoOptionExtra in the presets.json file makes the gui work. Just create a custom profile and add that option to the preset file. A good "so far" solution until something is implemented.
I can confirm that it works, and I now have Intel QSV working perfectly.  I did have to restart the container.
@jlesage , I implemented this on your testbuild.  I'll cross-post to the other issue.

@johnvick, can you verify with the latest Docker image and possibly with the workaround if it's working for you ?

Yes this is working with custom profile QSV 265 and lowpower=0 addition on the Tiger Lake device at 150 fps - starting manually. I'll test the watch folder later and report back. Thanks for sorting this.

Thanks for the update!
Wait for the next image before testing the watch folder.  It is currently broken :(.  A new image is currently being built and should be available in few hours.


Shard download space fix

High Level Overview of Change
With this change, the DatabaseShardImp will confirm that sufficient space is available to store all the shards queued for download, instead of confirming that each shard will fit without considering the additional space required by the others.

Context of Change
This PR is stacked on two unmerged PRs. The relevant code starts with commit 6e1b74e.
Type of Change

[ X] Bug fix (non-breaking change which fixes an issue)

Hi @undertome, it would be useful for me to have a little more background on this bug fix.  It would be handy to know:

The user observed nature of the bug before the fix.
The nature of the improved user experience after the bug fix.

Also it looks like the fix is associated with being low on resources.  Does the user have any recourse once they discover that they are too low on resources to handle the shards?
Thanks.

Hi @undertome, it would be useful for me to have a little more background on this bug fix. It would be handy to know:
* The user observed nature of the bug before the fix.

* The nature of the improved user experience after the bug fix.

Also it looks like the fix is associated with being low on resources. Does the user have any recourse once they discover that they are too low on resources to handle the shards?
Thanks.

Hi Scott,

Before the fix, the user would be able to initiate a batch of shard downloads when there was insufficient space to store all the shards, but sufficient space to store each shard in isolation. The user would see the first download complete, but subsequent downloads would fail.

With the fix, a request to download a batch of shards will fail immediately if there is insufficient space to store all the shards and a descriptive log message will be created.

If the user is low on resources, the DatabaseShardImp will log a warning, discontinue acquiring shards, and reject requests to download shards. At this point, it would be the user's responsibility to provide more storage to accommodate additional shards.
Hope that helps.

A couple of your commit messages don't follow these guidelines: https://chris.beams.io/posts/git-commit/  May I suggest rewordings?  Consider:
Delay SSLHTTPDownloader dtor until current session finishes

and
Download queued shards only if there is space for them all

Or perhaps you can come up with something better.

FWIW, I was running the unit tests for code coverage on this branch using macOS and I got this error message during the test run:
ERR:ShardArchiveHandler async_connect
ERR:ShardArchiveHandler Downloading shard id 10 from URL <IP_ADDRESS>/10.tar.lz4
rippled(2843,0x13c67c5c0) malloc: *** error for object 0x13eed4000: pointer being freed was not allocated
rippled(2843,0x13c67c5c0) malloc: *** set a breakpoint in malloc_error_break to debug
make[3]: *** [CMakeFiles/coverage_report] Abort trap: 6
make[2]: *** [CMakeFiles/coverage_report.dir/all] Error 2
make[1]: *** [CMakeFiles/coverage_report.dir/rule] Error 2
make: *** [coverage_report] Error 2

It did not leave a stack trace behind.  I'll see if I can reproduce the problem.

Not surprisingly the bug is not easy to reproduce.

Running the unit tests alone (once) did not recreate it.
Running code coverage on just ShardArchiveHandler did not reproduce it.
Running all of the unit tests under code coverage again did not reproduce it.

I'm going to start a loop running the ShardArchiveHandler test to see if I can reproduce the bug with enough repetitions.

I continued running a test over the weekend to try and get more information about the crash observed on Friday.  I ran just the ShardArchiveHandler unit test in a loop.  After 5928 iterations I got a different crash from the originally reported crash.  Here's link to the stack trace of this most recent crash in case it's useful: https://gist.github.com/scottschurr/cec16dc035d9ec83fe61e6411cee0ddf

Success!  By running just the ShardArchiveHandler under code coverage in a loop I was able to reproduce the original symptom quickly.  And this time I got a stack trace.  On the console I see:
ERR:ShardArchiveHandler Downloading shard id 9 from URL <IP_ADDRESS>/9.tar.lz4
ERR:ShardArchiveHandler async_connect
ERR:ShardArchiveHandler Downloading shard id 10 from URL <IP_ADDRESS>/10.tar.lz4
rippled(2685,0x148d915c0) malloc: *** error for object 0x13f548000: pointer being freed was not allocated
rippled(2685,0x148d915c0) malloc: *** set a breakpoint in malloc_error_break to debug
make[3]: *** [CMakeFiles/coverage_report] Abort trap: 6
make[2]: *** [CMakeFiles/coverage_report.dir/all] Error 2
make[1]: *** [CMakeFiles/coverage_report.dir/rule] Error 2
make: *** [coverage_report] Error 2

You can find the stack trace here: https://gist.github.com/scottschurr/f43be7a61290562738728bb4b668a6e8

I continued running a test over the weekend to try and get more information about the crash observed on Friday. I ran just the ShardArchiveHandler unit test in a loop. After 5928 iterations I got a different crash from the originally reported crash. Here's link to the stack trace of this most recent crash in case it's useful: https://gist.github.com/scottschurr/cec16dc035d9ec83fe61e6411cee0ddf

Are you sure you're running code from this branch? That stack trace references SSLHTTPDownloader which no longer exists.

Are you sure you're running code from this branch? That stack trace references SSLHTTPDownloader which no longer exists.

That's a fair question.  Here's the git log of the branch I've been testing:
commit fb1ca671319fcfdbeb3943f5d1fc554c3d7d230c (HEAD -> devon-shard-download-space-fix, devon/shard-download-space-fix)
Author: Devon White<EMAIL_ADDRESS>Date:   Thu Sep 17 17:26:02 2020 -0400

    [FOLD] Clang format patch

commit 09827143f62683907745a23842f6876d3612ade4
Author: Devon White<EMAIL_ADDRESS>Date:   Thu Sep 17 17:21:28 2020 -0400

    [FOLD] Add unit tests

commit f9b9647a5bd6b47133f2e58e1eb6585944c2d0fa
Author: Devon White<EMAIL_ADDRESS>Date:   Wed Sep 9 17:15:13 2020 -0400

    Download queued shards only if there is space for them all

commit e31c02ba89b6b7727eb2a8246d3d4ad4f227b104
Author: Devon White<EMAIL_ADDRESS>Date:   Fri Aug 14 15:45:56 2020 -0400

    Delay HTTPDownloader dtor until current session
    finishes

commit 649d4a741e112777575e9f9d10100c6588037d08
Author: Devon White<EMAIL_ADDRESS>Date:   Tue Aug 11 12:10:10 2020 -0400

    Allow shard downloads via HTTP or HTTPS

commit a8d481c2a54267d80bd539003f56481424b9ae63 (upstream/develop, origin/develop, develop)
Author: Nik Bougalis<EMAIL_ADDRESS>Date:   Tue Sep 1 10:43:26 2020 -0700

    Set version to 1.7.0-b1

That said, even though that's the source code currently in place, it's possible (but seems unlikely) that I did not do a proper build before running the ShardArchiveHandler unit test in a loop over the weekend.  If that's the case please accept my apologies.
For the malloc failure under Codecov, however, I'm pretty confident that I have both the correct sources and that those are the source files I'm running.  The rebuild is part of how Codecov is invoked.

That said, even though that's the source code currently in place, it's possible (but seems unlikely) that I did not do a proper build before running the ShardArchiveHandler unit test in a loop over the weekend. If that's the case please accept my apologies.
For the malloc failure under Codecov, however, I'm pretty confident that I have both the correct sources and that those are the source files I'm running. The rebuild is part of how Codecov is invoked.

The malloc failure seems current.

I took a quick look at the stack @scottschurr produced and observed the following. The crash comes from within a journal. From ShardArchiveHandler_test::testRedundantShardFailure, on line 612, the test has only created Env called close() which calls RPCCall::fromNetwork. A possible culprit I see is that inside fromNetwork a local journal j is declared and passed into the callbacks and the HTTPClient object. This might be a problem as HTTPClient::request is non blocking and j's sink object may have gone out of scope when the callbacks or invokeComplete are executed. Its possible the sink may be still be valid at this point as it might be kept alive by someone but I am not sure.

I took a quick look at the stack @scottschurr produced and observed the following. The crash comes from within a journal. From ShardArchiveHandler_test::testRedundantShardFailure, on line 612, the test has only created Env called close() which calls RPCCall::fromNetwork. A possible culprit I see is that inside fromNetwork a local journal j is declared and passed into the callbacks and the HTTPClient object. This might be a problem as HTTPClient::request is non blocking and j's sink object may have gone out of scope when the callbacks or invokeComplete are executed. Its possible the sink may be still be valid at this point as it might be kept alive by someone but I am not sure.

Thanks for looking into it, I will see if this checks out.

Codecov Report

:exclamation: No coverage uploaded for pull request base (develop@a8d481c). Click here to learn what that means.
The diff coverage is n/a.

New commit fixes a Travis error caused by an "unused" (it's used in an assert statement) variable. There was also a bug I discovered that was caused by the Checkpointer outliving a DatabaseCon.


fix(deps): update launch-editor fix path contains special characters

Description
https://github.com/yyx990803/launch-editor/pull/78

It has been included in #18050.


ServerConfigSelector, provider and filter

Broken from #27388
This PR is slightly awkward in that it adds a library which is not being used. The user (xDS) and tests (already available in #27388) would be available in upcoming PRs.

This change is

Issues: #21858, #27603

Thanks for reviewing!


[Empire in Ruins] Duel wielding added 10 advantage

I was duelwielding an attack and on the secund attack with the char Heinrich and the value went negative and gave med 1+ advantage.From: Loofinator#4285foundry: 11.315wfrp4e: 7.2.4wfrp4e-core: 5.0.7wfrp4e-dotr: 5.0.5wfrp4e-eis: 5.0.3wfrp4e-empire-ruins: 5.0.4wfrp4e-horned-rat: 5.0.4wfrp4e-middenheim: 5.0.2Active Modules: dice-calculator, dice-so-nice, dungeon-draw, levels, lib-wrapper, pin-cushion, pointer, socketlib, tagger, theripper-premium-hub, token-action-hud, wall-height, wfrp4e-pbth-maps, forge-vtt

Why is this classified under Empire in Ruins?
What does "value went negative" mean? What value?
What does "gave med 1+ advantage" mean?


vSphere Agent

this agent is in developing

Running OCD test at https://ocd.appscale.com:8080/job/Daily Build/251 to check for regression. Thanks for the submission! Let us know when it's ready to review. Perhaps you can also create a wiki page documenting how to use and test this agent?

404 error

We haven't managed to get our internal builds available outside the intranet ... We use it as an internal reference when reviewing the code. Sorry about the confusion.

The builds fails to pass the unit tests. You can try them typing 'rake' in the appscale directory.

ok i try it

you have install pysphere?

You can say all the commands useful to test my code in appscale ?

I correct syntax errors, now with the package installed pypshpere "pip install -U pysphere", exceeds 19 test successfully. I can do other tests.

Great. If you need to install a new dependency, you can look into debian/appscale_install_functions.sh how to do it: we usually create a function and we have a wrapper for pip. Then add your function to appscale_install.sh.

Ok.

i write the pip wrapper for pysphere

Closing this PR as inactive. @marticus88 if you're considering to keep working on this and maybe do a demo, you'd have to merge with current master first.


Moves nepali language translations from np directory to ne

This PR moves the existing nepali translations from np directory to ne directory
In Laravel community packages, the Nepali language locale is recognized as ne. This change aligns Filament translations with the standard directory used by other packages and translation managers, ensuring compatibility and consistency.

This is a breaking change for anyone currently using this language code.

This has been targeted for v4 and will be a step in the upgrade guide.

I would rather not have duplicate files.


Batch training doubts

❓ Questions & Help

I am trying to train a network with GCNConv layers. My batch_size is 4, the graph has 293 nodes(1 feature each) and the same edge_index. I used the following to initialize the network and in the fwd methods:-
self.conv1 = GCNConv(293*batch_size, 256, node_dim=1)    self.conv2 = GCNConv(256, 1)
def forward(self, data):         x, edge_index, batch = data.x, data.edge_index, data.batch         x = x.view(batch_size,293, -1)         print(x.shape, edges_index.shape) #[4, 293, 1]         x = self.conv1(x, edge_index)
I get a size mismatch error:-
size mismatch, m1: [1172 x 1], m2: [1172 x 256]
Unsure where I am going wrong. I read the docs and referred to a related issue as well, but couldn't solve my issue.

Is this resolved? You need to define GCNConv as GCNConv(1, 256).

I solved the (1,256) part.
So if I have a batch_size of 4, do I just have to pass
GCNConv(1, 256, node_dim=1)
and resize in the forward function?
Also, do I need node_dim=1 in the second layer as well?

Yes, resize to format [batch_size, num_nodes, num_features] and you are good to go. All convs that operate on those batches should have node_dim=1 set.

So the resize needs to happen before feeding to each GCNConv?

I went ahead with resizing x everytime before a conv layer but I get a cuda memory access error in scatter.py
`
Traceback of TorchScript (most recent call last): File
"/home/ph1391/.conda/envs/simclr/lib/python3.7/site-packages/torch_scatter/scatter.py", line 23, in scatter_sum
        size[dim] = int(index.max()) + 1
    out = torch.zeros(size, dtype=src.dtype, device=src.device)
    return out.scatter_add_(dim, index, src)
           ~~~~~~~~~~~~~~~~ <--- HERE

else:
    return out.scatter_add_(dim, index, src)

 RuntimeError: cuda runtime error (700) : an illegal memory access was encountered at
 /pytorch/aten/src/THC/generic/THCTensorScatterGather.cu:296`

My inputs are correctly shaped as well. I would appreciate any help!


Translated "Responding to Events" page

Hindi translation for Responding to Events is done, opening PR for review.

Done