Implement percentage-closer soft shadows (PCSS).

Percentage-closer soft shadows are a technique from 2004 that allow shadows to become blurrier farther from the objects that cast them. It works by introducing a blocker search step that runs before the normal shadow map sampling. The blocker search step detects the difference between the depth of the fragment being rasterized and the depth of the nearby samples in the depth buffer. Larger depth differences result in a larger penumbra and therefore a blurrier shadow.
To enable PCSS, fill in the soft_shadow_size value in DirectionalLight, PointLight, or SpotLight, as appropriate. This shadow size value represents the size of the light and should be tuned as appropriate for your scene.  Higher values result in a wider penumbra (i.e. blurrier shadows).
When using PCSS, temporal shadow maps (ShadowFilteringMethod::Temporal) are recommended. If you don't use ShadowFilteringMethod::Temporal and instead use ShadowFilteringMethod::Gaussian, Bevy will use the same technique as Temporal, but the result won't vary over time. This produces a rather noisy result. Doing better would likely require downsampling the shadow map, which would be complex and slower (and would require PR #13003 to land first).
In addition to PCSS, this commit makes the near Z plane for the shadow map configurable on a per-light basis. Previously, it had been hardcoded to 0.1 meters. This change was necessary to make the point light shadow map in the example look reasonable, as otherwise the shadows appeared far too aliased.
A new example, pcss, has been added. It demonstrates the percentage-closer soft shadow technique with directional lights, point lights, spot lights, non-temporal operation, and temporal operation. The assets are my original work.
Both temporal and non-temporal shadows are rather noisy in the example, and, as mentioned before, this is unavoidable without downsampling the depth buffer, which we can't do yet. Note also that the shadows don't look particularly great for point lights; the example simply isn't an ideal scene for them. Nevertheless, I felt that the benefits of the ability to do a side-by-side comparison of directional and point lights outweighed the unsightliness of the point light shadows in that example, so I kept the point light feature in.
Fixes #3631.
Changelog
Added

Percentage-closer soft shadows (PCSS) are now supported, allowing shadows to become blurrier as they stretch away from objects. To use them, set the soft_shadow_size field in DirectionalLight, PointLight, or SpotLight, as applicable.

The near Z value for shadow maps is now customizable via the shadow_map_near_z field in DirectionalLight, PointLight, and SpotLight.

Screenshots
PCSS off:

PCSS on:

I don't know the domain well- why do the shadows here:

Have curve-like imperfections?
I noticed the article you linked has this image which has a different type of noise:

Is it related to the geometry which casts the shadow or something else?
Also I was wondering how the shadows perform when the light source is moving.
I changed the example a bit to have the light undulate.
pcss-example-undulate.patch.txt
Feel free to git apply pcss-example-undulate.patch.txt if you want to try the same or add it to the PR.
https://github.com/bevyengine/bevy/assets/52322338/2a6530dd-c0f8-4fde-b6c7-fa901716b128
I noticed an effect that spot lights have this effect where it looks like the shadow density changes a lot, perhaps related to shadow map resolution?

The patterns are because of the interleaved gradient noise. This is generally considered more desirable than the white noise that it looks like your other example has.
There isn't a whole lot we can do about the noise in this patch without generating mip chains for the shadow map, which would require https://github.com/bevyengine/bevy/pull/13003 (and even if that were in I wouldn't want to do it in the first patch because this patch is too big as it is). When I asked in Discord I was told that PCSS in games tends to be noisy; that's just the way it is.

I considered allowing the user to move the light but I realized that this will cause problems with the Z near value, which is carefully tuned so that the palm tree makes maximum use of the shadow map. So I'm less inclined to make the light movable, though I could be convinced otherwise.

I suspect what's going wrong with spot lights is that the blocker search starts missing blockers. I'm not sure if this is something that can reasonably be fixed; it's always possible that the blocker search can miss things. The solution is for artists to tweak the soft_shadow_size manually.

Yeah. Unity doesn't implement PCSS at all and in this PR I pretty much learned why. PCSS is just really hard to tune.

@pcwalton Unity does, in fact, implement PCSS as part of their High Definition Render Pipeline package. See the "Shadows" section on this   changelog page, for example. If you ever want to know what Unity does and doesn't support then that repository is a fantastic place to search.

Please don't merge this yet as spot shadows are broken somehow.

This is ready for review again. I fixed the regression. I also moved the radio button code into a reusable helper module. That way, I don't have to keep recreating it for every new example.

@pcwalton I'm inclined to merge this basically as is. When you have a moment, could you clean up merge conflicts? I don't feel confident doing so myself here.
@superdump's comments are solid as well, but I don't feel that they're ultimately blocking (feel free to correct me).

@Olle-Lukowski It does work with point lights. This just isn't the prettiest scene for them.

Rebased and comments addressed. I think this is OK to go unless it needs a rereview.

DirectX12 failures: https://github.com/bevyengine/bevy/actions/runs/10909293058/job/30277186297

@alice-i-cecile I think I fixed DX12.

This failure is because directional_shadow_textures can be sampled with both linear and PCF with PCSS and WebGL 2 doesn't support that. Grr. Trying to see if there's a way I can just ifdef off all of PCSS in WebGL 2.

Should be fixed again.

I love WebGL2 🥲 Thanks for fixing this up <3

Thank you to everyone involved with the authoring or reviewing of this PR! This work is relatively important and needs release notes! Head over to https://github.com/bevyengine/bevy-website/issues/1688 if you'd like to help out.


[Feature Requests] Publish pre-built docker containers to a public repo

As a user I would like to not have to work through the steps to build a docker container for use. One should be readily available. It would also alleviate build issues based on machine as see in PR #8.
Pushing to the public ECR repo seems like a logic choice since this is an AWS project but pushing to DockerHub could work as well. Would effectively make this comment and running in Github less headache if every end user of LZA didn't have to build the same image. https://github.com/aws-samples/lza-validator/issues/7#issuecomment-2167207127
But also makes implementing #7 request easier as well since github actions can just be a layer on top of that docker container. https://docs.github.com/en/actions/creating-actions/creating-a-docker-container-action

@jc1518 - does this belong to you? https://hub.docker.com/r/jc1518/lza-validator


Using sprite as normalmap change sprite import options

Godot version:
3.2 beta 1
OS/device including version:
Ubuntu 19.10
Issue description:
It is really strange, that using sprite as normalmap in spatialmaterial, change import settings in icon.png.import from
[remap]

importer="texture"
type="StreamTexture"
path="res://.import/icon.png-487276ed1e3a0c39cad0279d744ee560.stex"
metadata={
"vram_texture": false
}

[deps]

source_file="res://icon.png"
dest_files=[ "res://.import/icon.png-487276ed1e3a0c39cad0279d744ee560.stex" ]

to:
[remap]

importer="texture"
type="StreamTexture"
path.s3tc="res://.import/icon.png-487276ed1e3a0c39cad0279d744ee560.s3tc.stex"
path.etc2="res://.import/icon.png-487276ed1e3a0c39cad0279d744ee560.etc2.stex"
metadata={
"imported_formats": [ "s3tc", "etc2" ],
"vram_texture": true
}

[deps]

source_file="res://icon.png"
dest_files=[ "res://.import/icon.png-487276ed1e3a0c39cad0279d744ee560.s3tc.stex", "res://.import/icon.png-487276ed1e3a0c39cad0279d744ee560.etc2.stex" ]

Steps to reproduce:

Minimal reproduction project:
Tanks.zip

Normal maps make use of RGTC compression to save on VRAM (and improve quality), so Godot is likely turning on VRAM compression automatically if you set a texture to be used as normal map.

This is the same issue as the very first bug report I opened for Godot: #15920

This does not happen if you have "Detect 3D" import setting disabled. So it's likely intended.

Is there possibly a way to detect that the texture is being used for albedo, and un-set the normal mapping compression? I found some docs here that suggest albedo textures use SRGB, perhaps SRGB being enabled should disable RGTC.
I tried adding singleton->make_flags[path] &= ~MAKE_NORMAL_FLAG; after line 48 of resource_importer_texture.cpp and it doesn't seem to make a difference.

So it's likely intended.
What exactly is the intention here? See my issue.

Here is another simple replication of this issue:
I had some other TileMap issues but was having trouble with changing import settings.
I found this post searching and solved my problem by changing sprites import settings from auto-detect to 2d?
https://github.com/rakkarage/NewTileMapTest

Open project works fine.
Change setting of doodad.png back to default auto-detect and reimport.
Open banner1.tres (shader material that uses that png in a shader) get errors and it changes import settings.
Now with the changed settings looks weird in TileSet editor.

If you change import settings back to 2d and do this all over it works fine?
If you remove the hint_albedo from the shader and do this all over it works fine?
I am not even sure what hint_albedo is for, it seemed to not work in 3 without it? Now it does not work with it? It is not needed? Maybe causing this problem?
But this does not seem intended? Auto detect should auto detect?
I should not have to change all my sprites in all my projects to 2D to ward against changes?
Or maybe I should remove all my hint_albedo?
Thanks.
Before:
[remap]

importer="texture"
type="StreamTexture2D"
uid="uid://gwkob421rs7s"
path="res://.godot/imported/Doodad.png-c1a5ceea497e429d040ae291cdf297c5.stex"
metadata={
"vram_texture": false
}

[deps]

source_file="res://Doodad.png"
dest_files=["res://.godot/imported/Doodad.png-c1a5ceea497e429d040ae291cdf297c5.stex"]

[params]

compress/mode=0
compress/lossy_quality=0.7
compress/hdr_compression=1
compress/bptc_ldr=0
compress/normal_map=0
compress/channel_pack=0
compress/streamed=false
mipmaps/generate=false
mipmaps/limit=-1
roughness/mode=0
roughness/src_normal=""
process/fix_alpha_border=true
process/premult_alpha=false
process/normal_map_invert_y=false
process/HDR_as_SRGB=false
process/size_limit=0
detect_3d/compress_to=0

After:
[remap]

importer="texture"
type="StreamTexture2D"
uid="uid://gwkob421rs7s"
path.s3tc="res://.godot/imported/Doodad.png-c1a5ceea497e429d040ae291cdf297c5.s3tc.stex"
path.etc2="res://.godot/imported/Doodad.png-c1a5ceea497e429d040ae291cdf297c5.etc2.stex"
metadata={
"imported_formats": ["s3tc", "etc2"],
"vram_texture": true
}

[deps]

source_file="res://Doodad.png"
dest_files=["res://.godot/imported/Doodad.png-c1a5ceea497e429d040ae291cdf297c5.s3tc.stex", "res://.godot/imported/Doodad.png-c1a5ceea497e429d040ae291cdf297c5.etc2.stex"]

[params]

compress/mode=2
compress/lossy_quality=0.7
compress/hdr_compression=1
compress/bptc_ldr=0
compress/normal_map=0
compress/channel_pack=0
compress/streamed=false
mipmaps/generate=true
mipmaps/limit=-1
roughness/mode=0
roughness/src_normal=""
process/fix_alpha_border=true
process/premult_alpha=false
process/normal_map_invert_y=false
process/HDR_as_SRGB=false
process/size_limit=0
detect_3d/compress_to=0

As I said above, the fact that import options are changed for normal maps is expected. What shouldn't happen is VRAM compression being forced for small textures (<= 256×256 by default), but this is something that needs to be implemented separately. I attempted this a while ago, but it was rejected as it needs to be done in a cleaner, user-adjustable manner.

my code does not use Normal Maps... just a sprite and a shader yet the import options are changed

I am not even sure what hint_albedo is for, it seemed to not work in 3 without it? Now it does not work with it? It is not needed? Maybe causing this problem?

hint_albedo should generally only be used in 3D shaders. It hints that sRGB -> linear conversion should be performed by the shader. This is not needed in 2D as all rendering occurs in sRGB there.

Using filter_linear or filter_nearest (maybe anything) (sampler2D being passed into shader with a hint) also seem to cause this unwanted and unneeded change of import settings causing the TileMap and TileSet to display wrong in this example.

It does seem seem to work fine with no hints but need filter_nearest to fix this other problem...
Which is originally from here:
https://github.com/godotengine/godot/issues/57679
Don't mean to bud in here but seems like same problem and just trying to help by providing more info.
Thanks.

This is now made clear by https://github.com/godotengine/godot/pull/61497 (https://github.com/godotengine/godot/pull/62004 in 3.x), closing.
https://github.com/godotengine/godot/pull/62023 also disables VRAM compression for small textures automatically, so icon.png will no longer use VRAM compression when used in 3D.


fixproject should be done inside subquery

Proposed changes
Issue Number: close #xxx
WITH t0 AS(
SELECT report.date1 AS date2 FROM(
SELECT DATE_FORMAT(date, '%Y%m%d') AS date1 FROM cir_1756_t1
) report GROUP BY report.date1
),
t3 AS(
SELECT date_format(date, '%Y%m%d') AS date3
FROM cir_1756_t2
)
SELECT row_number() OVER(ORDER BY date2)
FROM(
SELECT t0.date2 FROM t0 LEFT JOIN t3 ON t0.date2 = t3.date3
) tx;
The DATE_FORMAT(date, '%Y%m%d') was calculated in GROUP BY node, which is wrong. This expr should be calculated inside the subquery.
Checklist(Required)

[ ] Does it affect the original behavior
[ ] Has unit tests been added
[ ] Has document been added or modified
[ ] Does it need to update dependencies
[ ] Is this PR support rollback (If NO, please explain WHY)

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


Use $state location replace API

otherwise angular will always history.replaceState(); after sign in

Changes Unknown when pulling 504dc0bcb4ec885083288ca23f4b03c20994e5e5 on graingert:patch-1 into ** on Narzerus:master**.

Yes if I get the time. Either way the options: location: replace is the way
you're supposed to do this with ui-router.
On 19 Apr 2016 7:18 pm, "Błażej Krysiak"<EMAIL_ADDRESS>wrote:

@graingert https://github.com/graingert Could you please provide
example on jsfiddle to verify the case?
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/Narzerus/angular-permission/pull/219#issuecomment-212052704

Coverage remained the same at 100.0% when pulling d4bcab6d81ab97674f0b7dc6fef88ae21d911515 on graingert:patch-1 into d402951de54ecb00f41c6d8a0f24bd6efea3cb3c on Narzerus:master.


Loading icon not animating

In the popup/modal, the loading icon (leftmost icon) is not animating. This is a problem because it just looks like an asterisk otherwise. To the ordinary users who this tool is intended to reach, since they are not graphic designers working on a net neutrality campaign, they have never seen this icon "not animating", which means it's broken as far as communicating the necessary message. Sorry if this crucial requirement doesn't make for "tidy" graphic design, but it's important to stress the current failure of the modal, which is the first thing anyone sees, to explain "the web without Net Neutrality" (what is that exactly? A web full of cryptic icons that only have obvious ominous significance to people who are already net neutrality activists?) To be effective, the modal must have some kind of immediately, instantaneous meaning, and for many users the only instant pathway to mapping this political issue onto their existing brain circuitry is with the "loading animation" - this was surely a part of the success of the iconic previous campaign.

You can you the theme: 'slow' variation where the icon spins if you want.

I'd like to leave this open if we have time. People can still change.


Array in MongoDB in JDL file.

I have a JDL file  listing all my entities and one of them is something like
entity Section {
sectionName String,
code String,
strength Integer,
courseId String,// Course,
locationId String,//  Room,
mainTeacherId String,//Teacher,
eventsId String //Event
}
Here I want eventsId to be list of Strings...Is there any way that I can achieve this?
Versions

JHipster
JHipster UML
Editor & version

JHipster Version
jhipster-uml -v

@mayurbais There's no way to do it in the JDL, and you're in the wrong project! Go there the next time please!


Alternative syntax for unions requires Python 3.10 or newer

Envirement data

pylance version: 2020.10.2
vscode version: 1.50.1
python version: 3.8.5
Djagno version: 3.0.3
OS platform: MacOs 10.15.7

My django program also alert this and I don't know why, I want to know why and how to ignore this warning?

Experiencing the same issue, pylance version: 2020.10.2 (pyright d6db7c90), Python 3.7, Django 2.2
Same as the op, we are using Django Rest Framework.

The permission classes which contain the | between them inherit from this class
@six.add_metaclass(BasePermissionMetaclass)
class BasePermission(object):
    """
    A base class from which all permission classes should inherit.
    """

    def has_permission(self, request, view):
        """
        Return `True` if permission is granted, `False` otherwise.
        """
        return True

    def has_object_permission(self, request, view, obj):
        """
        Return `True` if permission is granted, `False` otherwise.
        """
        return True

Further related code can be found at https://github.com/encode/django-rest-framework/blob/master/rest_framework/permissions.py#L97 and near that.
Even manually adding a metaclass with defined __or__ property to the Permission classes does not remove the pylance error. Please let me know if further information is required.

Thanks for the bug report. I've updated the logic so it detects the case where a class has a custom metaclass that supports the __or__ or __ror__ method and use that for the | operator rather than assuming a misuse of of PEP 614 prior to Python 3.10. This fix will be in the next version of pylance.

@erictraut The | operator for type annotations won't be exclusive to 3.10, since postponed evaluation of annotations is enough to support it, as long as the type checker understands it. It can even be used in type comments with Python 2. https://github.com/python/typing/issues/429#issuecomment-690594334 and https://github.com/python/mypy/pull/9647
It might be an idea to limit the warning to isinstance and issubclass calls, or check if from __future__ import annotations is used with type annotations.

@cdce8p, that's a good point. I've changed the code to suppress that error in any situation where the evaluation of the type annotation is postponed (within quotes and with from __future__ import annotations enabled). The diagnostic will still be reported in cases where you're using <3.10 and the evaluation is not postponed (e.g. in a type alias declaration or an isinstance call).

This issue has been fixed in version 2020.10.3, which we've just released. You can find the changelog here: https://github.com/microsoft/pylance-release/blob/master/CHANGELOG.md#2020103-28-october-2020

@jakebailey The problem is not fixed in 2020.10.3
[Info  - 9:19:46 AM] Pylance language server 2020.10.3 (pyright b881f28d) starting

The specific class is from DRF ( details in my comment above ).
Should I create a new issue, or can we reopen this?
Thank you.

Same with type annotations. It seems mostly fixed, however I'm encountering a wired edge case with . notation.
E.g. time | None works, whereas dt.time | None dosen't.

@Zajozor, the case involving the Django Permission class will require an update to the Django type stubs so the metaclass declaration properly reflects the fact that it supports a custom __or__ overload method. Please file a separate issue on this, since it's a different issue from the type checker logic.
@cdce8p, I'm able to repro this problem in pylance but not in pyright, which is strange. I suspect it's related to the semantic token highlighting implemented in pylance. I'll investigate further.

@cdce8p, I found the root cause of the problem you reported. As I suspected, it was a bad interaction between the semantic token provider and the type evaluator. When the semantic token provider asks the type evaluator "what is the type of this token?", there are cases where we need to walk up the parse tree and evaluate the expression in a broader context. In this case, for example, we need to evaluate the entire type annotation expression so we know that the expression has a postponed evaluation.
The fix is in place, and it will be included in the next release.

@erictraut Thanks for spending the time investigating and fixing the issue 👍

This issue has been fixed in version 2020.11.0, which we've just released. You can find the changelog here: https://github.com/microsoft/pylance-release/blob/master/CHANGELOG.md#2020110-4-november-2020

For those wondering about the DRF Stubs issue, a very temporary fix is it edit your stubs locally, just go to rest-framework-stubs/permissions.pyi:34 (or just "Go to Definition" of BasePermission) and replace this:
class BasePermission(object):
with this:
class BasePermission(object, metaclass=BasePermissionMetaclass):
Everything else is already there, the metaclass itself is in the stub and it properly inherits from OperationsHolderMixin which has the __or__ method defined.

@erictraut There still seems to be an issue when using more the two types.
from __future__ import annotations
var: int | str | list | None = None

This would be a valid syntax: https://www.python.org/dev/peps/pep-0604/#simplified-syntax

@cdce8p, this is a different issue. You're using annotations future with a version of Python older than 3.10. This will be fixed in the next release.

@erictraut The | operator for type annotations won't be exclusive to 3.10, since postponed evaluation of annotations is enough to support it, as long as the type checker understands it. It can even be used in type comments with Python 2. python/typing#429 (comment) and python/mypy#9647
It might be an idea to limit the warning to isinstance and issubclass calls, or check if from __future__ import annotations is used with type annotations.

Thank you soo much!!! This worked for me!


bug: ion-content height on iPhone XR causes ion-footer to disappear for landscape web users with 2+ tabs open

With Ionic 4.3 on iPhone XR (and probably all +4.7 screen sizes), when launching a modal in landscape mode, then the ion-content height is set too tall, so ion-footer does not display, for web-based Ionic (non-native).
It seems iOS Safari now shows address bar and tabs when in landscape mode on iPhone XR, but ion-content sets height as if the address bar and tabs did not show. As a result, ion-footer is displayed off screen for these users.

Tested and confirmed as an issue with Ionic 4.4.0 with a fresh starter repo... see https://github.com/thmclellan/ionic-content-test for working example.
If you open the app on iPhone XR, with or without tabs, you can see the window.innerHeight being changed dynamically, but ion-content does not resize as expected, so ion-footer is hidden.
Screenshots:

After investigating it seems the root cause is how Safari treats body.height of 100%, based on the device's viewport when the user is scrolling, not based on window.innerHeight when tabs/address bar are visible. See https://medium.com/@susiekim9/how-to-compensate-for-the-ios-viewport-unit-bug-46e78d54af0d.
We're using ion-footer to show some essential navigation on a modal in landscape mode. In case it helps anyone else, one workaround is to set body height explicitly based on window.innerHeight. For example, before launching the modal, if the user is on iOS, we're setting body height as follows:
  document.body.style.height = window.innerHeight + 'px';

And we'll likely revert back to the previous setting on modal dismiss.  Another approach would be to add this logic when first initializing and add a window.onresize event listener.
Repo at https://github.com/thmclellan/ionic-content-test has been updated with this workaround and a toggle button to alternate between natural body height (100%) and explicit height based on window.innerHeight.
Probably worth adding this logic to the core framework for PWA's on iOS. The UX reasons for Safari's handling to 100% height don't seem relevant when working with ion-content and ion-footer.


Feature: Add prettier for formatting

We can use the prettier module and set up its config file to autoformat the code on save. This will help to maintain the format of the code even when the contributors make changes to the code.

Yes that was much required. Do it with the priority. Assigning u.


git-revise: init at 0.4.2

Motivation for this change
A flexible and fast alternative to git commit --amend/git rebase.
See also:

https://mystor.github.io/git-revise.html
https://lobste.rs/s/5gdnrt/introducing_git_revise
https://git-revise.readthedocs.io/en/latest/

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg build git-revise


Allow exposing dashboard via configurable entrypoints

It's common practice to access privileged interfaces via kubectl proxy and port-forwarding tools. This PR exposes the dashboard via the unexposed default traefik entrypoint which is available on port 9000 on the traefik pods. This change makes the helm chart more plug and play without negatively affecting security.
Fixes #79

Hi @ncmans
Thanks for you contribution.
Could you please rebase and bump chart version.

@mmatur done :+1:

@ncmans Thanks for the quick answer.
After a discussion with the Traefik team we have decided to not give the possibility to customise the dashboard entry-point and to serve the dashboard on traefik entry-point that is not exposed by default.
I am closing this PR in favor of #100


failing upstream-dev polyfit warning test

In xarray/tests/test_dataset.py::TestDataset::test_polyfit_warnings, the second call to polyfit:
ds.var1.polyfit("dim2", 10, full=True)

emits a warning with the current upstream-dev numpy:
xarray/tests/test_dataset.py::TestDataset::test_polyfit_warnings
  .../xarray/core/alignment.py:307: FutureWarning: Index.__or__ operating as a set operation is deprecated, in the future this will be a logical operation matching Series.__or__.  Use index.union(other) instead
    index = joiner(matching_indexes)

since the test makes sure only the first call emits a warning (a np.RankWarning), the test fails.

Our alignment is based on this. So this may have quite some implications. I am surprised there are not more warnings (by number) in the test suite.


Fixed login bug.

When attempting to login, I would get this error (in debug mode):
[DEBUG] unexpected end of JSON input
[ERROR] login verification failed
When the email is not confirmed, the post request returns an empty body which raises an error, preventing the timer from re-running and effectively stopping the login process.
I simply made it so that when the body is empty, it returns both an empty response and a nil error, allowing the timer to re-run.

Looks good. Merging this.

Looks good. Merging this.


Fleks 2.0 (WIP)

Progress:

[x] Breakthrough
[x] Code adjustments
[x] Unit test adjustments
[ ] Update comments
[ ] Update wiki
[ ] Update ReadMe
[ ] Update version and combine JVM and KMP. Maybe make the current master a new JVM_history branch to keep the history
[ ] Verify new version in Dinoleon and Mystic Woods

This is currently a WIP for Fleks 2.0. I did most of the code adjustments  but I am still missing the unit test adjustments and comments.
I did a test driven development approach (fleks2-tdd.kt) which I kinda liked for this exercise. It helped to get the API faster in the way I wanted it.
Also, I want to discuss the changes that I have applied, if they make sense to everyone or not. Be aware, here comes a wall of text ;) Questions are highlighted in bold and with a ❓ emoji.
To easily create the boilerplate code for the new Fleks component approach, I suggest to create a Live Template in the IDE. For example in IntelliJ we can do the following. You then just type flekscomponent or whatever abbreviation you want to use to get the necessary code.

Template text

private class $FLEKS_COMPONENT$ : Component<$FLEKS_COMPONENT$> {
	override fun type() = $FLEKS_COMPONENT$

	companion object : ComponentType<$FLEKS_COMPONENT$>()
}

TLDR: imo the Fleks codebase gets a little smaller and easier to understand with those changes. Also, with the removal of reflection and factory methods it is easier to understand, use and debug. The performance is slightly faster than before. I guess because of the removal of factory method calls and the conversion from listeners to hooks.

Changes
Components
It is no longer necessary to add components to the WorldConfiguration. ComponentMapper are created automatically similar to the JVM version, and we also no longer need any reflection or factory methods.
In order for that to work, a new interface and abstract class was added and every component now must implement the new Component interface.
More about the benefits below in the "Systems" section.
A position component now looks like this:
data class Position(
    var x: Float,
    var y: Float,
) : Component<Position> {
    override fun type(): ComponentType<Position> = Position

    companion object : ComponentType<Position>()
}

The ComponentType is important to give each type of component a unique ID which is necessary for Fleks internal stuff to optimize the way components are stored.
With Kotlin's unnamed companion object we can then achieve imo some nice syntax sugar (see below in "Systems" section).
With this change it is now also possible to add the same component multiple times to an entity. This is not possible in Fleks 1.x. This is something new! :) Here is an example of a Sprite component that can either be a background or foreground of an entity:
ata class Sprite(
    val background: Boolean,
    var path: String = "",
) : Component<Sprite> {
    override fun type(): ComponentType<Sprite> {
        return when (background) {
            true -> SpriteBackground
            else -> SpriteForeground
        }
    }

    companion object {
        val SpriteForeground = object : ComponentType<Sprite>() {}
        val SpriteBackground = object : ComponentType<Sprite>() {}
    }
}

Because type() is a function, we can basically do whatever we want and therefore reuse the same component class for different purposes.
Systems
Since we cannot get annotations in KMP, I adapted the existing approach of defining the family configuration in a system's constructor.
For that, I added a new family function with a new FamilyDefinition configuration to define the family and removed the allOf, anyOf, noneOf arguments with a single family argument.
This has two advantages imo:

Users must provide a family and cannot forget it anymore. Before this was not immediately visible because allOf, anyOf and noneOf had a default to an empty collection.
The amount of constructor arguments for an IteratingSystem gets reduced

Here is an example of the new syntax:
private class PositionSystem : IteratingSystem(family { all(Position) }) {
    override fun onTickEntity(entity: Entity) {
        entity[Position].x++ // <-- new way of retrieving components
    }
}

Note, that ComponentMapper are no longer necessary from a user's point of view. Via entity[Position] we can now simply access the position component of an entity.
Also, family definitions no longer need the ::class syntax. The allI, none and any functions of a FamilyDefinition take a vararg of ComponentType.
❓ Shall we use the familyDefinition approach? Or is the previous approach nicer?

The configureEntity function got replaced with an extension function for an Entity. Instead of writing configureEntity(entity) { ... } it is now entity.configure { ... }. This is a minor change but I think it is easier to read. Also, the configuration API got changed from mappers to the new component approach.
Here is an example of one of the benchmark systems that removes a FleksPosition component and adds a FleksLife component:
class FleksSystemComplex1 : IteratingSystem(family { all(FleksPosition).none(FleksLife).any(FleksSprite) }) {
    private var actionCalls = 0

    override fun onTickEntity(entity: Entity) {
        if (actionCalls % 2 == 0) {
            entity[FleksPosition].x++
            entity.configure { it += FleksLife() } // <-- new way of adding components
        } else {
            entity.configure { it -= FleksPosition } // <-- new way of removing  components
        }
        entity[FleksSprite].animationTime++
        ++actionCalls
    }
}

Finally, a new compareEntityBy function was added for easier sorting. You can now make any component implement the Comparable interface and then write the sorting code in that specific component. With that you can achieve something like this:
private data class SystemTestComponent(
    var x: Float = 0f
) : Component<SystemTestComponent>, Comparable<SystemTestComponent> {
    override fun type() = SystemTestComponent

    override fun compareTo(other: SystemTestComponent): Int {
        return other.x.compareTo(x)
    }

    companion object : ComponentType<SystemTestComponent>()
}

private class SystemTestIteratingSystemSortManual : IteratingSystem(
    family = family { all(SystemTestComponent) },
    comparator = compareEntityBy(SystemTestComponent), // <-- easy and concise way to create an EntityComparator
    sortingType = Manual
) {
    // ... rest of the code omitted
}

Entities
As seen above, entity configuration and also the creation got adjusted to the new components implementation. Here are some examples:
    @Test
    fun testComponentHooks() {
        val addComponent = Position(0f, 0f)
        val removeComponent = Position(0f, 0f)
        lateinit var testWorld: World
        testWorld = world {
            components {
                // example of an onAdd hook for a position component. 'component' is of type Position
                onAdd(Position) { world, entity, component ->
                    component.x = 1f
                    assertEquals(testWorld, world)
                    assertTrue { entity.id in 0..1 }
                    assertTrue { component in listOf(addComponent, removeComponent) }
                }

                // example of an onRemove hook for a position component. 'component' is of type Position
                onRemove(Position) { world, entity, component ->
                    component.x = 2f
                    assertEquals(testWorld, world)
                    assertEquals(Entity(1), entity)
                    assertEquals(removeComponent, component)
                }
            }
        }

        // entity that triggers onAdd hook
        testWorld.entity { it += addComponent }
        // entity that triggers onRemove hook
        val removeEntity = testWorld.entity { it += removeComponent }
        testWorld.configure(removeEntity) { it -= Position }

        assertEquals(1f, addComponent.x)
        assertEquals(2f, removeComponent.x)
    }

    @Test
    fun testFamilyHooks() {
        lateinit var testWorld: World
        lateinit var testFamily: Family
        var numAddCalls = 0
        var numRemoveCalls = 0
        testWorld = world {
            testFamily = family { all(Position) }
            families {
                // example of an onAdd hook for a family with allOf=Position
                onAdd(testFamily) { world, entity ->
                    ++numAddCalls
                    assertEquals(testWorld, world)
                    assertTrue { entity.id in 0..1 }
                }

                // example of an onRemove hook for a family with allOf=Position
                onRemove(testFamily) { world, entity ->
                    ++numRemoveCalls
                    assertEquals(testWorld, world)
                    assertEquals(Entity(1), entity)
                }
            }
        }

        // entity that triggers onAdd hook
        testWorld.entity { it += Position(0f, 0f) }
        // entity that triggers onRemove hook
        val removeEntity = testWorld.entity { it += Position(0f, 0f) }
        testWorld.configure(removeEntity) { it -= Position }
        // trigger family update to call the hooks
        testFamily.updateActiveEntities()

        assertEquals(2, numAddCalls)
        assertEquals(1, numRemoveCalls)
    }

ComponentListeners and FamilyListeners
I thought about this and in my opinion it is actually useless to have a list of listeners because why would someone register multiple listeners for the same component/family? At least I never used that and I could not come up with why I would ever do that.
That's why I changed the listener approach to a "hook" approach. It is basically the same but you specify only one add/remove hook per component/family. That way we don`t need to iterate over a list of listeners all the time which always has the size 1.
Important to note is that a hook always gets the world instance as an argument and the world now keeps the registered injectables and does not remove them anymore after creation. That way you can still get any relevant object into your hook code.
Here is an example of a component and a family hook. Fleks distinguishes between an add and remove hook. It is not necessary to add the code directly to the WorldConfiguration. It can of course link to a function in a different file.
    @Test
    fun testComponentHooks() {
        val addComponent = Position(0f, 0f)
        val removeComponent = Position(0f, 0f)
        lateinit var testWorld: World
        testWorld = world {
            components {
                // example of an onAdd hook for a position component. 'component' is of type Position
                onAdd(Position) { world, entity, component ->
                    component.x = 1f
                    assertEquals(testWorld, world)
                    assertTrue { entity.id in 0..1 }
                    assertTrue { component in listOf(addComponent, removeComponent) }
                }

                // example of an onRemove hook for a position component. 'component' is of type Position
                onRemove(Position) { world, entity, component ->
                    component.x = 2f
                    assertEquals(testWorld, world)
                    assertEquals(Entity(1), entity)
                    assertEquals(removeComponent, component)
                }
            }
        }

        // entity that triggers onAdd hook
        testWorld.entity { it += addComponent }
        // entity that triggers onRemove hook
        val removeEntity = testWorld.entity { it += removeComponent }
        testWorld.configure(removeEntity) { it -= Position }

        assertEquals(1f, addComponent.x)
        assertEquals(2f, removeComponent.x)
    }

    @Test
    fun testFamilyHooks() {
        val testFamilyDef = familyDefinition { allOf(Position) }
        var numAddCalls = 0
        var numRemoveCalls = 0
        lateinit var testWorld: World
        testWorld = world {
            families {
                // example of an onAdd hook for a family with allOf=Position
                onAdd(testFamilyDef) { world, entity ->
                    ++numAddCalls
                    assertEquals(testWorld, world)
                    assertTrue { entity.id in 0..1 }
                }

                // example of an onRemove hook for a family with allOf=Position
                onRemove(testFamilyDef) { world, entity ->
                    ++numRemoveCalls
                    assertEquals(testWorld, world)
                    assertEquals(Entity(1), entity)
                }
            }
        }

        // entity that triggers onAdd hook
        testWorld.entity { it += Position(0f, 0f) }
        // entity that triggers onRemove hook
        val removeEntity = testWorld.entity { it += Position(0f, 0f) }
        testWorld.configure(removeEntity) { it -= Position }
        // trigger family update to call the hooks
        testWorld.family(testFamilyDef).updateActiveEntities()

        assertEquals(2, numAddCalls)
        assertEquals(1, numRemoveCalls)
    }

❓ Shall we keep the hook approach? Or shall we go back to the listeners approach?
Injectables
As mentioned before, injectables are no longer cleared after the world is created. The main reason is the new hook approach.
But there are also other changes like e.g. the removal of the Inject object. Since the world is now keeping references to the different injectables, we can directly use the world to inject stuff.
Also, it is now possible to do constructor DI instead of just property DI. Imo constructor DI is always preferable because it creates cleaner code and makes it easier to test classes when you directly see their dependencies.
In order for that to work, the world creation process was adjusted a little bit and a new inject function was added.
Here is an example of a system that showcases both DI mechanisms:
private class SpriteSystem(
    val cstrInjectable: String = inject() // <-- constructor DI via the global inject function
) : IteratingSystem(family { any(SpriteBackground, SpriteForeground) }) {
    val propInjectable: String = world.inject("qualifiedString") // property DI via world.inject. The global function will work here as well

    override fun onTickEntity(entity: Entity) = Unit
}

WorldConfiguration
As already seen in some examples above, the configuration of the world has also changed slightly because of the removal of the listeners but also, because of the removal of factory methods for systems. Systems are now directly created which makes it imo easier to understand the code and also to debug the creation process, if necessary.
It is important, that the systems block comes last in the configuration when family or component hooks are registered. Fleks takes care of that and throws a new FleksWrongConfigurationOrder exception if users fail to do so.
The reason is that system constructors can create new entities and if user hooks are registered after system creation, then of course they are not called correctly for such entities.
Here is an example test case with component hooks and a system:
    @Test
    fun notifyComponentHooksDuringSystemCreation() {
        var numAddCalls = 0
        var numRemoveCalls = 0

        world {
            components {
                onAdd(WorldTestComponent) { _, _, _ -> ++numAddCalls }
                onRemove(WorldTestComponent) { _, _, _ -> ++numRemoveCalls }
            }

            systems {
                add(WorldTestInitSystem())
            }
        }

        assertEquals(1, numAddCalls)
        assertEquals(0, numRemoveCalls)
    }

I just read through the text. It sounds awesome. I think I will like the new API very much :)
Is it possible to build this branch and release locally and use it already at this state? I would like to get a feeling for the changes by adapting my code where I implemented an animation script system with fleks.

Thank you for checking it out and for your feedback!
I think it should work but comments are not updated yet and I want to bring in the old mapper API for update entities to make it easier for people to move from 1.x to 2.x. Then at least inside system code you don't have to migrate immediately to the new API.
I will try to do some tests on my own next week. I let you know if it worked or not and if it does work I will convert this PR draft to a real PR 😉

To be honest I wouldn't need the old mapper API and I would not miss it :)
I was always a bit confused when I wanted to change some component in an existing entity.
First one need to call configureEntity and than "search" for the correct mapper and call addOrUpdate on it.
Does the new entity.addOrUpdate {...} API already work? It looks so much more streamlined to the actual process what one want to do :)

❓ Shall we use the familyDefinition approach? Or is the previous approach nicer?

The familyDefintion approach looks nicer IMO.

❓ Shall we keep the hook approach? Or shall we go back to the listeners approach?

Definitely go with the new hook approach. It looks more intuitive to me!

Indeed everything should work already since the tests are passing.
About the mapper API: I personally prefer the new API with ComponentType and imo it is fast enough from performance point of view. Therefore, I'd remove the mapper API completely to "limit" users with options and don't overwhelm them.
However, there might be some bigger projects already existing and migrating to the new API could be time consuming. So maybe we should keep it still for 2.0 but mark it with a deprecation warning and remove it completely in the next release?

Ok, sounds good. Indeed good idea to give a deprecate warning :)

Currently in an onAdd hook of a family I cannot use entity["Component"] as the "get" operator for Entity is only implemented in an IteratingSystem. Will this be somehow also possible in the hooks or do we need here the "old" mapper API to get a component from an entity?

I think world.configure(entity) should do the trick but you have a point. Maybe hooks should already run within an EntityUpdateConfiguration scope to be able to easily modify the entity.
I will have a look at that 🙂

No, it doesn't work. Because there is no "Entity.get" implemented in EntityUpdateCfg class.

@jobe-m: Please check out the last commits. It is now possible to easily access components via the get operator inside a component/family hook, entity create block or entity update block.
I also fixed the WorldCfg DSL so that nesting of components, families and systems blocks is no longer possible. The compiler will now throw an error if a user does that. I used the @DslMarker annotation wrongly until now.
I hope this is now more intuitive!

Sound good. I will check it out :)

Ah, I didn't do a reload of the pull request page. Now I see your edits.
Great. I like in "edit2" the onAddPosition syntax. Even better with the type alias. :)

What do you think if the component hooks are added to the component data class?
I guess that way the hook variables do not pollute the global name space. And the code would be visually located where the data is.
You could even expand the Intellij Template. Thus the code is already there and if not needed can be easily deleted again or not added to the components section in the world.
Here an example how it looks in one of my components:
/**
 * This component holds all needed details to animate an entity's component.
 */
data class AnimationScript(
    var debugAnnotate: Boolean = false,  // TODO not implemented yet
    var tweens: List<TweenBase> = listOf(),

    // Internal runtime data
    var index: Int = 0,            // This points to the animation step which is currently in progress
    var timeProgress: Float = 0f,  // Elapsed time for the object to be animated
    var waitTime: Float = 0f,
    var active: Boolean = false
) : Component<AnimationScript> {
    override fun type(): ComponentType<AnimationScript> = AnimationScript

    companion object : ComponentType<AnimationScript>() {
        /**
         * Initialize internal waitTime property with delay value of first tweens if available.
         */
        val onComponentAdded: ComponentHook<AnimationScript> = { world, entity, component ->
            if (component.tweens.isNotEmpty()) component.waitTime = component.tweens.first().delay ?: 0f
        }

        val onComponentRemoved: ComponentHook<AnimationScript> = { world, entity, component ->
            // do something
        }
    }
}

In the world it could then look like this:
import components.*
import components.AnimationScript.Companion.onComponentAdded
import components.AnimationScript.Companion.onComponentRemoved

[...]

        components {
            onAdd(AnimationScript, onComponentAdded)
            onRemove(AnimationScript, onComponentRemoved)
        }

I think this is totally valid and fine but it is up to the users where they want to put the code. I personally would follow the same approach as you because of personal preference ;)

My last commit makes the second line possible within an EntityHookContext. I think this is more readable :)

I will do the same for getOrNull(). The first looks much better :)

Added Entity.get, Entity.getOrNull and Entity.contains now in the contexts, family and system and I actually identified a wrong behavior for contains calls inside a component removehook. It returned true although it should be false of course because the components gets removed ;)

if(entity has Sprite) {}

Very good idea. I like it, too.

[...]
I added a new query function to the world which I use already as a replacement in some unit test code. It executes a query to the world within an EntityHookContext for a given entity and returns the result of the query.

The new query function looks interesting. Is the "query" more expensive than if the same functionality is used inside a system or the other scopes? (I mean more expensive in terms of runtime e.g.)
I have quite some code which creates and configures entities outside of the world or the systems. I want to keep the entity creation outside of systems since entities are depending on e.g. the game level and I do not want to have a dependency of my systems to specific parts (levels) of my game. (At least I try so...)
Also I am adding a component to an entity in a familyHook function. (That added component is not part of the family definition for the FamilyHook to not trigger recursive calls of the hook).
Also in that FamilyHook I am using injected objects.
I have already seen that this makes a problem with using injected objects in hook context. I get there an exception that an injected object is not used, because I do not use it on creation time of the systems (?). So I guess that this comes from the fact that the inject() is used later at runtime when an entity is created.
Thus currently I need to comment out the throw FleksUnusedInjectablesException(...) in world.kt to get my code working...
Maybe we need another mechanism to use "world external" objects inside the hook functions?

// note that we are outside of a family, system or hook scope and only have a world
// 1) We can get the component of an entity
val pos: Position = world.query(entity) { it[Position] }
// 2) We can also check if an entity has a component
if( world.query(entity) { it has Position }) {

This again looks nice. I will try it out soon...

Lastly, the EntityComparator functionality had to change a little bit. Both compareEntity and compareEntityBy take a world argument now which defaults to the CURRENT_WORLD which means a user does not need to provide anything when using those functions within a WorldConfiguration block. compareEntity now then also simply runs within an EntityHookContext to provide the base entity extension functions to easily retrieve components.

Currently I have no use case yet for sorting of entities in my systems. So I am not using it and have no experience yet.

I think it is basically the same as Kotlin's let, apply, run, etc.. I don't see any expensive overhead in the query functionality. Should be the same as within a system.
You have a good point again with the injectables. I did not think about that yet. Shall we simply remove this check completely and provide a function in the world like getUnusedDependencies? Then users can decide if they want to analyze and debug that or not.
But for sure it is no longer an error because as you mentioned this does not make sense with hooks.

I think it is basically the same as Kotlin's let, apply, run, etc.. I don't see any expensive overhead in the query functionality. Should be the same as within a system.

👍

You have a good point again with the injectables. I did not think about that yet. Shall we simply remove this check completely and provide a function in the world like getUnusedDependencies? Then users can decide if they want to analyze and debug that or not. But for sure it is no longer an error because as you mentioned this does not make sense with hooks.

Good idea to have that checking function. I had the idea to transform the exception into a warning message. But I guess that it does not need a warning since the inject functionality is now different. The injected objects are there forever and can be used whenever needed. :)

I updated Dinoleon to 2.0 and everything works out! FPS are also the same although I did not optimize some things with the migration (runs at stable ~1300 FPS on my machine before 2.0 and after).
I did not push the changes yet because I want to wait for the official 2.0 maven release but so far looks good!
Will try to update MysticWoods tomorrow and if that also looks good then I will update the wiki (fun! :P).
Maybe we can already release 2.0 in 1 or 2 weeks.

Awesome 👍😎
I did not have luck in updating my code to 2.0. It compiles successfully, but when I run it my animation system does nothing. It looks I did some mistake while adapting the code. I need still to debug it ;)

MysticWoods is also working and runs ~50FPS faster than before (from 1170FPS to 1220FPS).
What I like is that some lateinit var in components can be changed to val since a component is no  longer created via reflection and therefore I can immediately set certain values. Same for some var values that can also be changed to val.

I also need to use the query function a few times in this project because I have an entity AI wrapper class for AI stuff. It wraps a world and an entity and provides some utility functionality for AI. Since mappers are gone but I still need to check for some components or access them, I need to use the query function. I ended up writing my own Entity. extension functions using the world.query implementation internally inside that AIEntity class.
I also had to use that in a PlayerController which is responsible for player input handling and a GameModel that I use for the UI in an MVVM approach.

What I did not like that I cannot negate the new infix has function. From what I read this is simply not supported by Kotlin (yet?). I was surprised since you can already write !in or !is in standard Kotlin but !has did not work for me. So, I ended up writing it like this if (!(entity has AnimationComponent)) .

@jobe-m: let's wait until you also were able to migrate your application to 2.0 to be sure that there is no problem in the code changes.
@czyzby: I have two questions for you again where you might be able to help please. First of all, thank you for the different suggestions you proposed early in the development of Fleks. With the new 2.0 version I used some of them :)

Is it possible to support negation of infix functions that return a boolean? For example I have a has function for an entity to return true if it has a specific component or not. I can write entity has MoveComponent  but I cannot write entity !has MoveComponent.
Is this somehow possible? Or can we only write !(entity has MoveComponent)?

Within the context of Fleks (like systems, families or hooks) it is possible to provide entity extension functions e.g. to add a component or remove a component. We can then write entity += MoveComponent(speed =3f) or entity -= MoveComponent. This is possible because systems, families and hooks internally have a reference to the component service and can therefore interact with it to get component information.
However, sometimes you write code outside of Fleks like e.g. some special event handling of your game. I did not see an option to provide those entity extension functions outside because an entity is just an ID and therefore has no access to anything.
What I ended up with is defining a query function to the fleks World which those external classes/logics should have access to. The query then allows the support of the entity extensions again. Here is an example of my ViewModel that reacts on a so called "AggroEvent":
// ...
is EntityAggroEvent -> {
            val source = event.aiEntity
            val sourceType = world.query(source) { it.getOrNull(AnimationComponent) }?.atlasKey
            val target = event.target
            val isTargetPlayer = world.query(target) { it has PlayerComponent }
            if (isTargetPlayer && sourceType != null) {
                updateEnemy(source)
            }
        }

And here is how I defined the query function of the world in Fleks. The EntityHookContext is the thing that provides the entity extension functionality:
/**
 * Executes and returns the result of a [query] for the given [entity].
 * This can be used to retrieve [components][Component] of the [entity] or check
 * if it has certain [components][Component].
 */
inline fun <reified T> query(entity: Entity, query: EntityHookContext.(Entity) -> T): T {
    return query(hookCtx, entity)
}

Do you see a better approach to solve this?

@Quillraven

Is it possible to support negation of infix functions that return a boolean?

I don't think so. Here's the associated issue: KT-5351. It's not ideal, but you might want to provide hasNot infix method instead.

Do you see a better approach to solve this?

What about recommending the built-in with?
with(world) { entity has Component }

It does change this within the context, but I think it introduces less mental overhead when reading the code - you can see that something is being done with a specific entity in the context of the chosen world.

With this change locally in component.kt file my game how also works :)

@jobe-m: good to hear that your game is also working, nice! And I have nothing against changing the logic. I will do that over the weekend and adjust code, comments and tests.
@czyzby: thank you for your quick response and information. I will try out the with(world) version which has the downside that I need to copy&paste the extension functions to the world as well but it should work and make it easier to read, as you mentioned.
Or do you know a solution how to solve the following:

We have some base entity extension functions which are available in certain contexts like systems, families, hooks and soon also world.
Those functions are Entity.get, Entity.getOrNull, Entity.contains and Entity.has/**
 * Returns a [component][Component] of the given [type] for the [entity][Entity].
 *
 * @throws [FleksNoSuchEntityComponentException] if the [entity][Entity] does not have such a component.
 */
inline operator fun <reified T : Component<*>> Entity.get(type: ComponentType<T>): T =
    compService.holder(type)[this]

/**
 * Returns a [component][Component] of the given [type] for the [entity][Entity]
 * or null if the [entity][Entity] does not have such a [component][Component].
 */
inline fun <reified T : Component<*>> Entity.getOrNull(type: ComponentType<T>): T? =
    compService.holder(type).getOrNull(this)

/**
 * Returns true if and only if the [entity][Entity] has a [component][Component] of the given [type].
 */
inline operator fun <reified T : Component<*>> Entity.contains(type: ComponentType<T>): Boolean =
    compService.holder(type).contains(this)

/**
 * Returns true if and only if the [entity][Entity] has a [component][Component] of the given [type].
 */
inline infix fun <reified T : Component<*>> Entity.has(type: ComponentType<T>): Boolean =
    compService.holder(type).contains(this)

I cannot put them on a global scope, I guess, because they require the ComponentService.
My idea then was to use Kotlin's delegation and create an interface with those four functions, one implementation and then in any class that provides those extensions, I simply let it extend the interface and delegate it to the implementation.
However, this does not seem to work because I use the reified keyword and virtual functions cannot have this keyword, for whatever reason.
Do you see any solution to solve that? Or do I need to stick to the current approach and copy&paste those functions everywhere, where needed?

@Quillraven From what I understand, the problem is that each class provides the ComponentService in a different way, but you cannot define an interface with an inline method that has a custom receiver. That's true, but the same limitation does not apply to abstract classes, if they're not a deal breaker. Here's a proof of concept:
typealias Entity = Int

// Example abstract class:
abstract class ComponentProvider {
  abstract val componentService: ComponentService

  // Example reified extension methods:
  inline fun <reified T> Entity.getOrNull() = componentService[this]
}

// Example implementations:
class ComponentService : ComponentProvider() {
  operator fun get(entity: Entity) = entity

  override val componentService: ComponentService
    get() = this
}

class World : ComponentProvider() {
  override val componentService = ComponentService()
}

// Usage:
fun example() {
  val entity: Entity = 0
  with(ComponentService()) { entity.getOrNull<String>() }
  with(World()) { entity.getOrNull<String>() }
  // Does not compile:
  // entity.getOrNull<String>()
}

You'd probably have to put this somewhere in your class hierarchy for every relevant class, but it beats code duplication.

That works for family, world and hooks but unfortunately won't work for IteratingSystem as it already inherits from IntervalSystem and therefore cannot inherit from a second class. I will think about it some more over the weekend. Maybe I can restructure a little bit and otherwise, I can at least avoid code duplication in most areas.
Thanks!

@czyzby: I was able to use an abstract class for all the different areas - thanks again for the hint!
@jobe-m: I adjusted the addOrUpdate functionality as you suggested. Also, the world.query function and EntityHookContext class got removed because the world itself is now providing those extensions.
For that, a new BaseEntityExtension class got introduced to provide those basic extensions in different areas. Following classes inherit from it:

Family
IntervalSystem
World
EntityCreateContext

With this change, the hooks now take one less argument because the world is no longer passed. The hook itself runs within the world context which can be referenced via this.
Finally, I added a notHas infix function.
Now, instead of writing
world.query(entity) { it has Position }

you can write
with(world) { entity has Position }

with(world) { entity notHas Position }

It doesn't sound quite right, but doesNotHave is way too verbose for an infix such as this. Perhaps hasNo makes sense? entity hasNo Position sounds more or less like a sentence.

@Quillraven Just one more idea on how to approach has. You cannot add the ! operator to the infix method itself, but there's nothing stopping you from overloading the operator for the companion object itself. Quick proof of concept:
typealias Entity = Int

abstract class ComponentType<T>(val componentClass: Class<T>) {
  // Add this operator:
  operator fun not() = Negated(this) // Perhaps cache this.
}

// Simple component type storage that marks it for negation:
data class Negated<T>(val componentType: ComponentType<T>)

// Example component:
class Position {
  companion object : ComponentType<Position>(Position::class.java)
}

// Relevant world methods:
class World {
  infix fun Entity.has(componentType: ComponentType<*>): Boolean = true // TODO
  // Add this:
  infix fun Entity.has(negated: Negated<*>): Boolean = !has(negated.componentType)
}

// Usage example:
fun example() {
  with(World()) {
    val entity: Entity = 0
    entity has Position
    entity has !Position
  }
}

Whether entity has !Position is easier to understand than entity hasNo Position, I guess it's a personal preference thing.

I will go for the hasNo version because I think it is easier to read from left to right. Your Negated version is a nice trick though! Did not know about that not operator :)

Tested the new changes in Dinoleon and MysticWoods again, looks good! I like the new with(world) approach a lot more.
E.g. in the AIEntity class I mentioned above, I can now simply wrap the functions in the with(world) construct and can then intuitively work with the ECS:

Also the event handling example from above looks cleaner now imho:

Added now also Entity.configure and Entity.remove to the EntityComponentContext. I guess this is everything that is needed. Here is an example from the test checking all the different functions:

Awesome 👍 I will update my game tomorrow to the latest changes!

Okay, wiki is updated. There are actually a few mistakes in the original documentation already (mainly code example typos) but I did not bother to fix them in the 1.x wiki as it will anyway die at some point.
In 2.0 everything should be correct :) Appreciate it, if anyone wants to read over it!

Well done! 👍 💯
I read through the wiki and also updated my game which still works flavlessly :)
One thing in the wiki I stumbed over. The following sentence on the "system_new" page is a bit hard to understand / follow:
The purpose of this function is performance reasons to trigger internal
calculations of Fleks only once instead of each time a component gets
added or removed.

With "this function" entity.configure is meant. But what does that mean? Currently it can be read also as "usage of the entity extention functions is also possible outside the configure block. Does this has performance penalties or does this work at all? (It shouldn't, I guess). Maybe you can rephrase it?

Thanks for checking! I updated this section in the wiki and also updated the titles to remove the "_new". The updated section now is:

I now removed the kmp branch. We have a history for 1.6-JVM and 1.6-KMP. Master is now up to date with Fleks2.
I will do some final checks tomorrow and then prepare the release.

@czyzby / @jobe-m: What is a good name for our addOrUpdate function? Since we changed its implementation it does not make sense to me anymore. This function is now always calling update and only calling add if the component does not exist.
Therefore, add OR update is not correct imo. But also updateAndAdd is not correct? updateAndAddIfNotExists is way too long :D
Any good ideas?
/**
     * Updates a [component][Component] of the given [type] for the [entity][Entity] by calling [update].
     *
     * If the [entity][Entity] does not have such a [component][Component] yet then [add] is called to
     * assign the returned [component][Component] to the [entity][Entity] before calling [update].
     */
    inline fun <reified T : Component<T>> Entity.addOrUpdate(
        type: ComponentType<T>,
        add: () -> T,
        update: (T) -> Unit,
    )

maybe refresh ?

That's a tough one if you'd like to have this as a single method. I'd probably go with 2 separate methods along the lines of addIfNull and update, or simply a single update method with the add parameter being optional.

Hmmmm 😅
Add is not really optional because it will be called if the component does not exist and also, the component instance will be created lazily via the lambda, because we don't want to create an instance that is not necessary.
Splitting it into two methods it will look like this:
entity.addIfNull{ Move() }.update{ cmp -> cmp.speed += 3f }

// or maybe if addIfNull simply returns either the new component or the existing one, we could do this
entity.addIfNull{ Move() }.speed += 3f

// for more complex components it can be combined with kotlin's 'also' or 'run' to do multiple things

I think that's maybe easier to just remove addOrUpdate completely and replace it with addIfNull ?

That's what I was proposing, yes. Maybe getOrAdd would be even better if you're going with a chainable method.
By the way, can you think of any way to define extension properties as a way to access and optionally create components? Something similar was recently added to ktx-ashley. It's tricky in your case, since entities are just type aliases, but I'm wondering if something along those lines would be cleaner.
import kotlin.reflect.KProperty

// Mock types:
typealias Entity = Int
interface Component

// Example component:
data class Position(var x: Float = 0f, var y: Float = 0f): Component

// Mock World implementation:
open class World {
  inline fun <reified T> Entity.has() = has(T::class.java)
  fun Entity.has(componentClass: Class<*>) = true
  fun <T> Entity.add(component: T): T = component
  fun <T> Entity.get(): T? = null
}

// Example implementation of property delegate:
class ComponentDelegate<T : Component>(
  private val world: World,
  private val componentType: Class<T>,
  private val provider: () -> T
) {
  operator fun getValue(entity: Entity, property: KProperty<*>): T = with(world) {
    if (entity.has(componentType)) entity.get<T>()!! else entity.add(provider())
  }

  operator fun setValue(entity: Entity, property: KProperty<*>, value: T) = with(world) {
    entity.add(value)
  }
}

// Factory method for property delegates:
inline fun <reified T: Component> World.propertyFor(noinline provider: () -> T) =
  ComponentDelegate(this, T::class.java, provider)

// Usage example #1 - global instance:
val world = World()
val Entity.position by world.propertyFor { Position() }

fun example() {
  val entity: Entity = 0
  entity.position.x = 10f

  // The problem is, it pollutes Int/original type API outside of world scope:
  val number: Int = 0
  number.position.x
}

// Usage example #2 - custom world extension:
class MyWorld : World() {
  val Entity.position by propertyFor { Position() }
}

fun usage() {
  val world = MyWorld()
  with(world) {
    // Only works on entities/ints within the world scope:
    val entity: Entity = 0
    entity.position.x = 10f
  }
}

Not sure if you can manage it without a global world instance or forcing the users to extend World, but this would probably be the most natural API you can get away with.

Hmm, not sure if we should rewrite the whole API for fleks 2 again ;)  But it looks like an interesting approach to define components for an entity.
@Quillraven if you rename the addOrUpdate method I would keep the "update" in the name because actually I am using it mainly to update a component of an entity. If that entity does not have that component I would be ok if it is silently added.
But also the getOrAdd approach with the following also is for me ok and well readable. I even like getOrAdd more than the old addOrUpdate pattern.

I actually checked out those commits a few weeks ago and also saw that tagFor functionality (I think it was called like that). I am not sure if we gain much with that given the new API because isn't the only difference than instead of writing entity[Position] you wirte entity.position? Or am I missing something?
I personally prefer the second option but I am not sure how easy it is to implement here. I could imagine your second example and combine it with the existing world DSL function and somehow use this given type of the user world instead of the fleks world internally to get access to the different properties. Something like:
// USE 'T' internally somehow instead of 'World'
fun <T : World> world(
    newWorld: T,
    // entityCapacity: Int = 512, <-- no longer necessary since the world is created outside
    cfg: WorldConfiguration.() -> Unit
): T {
    // !! NEW !! the line below becomes an argument to the function and we make World an open class
    // val newWorld = World(entityCapacity)
    CURRENT_WORLD = newWorld

    try {
        val worldCfg = WorldConfiguration(newWorld).apply(cfg)
        // assign world systems afterwards to resize the systems array only once to the correct size
        // instead of resizing every time a system gets added to the configuration
        newWorld.systems = worldCfg.systems.toTypedArray()
    } finally {
        CURRENT_WORLD = null
    }

    return newWorld
}

@jobe-m: I just updated the master with the new getOrAdd approach. I am currently migrating my three example games ;) Let me know, if this change is fine for you. After that I will release the version :)
I also don't see the property delegation approach for version 2 mainly because I don't have a lot of time anymore for the next couple of weeks/months and I want to get that update out with the version combining (JVM + KMP) and imo improvements over the old API.
Ideally, we can make the delegation approach, if feasible, as an update for 2.x and ideally it does not break anything meaning that users can stick to the current API but can also use the delegation properties if they prefer it.

I actually checked out those commits a few weeks ago and also saw that tagFor functionality (I think it was called like that). I am not sure if we gain much with that given the new API because isn't the only difference instead of writing entity[Position] you wirte entity.position? Or am I missing something?

The extra utility is that you can define how the missing components are handled, and the user does not have to pass a lambda on every call. As for the performance - not sure how big of an overhead the properties are, but getOrAdd doesn't seem that lightweight itself with its lambda parameter. You'd have to compare the two.
Also, tagFor is for "boolean" components like Visible where the existence of the component itself is important info in terms of game logic. propertyFor works similarly to what I've posted above.
Anyway, this wouldn't require a rewrite of the API, since the properties could be built upon the existing one. It could be an optional, complementary syntax to the existing entity[Position]. The only required change  make it work is that you'd probably have to make the World extensible so that the users can define their own Entity extensions within.


Suggested tweak

Summary of problem or feature request
When a table is generated, all columns are created, including those which are not going to be visible. The "visible=>false" columns are then removed, which looks untidy.
I have inserted one line of code to fix this:
if (array_key_exists('visible', $value) && FALSE == $value['visible']) continue;

Original code with addition:
/**
     * Set datatables columns from array definition.
     *
     * @param array $columns
     * @return $this
     */
    public function columns(array $columns)
    {
        $this->collection = new Collection;

        foreach ($columns as $key => $value) {
            if (! is_a($value, Column::class)) {
                if (is_array($value)) {

                  // Skip "invisible" columns:
                   if (array_key_exists('visible', $value) && FALSE == $value['visible']) continue;

                    $attributes = array_merge(
                        [
                            'name' => $value['name'] ?? $value['data'] ?? $key,
                            'data' => $value['data'] ?? $key,
                        ],
                        $this->setTitle($key, $value)
                    );
                } else {
                    $attributes = [
                        'name'  => $value,
                        'data'  => $value,
                        'title' => $this->getQualifiedTitle($value),
                    ];
                }

                $this->collection->push(new Column($attributes));
            } else {
                $this->collection->push($value);
            }
        }

        return $this;
    }

-->
System details

Operating System  Ubuntu 16.04
PHP Version  7.3
Laravel Version 5.7
Laravel-Datatables Version

Thanks for the idea but it will break the installations specially this trick which I often used. I often add hidden column to enable searching on it. An example would be for the name. Having a column first & last name, we often display it as full_name and named as first_name but this limits the search to first_name column. To allow searching on last_name, we can then add a hidden column for it and it will be included in the search.
columns: [
  {data: 'full_name', name: 'first_name'},
  {data: 'last_name', name: 'last_name', visible: false},
]

Ok, I see. Would it be possible to incorporate it as an option? I know that if I update the package, I will lose my tweak.


Flood records in console window: [server] WARN: Colud not get any audio timestamp

Environment

OS: Windows 10 x64
scrcpy version: 2.0
installation method: Windows release.
device model: Motorola G9 Play
Android version: 11

Flood records in console window (several lines per second(!)):
[server] WARN: Colud not get any audio timestamp

Thank you for your report.
https://github.com/Genymobile/scrcpy/blob/1a80333747de602b03ca7ae6b8e58790eeefc3dc/server/src/main/java/com/genymobile/scrcpy/AudioCapture.java#L125
So this means that audioRecord.read() always returns 0, which is unexpected.
According to the documentation, this happens when the buffer is not a direct buffer, which is also unexpected (it's a buffer from MediaCodec).
Could you please paste the output of scrcpy -Vdebug (with just the beginning and the firsts warnings, not the thousand ones :wink:)?

I can reproduce on my Android 11 device by disabling some workarounds:
diff --git a/server/src/main/java/com/genymobile/scrcpy/Server.java b/server/src/main/java/com/genymobile/scrcpy/Server.java
index 5800487da..3e5985ac8 100644
--- a/server/src/main/java/com/genymobile/scrcpy/Server.java
+++ b/server/src/main/java/com/genymobile/scrcpy/Server.java
@@ -83,11 +83,6 @@ public final class Server {
         //  - <https://github.com/Genymobile/scrcpy/issues/994>
         boolean mustFillAppInfo = Build.BRAND.equalsIgnoreCase("meizu");

-        // Before Android 11, audio is not supported.
-        // Since Android 12, we can properly set a context on the AudioRecord.
-        // Only on Android 11 we must fill app info for the AudioRecord to work.
-        mustFillAppInfo |= audio && Build.VERSION.SDK_INT == Build.VERSION_CODES.R;
-
         if (mustFillAppInfo) {
             Workarounds.fillAppInfo();
         }

On dev branch, commit 337d6c2fd351226ede8b7b2b164ffd3d67a6953f avoids to flood the console, but does not fix the audio problem.

Here are the required log entries:
scrcpy_debug_log.txt

Oh, there logs show a different issue, it's not "Could not get any audio timestamp" but "Buffering threshold exceeded".
See #3793.
In particular, replace this binary: https://github.com/Genymobile/scrcpy/issues/3793#issuecomment-1465713098
And run scrcpy --audio-output-buffer=10.


(AzureCXP) Update Action Types to make current

Based on customer feedback on this GitHub , created this PR.
https://github.com/MicrosoftDocs/azure-docs/issues/50996
Based on the current Action Types for creating Action Groups we see in the Azure Portal UI.

@BharathNimmala-MSFT : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off


feat: local ml flow

@nkoppisetty make sure that when you squash merge, just keep one main commit with a sign-off. Remove everything else

@nkoppisetty make sure that when you squash merge, just keep one main commit with a sign-off. Remove everything else

I have defaulted the merge option to the pull title.


(Paper): Implementation differences KernelShap with shapley sampling values

Hi,
I've been digging a bit deeper into the SHAP paper and the shapley sampling values paper and would like to understand the differences better, so I have several questions:

Both SHAP and shapley sampling values use the expectation value conditional on the features in a given subset of features over the data distribution to integrate out the non-included ("toggled off") features. Is that correct? If so, what are the theoretical differences between SHAP values and shapley sampling values?

Shapley sampling values are calculated by sampling both from the data distribution and permutation of features distribution at the same time. KernelShap on the other hand, does not sample the data distribution, but does sample the permutation distribution and then calculates the model outcomes for that feature subset over the whole given dataset? (which could be the training set, kmeans of the training set or just the mean). If so, in theorem 2, the equivalent of sampling from all possible permutations is sampling a number of z' from the Z space, which is possible vectors in the M-dimensional binary space? f(h_x(z')) is then calculated by integrating out the whole given background dataset for the features not included in z'. The final step is then to solve a linear regression where f(h_x(z')) are the 'y-values' and z' the M-dimensional independent variable space and the SHAP values the coefficients to be solved. Is this correct?

If 2 is correct, is it correct that you need a minimum of M+1 samples of the permutation space to be able to solve the linear regression?

I hope these questions make sense and thanks for the package again. :)

Good questions.

At a high level, the Štrumbelj and Kononenko paper on shapley sampling values is where using Shapley values to explain machine learning models was first introduced, while our work on SHAP values represents a unification of these ideas with several other approaches and associated improvements in their estimation and justification. I think they don't do a conditional expectation but just an expectation without conditioning on other features (equivalent to the feature independence assumption of Kernel SHAP). They also reference a different older set of properties for the uniqueness of Shapley values that are less applicable to ML interpretation (for example they don't use the consistency property). So the values they estimate are the same as Kernel SHAP while the estimation method they use (Shapley sampling) is different and based on the original definition by Shapley. I should note that we used the term SHAP values to distinguish Shapley values (which could apply to any definition function of subsets) from Shapley values applied to conditional expectations. We even have a SamplingExplainer object now that is an improved version of the Shapley sampling values approach.

Correct, with the small clarification that the Shapley kernel has infinite weights that effectively eliminate two of the free parameters before the linear regression. I should also note that because Kernel SHAP separates the sampling from the expectations is can also support a non-linear link function.

Well, only if you have no regularization. By default Kernel SHAP will try and use L1 regularization, so if you have 10000 features where only a few matter than you don't need 10001 samples.

Hope that helps!

I have spent a few days looking at some of these paper and implementing some practical techniques. In the process, I came to a few conclusions regarding the issue of multicollinearity damping feature importance.
Without removing features you can make significant use of column subsampling techniques (LightGBM  - params["bagging_fraction"] = 0.20). Second, you can further correct for multi-collinearity-dampening by mean adjusting variable A's aggregate absolute shapely values with the product of all other variable's to variable A correlations. Thus when a variable has a low Shapley value, but many others are slightly correlated with it, as opposed to not correlated with it, it will have a higher absolute aggregate adjusted value. Once I create something more formally, I will add it here.

@snowde Thanks for the comments! Bagging features does seem like it would spread credit among correlated features. I would also be interested in anything you come up with for post-adjusting the importances. A formally motivated way to do that would be useful in many circumstances.

Scott, thanks for the answers on my question!

@slundberg concerning your answer to Toekan's question 3:
"3. Well, only if you have no regularization. By default Kernel SHAP will try and use L1 regularization, so if you have 10000 features where only a few matter than you don't need 10001 samples."
Theorem 2 in your NIPS '17 paper specifies $\Omega(g) = 0$ which would mean "no regularization". If we add regularization (e.g. L1 as you mention), do we still get the SHAP values by solving the weighted linear regression?
Also, from Section "5.1 Computational Efficiency" of the paper I understand that Kernel SHAP is better than Shapley sampling "particularly when regularization is added to the linear model (Figure 3)". This suggests again that regularization is applied to the weighted linear regression giving the SHAP values of g (which again, to my understanding, does not fit to Theorem 2).
Did I get anything wrong?

@mateisc great questions! If you want to exactly compute Shapley values with weighted linear regression then you need to completely enumerate all subsets of features. In practice you won't do that if you have more than 10 or 15 features so you will instead sample from the Shapley kernel. This results in an unbiased approximation of the Shapley values. But, as is typical for regression, you can achieve lower error by introducing some bias. L1 regularization adds bias towards sparse solutions (which are expected for many high dimensional problems). You are right that with L1 regularization you now have a biased estimate of the Shapley values, but it now has lower variance and will still converge to the correct values in the limit of many samples (assuming your regularization term scales with sample size).