[3.x] Add option to use handles to RID

Adds an option to compile an alternative implementation for RIDs, which allows checks for erroneous usage patterns as well as providing leak tests.
Note : This is compiled out during a regular build so there should be no differences to operation. Handles are only compiled in (replacing the regular RIDs) when passing an argument to scons.
Introduction
We've had a number of very difficult issues over the years concerned with the lifetimes of objects. There are two lifetimes of major interest - scene side Nodes (and objects derived from Reference, Object), and server objects, which are referenced using RIDs. It is this latter objects that are addressed here.
An example of the difficult to investigate problems caused by RID lifetimes is #53374. This and similar bugs can be debugged and fixed using this PR.
The implementation of RID in 3.x is basically a glorified pointer to the object (stored in RID_Data, the actual server objects are derived from this). This means that if you copy a RID, all the RIDs point to the same object.
This is great and quite efficient, in that you can basically just look up the pointer to get the object, the problem comes when you come to delete objects, you often get dangling RIDs (pointers) that are pointing to an object that has been deleted, or worse, another object that has replaced the old object.
One widely used solution to this problem is handles containing an indirect lookup for the object rather than a direct link. That way when you delete an object, a further erroneous lookup of the handle can detect this situation,   flag an error and return a NULL. There are a number of other advantages of handles - the object is relocatable, and you can detect use of an invalid handle ID using revisions, etc etc.
https://en.wikipedia.org/wiki/Handle_(computing)
The downside in this case is that the lookup requires an extra indirection, which may potentially cause a cache miss. There is a little extra housekeeping versus a straight pointer, but not a lot, and certainly nothing compared to the cache implications.
Godot 4.x has moved to using handles for the safety aspects.
This PR
What I've done here is add a simple alternative implementation for RIDs that uses handles, and is complete with error checking, and also can pinpoint the original files / lines of leaks etc (to a limited extent). This ability to do tracking hasn't been added to 4.x, it's somewhat difficult logistically but I've managed to get it working via a small macro when creating RIDs (you wrap them in a RID_PRIME()), which compiles out when tracking is switched off.
This is a small extra hassle when creating RIDs, but it's no big deal if you forget to add one, it just means you won't get tracking for that object.
Handle system
The handle system itself is pretty simple, it doesn't attempt some of the optimizations in 4.x, and it reuses the pooled_list templates which have already been created for handles in the BVH and portals, which makes the new code quite easy to understand.
Building with handles
Nothing has changed in the regular builds of Godot, in order to build with handles you need to add an scons argument:
rids=pointers [default]
rids=handles
rids=tracked_handles

The tracking version does a little more than the straight handles implementation because it has to record files and line numbers. But even the tracking version uses less memory than the standard build, around 20 megs less in the small tests I used so far, so the old system is quite memory hungry presumably with red black trees etc.
Implementation Details
The RID_Database is just a global currently. It needs to be readily accessible for the RID_PRIME systems to work efficiently. It could be a singleton, but there's no real need for this and it will probably make it slower. It could also potentially be tagged onto another existing object, but the lifetime issues are quite important here, it needs to be available until other systems have been destroyed, so having it some form of global is a safe bet in this regard.
Performance
I haven't tested yet to any extent, I get the impression the handles version may be very slightly slower in some situations (and possibly faster in others), and it is not highly optimized, indeed many of the calls are in cpp files rather than headers to avoid the unfortunate side effect of recompiling the engine when making changes, as RID is quite deep in the dependency tree.
The main aim to start with however is to provide a debugging build that can be used to investigate and test for RID lifetime issues. We may decide at a later date to swap completely for handles, but there's no need to make this decision at this time.
Example Leak Output
ERROR: RID_Database leaked 10 objects at exit.
   at: shutdown (core/rid_handle.cpp:140)
ERROR: 1 RID objects leaked
   at: N22RasterizerStorageGLES38MaterialE (drivers/gles3/rasterizer_scene_gles3.cpp:4953)
ERROR: 1 RID objects leaked
   at: N22RasterizerStorageGLES36ShaderE (drivers/gles3/rasterizer_scene_gles3.cpp:4924)
ERROR: 1 RID objects leaked
   at: N22RasterizerStorageGLES38MaterialE (drivers/gles3/rasterizer_scene_gles3.cpp:4926)
ERROR: 1 RID objects leaked
   at: N22RasterizerStorageGLES36ShaderE (drivers/gles3/rasterizer_scene_gles3.cpp:4929)
ERROR: 1 RID objects leaked
   at: N22RasterizerStorageGLES38MaterialE (drivers/gles3/rasterizer_scene_gles3.cpp:4930)
ERROR: 1 RID objects leaked
   at: N22RasterizerStorageGLES36ShaderE (drivers/gles3/rasterizer_scene_gles3.cpp:4936)
ERROR: 1 RID objects leaked
   at: N22RasterizerStorageGLES38MaterialE (drivers/gles3/rasterizer_scene_gles3.cpp:4938)
ERROR: 1 RID objects leaked
   at: N22RasterizerStorageGLES36ShaderE (drivers/gles3/rasterizer_scene_gles3.cpp:4941)
ERROR: 1 RID objects leaked
   at: N22RasterizerStorageGLES38MaterialE (drivers/gles3/rasterizer_scene_gles3.cpp:4942)
ERROR: 1 RID objects leaked
   at: N22RasterizerStorageGLES36ShaderE (drivers/gles3/rasterizer_scene_gles3.cpp:4950)

This is actually the standard situation running the engine, these are some default materials / shaders that are not cleaned up in the normal way. They are not problematic, but could be fixed.
In use there may be other leaks in particular projects, which indicate problems that need to be investigated.

PR meeting: reduz said this is good, needs review by @RandomShaper

Since it's an optional build (which I don't think it should be), the CI/CD checks are meaningless.
I tried to build this with rids=handles. The build failed on win10/64 msvc2019.
1>core\rid_handle.cpp(103): error C2039: 'owner_name_id': is not a member of 'RID_Database::PoolElement'
1>Compiling ==> core\string_builder.cpp
1>D:\GD\godot\core\rid_handle.h(123): note: see declaration of 'RID_Database::PoolElement'
1>core\rid_handle.cpp(108): error C2039: 'owner_name_id': is not a member of 'RID_Database::PoolElement'
1>D:\GD\godot\core\rid_handle.h(123): note: see declaration of 'RID_Database::PoolElement'
1>core\rid_handle.cpp(145): error C2039: 'line_number': is not a member of 'RID_Database::PoolElement'
1>D:\GD\godot\core\rid_handle.h(123): note: see declaration of 'RID_Database::PoolElement'
1>core\rid_handle.cpp(145): error C2039: 'owner_name_id': is not a member of 'RID_Database::PoolElement'
1>string_builder.cpp
1>D:\GD\godot\core\rid_handle.h(123): note: see declaration of 'RID_Database::PoolElement'
1>core\rid_handle.cpp(145): error C2039: 'filename': is not a member of 'RID_Database::PoolElement'
1>D:\GD\godot\core\rid_handle.h(123): note: see declaration of 'RID_Database::PoolElement'Compiling ==> core\string_name.cpp
1>
1>core\rid_handle.cpp(145): error C2660: 'RID_Database::register_leak': function does not take 1 arguments
1>core\rid_handle.cpp(116): note: see declaration of 'RID_Database::register_leak'
1>string_name.cpp
1>Compiling ==> core\translation.cpp
1>scons: *** [core\rid_handle.windows.opt.tools.64.obj] Error 2

Building with rids=tracked worked fine. It seems there are several RIDs that leak in the stock engine, without any project. Just quitting the project manager causes it to report those 10 leaks you quoted. I think they should be cleaned up in this PR. Otherwise, they'll be reported as issues and confusing for users who think they are problems in their game instead of the engine.
I'll start using this build for my project.

Ah good spot I'll fix that.  I was mostly checking the tracked build and it takes a while to recompile.
You are correct about CI builds, but in practice it's not hugely important as it is just for special tracking builds at this stage, and once up and running it is unlikely many things would break it (except work on the RIDs themselves). We really want to keep the CI builds as streamlined as possible and limiting the permutations is already done.
If you want to use the handles build in production that is probably fine if you desire (It's preferable to the tracked build as that will bloat the executable, and may be a little slower). It could be optimized a bit for release use but it's not that bad.
We can definitely consider swapping the standard builds to handles in the future, but I'm not sure it's a good idea to jump the gun on this, mainly because this approach and any implementation needs thorough testing, as RIDs are central to the engine. There's also a trade off - handles are safer, but pointer RIDs may be a bit faster in some situations, as they avoid an indirection.

As a compromise I've added a second commit which fixes the cleanup in GLES3 and GLES2. Rather amusingly, there was not even an attempt to cleanup RasterizerSceneGLES2 (there was no destructor!). :grinning:
I still have a RID leak in Environment in GLES2 but don't want to get things out of hand in this PR.

We can definitely consider swapping the standard builds to handles in the future, but I'm not sure it's a good idea to jump the gun on this,

I understand about testing. But the whole purpose of fixing the RIDs ability to determine if they are valid or not was to fix leakage and spatialeditor destroying memory that doesn't belong to it. If this isn't set as the default, then it doesn't meet the primary motivation for writing it, and I need to fix spatialeditor and visual server, and maybe physics server by other means, and assuming the RIDs are still unreliable. Not a deal killer, but worth discussing.

I understand about testing. But the whole purpose of fixing the RIDs ability to determine if they are valid or not was to fix leakage and spatialeditor destroying memory that doesn't belong to it. If this isn't set as the default, then it doesn't meet the primary motivation for writing it, and I need to fix spatialeditor and visual server, and maybe physics server by other means, and assuming the RIDs are still unreliable. Not a deal killer, but worth discussing.

The existing behaviour of RIDs follows the new / delete idiom. The client code is responsible for keeping track of the lifetime and calling delete once, but only once. Doing it more than more, or using the object after calling delete is a bug. Normally that would crash, but with the handles we have the opportunity to more gracefully handle this situation, by not crashing and outputting an error message.
Now the handles are written in such a way as it can detect such use and recover from it, but this pattern of double deletes etc is imo not expected behaviour by the client and should not be relied on, or used as a replacement for a proper bug fix. This behaviour is the same in the handles in 4.x afaik.
So currently using double deletes with handles might work, but it will flag errors, and would be considered a hack, rather than a proper fix. The proper fix is to keep proper track of lifetimes in the client code. There is no reason why you can't use e.g. reference counting for this or other methods if they are warranted in a situation.
Altering the "contract" in the way you suggest is quite a major change and would need to go through a proposal etc.

Yes on balance I think I will split the fixes into another PR and shrink this one to just the tracking. Hopefully won't take too long. :+1: :grinning:

Thanks!

Just to note here, that at the moment, you won't see the leak list at the end when running a game from the Godot IDE. Instead you need to run a game (or the Godot Editor) directly from a console, or c++ debugger for the leak list to show up.
The reason for this is that the machinery for print_line, ERR_PRINT etc to transfer these from the game to the Godot IDE, is shutdown before the leak checks can be done (i.e. after the Rasterizers etc are cleaned up). This is actually a surprisingly common problem for leak checks. :grinning:
This is a bit annoying as it would be nicer to be able to see the leaks directly in the Godot IDE. I'm just seeing if we can fix this by simply moving the call site around for the rid_database shutdown (which lists the leaks), but no joy so far. I'm not sure exactly which parts of the engine are responsible for debug output. So at the moment we either get no output in the Godot IDE, or we get 'fake leaks'.
An alternative could be to move the RID cleanup code for the rasterizers to their finalize() method rather than the destructor (this is just a stub in some cases, but it is obviously intended to help deal with this type of order of destruction problem), and call this before the engine debug output system is shut down.
We could also potentially try moving the cleanup of the debug output to later in the cleanup sequence, but I'm loathe to touch this if possible, as this order has obviously been tested on all the platforms and works.
Having said that the errors for misuse at runtime should show up fine in the Godot IDE, it's just a gotcha to watch out for.

@lawnjelly How do we free the handles? When opening my project in the editor, not running it, then closing it, the console reports 21 leaks in CanvasItem and Texture:
ERROR: RID_Database leaked 21 objects at exit.
   at: RID_Database::shutdown (core\rid_handle.cpp:142)
ERROR: 18 RID objects leaked
   at: struct VisualServerCanvas::Item (scene\2d\canvas_item.cpp:1268)
ERROR: 3 RID objects leaked
   at: struct RasterizerStorageGLES3::Texture (scene\resources\texture.cpp:828)

Looking at texture.cpp:828, it creates an RID for a StreamTexture. However that is freed in the destructor. Since the RID is freed in the destructor, is this falsely reporting a leak?
StreamTexture::StreamTexture() {
        ...
	texture = RID_PRIME(VS::get_singleton()->texture_create());
}

StreamTexture::~StreamTexture() {
	VS::get_singleton()->free(texture);
	texture = RID();
}

Both go to RasterizerStorageGLES3, where the texture_owner.free(p_rid) is ran.
CanvasItem shows the same thing:
CanvasItem::CanvasItem() :
		xform_change(this) {
	canvas_item = RID_PRIME(VisualServer::get_singleton()->canvas_item_create());
...
}

CanvasItem::~CanvasItem() {
	VisualServer::get_singleton()->free(canvas_item);
}

Both go to VisualServerCanvas where canvas_item_owner.free(p_rid); is ran.

That looks correct in theory, but you need to debug it (or post a min reproduction project and I can take a look). The things of interest are the RID _id, and the _revision.
False leaks are in theory possible, but unlikely (most likely a valid bug), because the RID_Database should be shutdown last. There could also be a bug in the tracking (as it is quite new), or most obvious this could be a genuine error that needs fixing.

How about making this the default build option in the next beta release for editor builds @akien-mga?
Suggested here: https://github.com/godotengine/godot/issues/60406#issuecomment-1105826221
The bottom lines are:

The RID system and the way they are free()'d in the servers is inherently flawed and will continue to produce bugs until it is redesigned.
99% of users cannot debug RID issues, and will need dev help.
The RID handle implementation is a useful tool for devs, but it won't be enabled by most users unless special builds are made available.
Users and devs can save time if users can provide the error database on the first go-round. However, now the console has been hidden from them. :/

@lawnjelly said this:

This is perhaps something we could consider now some time has passed.

As far as I remember it from the time:

The handles builds needed testing - I've now been using the rids=tracked_handles builds for 5 months or so now as a daily with no problems at all, for debug and release_debug editor builds. This is a sample size of one, on one platform, but on the flip side the type of problems likely to occur will probably occur on all platforms.
The tracked_handles may bloat the executable size a bit and be a little slower. This could be tested. (On the plus side, memory use was actually lower when I tested this.)

The other question is whether we might use tracked_handles for the editor, and pointers for release templates (as ultimately these should be fastest, in theory at least. We need to profile of course though. In practice, the difference might be negligible, in which case robustness would win.). Given that these two are quite different there is the opportunity for bugs if the implementation of either diverges. An alternative is to make release templates with rids=handles, which has all the same robustness but can't provide such detailed error messages. This is the approach that Godot 4 takes, if I remember correctly.

On the other hand there is always the potential to introduce new nasty hard to find bugs if we change this in the release templates, particularly threading bugs, which the handles is more susceptible to. So on balance I am probably still slightly in favour of keeping the current release template to use pointers (if it ain't broke etc).

I'm personally fairly agnostic on this really, and we should also weigh in how often RID errors happen in practice (not very often it seems now that we have fixed the major existing bugs), and we do have the tools to investigate them now when they do crop up.

Another option is to build with rids=tracked_handles for DEV_ENABLED builds, so that more eyes are likely to see RID bugs earlier. Or we could publish a debug build of the editor with all debugging enabled and rids=tracked_handles with each release, for windows only might be okay as linux users could run it under wine. I think there was some interest in having debug builds available so maybe this could be a "kill two birds with one stone" thing.

Next build should RC1, so it's not the time for experiments anymore for 3.5. But we can consider it for 3.6 beta 1.

I'll remember to use use rids=tracked_handles in my debug builds once 3.5 is released.


Fix a bug in R.values test file

It makes no sense to test for R.keys in R.values testfile. R.keys accidently passed the test clause, but it needs to fix for sure.

You're right that this needs to be fixed.  But I wonder if it would be clearer altogether to skip the list and call each one explicitly:
  it('returns an empty object for primitives', function() {
    eq(R.values(null), [])
    eq(R.values(undefined), [])
    eq(R.values(55), [])
    eq(R.values('foo'), [])
    eq(R.values(true), [])
    eq(R.values(false), [])
    eq(R.values(NaN), [])
    eq(R.values(Infinity, [])
    eq(R.values([]), [])
  });

I agree with you it's more easy to read when writing in this way. If we want to update this, it's better to update R.keys counterpart simultaneously.

Perfect.  Thank you very much!


Add Automatic-Module-Name in manifest

Add Automatic-Module-Name in manifest so Java9 modular applications can depend on this library.
It is recommended for java libraries in public repositories like Maven Central to add Automatic-Module-Name manifest entry. This allows the consumers of the library to move to Java 9 modularization and depend on the Automatic-Module-Name value instead of the name derived from the Jar file. This way, your consumers won't need to make changes when your library is eventually converted to a Java 9 module.
Adding Automatic-Module-Name doesn't affect any code and will simply make a name reservation until an actual module-info.java will be added eventually. More details can be found in this article.
This article has best naming practices for JPMS modules. software.amazon.ion seems suitable as it follows your package naming strategy.
Description of changes:
Adds Automatic-Module-Name header to manifest file so other Java 9 application can depend on a concrete name instead of the name derived from java file (which is ion-java).
Instead of this change, You can also add the module-info.java which makes the library Java 9 compatible.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Yes, I confirm

Thanks. Can I know when a new version with this change will be released to Maven?

I think we should consider a more focused module name, that allows us to vend additional modules in a sensible way.   We know we want to split out the DOM, and we'll certainly have other components in the future.   Something like software.amazon.ion.core would allow us to add software.amazon.ion.dom and other sibling projects.

@therapon
@toddjonker's suggestion sounds good to me. Please consider the future extensibility concerns mentioned by todd and finalize the module name before releasing the next version to maven.

Will do.

To be clear, "core" was a strawman placeholder proposal.  We should do a bit of active planning to scope out what we might release in the future, and then come back to clear and meaningful names.

When do you plan to release a new version to Maven that has this change?

We are still discussing if we only want a single module for ion-java as this is a one way door decision we need to be careful with it.
We should have an ETA for releasing it by end of the week, however if we end up going with multiple modules we may need to some additional work before releasing it to allow for multiple modules

new version released with this pull request, available on maven central
https://github.com/amzn/ion-java/releases/tag/v1.2.0


Regression test for more complex profile

Feature Request
Description of Problem:
The current tests are for relatively simple profiles, with only 1 or 2 fields, and not many decisions.
This doesn't fully ensure our system works for complex profiles, especially for regression purposes
Should this test interesting, random, or full exhaustive?
Acceptance criteria

[ ] Regression testing

Come up with few complex profiles so we can later decide which ones can be/could be automated and which ones could we manually run it.

I would suggest couple of pairing up sessions for generating some more profiles which we think are complex and then start thinking on how we should test it  on a regular basis.

should we close #721

https://docs.google.com/document/d/1D7-mK1t3KXuJ0_8IpNmtzIuvXNK4_HDwOg38d_Vp_qs/edit?usp=sharing


Upgrade "sbt-compiler-api" dependency version to 1.0.0-beta9

Upgrade sbt-compiler-maven-plugin:sbt-compiler-api dependency version from 1.0.0-beta8 to 1.0.0-beta9.
For more details see sbt-compiler-maven-plugin 1.0.0-beta9 issues

New 1.0.0-beta6-SNAPSHOT snapshot deployed.

Documentation for version 1.0.0-beta6-SNAPSHOT updated.


🍒 Cherry pick PR #4742 to staging 🍒

🍒 Cherry pick https://github.com/Expensify/App/pull/4742 to staging 🍒

🚀 Cherry-picked to staging by @roryabraham in version: 1.0.86-10 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

@Expensify/applauseleads please QA this PR and check it off on the deploy checklist if it passes.


Importing BibTex file help video

Description:
Document the process of cleaning and importing a BibTeX file.
Tasks:

[x] Create video that walks users through the process of cleaning and uploading a BibTeX file in the references section.

Acceptance Criteria:

[ ] Video created and link shared with DevSeed.

Blocked until BibTeX files are uploaded appropriately - all fields populating correctly (e.g., DOI, Titles, etc.).

@naomatheus – see how we did it in SNWG. @deborahUAH said that using that approach (adding their Google Drive link into the help text) would work well.

@deborahUAH
Here is the video: https://www.youtube.com/watch?v=1RZRPEoARyo
I will send it to DevSeed if you are satisfied with the video.

@bwbaker1
I like the video content and the pace of presentation is good.  I do not like the automated voice.  Any reason for using this?  Can you make the video with your reading the script?

@deborahUAH
Used AI voice for pace and smoothness. I recorded audio first and listened to it as I made the screen recording.  Then merged audio and screen recording. I can add a third step and record myself reading the script (easy fix).  I will use the same three-step process as I make other help videos.


Ensure __decay builtin is not part of function signatures.

Example in readme fails to compile with at least gcc 14.2.1 and trunk: https://godbolt.org/z/xr536qvEo
Wrap call to __decay builtin in class template, same as #1124.

/ok to test

on clang 18 i get: error: invalid application of 'sizeof' to an incomplete type with sizeof(_Ty) == ~0ul as arg, template <bool = false>, and __decay_<>::template compiles and tests fine on both gcc 14 and clang 18, but maybe there was a reason for using sizeof?
   namespace __tt {
-    template <bool>
+    template <bool = false>
     struct __decay_ {
       template <class _Ty>
       using __f = __decay(_Ty);
     };
   } // namespace __tt
   template <class _Ty>
-  using __decay_t = typename __tt::__decay_<sizeof(_Ty) == ~0ul>::template __f<_Ty>;
+  using __decay_t = typename __tt::__decay_<>::template __f<_Ty>;

/ok to test

/ok to test


Make ivy-prescient-mode work even if enabled before loading Counsel

Problem with load order is easily resolved https://github.com/raxod502/prescient.el/issues/45#issuecomment-546194457 and documented https://github.com/raxod502/prescient.el/commit/d9b39c2ba5914a567d97495bbcaf5559355bf8cc but this still sucks for users as it's an easy trap to walk into. We could probably rewrite our hacks so that they work regardless of load order (probably by ignoring the API that Ivy tries to provide, because it's kind of a Jenga tower).

I no longer care about this issue since I have developed a replacement for Swiper, namely CTRLF. Comment and I will reopen.


feat: Add target group attachment capabilities

Description
Allows for the creation of a "targets" index inside the target_groups variable, where the parameters of the attachment can be defined. The same arguments that are used for the target_group_attachment resource, are the same arguments that should/can be defined. Any additional arguments are created during the "product mapping", but are not used in the resource, but allows for easy upgrading in the future if more arguments are added (only have to update on the resource function)
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lb_target_group_attachment#argument-reference
Motivation and Context
Allow attaching EC2 instance(s) and Lambda functions to the target groups created by this module so all resources can be managed in one location.  Currently, there's no functionality to add any attachments to the target groups this module creates and everything needs to be additionally defined/coded

https://github.com/terraform-aws-modules/terraform-aws-alb/issues/134
https://github.com/terraform-aws-modules/terraform-aws-alb/issues/189

Breaking Changes
This should be backwards compatible, if no "targets" are defined on the target_group variable, no attachments are created.
How Has This Been Tested?
[x] I have tested and validated these changes using one or more of the provided examples/* projects
This was manually tested in a sandbox AWS environment on actual ALB/EC2 resources. I currently don't work w/ lambda function attached to any type of LB, so I did not test that. But since the function calls are the same on the target_group_attachment resource between an EC2 instance and a Lambda Function and this is just a mapping of parameters to that function, I really don't anticipate any issues.
variable definition can look like this now
    target_groups = [
        {
            name = "tg-with-targets"
            backend_protocol = "HTTP"
            backend_port = "80"
            target_type = "instance"
            deregistration_delay = 300
            health_check = { ... }
            stickiness = { ... }
            targets = [
                {
                    target_id = "i-0123456789abcdefg"
                    port = 80
                },
               {
                    target_id = "i-a1b2c3d4e5f6g7h8i"
                    port = 8080
                }
            ]
        },
        {
            name = "tg-without-targets"
            backend_protocol = "HTTP"
            backend_port = "80"
            target_type = "instance"
            deregistration_delay = 300
            health_check = { ... }
            stickiness = { ... }
        }
    ]

@antonbabenko, this look good to merge?

Amazing piece of code! Just a few comments and questions.

I'll get all these reviewed quickly so we can merge, just been buried with work

@veilig2000 I will work on this later today.

Thanks for your contribution, @veilig2000 !
v5.14.0 has been just released.

this PR introduces a breaking change
I'm using terraform 0.12.29, and depend on the module with ~> 5.0
so if I'm running a clean init, I'm getting:
Module xxxx (from "terraform-aws-modules/alb/aws") does not
support Terraform version 0.12.29

because of the updated version constraint
IMO, this should have been a major version change

@lev112 https://github.com/terraform-aws-modules/terraform-aws-alb/issues/195#issuecomment-820316313


Fix kubelet proxy configuration race condition

Trying to fix the race condition in Configure kubelet proxy
First round: alter the way initial node labels are set

Specs and e2e pass, can't confirm if this fixes the
ERROR: Pharos::PhaseManager::Error : Phase failed on 2 hosts:
- 667-do-charming-tapir:
RuntimeError: can't add a new key into hash during iteration
- 667-do-first-wildcat:
RuntimeError: can't add a new key into hash during iteration

Looks like merge broke it.


Redirect Does Not Work in Lecture-5.71

The redirect method employed in the /add-product route fails by redirecting to the 404.html view, when it should redirect to /.
I will continue to investigate and have a message in to the instructor and his assistants. For now it remains an open issue associated with this commit: https://github.com/RHieger/nodejs-the-complete-guide/commit/76f60eaacc8ae91df301e8c163d9832b7507904c

Code in lecture 5.71-fix branch resolves this issue.


Question: Disk Types

Now that the DISK_TYPE dictionary has been refactored to each provider's disk implementation, what's the best way to represent local/remote disks that can both be spinning disks and/or SSD disks?
Under disk.py I can only see these types defined
STANDARD = 'standard'
REMOTE_SSD = 'remote_ssd'
PIOPS = 'piops'  # Provisioned IOPS (SSD) in AWS
LOCAL = 'local'

In Rackspace's case, depending on the flavor (a.k.a machine-type/instance-type), disks could be local or remote, and they can be either spinning disks or SSDs.
Perhaps introduce REMOTE_HDD constant? What do you think?

REMOTE_HDD is equivalent to STANDARD currently.
To handle all of these cases, we'd need to:

Rename STANDARD to REMOTE_HDD.
Rename LOCAL to LOCAL_SSD.
Introduce LOCAL_HDD.

@ehankland, what do you think?

@cmccoy I thought STANDARD meant whatever is the default when booting an instance.
Let me give some more context, to make sure you have all the information.
In Rackspace, the default type of storage depends on the flavor, for example:

compute1-* and memory1-* by default uses what we are calling REMOTE_SSD
it could also be REMOTE_HDD
general1-* flavors use LOCAL_HDD,
io-* flavors use LOCAL_SSD,
onmetal-* use something like LOCAL_SSD but actually they are NVMe SSDs. (These are fast!!)

Disk Volumes can only be attached to certain flavors, and can be either SSD or HDD.

@cmccoy
I think being explicit about remote_hdd is probably a good idea since it's obviously caused some confusion. Splitting up local into local hdd and local ssd would be a bit more difficult. We'd either have to keep track of which instances have local ssd/hdd or we'd have to do the detection after the instances have already booted (this also assumes that all disks correctly report whether they are rotational which may not be universally true). My assumption would be that if a user specified "local_hdd", but the instance only had local ssd (or vice versa) we'd raise an exception and refuse to run. If we did the detection after boot this could be fairly annoying, since all the resources would be created and then torn down. That still leaves the keeping track of machine type/disk type option. I'm open to that if everyone agrees that we should do it, but we basically already encode the disk type in sample metadata since we record the cloud and machine type, so the only real advantage we'd get out of splitting them (at least that I can see) is refusing to run. Another option is simply adding more explicit metadata about the disk type if that's what people wanted out of the split. If anyone thinks there are compelling reasons to split local, please continue the discussion in this thread - it would be useful to hear more opinions.
@meteorfox
I couldn't seem to find info about Rackspace's local hdd offerings - I could only see local ssds on the pricing page: http://www.rackspace.com/cloud/public-pricing (the page says that "general" types have ssds for the system disk and don't have local data disks) - this page seems to agree with that as well: http://docs.rackspace.com/servers/api/v2/cs-devguide/content/server_flavors.html

Good idea.  In fact Eric and I debated the naming of STANDARD to be
REMOTE_HDD or not.  We should just do it.
In terms of LOCAL_SSD vs LOCAL_HDD we also discussed this.  This would
require a lot more knowledge of VM's under test than PKB currently has.  We
should discuss this point more.
On Thu, May 28, 2015 at 2:03 PM, Carlos Torres<EMAIL_ADDRESS>wrote:

Now that the DISK_TYPE dictionary has been refactored to each provider's
disk implementation, what's the best way to represent local/remote disks
that can both be spinning disks and/or SSD disks?
Under disk.py I can only see these types defined
STANDARD = 'standard'
REMOTE_SSD = 'remote_ssd'
PIOPS = 'piops'  # Provisioned IOPS (SSD) in AWS
LOCAL = 'local'
In Rackspace's case, depending on the flavor (a.k.a
machine-type/instance-type), disks could be local or remote, and they can
be either spinning disks or SSDs.
Perhaps introduce REMOTE_HDD constant? What do you think?
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/284.

--
Anthony F. Voellm (aka Tony)
Google Voice:<PHONE_NUMBER>
https://www.google.com/voice/b/0?pli=1#phones
Blog: http://perfguy.blogspot.com/

@ehankland whoops, that's my mistake. You are right, I confused the general flavors with our old standard flavors. They renamed all of them recently, and I still get confused. :)
Now these are the correct ones:

Local SSDs

general1-* (Multiple RAID10 SSDs)
io1-*  (Multiple RAID10 SSDs)
onmetal-(compute|memory)1 (32GB SATADOM)
onmetal-io1  (32GB SATADOM & 2 x 1.6 TB PCIe Flash, bare-metal)

Local HDDs

standard  (1x RAID10 SATA HDDs) - Deprecated

Remote volume (SSD and/or HDD) can be attached to any of these flavors

compute1-*
memory1-*
general1-*
io1-*

@cmccoy I'm trying to understand how the ScratchDisk abstraction works.
By default some of our flavors only use local disks (e.g. general1-*, io1-*, onmetal-*, standard) with the option to attach remote volumes (except for onmetal, and standard). So whenever the user executes pkb.py with any of those flavors, I want the default to use those local disks provided by the flavor and skip the remote storage volume creation. If there are more than extra 1 local disk available, then it would stripe them together in RAID0. If the flavor does not have extra local disks, then perfkit would just use that (the system disk).
Optionally, I want to let the user specify any number of remote storage volumes (up to 14, and if supported by the flavor), that he/she could attach, then also stripe them together (only remote volumes would be striped, if this is specified), in this case, perfkit would use these volumes for the benchmarks.
The part I don't understand completely is, how is the system disk (e.g. where the OS is running) considered? Is that considered a local scratch disk? I guess my confusion comes from not understanding the whole design to handle disk types, from the CLI flags available to the user, to the actions emitted by perfkit.
It'd be awesome if there we some kind of guide explaining these abstractions, and show examples on how one would go about implementing for a provider. :)
Thanks in advance!

Up until now we haven't let users make use of the system disk as a "scratch disk", but as we are adding more cloud providers, we are running into more clouds where it makes sense to allow that ( @klausw encountered this when adding digital ocean support).  So it's pretty likely that we'll make changes in order to support it.
There are really only 3 flags which govern which disks get used:
flags.DEFINE_enum(
'scratch_disk_type', disk.STANDARD,
[disk.STANDARD, disk.REMOTE_SSD, disk.PIOPS, disk.LOCAL],
'Type for all scratch disks. The default is standard')
flags.DEFINE_integer('num_striped_disks', 1,
'The number of disks to stripe together to form one '
'"logical" scratch disk. This defaults to 1 '
'(except with local disks), which means no striping. '
'When using local disks, they default to striping '
'all disks together.',
lower_bound=1)
flags.DEFINE_integer('scratch_disk_size', 500, 'Size, in gb, for all scratch '
'disks, default is 500')
The scratch_disk_type flag determines whether we try to use  remote magnetic ("standard"), remote ssd, or local disks (piops is provisioned iops and is basically AWS specific - at least with the clouds that have been added so far) .
The num_striped_disks flag determines how many disks of the given type you want to stripe together and treat as a single device. A value of 1 (the default) means no striping - a value of 14 would mean creating 14 disks, striping them together and using that as the "scratch disk" for the benchmark.
The scratch_disk_size flag determines how large each disk should be if that can be controlled (it can't be in the local scratch disk case).
A virtual machine's CreateScratchDisk method takes a disk.BaseDiskSpec (which has been filled out according to those flags) and does the work of creating, attaching, striping, formatting, and mounting the scratch disk.  Assuming you create a disk.BaseDisk subclass for Rackspace disks, and implement the CreateScratchDisk method, that should be just about everything you need.
There are a few optional steps:

If necessary, you can implement SetupLocalDisks() - if there's anything you want/need to do (such as unmounting an automounted drive or running a script) in order to prepare the local disks, you can do it here. This will only be executed if --scratch_disk_type=local is specified.

There is a virtual_machine.BaseVirtualMachine attribute "max_local_disks":
max_local_disks: The number of local disks on the VM that can be used as
scratch disks or that can be striped together.
It defaults to 0, but if you set it to the number of local drives on the system, PKB will automatically stripe all of them together when running with --scratch_disk_type=local (unless you specify --num_striped_disks which will override that).

Implement GetLocalDisks() which will provide a list of all the raw device paths on the vm. Only one benchmark makes use of this currently (aerospike) and only when using --scratch_disk_type=local. This will probably be removed at some point since it doesn't fit with the way the rest of the benchmarks make use of disks.

If you need more info or if anything I said wasn't clear, please let me know.

Based on my current experiments, I think it would help to split the "scratch disk" abstraction into three separate cases:

single data directory: a benchmark needs to store a substantial amount of data, for example fio or bonnie++, but just uses the mount point it gets from vm.GetScratchDir() as a plain directory. This could be striped for performance, but that isn't strictly necessary to execute the benchmark, and could use a plain directory within the base image. It would help if the benchmark would specify how much storage it actually needs - assuming the current default of 500G for scratch disks as a minimum seems excessive.

multiple data directories: a benchmark uses multiple directories that are assumed to be on separate disks, for example copy_throughput which uses vm.GetScratchDir(N). This could technically work by creating multiple directories, but that's probably not a good idea if the results would not be comparable to using actual independent disks.

disk devices: a benchmark specifically needs one or more disk devices (not directories) to use internally, i.e. for raw data storage or custom filesystems. This includes aerospike.

I'm currently using a hack which just creates and returns a plain directory for vm.GetScratchDir(), this seems initially promising and gets a few scratch disk based benchmarks working. But I think it would be cleaner to explicitly distinguish the cases through the API.

@ehankland This is awesome. Ok, it's starting to make sense. I'll let you know if I other questions come into mind. But as you mentioned, Rackspace is also one of those providers that gives you extra local drives, separate from the system disk. I'll probably end up doing the same hack @klausw did, to support using a directory in the system disk as a ScratchDir.
@klausw I think you are onto something.  Some benchmarks need a mounted path in the file system (choice of file system matters), while others might take the path to the block device directly. In the case of benchmarks that only take a mounted path, then it will depend how you chose to setup the block devices behind that mount (e.g. stripe, mirror, writethrough, writeback, etc). Perhaps we should push the task of setting up the file system mounts, and the block devices to the benchmark, then the benchmark decides, where and how to create the ScratchDir for the benchmark to run. But probably each disk will need some metadata describing the nature of the block devices (e.g. system disk, remote, local, ssd, hdd), if they are mounted or not, where are they mounted etc. All of this seems to be the motivation behind BaseDisk, as far as I can tell.
So, maybe, drop GetScratchDir() from the BaseVirtualMachine, and delegate the responsibility of setting up the storage to the benchmarks, by adding a new method to BaseVirtualMachine that returns all the available block devices?

While we are giving this should we also include ROOT_DISK?
On Jun 1, 2015 7:00 PM, "Carlos Torres"<EMAIL_ADDRESS>wrote:

@ehankland https://github.com/ehankland This is awesome. Ok, it's
starting to make sense. I'll let you know if I other questions come into
mind. But as you mentioned, Rackspace is also one of those providers that
gives you extra local drives, separate from the system disk. I'll probably
end up doing the same hack @klausw https://github.com/klausw did, to
support using a directory in the system disk as a ScratchDir.
@klausw https://github.com/klausw I think you are onto something. Some
benchmarks need a mounted path in the file system (choice of file system
matters), while others might take the path to the block device directly. In
the case of benchmarks that only take a mounted path, then it will depend
how you chose to setup the block devices behind that mount (e.g. stripe,
mirror, writethrough, writeback, etc). Perhaps we should push the task of
setting up the file system mounts, and the block devices to the benchmark,
then the benchmark decides, where and how to create the ScratchDir for the
benchmark to run. But probably each disk will need some metadata describing
the nature of the block devices (e.g. system disk, remote, local, ssd,
hdd), if they are mounted or not, where are they mounted etc. All of this
seems to be the motivation behind BaseDisk, as far as I can tell.
So, maybe, drop GetScratchDir() from the BaseVirtualMachine, and delegate
the responsibility of setting up the storage to the benchmarks, by adding a
new method to BaseVirtualMachine that returns all the available block
devices?
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/284#issuecomment-107772331
.

@meteorfox - Benchmarks should have the option of setting up their own storage (aerospike already does this), but I think it would still be useful to let a benchmark say "give me a data directory" where the details of this are taken care of by the specific VM implementation combined with flag settings. So I'd want to keep something like GetScratchDir(), just being a bit more explicit about what the benchmarks wants to do with it. For example, the disk spec could say "minimum 20G space, lots of I/O". On GCP this would pick a 500G scratch disk to ensure fast I/O, and on DigitalOcean it would just give you a directory within the image's base partition.
Changing the VM API to only exports block devices would be annoying for DigitalOcean since there's only one device that's already in use for the OS filesystem, and according to this post it's not resizable.

@klausw Great point. I agree, seen from that perspective I can see the value of something like GetScratchDir() taking care of the details.

@ehankland Now that I wrote the scratch_disk_type and machine_type handling, I think I've settled on a way to manage the different options.
Right now:
STANDARD, REMOTE_SSD, LOCAL, and PIOPS are the support scratch_disk_types.
Except for PIOPS, I tried to map each scratch_disk_type to a Rackspace supported storage, for those that make sense.
Here's how I'm doing the mapping, this follows what I think is 'best fit' as compared to other providers:

What I want:
I think another approach is to use STANDARD to represent the machine_type's strengths in terms of storage. Introduce REMOTE_HDD, to specify non-SSD remote volumes, and introduce BOOT_DISK to specify that a path in the system disk should be use.
They idea of using STANDARD is that if a user doesn't specify a scratch_disk_type, then it would default to what we think is the disk type in terms of performance.
Here's how the compatibility matrix would look for Rackspace with this approach:

What do you think?

I talked this over with @voellm and @klausw and we liked your suggestion. From the discussion, I think we also want to rename the new STANDARD type to RECOMMENDED and LOCAL to LOCAL_EPHEMERAL (in addition to the name changes you proposed)  to be very explicit about the types so there isn't any confusion.

Looks like we all agreed on what to do here.  Ok to close?

I'm not sure if we should close yet, should we keep track of the changes here? or should we open a new issue to track those changes?

Carlos covered this at the recent community meeting here - https://docs.google.com/presentation/d/1XLx3pnwAzI0XHohWxcNQbxnN91jcJXYSAPd6xFfD6Hw/edit#slide=id.g64899ba87_1_9
@noahl is going to work on the next set of changes with Carlos.

Hi @meteorfox
I've been looking through the code and at your slides from the community meeting. There are several different issues brought up here - what the interface for getting scratch disk / directories should be, what should happen when a provider doesn't have a certain type of disk - but I don't think we should let the other issues distract from the renaming project you're proposing. Changing the names of disk categories is something we can do right now, with relatively little work, and it will have a positive impact on PKB. I think we should start with that and hopefully get it merged using the existing infrastructure, while also thinking about what the disk infrastructure should look like.
And speaking of the disk categories, there's an issue I'm struggling with on disks like PD or EBS. In a situation where you choose the size of your drive, and pay more for more performance, what is the "default" size? For best performance, I'd have to recommend spending 100% of your budget on a gigantic PD drive with a huge number of IOPS -- but that's not the best choice for anyone. It's also not clear how to compare the drives. We could try to match drives of the same size, or same maximum IOPS, or same cost. I think same cost is the best choice, but I'm not sure. @voellm?

@noahl
I agree with you, I think we can focus on the renaming pieces first, and maybe add the remote_hdd disk type as a first step.
After that, I suggest to pay our attention on deciding what standard should be replaced with.  I agree that when defining the default disk type, it should not be what is the best disk. Maybe instead we could make it be whatever is the common denominator of all providers, say, using the disk with the root partition, one benefit, is that this will probably be the cheapest option too, and follows the same 'philosophy' as the default flavor. If we go with this approach, then there wouldn't be a need for the root_disk type.
Alternatively, the default disk could be whatever disk that comes by default with the machine type. Example, in Rackspace, a machine type of the general1 class comes with 1 disk, the root partition, even though you can attach remote volumes to the instance, the default for this one would be root_disk. Now, for the onmetal-io1 machine type, the default could be local_ephemeral since this machine type already comes with the extra PCI flash drive, and the user is not incurring on an extra cost as opposed to the remote volumes.
That being said, I know this will bring up the issue of how do we handle the case for when benchmarks use the raw blk device, instead of a file. Possibly we could make the BaseDisk object include a new method or property that indicates the disk cannot be used as a raw device, then have the benchmarks explicitly define that they need a raw device, and whenever the scratch disk does not satisfy such requirement then PKB could just throw a NotSupportedConfiguration error.
What are your thoughts? @voellm

Hi @meteorfox,
I just wanted to let you know that we haven't dropped this. We're talking about the taxonomy, and we'll get back to you with our thoughts on the best way to describe disks.
Thanks for your patience,
Noah

We at Google have been talking about this more, and we have a proposed design. Anyone who is interested, please comment on this doc: https://docs.google.com/document/d/1ruXqIEg_QAGAD9kpH5rskFVltLtM6gZLqAR67UQg9Es/edit?usp=sharing

The disk types refactor was merged in #599. Closing this for now. Please reopen if there is more to do.


Suggestion for a VS Code extension to include in you config

Clear and concise description of the problem
Writing markdown in VS Code is great for the linting but not so great for the smaller quality of life stuff. I've been switching between VS Code and Obsidian but staying in one editor would be nice.
Suggested solution
I just came across a project called Milkdown and it looks excellent. It turns out there is a VS Code extension using Mikdown for editing rendered markdown with a block style editor and slash commands (aka Notion).
https://marketplace.visualstudio.com/items?itemName=mirone.milkdown
See what you think.
Alternative
No response
Additional context
No response
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guide.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

Ugh IMO you are posting in a wrong place
This repository is to share @antfu's personal vscode preferences, not aimed to be a unified config for everybody


DEV: negative adpative scan init

Codecov Report

Merging #871 into master will decrease coverage by 0.01%.
The diff coverage is 83.33%.

@@            Coverage Diff             @@
##           master     #871      +/-   ##
==========================================
- Coverage   85.32%   85.31%   -0.02%
==========================================
  Files          18       18
  Lines        4975     4978       +3
==========================================
+ Hits         4245     4247       +2
- Misses        730      731       +1

Impacted Files
Coverage Δ

bluesky/plans.py
91.01% <83.33%> (-0.07%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7ae4817...6c3ad83. Read the comment docs.

tests added.

also added another test in relative_adaptive_scan.

OK, great. In order to keep commit history clean, I think it's worth squashing this down to one commit before we merge. Can you squash and force-push?

yep, that's also what I am thinking.

Good to go! Merging.


feat: update schema (multiple units for account)

[x] tests and linter show no problems (npm t)
[x] code is properly formatted (npm run fmt)
[x] update .d.ts typings

Landed in https://github.com/metarhia/metasql/commit/33a12ab171c491281ae3ab6a7287952d0b9ca77f


(bug) CI: The assign/unassign GitHub Action is not working

Observed behaviour
A GitHub Action is used to assign/unassign people to an issue.
https://github.com/thomaspoignant/go-feature-flag/blob/main/.github/workflows/unassign-issue.yml
This GitHub action works fine to assign the issues but is not working well for unassigning.
Expected Behavior
The action should unassign the issue assignment after 10 days.

/assign-me


How can I get HtmlHelper from BootstrapConfig?

Hi everyone.
I'm writing some extensions for my project to simplify my views and to make fluent bootstrap more flexible.

How can I get HtmlHelper from FB Config? As I can see it's not a public property helper.GetConfig().HtmlHelper.
Thanks.

There's extensions to help with this. They're hidden inside a special namespace so that normal consumers don't accidentally see them and get confused. Just add using FluentBootstrap.Mvc.Internals; to the top of your file to bring the extensions into scope and you should be able to write something like helper.GetConfig().GetHtmlHelper() to  get your typed HtmlHelper<TModel>.
As a little side note, the way that HtmlHelper<TModel> stores the model type for the view is the whole reason FluentBootstrap needs all these crazy generics. The underlying FluentBootstrap code would be so much simpler if it weren't for the need to carry that model type with us everywhere.


Building still uses build number(eg build_91) in tags instead of version

Ohh I think you need something like this: https://github.com/FairwindsOps/polaris/blob/master/.circleci/config.yml#L13

I thought so too .. but not sure with the current configuration if we define set-environment we won't have a way to define it as a step in docker_build_push job. Taking a page from https://github.com/FairwindsOps/pluto/blob/master/.circleci/config.yml not sure why this does not have that but builds fine for tags.

Ohh - because you're using the orb, you don't need set_environment_variables.
I don't understand why this would work, but worth a shot


Is there a generic "person status" entity?

I'm wondering whether there is a generic entity that describes a person (whether this person is at home) without using the device tracker.
In my opinion the device_tracker component is one way how to implement that person status, but there can be more ways. For example, there are apps like Geofancy that can send a request to a given url based on geolocation / iBeacon presence.
Having a generic entity having the STATE_HOME / STATE_NOT_HOME states would be the most simple solution since then a POST request to /api/states/<entity_id> could be made.

There is currently no platform for the device tracker component that supports entities to be controlled by events/service calls.
In case of geofancy you are able to send POST requests to URL endpoints. You can have it target the update state API.

So, could the following be a solution?
I could make a non-existing entry in the known_devices.csv for the nmap device tracker like "00:00:00:00:00:00". Since this one will never be detected, I can set its state with POST requests to /api/states/[entity_id].

You don't even have to make a non-existing entry. Just fire away at the API. If the state does not exist it will be created.

Just noticed an issue in the API docs. The JSON body for setting a state is
{
    "state": "home",
    "attributes": {
        "entity_picture": "http://graph.facebook.com/schoutsen/picture"
    }
}

About not creating a non-existing entity - I think an entity is needed to visualize it on the dashboard.

It would be pleasant if the device_sun_light_trigger could also be used with manually-created entities, instead of depending on the device_tracker

You should be able to have it work with the device sun light trigger using the following steps:

Create a group that groups your manual created device tracker entities.
Point the device sun light trigger at your custom group.

Interesting, I'll try it :)

This is working pretty well. The only problem I encountered is that geofency is sending their own POST Body, which is not JSON.
This is causing the API to abort the request and not switch the state.
I have created a pull request (https://github.com/balloob/home-assistant/pull/124), which allows non JSON Body and simply ignores it. The API request still works since the request query parameters are added to the data dictionary.
Do you guys have any ideas of how to implement the following:

Automatically make all entities of the type "people" part of a group without having to configure each one separately in the config

How to set the default state of a group (in this case "not_home"), since if no entity belongs to the group it is just "unknown"

How to have 2 conditions in the automation: people group changes to "home" and sun is "below_horizont"?

KimchaC, in the future, please use the mailing list for these questions.
And about your questions, all of them are currently not possible out of the box. 1 and 3 can be solved easily with the built-in components:
To solve 1, listen for state changes and add each person that is not part of the group to the group
To solve 3, listen for state changes on the people group and wrap your logic in if not sun.is_on(hass):
The example component should help you get started with custom component development.


./bin/kafka-manager does not use ./conf/application.conf

It seems I can't have kafka-manager to use the conf/application.conf file... Have to specify the config options on the commandline using -D...
Is this normal behaviour ?

I see this as well.  The good news is that this works when providing the flags at the command line

you're lucky... once started, kafka-manager does not see any of my 3 brokers even if connected to them (seen using lsof...)... still trying...

Did you try adding a cluster once the application is up?  Kafka manager has its own zookeeper configuration which it uses to store its own configuration.  In order to manage a cluster, you need to goto Cluster menu and Add Cluster and enter the config of the kafka zookeepers and version.

you can use -Dkafka-manager.zkhosts="host:2181"

I realized that without the command line option (-D...) kafka-manager tries to connect with the zkhost specified at the time of sbt clean dist. This error shows it connecting to localhost:49955 the zkhost I gave during compilation:
 [error] o.a.c.ConnectionState - Connection timed out for connection string (<IP_ADDRESS>:49955) and timeout (15000) / elapsed (15031)

I think this is related to https://github.com/yahoo/kafka-manager/issues/16.

@srangwal How can you specify zk urls when sbt clean dist, I follow the README and find no any errors.

you must "Add cluster" at first.


[Regression] injector.BoundKey

Hi,
I have found a small regression in this snippet, which is provided as example on how to use the BoundKey here: https://injector.readthedocs.io/en/latest/api.html#injector.BoundKey
from injector import Injector, BoundKey, InstanceProvider

class A:
    def __init__(self, a, b):
        self.a = a
        self.b = b

InjectedA = BoundKey(A, a=InstanceProvider(1), b=InstanceProvider(2))
injector = Injector()
a = injector.get(InjectedA)
a.a, a.b

It works in 0.15, 0.16 but fails in 0.17 with the following error:
UnknownProvider: couldn't determine provider for InstanceProvider(1) to None
Hope this helps :)
Have a good day !

The fix for this has just been released as part of the version 0.18.1.


update build

update gradle
update gh action
add renovate

Welcome to Codecov :tada:
Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.
Thanks for integrating Codecov - We've got you covered :open_umbrella:


How to use GLSL lark parser on a QScintilla

Hi, nice to meet you.
Today I've discovered this really promising project and although I haven't tested it yet from what I've read I can say it looks very promising to use. Few days ago I was testing this little pyparsing glsl parser and the results I was getting weren't too great for simple piece of code (~0.2-0.3s) so I wanted to give it a shot to lark to see whether I'll be getting faster results.
That said, I've opened this question couple of days ago but since I've discovered lark I was thinking that instead considering using a QsciLexerCPP subclass to address this problem (that is just wrong) I'd write my own QsciLexerCustom subclass using lark, more info about here. But some questions arise to me:

Assuming I've got the lark grammar and built the AST, how would you create the styles with it on a QSciScintilla, could we come up with a little example to illustrate this? Could we come up with a 5-min example showing how to use one of the lark examples on a qscintilla?
I'll want proper glsl syntax highlighting on the qscintilla editor but I'll also want to show autocompletions of function definitions, function declarations, ... , will be that possible to achieve by using lark?
Is there any freenode IRC channel / forum / SO tag to talk/discuss about lark? I'm posting questions here assuming you'll have the tag "help_wanted" :)

Got few more questions but I don't want to make the thread longer, summing up, trying to learn how to use lark to syntax highlight text on a QScintilla component properly.
Thanks in advance!
Bruno

Hi Bruno,
It looks doable with Lark.
In terms of pseudo-code, I presume it might look something like:
def styleText(self, start, end):
        self.startStyling(start)
        text = self.parent().text()[start:end]

        for token in self.lark_parser.lex(text):
            self.setStyling( len(bytearray(token, "utf-8")), self.style.get(token.type, 0))

Where self.style is a dictionary of token_name -> style-number.
Lark has a newgroup: https://groups.google.com/forum/#!forum/lark-parser
We can continue the discussion there.
Good luck,
Erez

Erez,
Thanks for the fast response. I'd like to give it a shot to your previous snippet... But... are you sure that code is correct? Here's how styleText works... Asking you because I'm still trying to wrap around that in my head ;)

I didn't test it, and the docs aren't exactly superb, but it looks about right.

Mmm, i ask you cos couple hours ago i'd created a little example using the json parser included in the examples and the output was just wrong... Im on my phone now so i cant attach you any shot but my first thought was that the styletext overrided function was missing something... I'll read again the docs though

Well, asking me if I'm sure won't get us anywhere. You can post your version of the code with a screenshot and maybe I can spot your mistake.
Btw

Lark has a newgroup: https://groups.google.com/forum/#!forum/lark-parser

We can continue the discussion there.

@erezsh You're absolutely right... forget about my 2 previous messages (cleaning them up now) as they weren't providing any relevant information, they were just noise... just texted them while I was on my phone. Anyway, here's my little test:
import sys
import textwrap
import re

from PyQt5.Qt import *  # noqa

from PyQt5.Qsci import QsciScintilla
from PyQt5.Qsci import QsciLexerCustom

class JsonLark(QsciLexerCustom):

    def __init__(self, parser, parent=None):
        super().__init__(parent)

        # Default settings
        self.setDefaultColor(QColor("#ff000000"))
        self.setDefaultPaper(QColor("#ffffffff"))
        self.setDefaultFont(QFont("Consolas", 14))

        # Styles (color/paper/font)
        styles = [
            ("#ff000000", "#ffffffff", "Consolas", {
             "pointSize": 14, "weight": QFont.Bold}),
            ("#ff7f0000", "#ffffffff", "Consolas", {
             "pointSize": 14, "weight": QFont.Bold}),
            ("#ff0000bf", "#ffffffff", "Consolas", {
             "pointSize": 14, "weight": QFont.Bold}),
            ("#ff007f00", "#ffffffff", "Consolas", {
             "pointSize": 14, "weight": QFont.Bold}),
        ]

        for style, (color, paper, font_family, font_kwargs) in enumerate(styles):
            self.setColor(QColor(color), style)
            self.setPaper(QColor(paper), style)
            self.setFont(QFont(font_family, **font_kwargs), style)

        self.token_styles = {
            "__COLON": 0,
            "__COMMA": 0,
            "__FALSE1": 0,
            "__LBRACE": 3,
            "__LSQB": 0,
            "__NULL2": 0,
            "__RBRACE": 3,
            "__RSQB": 0,
            "__TRUE0": 0,
            "ESCAPED_STRING": 1,
            "SIGNED_NUMBER": 2
        }

        self.parser = parser

    def language(self):
        return "JsonLark"

    def description(self, style):
        return {v: k for k, v in self.token_styles.items()}.get(style, "")

    def styleText(self, start, end):
        self.startStyling(start)
        text = self.parent().text()[start:end]

        for token in self.parser.lex(text):
            self.setStyling(
                len(bytearray(token, "utf-8")),
                self.token_styles.get(token.type, 0)
            )

class Foo(QsciScintilla):

    def __init__(self, parent=None):
        super().__init__(parent)
        parser = lark_parser()
        lexer = JsonLark(parser, self)

        self.setLexer(lexer)

        QShortcut(QKeySequence("Ctrl+7"), self, self.test)

    def test(self):
        import time
        parser = self.lexer().parser
        start = time.time()
        for token in parser.lex(self.text()):
            print(token.type)
        print(time.time() - start)

def lark_parser():
    from lark import Lark, inline_args, Transformer

    json_grammar = r"""
        ?start: value
        ?value: object
              | array
              | string
              | SIGNED_NUMBER      -> number
              | "true"             -> true
              | "false"            -> false
              | "null"             -> null
        array  : "[" [value ("," value)*] "]"
        object : "{" [pair ("," pair)*] "}"
        pair   : string ":" value
        string : ESCAPED_STRING
        %import common.ESCAPED_STRING
        %import common.SIGNED_NUMBER
        %import common.WS
        %ignore WS
    """

    class TreeToJson(Transformer):
        @inline_args
        def string(self, s):
            return s[1:-1].replace('\\"', '"')

        array = list
        pair = tuple
        object = dict
        number = inline_args(float)

        def null(self, _): return None

        def true(self, _): return True

        def false(self, _): return False

    # json_parser = Lark(json_grammar, parser='earley', lexer='standard')
    # def parse(x):
    #     return TreeToJson().transform(json_parser.parse(x))

    return Lark(json_grammar, parser='lalr', transformer=TreeToJson())

def main():
    app = QApplication(sys.argv)
    ex = Foo()
    ex.setText(textwrap.dedent("""\
        {
            "empty_object" : {},
            "empty_array"  : [],
            "booleans"     : { "YES" : true, "NO" : false },
            "numbers"      : [ 0, 1, -2, 3.3, 4.4e5, 6.6e-7 ],
            "strings"      : [ "This", [ "And" , "That", "And a \\"b" ] ],
            "nothing"      : null
        }\
    """))
    ex.resize(800, 600)
    ex.show()
    sys.exit(app.exec_())

if __name__ == "__main__":
    main()

And here's what I mean by broken](https://dl.dropboxusercontent.com/s/bmn9bgh6fav7dqx/2018-05-23_21-25-53.mp4) output. Do you know how to fix the above code?

Lark has a newgroup: https://groups.google.com/forum/#!forum/lark-parser
We can continue the discussion there.

Never used google groups? Is there any benefit with respect to github? Do you mind if we continue this one here till I learn a little bit more about the other one... I do really love the fact github is feature-complete and have such a nice friendly interface to post code, images, videos, closing comments... feels cozy ;D
Yours,
Bruno

Hi Bruno,
It looks like a good start!
I think the reason for the confused coloring, is that setStyling, quite reasonably, needs the whitespace as well!
Let me try to re-write the function, and hopefully I'll get it right, though it's a bit tricky:
def styleText(self, start, end):
    self.startStyling(start)
    text = self.parent().text()[start:end]

    last_pos = 0
    for token in self.parser.lex(text):
        ws_len = token.pos_in_stream - last_pos
        if ws_len:
            self.setStyling(ws_len , 0 )    # whitespace

        token_len = len(bytearray(token, "utf-8"))
        self.setStyling(token_len, self.token_styles.get(token.type, 0))

        last_pos = token.pos_in_stream + token_len

Hopefully that's close enough to get you started.

P.S. You should probably name all the tokens, for example _NULL: "null". That way your code won't break if the auto-numbering changes!

@erezsh Awesome, thank you very much, that function is much better than its predecesor one ;)
The first tests looks ok, I'll be testing further though so maybe I can find some corner cases, the real future editor will have multiselection support & folding so I also need to check the current code already supports that with no problem, also I'll be testing performance on long texts. GLSL grammar will be slightly more complex than json one and the text chunks will be in the range of [0-100kb].
Btw, is there any programmatic way to extract all tokens from a grammar using Lark? Also, I didn't get your last comment about _NULL: "null", could you please elaborate?
One last thing, please feel free to add the above example to your project if you feel that can be useful for other users... Although unfortunately QScintilla niche isn't very big one :) , just look at Stackoverflow how little traffic it has ;D
Again, thanks for your help, only 30min of experience using lark but it already feels like can become a good choice.

I'll be testing performance

Nice, let me know how it turns out. Complexity should be linear, and speed should be relatively close to Python's maximum speed.

is there any programmatic way to extract all tokens from a grammar using Lark?

Yes. It's not an official API, so it might change in the future, but it probably won't.
[t.name for t in lark_parser.lexer.tokens]

I didn't get your last comment

In your code, the NULL token is called __NULL2, which I presume you copied from the resulting parse tree. This is because anonymous terminals in Lark are named as meaningful description if possible + automatic counter. The counter exists to prevent collisions. However, it depends on the order in which the terminals appear in the grammar. If you change the order, you might find it become __NULL3, and so on.

please feel free to add the above example

Sure. If you can provide a minimal example demonstrating this, I'll appreciate it! Open-source is something we do together :)
Good luck, and let me know if you have other questions.

Erez,

Nice, let me know how it turns out. Complexity should be linear, and speed should be relatively close to Python's maximum speed.

that's wonderful news, I've seen your benchmarks about json in the docs and I must to say the results are quite astonishing, awesome job!  Maybe it's better to come up with a more simplified one like this or this? I'd like my editor to be feature-complete (getting function definitions, function declarations, autocomplete)... on the other hand, glsl compiler itself is rock-solid tech and I'd like to have just the minimal code as possible granting me the features. Plus, performance is the highest priority here, the bucket for lexing/syntax_highlighting/autocompletion in 1s should be as small as possible
How hard would be to convert the above official grammar to lark? (Reading the lark docs/examples/tutorial as we speaking ;))

Sure. If you can provide a minimal example demonstrating this, I'll appreciate it! Open-source is something we do together :)

I do like contribute to awesome projects like this ones each time I've got some spare time, although compilers area is not my expertise area (I'm more into the 3d graphics & maths actually :)), as a casual user, I'll be glad to provide more examples like the little 15min qscintilla json example. I think one of the main drivers for a project to become popular is the number/type of examples it provides.
On the PR you can see I've cleaned it up a little bit and now it's become a mcve with a look&feel more similar to SublimeText(yeah, ST is my favourite editor :)), also added the requirements-example.txt as the examples will need PyQt5/QScintilla deps.

Good luck, and let me know if you have other questions.

That's awesome, thank you very much! Actually, right now I was trying to figure out how to add folding support to custom lexers like Json or the future GLSL, one of the good things about subclassing qscilexercpp.cpp was I would be getting all the folding stuff functionality for free but when having a custom lexer from scratch I'm a little bit lost how that can be done :/
For instance, if you wanted folding been enabled when using a QSciLexerCpp subclass you just have to do:
    # http://www.scintilla.org/ScintillaDoc.html#Folding
    self.setFolding(QsciScintilla.BoxedTreeFoldStyle)

    lexer = QsciLexerCPP()
    lexer.setFoldAtElse(True)
    lexer.setFoldComments(True)
    lexer.setFoldCompact(False)
    lexer.setFoldPreprocessor(True)
    self.setLexer(lexer)

and that's it, you've got folding feature for free but with a custom json/glsl lexer... I've got no clue where to start, any idea?
Regards,
Bruno

do you think is a good idea to use such a meaty grammar?
If you only need a lexer, you don't really have to implement any rules, just define the terminals. At the same time, the amount of rules won't affect lexer speed.
Also, many of the rules in this "meaty" grammar are just plain repetitions that can be abstracted away.
For example

compound_statement_no_new_scope :
    LEFT_BRACE RIGHT_BRACE
    LEFT_BRACE statement_list RIGHT_BRACE

Can be:
compound_statement_no_new_scope: "{" statement_list? "}"

You might find it ends up shorter than you expected.
Converting the grammar shouldn't be too hard. Maybe just a bit of a "dirty" work.

I think one of the main drivers for a project to become popular is the number/type of examples it provides.

I completely agree. I'm always happy to accept new examples and/or tutorials.
See my comment on the PR.

was trying to figure out how to add folding

I'm sorry, I never got into qt, so I don't know how it implements folding. But it doesn't sound like Lark will be the barrier.

Now I can create more readable github comments

That's nice! Simple html5, but I admit I didn't consider it.

Converting the grammar shouldn't be too hard. Maybe just a bit of a "dirty" work.

Ok, today later I'll give a shot, couple of questions before adapting:
Is there any difference in terms of performance between grammar1, grammar2 and grammar3?
grammar1 = """
    type_specifier_nonarray : VOID
        | FLOAT
        | DOUBLE
        | INT
        | UINT
        | BOOL
        | VEC2
        | VEC3
        | VEC4
    VOID      : "void"
    FLOAT     : "float"
    DOUBLE    : "double"
    INT       : "int"
    UINT      : "uint"
    BOOL      : "bool"
    VEC2      : "vec2"
    VEC3      : "vec3"
    VEC4      : "vec4"
"""

grammar2 = """
    type_specifier_nonarray : "void"
        | "float"
        | "double"
        | "int"
        | "uint"
        | "bool"
        | "vec2"
        | "vec3"
        | "vec4"
"""

grammar3 = """
    type_specifier_nonarray :
        | "void"
        | "float"
        | "double"
        | "int"
        | "uint"
        | "bool"
        | "vec[234]"
"""

Haven't checked the above examples are valid EBNF grammar though.

See my comment on the PR.

Next time, just leave the PR opened so you'll give me time to read your comments/suggestions/critics and editing the stuff so I don't need to bother you opening more PRs ;)

Nope, all same performance.

Cool, good to know, that means I'll be able to apply succesive transformations to simplify G0 => G1=T(G0) => G2=T(G1) => ... => Gn+1=T(Gn) where Gn will be a much cleaner grammar than G0 but with the same performance. Probably I'll create some script to apply the transformations on top of the grammar coming from the official khronos specs, that will save me future job of adapting more recent versions of GLSL.

Generally speaking, as long as you keep your parser LALR(1), it's going to be very fast no matter what.

Related question

@erezsh Hiya, I was making some little tests before trying to use the official glsl4.60 grammar and I've got stuck with this:
editor_glsl.py

import sys
import textwrap
import re

from PyQt5.Qt import *  # noqa

from PyQt5.Qsci import QsciScintilla
from PyQt5.Qsci import QsciLexerCustom

from lark import Lark, inline_args, Transformer

class LexerGlsl(QsciLexerCustom):

    def __init__(self, parent=None):
        super().__init__(parent)
        self.create_grammar()
        self.create_styles()

    def create_styles(self):
        deeppink = QColor(249, 38, 114)
        khaki = QColor(230, 219, 116)
        mediumpurple = QColor(174, 129, 255)
        mediumturquoise = QColor(81, 217, 205)
        yellowgreen = QColor(166, 226, 46)
        lightcyan = QColor(213, 248, 232)
        darkslategrey = QColor(39, 40, 34)

        styles = {
            0: mediumturquoise,
            1: mediumpurple,
            2: yellowgreen,
            3: deeppink,
            4: khaki,
            5: lightcyan
        }

        for style, color in styles.items():
            self.setColor(color, style)
            self.setPaper(darkslategrey, style)
            self.setFont(self.parent().font(), style)

        self.token_styles = {
            "KEYWORDS": 3
        }

    def create_grammar(self):
        grammar = '''
            start: KEYWORDS+
            KEYWORDS: "atomic_uint" | "attribute"
                | "bool" | "break" | "buffer" | "bvec2" | "bvec3" | "bvec4"
                | "case" | "centroid" | "coherent" | "const" | "continue"
                | "default" | "discard" | "dmat2" | "dmat2x2" | "dmat2x3" | "dmat2x4" | "dmat3" | "dmat3x2" | "dmat3x3" | "dmat3x4" | "dmat4" | "dmat4x2" | "dmat4x3" | "dmat4x4" | "do" | "double" | "dvec2" | "dvec3" | "dvec4"
                | "else"
                | "false" | "flat" | "float" | "for"
                | "highp"
                | "if" | "iimage1D" | "iimage1DArray" | "iimage2D" | "iimage2DArray" | "iimage2DMS" | "iimage2DMSArray" | "iimage2DRect" | "iimage3D" | "iimageBuffer" | "iimageCube" | "iimageCubeArray" | "image1D" | "image1DArray" | "image2D" | "image2DArray" | "image2DMS" | "image2DMSArray" | "image2DRect" | "image3D" | "imageBuffer" | "imageCube" | "imageCubeArray" | "in" | "inout" | "int" | "invariant" | "isampler1D" | "isampler1DArray" | "isampler2D" | "isampler2DArray" | "isampler2DMS" | "isampler2DMSArray" | "isampler2DRect" | "isampler3D" | "isamplerBuffer" | "isamplerCube" | "isamplerCubeArray" | "ivec2" | "ivec3" | "ivec4"
                | "layout" | "lowp"
                | "mat2" | "mat2x2" | "mat2x3" | "mat2x4" | "mat3" | "mat3x2" | "mat3x3" | "mat3x4" | "mat4" | "mat4x2" | "mat4x3" | "mat4x4" | "mediump"
                | "noperspective"
                | "out"
                | "patch" | "precise" | "precision"
                | "readonly" | "restrict" | "return"
                | "sample" | "sampler1D" | "sampler1DArray" | "sampler1DArrayShadow" | "sampler1DShadow" | "sampler2D" | "sampler2DArray" | "sampler2DArrayShadow" | "sampler2DMS" | "sampler2DMSArray" | "sampler2DRect" | "sampler2DRectShadow" | "sampler2DShadow" | "sampler3D" | "samplerBuffer" | "samplerCube" | "samplerCubeArray" | "samplerCubeArrayShadow" | "samplerCubeShadow" | "shared" | "smooth" | "struct" | "subroutine" | "switch"
                | "true"
                | "uimage1D" | "uimage1DArray" | "uimage2D" | "uimage2DArray" | "uimage2DMS" | "uimage2DMSArray" | "uimage2DRect" | "uimage3D" | "uimageBuffer" | "uimageCube" | "uimageCubeArray" | "uint" | "uniform" | "usampler1D" | "usampler1DArray" | "usampler2D" | "usampler2DArray" | "usampler2DMS" | "usampler2DMSArray" | "usampler2DRect" | "usampler3D" | "usamplerBuffer" | "usamplerCube" | "usamplerCubeArray" | "uvec2" | "uvec3" | "uvec4"
                | "varying" | "vec2" | "vec3" | "vec4" | "void" | "volatile"
                | "while" | "writeonly"
            %import common.WS
            %ignore WS
        '''

        # grammar = '''
        #     ?start: keywords
        #     KEYWORDS: "in"
        #         | "out"
        #         | "vec2"
        #         | "vec3"
        #         | "vec4"
        #         | "void"
        #     %import common.ESCAPED_STRING
        #     %import common.SIGNED_NUMBER
        #     %import common.WS
        #     %ignore WS
        # '''

        self.lark = Lark(grammar, parser='lalr')
        # All tokens: print([t.name for t in self.lark.parser.lexer.tokens])

    def defaultPaper(self, style):
        return QColor(39, 40, 34)

    def language(self):
        return "Json"

    def description(self, style):
        return {v: k for k, v in self.token_styles.items()}.get(style, "")

    def styleText(self, start, end):
        self.startStyling(start)
        text = self.parent().text()[start:end]
        last_pos = 0

        try:
            for token in self.lark.lex(text):
                ws_len = token.pos_in_stream - last_pos
                if ws_len:
                    self.setStyling(ws_len, 0)    # whitespace

                token_len = len(bytearray(token, "utf-8"))
                self.setStyling(
                    token_len, self.token_styles.get(token.type, 0))

                last_pos = token.pos_in_stream + token_len
        except Exception as e:
            print(e)

class EditorAll(QsciScintilla):

    def __init__(self, parent=None):
        super().__init__(parent)

        # Set font defaults
        font = QFont()
        font.setFamily('Consolas')
        font.setFixedPitch(True)
        font.setPointSize(8)
        font.setBold(True)
        self.setFont(font)

        # Set folding defaults (http://www.scintilla.org/ScintillaDoc.html#Folding)
        self.setFolding(QsciScintilla.CircledFoldStyle)

        # Set margin defaults
        fontmetrics = QFontMetrics(font)
        self.setMarginsFont(font)
        self.setMarginWidth(0, fontmetrics.width("000") + 6)
        self.setMarginLineNumbers(0, True)
        self.setMarginsForegroundColor(QColor(128, 128, 128))
        self.setMarginsBackgroundColor(QColor(39, 40, 34))
        self.setMarginType(1, self.SymbolMargin)
        self.setMarginWidth(1, 12)

        # Set indentation defaults
        self.setIndentationsUseTabs(False)
        self.setIndentationWidth(4)
        self.setBackspaceUnindents(True)
        self.setIndentationGuides(True)

        # self.setFolding(QsciScintilla.CircledFoldStyle)

        # Set caret defaults
        self.setCaretForegroundColor(QColor(247, 247, 241))
        self.setCaretWidth(2)

        # Set selection color defaults
        self.setSelectionBackgroundColor(QColor(61, 61, 52))
        self.resetSelectionForegroundColor()

        # Set multiselection defaults
        self.SendScintilla(QsciScintilla.SCI_SETMULTIPLESELECTION, True)
        self.SendScintilla(QsciScintilla.SCI_SETMULTIPASTE, 1)
        self.SendScintilla(
            QsciScintilla.SCI_SETADDITIONALSELECTIONTYPING, True)

        lexer = LexerGlsl(self)
        self.setLexer(lexer)

def main():
    app = QApplication(sys.argv)
    ex = EditorAll()
    ex.setWindowTitle(__file__)
    # ex.setText(textwrap.dedent("""\
    #     void mainImage( out vec4 fragColor, in vec2 fragCoord )
    #     {
    #         // Normalized pixel coordinates (from 0 to 1)
    #         vec2 uv = fragCoord/iResolution.xy;

    #         // Time varying pixel color
    #         vec3 col = 0.5 + 0.5*cos(iTime+uv.xyx+vec3(0,2,4));

    #         // Output to screen
    #         fragColor = vec4(col,1.0);
    #     }\
    # """))

    ex.setText("atomic_uint attribute bool break buffer bvec2 bvec3 bvec4 case centroid coherent const continue default discard dmat2 dmat2x2 dmat2x3 dmat2x4 dmat3 dmat3x2 dmat3x3 dmat3x4 dmat4 dmat4x2 dmat4x3 dmat4x4 do double dvec2 dvec3 dvec4 else false flat float for highp if iimage1D iimage1DArray iimage2D iimage2DArray iimage2DMS iimage2DMSArray iimage2DRect iimage3D iimageBuffer iimageCube iimageCubeArray image1D image1DArray image2D image2DArray image2DMS image2DMSArray image2DRect image3D imageBuffer imageCube imageCubeArray in inout int invariant isampler1D isampler1DArray isampler2D isampler2DArray isampler2DMS isampler2DMSArray isampler2DRect isampler3D isamplerBuffer isamplerCube isamplerCubeArray ivec2 ivec3 ivec4 layout lowp mat2 mat2x2 mat2x3 mat2x4 mat3 mat3x2 mat3x3 mat3x4 mat4 mat4x2 mat4x3 mat4x4 mediump noperspective out patch precise precision readonly restrict return sample sampler1D sampler1DArray sampler1DArrayShadow sampler1DShadow sampler2D sampler2DArray sampler2DArrayShadow sampler2DMS sampler2DMSArray sampler2DRect sampler2DRectShadow sampler2DShadow sampler3D samplerBuffer samplerCube samplerCubeArray samplerCubeArrayShadow samplerCubeShadow shared smooth struct subroutine switch true uimage1D uimage1DArray uimage2D uimage2DArray uimage2DMS uimage2DMSArray uimage2DRect uimage3D uimageBuffer uimageCube uimageCubeArray uint uniform usampler1D usampler1DArray usampler2D usampler2DArray usampler2DMS usampler2DMSArray usampler2DRect usampler3D usamplerBuffer usamplerCube usamplerCubeArray uvec2 uvec3 uvec4 varying vec2 vec3 vec4 void volatile while writeonly")

    ex.resize(800, 600)
    ex.show()
    sys.exit(app.exec_())

if __name__ == "__main__":
    main()

This simple test is intended to help to understand how to apply a particular style on a bunch of keywords separated by spaces.
Problem is you run it you'll get this. Guess it's related to token priorities?
What's the proper way to fix cases like these ones? The first idea it comes to my mind would be using regexes like mat[234](x[234])?/.?    but I'd prefer to keep a list of separated tokens so I can create the lark grammar automatically using some scripts.
Please advice.
B.

Lark knows how to resolve the priority automatically in simple cases such as this. However, it won't mess with the internal structure of your terminal patterns.
In other words, by defining them as KEYWORDS, you're telling Lark to create a single terminals with many options, and the order you defined them in is preserved.
You should change the definition to keywords, which means it's a rule with many different terminals, and everything should work as expected.
            start: keywords+
            keywords: "atomic_uint" | "attribute" | ...

Mmmm, slowly slowly catchy the monkey :D . It's taking me a while to grasp all the concepts so I can start adapting the official glsl ebnf grammar to lark, I must to admit all this stuff isn't easy... plus, it's been like 10 years since I took all these courses about compilers, automatas, FSM ... nowadays, I forgot all of it, I don't remember not even one single theorem from the zillion existing ones ;D
Btw, do you think it'd be very tricky to give support to something similar to this syntax:
grammar = '''
    start: jump_statement+
    jump_statement :
        | CONTINUE SEMICOLON
        | BREAK SEMICOLON
        | RETURN SEMICOLON
        | DISCARD SEMICOLON // Fragment shader only
    CONTINUE : "continue"
    SEMICOLON : ";"
    BREAK : "break"
    RETURN : "return"
    DISCARD : "discard"
    %import common.WS
    %ignore WS
'''

instead the current syntax:
grammar = '''
    start: jump_statement+
    jump_statement : CONTINUE SEMICOLON
        | BREAK SEMICOLON
        | RETURN SEMICOLON
        | DISCARD SEMICOLON // Fragment shader only
    CONTINUE : "continue"
    SEMICOLON : ";"
    BREAK : "break"
    RETURN : "return"
    DISCARD : "discard"
    %import common.WS
    %ignore WS
'''

Rationale: That way the EBNF grammar will become much more readable, not only that, you'd be able to fold long grammars easily on your favourite text editor, example here. I've noticed the EBNF syntax used on the glsl specs consider indented lines instead | , although I guess that'd be even trickier to implement? What do you think about the above suggestion? very ugly? or maybe something like:
primary_expression :
    variable_identifier
    | INTCONSTANT
    | UINTCONSTANT
    | FLOATCONSTANT
    | BOOLCONSTANT
    | DOUBLECONSTANT
    | LEFT_PAREN expression RIGHT_PAREN

mmm, if you ask me I think the cleanest one is the used by the glsl docs, I think the OR operator just add verbosity... but I guess there are good reasons to be used.
Anyway, just a little suggestion.

Interesting, the Antlr4 syntax is also quite clean, example:
variable_identifier :
    : IDENTIFIER
    ;
primary_expression :
    : variable_identifier
    | INTCONSTANT
    | UINTCONSTANT
    | FLOATCONSTANT
    | BOOLCONSTANT
    | DOUBLECONSTANT
    | LEFT_PAREN expression RIGHT_PAREN
    ;
postfix_expression :
    : primary_expression
    | postfix_expression LEFT_BRACKET integer_expression RIGHT_BRACKET
    | function_call
    | postfix_expression DOT FIELD_SELECTION
    | postfix_expression INC_OP
    | postfix_expression DEC_OP
    ;

@erezsh Could you please explain me how to implement the below terminals?
BOOLCONSTANT: like true or false, the value itself is side data, same for all the following
DOUBLECONSTANT: like 3.92e+2LF
FIELD_SELECTION: like structure.member
FLOATCONSTANT: like 3.14159
IDENTIFIER: a name, a variable, etc., in the source code, like i or foo
INTCONSTANT: like 42
TYPE_NAME: like the second S in struct S {...}; S foo;
UINTCONSTANT: like 42u

I've got a rough idea about how the regexes of {BOOLCONSTANT, DOUBLECONSTANT, FLOATCONSTANT, INTCONSTANT, UINTCONSTANT} could look like but not clue about {FIELD_SELECTION, IDENTIFIER, TYPE_NAME} ones
Once I add these ones I'll be able to start testing the whole grammar... once is well tested and you've pushed this one I'll make a PR with a new example syntax highlighting glsl so we can close this one.
Just for the record, yesterday I was having some issues testing your above method with my previous example:
start: keywords+
keywords: "atomic_uint" | "attribute" | ...

I hadn't figured out what was wrong at that time... it was a test example but good to know what was wrong though :/

Sounds like IDENTIFIER and TYPE_NAME are the same terminal.
Meanwhile,
FIELD_SELECTION: IDENTIFIER "." IDENTIFIER

No?
Or maybe it's better as a rule.


[sql] Add more keywords to the default highlighter

VSCode Version: 1.10.2
OS Version: Windows 10

Steps to Reproduce:

The default SQL highlighter does not see some of the keywords.

duplicate of https://github.com/textmate/sql.tmbundle/issues/21


Extract kubernetes username within each exec log

Is there an existing issue for this?

[X] I have searched the existing issues

Is your feature request related to a problem?
No response
Describe the feature you would like
I want to get kubernetes (or openshift) username of user that executes commands in a pod. How can I do this?
Describe your proposed solution
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

The discussion around #2030 may be relevant.

Sorry nobody responded directly here, the same discussion happened in another issue later:

https://github.com/cilium/tetragon/issues/2015

I will close this one to concentrate on things where we have the most context on this, it's not resolved though!


Remove border left on warning panels

What is the context of this PR?

Noticed this issue with footer warning panels. This PR will make sure the unneeded margin left is removed. This will also make sure warning panels align with the edge of the page.
How to review

Check footer warning looks as it should without the border left
Check that warning panels no longer have a border left and align with edge of page

 Remove border left on warning panels


Heatmap with time as the x-axis

Question: Is there a way to use actual x and y axis values instead of indexes
ScottPlot Version: 5.0.34
I have been using the NumericAutomatic tick generators which is great, but the issue is that I have combined files that are at potentially at different time sample rates and time gaps between the groups of data.   With a scatter plot this is not an issue but with a heatmap that uses index values this is an issue.  What I am having to do is insert null records to fill in the timeline gaps so that the index values can be correlated with the actual time values.
I would be great if there was a scatter plot form of a heatmap.  Am I missing something?

I have the same question.


Empty data array causes crash

I know it is not correct to use empty data array but I just hope the components can be more robust.

Sure! Totally agree:)

It will be best if there is an option to display something indicating "no data available" rather than an blank board.

Good idea, I was just thinking about how to signal missing data.

For now fixed the crash by supplying a default dataset to the chart. Might not be the best solution but better than crashing, until I have time to think of something better.

Currently I changed your code so that when the data is null or undefined, there will be a spinning animation and a text showing "loading data...", and when the data is an empty array, there will be a text showing "no data available" and the x-axis and y-axis will still be drawn if given. I think this can be set as the default behavior unless user specifies the element for it when data is not a valid data array.

Can you send a pullreq or something? I would integrate that :)

No, I only modified LineChart (since I only use that). Other components may require the same respect before sending a pr.

True, anyhow how did you implement the spinner?

Very easy, it is done by setting background-image to a spinning gif file using css.

Oh well, lol :)


Streaming seg does not support Patch updates

Issue in ./help/segmentation/ui/streaming-segmentation.md
"Streaming segmentation works on all data that was ingested using a streaming source. Data ingested using a batch-based source will be evaluated nightly, even if it qualifies for streaming segmentation."
We have discovered that streaming segmentation does not currently support PATCH updates on the profile. So it would be helpful to have this information added to the documentation.

Thank you for submitting this issue. We will review the documentation and make the necessary updates as quickly as possible. Tracking internally in PLAT-169451.

Closing this issue based on documentation team's assessment.


Distinguish ErrorCodeManifestBlobUnknown message

Both "ErrorCodeManifestBlobUnknown" and "ErrorCodeBlobUnknown" have the same message (which gets logged as error.code field), making it hard to distinguish.
We're seeing error.code="blob unknown" in the logs and can't distinguish which one is which.

Current coverage is 51.34% (diff: 100%)

Merging #2050 into master will decrease coverage by 10.09%

@@             master      #2050   diff @@
==========================================
  Files           126        126
  Lines         11299      11299
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
- Hits           6942       5801   -1141
- Misses         3469       4751   +1282
+ Partials        888        747    -141

Powered by Codecov. Last update ac0493d...8b3fee0

Does this need API docs regeneration?

Great question, I'll take a look at this later.

This is no longer necessary.


Refactor: split up feature model anomaly solver into different anomaly types

the only real overhead of this approach is the additional meta solver, which is valid and perhaps even desired.

We might even be able to get rid of that. We could implement an  abstract SingletonProblemSolver extends ProblemSolver implements MetaSolver that implements the MetaSolver methods automatically. This could be used for problems with only a single solver.

We might even be able to get rid of that. We could implement an  abstract SingletonProblemSolver extends ProblemSolver implements MetaSolver that implements the MetaSolver methods automatically. This could be used for problems with only a single solver.

I'd rather keep it as it is, as a SingletonProblemSolver kind of goes against the point of a MetaSolver which should handle multiple solvers. We might not want to create another solver for something like a dead features anomaly in feature models, but at least we support it.

I've fixed toolbox-web to use this new route scheme in https://github.com/ProvideQ/toolbox-web/pull/25

I've finally gotten to revising this PR now :sweat_smile:
If this works for you now, please merge this PR into refactor/controllers first so the tests over there pass as well. Then we can merge #39 as well :)


Using Mailchimp for newsletters

Maintaining Subscriber's list - We already have a mailchimp account and existing subscribers imported to a list.
Adding a popup window on website for new subscribers + Making sure all new let's DevOps slack group members are automatically added to that list.

@onkarkundargi Can you please work with @ramitsurana to make sure this gets completed in time?


Improve test coverage for complex function chosen by Elias

parseLine<EMAIL_ADDRESS>

Full coverage can already be accomplished by running on different types of systems. No need to improve coverage