[BEAM-5919] Exclude META-INF from validation of shaded jar files

The rule used to match a "class" file added by Java 9 that is put underneath META-INF

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.


internal hack because of old systemd

just a quick hack so that we get splay in systemd timers without   'RandomizedDelaySec' feature

verified working:
/opt/chef/embedded/bin/ruby --disable-gems /usr/bin/chef-client -c /etc/chef/client.rb -s 300


scons: Add an option specifying the path to mold linker binary

To use mold linker with gcc of version older than 12.1.0, the user has to pass the -B option to specify where the linker is. [1]
Currently in gem5, scons only looks for the mold binary at conventional places, such as /usr/libexec/mold and /usr/local/libexec/mold. There's no option to manually specify the path to the linker.
gcc-12 and mold are not widely available on older systems. Having an option to manually input the linker path allows user to build and use mold without sudo permission.
[1] https://github.com/rui314/mold#how-to-use
Change-Id: Ifb2366a0c2342bf4e7207df8db6196e14184a9d4

To use mold linker with gcc of version older than 12.1.0, the user has to pass the -B option to specify where the linker is. [1]

Why? I don't get why the user must pass the -B flag to enable us to use gcc versions >12.1.0. We already pass the -B flag to GCC for versions <12.1.0. What's so special about the user doing it? Mold surely installs in the same place regardless as to what GCC version it's going be linked into.
Also, this path doesn't remove the check that GCC must be <12.1.0, so this patch doesn't even enable gcc versions >12.1 with mold.

Currently in gem5, scons only looks for the mold binary at conventional places, such as /usr/libexec/mold and /usr/local/libexec/mold. There's no option to manually specify the path to the linker.

Fair... but what does this have to do with gcc versions >12.1.0?

gcc-12 and mold are not widely available on older systems.

What do you mean not available? If they are not available on older systems, why do we care? We can't help users who can't get mold on their system and in those cases where they can't use gcc 12, we're not fixing anything here, right? This patch is to enable mold with gcc-12.

Having an option to manually input the linker path allows user to build and use mold without sudo permission.

Ok I think I understand this. I just confused as I read the last part to mean "users need sudo to build mold". Is the following a fair reword?: "Having an option to manually input the mold linker path allows users to link towards built mold instances anywhere on the system, not just the default locations in /usr where the may not have permission to install mold install (i.e., no sudo permissions).
If so, this is fair case.

[1] https://github.com/rui314/mold#how-to-use
Change-Id: Ifb2366a0c2342bf4e7207df8db6196e14184a9d4
already pa

To use mold linker with gcc of version older than 12.1.0, the user has to pass the -B option to specify where the linker is. [1]
Why? I don't get why the user must pass the -B flag to enable us to use gcc versions >12.1.0.

The context is, when using gcc, user must pass the path to the mold binary the -B flag or -fuse-ld to gcc to use mold with gcc. Prior to 12.1.0, gcc doesn't know what mold is.

We already pass the -B flag to GCC for versions <12.1.0.

The existing -B flags don't work for all systems. Those work nicely if you have mold installed at the default location, but it doesn't work when you have the linker somewhere else.

Fair... but what does this have to do with gcc versions >12.1.0?

Nothing. The mold-path option is only evaluated when the gcc version is < 12.1.0.

Having an option to manually input the mold linker path allows users to ~link towards~ use built mold instances anywhere on the system, not just the default locations in /usr where [they] may not have permission to install mold (i.e., no sudo permissions).

I believe this is better. Will update.

After some testing, I found the following,

According to https://gcc.gnu.org/onlinedocs/gcc-9.4.0/gcc/Directory-Options.html#index-B,

For each subprogram to be run, the compiler driver first tries the -B prefix, if any. If that name is not found, or if -B is not specified, the driver tries two standard prefixes, /usr/lib/gcc/ and /usr/local/lib/gcc/. If neither of those results in a file name that is found, the unmodified program name is searched for using the directories specified in your PATH environment variable.

This basically means you can pass anything to "-B $PREFIX" and gcc will fall back to ld if "$PREFIX/ld" does not exist.
So, it is possible for gcc to pass the current test for mold, isdir(somepath) and conf.CheckLinkFlag("-B"+somepath), and gcc will still use ld.

Also, mold will install ld.mold to the installation place, not ld. It seems that my version of gcc does not take ld.mold into account even though I added -B/somepath where /somepath/ld.mold exists.

So, I guess the new check for mold should be,
if gcc_version < 12.1.0 and \
  (isdir(mold_path) and isfile(mold_path+"/ld") and conf.CheckLinkFlag("-B"+mold_path)

Could anyone explain how to actually use this with scons?  Building mold from source installs in /usr/local/bin, but if I use --mold-path=/usr/local/bin it does not work.  It used to work fine with gcc 9.4 for me, so I will revert this change locally until there is some example of how to use this new option with scons.

I'm surprised that it worked for g++ 9.4 without having the path to mold direct because, prior to this, scons only checks for mold at 2 places,
- /usr/local/libexec/mold
- /usr/libexec/mold

I'm not opposed to reverting this though.


polkadot v0.9.26

@clangenb  so we stay with the current commit, right?

Then we would define this order as the new standard in our repositories?
optional
default-features
features
git
branch


Inconsistent enum types between Grpc/Rest Connection implementations

Note: this only affects Pubsub and Logging as they are currently the only clients that have both a Rest and Grpc implementation (for now).
For enums, the REST API returns a capitalized string value, while gRPC returns an integer. Example: logging client metrics:
<?php
require_once 'vendor/autoload.php';
$client = new Google\Cloud\Logging\LoggingClient();
// assuming you've already created this metric at some point.
$metric = $client->metric('test-metric');
var_dump($metric->info()['metricDescriptor']['metricKind']);

Without grpc extension:
string(5) "DELTA"

With grpc extension:
int(2)

If using the REST transport with the Gapic client, the response is also int(2).
This problem should go away when we consolidate an only use the Gapic client.

Just an update on this. We are in the process of consolidating the 2 layers and will likely use the GAPIC client only, even for REST transports.

We released v2 for Pubsub last week and with this release this issue should be fixed.
The responses that the client receives will now be consistent agnostic to the transport used.
We are aiming to do the same for a few more handwrittens within the next month including Logging.
Hence, closing the issue.
Thanks again for reporting this @chingor13 :)


[DO NOT MANUAL MERGE] Release Qualification

Trigger release qualification jobs
none

@silverdev: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/daily-e2e-cluster_wide-auth-default.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-cluster_wide-auth-default

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@silverdev: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/daily-e2e-cluster_wide-auth-default.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-cluster_wide-auth-default

prow/daily-e2e-cluster_wide-auth.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-cluster_wide-auth

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@silverdev: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/daily-e2e-cluster_wide-auth-default.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-cluster_wide-auth-default

prow/daily-e2e-cluster_wide-auth.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-cluster_wide-auth

prow/daily-e2e-rbac-auth-default.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-rbac-auth-default

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@silverdev: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/daily-e2e-cluster_wide-auth-default.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-cluster_wide-auth-default

prow/daily-e2e-cluster_wide-auth.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-cluster_wide-auth

prow/daily-e2e-rbac-auth-default.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-rbac-auth-default

prow/daily-e2e-rbac-no_auth-default.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-rbac-no_auth-default

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@silverdev: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/daily-e2e-cluster_wide-auth-default.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-cluster_wide-auth-default

prow/daily-e2e-cluster_wide-auth.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-cluster_wide-auth

prow/daily-e2e-rbac-auth-default.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-rbac-auth-default

prow/daily-e2e-rbac-no_auth-default.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-rbac-no_auth-default

prow/daily-e2e-rbac-no_auth.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-rbac-no_auth

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@silverdev: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/daily-e2e-cluster_wide-auth-default.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-cluster_wide-auth-default

prow/daily-e2e-cluster_wide-auth.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-cluster_wide-auth

prow/daily-e2e-rbac-auth-default.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-rbac-auth-default

prow/daily-e2e-rbac-no_auth-default.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-rbac-no_auth-default

prow/daily-e2e-rbac-no_auth.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-rbac-no_auth

prow/daily-e2e-rbac-auth.sh
4e0f380c2c8a065b78350e294d7e2c77439cee9c
link
/test daily-e2e-rbac-auth

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


Forget items from leaves of tree upwards

When the chrome extension starts forgetting items due to the predefined limit, it often starts with the top level parent stories that provide structure to the rest of what's going on. This usually results in nonsensical, confusing output.
When stuff is being forgotten, it should start with the log items from the leaf stories (say, an even selection of the oldest log items from every leaf story). Only once a story has no more children should it be deleted.

Hi, and thanks for the report!
Could you maybe share some steps to reproduce, or maybe screenshots of what you mean? Even better: maybe you could consider adding a failing test to the suite? In principle (see this test), the forgetting algorithm should work just as you describe…


Diff pr

Description:

Affected area:

New Functionality      [ ]
Documentation          [ ]
Install                [ ]
Control Plane          [ ]
CLI Tool               [ ]
Certificate Management [ ]
Networking             [ ]
Metrics                [ ]
SMI Policy             [ ]
Security               [ ]
Tests                  [ ]
CI System              [ ]
Performance            [ ]
Other                  [ ]

Please answer the following questions with yes/no.

Does this change contain code from or inspired by another project? If so, did you notify the maintainers and provide attribution?

Codecov Report

Merging #58 (3dcd077) into main (2581cbd) will increase coverage by 0.19%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main      #58      +/-   ##
==========================================
+ Coverage   59.22%   59.42%   +0.19%
==========================================
  Files         156      156
  Lines        7054     7054
==========================================
+ Hits         4178     4192      +14
+ Misses       2860     2846      -14
  Partials       16       16

Flag
Coverage Δ

unittests
59.42% <ø> (+0.19%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

cmd/cli/dashboard.go
0.00% <ø> (ø)

pkg/configurator/validating_webhook.go
80.00% <0.00%> (+9.65%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2581cbd...3dcd077. Read the comment docs.


No JSON content found. Path '', line 0, position 0.

Hello,
great script! I try to install this script on a fresh installed Debian 10. Installation (with user eco) works great, but the server doesnt start:
[13:12:52.847] [1] [ Info] [Eco] >>> START EcoServer, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=null
[13:12:52.859] [1] [ Info] [Eco] Eco Server <IP_ADDRESS> beta

[13:12:52.934] [1] [ Info] [Eco] Server Initialization...
[13:12:52.934] [1] [ Info] [Eco] Starting LocalizationPlugin...
[13:12:53.235] [1] [ Info] [Eco] Starting ModKitPlugin...
[13:12:53.236] [1] [ Info] [Eco] Loading mods...
[13:12:53.280] [1] [ Info] [Eco] Compiling mods...
[13:12:54.879] [1] [ Info] [Eco] Loading Eco.IntegrationTests...

[13:12:54.886] [1] [ Info] [Eco] Loading mods finished in 1.649,3ms

[13:12:54.886] [1] [ Info] [Eco] Starting NetworkManager...
[13:12:54.891] [7] [ Info] [Eco] Initializing Simulation Types...
[13:12:55.216] [9] [ Info] [Eco] Starting BackupPlugin...
[13:12:55.223] [9] [ Info] [Eco] Starting StorageManager...
[13:12:55.235] [8] [ Info] [Eco] Starting WorldGeneratorPlugin...
[13:12:55.360] [5] [ Info] [Eco] Starting GameplayStatsPlugin...
[13:12:55.811] [7] [ Info] [Eco] Initializing items...
[13:13:01.071] [9] [ Info] [Eco] Starting PausePlugin...
[13:13:03.706] [7] [ Info] [Eco] Initializing skills...
[13:13:03.849] [9] [ Info] [Eco] Starting StatDatabase...
[13:13:03.849] [6] [ Info] [Eco] Starting WorldPlugin...
[13:13:03.896] [6] [ Info] [Eco] Initializing world...
[13:13:03.901] [6] [ Info] [Eco] Starting TimePlugin...
[13:13:03.901] [6] [ Info] [Eco] Starting ReputationPlugin...
[13:13:03.901] [6] [ Info] [Eco] Starting PlayerHousingManager...
[13:13:03.904] [6] [ Info] [Eco] Starting UserManager...
[13:13:03.927] [6] [ Info] [Eco] Initializing inventories...
[13:13:03.961] [6] [ Info] [Eco] Starting EcoObjectManager...
[13:13:03.961] [1] [ Info] [Eco] Server Initialization finished in 11.028,8ms

[13:13:03.968] [1] [ Info] [Eco] Failed to load server, Exception was Exception: JsonSerializationException
Message:No JSON content found. Path '', line 0, position 0.
Source:Newtonsoft.Json

Newtonsoft.Json.JsonSerializationException: No JSON content found. Path '', line 0, position 0.
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Populate (Newtonsoft.Json.JsonReader reader, System.Object target) [0x00037] in <b1d2c1d6f4dd47bea2964bdd6464f67d>:0
  at Newtonsoft.Json.JsonSerializer.PopulateInternal (Newtonsoft.Json.JsonReader reader, System.Object target) [0x00051] in <b1d2c1d6f4dd47bea2964bdd6464f67d>:0
  at Newtonsoft.Json.JsonSerializer.Populate (Newtonsoft.Json.JsonReader reader, System.Object target) [0x00000] in <b1d2c1d6f4dd47bea2964bdd6464f67d>:0
  at Newtonsoft.Json.JsonConvert.PopulateObject (System.String value, System.Object target, Newtonsoft.Json.JsonSerializerSettings settings) [0x00013] in <b1d2c1d6f4dd47bea2964bdd6464f67d>:0
  at Eco.Core.Serialization.SerializationUtils.PopulateObjectWithJson (System.String json, System.Object instance) [0x00009] in <9dd284aca03c49598f63e471517bb8c1>:0
  at Eco.Core.Plugins.PluginConfig`1[T].Load (System.String filename) [0x00049] in <9dd284aca03c49598f63e471517bb8c1>:0
  at Eco.Core.Plugins.PluginConfig`1[T]..ctor (System.String name, System.Boolean preserveReferences, System.Boolean buildConfigProperties) [0x0001b] in <9dd284aca03c49598f63e471517bb8c1>:0
  at Eco.WorldGenerator.WorldGeneratorPlugin..ctor () [0x00006] in <ef9b11cea30c46f9bf63f068ca18019e>:0
  at (wrapper managed-to-native) System.Reflection.RuntimeConstructorInfo.InternalInvoke(System.Reflection.RuntimeConstructorInfo,object,object[],System.Exception&)
  at System.Reflection.RuntimeConstructorInfo.InternalInvoke (System.Object obj, System.Object[] parameters, System.Boolean wrapExceptions) [0x00005] in <7b90a8780ac4414295b539b19eea7eea>:0

root@eco ~ # ps ax|grep -i mono
  502 ?        Sl     0:00 /usr/bin/mono /usr/lib/mono/4.5/xsp4.exe --port 8084 --address <IP_ADDRESS> --appconfigdir /etc/xsp4 --nonstop
24951 pts/8    S+     0:00 grep -i mono
root@eco ~ #

on every startup, i got this error. Any ideas?
If i start the server without ELSM and screen:
eco@eco:/opt/ELSM/Server/TestServer$ mono EcoServer32.exe -nogui
[01:34:54] Eco Server <IP_ADDRESS> beta
[01:34:54] Server Initialization...
[01:34:54] Starting LocalizationPlugin...
[01:34:55] Starting ModKitPlugin...       Finished in 656,5ms
[01:34:55] Loading mods...
[01:34:55] Compiling mods...
[01:34:59] Loading Eco.IntegrationTests....418,3ms
[01:34:59] Loading mods finished in 3.527,1ms
[01:34:59] Starting NetworkManager... nished in 3.528,8ms
[01:34:59] Initializing Simulation Types...
[01:34:59] Starting WorldGeneratorPlugin... ed in 35,6ms
[01:34:59] Starting GameplayStatsPlugin...
[01:34:59] Starting BackupPlugin...        Finished in 0,1ms
[01:34:59] Starting StorageManager... nished in 22,5ms
[01:35:02] Initializing items...            Finished in 3.005,1ms
[01:35:09] Starting PausePlugin...    Finished in 10.272,1ms
[01:35:12] Initializing skills... inished in 10.809,4ms
[01:35:13] Starting StatDatabase... nished in 171,4ms
[01:35:13] Starting WorldPlugin...
[01:35:13] Initializing world...    Finished in 15,3ms
[01:35:13] Starting TimePlugin... iFinished in 52,1ms
[01:35:13] Starting ReputationPlugin... ed in 0,8ms
[01:35:13] Starting PlayerHousingManager... shed in 1,3ms
[01:35:13] Starting UserManager...          Finished in 0,7ms
[01:35:13] Initializing inventories...
[01:35:13] Starting EcoObjectManager... hed in 54,2ms,9ms
[01:35:13] Server Initialization finished in 18.297,5msms
[01:35:13] Failed to load server, Exception was Exception: JsonSerializationException
Message:No JSON content found. Path '', line 0, position 0.
Source:Newtonsoft.Json

Newtonsoft.Json.JsonSerializationException: No JSON content found. Path '', line 0, position 0.
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Populate (Newtonsoft.Json.JsonReader reader, System.Object target) [0x00037] in <b1d2c1d6f4dd47bea2964bdd6464f67d>:0
  at Newtonsoft.Json.JsonSerializer.PopulateInternal (Newtonsoft.Json.JsonReader reader, System.Object target) [0x00051] in <b1d2c1d6f4dd47bea2964bdd6464f67d>:0
  at Newtonsoft.Json.JsonSerializer.Populate (Newtonsoft.Json.JsonReader reader, System.Object target) [0x00000] in <b1d2c1d6f4dd47bea2964bdd6464f67d>:0
  at Newtonsoft.Json.JsonConvert.PopulateObject (System.String value, System.Object target, Newtonsoft.Json.JsonSerializerSettings settings) [0x00013] in <b1d2c1d6f4dd47bea2964bdd6464f67d>:0
  at Eco.Core.Serialization.SerializationUtils.PopulateObjectWithJson (System.String json, System.Object instance) [0x00009] in <9dd284aca03c49598f63e471517bb8c1>:0
  at Eco.Core.Plugins.PluginConfig`1[T].Load (System.String filename) [0x00049] in <9dd284aca03c49598f63e471517bb8c1>:0
  at Eco.Core.Plugins.PluginConfig`1[T]..ctor (System.String name, System.Boolean preserveReferences, System.Boolean buildConfigProperties) [0x0001b] in <9dd284aca03c49598f63e471517bb8c1>:0
  at Eco.WorldGenerator.WorldGeneratorPlugin..ctor () [0x00006] in <ef9b11cea30c46f9bf63f068ca18019e>:0
  at (wrapper managed-to-native) System.Reflection.RuntimeConstructorInfo.InternalInvoke(System.Reflection.RuntimeConstructorInfo,object,object[],System.Exception&)
  at System.Reflection.RuntimeConstructorInfo.InternalInvoke (System.Object obj, System.Object[] parameters, System.Boolean wrapExceptions) [0x00005] in <7b90a8780ac4414295b539b19eea7eea>:0
eco@eco:/opt/ELSM/Server/TestServer$

This is an issue with ECO, I've long since abandoned this project until ECO becomes more stable with linux servers.


Typo issue in LeafletDirective section of the documentation

In https://github.com/Asymmetrik/angular2-leaflet#inject-leafletdirective-into-your-component, the variable leafletDriective is misspelled.

This fix will go in soon with a new release I'm working on.

Should be fixed in the latest release, thanks!


V2 release candidate

The purpose of this pull request is to release github.com/MicahParks/keyfunc/v2 to be compatible with github.com/golang-jwt/jwt/v5.

This looks ready to publish to me. I'm going to leave it leave it open until after business hours today Easter Daylight Time (EDT).
Please make sure to leave any final comments before then.


[Darwin] Subscription reports should be streamed rather than wait for OnReportEnd

Reproduction steps
Currently both MTRDevice and MTRBaseDevice subscriptions report at OnReportEnd time. Given SubscriptionCallback is using Buffered callback, all OnXData calls should be complete and reportable.
We should stream the reports for more responsive subscriptions.
Platform
darwin
Platform Version(s)
1.0
Type
Platform validated
(Optional) If manually tested please explain why this is only manually tested
No response
Anything else?
No response

Reopening since the reachable PR is related but doesn't fix this.


[SQL Lab] Reduce number of queries needed to create a chart

The number of query executions needed to get from SQL->Dashboard is inefficient. Stresses databases (more redundant query executions), slows down data analysis (waiting for queries to complete)
Typical workflow to create a chart
1st SQL run --> SQL Lab to execute query and get result sets
2nd SQL run --> Clicking Explore, which instantiates the query as a datasource off of the result set
3rd and more SQL runs --> Changing viz type (because user lands on a count of # of rows upon clicking Explore) and subsequent changes on the datasource all require re-execution
Proposal:
(1) temporarily cache result sets for SQL queries run on a tab? Can be quickly discarded based on last query run on a tab. Persist as long as tab remains open or something reasonable like 24 hours?
(2) If user click on Explore, persist result set in cache, such that subsequent actions don't require re-execution
Open to other suggestions which solve the problem of too many SQL re-executions in order to explore/chart a result set which remains static.
Make sure these boxes are checked before submitting your issue - thank you!

[ ] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[ ] I have reproduced the issue with at least the latest released version of superset.
[ ] I have checked the issue tracker for the same issue and I haven't found one similar.

Superset version
Expected results
Actual results
Steps to reproduce

This is pretty relevant.  More generally, the results of an SQL query should probably be savable as a data source from the "save as" option.  Then can just use alongside any database table as basis for charts.

+1


TPM unlocking fails: stat /dev/tpmrm0: no such file or directory

I have used systemd-cryptenroll to add a TPM2 key. When I try to boot an image generated using booster (no arguments, empty /etc/booster.yaml), I get a password prompt instead of automatic TPM unlocking, and with booster.log=debug I see
Nov 26 21:11:56 mike-xps booster: recovering systemd-tpm2 token #1 failed: stat /dev/tpmrm0: no such file or directory
Nov 26 21:11:56 mike-xps kernel: fbcon: Taking over console
Nov 26 21:11:56 mike-xps kernel: Console: switching to colour frame buffer device 240x67
Nov 26 21:11:56 mike-xps booster: udev event {Header:add@/devices/virtual/vtconsole/vtcon1 Action:add Devpath:/devices/virtual/vtconsole/vtcon1 Subsystem:vtconsole Seqnum:1340 Vars:map[ACTION:add DEVPATH:/devices/virtual/vtconsole/vtcon1 SEQNUM:1340 SUBSYSTEM:vtconsole]}
Nov 26 21:11:56 mike-xps booster: udev event {Header:add@/devices/platform/MSFT0101:00/tpm/tpm0 Action:add Devpath:/devices/platform/MSFT0101:00/tpm/tpm0 Subsystem:tpm Seqnum:1341 Vars:map[ACTION:add DEVNAME:tpm0 DEVPATH:/devices/platform/MSFT0101:00/tpm/tpm0 MAJOR:10 MINOR:224 SEQNUM:1341 SUBSYSTEM:tpm]}
Nov 26 21:11:56 mike-xps booster: udev event {Header:add@/devices/platform/MSFT0101:00/tpmrm/tpmrm0 Action:add Devpath:/devices/platform/MSFT0101:00/tpmrm/tpmrm0 Subsystem:tpmrm Seqnum:1342 Vars:map[ACTION:add DEVNAME:tpmrm0 DEVPATH:/devices/platform/MSFT0101:00/tpmrm/tpmrm0 MAJOR:236 MINOR:65536 SEQNUM:1342 SUBSYSTEM:tpmrm]}

in the logs. It seems that the device node /dev/tpmrm0 is being created too late, after booster already tried to open it.
If I set booster.log=debug,console, the TPM unlock is successful. I'm guessing that all the text being printed slows things down so that things happen in the correct order, because in the logs I see that tpmrm0 is being created before the disk is unlocked.
This is happening with both booster-git and 0.7.
$ uname -r
5.15.5-arch1-1
$ pacman -Q booster
booster-git 0.7.r24.g754a0d3-1

Yes indeed it is a race condition between discovering required devices and processing LUKS partition. The LUKS partition is handled so fast that it happens before TPM driver initializes devices.
It belongs to the same bucket of problems as with other hardware dependencies:

booster tries to unlock a tang binding before the network is available
booster tries to unlock a Yubikey bound device before it is initialized (or if it was inserted later, after the boost process started)

I want to look at these issues before 0.8 release.

@Axelen123 thank you very much for the fix. The code looks good. I ran integration tests for an hour and I do not see any errors with it. The PR is merged to wip branch.
FYI Here is a related ticket that aims to create an event synchronization mechanism that would handle hardware event like TPM chip is available, Network is available, Yubikey is inserted, ... https://github.com/anatol/booster/issues/100

The fix has been merged to master


Add tests for keyframes

Closes #160
tweak the indentation on keyframes output as well

Coverage decreased (-4.74%) to 95.26% when pulling 3d8402cf24151e5bfd0c4b771c6328aac51c2eea on keyframes-tests into e8a0f83d99e4ad936e765f86849fb20e10d0b1db on master.

Coverage remained the same at 100.0% when pulling 0ae20e5c0f27a9d127cfadc50cd4077cb94dc4cb on keyframes-tests into e8a0f83d99e4ad936e765f86849fb20e10d0b1db on master.

Coverage remained the same at 100.0% when pulling ef504c32df3cc8c2bec4241b9ea975d4c6dc6199 on keyframes-tests into e8a0f83d99e4ad936e765f86849fb20e10d0b1db on master.

:+1:


Execute some commands asynchronously in Neovim

Neovim has job control feature and it enables to run commands asynchronously easily. (In Vim, it requires a trick.)
Running spec and implementation tools improves coding Crystal.

Vim will have job control feature.  I may be able to run tests asynchronously also in Vim, not only Neovim.


[Arc] Fix version check issues in Arc Extension

Fixes: https://github.com/raycast/extensions/issues/4246
Maybe this too: https://github.com/raycast/extensions/issues/4258
Screenshot with changing required version to 0.85.1

Thanks for the fix! I went and improved the code a bit further with two things:

I rely on the cached promise for the version check. This way it's faster than waiting for the AppleScript to finish loading.
I added an image to make the Version Check look a bit better.

And I added you as a contributor 😉

Awesome, Thank You! I missed that caching part 🙈


Add a static factory for Integrated LCache object.

This PR provides a static factory method for the Integrated LCache object:
$integrated = \LCache\Integrated::create($dbh);

This allows canonical clients to reference only Address and Integrated classes.
This PR does not test the new LCache:: setOverheadThreshhold() method, but I could add a test to pick that up to keep us at 100% coverage if this general concept (reducing required API touch-points for canonical clients) is desired.

Coverage decreased (-0.7%) to 99.295% when pulling<PHONE_NUMBER>eddd4d08a99b182b9982bae391fd3f on integrated-factory into 375afc39465403da4e365917bc7cb94e2e21c41a on master.

Coverage decreased (-0.7%) to 99.295% when pulling<PHONE_NUMBER>eddd4d08a99b182b9982bae391fd3f on integrated-factory into 375afc39465403da4e365917bc7cb94e2e21c41a on master.

@davidstrauss LMK if you like the PR, and I'll write the test needed to go back to 100% if you do.

We could separate this into a separate Factory class, but that's not really needed at this point. Closing for now.


[$1000] Multiline inputs have cursor pointer over its label name but single line inputs have 'I' cursor over its label name

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Open the app
Open settings
Open Security
Open close account
Click and Hover over message label i.e. hover over 'Enter message here' to see that cursor is displayed as pointer
For single line input, visit settings->profile->display name and click and hover over first name and last name input label to observe that it displays 'I' cursor

Expected Result:
App should maintain consistency and display 'I' cursor over label of inputs like it does for single line input
Actual Result:
App displays pointer cursor on label of multiline input instead of single line input
Workaround:
Can the user still use Expensify without this being fixed? Have you informed them of the workaround?
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number: 1.3.36-1
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://github.com/Expensify/App/assets/43996225/71bf1312-d25b-4535-9cb1-50ad15d50ca0
https://github.com/Expensify/App/assets/43996225/19d0e600-e079-4567-8cf9-19310869513a
Expensify/Expensify Issue URL:
Issue reported by: @dhanashree-sawant
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1685951694011049
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01688c55b644b2f2fb
Upwork Job ID:<PHONE_NUMBER>182765056
Last Price Increase: 2023-07-06

Proposal
Please re-state the problem that we are trying to solve in this issue.
The BaseTextInput has an area above the input per se, that displays the wrong cursor. This area is overlapped for single line inputs, not for multiline ones.
What changes do you think we should make in order to solve the problem?
I changed the style of the PressableButton in BastTextInput, to add the required cursor.

@raphaelppp Thanks for the proposal. I don't think your RCA is correct. Also please follow the proposal template.

@samh-nl Thanks for the proposal. I don't think your RCA is correct.

Proposal
Please re-state the problem that we are trying to solve in this issue.
Multiline inputs have cursor pointer over its label name but single line inputs have 'I' cursor over its label name
What is the root cause of that problem?
The textinput is wrapped by PressableWithoutFeedback, see
https://github.com/Expensify/App/blob/60f48175c9e3342ef192d878218820e5ed5a76cd/src/components/TextInput/BaseTextInput.js#L248
and frontend inspection screenshot

Its style is calculated from here.
Below is the inspection screenshot of View to wrap RNTextInput

There's padding area(added by here) above the View displays cursor as pointer according to style calculated as mentioned above.
Both multi-line and single line input also have another similar issue, like moving cursor near the bottom border of the input. Their root causes are similar, parent container with cursor: "pointer" style is not fully filled by the container of textinput which displays text type pointer.
What changes do you think we should make in order to solve the problem?
To fix this issue, we can set accessibilityRole below this line
https://github.com/Expensify/App/blob/60f48175c9e3342ef192d878218820e5ed5a76cd/src/components/TextInput/BaseTextInput.js#L251
like
accessibilityRole={CONST.ACCESSIBILITY_ROLE.TEXT}

to get right pointer for cursor this line
https://github.com/Expensify/App/blob/60f48175c9e3342ef192d878218820e5ed5a76cd/src/components/Pressable/GenericPressable/BaseGenericPressable.js#L146
I think we should also define property accessibilityRole
accessibilityRole: PropTypes.oneOf(_.values(CONST.ACCESSIBILITY_ROLE)),

below this line.
What alternative solutions did you explore? (Optional)
N/A

I have been checking this. I think it's a regression from https://github.com/Expensify/App/pull/19322. Previously we used TouchableWithoutFeedback in which the cursor was not explicitly set. I think we can get back to that state (and kinda fix the issue) by just adding styles.cursorInitial. However even back then we had a cursor inconsistency in the label where the multi-line label had a default cursor instead of text.

@eh2077 Thanks for the proposal. I don't think your RCA is accurate. The bug is not due to added padding but to the border of textInputContainer (for single and multiline inputs) and to this View (for multiline only).

Their root causes are similar, parent container with cursor: "pointer" style is not fully filled by the container of textinput which displays text type pointer.

That's correct :+1: Also please check the comment above ^ it may be helpful.

PS: Let's try avoid any accessibilityRole solution, accessibilityRole=text is not the right use here. This is used when the element should be treated as static text that cannot change.^1

Proposal
Updated RCA as pointed out and the solution to set style styles.cursorInitial for PressableWithoutFeedback of BaseTextInput

@eh2077 Thanks for the quick follow up. I didn't mean to control the cursor that way. I was just trying to actually unset the cursor.

Before when we used TouchableWithoutFeedback the button style didn't include cursor css at all
Now using PressableWithoutFeedback the button style includes cursor: pointer

I'm not sure if we can remove the new cursor style but setting it to auto, initial or unset mimics that.

For simplify, let's assume that PressableWithoutFeedback has pointerEvents="none". Now we are back to how things used to work (before https://github.com/Expensify/App/pull/19322), but as mentioned earlier the multiline label still has default cursor instead of text. How can we fix that?

Although controlling the cursor based on isEditable would work, I prefer to avoid this solution for now and instead have the cursor being set naturally.

@samh-nl Thanks for the update. I think the same note above applies here. Let's try avoid controlling the cursor and instead work on the View and other padding/margin values that may be overlapping with the Pressable.

For simplify, let's assume that PressableWithoutFeedback has pointerEvents="none". Now we are back to how things used to work (before #19322), but as mentioned earlier the multiline label still has default cursor instead of text. How can we fix that?

If we set pointerEvents="none" below this line
https://github.com/Expensify/App/blob/60f48175c9e3342ef192d878218820e5ed5a76cd/src/components/TextInput/BaseTextInput.js#L251
, then I think we only need to fix the major cursor issue of the background View in multiple line case by changing
https://github.com/Expensify/App/blob/60f48175c9e3342ef192d878218820e5ed5a76cd/src/components/TextInput/BaseTextInput.js#L272
to pointerEvents="auto"
and add
...cursor.cursorText,

for textInputLabelBackground

@eh2077 Thanks for the update. That would work but we'd be still controlling the cursor explicitly.

@samh-nl Thanks for the update. I was thinking of doing something in that sort but I think the challenge here is to ensure that the label does not overlap with the content and that the scrolling bar is not cut (you first need to fill in the input with some multiline text to test that). Can you confirm whether this works correctly in such case?

@s77rt
I updated my proposal to revise the RCA, and better suit the template. I also detailed my solution.

@samh-nl I think we are just evaluating different approaches for now.

In another scenario where the label does not overlap with the textarea, one can question whether the "text" cursor is applicable. It does not behave identically to single-line inputs as click + drag to select text would not work. It'd only be for activating focus upon click.

Indeed, I think whatever cursor and behaviour we go with it should be consist on both single and multiline input. For instance checking the mui text input it uses the default cursor on the label but that behaviour is consist.

@kevinksullivan I see I was assigned to this when the External label was applied, but it looks like you're still assigned. Did you need me to take this on?

@raphaelppp Thanks for the update. That solution has been proposed before, currently we are looking for better approaches.

@s77rt
I used the solution from @samh-nl, but instead of padding I used a top border of 23 px. The scroll bar is preserved.

@raphaelppp Thanks for the update. This seems going well but the height is not being updated correctly i.e. after pressing enter the height is less than expected.

@samh-nl I'm not sure if this fixed the issue (does not seem so from my end) but regardless we should avoid making any change to box-sizing for consistency. Native only supports border-box.

@raphaelppp Instead of adding the 23 value to textInputHeight let's adjust the actual height of the hidden input https://github.com/Expensify/App/blob/0bdca706efb6344adfbccc6cff9f02a74f5a3650/src/components/TextInput/BaseTextInput.js#L387-L395. Would that fix the other bug?

@s77rt
There is no need to add 23px to textInputHeight itself because it already includes a 23px+8px padding. This is because the hidden text used for measurement has props.inputStyle, which includes baseTextInput. Adding 23px on top of that would yield an incorrect height for the root view, since it also uses textInputHeight.
The 23px added to compute the height of the actual text input is because this input now includes a 23px border besides the text itself.
And if you wonder why we needed to add 23px on top of something that already factored in a 23px, this is because getAutoGrowHeightInputStyle is a strangle fellow that returns maxHeight or maxHeight-25 depending on whether textInputHeight is greater than maxHeight. The result is that a multi-line input offsets the controls below it! This is another undiscovered bug, unrelated to the rest.

https://github.com/Expensify/App/blob/08fa90c6ec2851db3b0c6d1541a7e00de5f3309e/src/styles/StyleUtils.js#L331-L346
As for the other bug I mentioned (the incorrect textInputHeight on a new line), it is unrelated to all of this.

@s77rt
There is no need to add 23px to textInputHeight because the hidden text used for measurement already factors in a 23px+8px padding. It comes from props.inputStyle, which includes styles.baseTextInput. Adding another 23 px would mess up the height of the root view, since it also uses textInputHeight.
The reason we need to add 23px to the height of the input per se, is because we added a 23px border to it.
Now you may wonder why we needed to do that since it already factored in a 23px padding in the first place. This is because getAutoGrowHeightInputStyle is a counter-intuitive fellow: it either returns maxHeight or maxHeight-25, depending on whether textInputHeight is bigger than maxHeight. The text is never stretched, it merely pushes down the controls below.
https://github.com/Expensify/App/blob/08fa90c6ec2851db3b0c6d1541a7e00de5f3309e/src/styles/StyleUtils.js#L331-L345

At first glance it seems that getAutoGrowHeightInputStyle is incorrect. For the start it seems to use the wrong padding (23+2 rather than 23+8). But it produces correct results, and modifying it proved to be non-trivial.
As for the other bug mentioned (the incorrect text height on a new line), it is completely orthogonal. I just reported it to the Slack channel.

@raphaelppp

The reason we need to add 23px to the height of the input per se, is because we added a 23px border to it.

That's the part that does not seem to make sense. The box-sizing is border-box, the height already includes the width of the border. Maybe the actual formula is textInputHeight + borders/paddings > maxHeight (similar to how it's done in the second phase)?

The other bug is unrelated, I though it was a regression, it's a bug that exists on main too. This can be handled separately.

I have just found this PR https://github.com/Expensify/App/pull/22397 that applied part of the solution, now the cursor is no longer a pointer but there still exist inconsistency.

@s77rt
The textInputHeight is not computed from the actual input with the border. It is computed from the hidden text input that already had, and still has, an equivalent padding.
Therefore we changed the needed space for the former, but textInputHeight did not change.

@raphaelppp In that case why don't we add the border to the hidden input as well? The padding is also applied to both the real and the hidden input, isn't it?

@s77rt
The real input does not have any padding because it is removed by pv0:
https://github.com/Expensify/App/blob/60f48175c9e3342ef192d878218820e5ed5a76cd/src/components/TextInput/BaseTextInput.js#L320
To tidy things up, I propose to:

remove pv0 for the multiline case (keep it for !label)
moves the zero padding and the border to textInputMultiline, which is already part of props.inputStyle and shared by the actual and hidden inputs. Currently, textInputMultiline merely specifies the scroll padding. It is not used anywhere else.

@raphaelppp Looks that we may have some overlapping logic. Can you please update your proposal with the latest suggestions?

@s77rt
updated

@s77rt
Right after doing the "tidying" we discussed earlier, I noticed it causes a new bug.
This brought me back to getAutoGrowHeightInputStyle, and the 23px offset I was passing to it. I now removed this offset and instead modified this function:
-height: maxHeight - styles.textInputMultilineContainer.paddingTop - styles.textInputContainer.borderBottomWidth
+height: maxHeight - styles.textInputContainer.borderBottomWidth - styles.textInput.paddingBottom,

This now works flawlessly, and it is logical.

@s77rt
updated

@raphaelppp Thanks for the update.

On Input, I modified isLabeledMultiline by using props.autoGrowHeight rather than props.multiline

This change does not make sense. Did you mean to update the isLabeledMultiline to be true when either of this.props.multiline, this.props.autoGrowHeight is set?

@s77rt
The problem is that no input is ever declared as multiline: instead people use autoGrowHeight to mean multiline.
Technically there could be a multiline input without autoGrowHeight, so they are supposed to be different. But this is not how they are employed, and I doubt a non-growing multiline input would work correctly given the existing code. But a proper clean-up is out of the scope of this issue.
Stuck between an anvil and a hammer, I choose to use autoGrowHeight. But I am fine using a disjunction.

@s77rt
For the multiline and unlabelled cases, pv0 was replacing the 10px bottom padding by a 0px padding on the actual input. I am pretty sure this is unintended, that only the top padding was supposed to be removed.
If this is intended, then this padding should also be added to styles.textInputMultiline, to affect the hidden input.
, not on the hidden one. I am not sure

@s77rt
There is a difference between how  is declared in index.js and index.native.js: the latter does not change the style for the isMultilineLabelled case.
It was inconsequential before, since isMultilineLabelled was never true, when the actual and hidden inputs had different styles.
But now it seems wiser to move this part to BaseInput instead.

@s77rt
Just an update to say that I have checked Storybook but I am still struggling to try to run Expensify on Android.
Regarding storybook, everything is fine. Note however that the autoGrowHeight story does not use a scrollbar (maxHeight on textInputContainerStyles rather than containerStyles), and was not behaving properly even before my modifications. However everything is fine with my changes and the fix for https://github.com/Expensify/App/issues/21271 (the other issue I previously mentioned).  I am considering adding a story for that.

@raphaelppp

I am still struggling to try to run Expensify on Android.

Can you please check in iOS instead?

@s77rt

Can you please check in iOS instead?

Because you think it will be faster (I doubt so), or because you think compatibility is more at risk on this platform?

@raphaelppp Building iOS is much faster than Android. You can test on either. Just need to confirm that the behaviour of TextInput is consist (since we have different files index.js and index.native.js).

@s77rt
The behavior of the Android layout engine is unfortunately different: the text is not offset by the border ! I therefore had to add padding besides the border, but just on the visible input, not the hidden one.
I now need to test iOS and the various browsers, to check their behavior.

@raphaelppp Any updates here?

@s77rt
I am still struggling to build the iOS version. Hopefully the end is near.

@s77rt
Everything is ok. Storybook, android, ios, firefox, chrome, edge, safari.
[https://github.com/Expensify/App/issues/22121#issuecomment-1618803732](Updated proposal)

@raphaelppp Thanks for the update. I have question regarding this point

I added a 23px top padding on the visible input on native platforms, since on native the border does not offset the position

How come we have inconsistency between web and native? Is this related to E/App or RN/RNW itself? It would help if you can make a reproducible example in snack (if this is a bug in RN/RNW).

@s77rt
I sort of get a repro, but the problem occurs on the Text, rather than the TextInput, which is the opposite of what i am seeing with the App. Note how the "text" appears over the border on Android, but below the border on other platforms.
https://snack.expo.dev/2zzjJg5Bp
NB: I forgot to mention after testing on iOS, but the problem does not occur on this platform: this is android-specific.
NB: In snack the border is not visible on iOS, but this does not occur with App.

@s77rt
I am starting to wonder if it would not be easier to just use a dark view under the label and over the input (so that the input can still fill the whole space), and a padding on the right to leave the scroll bar visible.
At least it would work equally well on native + web.

@s77rt
I did just that and it works well. The only con is that the scroll bar starts above the label, not below. But it seems to be a good compromise.

@s77rt
I updated the proposal to add the aforementioned alternative.

@s77rt
As I rushed a solution yesterday I just used a constant right margin. Actually it needs to be computed with another hidden element. I am going to update the proposal.

@raphaelppp Please tag me once you update your proposal.

@s77rt
It has been updated already, but I did not yet test the latest mentioned trick. I will do so within a couple of hours.

@s77rt
The alternative solution I proposed has a flaw that I do not know how to fix: when we use the up arrow to go back to the top of the text, the top line is not brought into view.
This sends us back to the original solution, with the border. It works flawlessly in the Android and iOS simulators, for the versions I tested, but it may be vulnerable to future changes.

@raphaelppp So in Android we add both padding and border where in others we just add the border? (to the visible input)

@s77rt
Exactly.

@raphaelppp Okay I see and do we have this issue reported in RN?

@s77rt
No, since I do not have a repro for the textInput (only for text).
I tried to get a better repro but Snack has started to fail when I set the input multiline. I should try an empty projet with the simulator.

Looks like this issue is fixed by https://github.com/Expensify/App/pull/22397

@eh2077
The merge solution poses problems though: it gives the user the impression he can click to focus or drag to select, which is not the case. This is why we were still working on an alternative.

@raphaelppp Thanks for continuous work here. I think we can move on and work the rest on the PR.
:ribbon: :eyes: :ribbon: C+ reviewed
Link to proposal

@s77rt
Thank you very much! I will submit the PR within a few hours.
For future reference I leave here the two related React native bugs I submitted. This allowed me to refine my solution.
https://github.com/facebook/react-native/issues/38517
https://github.com/facebook/react-native/issues/38518

@raphaelppp Please do not raise a PR yet. Wait for @neil-marcellini to review your proposal first.

@neil-marcellini
small proposal update to propose a better workaround regarding the Android bug I clarified today.

I'm no longer able to reproduce because the only mutliline text inputs we have in App right now are the composer and the editing composer, and I'm not able to reproduce the issue there. If we add a multiline text input in the future we can reopen, but for now I suggest we close this. CC @johncschuster.

@neil-marcellini
While the initial cursor problem has been fixed because of https://github.com/Expensify/App/pull/22397, it has created a new problem that we were trying to fix: the cursor gives the user the impression he can click to focus the input, or drag to select. This is not the case with the merged fix.

@raphaelppp If there is another problem please create a new bug report in #expensify-bugs.

@johncschuster I've recommended we close this. Please lmk if you need anything to get this closed out.

Oh, I missed your recommendation! I'll go ahead and close it.


Make the DID standard useful for our Certificate System

https://w3c-ccg.github.io/did-spec/
https://identity.foundation/

or are there more standards? e.g. ERC725


Renovations of the project

Renovations

The repository for ipxe changed - ipxe was complainging with an error that no longer exists;
Using debian:bullseye

"Features"

Show output from build script

Helps to know that the container is working, especially when using this for the first time.
Helps to know when it finished.

add 'runDocker.sh'

Makes it easier to run and rerun - also uses modern docker compose.
Added comment to show how to scp the result to a target server

Helps to understand where one of the output files is located exactly.

Use local git copy as the context (docker-compose.yml).

I made several updates to the build script/Dockerfile without any effect until I saw that the context was the remote git repository.

Added some of your suggestions in 4c830caeb654d0bcbd10e6eb4a9790d3930c39f8 and 975f860eb5de1d94fd91ba92118ef20ce779f3fa


Same client address and ssh error

Good Morning,
I am testing the program, but I have noticed that the client's ip in the "client Address" menu is always <IP_ADDRESS>. Shouldn't it be the client's ip selected in the menu?
On the other hand, I get this error in the windows version.
BoreTunnel error:  Failed to dial: %!(EXTRA *errors.errorString=ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain)
Thanks in advanced!!

I can see this in ssh log
Bad options in /root/.ssh/authorized_keys file, line 1: permitlisten="<IP_ADDRESS>:36299" ssh-rsa "numberhere"

Hi @Ataraxiall, boringproxy requires version 7.8+ (that's when permitlisten was added). permitlisten is a security feature to ensure a given private key can only open tunnels on a specific port. This isn't so important if you're the only user on your server (which is likely), but there's not currently a way to disable it, and it's not planned.
If you want to self-host a simple single-user tunnel proxy, I'd recommend you take a look at SirTunnel, or check my list here for other projects.

Thanks a lot!
I have updated it and now it works great!
Thanks!


RFC: A type-safe readResult helper, to make writing tests easier

Summary
Tada is an awesome library, solves almost all the pain points of codegen preset-client, except one: writing test fixtures with fragment masking turned on.
Currently, when writing a test fixture, you have two options:

Use the unsafe_readResult function, which treats fragments as type any and hence is unsafe.
Mask each individual fragment explicitly, causing the test fixture to be entangled with the implementation of the query & hierarchy of fragments (and thus UI components).

Here's a simplified example from our code base for case 2:

import { graphql, maskFragments, ResultOf } from 'gql.tada';

export const FavouritesCardFragment = graphql(`
  fragment FavouritesCardFragment on Location {
    id
    name
  }
`);
const FavouritesQuery = graphql(
  `
    query Favourites {
      favouritedLocations {
        edges {
          node {
            id
            ...FavouritesCardFragment
          }
        }
      }
    }
  `,
  [FavouritesCardFragment],
);
const mockedFavouritesData: ResultOf<typeof FavouritesQuery> = {
  favouritedLocations: {
    edges: [
      {
        id: '1',
        name: 'Location A',
      },
      {
        id: '2',
        name: 'Location B',
      },
    ].map(node => {
      return {
        node: {
          id: node.id,
          ...maskFragments([FavouritesCardFragment], node),
        },
      };
    }),
  },
};
console.log(mockedFavouritesData);

Each time any of the internals of query Favourites are changed, such as which fragments it uses or  which components they come from, our test fixtures also need to change, despite the actual behaviour  (how the server sees it) and thus the fixture itself remaining unchanged.
Proposed Solution

In theory, unsafe_readResult doesn't need to be unsafe. However, the problem right now is that TadaDocumentNode does not keep references to the fragments that were passed into the GraphQLTadaAPI call signature. If the list of fragments was to be retained somehow, then readResult (safe version of unsafe_readResult) can traverse the ResultOf<Document> type and instead of using omitFragmentRefsRec<ResultOf<Document>> these refs can be looked up in the fragments list and added back to the result type.
In a very simplified way:
type getDocumentNode<
  Document extends DocumentNodeLike,
  Introspection extends SchemaLike,
  Fragments extends {
    [name: string]: any;
  } = {},
  isMaskingDisabled = false,
 > = getDocumentType<Document, Introspection, Fragments> extends never
  ? never
  : TadaDocumentNode<
      getDocumentType<Document, Introspection, Fragments>,
      getVariablesType<Document, Introspection>,
-     decorateFragmentDef<Document, isMaskingDisabled>
+     decorateFragmentDef<Document, isMaskingDisabled>,
+     Fragments
    >;

interface TadaDocumentNode<
  Result = {
    [key: string]: any;
  },
  Variables = {
    [key: string]: any;
  },
  Decoration = void,
+ Fragments extends {
+   [name: string]: any;
+ } = {},
> extends DocumentNode,
    DocumentDecoration<Result, Variables>,
-   DefinitionDecoration<Decoration> {}
+   DefinitionDecoration<Decoration> {
+     __fragments: Fragments,
+   }

And then a safe readResult that uses something similar to omitFragmentRefsRec can be used.
Requirements

TadaDocumentNode retains information about what fragments were used/masked.
A new readResult function is introduced, that is type safe (retains type information about what's been masked by fragment masking)

This is on my radar, but to be honest, I'm not a fan of this approach and am rather trying to think of alternative APIs that make composing results easier.

If the list of fragments was to be retained somehow, then readResult (safe version of unsafe_readResult) can traverse the ResultOf type and instead of using omitFragmentRefsRec<ResultOf> these refs can be looked up in the fragments list and added back to the result type.

This is imho not acceptable and will lead to a degredation of TypeScript performance. While this is somewhat acceptable with disableMasking, passing through all fragments leads to:

more noise in the type output
potentially fewer lazy evaluations by TS (and hence a negative effect on all TS features)
even if all fragments are available there, they'd still have to be unwrapped in a second step

I can imagine a type that recursively unwraps fragments. This is potentially just as costly but isolated to a usage site in the tests, similar to maskFragments(), but on demand.
That said, I still don't 100% agree with that API, because this assumes that we want to have exact result types as test fixtures, which isn't always the case. I basically am not convinced that it's ergonomic and nice to create huge mock results in this manner, and would still recommend an unsafe cast for that, to encourage these data fixtures to not be handwritten (i.e. generated from real data)

I agree with you, especially about test fixtures. Most of the time you don't need 100% of the result type for a test fixture, but this isn't something that can't be solved with Partial or RecursivePartial type.
To address your points:

I don't see much noise that's unmanageable by adding fragments list to TadaDocumentNode as a type argument. That type is almost never used on its own, and the suggestion here would not affect ResultsOf and VariablesOf, unless you had something else in mind?
I'm not really familiar with lazy evaluation in typescript, and couldn't find much when I looked it up. Can you elaborate here or share any links to relevant resources?
Yes and that's unescapable, but still solves the problem of having tests need to reflect implementation of which fragments are used in which query

I really dislike the idea of having coded fixtures in general, but there is another more acceptable use-case than the one I mentioned: components that use other components (fragments referencing other fragments), and need manual fixtures such as for storybooks.
const VenueCardFragment = graphql(`
fragment VenueCardFragment on Venue {
  id name image
}
`);
const VenueCard: React.FC<{venue: ResultOf<typeof VenueCardFragment>}> = ({venue}) => { /* ... */ }

const VenuesCarouselFragment = graphql(`
fragment VenuesCarouselFragment on VenuesConnection {
  edges { node { ...VenueCardFragment } }
  pageInfo { hasNext nextCursor }
}
`);
/** A carousel of venues with pagination/infinite scroll etc */
const VenuesCarousel: React.FC<{data: ResultOf<typeof VenuesCarouselFragment>}> = ({data}) => { /* ... */ }

The problem with splitting components between presentational & structural is it either encourages immature abstractions, where people start creating very generic components too early, or it gets too verbose, where people have to re-declare the query return type as a props type in their presentational components to avoid coming up with a new abstraction too early.
I am happy to see any other proposals you might have that solve the main requirement here really: users have the ability to get the unmasked return type from the return value of graphql() function, without needing to re-declare what was passed into that function (i.e. fragments list).

Just to emphasis: I've been pushing my teams to start adopting fragment co-location, and tada is the only tool that allows incremental adoption with decent DX, so really appreciate the work that was done here. Just trying to make that adoption barrier lower for my teams and the community overall.

I think personally I am in a similar boat to Phil, when we talk about unit-testing I like to think that we test a component in isolation and not the children/... I do realise that this leans towards a different discussion in terms of the whole shallow vs non-shallow rendering during tests. I do however believe that this is a similar problem space, we co-locate our fragments so we don't have to care in the parent-component about what data is needed by the children.

I'm not really familiar with lazy evaluation in typescript, and couldn't find much when I looked it up. Can you elaborate here or share any links to relevant resources?

There are sadly very little resources on this but the gist of it is that TypeScript as it evaluates types will often try to defer work to when it's needed/used/... This means that de-normalizing a type like the Fragment by immediately including it will force TS to evaluate it and hence incur more work. This is a pretty big cost to make testing easier imho.
I do see the case for storybook and visual regression testing there, we have faced something similar where I work. We do however use the approach you describe initially where we keep calling maskFragments. My thinking is more going into the direction of an alternative helper that doesn't change the core types as we worked a lot on performance of this library to be acceptable.

Just bumping this here 😄 I know, unusual for a maintainer to bump a feature request.
I want to basically lay out some requirements, if anyone has an alternative proposal for maskFragments and unsafe_readResult (or improvements) that:

improve the ability to build up mock result data
ensure that fragment masking is still enforced (somewhat)
fix the problem of mocked data and/or fake data specifically (including for cache updaters)
and; does not break internal type separation (i.e. doesn't imply that @_unmask is used everywhere and doesn't pass on fragment types on each type, which would worsen TS laziness and performance)

Then we're happy to look into this. From our end, we currently don't have any proposals or solutions that improve this. Sorry! 😅

I haven't looked at your codebase since opening the ticket above, so apologies if what I'm saying makes no sense, but what if every document node came with two result types: Masked and Unmasked
The unmasked type would be generated identically to the masked type, if the masking option was turned off
Then you can have a utility UnmaskedResultOf type that can extract that
The unmasked result type can be exposed either as a property on the document (similar to how I was proposing exposing __fragments) or you can store it on some interface type by giving each doc an id and augmenting that interface (similar to your cache/turbo type I believe)


Doc: Fix missing parenthesis

Checklist

[x] Use a separate branch in your local repo (not master).
[x] Test coverage is 100% (or you have a story for why it's ok).

@googlebot I signed it!


SQL should not be cached if it does not contain a  question mark

For now, SQLs will be cached when use a PreparedStatement regardless of the question mark. This may lead to all SQLs of a service be cached and performance declined.

What dose the rules should be
rule A: SQL include a question mark ? cache it : no cache
rule B: Use jdbc.PreparedStatement -> cache it; Use jdbc.Statement -> no cache

I think the rule should be: Use PreparedStatement and contains question mark.
An SQL without question mark will never be hit in most scenarios.

We should not consider about whatever SQL contains parameter marker or not.
For PreparedStatement, we should cache the SQL parse result; for Statement, do not cache.

However, this could be an extension point of our shardingsphere plugin system

`
MyFactory extends DefaultShardingSpherePreparedStatementFactory {
boolean cacheThePreparedStatement ( it ) {
if ( not contains parameter marker ) {
return false;
}
	return super.cachePreparedStatement( it );
}

}
`


Allow multiple breakpoints for sidebars

For Electron (and web), I would like the (left) sidebar to be a column of (large) icons. When hovering over the panel, it should expand (cover) to see the names related to the icons.

I guess, you mean something like this https://framework7io.github.io/framework7-recipes/admin-dashboard/ ?
I think, it makes sense to add it to the core

Yes, exactly! Also, nice option on desktop/electron when hovering over icon to have it expand (cover) in case we want to always have it as an icon view.


Use jl.permutedims in permute_dims

Hi @hameerabbasi @willow-ahrens,
This small fix makes permute_dims use jl.permutedims instead of jl.swizzle.
It's needed to make transpositions work in the lazy mode, because for swizzle(lazy(tensor), ...) we get:
ERROR: MethodError: swizzle(::Finch.LazyTensor{Float64, 3}, ::Int64, ::Int64, ::Int64) is ambiguous.

Candidates:
  swizzle(body, dims::Int64...)
    @ Finch ~/JuliaProjects/Finch.jl/src/tensors/combinators/swizzle.jl:80
  swizzle(arr::Finch.LazyTensor, dims...)
    @ Finch ~/JuliaProjects/Finch.jl/src/interface/lazy.jl:63

and permutedims doesn't raise it.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

:exclamation: No coverage uploaded for pull request base (main@0c3a202). Click here to learn what that means.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #44   +/-   ##
=======================================
  Coverage        ?   91.37%
=======================================
  Files           ?        7
  Lines           ?      649
  Branches        ?        0
=======================================
  Hits            ?      593
  Misses          ?       56
  Partials        ?        0

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

I think we should fix that ambiguity error too though. I'll add the issue to finch.jl

Sure! For now it's a small fix here in finch-tensor.

Can we add a regression test? Something which ensures permute_dims will always compile?

Sure! Done! I extended test_permute_dims so it also runs lazy mode (it passes here and fails in a main branch).

Thanks for the diagnosis and the work put towards fixing this, @mtsokol!


APP-1292 - Sending an incomplete or malformed to Universal Webhook returns the wrong status code

RemoteApiExceptions weren't being properly handled on the WebHookDispatcherResource, I've added an exception handler to this class, however if in the future we discover more exceptions are not being handled I believe a whole exceptionHandler class should be made.

Codecov Report

Merging #129 into dev will decrease coverage by 0.63%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##              dev     #129      +/-   ##
==========================================
- Coverage   90.19%   89.56%   -0.64%
==========================================
  Files         141      171      +30
  Lines        2785     3506     +721
  Branches      261      326      +65
==========================================
+ Hits         2512     3140     +628
- Misses        215      281      +66
- Partials       58       85      +27

Impacted Files
Coverage Δ

...ration/web/resource/WebHookDispatcherResource.java
97.29% <0%> (-2.71%)
:arrow_down:

.../exception/ConfigurationProvisioningException.java
50% <0%> (ø)

...ation/provisioning/client/AppRepositoryClient.java
95% <0%> (ø)

...xception/IntegrationProvisioningAuthException.java
100% <0%> (ø)

.../integration/provisioning/service/UserService.java
94.59% <0%> (ø)

...rovisioning/service/CompanyCertificateService.java
82.6% <0%> (ø)

.../properties/IntegrationProvisioningProperties.java
0% <0%> (ø)

...on/provisioning/exception/UserSearchException.java
100% <0%> (ø)

...tegration/provisioning/service/KeyPairService.java
91.66% <0%> (ø)

...ng/exception/ApplicationProvisioningException.java
100% <0%> (ø)

... and 21 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 594a568...441c051. Read the comment docs.

Build started sha1 is merged.


hash based routing

Hi guys,
I'm evaluating router5, but couldn't find info regarding hash based routing instead of history based.
Can some one provide info regarding this topic? I just wan't to know is hash based routing supported and does it play well with redux..
Thanks a lot in advance,
Emil Dimitrov

Hi Emil,
The router is a state machine, outputting state updates to whoever want to listen to them: it plays very well with redux. See https://github.com/router5/redux-router5 for more details, example here https://github.com/router5/examples/tree/master/apps/react-redux.
There is hash based routing available but it uses HTML5 history (i.e. it uses the history API and popstate events, not hashchange events, so you need history support). For syncing router state with your history, you need to use the browserPlugin. I've just updated the docs: http://router5.github.io/docs/history.html

Well then, sorry for my misunderstanding.. my brain is just really used to the default hashChange & history driven routes..
So instead of choosing between #user/details & /user/details we can just stop worrying about whats going out in the url bar... and even not use it for transition between our screens ( in SPA )..

The decision to use # or not is more a backend-driven decision (i.e. can you serve an index.html on any route or just on your domain root). I don't support hashchange since I don't need it myself and it has not been requested / implemented by users until now. That said, it's wouldn't be that difficult to implement a plugin. It would basically consist of:

On transition success, update the hash URL
On hash change event, ask the router to navigate to the new path

My concern was the browser support.. some support hash change some history api.. sad story. With state machine driven approach this doesn't look like issue, but and ux decision. After thinking about this approach i found it flawless ( in the context of client side SPA ), because by nature the url bar is meant for get requests to server and the # is meant for navigation to id.. not for client side routing..  in general i really feel natural a state machine for content transition & similar interactions.
Really appreciate your work! This is future.


added fetch-balance method to sparkswap exchange

Adds fetchBalance to the sparkswap CCXT exchange. This includes an example file for how users need to interact w/ ccxt.
The base branch fort this commit is adding-sparkswap
Example:

 add fetchBalance to ccxt implementation


Improve README and update the screenshots

Idk why but the screenshots didn't change...

Idk why but the screenshots didn't change...

they did update but github's imaging cache takes longer than expected.


[SPARK-49106][DOCS] Documented Prometheus endpoints

What changes were proposed in this pull request?
Since SPARK-46886 enables spark.ui.prometheus.enabled by default, this PR aims to provide clear documentation on the endpoints exposed by this PR.

Why are the changes needed?
Provide a better documentation.
Does this PR introduce any user-facing change?
No
How was this patch tested?
N/A
Was this patch authored or co-authored using generative AI tooling?
No

For the record,

I filed a new JIRA for you, SPARK-49106.
I removed the unmatched ending th tag, (		) to generate a correct HTML table.
I re-arranged the code in order to avoid mixing list and table.
I manually regenerated the Apache Spark webpage and verified. Also, the new screenshot is attached to the PR title.
I revised the PR title and description accordingly.

Do you have a JIRA ID? Please let me know. I can assign SPARK-49106 to you.

Merged to master for Apache Spark 4.0.0-preview2.

duuuude, it is an honour. my Jira ID is jerrdeh. thanks so much for the review, and excited to continue contributing.

I added you to the Apache Spark contributor group and assigned SPARK-49106 to you. You are at both Reporter and Assignee fields.

Congratulations to your first commit. Welcome to the Apache Spark community. :)


metrics: Add additional operational excellence metrics

Fixes https://github.com/numaproj/numaplane/issues/97
Modifications
Add additional operational excellence metrics
metrics data
# HELP isb_services_synced_total The total number of ISB service synced
# TYPE isb_services_synced_total counter
isb_services_synced_total{intuit_alert="true"} 1
# HELP leader_election_master_status Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master. 'name' is the string used to identify the lease. Please make sure to group by name.
# TYPE leader_election_master_status gauge
leader_election_master_status{name="numaplane-controller-lock"} 1
# HELP numaflow_controller_kube_request_total The total number of kubernetes request for numaflow controller
# TYPE numaflow_controller_kube_request_total counter
numaflow_controller_kube_request_total{intuit_alert="true"} 103
# HELP numaflow_controller_kubectl_execution_total The total number of kubectl execution for numaflow controller
# TYPE numaflow_controller_kubectl_execution_total counter
numaflow_controller_kubectl_execution_total{intuit_alert="true"} 16
# HELP numaflow_controller_running Number of Numaflow controller running
# TYPE numaflow_controller_running gauge
numaflow_controller_running{intuit_alert="true",version="0.0.2"} 1
# HELP numaflow_controller_synced_total The total number of Numaflow controller synced
# TYPE numaflow_controller_synced_total counter
numaflow_controller_synced_total{intuit_alert="true"} 1
# HELP numaflow_isb_services_running Number of Numaflow ISB Service running
# TYPE numaflow_isb_services_running gauge
numaflow_isb_services_running{intuit_alert="true"} 1
# HELP numaflow_kube_resource_cache Number of kubernetes resource object in cache
# TYPE numaflow_kube_resource_cache gauge
numaflow_kube_resource_cache{K8SVersion="1.26",intuit_alert="true"} 94
# HELP numaflow_kube_resource_monitored Number of monitored kubernetes resource object in cache
# TYPE numaflow_kube_resource_monitored gauge
numaflow_kube_resource_monitored{intuit_alert="true"} 0
# HELP numaflow_pipelines_running Number of Numaflow pipelines running
# TYPE numaflow_pipelines_running gauge
numaflow_pipelines_running{intuit_alert="true"} 1
# HELP numaplane_reconciliation_duration_seconds Duration of pipeline reconciliation
# TYPE numaplane_reconciliation_duration_seconds histogram
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="numaflow-controller-rollout-controller",le="0.005"} 0
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="numaflow-controller-rollout-controller",le="0.01"} 0
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="numaflow-controller-rollout-controller",le="0.025"} 0
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="numaflow-controller-rollout-controller",le="0.05"} 0
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="numaflow-controller-rollout-controller",le="0.1"} 0
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="numaflow-controller-rollout-controller",le="0.25"} 0
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="numaflow-controller-rollout-controller",le="0.5"} 0
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="numaflow-controller-rollout-controller",le="1"} 0
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="numaflow-controller-rollout-controller",le="2.5"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="numaflow-controller-rollout-controller",le="5"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="numaflow-controller-rollout-controller",le="10"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="numaflow-controller-rollout-controller",le="+Inf"} 1
numaplane_reconciliation_duration_seconds_sum{intuit_alert="true",phase="create",type="numaflow-controller-rollout-controller"} 1.832084167
numaplane_reconciliation_duration_seconds_count{intuit_alert="true",phase="create",type="numaflow-controller-rollout-controller"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="pipeline-rollout-controller",le="0.005"} 0
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="pipeline-rollout-controller",le="0.01"} 0
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="pipeline-rollout-controller",le="0.025"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="pipeline-rollout-controller",le="0.05"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="pipeline-rollout-controller",le="0.1"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="pipeline-rollout-controller",le="0.25"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="pipeline-rollout-controller",le="0.5"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="pipeline-rollout-controller",le="1"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="pipeline-rollout-controller",le="2.5"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="pipeline-rollout-controller",le="5"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="pipeline-rollout-controller",le="10"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="create",type="pipeline-rollout-controller",le="+Inf"} 1
numaplane_reconciliation_duration_seconds_sum{intuit_alert="true",phase="create",type="pipeline-rollout-controller"} 0.023695542
numaplane_reconciliation_duration_seconds_count{intuit_alert="true",phase="create",type="pipeline-rollout-controller"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="update",type="pipeline-rollout-controller",le="0.005"} 0
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="update",type="pipeline-rollout-controller",le="0.01"} 0
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="update",type="pipeline-rollout-controller",le="0.025"} 0
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="update",type="pipeline-rollout-controller",le="0.05"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="update",type="pipeline-rollout-controller",le="0.1"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="update",type="pipeline-rollout-controller",le="0.25"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="update",type="pipeline-rollout-controller",le="0.5"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="update",type="pipeline-rollout-controller",le="1"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="update",type="pipeline-rollout-controller",le="2.5"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="update",type="pipeline-rollout-controller",le="5"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="update",type="pipeline-rollout-controller",le="10"} 1
numaplane_reconciliation_duration_seconds_bucket{intuit_alert="true",phase="update",type="pipeline-rollout-controller",le="+Inf"} 1
numaplane_reconciliation_duration_seconds_sum{intuit_alert="true",phase="update",type="pipeline-rollout-controller"} 0.04209125
numaplane_reconciliation_duration_seconds_count{intuit_alert="true",phase="update",type="pipeline-rollout-controller"} 1
# HELP pipeline_rollout_queue_length The length of pipeline rollout queue
# TYPE pipeline_rollout_queue_length gauge
pipeline_rollout_queue_length{intuit_alert="true"} 0
# HELP pipeline_synced_total The total number of pipeline synced
# TYPE pipeline_synced_total counter
pipeline_synced_total{intuit_alert="true"} 4
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter

Verification

Fixes #97
Modifications
Add additional operational excellence metrics
metrics data
# HELP numaflow_controller_running Number of Numaflow controller running
# TYPE numaflow_controller_running gauge
numaflow_controller_running{name="numaflow-controller",namespace="example-namespace",version="1.2.1"} 1
# HELP numaflow_isb_service_running Number of Numaflow ISB Service running
# TYPE numaflow_isb_service_running gauge
numaflow_isb_service_running 1
# HELP numaflow_kube_request_total The total number of kubernetes request for numaflow controller
# TYPE numaflow_kube_request_total counter
numaflow_kube_request_total 235
# HELP numaflow_kube_resource_cache Number of kubernetes resource object in cache
# TYPE numaflow_kube_resource_cache gauge
numaflow_kube_resource_cache{K8SVersion="1.26"} 95
# HELP numaflow_kube_resource_monitored Number of monitored kubernetes resource object in cache
# TYPE numaflow_kube_resource_monitored gauge
numaflow_kube_resource_monitored 0
# HELP numaflow_kubectl_execution_total The total number of kubectl execution for numaflow controller
# TYPE numaflow_kubectl_execution_total counter
numaflow_kubectl_execution_total 32
# HELP numaflow_pipelines_running Number of Numaflow pipelines running
# TYPE numaflow_pipelines_running gauge
numaflow_pipelines_running 1
# HELP numaplane_reconciliation_duration_seconds Duration of pipeline reconciliation
# TYPE numaplane_reconciliation_duration_seconds histogram
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="isbsvc-rollout-controller",le="0.005"} 0
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="isbsvc-rollout-controller",le="0.01"} 11
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="isbsvc-rollout-controller",le="0.025"} 13
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="isbsvc-rollout-controller",le="0.05"} 13
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="isbsvc-rollout-controller",le="0.1"} 13
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="isbsvc-rollout-controller",le="0.25"} 14
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="isbsvc-rollout-controller",le="0.5"} 14
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="isbsvc-rollout-controller",le="1"} 14
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="isbsvc-rollout-controller",le="2.5"} 14
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="isbsvc-rollout-controller",le="5"} 14
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="isbsvc-rollout-controller",le="10"} 14
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="isbsvc-rollout-controller",le="+Inf"} 14
numaplane_reconciliation_duration_seconds_sum{phase="create",type="isbsvc-rollout-controller"} 0.22133087599999995
numaplane_reconciliation_duration_seconds_count{phase="create",type="isbsvc-rollout-controller"} 14
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="numaflow-controller-rollout-controller",le="0.005"} 0
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="numaflow-controller-rollout-controller",le="0.01"} 0
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="numaflow-controller-rollout-controller",le="0.025"} 0
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="numaflow-controller-rollout-controller",le="0.05"} 0
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="numaflow-controller-rollout-controller",le="0.1"} 0
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="numaflow-controller-rollout-controller",le="0.25"} 1
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="numaflow-controller-rollout-controller",le="0.5"} 2
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="numaflow-controller-rollout-controller",le="1"} 2
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="numaflow-controller-rollout-controller",le="2.5"} 2
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="numaflow-controller-rollout-controller",le="5"} 2
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="numaflow-controller-rollout-controller",le="10"} 2
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="numaflow-controller-rollout-controller",le="+Inf"} 2
numaplane_reconciliation_duration_seconds_sum{phase="create",type="numaflow-controller-rollout-controller"} 0.652376542
numaplane_reconciliation_duration_seconds_count{phase="create",type="numaflow-controller-rollout-controller"} 2
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="pipeline-rollout-controller",le="0.005"} 1
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="pipeline-rollout-controller",le="0.01"} 8
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="pipeline-rollout-controller",le="0.025"} 13
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="pipeline-rollout-controller",le="0.05"} 13
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="pipeline-rollout-controller",le="0.1"} 13
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="pipeline-rollout-controller",le="0.25"} 13
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="pipeline-rollout-controller",le="0.5"} 13
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="pipeline-rollout-controller",le="1"} 13
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="pipeline-rollout-controller",le="2.5"} 13
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="pipeline-rollout-controller",le="5"} 13
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="pipeline-rollout-controller",le="10"} 13
numaplane_reconciliation_duration_seconds_bucket{phase="create",type="pipeline-rollout-controller",le="+Inf"} 13
numaplane_reconciliation_duration_seconds_sum{phase="create",type="pipeline-rollout-controller"} 0.11895362499999998
numaplane_reconciliation_duration_seconds_count{phase="create",type="pipeline-rollout-controller"} 13
# HELP pipeline_rollout_queue_length The length of pipeline rollout queue
# TYPE pipeline_rollout_queue_length gauge
pipeline_rollout_queue_length 1
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 3.6
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 11
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.05197568e+08
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.7224031732e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 5.611327488e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP rest_client_requests_total Number of HTTP requests, partitioned by status code, method, and host.
# TYPE rest_client_requests_total counter
rest_client_requests_total{code="200",host="<IP_ADDRESS>:443",method="GET"} 1147
rest_client_requests_total{code="200",host="<IP_ADDRESS>:443",method="PATCH"} 12
rest_client_requests_total{code="200",host="<IP_ADDRESS>:443",method="PUT"} 869
rest_client_requests_total{code="201",host="<IP_ADDRESS>:443",method="POST"} 1

Verification

why this doesn't include the total number of ISB service synced, etc?

for any metrics that are particular to Numaflow Controller can we have "numaflow_controller" or "numaflow_ctlr" in the name?

Why is numaflow_controller_running parameterized by namespace but numaflow_pipelines_running and numaflow_isb_service_running aren't?

why this doesn't include the total number of ISB service synced, etc?
Fixed, It was missed register that variable

Why is numaflow_controller_running parameterized by namespace but numaflow_pipelines_running and numaflow_isb_service_running aren't? (this question is based on the PR description console output)

numaflow_isb_service_running and numaflow_pipelines_running is also parameterized with namespace but it's not added in label also for numaflow_controller_running I have removed the label namespace but left to update PR description. I'll update it. Thanks

Why is numaflow_controller_running parameterized by namespace but numaflow_pipelines_running and numaflow_isb_service_running aren't? (this question is based on the PR description console output)

I think we initially thought numaflow_controller_running should parameterize by namespace because per namespace only one controller, but now this may no longer hold true with no downtime upgrade. @chandankumar4 Do you mind having an issue to track this, you can use one issue to track all the left tasks in this PR. Thanks!

I'm kind of thinking about in a future PR, I wonder if we should consider avoiding doing metrics in the inner code and trying to as much as possible do them in one place, like in reconcile() or Reconcile(). I'm worried about future code in which we accidentally add some new code and miss adding in the metric there. Like in the case of the syncTime metrics, where we need to know if we created, updated, or did nothing, what if we returned a value from the inner functions to indicate the operation that was performed? Would it make the code too ugly? It does seem like less opportunity for error at least.

Why is numaflow_controller_running parameterized by namespace but numaflow_pipelines_running and numaflow_isb_service_running aren't? (this question is based on the PR description console output)

I think we initially thought numaflow_controller_running should parameterize by namespace because per namespace only one controller, but now this may no longer hold true with no downtime upgrade. @chandankumar4 Do you mind having an issue to track this, you can use one issue to track all the left tasks in this PR. Thanks!

btw I have already remove the namespace label from metrics but do we also wanted to revert this change and allow multiple numaplane-controller in one namespace?

Right, we may need to revert that change in future, but if you remove the namespace label in this PR then no need to open an issue for it. The revert will be tracked by no downtime upgrade feature, Thanks!


Subscribe not constantly pulling data

Subscription doesnt update when changes are made
here is my code:
return Subscription<Map<String, dynamic>>(
      'MyQuery',
      r'''
        query MyQuery {
          teams {
              id
              name
            }
          }
      ''',
      builder: ({dynamic loading, dynamic payload, dynamic error}) {
        if (error != null) {
          return Text(error.toString());
        }

        if (loading == true) {
          return Center(
            child: const CircularProgressIndicator(),
          );
        }
        return Text(payload.toString());
      },
    );`

The subscription widget only works with with graphql subscriptions such as you can use with apollo, which have a special schema type:
https://github.com/zino-app/graphql-flutter/blob/dd7e6bf3cc689d59a24b8171225433ec97ab1f18/examples/starwars/lib/reviews/review_subscription.dart#L12-L16


Some way to avoid having to wrap things in an if

It's slightly annoying that, even for commands that have no body, I still have to wrap the entire body in an if statement. For the common case where you only want to handle fully-formed valid commands, maybe an API like
#[derive(Parser, ConsoleCommand)]
struct DoSomething {
  arg: u32
}

fn do_something_system(
  In(command): In<DoSomething>,
  resource: Res<SomeResource>
) {
  todo!()
}

would work. The in-parameter can then be inferred so you wouldn't need to pass it again in add_console_command. Could also allow In(command): In<Result<DoSomething>> if you want to handle parse errors in your system.

Hi, it's not really possible to avoid the condition within the system, since the SystemParam needs to always return something, in the case that the system run but no command was available, or in the case of the parse error, we need to be able to provide some variant of the type.
We could provide another type of SystemParam, let's also call it In, which would operate over a different type of event, let's call it ParsedConsoleCommand which would always be a valid command, but then it is always possible the system runs in a state where no such events are available, meaning it would need to panic. And if it were not to panic, we'd need an Option involved, turning the signature into In<Option<T>> which does not really help the problem.
We could (maybe idk if it's sound) avoid panics by separating the command systems into "parsed" and "unparsed" command systems with the appropriate guarantess via run conditions, but that adds a whole lot of complexity and I am not convinved the benefits outweigh the cons in this case.


Network Connection is unable to get to speed test

I am running magic mirror on my Pi 400
It has been running without a problem for over a year.  I  made no changes.
I noticed that the network connection module is not loading any more and I see the following:
[2023-01-16 18:09:59.160] [ERROR]  Whoops! There was an uncaught exception...
[2023-01-16 18:09:59.161] [ERROR]  Error: No server found, verify your proxy/network
at gotServers (/home/pi/MagicMirror/modules/MMM-NetworkConnection/node_modules/speedtest-net/index.js:621:25)
at /home/pi/MagicMirror/modules/MMM-NetworkConnection/node_modules/speedtest-net/index.js:107:16
at /home/pi/MagicMirror/modules/MMM-NetworkConnection/node_modules/speedtest-net/index.js:301:7
at Parser. (/home/pi/MagicMirror/modules/MMM-NetworkConnection/node_modules/xml2js/lib/parser.js:303:18)
at Parser.emit (events.js:200:13)
at SAXParser.onclosetag (/home/pi/MagicMirror/modules/MMM-NetworkConnection/node_modules/xml2js/lib/parser.js:261:26)
at emit (/home/pi/MagicMirror/modules/MMM-NetworkConnection/node_modules/sax/lib/sax.js:624:35)
at emitNode (/home/pi/MagicMirror/modules/MMM-NetworkConnection/node_modules/sax/lib/sax.js:629:5)
at closeTag (/home/pi/MagicMirror/modules/MMM-NetworkConnection/node_modules/sax/lib/sax.js:889:7)
at SAXParser.write (/home/pi/MagicMirror/modules/MMM-NetworkConnection/node_modules/sax/lib/sax.js:1436:13)
[2023-01-16 18:09:59.166] [ERROR]  MagicMirror will not quit, but it might be a good idea to check why this happened. Maybe no internet connection?
[2023-01-16 18:09:59.166] [ERROR]  If you think this really is an issue, please open an issue on GitHub: https://github.com/MichMich/MagicMirror/issues
Node Version: I just upgraded to node 10.23 (along with  upgrading NPM to 6.14) and restarted and still get the same problem
MagicMirror² Version: 2.13
Steps to Reproduce: use NPM start
Expected Results: module should start up and show ping, upload and download speed
Actual Results: Mirror just shows "loading" where this module should display
Configuration: config.js attached
config.txt

This repo is only for issues with mm core or the default modules. We don't support foreign modules here, please open an issue in the gitgub repository of the module or ask in the forum (Link on main page of this repo).


Fixes #480. Add missing alt attribute that breaks build

Issue: https://github.com/vercel/next-learn/issues/480
Otherwise it breaks the build on vercel when following Connect and deploy your project on Chapter 6

Thank you @ernestd 🙏🏼


Use INET string to determine protocol for rule destination

Fixes #1453

Pull Request Test Coverage Report for Build 096cddfd6f3e8bfbcf0ef4f5b6e4c719fdd36223-PR-1454

2 of 2 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.1%) to 66.508%

Totals

Change from base Build 0bcabb64ca88feeb999ea59c0490130b1539fc52:
0.1%

Covered Lines:
1674

Relevant Lines:
2517

💛  - Coveralls

Thank you!