HBASE-28107 Limit max count of rows filtered per scan request.

Detail: https://issues.apache.org/jira/browse/HBASE-28107

@bbeaudreault hi, could you review code and provide some suggestions if you are available? Thanks!

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 25s
Docker mode activated.

Are you sure this is necessary? I agree that heavily filtered scans can cause problems. As of recently, this should be mostly mitigated with the improvements around blockBytesScanned (HBASE-27227, HBASE-27532, HBASE-27558).
So with those jiras, a heavily filtered scan will have a high volume of blockBytesScanned. This will cause those scans to checkpoint more often due to max scan size limits, so they won't hold up RPC handlers. You can then use quotas to limit these scans, so they don't have to be hard failed instead slowed down.
I prefer that approach over this one because in my experience failing a request is an extreme response which can have consequences for users. Fixing the scan requires figuring out why and pushing code to production, which is time consuming and in the meantime the user queries are failing. Another big problem with filtered rows is it doesn't necessarily happen right as you deploy a new scan workload. It may work fine for a while but over time rows get written that dont match your filters, so one day your scan just starts failing due to the limits.
What do you think? Do you want to give the above jiras a try?

@bbeaudreault hi, Thank you very much for your reply. I roughly looked at above jiras, they are indeed very useful. But I found a scenario that seems not to be covered? If the data that needs to be filtered only exists in the memstore, can the user's scan request be restricted? Besides, I think my implementation is relatively simple, maybe it can also be an option for users to quickly kill heavily filtered scan requests?
What do you think? cc @Apache9

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 26s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+1 :green_heart:
mvninstall
3m  9s
master passed

+1 :green_heart:
compile
2m 29s
master passed

+1 :green_heart:
checkstyle
0m 34s
master passed

+1 :green_heart:
spotless
0m 39s
branch has no errors when running spotless:check.

+1 :green_heart:
spotbugs
1m 25s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 33s
the patch passed

+1 :green_heart:
compile
2m 23s
the patch passed

+1 :green_heart:
javac
2m 23s
the patch passed

-0 :warning:
checkstyle
0m 32s
hbase-server: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
9m 31s
Patch does not cause any errors with Hadoop 3.2.4 3.3.6.

+1 :green_heart:
spotless
0m 39s
patch has no errors when running spotless:check.

+1 :green_heart:
spotbugs
1m 30s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m  8s
The patch does not generate ASF License warnings.

31m 46s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/3/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5428

Optional Tests
dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile

uname
Linux 418a6fb9063e 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 6cb15b14d4

Default Java
Eclipse Adoptium-11.0.17+8

checkstyle
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt

Max. process+thread count
77 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/3/console

versions
git=2.34.1 maven=3.8.6 spotbugs=4.7.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Are you sure this is necessary? I agree that heavily filtered scans can cause problems. As of recently, this should be mostly mitigated with the improvements around blockBytesScanned (HBASE-27227, HBASE-27532, HBASE-27558).
So with those jiras, a heavily filtered scan will have a high volume of blockBytesScanned. This will cause those scans to checkpoint more often due to max scan size limits, so they won't hold up RPC handlers. You can then use quotas to limit these scans, so they don't have to be hard failed instead slowed down.
I prefer that approach over this one because in my experience failing a request is an extreme response which can have consequences for users. Fixing the scan requires figuring out why and pushing code to production, which is time consuming and in the meantime the user queries are failing. Another big problem with filtered rows is it doesn't necessarily happen right as you deploy a new scan workload. It may work fine for a while but over time rows get written that dont match your filters, so one day your scan just starts failing due to the limits.
What do you think? Do you want to give the above jiras a try?

@bbeaudreault hi, Thank you very much for your reply. I roughly looked at above jiras, they are indeed very useful. But I found a scenario that seems not to be covered? If the data that needs to be filtered only exists in the memstore, can the user's scan request be restricted? Besides, I think my implementation is relatively simple, maybe it can also be an option for users to quickly kill heavily filtered scan requests? What do you think? cc @Apache9

Yea currently it doesn't handle the memstore, but we are hoping to add that in the future. Since this feature here is optional, I don't have a problem with adding it. I just think in general hard failures like this are hard to react to in production, so hopefully disabled by default.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 11s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
2m 21s
master passed

+1 :green_heart:
compile
0m 39s
master passed

+1 :green_heart:
shadedjars
4m 40s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 23s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 13s
the patch passed

+1 :green_heart:
compile
0m 40s
the patch passed

+1 :green_heart:
javac
0m 40s
the patch passed

+1 :green_heart:
shadedjars
4m 42s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 24s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
219m 56s
hbase-server in the patch passed.

240m 23s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5428

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 078d5b249ed3 5.4.0-153-generic #170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 6cb15b14d4

Default Java
Temurin-1.8.0_352-b08

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/3/testReport/

Max. process+thread count
4687 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/3/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 38s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
2m 45s
master passed

+1 :green_heart:
compile
0m 48s
master passed

+1 :green_heart:
shadedjars
4m 50s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 25s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 38s
the patch passed

+1 :green_heart:
compile
0m 47s
the patch passed

+1 :green_heart:
javac
0m 47s
the patch passed

+1 :green_heart:
shadedjars
4m 47s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 25s
the patch passed

_ Other Tests _

-1 :x:
unit
228m 45s
hbase-server in the patch failed.

251m 41s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5428

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux bfa4dd5f0876 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 6cb15b14d4

Default Java
Eclipse Adoptium-11.0.17+8

unit
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/3/testReport/

Max. process+thread count
4452 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/3/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Are you sure this is necessary? I agree that heavily filtered scans can cause problems. As of recently, this should be mostly mitigated with the improvements around blockBytesScanned (HBASE-27227, HBASE-27532, HBASE-27558).
So with those jiras, a heavily filtered scan will have a high volume of blockBytesScanned. This will cause those scans to checkpoint more often due to max scan size limits, so they won't hold up RPC handlers. You can then use quotas to limit these scans, so they don't have to be hard failed instead slowed down.
I prefer that approach over this one because in my experience failing a request is an extreme response which can have consequences for users. Fixing the scan requires figuring out why and pushing code to production, which is time consuming and in the meantime the user queries are failing. Another big problem with filtered rows is it doesn't necessarily happen right as you deploy a new scan workload. It may work fine for a while but over time rows get written that dont match your filters, so one day your scan just starts failing due to the limits.
What do you think? Do you want to give the above jiras a try?

@bbeaudreault hi, Thank you very much for your reply. I roughly looked at above jiras, they are indeed very useful. But I found a scenario that seems not to be covered? If the data that needs to be filtered only exists in the memstore, can the user's scan request be restricted? Besides, I think my implementation is relatively simple, maybe it can also be an option for users to quickly kill heavily filtered scan requests? What do you think? cc @Apache9

Yea currently it doesn't handle the memstore, but we are hoping to add that in the future. Since this feature here is optional, I don't have a problem with adding it. I just think in general hard failures like this are hard to react to in production, so hopefully disabled by default.

Thank you very much for your suggestion, I have added it to the RegionScannerLimiter doc comments.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 24s
Docker mode activated.

@Apache9 @bbeaudreault hi, I have changed code a bit, with the limitation of the maximum filtered rows for each scan request, and allowing users to optionally kill the scan request. Could you review code for me if you are available?

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+1 :green_heart:
mvninstall
2m 56s
master passed

+1 :green_heart:
compile
2m 23s
master passed

+1 :green_heart:
checkstyle
0m 33s
master passed

+1 :green_heart:
spotless
0m 39s
branch has no errors when running spotless:check.

+1 :green_heart:
spotbugs
1m 22s
master passed

_ Patch Compile Tests _

-1 :x:
mvninstall
1m 13s
root in the patch failed.

-1 :x:
compile
0m 13s
hbase-server in the patch failed.

-0 :warning:
javac
0m 13s
hbase-server in the patch failed.

-0 :warning:
checkstyle
0m 32s
hbase-server: The patch generated 1 new + 11 unchanged - 0 fixed = 12 total (was 11)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

-1 :x:
hadoopcheck
1m 16s
The patch causes 10 errors with Hadoop v3.2.4.

-1 :x:
hadoopcheck
2m 32s
The patch causes 10 errors with Hadoop v3.3.6.

+1 :green_heart:
spotless
0m 38s
patch has no errors when running spotless:check.

-1 :x:
spotbugs
0m 13s
hbase-server in the patch failed.

_ Other Tests _

+1 :green_heart:
asflicense
0m  7s
The patch does not generate ASF License warnings.

14m 26s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/6/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5428

Optional Tests
dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile

uname
Linux 352f7c0d7c51 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 865a59547b

Default Java
Eclipse Adoptium-11.0.17+8

mvninstall
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/6/artifact/yetus-general-check/output/patch-mvninstall-root.txt

compile
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/6/artifact/yetus-general-check/output/patch-compile-hbase-server.txt

javac
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/6/artifact/yetus-general-check/output/patch-compile-hbase-server.txt

hadoopcheck
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/6/artifact/yetus-general-check/output/patch-javac-3.2.4.txt

hadoopcheck
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/6/artifact/yetus-general-check/output/patch-javac-3.3.6.txt

spotbugs
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/6/artifact/yetus-general-check/output/patch-spotbugs-hbase-server.txt

Max. process+thread count
79 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/6/console

versions
git=2.34.1 maven=3.8.6 spotbugs=4.7.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 38s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
2m 40s
master passed

+1 :green_heart:
compile
0m 47s
master passed

+1 :green_heart:
shadedjars
4m 50s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 24s
master passed

_ Patch Compile Tests _

-1 :x:
mvninstall
1m 19s
root in the patch failed.

-1 :x:
compile
0m 17s
hbase-server in the patch failed.

-0 :warning:
javac
0m 17s
hbase-server in the patch failed.

-1 :x:
shadedjars
3m 19s
patch has 10 errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 25s
the patch passed

_ Other Tests _

-1 :x:
unit
0m 16s
hbase-server in the patch failed.

15m 44s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5428

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 271b3d38aa45 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 865a59547b

Default Java
Eclipse Adoptium-11.0.17+8

mvninstall
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/6/artifact/yetus-jdk11-hadoop3-check/output/patch-mvninstall-root.txt

compile
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/6/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt

javac
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/6/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt

shadedjars
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/6/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt

unit
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/6/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/6/testReport/

Max. process+thread count
80 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5428/6/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 26s
Docker mode activated.

Gentle ping @Apache9  If u have time, could u help review this? Thanks!

@Apache9 @bbeaudreault hi, I have made some code modifications, could you review code and provide some suggestions if you are available? Thanks!

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 32s
Docker mode activated.


[FLINK-13543][table-planner-blink] Enable reuse forks for integration tests in blink planner

What is the purpose of the change
Enable reuse forks for integration tests in blink planner.
Brief change log
Override maven-surefire-plugin configuration in blink planner pom.xml.
Verifying this change
This change is already covered by existing tests.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Merging...


Use preview mode for tool.uv.sources

Only allow using tool.uv.sources with preview mode, the design isn't finalized yet.
Not sure what to label this, do we want a preview section and label for the release notes?

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#3277  👈
#3263
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @konstin and the rest of your teammates on  Graphite

Added the label!


Update understanding-docker.md

Modify some wrong words and sentences.

Please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "vanpire110-patch-1" git@github.com:vanpire110/docker.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

Ammending updates the existing PR. You DO NOT need to open a new one.

Thanks @vanpire110, I added my comments above

@vanpire110 you don't need to create a new pull request; if you make the changes, then git commit --amend, and do a force push, GitHub will automatically update the existing pull request

Closing, as this has been replaced by https://github.com/docker/docker/pull/18137


Missing piece in upgrade guide to 0.59

Local development builds fail with targetSdkVersion 28 because starting from Android 9, clearText support is disabled (https://developer.android.com/training/articles/security-config#CleartextTrafficPermitted). It is not mentioned anywhere in the docs, maybe we could include it somewhere?
To make sure the app is able to communicate with metro bundler, you need to add android:usesCleartextTraffic="true" to your AndroidManifest.xml file so it should finally look something like:
 <application
      android:name=".MainApplication"
      //...
      android:icon="@mipmap/ic_launcher"
	  android:usesCleartextTraffic="true" <-- add this
    // ....
>

If you don't add this, your app would build and launch but would fail with the following screen:

Throws the same error for me even after adding android:usesCleartextTraffic="true"in AndroidManifest.xml

@ehsan680 did you rebuild the app?

Hey all 👋 this was documented here: https://facebook.github.io/react-native/docs/integration-with-existing-apps#network-security-config-api-level-28 (android tab)
I'll close this for now but I'll see if I can get this added to the release notes/upgrade guide also.


Temperature Sensor: LWM2M objects 3303 created

Hi everyone,
Working with wakaama client implementation of the LWM2M and Fiware Orion (of course using IoTAgent). I already have raspberry Pi sensor collecting temperature and stores in SQLite DB.
The scenario I want implement is to have the IoTAgent observes the wakaama LWM2M client (I mean wakaama client send the data to the IoTAgent.
I already created the 3303 object in wakaama/example/client/object_temperature.c then build, and I can confirm the client connects to the IoTAgent as far IoT logs (see below).
time=2019-01-24T12:51:15.270Z | lvl=DEBUG | corr=3e578c67-f589-4348-afaf-225509e9c787 | trans=3e578c67-f589-4348-afaf-225509e9c787 | op=IoTAgentNGSI.MongoDBDeviceRegister | srv=n/a | subsrv=n/a | msg=Looking for device with id [raspiSensorTV]. | comp=IoTAgent
time=2019-01-24T12:51:15.299Z | lvl=DEBUG | corr=n/a | trans=n/a | op=LWM2MLib.Registration | msg=Registration request ended successfully
time=2019-01-24T12:51:15.349Z | lvl=DEBUG | corr=n/a | trans=n/a | op=LWM2MLib.InformationReporting | msg=Observing value from resource /3303/0/0 in device [1]

But unfortunately, no are measure is sent by the client. Is anything wrong with my object_temperature.c? Code reproduced:
#include "liblwm2m.h"
#include "lwm2mclient.h"

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <ctype.h>
#include <time.h>
#include <sqlite3.h>

//Resource Id's
#define RES_SENSOR_VALUE  5700
#define RES_SENSOR_UNITS  5701

typedef struct _prv_instance_
{
  struct _prv_instance_ * next;
  uint16_t     shortID;
  double       temp;
   char        unit[10];
} prv_instance_t;

static uint8_t prv_temperature_read(uint16_t instanceId,
                               int * numDataP,
                               lwm2m_data_t ** dataArrayP,
                               lwm2m_object_t * objectP)
{
   prv_instance_t * targetP;
   int i;
   sqlite3 *db;
   sqlite3_stmt *res;
   char temperature_val[30];

targetP = (prv_instance_t *)lwm2m_list_find(objectP->instanceList, instanceId);
if (NULL == targetP) return COAP_404_NOT_FOUND;

fprintf(stderr, "----------------- Entering in prv_temperature ----------------\n");
   // connect to the backend
int rc = sqlite3_open("../../urbansense.sqlite3", &db);

if (rc != SQLITE_OK) {
   fprintf(stderr, "Cannot open database: %s\n", sqlite3_errmsg(db));
   sqlite3_close(db);
   return 1;
   }
   else {
   fprintf(stderr, "Connection successful.\n" );
   }
rc = sqlite3_prepare_v2(db, "SELECT temperature_data FROM basic_environment ORDER BY ID DESC LIMIT 1", -1, &res, 0);
if (rc != SQLITE_OK) {
  fprintf(stderr, "Failed to fetch data: %s\n", sqlite3_errmsg(db));
  sqlite3_close(db);
  return 1;
  }
rc = sqlite3_step(res);
if (rc == SQLITE_ROW) {
  sprintf(temperature_val, "%s", sqlite3_column_text(res, 0));
  fprintf(stdout, "Temperature value ==== : %s\n", sqlite3_column_text(res, 0));
  sqlite3_finalize(res);
  sqlite3_close(db);
  }
 if(*numDataP == 0)
  {
     *dataArrayP = lwm2m_data_new(1);
     if (*dataArrayP == NULL) return COAP_500_INTERNAL_SERVER_ERROR;
     *numDataP = 1;
     (*dataArrayP)[0].id = RES_SENSOR_VALUE;
     (*dataArrayP)[1].id = RES_SENSOR_UNITS;
   }
 for (i = 0; i < *numDataP; i++)
  {
     switch((*dataArrayP)[i].id)
     {
      case RES_SENSOR_VALUE:
          lwm2m_data_encode_float(targetP->temp, *dataArrayP +1);
           break;
      case RES_SENSOR_UNITS:
          lwm2m_data_encode_string(targetP->unit, *dataArrayP +1);
          break;
      default:
          return COAP_404_NOT_FOUND;
     }
  }
 return COAP_205_CONTENT;
}

lwm2m_object_t * get_object_temperature()
{
    /*
     * The get_object_temperature function create the object itself and return a pointer to the structure that represent it.
     */
    lwm2m_object_t * temperatureObj;

    temperatureObj = (lwm2m_object_t *)lwm2m_malloc(sizeof(lwm2m_object_t));

    if (NULL != temperatureObj)
    {
        memset(temperatureObj, 0, sizeof(lwm2m_object_t));

        /*
         * Assigns it's unique ID 3303
         */
        temperatureObj->objID = LWM2M_TEMPERATURE_OBJECT_ID;

        /*
         * and its unique instance
         *
         */
        temperatureObj->instanceList = (lwm2m_list_t *)lwm2m_malloc(sizeof(lwm2m_list_t));
        if (NULL != temperatureObj->instanceList)
        {
            memset(temperatureObj->instanceList, 0, sizeof(lwm2m_list_t));
        }
        else
        {
            lwm2m_free(temperatureObj);
            return NULL;
        }

        temperatureObj->readFunc     = prv_temperature_read;
        temperatureObj->userData = lwm2m_malloc(sizeof(prv_instance_t));

    }

    return temperatureObj;
}

void free_object_temperature(lwm2m_object_t * objectP)
{
    if (NULL != objectP->userData)
     {
       lwm2m_free(objectP->userData);
       objectP->instanceList = NULL;
      }
    if(NULL != objectP->instanceList)
    {
      lwm2m_free(objectP->instanceList);
      objectP->instanceList = NULL;
    }

    lwm2m_free(objectP);
}

@sbertin-telular @dnav
I'm very sorry to write on this issue again, but I'm still having trouble. I would appreciate if you could kindly take a look at my code to see what I'm doing wrong. I ignored the database transaction stuff, instead get temperature value change only. To simplify my research, I used wakaama files as they are, then add following:
Temperature object as above (now modified) in /example/client/object_temperature.c reproduced below
#include "liblwm2m.h"
#include "lwm2mclient.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>

// Resource Id's:
#define RES_SENSOR_VALUE    5700
#define RES_SENSOR_UNITS    5701

typedef struct
{
    uint26_t    shortID;
    float       temp;
    char        unit[10];
    unsigned long timestamp;
} temperature_data_t;

static uint8_t prv_set_value(lwm2m_data_t* dataP,
                           temperature_data_t* tempData)
{
    uint8_t ret = COAP_205_CONTENT;
    switch (dataP->id)     // temperature resourceId
    {
    case RES_SENSOR_VALUE:
        lwm2m_data_encode_float(tempData->temp, dataP);
        break;
    case RES_SENSOR_UNITS:
        //not interested in temperature unit value
        ret = COAP_405_METHOD_NOT_FOUND;
        break;
    default:
        ret = COAP_404_NOT_FOUND;
        break;
    }

    return ret;
}

static uint8_t prv_temperature_read(uint16_t objInstId,
                                 int*  numDataP,
                                 lwm2m_data_t** tlvArrayP,
                                 lwm2m_object_t*  objectP)
{
    int     i;
    uint8_t result = COAP_500_INTERNAL_SERVER_ERROR;
    temperature_data_t* tempData = (temperature_data_t*)(objectP->userData);

    // defined as single instance object!
    if (objInstId != 0) return COAP_404_NOT_FOUND;

    if (*numDataP == 0)     // full object, readable resources!
    {
        uint16_t readResIds[] = {
                RES_SENSOR_VALUE,
                RES_SENSOR_UNITS
        }; // readable resources!

        *numDataP  = sizeof(readResIds)/sizeof(uint16_t);
        *tlvArrayP = lwm2m_data_new(*numDataP);
        if (*tlvArrayP == NULL) return COAP_500_INTERNAL_SERVER_ERROR;

        // init readable resource id's
        for (i = 0 ; i < *numDataP ; i++)
        {
            (*tlvArrayP)[i].id = readResIds[i];
        }
    }

    for (i = 0 ; i < *numDataP ; i++)
    {
        result = prv_set_value((*tlvArrayP)+i, tempData);
        if (result != COAP_205_CONTENT) break;
    }

    return result;
}

lwm2m_object_t * get_object_temperaturen()
{
    lwm2m_object_t * temperatureObj;

    temperatureObj = (lwm2m_object_t *)lwm2m_malloc(sizeof(lwm2m_object_t));

    if (NULL != temperatureObj)
    {
        memset(temperatureObj, 0, sizeof(lwm2m_object_t));

        temperatureObj->objID = LWM2M_TEMPERATURE_OBJECT_ID;

        // and its unique instance
        temperatureObj->instanceList = (lwm2m_list_t *)lwm2m_malloc(sizeof(lwm2m_list_t));

        if (NULL != temperatureObj->instanceList)
        {
            memset(temperatureObj->instanceList, 0, sizeof(lwm2m_list_t));
        }
        else
        {
            lwm2m_free(temperatureObj);
            return NULL;
        }

        temperatureObj->readFunc    = prv_temperature_read;
        temperatureObj->userData    = lwm2m_malloc(sizeof(temperature_data_t));

        // initialize private data structure containing the needed variables
        if (NULL != temperatureObj->userData)
        {
            temperature_data_t* data = (temperature_data_t*)temperatureObj->userData;
            data->shortID = 0;
            data->temp = 0.0;
            data->unit = "Cel";
            data->timestamp   = time(NULL);
        }
        else
        {
            lwm2m_free(temperatureObj);
            temperatureObj = NULL;
        }
    }

    return temperatureObj;
}

void free_object_temperature(lwm2m_object_t * object)
{
    lwm2m_list_free(object->instanceList);
    lwm2m_free(object->userData);
    lwm2m_free(object);
}

Now creates a set_temperature_value() in lwm2mclient.c in which I call the lwm2m_resource_value_changed code below
void set_temperature_value(lwm2m_context_t * contextP, lwm2m_object_t * objectP,
                          double value)
{

    temperature_data_t * targetP;

    targetP = (temperature_data_t *)(objectP->instanceList);

    if (targetP != NULL)
    {
        targetP->x_value = value;

        lwm2m_uri_t joystickUri = {

            (LWM2M_URI_FLAG_OBJECT_ID|LWM2M_URI_FLAG_INSTANCE_ID|LWM2M_URI_FLAG_RESOURCE_ID),

            OBJECT_ID_joystick, targetP->shortID, JOYSTICK_X_VALUE };

        lwm2m_resource_value_changed(contextP, &joystickUri);

    }

}

Yet, no measure is observed. What I am doing worng here please?

I wouldn't even expect this to compile. You're mixing temperature and joystick in set_temperature_value.

It was a copy-paste mistake. I corrected with what I actually have written.

Still has copy/paste errors, but I get the idea.
Back in your first message I see in the IoT Agent log "Observing value from resource /3303/0/0". The resource that should be observed is /3303/0/5700.

I should check with Agent log again. Just wanted to be sure I'm on right track. Do I have to make a call to set_temperature_value to initiate observation (if so, from òbject_temperature.c)? It seems I only define set_temperature_value` without being called anywhere.

The observation is initiated by the agent, not any code on the device. In order to notify the agent of changes after it observes the temperature you will need to call set_temperature_value when the temperature changes. I would recommend the set_temperature_value function be in object_temperature.c (so there is no need to expose temperature_data_t) and be called from lwm2mclient.c.

Still has copy/paste errors, but I get the idea.
Back in your first message I see in the IoT Agent log "Observing value from resource /3303/0/0". The resource that should be observed is /3303/0/5700.

I can confirm the Agent is still observing inappropriate resource /3303/0/0 instead of /3303/0/5700 despite that I declared the resource id in my object_temperature.c
// Resource Id's:
#define RES_SENSOR_VALUE    5700
#define RES_SENSOR_UNITS    5701

What am I missing?

You're missing something on the IoT Agent side. I'm not familiar with it, so can't offer any guidance.

You're missing something on the IoT Agent side. I'm not familiar with it, so can't offer any guidance.

This one too is a guide. Got to check with IoTAgent. Issue closed.


Fix crash when new storage location is added

Any event putting inventory into a storage location crashed if the location was created after the most recent snapshot event. This fixes it!

(Though I would like to see the events added to the staging refresh too!)

@cielf I've updated the PR to add events to the seeds. In general I think it's easier to follow now since it's not manually updating inventory, it just goes through the appropriate service classes.

Uh-oh.... I thought this was passing -- Did something happen?

@dorner Looks like adding that in broke a few things.   I'd be good with putting the seed change into a separate PR.   We do need it, but not as urgently as the original change.


Documentation Gap: GRU recurrent weights and layer connection

Hi,
I searched for explanations about how the layers of LSTMs/GRUs are connected to each other, and how or if the layer's cells are connected to each other. Another question is how the first layer is connected to the input.
I found some older issues in the repo about that. I will list and comment them later.
First I want to show you my test model with 3 GRU cells with 30 inputs, and 1 Dense layer with 30 outputs which I used to figure out how many weights the model gets and how they are used:
model.add(GRU(3,                       return_sequences=False,                       input_shape=(30, 1),                       name="GRU_1",                       use_bias=True))  model.add(Dense(30,                       activation=ACTIVATION,                       name="OUTPUT_LAYER"))
When I use get_weights to get the weights of a layer I get some matrixes.
the first one is the kernel, the second one the recurrent, the third one the bias.
This is the plot I made:

source code of keras GRU:
736: self.kernel = self.add_weight(shape=(self.input_dim, self.units * 3).... ...)
741: self.recurrent_kernel = self.add_weight(shape=(self.units, self.units * 3)....  ...)
749: self.bias = self.add_weight(shape=(self.units * 3,),.... ...)
that explains the 3 weight matrices and the number of weights in it.
So the kernel weights are the input weights, 33 makes sense here. One input for each gate of each cell, right?
But why do they do unit * unit * 3 weights for the recurrent layer? The 3 is the number of gates. Because I used GRU, there is a 3. If I would use LSTM, then we get 4 (1 more gate)
In issue 4149 the people said that the cells aren't connected to each other on the same layer. But when I plot the weights of my first layer (the GRU layer) It looks like they are connected to each other. What else should be the reason for 3(33) weights for the recurrent layer? When only one cell is recurrent to ifself it would be 3*3 weights.
In 4149 it's also stated that the layers are fully connected, which seams to be correct.
I made some more plots for different GRU unit counts, and saw that the weights are independent to the input/output shape (because i always use 1 feature dim)
Thats correct, the source says: self.input_dim = input_shape[2].
In issue 2673 they said that the first layer is not fully connected to the input. I think, that this is corrent,..
But how are my 30 inputs mapped to this 9 input weights? With 3 inputs it would make sense to have 3 or 9 weights (for each gate input)... but with 30 inputs?
In the original paper  they show on page 9, that ALL inputs are connected to ALL gates. I think that this is also done in keras.
Are the inputs then  applied like:

1:  input[0:3]
2:  input[1:4]
3: ...
n:  input[n-3:,n]  ?

BR

Surprisingly little web-info on this topic; my findings:
Kernel weights    = [input_dim] x [num_gates *  layer_dim]
Recurrent weights = [layer_dim] x [num_gates * layer_dim]
Bias weights     = ________[1] x [num_gates * layer_dim]
num_gates = 3 for GRU, = 4 for LSTM. layer_dim = # of hidden units
Keras GRU gates: UPDATE (z), RESET (r), NEW* (h)-- IMG1
It's often of interest to know which gate the weights correspond to; in your example, layer_dim=3, and you see 3*3=9 columns for each weight matrix; which three belong to, say, the update gate? Digging further into the source code -- IMG2 -- the weights appear to be aligned with order of declaration:
[UPDATE -- hidden_00] [UPDATE -- hidden_01] [UPDATE -- hidden_02] | [RESET -- hidden_00] ...
[UPDATE -- hidden_10] [UPDATE -- hidden_11] [UPDATE -- hidden_12] | [RESET -- hidden_01] ...
[UPDATE -- hidden_20] [UPDATE -- hidden_21] [UPDATE -- hidden_22] | [RESET -- hidden_02] ...

I'm not familiar with TensorFlow's graph build syntax, but Keras' model.layers[].get_weights() orders weights exactly in order of source code declaration; can be verified further by printing individual weights and comparing against a heatmap plot.
Fine inquiry.

IMG1

IMG2


how to add or use tfx python module for catkin?

i download a repository  i am using catkin_workplace when ever i run python code
python simulation_one_motion_queue.py
every time error comes tfx not found as;
raise ResourceNotFound(name, ros_paths=self._ros_paths)
rospkg.common.ResourceNotFound: tfx
ROS path [0]=/opt/ros/indigo/share/ros
ROS path [1]=/home/mudasser/p/src
ROS path [2]=/opt/ros/indigo/share
ROS path [3]=/opt/ros/indigo/stacks

Please do not open duplicate tickets for this (#814). As already mentioned there please ask questions on https://answers.ros.org. Please also include enough information for others to understand / reproduce your problem (e.g. show or link to the Python code which actually raise the exception).

they are not answering this problem you can see i have given enough info

Please post the link to your question on answers.ros.org for people to find it from this link.
From the information provided here I can't see what your code is actually doing hence can't recommend anything to help. So more information / context might help to "attract" answers to your question.
But even if your question hasn't been answered yet cross posting is not appropriate.

Please post the link to your question on answers.ros.org for people to find it from this link.
From the information provided here I can't see what your code is actually doing hence can't recommend anything to help. So more information / context might help to "attract" answers to your question.
But even if your question hasn't been answered yet cross posting is not appropriate.
Please also see the support guidelines in the wiki: http://wiki.ros.org/Support


expose driver from media source

Instead of only exposing the properties from the driver I'm exposing the driver itself. To ensure we can reconnect to cameras I'll need the label from driver.Info().Label. This is needed for https://viam.atlassian.net/browse/DATA-597. Its PR is https://github.com/viamrobotics/rdk/pull/1644.

+1 to this entire PR. I almost recreated it myself (https://github.com/kharijarrett/gostream/tree/InfoFromSrc)


Fix required prop warning in settings tab

This pr fixes required prop warning in "settings" tab

Thanks for your PR! However, this seems to be an issue in vuejs/ui instead. I'll take a look at it.


ETERNAL LOADING [BUG]

Describe the bug
downloaded two days ago, played, today I go again and see how the game page endlessly loads
not the game itself is endlessly installing, but its page
below is a screenshot
Steps to Reproduce

Launched Hydra
Opened the game page
I see the page loading forever

Expected behavior
a request and a plea to find out what is wrong and how to fix it
Screenshots

Operating System
Windows 10 Pro
Hydra Version
v2.0.3 "Leviticus"
Additional Information
No response

Should be fixed by now. Could you confirm?


Action Recognition -- Separating the encoder and the decoder

The Inference Engine (IE) action recognition sample takes  separated encoder and decoder pre-trained models as input arguments. On the other hand, when we export the Pytorch model using the instructions provided, we get a single encoder-decoder architecture. I have two main questions:

Is there any way to export a Pytorch model into Openvino encoder model and Openvino decoder model separately so that they can be plugged into the action recognition IE sample?
Is there any way to train just the decoder part of the architecture?

@vadimadr can you, please, comment?

Hi, @Dolokhow!

You can export encoder and decoder models separately. To do this you need to provide *-vtn_encoder or *-vtn_decoder in --model. Note that if you use IE sample for inference, you must provide normalization and reverse channel order options during conversion of encoder in mo.py, decoder model does not require any pre-processing.
There is no option to train encoder-only or decoder-only, but you can set zero learning rate to the parameters that you want to freeze. To do this you need to insert regex for parameter names you want to freeze in trainable_parametrs() method of corresponding model. This, however, won't save you any memory usage (pytorch still saves gradients). Hence if you want to fine-tune decoder and save some GPU memory you can try to wrap non trainable parts of the architecture in with torch.no_grad()

Hi @vadimadr,
Thank you for a very quick response. I've just tested what you have suggested and I have just two more questions before I close this issue. I'll also be adding a couple of comments and clarifications so that others can repeat this process more easily, please correct me if I am wrong somewhere.
Questions:

For conversion from ONNX -> VINO format of the encoder, you've said that one needs to provide normalization and reverse channel order options. The reverse channel option is --reverse_input_channels, can you give me the explicit flag name for the normalization option?
[Read after reading Clarifications section below]: If one wanted to convert the entire architecture (encoder+decoder) into a single VINO .xml + .bin pair, given the fact that --reverse_input_channels option does not work, how should one solve that situation? Specifically, if I understand correctly, if channels are not reversed then that would have significant impact on the accuracy of the model, and if there is no way to apply this operation on the entire ONNX architecture (encoder+decoder) then the only way to use the model is to separate it into the encdoer part and the decoder part?

Clarifications:

Using --reverse_input_channels will not work when trying to convert the entire architecture (encoder+decoder) defined in a single ONNX file to VINO format. The error raised is below:

[ ERROR ]  Reverse input channels are not applied -- appropriate convolutions were not found

The --input_shape for the entire (encoder+decoder) architecture when coverting from ONNX to VINO is '[1,16,3,224,224]' as is given in the instructions. However, when converting just the extracted ONNX encoder, the --input_shape is [1,3,224,224].
The --input_shape when converting the ONNX decoder to VINO format is [1,16,512] if you use --clip-size 16 during training.

The options for normalization are: --mean_value and --scale_values. @AlexanderDokuchaev could you please provide the correct line with exact values?
You are correct about impact of incorrect pre-processing on model accuracy.  You can always skip the pre-processing  options in mo.py and rely on normalization in your sample code. In this case, you need to implement the same operations that you used to prepare batch during the training in your code (e.g. subtract men, divide by scaling factor, convert color to BGR). But I hope the bug with pre-processing of 3D inputs will be fixed soon.

@AlexanderDokuchaev Could you please give me the exact values, or tell me where to look for them?

Sorry, --mean_values=[103.53,116.28,123.675] --scale_values=[58.395,57.12,57.37499999999999]

Thank you all for your help! I am closing the issue.


Datepicker - format input once it loses focus

Proposal:
Format the date in the datepicker input once it loses focus.
What is the expected behavior?
Here's a demonstration (Google Flights):

What is the current behavior?
The datepicker doesn't format the input unless selected from the pop-up directly.
Which versions of Angular, Material, OS, TypeScript, browsers are affected?
2.0.0-beta.10

This should work now, you can see it in action on the docs site: https://material.angular.io/components/datepicker/overview. Type "Jan 2018" in the input, move focus away, observe that it changes to "1/1/2018"


Fix counter_culture option name of README.md

I'm using it but option name is invalid.
So I fixed option name of Usage of README.md.

Thank you!


Publish redirects to live

Docs Build status updates of commit 63c987a:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

.openpublishing.redirection.json
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 3b35031:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

.openpublishing.redirection.json
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


ci(graphql): tests for staging variant

Description
Set-up testing for staging variant of sui-graphql-rpc:

Split E2E tests into stable and staging variants (each in their own sub-directories -- existing tests are all stable). Only run the stable tests by default, and additionally run the staging tests if the staging feature is enabled.
Generate a separate staging schema and snapshot test.
Add a CI step to run GraphQL tests with the staging flag enabled.

Test plan
sui$ cargo nextest run --profile ci --features staging  \
  -E 'package(sui-graphql-rpc)'                         \
  -E 'package(sui-graphql-e2e-tests)'

And also check CI.

Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

[ ] Protocol:
[ ] Nodes (Validators and Full nodes):
[ ] Indexer:
[ ] JSON-RPC:
[x] GraphQL: Introduce staging.graphql. This schema includes changes that are being developed and tested but have not been productionised yet.
[ ] CLI:
[ ] Rust SDK:
[ ] REST API:

Yes there is -- I will be adding it in #19365


Error: [libs/sql-schema-describer/src/mssql.rs:251:30] Couldn't parse default value

Command: prisma introspect
Version: 2.11.0
Binary Version:<PHONE_NUMBER>2e47bdcec77a2f1e7c1fb83a463918
Report: https://prisma-errors.netlify.app/report/12038
OS: x64 darwin 18.7.0
JS Stacktrace:
Error: [libs/sql-schema-describer/src/mssql.rs:251:30] Couldn't parse default value
    at ChildProcess.<anonymous> (/Users/tbrown/sites/ez/ezprisma/node_modules/@prisma/cli/build/index.js:4947:28)
    at ChildProcess.emit (events.js:315:20)
    at ChildProcess.EventEmitter.emit (domain.js:482:12)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:275:12)

Rust Stacktrace:
   0: backtrace::backtrace::trace
   1: backtrace::capture::Backtrace::new
   2: user_facing_errors::Error::new_in_panic_hook
   3: user_facing_errors::panic_hook::set_panic_hook::{{closure}}
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::begin_panic_handler::{{closure}}
   6: std::sys_common::backtrace::__rust_end_short_backtrace
   7: _rust_begin_unwind
   8: core::panicking::panic_fmt
   9: core::option::expect_failed
  10: <sql_schema_describer::mssql::SqlSchemaDescriber as sql_schema_describer::SqlSchemaDescriberBackend>::describe::__describe::{{closure}}::{{closure}}
  11: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  12: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  13: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  14: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  15: <futures_util::compat::compat03as01::Compat<Fut> as futures::future::Future>::poll
  16: <futures::future::map::Map<A,F> as futures::future::Future>::poll
  17: <futures::future::either::Either<A,B> as futures::future::Future>::poll
  18: <futures::future::lazy::Lazy<F,R> as futures::future::Future>::poll
  19: futures::future::chain::Chain<A,B,C>::poll
  20: <futures::future::then::Then<A,B,F> as futures::future::Future>::poll
  21: <futures::future::map::Map<A,F> as futures::future::Future>::poll
  22: <futures::future::either::Either<A,B> as futures::future::Future>::poll
  23: futures::task_impl::std::set
  24: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  25: tokio::runtime::thread_pool::ThreadPool::block_on
  26: tokio::runtime::context::enter
  27: tokio::runtime::handle::Handle::enter
  28: introspection_engine::main
  29: std::sys_common::backtrace::__rust_begin_short_backtrace
  30: std::rt::lang_start::{{closure}}
  31: std::rt::lang_start_internal
  32: _main

@divyenduz would you ping me when this error bubbles up again? We now get the default value printed which will help us debug.


locale="ms" is not taking any effect

https://jsfiddle.net/mjvdbfct/
Your Environment

flatpickr version used: 4.6.13
Browser name and version: Chrome Version 102.0.5005.61
OS and version: Mac

Translation for Malaysian language not working.

Same here, using locale='zh-tw' ended up "Error: flatpickr: invalid locale" message in browser console.

Same here, using locale='zh-tw' ended up "Error: flatpickr: invalid locale" message in browser console.

Updated the PR to fix your reported issue as well.
@chmln Can you please check this PR once.
https://github.com/flatpickr/flatpickr/pull/2738

Same here, using locale='zh-tw' ended up "Error: flatpickr: invalid locale" message in browser console.

We can use locale='zh_tw' in order to get this working now. @jh961202


chebi roles

CHEBI has a role hierarchy:

CHEBI:50906 ! role

CHEBI:24432 ! biological role
CHEBI:33232 ! application
CHEBI:51086 ! chemical role

These are linked from structures via (their own) has-role relation. It could be useful to bring these in.
The challenge is:

it's not clear these are truly roles in the BFO sense
we need to 'knock out' CHEBI role and rewire the subclasses to whatever appropriate BFO class

Related: a while ago I tried axiomatizing all the CHEBI roles https://github.com/cmungall/chiro

There is not a clear use case for ChEBI roles to be in core.


REST: Fix PATCH method

Description
Use a converter to reconcile the PATCH body with the existing instance.
Fixes https://issues.redhat.com/browse/RHOAIENG-8534
How Has This Been Tested?

Refer to reproducers in the Jira.
Merge criteria:

[x] The commits and have meaningful messages; the author will squash them after approval or will ask to merge with squash.
[x] Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
[x] The developer has manually tested the changes and verified that the changes work

cc @tarilabs

/lgtm and thank you so much @isinyaaa
before approving one remaining question: since this was raised by UI team, would you think an update to RobotFramework with a user-story dedicated to using Patch with this PR, on in a subsequent PR, please?

that's surely a good idea, will look into it asap :)

Added a simple test for the use-case. wdyt @tarilabs ?


Release App to Playstore or something similar

We have access to a Playstore-Account now.

Created a project board on GitHub.


Maxlength 8 and user now can just type numbers

Hey @gabrielduessmann just tested it and works very well :ok_hand:  Thanks a lot for your contribution, really appreciate it :smile: :tada:


Upgrade to Lucene 9.9.0

This commit upgrades Anserini to use Apache Lucene 9.9.0.
This is just the basic changes to get Anserini onto Lucene 9.9.0. Further work can be done to leverage improvements like, say int8 scalar quantization, etc.
closes #2288

Thanks @ChrisHegarty !
I've got a major code dump queued up, referenced here: https://github.com/castorini/anserini/issues/2288#issuecomment-1840729978
Let me do this after? Thanks!

No problem. Take your time @lintool. I just wanted to get this into a clean state so that we could better understand the impact of the upgrade on the source base - which appears small.

Closing in favor of #2302


fix: parse removal messages for ZWLR nodes correctly

16-bit node IDs weren't parsed correctly when removing a node, causing Z-Wave JS to think that the node is still around until restart.

@zwave-js-bot automerge


New specialized parser

This is a refactor to the existing parser extension in Zephir. It takes advantage of the preferred memory management for php5 (heap allocation) and php7 (stack allocation) for zvals. The approach allows us to reduce the boilerplate and the complexity of supporting both Zend Engines in a single parser. The drawback is that for now both parsers must be maintained.
Compiling Phalcon on Linux 64bits/php 7.0.5 before this refactor requires: 318MB of memory
Compiling Phalcon on Linux 64bits/php 7.0.5 after this refactor now requires: 153MB of memory
While 153MB is still high, this refactor reduces approximately 45%-55% memory overall usage when compiling a large Zephir project.
Also, my conclusion is that I must follow this approach in Phalcon in PHQL/Volt parsers.

@andresgutierrez
On the long term, we should aim for PHP_VERSION_ID or similar
insteadof php5/php7 since you'll also need a recompilation of the parser
if you update PHP (or it get's updated)

Yes, that's the approach I will pursue on PHQL, if everything works fine I'll port it back to Zephir

I don't know why all of you are writeing size od debugged version of extensions ? You shouldn't never use debugged php and phalcon on production enviroment and for me it's obvious. Non debug version is 4.6 mb.

This is not the size of the executable but the peak memory used in RAM to generate the code.

Oh okay. Then it's great idea to refractor this, beacause it can be tough sometimes to compile phalcon on vps with 512 mb ram and without ability to create swap.

Now when I'm trying to build phalcon on php5 I'm receiving:
sh build_linux.sh
[   52s] parser..lemon: Can't open this file for reading.
[   52s] error: couldn't generate parser
[   52s] error: Bad exit status from /var/tmp/rpm-tmp.YgjNB6 (%build)

https://build.opensuse.org/package/live_build_log/home:mruz/php5-phalcon/openSUSE_13.2/x86_64

@mruz If you are running build_linux.sh manually now you have to pass the parser you want to build: build_linux.sh php5 and build_linux.sh php7

Ok, thanks. I'll try.
----- Reply message -----
Od: "Andy Gutierrez"<EMAIL_ADDRESS>Do: "phalcon/zephir"<EMAIL_ADDRESS>DW: "Mariusz Łączak"<EMAIL_ADDRESS>"Mention"<EMAIL_ADDRESS>Temat: [phalcon/zephir] New specialized parser (#1248)
Data: czw., kwi 28, 2016 17:08
@mruz If you are running build_linux.sh manually now you have to pass the parser you want to build: build_linux.sh php5 and build_linux.sh php7
—You are receiving this because you were mentioned.Reply to this email directly or view it on GitHub

We need to do it where ? Just zephir root/parser/parser ? And then just zephir build as always ?

I guess we have to tell phalcon to use php7 files the same way ?

Wojtek, only when you rebulid the parser manually, eg.
cd zephir/parser/parser
sh build_linux.sh php5
cd ..
phpize
./configure --enable-zephir_parser
make

Well already tested it and if i delete parser.php5 or parser.php7 from phalcon it causing error so i guess is just working.


Make direct AlertRule option

Get possible calculation times from list in API.
This will be added to /taxonomy/lists and will have a (first) option of 0 minutes.
This should be the default alert calculation time, it triggers on every incoming device data having the alertrule measurement.

done @ app-test


Redirects with .asp extension are downloading

I have an old site with a /file.asp path, that I'd like to redirect to /file. However, the browser is interpreting the .asp as a file and downloading it instead of redirecting.

This is somewhat, the intended behavior, if unfortunate, in that you could have a .asp file on your site which you'd like other to download. The best workaround is to set the redirect to /file.asp/, which will really create /file.asp/index.html, which should serve the redirect with an HTML document type.


Unconsistent confidence score

Hello,
I'm using spacy-langdetect with Python 3.7.6 on Fedora 31 with an Intel(R) Core(TM) i7-7500U CPU
I'm using it to detect the language of sentences in a text (which contains both french and english sentences)
I've ran into a strange problem when executing
english_nlp_model = spacy.load("en_core_web_sm")
english_nlp_model.add_pipe(LanguageDetector(), name='language_detector', last=True)

doc = english_nlp_model(my_text)
for sent in doc.sents:
    print(f"-> {sent._.language} | {sent._.language}")

For most of the results the two values are quite similar (for a float) and I did not show it in this excerpt, but you can see that for some of the values, it's quite different:
-> {'language': 'fr', 'score': 0.7142844894042983} | {'language': 'fr', 'score': 0.9999969204393719}
-> {'language': 'it', 'score': 0.9999932252642086} | {'language': 'it', 'score': 0.7142820534070277}
-> {'language': 'fr', 'score': 0.9999980859423542} | {'language': 'fr', 'score': 0.9999945446302498}
-> {'language': 'nl', 'score': 0.8571380236556732} | {'language': 'nl', 'score': 0.9999957626562326}

Any ideas on what could cause this?
Have a great day
EDIT: some languages change as well, but it's quite rare (imo it's just because there is less languages possible, so it's not as visible, but both are affected by the same problem)
-> {'language': 'fi', 'score': 0.5714279768513012} | {'language': 'de', 'score': 0.5714273200401079}

EDIT2: I tried to print it 3 times just to be sure, and it's often different in all of them:
-> {'language': 'fr', 'score': 0.9999958708843195} | {'language': 'fr', 'score': 0.7142804945594394} | {'language': 'fr', 'score': 0.8571393420155614}

EDIT3: Just for information, my text is mostly composed of notes taken quickly, which obviously doesn't help detection but helps highlighting the problem I think, I've tried with normal english text and the scores do change a little bit, but it's not as visible as in mine.

I have also noted inconsistent scores.  The code I was using went like this:
def get_lang_detector(nlp, name):
    return LanguageDetector()

eng_nlp = spacy.load("en_core_web_lg")
Language.factory("language_detector", func=get_lang_detector)
eng_nlp.add_pipe('language_detector', last=True)
...........................................
docs = self.ent_nlp.pipe(sentences)
for doc in list(docs):
  doc = self.eng_nlp(sentence)
  langs = []
  scores = []
  sentence_threshold = 0.90
  for i in range(5):
      lang = doc._.language["language"]
      score = doc._.language.["score"]
      langs.append(lang)
      scores.append(score)
  oops = False
  for i in range(5):
      oops = oops or langs[0] != langs[i]
  if not oops and lang != "en":
      sign_0 = math.copysign(1, scores[0]-sentence_threshold)
      sign_i = math.copysign(1, scores[i]-sentence_threshold)
      oops = oops or sign_0 != sign_i
  if oops:
      print langs and scores

I also do the same thing for the paragraph using " ".join(sentences)
I get this (PMID is a PubMed paper ID.)  This complaint is about one of the "sentences" in the abstract of that paper, and then about the entire abstract.
PMID 134435 Sentence oops:  Langs: ['en', 'en', 'en', 'de', 'en']
PMID 134435 Sentence oops:  Scores: [0.8571390606714735, 0.8571390295157906, 0.9999958831576141, 0.5714292463415062, 0.7142828647104207]
PMID 134435 Paragraph oops.  Langs: ['en', 'en', 'en', 'en', 'de']
PMID 134435 Paragraph oops.  Scores: [0.7142831346993775, 0.7142851057380656, 0.7142834277378337, 0.7142846675001696, 0.7142847751645527]

and this.  There is only trouble with the score in this one, but it's all over the map:
PMID 55295 Paragraph oops.  Langs: ['en', 'en', 'en', 'en', 'en']
PMID 55295 Paragraph oops.  Scores: [0.8571390387840068, 0.9999961964510007, 0.8571401002100192, 0.7142855836268951, 0.8571390751698879]

I understand that outcome may be determined by a random start in fitting a model, but this business of getting a different answer every time I check the result is more than a little counterintuitive.
Spacy is correct to be confused abut 134435, since half the abstract is in English and the other half is in German.  However in this case the "sentence" as defined in the input data was the complete "paragraph".  Both were just over 200 words.  I would have expected a little more consistency.  It doesn't matter here because the score is relatively low, but I wish it would be more consistent in cases that do matter.
The abstract for 55295 is only 38 words, and while it is in English, there are a lot of medical terms.  I could understand a somewhat low score, but the score here varied between 0.71 and 0.999996 the 5 times I read it.
This was with Spacy 3.0.6 and spacy-langdetect 0.1.2
As I was writing this, I had a sudden thought.  This is being done in a multiprocessing environment.  The creation of eng_nlp is done after each individual process has started up.  Is it possible that something is shared when it shouldn't be?  It doesn't seem like it, but I thought it worth mentioning.  The processes are not spawned by langdetect, for whatever that's worth.

The problem is that this package is mostly a wrapper around the langdetect one. It uses the factory to create a lang detector but the later init it each time the function detect_langs is called. So since, the wrapper calls this function one time for all the doc, one time for each sentence of the doc and one time for each token of each sentence, it init constantly.  For example, for a doc of 1 sentence of 10 tokens, it does it 12 times. That process has no sense at all, but for langdetect it does since you call detect_langs only one time on a text.
So yes, the seed is not the same all the time. But, you can set the seed on the factory of langdetect or manually can set the seed of the package. It's not clean but here is a quick fix
from langdetect import DetectorFactory
from spacy_langdetect import LanguageDetector

def get_lang_detector(nlp, name):
    DetectorFactory.seed = 42 # we set the seed of for the langdetect package
    return LanguageDetector()

To make it simpler, I've released a package base on the fork of this project.

Unfortunately setting the seed only solves half the problem.  Once you set the seed, you get the same score every time for the same text.  However, if you change the seed you can get a very different score and language for the same text.  If you make the central loop in my example something like this (yes I know there was a stray period in the original):
  for i in range(5):
      DetectorFactory.seed = 42 + 10*i
      d = doc._.language
      langs.append(d["language"])
      scores.append(d["score"])

you can still get answers that are very different.  Setting the seed just makes them the same every time you run the code.  (Doing this also eliminates the "it changes every time I look at it" problem if you don't set the seed.)
I have finally settled on setting on asking for N different results using different seeds. (N=5 mostly.)  If the languages differ I throw the text away as being possibly multi-lingual.  If the languages are all the same but the scores are too low (look at the min or the median) I also throw the text away.  If the languages are all the same and the scores are high enough I keep the text.  This isn't fool proof, but it's not horrible.  It gets fooled by paragraphs that are mostly recitations of the names of organic compounds, and by short English paragraphs that contain the names/addresses of hospitals in non-English speaking countries.

Do you have a reason to do the execution 5 times?
It's sure that if you have a different seed since it is a statistical model, it will give different results each time. But, if you fix the seed FOR ALL the execution, then it will always be the same. Also, I don't think langdetect is that good... Like Hello is not classified as English...

5 is just a fairly arbitrary small number.  3 is just a little too small.
Why multiple tries?  Setting the seed just hides the inconsistent behavior.  If I use two different seeds, I can sometimes get two very different scores, e.g. 0.75xxxx and 0.9999xxxx.  Assuming langdetect gives a consistent answer for the language, the score is a random variable if the seed is chosen at random.  Think about using the median to summarize that distribution.  (The median is better than the mean given the skewness of the distribution.)  A sample mean is a good estimate of the distribution mean.  You can use this to do a test of how well the text matches the principal language.  I'm not sure how easy it would be to make probability statements.


Add missing contrib modules / pl extensions to alpine image

The readme on dockerhub says:

When using the Alpine variants, any postgres extension not listed in postgres-contrib will need to be compiled in your own image (again, see github.com/postgis/docker-postgis for a concrete example).

However, not all extensions in contrib are available. The language extension plperl, plpython and pltcl, as well as related contrib modules (e.g. hstore_*, jsonb_*, ...) are missing.
Only in very rare cases, when the postgres version matches the one distributed in the current alpine distribution, one can install e.g. plpython via apk add postgresql-plpython3 (with a bit of moving files around to their proper location). However, this is not sustainable: It currently works for pg 13.x on alpine 3.14, but not for pg 14.x anymore.
Compiling those extensions from source kind of defeats the purpose of deriving from this image in the first place.
Currently, those extensions seem to be left out on purpose:
https://github.com/docker-library/postgres/blob/a11e908fb50cacb6192d1db93dcf911bc1a724e6/Dockerfile-alpine.template#L104-L110
I investigated the potential increase in image size a little bit. It seems like the biggest chunk results from additional dependencies in perl, python and tcl. However, those can be avoided, if we build the extension, but then remove the dependencies for them. This means the above mentioned extensions will only be usable after installing the dependencies in a derived image again - but that's already a huge step forward.
As an example, using plpython would be as simple as:
FROM postgres:14-alpine
RUN apk add --no-cache python3

I looked at image size in 4 cases:

master: current master branch
+PLfull: compiled language extension, but kept the dependencies
+PL: compiled language extensions, but removed dependencies
+PL+auth: same as before, but also compiled --with-krb5, --with-gssapi and --with-ldap

The last one was just because those were commented out, too, and I wondered what it would cost to enable them now.

postgres
master
+PLfull
+PL
+PL+auth

14.1-alpine
195
280
196
199

13.5-alpine
192
276
192
195

12.9-alpine
190
275
191
194

11.14-alpine
188
273
189
192

10.19-alpine
73
157
73
76

9.6.24-alpine
37
92
51
54

The data shows that indeed the biggest chunk comes from perl, python and tcl. The increase in image size for the other two cases however is really small.
I suggest to enable both the language extensions as well as --with-krb5, --with-gssapi and --with-ldap, but leave out the 3 major dependencies to keep image size small. This will make the alpine image much more usable.

I think this is going to lead to a non-trivial image size increase.  See also https://github.com/docker-library/postgres/issues/384 for a useful previous discussion on this topic.

... @yosifkit kindly pointed out that the table in your message covers this :facepalm:
Looking at the builds themselves, 14-alpine goes from ~195MB up to ~199MB. :sweat_smile:

@yosifkit yosifkit merged commit bafaf65 into docker-library:master 3 minutes ago

:tada:
Just a question regarding releases: Am I right to assume that this change will only be released on docker hub once new postgres minor versions are released?

Nope, it'll go up as soon as we decide to make (and merge) a PR over to official-images. See also https://github.com/docker-library/postgres/tree/bafaf65a2356a05810c12a03d4412e1003cd1bb8#see-a-change-merged-here-that-doesnt-show-up-on-docker-hub-yet.

Nope, it'll go up as soon as we decide to make (and merge) a PR over to official-images.

Ok, cool. I'll watch that closely.

See also https://github.com/docker-library/postgres/tree/bafaf65a2356a05810c12a03d4412e1003cd1bb8#see-a-change-merged-here-that-doesnt-show-up-on-docker-hub-yet.

Yeah, I went through that (and the FAQ the link points to) already, but the change in that example is about a version update. That creates new tags, so it all kinds of makes sense. I couldn't tell from the FAQ what that means for "non-version-update" changes, though. Would those recreate the latest version tag? I always assumed that, once I reference a "full-version" image (e.g. postgres:14.1-alpine, right now), I would always get the same image. But apparently those are replaced, too?
I probably just missed that part in the FAQ.

Yeah, any tag is mutable. We rebuild images when either a PR is merged to official-images that changes its build context (Dockerfile, shell scripts, etc) or any of its parent images can cause a rebuild (e.g. our rebuilding of Debian every ~30 days causes all dependent images to rebuild). Even when rebuilding for just a Dockerfile change, docker build cache is used as much as possible to minimize change between images where possible.
If anyone needs immutable references, then they should use the content addressable digest (postgres@sha256.....). This is a reference to that exact image build and will always be the same image.


Consultant Gets Error "Cannot read property 'status' of undefined" When Opening Any Project

One particular Consultant is getting this error on every engagement page he tries to visit. I cannot reproduce in live, but I have observed on screen share. User says he's been experiencing issues like this for two weeks though bug report is from last four days +-.
User: https://cordfield.com/users/5c5dfbd41e560f7f00c0a60f
Engagements: (All assigned to him, but first two on the list are 5d2de693abc8f499add79463 and 5d3ee7e2abc8f41d07e331c4 )
Error screenshot:

Log Rocket Log: https://app.logrocket.com/x6d22y/cord-prod/s/4-86dd486a-24ee-43db-a466-fb3726a9a268/0?filters=%255B%257B%2522id%2522%253A%2522SJOG95uwd%2522%252C%2522type%2522%253A%2522email%2522%252C%2522operator%2522%253A%257B%2522name%2522%253A%2522is%2522%252C%2522type%2522%253A%2522IS%2522%252C%2522hasStrings%2522%253Atrue%252C%2522autocompleteEnabled%2522%253Atrue%257D%252C%2522strings%2522%253A%255B%2522paul_schmidt%2540tsco.org%2522%255D%257D%255D&fromTab=latest

Support conversation: https://github.com/SeedCompany/management/issues/164

I've reproduced locally with his account in a prod-like environment. Investigating further.

From Slack:
We have two roles in the db, BibleTranslationLiaison and Liaison. I’m guessing we replaced the former with the latter and forgot to update the role in the db. It doesn’t look like BibleTranslationLiaison is implemented as a role in the API. I updated the user’s role to Liaison in a test environment and am able to view projects now, but I want to confirm that this is the same role. There are only 9 users who have this role in the db.
I checked the closed issues where we remapped old roles and these two weren’t mentioned.

I've changed his role in production to Liaison, which seems to be correct.
Confirming with user.

Checking with user again

I plan to close this with the end of the sprint as we've had no further feedback from the user.

Hi @ParkeBrown @CarsonF @parashara1
I've verified on the latest develop e8bb4ac and can't reproduce this issue.
See video:
https://user-images.githubusercontent.com/5212585/134857276-c5ef9e21-b530-4767-945f-3c833416c5c2.mp4


Feat/hightouch suggestions

Hi Firebolt team!
I was working on the Firebolt integration and I was adding a couple of missing features to our system, but I've figure out you might want to have them in the SDK. These are all suggestions. Let me know what would you like to add and we could check together.
NOTE: I don't know how pagination works with your REST API, so that part is broken, but it retrieves the first page.

Amazing contribution! Thanks! I will research how to do pagination and we really liked this features around waiting for engine to start and database service.


Errors after upgrading to v1.0.0

No problems with running this plugin in v0.3.3 but updateing to v1.0.0 this comes up when calling:
error:
[14:34:29] Starting nightwatch...

[Ilb2] Test Suite
=================
    POST  /wd/hub/session - ECONNREFUSED
Error: connect ECONNREFUSED <IP_ADDRESS>:4444
   An occurred error while retrieving a new session: "connect ECONNREFUSED <IP_ADDRESS>:4444"
       at Socket.socketErrorListener (_http_client.js:387:9)
_________________________________________________

TEST FAILURE: 1 error during execution 0 tests failed, 0 passed. 1.218s

 × ilb2

  An occurred error while retrieving a new session: "connect ECONNREFUSED <IP_ADDRESS>:4444"
       at Socket.socketErrorListener (_http_client.js:387:9)

   SKIPPED:
   - Open and Close lightbox page
   - Caption
   - Deep links
   - Dynamic add
   - Manual trigger
   - Navigation

[14:34:31] '<anonymous>' errored after 2.18 s
[14:34:31] Error in plugin "gulp-nightwatch"
Message:
    nightwatch exited with code 5
Details:
    domainEmitter: [object Object]
    domain: [object Object]
    domainThrown: false

[14:34:31] 'night:js' errored after 3.32 s

my config:
{
    "src_folders" : [ "./test/src" ],
    "output_folder" : "./test/reports",

    "selenium" : {
        "start_process" : true,
        "server_path" : "./test/resources/selenium-server-standalone-3.13.0.jar",
        "log_path" : false,
        "port" : 4444,
        "cli_args" : {
            "webdriver.chrome.driver" : "./test/resources/chromedriver.exe"
        }
    },

    "test_settings" : {
        "default" : {
            "launch_url" : "http://localhost",
            "selenium_port"  : 4444,
            "selenium_host"  : "localhost",
            "silent": true,
            "desiredCapabilities": {
                "browserName" : "chrome",
                "javascriptEnabled" : true,
                "acceptSslCerts" : true
            }
        }
    }
}

@rejas
As far as I saw the logs you pasted, a selenium server dose not seem to be running in your environment. Please access to http://localhost:4444 with a browser, and check whether a selenium server works fine.

Isnt gulp-nightwatch starting the selenium server itself? My gulp workflow only starts a dev server at http://localhost:8080/docs/

Isnt gulp-nightwatch starting the selenium server itself?

Yes. To be exact, gulp-nightwatch starts a selenium server via nightwatch when start_process set to true in nightwatch.js.
But, I've found that gulp-nightwatch v1.0.0 does not start a selenium server. Sorry, this is a bug.  I'll fix it.

Glad to know its a bug and not my incompetence for once :-)

@rejas
I've fixed this issue, and released as v1.0.1.

@tatsuyafw Works! Thx a lot!


The Resources Store for Tracks and Samples is not working.

Good morning, I'm learning to use osu! Framework, and I tried to learn to use his audio. But I was looking at the documentation and it says:
Game will automatically create the following stores that point to the respective paths for each store:

/Textures (TextureStore)
/Tracks (AudioManager.Track)
/Samples (AudioManager.Sample)
/Shaders (ShaderManager)
/Fonts (FontStore)

Okay, I set up the resource project folders for me to put an amazing theme song for my game.
But when I put:
        private ISampleStore sampleManager;
        private ITrackStore trackManager;

        [BackgroundDependencyLoader]
        private void load(AudioManager audio)
        {
            AddRangeInternal(new Drawable[]
            {
                ...its work...
            });

            sampleManager = audio.Samples;
            trackManager = audio.Tracks; // and
            Track myTrack = trackManager.Get("mainmenu");
            myTrack.Looping = true; // the error is caused by that line
        }

The game hits me right off the bat like this:
- System.NullReferenceException: 'Object reference not set to an instance of an object.'

- myTrack was null.

I tried all the ways, check if the folders are correct, the names of the files, but nothing works!

OBS: This is a fan game for Super Mario World! :3

To clarify, what is your project structure? Do you have a separate project for the Game itself and a separate project for the resources?
If so, did you make sure to register the appropriate DllResourceStore, as done in the FlappyDon template?
https://github.com/ppy/osu-framework/blob/e6fa465fb7b20e7e1a3b0d91e7bbec882c958ba6/osu.Framework.Templates/templates/template-flappy/FlappyDon.Game/FlappyDonGameBase.cs#L22-L23

Yes, I did, and there is a resource project that the template itself generates from the game.
Here the file where I imported the resource project:
    public abstract class PlatformGameBase : osu.Framework.Game
    {
        private TextureStore textures;
        private PixelTextureStore pixelTextures;
        private LevelAudioManager levelAudioManager;

        private DependencyContainer dependencies;

        [BackgroundDependencyLoader]
        private void load(FrameworkConfigManager config)
        {
            Resources.AddStore(new DllResourceStore(PlatformResources.ResourceAssembly));

            textures = new TextureStore(Textures);
            pixelTextures = new PixelTextureStore(Textures);

            dependencies.Cache(textures);
            dependencies.Cache(pixelTextures);

            var tracks = new ResourceStore<byte[]>();
            tracks.AddStore(new LevelAudioStore());

            levelAudioManager = new LevelAudioManager(Host.AudioThread, tracks, new ResourceStore<byte[]>())
            { EventScheduler = Scheduler };

            dependencies.Cache(levelAudioManager);

            config.GetBindable<FrameSync>(FrameworkSetting.FrameSync).Value = FrameSync.Unlimited;

            Host.Window.Title = "Platform Game";
        }

        protected override IReadOnlyDependencyContainer CreateChildDependencies(IReadOnlyDependencyContainer parent)
            => dependencies = new DependencyContainer(base.CreateChildDependencies(parent));
    }

Did you try explicitly specifying the extension of the audio file (as in mainmenu.mp3)? I'm pretty sure it's required.

As I said, I tried anyway, not even putting the extension with the name it doesn’t stick.
I also tried to check the number of tracks that the Audio Manager took, I put it like this:
Logger.Log(message: $"COUNT: {audio.Tracks.GetAvailableResources().ToList().Count});

and returned zero.

do you have a repository i can check out or something? it's counterproductive for me to do remote debugging over github comments.

I don't have a repository with this, but I will send it now.

https://github.com/zenpakuh/platform @bdach

I'm not seeing an exception after cloning here, the track is returned (and even started, if I start it) correctly.

Uai, how so? Why doesn't mine return?

I cloned the repository, compiled the project and returned the track.

I managed to fix it! Thanks a lot for the help!


Add unit tests

As the code grows, we should make sure that we add unit tests that can make sure the code runs properly. Ideally, most of the code should be covered by unit tests. You can see in the main glue repository how to write tests:
https://github.com/glue-viz/glue
We should make use of the py.test framework for consistency.
This is not something that one specific person needs to do - rather, we should all try and start adding unit tests over the next few weeks.

Started in #12


Refine the general versioning process

This issue describes a refinement of our general versioning process. This removes possible mistakes when adding new eras, thereby simplifying the addition of new eras.
We should implement the refinements described in https://github.com/IntersectMBO/ouroboros-consensus/issues/873.

This is done.


chore(main): release taiko-client 0.35.0

:robot: I have created a release beep boop
0.35.0 (2024-09-10)
Features

taiko-client: add two more new ZK related metrics (#18043) (e43eeac)
taiko-client: remove the legacy 290 tier (#18035) (5064037)
taiko-client: update ontakeForkHeight to Sep 24, 2024 (#18047) (a1ff620)
taiko-client: update contract binding & add Proposer (#18053) (d0554a2)
taiko-client: use proveBlocks by default for post ontake blocks (#18042) (15709af)

Bug Fixes

taiko-client: use proposed at, not timestamp when fetching blob (#18055) (32d95c1)

This PR was generated with Release Please. See documentation.

:robot: Created releases:

taiko-client-v0.35.0
:sunflower:


When merging objects, even keys for invalid values should be remembered

The primary reason for this change is to support null values within compiled style objects.
In addition to that, any invalid value should also override previous values for that key. AFAIK, this is what React Native does. (Hard to say for sure since some invalid values will often throw an error).

React Native apps will often crash on invalid values anyway. That's something on the roadmap to fix.

So the intention is to change RN to not throw on invalid values and ignore them instead? (like StyleQ does right now?)

Will update with more tests shortly.

Yeah. Not throwing on invalid values is something Expo have been asking for

Yeah. Not throwing on invalid values is something Expo have been asking for

We would need to ensure that invalid values don’t behave like “null” then.


⚠ Samsung has degraded performance

In c04d108, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2113 ms

Resolved: Samsung performance has improved in 4758074.


Implement rollback --all feature to rollback all migrations

It would be super handy to be able to rollback all the migrations in one command.
Currently I either have to bin/cake migrations rollback for every migrations, or I have to rollback to a specific timestamp by doing a bin/cake migrations rollback -t 0. Suggestion is to add a --all option to the rollback command, that performs a rollback on all the migrations.
Also having a refresh command that performs a rollback --all and then migrates could be super handy for development. Even cooler would be to extend this feature with an optional --seed that also seeds the database after a rollback->migrate has been performed.

All of this seconded & would make Phinx a real joy to use. Very eager to see refresh --seed in particular, it's super useful.

I'm waiting for this PR to be finished so we can reuse the logic to add --all https://github.com/cakephp/phinx/pull/945

Eager to use this.

As per http://docs.phinx.org/en/latest/commands.html#the-rollback-command you can rollback all migrations via:
phinx rollback -e development -t 0

So this can be closed then as a solution is available.

Yeah, the timestamp rollback thing is just semantically not very nice. Thanks though.

@lomholdt Are you able to provide a PR with your semantically improved version of this enhancement?

I can give it a go. Thanks @dereuromark .


Decide approach to code splitting / routing

https://github.com/sveltejs/kit/pull/2931

Hey @juranki
I'm trying to dig into your project and to understand what's happening :)
I understand that you use one Lambda@Edge function whose handler imports the whole SvelteKit app, does SSR when necessary, otherwise just instructs CloudFront to provide the pre-rendered page.
Could you please help to understand

why did you chose Lambda@Edge over API Gateway + {proxy+} + regular Lambda pattern? Isn't it considered best practice to have many small functions?
Is this exactly the topic of this issue, to split the application into several lambdas?

Thanks! And please keep up the good work!
This is the only AWS adapter working with the latest sveltekit version out in the wild - afaik. Cool!

I understand that you use one Lambda@Edge function whose handler imports the whole SvelteKit app, does SSR when necessary, otherwise just instructs CloudFront to provide the pre-rendered page.

Correct.

why did you chose Lambda@Edge over API Gateway + {proxy+} + regular Lambda pattern? Isn't there a risk that the code reaches the Lamda@Edge quotas?

The primary reason is the mechanism Lambda@Edge provides for handling pre-rendered pages. Secondary reasons include simplicity and less network hops, thus potentially smaller latency (haven't measured though)
There is a potential for hitting the quotas. The relevant ones are response size (1 MB) and compressed size of the handler (50 MB). For most use cases those should be reasonable, although it's easy to come up with cases that could be problematic.

Isn't it considered best practice to have many small functions?

With many small functions you can possibly reduce startup time and avoid function size quota, but at the same time the number of cold starts increases. Pick your poison :)

Is this exactly the topic of this issue, to split the application into several lambdas?

Yes, and currently I think this is in some-day-maybe category. The complexity is significant and I'm not sure if it's possible to map SvelteKit routes to CloudFront behaviour patterns.

Great work on this. I'm still trying to learn Sveltekit and dive deeper into CDK, so take my points with this in mind. I am starting at the end (i.e. deployment) because this has become my natural way of thinking.

I am curious as to how would the "many functions" option mentioned be implemented? Demo app only has one Server, so can you elaborate more on that?
It could be a good idea if there's a constructor prop that let's you choose Lambda or Lambda@Edge (though, maybe after your desired robustness is achieved)

Hello @aihaddad,

SvelteKit produces javascript files that implement the server. Not all routes need all files. SvelteKit also produces dependency information between routes and files, which could be used by adapters to create separate (and potentially smaller) handlers for different routes.
Lambda@Edge provides good facilities for handling pre-rendered pages when it's deployed as origin request handler. With lambda that would be much harder. On the other hand, with lambda you can control the region it runs on. Maybe worth considering at some point, but definitely not now.

Thank you @juranki
Regarding point 2, Lambda@Edge is definitely the better/best/only option for pre-rendered static and client pages, I was referring more to SSR. Basically everything beyond line 38 where it can probably be delegated to another Lambda, and that's one where the choice is given


enable click on components when toast is displayed

Hi,
I see this is closed but I still have this problem - when the toast is displayed nothing else can be tapped. How do I configure this so that other buttons are still active while the toast is displayed?

@DigNZ to create a dismissable notification you need to set the position to bottom otherwise if it's on top or center you can't dismiss it when pressing outside the notification
(notificationPosition = NOTIFICATION_POSITION.bottom)

@DigNZ yes you're right i'm working on it already

Hello :) how about this bug? :)

I'm also facing this issue - also, it would be nice if the default behavior for toasts was to clear them when tapping.

the toast is dismissable only when it's showed in the bottom, the dismissable values by default it's false you can set it as true in your call to the toast
I just published a new version with dismissable is set to true by default
https://pub.dev/packages/motion_toast
for now when showing the toast on top or center there's no dismissable behavior support in the package but I'm working on it
thanks for your support !!

@jagged91 @PawelSzymanski89 @DigNZ I just published a new  version, this time you can dismiss the toast when it's displayed on top or center also
check this new version
https://pub.dev/packages/motion_toast

@jagged91 @PawelSzymanski89 @DigNZ I just published a new version, this time you can dismiss the toast when it's displayed on top or center also check this new version https://pub.dev/packages/motion_toast

Yeah, the new version solves this.
I think it's safe to close this Issue.
Thanks @koukibadr for the package.


Touch coordinates are wrong on iOS if UIStatusBarHidden is false

I previously posted this on the RoboVM forums, but Niklas directed me this way.
I noticed my touches are reported as being half a centimeter or so lower than where I'm really pressing. This happens with a app on iOS when UIStatusBarHidden is set to false. When the status bar is hidden, touch coordinates are correct. The offset is about the size of the status bar, so I think this isn't being taken into account somehow? Test case here.
It's barely noticeable most of the time, but, for instance, if you hit the bottom of buttons in a stage they don't respond. Easy fix by hiding it anyway.
Everything else is glorious, keep it up! :D

Not an iOS expert, so everything below is only my humble opinion :) but looking at robovm backend code - it seems you might be correct about status bar not taken into account. Unfortunately, I have no iOS devices/development tools, but it seems something like following code is missing and might fix this issue:
    if (!uiApp.isStatusBarHidden()) {
        CGRect statusBarRect = uiApp.getStatusBarFrame();
        switch(uiApp.getStatusBarOrientation()) {
        //set correct touch offset basing on status bar frame size and orientation
        }
    }

Had a look into this, but couldn't actually get the status bar to show correctly. Anyone have the config for RoboVM that shows the status bar?  (For LibGDX 1.6.0 and RoboVM 1.2 that is)

This has been fixed.


WIP enable in Marathon secrets via plugin config

Could you please add a JIRA link to the description?


Add ability to set TTL

#175 rebased  (h/t @hamptonmoore)

We're blocked on CircleCI because it's not picking this up for some reason and "Required statuses must pass before merging"...

Yea, I don't know why CircleCI isn't seeing this PR.

@SuperQ should I just ignore the errors, log them as warning or fail fast by checking and returning the error?
what would be the most suitable approach?


Exception when search product based on selected category from search bar

Bug report
Title
Exception when search based on the selected category from search bar
Issue Description
Exception when search based on the selected category
Preconditions
1. framework Version- master

Steps to reproduce
1. step1- select any category from search bar
2. step2- pass any value in search bar

Expected result
should show all the product based on selected category
Actual result

Fixed


fix: fix token URI

This fix in TieredDrop has been patched here: https://github.com/thirdweb-dev/contracts/blob/main/contracts/tiered-drop/TieredDrop.sol#L502


Unable to determine why I am getting user errors on my Event Hubs based on description

I have an Event Hub that is receiving Resource Logs from various Azure resources in the same region where the Event Hub is located. I then have a Logic App that gets messages from the Event Hub (every 60 seconds) and sends them via the HTTP(S) action to my on-prem logging solution.
Occasionally this Event Hub reports that User Errors are occurring, however I can't make sense of them because the only thing the Event Hub is dealing with are Azure platform logs, and the only entity requesting messages from the Event Hub is my Logic App that uses the Event Hub trigger and the HTTP(S) action to send the logs to my logging solution.
In this setup, what would be causing the User Errors and what do they mean? I think the name "User Errors" is misleading here because the logs involved in the Event Hub are only ones that Azure generates, and the resource requesting logs from the Event Hub is a built in Azure service (Logic App).

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: cbfaf7ee-5223-8b19-e2a4-1f2d3290f2d7
Version Independent ID: 88cfd463-58f5-6923-22c4-db901b54d6fa
Content: Monitoring Azure Event Hubs data reference - Azure Event Hubs
Content Source: articles/event-hubs/monitor-event-hubs-reference.md
Service: event-hubs
GitHub Login: @spelluru
Microsoft Alias: spelluru

@mlcooper  Since this issue is not related directly to the doc, I would recommend you to create a thread on the forums - Microsoft Q&A or Stack Overflow
Once you post your issue on forums, it will have visibility across the community which is a better suited audience for such types of issues.

The documentation should explain what a user error is in the context I've described. I am not looking for a solution as to why I am getting the error in the first place.

@spelluru Could you please look into the above issue?

@mlcooper - here's the response from our dev. sorry for the delay.
"Logic Apps crates epoch receivers and they may move receivers from one node to another depending on the service load. During those moves, ReceiverDisconnectionExceptions during epoch receiver creation are counted as user errors on the EH service side. Logic Apps probably collects the failures from EventHubClients so customer can view them in the user logs."
I will add it as a note in the article. Thanks.
#please-close


Research EIP-712 compatibility

so far, the repo should be theoretically compatible with EIP 712 signatures according ot the latest spec
to see if it's so in practice, review metamask and test
potentially have tests which use metamask's source code to sign EIP 712 msgs

https://github.com/ethereum/EIPs/blob/master/EIPS/eip-712.md


fix: allowing multi-lines in env variables - envload

Description
This PR is to fix the issue when trying to add variable with multiple lines as a value
Now, it allows doing something like this:
KEY="ValueValue
VALUEVALUE
VALUE"

and also adding more variables with no problem
KEY="ValueValue
VALUEVALUE
VALUE"
KEY2="NEWVALUE"

This is great @MateoCaicedoW Should we document envload in the docs?

This is great @MateoCaicedoW Should we document envload in the docs?

We have it there!

Nice! Thanks!