[SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples

What changes were proposed in this pull request?
Now, DirectKafkaWordCount example is not support to visit kafka using kerberos authentication. Add Java/Scala DirectKerberizedKafkaWordCount.
How was this patch tested?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Hi, @hddong . Thank you for making a PR.

Since DirectKafkaWordCount.scala and JavaDirectKafkaWordCount.java is a pair for language parity, this PR should update them together consistently.
And, just out of curiosity, why do you want to have a Kerberized example at DirectKafkaWordCount? Do you want to change another examples like StructuredKafkaWordCount.scala, too?

Hi, @hddong . Thank you for making a PR.

Since DirectKafkaWordCount.scala and JavaDirectKafkaWordCount.java is a pair for language parity, this PR should update them together consistently.
And, just out of curiosity, why do you want to have a Kerberized example at DirectKafkaWordCount? Do you want to change another examples like StructuredKafkaWordCount.scala, too?

JavaDirectKafkaWordCount.java  done.

Most of our environment is Kerberized, when we do test if sparkstreaming can read from kafka normally in some case, we need to copy one more jar from somewhere else. So, I think it will be convenient if examples can test by default.

I'm glad to change other like StructuredKafkaWordCount.scala , I will make a new PR later

@hddong Please see my public repo for examples.

ok to test

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14034/
Test PASSed.

Test build #108963 has started for PR 25412 at commit f6d4a0b.

Test build #108963 has finished for PR 25412 at commit f6d4a0b.

This patch fails Java style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108963/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14035/
Test PASSed.

Test build #108964 has started for PR 25412 at commit 5321864.

Test build #108964 has finished for PR 25412 at commit 5321864.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108964/
Test PASSed.

While I don't think we want to overcomplicate these examples by showing all the ways it could be used, this is possibly OK.

@srowen +1 not overcomplicate. Not sure what you mean this is possibly OK? OK to give example on a deprecated feature?

I would not illustrate a deprecated feature. I just mean probably OK to include something like this at all.

Well, just checked and Kafka broker 1.1.0 introduced DT so for 1.1.0- users this is the way, so I'm fine with that. On the other hand I'm going to have some comments...

@srowen @gaborgsomogyi ,SSL and SASL_SSL need too more parameters,such as ssl.truststore.location ssl.truststore.password ssl.keystore.location ssl.keystore.password... Is it a better choice to create another example file?
cc, @dongjoon-hyun
if include something like this at all is better , I will make some changes to support ssl authentication on this file.

Yep. If you want, please make a new example for Kerberos.
And, let's revert all the current changes and keep the original examples simple.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14073/
Test PASSed.

Test build #109006 has started for PR 25412 at commit 641b12b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14074/
Test PASSed.

Test build #109007 has started for PR 25412 at commit 7fdd858.

Test build #109006 has finished for PR 25412 at commit 641b12b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109006/
Test PASSed.

Test build #109007 has finished for PR 25412 at commit 7fdd858.

This patch fails Java style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109007/
Test FAILed.

Test build #109008 has started for PR 25412 at commit cb62731.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14075/
Test PASSed.

Test build #109008 has finished for PR 25412 at commit cb62731.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109008/
Test PASSed.

Test build #109023 has started for PR 25412 at commit b9937f3.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14090/
Test PASSed.

Test build #109023 has finished for PR 25412 at commit b9937f3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109023/
Test PASSed.

Test build #109034 has started for PR 25412 at commit 2c27d7b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14097/
Test PASSed.

Test build #109034 has finished for PR 25412 at commit 2c27d7b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109034/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14161/
Test PASSed.

Test build #109095 has started for PR 25412 at commit 75f9ab8.

Test build #109095 has finished for PR 25412 at commit 75f9ab8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109095/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14206/
Test PASSed.

Test build #109138 has started for PR 25412 at commit f79328b.

Thasks for @HyukjinKwon . Move the comment here.
This file uses SASL_PLAINTEXT for simplicity; however, SASL_PLAINTEXT has no SSL encryption and likely be less secure. Consider using SASL_SSL.

Test build #109138 has finished for PR 25412 at commit f79328b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109138/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14387/
Test PASSed.

Test build #109325 has started for PR 25412 at commit 1d67fc5.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14392/
Test PASSed.

Test build #109328 has started for PR 25412 at commit 10c7de7.

Test build #109325 has finished for PR 25412 at commit 1d67fc5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109325/
Test PASSed.

Test build #109328 has finished for PR 25412 at commit 10c7de7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109328/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14400/
Test PASSed.

Test build #109335 has started for PR 25412 at commit fd7198b.

Test build #109335 has finished for PR 25412 at commit fd7198b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109335/
Test PASSed.

BTW, I like this example. Let me know if this looks good to you @gaborgsomogyi and @HeartSaVioR

Test build #109483 has started for PR 25412 at commit 24c2f30.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14542/
Test PASSed.

Test build #109484 has started for PR 25412 at commit 739cdb9.

Test build #109483 has finished for PR 25412 at commit 24c2f30.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109483/
Test PASSed.

Test build #109484 has finished for PR 25412 at commit 739cdb9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109484/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14596/
Test PASSed.

Test build #109540 has started for PR 25412 at commit 197c971.

Test build #109540 has finished for PR 25412 at commit 197c971.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109540/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14597/
Test PASSed.

Ooops, just seen the scala style failure. That must be solved.

Merged to master.

LGTM

Thanks for review and your advice.

Thanks all for review and advice.
And i will add a new file for structured streaming later.

@hddong ping me if review needed...


Some minor settings fixes

@rigrig as #272 is merged now your branch has some conflicts.

Thanks - just fixed. It was only the ChangeLog :)


test fix for mysql start timeout [WIP] [DO NOT MERGE]

Here I'm testing a fix in OSA, but changing the submodule to point at
my fork of OSA where I have made a patch.  I have to do it this way
because OSA doesn't do upgrade testing of mariadb between kilo and
liberty, which explains why we're only seeing this mostly in rpco.
The fix is essentially to wait longer between mysql start attempts after doing a mariadb upgrade.
Please, feel free to recheck_upgrade on this because that's what I'm testing here.
FYI this is the OSA fix I am testing https://github.com/mancdaz/openstack-ansible/commits/mysql-start
Connects  #1201

if this passes, please continue to recheck the upgrade job over and over again

RPC-AIO-Ceph-Liberty-PR: Fail

RPC-AIO-Liberty-Upgrade-PR: Fail

@mancdaz the upgrade job has failed, the log includes the following:
16:01:26 TASK: [galera_server | Start MySQL] *******************************************
16:01:36 failed: [jrpcaioiad-a69_galera_container-d18b7d77] => {"changed": true, "cmd": "for i in {1..3}; do\n (pgrep mysqld && exit 0) || (/etc/init.d/mysql start || true)\n sleep 10\n done\n echo \"Service failed to start\"\n exit 1", "delta": "0:00:10.006611", "end": "2016-07-01 16:01:36.581570", "rc": 1, "start": "2016-07-01 16:01:26.574959", "warnings": []}
16:01:36 stdout: 574
16:01:36 1492
16:01:36 Service failed to start
16:01:36

RPC-AIO-Liberty-Upgrade-PR: Fail

RPC-AIO-Liberty-Upgrade-PR: Fail

RPC-AIO-Ceph-Liberty-PR: Fail

RPC-AIO-Liberty-Upgrade-PR: Fail

RPC-AIO-Liberty-Upgrade-PR: Fail

recheck_upgrade

RPC-AIO-Liberty-Upgrade-PR: Fail

RPC-AIO-Liberty-Upgrade-PR: Fail

recheck_upgrade

recheck_upgrade

RPC-AIO-Liberty-Upgrade-PR: Fail

recheck_upgrade

RPC-AIO-Liberty-Upgrade-PR: Fail

RPC-AIO-Liberty-Upgrade-PR: Fail

back to square one:
17:02:01 TASK: [galera_server | Start MySQL] *******************************************
17:02:01 <<IP_ADDRESS>> ESTABLISH CONNECTION FOR USER: root
17:02:01 <<IP_ADDRESS>> REMOTE_MODULE service name=mysql state=started
17:02:01 <<IP_ADDRESS>> EXEC ssh -C -v -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=120 <IP_ADDRESS> /bin/sh -c 'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python'
17:22:01 Build timed out (after 20 minutes). Marking the build as aborted.
17:22:01 Build was aborted

http://jenkins.propter.net/job/RPC-AIO/8484/consoleFull

Seems like there's a sneaky start of mariadb 10 happening when the package is being installed, even though it's mean to be denied by the policy:
17:01:34 TASK: [galera_server | Install galera packages] *******************************                                                                                17:01:34 <<IP_ADDRESS>> ESTABLISH CONNECTION FOR USER: root                                                                                17:01:34 <<IP_ADDRESS>> REMOTE_MODULE apt pkg=mariadb-client,mariadb-galera-server-10.0,galera-3,qpress,rsync,socat state=latest                                                                                17:01:34 <<IP_ADDRESS>> EXEC ssh -C -v -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=120 <IP_ADDRESS> /bin/sh -c 'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python'                                                                                17:01:56 changed: [jrpcaioiad-dd9_galera_container-96956b05] => (item=mariadb-client,mariadb-galera-server-10.0,galera-3,qpress,rsync,socat) => {"attempts": 0, "changed": true, "item": "mariadb-client,mariadb-galera-server-10.0,galera-3,qpress,rsync,socat", "stderr": "All runlevel operations denied by policy                                                                                invoke-rc.d: policy-rc.d denied execution of stop.                                                                                Configuration file '/etc/init.d/mysql'                                                                                ==> Modified (by you or by a script) since installation.                                                                                ==> Package distributor has shipped an updated version.                                                                                ==> Keeping old config file as default.                                                                                Configuration file '/etc/mysql/conf.d/mysqld_safe_syslog.cnf'
 ==> Deleted (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
 ==> Keeping old config file as default.
All runlevel operations denied by policy
invoke-rc.d: policy-rc.d denied execution of stop.
160705 17:01:45 [Note] /usr/sbin/mysqld (mysqld 10.0.26-MariaDB-1~trusty-wsrep) starting as process 18684 ...
All runlevel operations denied by policy
invoke-rc.d: policy-rc.d denied execution of start.
", "stdout": "Reading package lists...
Building dependency tree...
Reading state information...
Suggested packages:
  mailx mariadb-galera-test tinyca
The following NEW packages will be installed:
  mariadb-galera-server-10.0 qpress
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 10.4 MB of archives.
After this operation, 98.3 MB of additional disk space will be used.
Get:1 https://repo.percona.com/apt/ trusty/main qpress amd64 11-1.trusty [26.4 kB]
Preconfiguring packages ...
Fetched 10.4 MB in 0s (11.2 MB/s)
Selecting previously unselected package mariadb-galera-server-10.0.
(Reading database ... 23708 files and directories currently installed.)
Preparing to unpack .../mariadb-galera-server-10.0_10.0.26+maria-1~trusty_amd64.deb ...
Unpacking mariadb-galera-server-10.0 (10.0.26+maria-1~trusty) ...
Selecting previously unselected package qpress.
Preparing to unpack .../qpress_11-1.trusty_amd64.deb ...
Unpacking qpress (11-1.trusty) ...
Processing triggers for man-db (<IP_ADDRESS>-1ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up mariadb-galera-server-10.0 (10.0.26+maria-1~trusty) ...
Installing new version of config file /etc/logrotate.d/mysql-server ...
Installing new version of config file /etc/mysql/conf.d/tokudb.cnf ...
Setting up qpress (11-1.trusty) ...
Processing triggers for ureadahead (0.100.0-16) ...
"}

Specifically note this line:
160705 17:01:45 [Note] /usr/sbin/mysqld (mysqld 10.0.26-MariaDB-1~trusty-wsrep) starting as process 18684 ...

what are you doing there buddy?

using jenkins build 8484:
galera log: http://jenkins.propter.net/job/RPC-AIO/8491/artifact/archive/openstack/jrpcaioiad-dd9_galera_container-96956b05/galera_server_error.log
specific bit of the galera log we are interested in showing the multiple starts/stops of mysqld/innodb/wsrep: http://paste.openstack.org/show/526090/
ansible console log: http://jenkins.propter.net/job/RPC-AIO/8491/consoleFull
specific bit of ansible log showing the sneaky mysqld start at the package install time:
http://paste.openstack.org/show/526091/

RPC-AIO-Liberty-PR: Fail

RPC-AIO-Ceph-Liberty-PR: Fail

@mancdaz are you sure it's starting, the following two lines in the log suggest it was prevented
160705 17:01:45 [Note] /usr/sbin/mysqld (mysqld 10.0.26-MariaDB-1~trusty-wsrep) starting as process 18684 ...
All runlevel operations denied by policy
invoke-rc.d: policy-rc.d denied execution of start.

@git-harry I'm not sure of anything tbh, just continuing to pour through the logfiles and try and note anything odd.
However, the timing of that line does suggest that something is trying to start up at the point of install, and indeed the mysql error log shows lots of activity around then. If it was truly being denied by policy-rc.d, I would not expect it to be assigned a PID.

More notes:
In a successful startup, the following is in the mysql log:
160704 21:37:03 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
160704 21:37:03 [Note] WSREP: restore pc from disk failed

In a failed startup:
[Note] WSREP: restore pc from disk successfully

The existence of a gvwstate.dat file is evidence of a non-graceful shutdown (i.e. crash) prior to this startup attempt. So we actually want it to not find that file.
May mean nothing, who knows

RPC-AIO-Liberty-Upgrade-PR: Fail

RPC-AIO-Liberty-PR: Fail

RPC-AIO-Liberty-Upgrade-PR: Fail

So, adding a task to pgrep for mysqld and look at netstat right after the mariadb-server package is installed (bearing in mind that policy-rc.d should not allow it to be running), shows that it is indeed running (or at least the galera wsrep part of it is listening on 4567):
10:15:54 TASK: [galera_server | pgrep mysqld] ******************************************
10:15:54 <<IP_ADDRESS>> ESTABLISH CONNECTION FOR USER: root
10:15:54 <<IP_ADDRESS>> REMOTE_MODULE command pgrep mysqld
10:15:54 <<IP_ADDRESS>> EXEC ssh -C -v -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=120 <IP_ADDRESS> /bin/sh -c 'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python'
10:15:54 changed: [jrpcaioiad-831_galera_container-6d11afca] => {"changed": true, "cmd": ["pgrep", "mysqld"], "delta": "0:00:00.004601", "end": "2016-07-06 10:15:54.512153", "rc": 0, "start": "2016-07-06 10:15:54.507552", "stderr": "", "stdout": "575\n1480", "warnings": []}
10:15:54
10:15:54 TASK: [galera_server | netstat] ***********************************************
10:15:54 <<IP_ADDRESS>> ESTABLISH CONNECTION FOR USER: root
10:15:54 <<IP_ADDRESS>> REMOTE_MODULE command netstat -pntl
10:15:54 <<IP_ADDRESS>> EXEC ssh -C -v -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=120 <IP_ADDRESS> /bin/sh -c 'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python'
10:15:54 changed: [jrpcaioiad-831_galera_container-6d11afca] => {"changed": true, "cmd": ["netstat", "-pntl"], "delta": "0:00:00.007537", "end": "2016-07-06 10:15:54.596204", "rc": 0, "start": "2016-07-06 10:15:54.588667", "stderr": "", "stdout": "Active Internet connections (only servers)\nProto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name\ntcp        0      0 <IP_ADDRESS>:22              <IP_ADDRESS>:*               LISTEN      528/sshd        \ntcp        0      0 <IP_ADDRESS>:4567            <IP_ADDRESS>:*               LISTEN      1480/mysqld     \ntcp6       0      0 :::22                   :::*                    LISTEN      528/sshd        ", "warnings": []}

Pulling out the important bits:
netstat:
command: ["netstat", "-pntl"]
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 <IP_ADDRESS>:22              <IP_ADDRESS>:*               LISTEN      528/sshd
tcp        0      0 <IP_ADDRESS>:4567            <IP_ADDRESS>:*               LISTEN      1480/mysqld
tcp6       0      0 :::22                   :::*                    LISTEN      528/sshd        "

pgrep:
command: ["pgrep", "mysqld"]
"575
1480"

In an upgrade job, nodes 1,2 of the cluster are restarted, and then node 0 is restarted.  In a job where the later mariadb upgrade fails, it's useful to note that earlier when those restarts happened, the cluster was left in a bad state:
160705 16:38:22 [Warning] WSREP: No re-merged primary component found.
160705 16:38:22 [Warning] WSREP: No bootstrapped primary component found.
160705 16:38:22 [ERROR] WSREP: gcs/src/gcs_state_msg.cpp:gcs_state_msg_get_quorum():794: Failed to establish quorum.
160705 16:38:22 [Note] WSREP: Quorum results:
	version    = 3,
	component  = NON-PRIMARY,
	conf_id    = -1,
	members    = 0/3 (joined/total),
	act_id     = -1,
	last_appl. = -1,
	protocols  = -1/-1/-1 (gcs/repl/appl),
	group UUID = 00000000-0000-0000-0000-000000000000
160705 16:38:22 [Note] WSREP: Flow-control interval: [28, 28]
160705 16:38:22 [Note] WSREP: Received NON-PRIMARY.
160705 16:38:22 [Note] WSREP: New cluster view: global state: e5c8e624-42c4-11e6-b8fa-13e99514039f:8970, view# -1: non-Primary, number of nodes: 3, my index: 2, protocol version -1
160705 16:38:22 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
160705 16:38:24 [Note] WSREP: (e5c841c6, 'tcp://<IP_ADDRESS>:4567') turning message relay requesting off

In a 'good' job, the cluster gets itself back into a good state after those container restarts:
160704 21:12:10 [Warning] WSREP: No re-merged primary component found.
160704 21:12:10 [Note] WSREP: Bootstrapped primary 00000000-0000-0000-0000-000000000000 found: 3.
160704 21:12:10 [Note] WSREP: Quorum results:
	version    = 3,
	component  = PRIMARY,
	conf_id    = -1,
	members    = 3/3 (joined/total),
	act_id     = 9075,
	last_appl. = -1,
	protocols  = 0/7/3 (gcs/repl/appl),
	group UUID = acf766fa-4221-11e6-92bc-d7b4acbf1b0e
160704 21:12:10 [Note] WSREP: Flow-control interval: [28, 28]
160704 21:12:10 [Note] WSREP: Restored state OPEN -> JOINED (9075)
160704 21:12:10 [Note] WSREP: New cluster view: global state: acf766fa-4221-11e6-92bc-d7b4acbf1b0e:9075, view# 0: Primary, number of nodes: 3, my index: 1, protocol version 3
160704 21:12:10 [Note] WSREP: SST complete, seqno: 9075
160704 21:12:10 [Note] WSREP: Member 1.0 (jrpcaioiad-265_galera_container-bf6e1285) synced with group.
160704 21:12:10 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 9075)
160704 21:12:10 [Note] WSREP: Member 2.0 (jrpcaioiad-265_galera_container-9c839293) synced with group.
160704 21:12:10 [Note] WSREP: Member 0.0 (jrpcaioiad-265_galera_container-25d21a22) synced with group.
160704 21:12:10 InnoDB: The InnoDB memory heap is disabled
160704 21:12:10 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160704 21:12:10 InnoDB: Compressed tables use zlib 1.2.8
160704 21:12:10 InnoDB: Using Linux native AIO
160704 21:12:10 InnoDB: Initializing buffer pool, size = 512.0M
160704 21:12:10 InnoDB: Completed initialization of buffer pool
160704 21:12:10 InnoDB: highest supported file format is Barracuda.
160704 21:12:11  InnoDB: Waiting for the background threads to start
160704 21:12:12 Percona XtraDB (http://www.percona.com) 5.5.49-MariaDB-37.9 started; log sequence number 23397856
160704 21:12:12 [Note] Plugin 'FEEDBACK' is disabled.
160704 21:12:12 [Note] Server socket created on IP: '<IP_ADDRESS>'.
160704 21:12:12 [Note] Event Scheduler: Loaded 0 events
160704 21:12:12 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.50-MariaDB-1~trusty-wsrep'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution, wsrep_25.14.r9949137
160704 21:12:12 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
160704 21:12:12 [Note] WSREP: REPL Protocols: 7 (3, 2)
160704 21:12:12 [Note] WSREP: Service thread queue flushed.
160704 21:12:12 [Note] WSREP: Assign initial position for certification: 9075, protocol version: 3
160704 21:12:12 [Note] WSREP: Service thread queue flushed.
160704 21:12:12 [Note] WSREP: Synchronized with group, ready for connections
160704 21:12:12 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
160704 21:12:13 [Note] WSREP: (acf68560, 'tcp://<IP_ADDRESS>:4567') turning message relay requesting off

OK so it's not mariadb 10 that's starting right on install, but an old lingering mariadb 5.5 daemon that never got shut down properly from earlier. We can see this by looking at the PID we saw earlier:
netstat:
command: ["netstat", "-pntl"]
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 <IP_ADDRESS>:22              <IP_ADDRESS>:*               LISTEN      528/sshd
tcp        0      0 <IP_ADDRESS>:4567            <IP_ADDRESS>:*               LISTEN      1480/mysqld
tcp6       0      0 :::22                   :::*                    LISTEN      528/sshd        "

And looking in the mariadb log for that container:
160706  9:52:02 [Note] /usr/sbin/mysqld (mysqld 5.5.50-MariaDB-1~trusty-wsrep) starting as process 1480

Shortly after this, the cluster fails to establish quorum and goes into error state:
160706  9:52:04 [Warning] WSREP: No bootstrapped primary component found.
160706  9:52:04 [ERROR] WSREP: gcs/src/gcs_state_msg.cpp:gcs_state_msg_get_quorum():794: Failed to establish quorum.
160706  9:52:04 [Note] WSREP: Quorum results:
	version    = 3,
	component  = NON-PRIMARY,
	conf_id    = -1,
	members    = 0/3 (joined/total),
	act_id     = -1,
	last_appl. = -1,
	protocols  = -1/-1/-1 (gcs/repl/appl),
	group UUID = 00000000-0000-0000-0000-000000000000
160706  9:52:04 [Note] WSREP: Flow-control interval: [28, 28]
160706  9:52:04 [Note] WSREP: Received NON-PRIMARY.
160706  9:52:04 [Note] WSREP: New cluster view: global state: c77fc194-4354-11e6-8737-6215be425fe5:9163, view# -1: non-Primary, number of nodes: 3, my index: 1, protocol version -1
160706  9:52:04 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
160706  9:52:05 [Note] WSREP: (c77ed8c5, 'tcp://<IP_ADDRESS>:4567') turning message relay requesting off

For whatever reason, this means that the mariadb 5.5 daemon fails to shut down properly when we come to install/start mariadb 10.
Need to understand what is causing the cluster to go into error (fail to reach quorum) around the time of the container restarts at the beginning of the upgrade.

 Turns out that the part of the lxc_container_create role that adds some autodev stuff for containers in liberty, thusly:
https://github.com/openstack/openstack-ansible/blob/liberty/playbooks/roles/lxc_container_create/tasks/container_create.yml#L46-L55
causes a secret container restart, so even though my patch is shutting mysql down prior to the later explicit container restart, there is a secret container restart which causes mysql to start again. Then, when it comes to the later explicit container restart, we're basically just back to forcefully crashing the mysql daemon without a graceful stop. le sigh.


L 5.3 problem..

Hello sir,
I need a help from you. I am about to start a project in office since today. I started in L5.3 and used your modules extenstion from GitHub. The problem is, after using modules plugin, no any class is being supported in my Controllers.
Eg: Input Class not found.
I used to do projects in L 5.1  where I used pingping-sky and everything was working well. When I compared those two modules created by pingpong in 5.1 and laravel-modules in 5.3, I found that
In modules created by ping ping : there is this line
use Pingpong\Modules\Routing\Controller;
set by default in Newly created module, which is not present in the module created by laravel/modules. Do you think this is the issue?
Hoping positive solution from you ! It would be an big help if you could take some of your time to reply this message.
Thanks !!

Hello
This isn't related to this package.
You need to correctly import your namespaces into your files.


Change the generated release template to docs/README.md

cc @thockin @erictune
I also changed the first couple of releases on the page, but not many.

LGTM, Jenkins and shippable both look like flakes, OK to merge without those

@k8s-bot test this

one shippable run passed, integration test timed out in the other one. Timeout isn't caused by doc change, merging :)


Editor Crashes using floating script window while refocusing it

Godot version
4.1.1
System information
Windows 10 - Godot v4.1.1.stable -Vulkan (Forward+)
Issue description
Editor crashes when the scripting window is in a floating state and un/focusing it with alt + tab.
Steps to reproduce
I've been able to reproduce this every time:

Open Godots script editor.
Put the script editor in floating window state.
Alt + Tab.
Refocus editor.

Minimal reproduction project
N/A

Cannot reproduce in 4.1.1 or other 4.1.x versions.
At most I can get an error to show platform/windows/display_server_windows.cpp:1160 - Condition "wd_window.transient_parent == p_parent" is true., but I can't get it to crash.

Closing as resolved, if someone can replicate this on a supported version please comment


Switch code to using the new LPRequest class

Behind a feature flag for safety

Can one of the admins verify this patch?


License?

Hello, would you mind providing a license file for this project? I'd like to use it as a starting point for my own flake but the current license is not indicated. Thanks!

Wow that was quick, thanks!


Three tests failing exclusively on Nvidia Turing GPUs

tf.math.digamma, tf.math.is_finite and tf.math.mod are all passing in Swiftshader and failing on Turing GPUs. (Errors in the first log are unrelated).
I don't have access to an Nvidia GPU so I can't really debug this. That said, these are the MHLO MLIR files for each of the functions:

digamma:

rank_1_digamma.mhlo.mlir
digamma.mhlo.mlir

is_finite:

is_finite.mhlo.mlir

mod:

rank_1_mod.mhlo.mlir
mod.mhlo.mlir

I included rank 1 versions of the functions since those are easier to debug, but I can't guarantee the error is reproducible with them. These can be passed to iree-run-mlir with the appropriate signatures.
This can also be reproduced by installing the python bindings on a machine with a GPU and running
python integrations/tensorflow/e2e/math/math_test.py --target_backends=iree_vulkan --functions=is_finite

is_finite looks related to one of my changes. IIRC they were enabled for a while? Do you know when do they start failing?

It looks like an issue in compare op. Does it block anything? Things might be different after moving to Linalg on tensors, I can take a look after the migration.

digamma and is_finite were disabled before https://github.com/google/iree/pull/5393. they started passing on swiftshader sometime in the past four months. mod has had this issue since the tests were created in November.
This doesn't block anything and shouldn't be prioritized as such. They are execution failures / miscompiles though, and from what I understand those are more problematic than compilation failures.

This might be fixed now. We should audit disabled tests - a few reference issues that are already fixed like https://github.com/iree-org/iree/blob/636bc44f6d741da36667cf4c8f596f146150fa8f/integrations/tensorflow/test/iree_tf_tests/math/vulkan__digamma.run#L1-L4


Add 3d detection and tracking ROS nodes

Added Voxel Based 3D Detection, AB3DMOT and Point Cloud dataset nodes

Thank you, @passalis @ad-daniel. I have updated the documentation.

@iliiliiliili just a few more things from my side


added doc for sparse_categorical_crossentropy

I also added a clarifying note to the source. This was in response to #2444.

Thanks, that's useful.

is there a way that if one of the labels in the groundtruth is greater than the number of classes-1, the function throws an error about it?
Currently it just shows Nan in the loss.


Future after shutting down Hyper.sh

Do you plan to stop active development after Hyper.sh shutdown?

What has happened to this project? It seems to have many interesting project that depend on it: https://github.com/kubernetes/frakti , https://github.com/codius/codiusd
However, it seems to be completely non functional. The hypercontainer.io website doesn't work. There are essentially no resources online. And I am completely unable to get this repo to provide any functionality ...
Is the project depreciated?

@teawater Could you comment?

https://github.com/codius/codiusd/issues/150#issuecomment-483040576

Thank you.


add new env vars MM_MODULES_DIR and MM_CUSTOMCSS_FILE …

… for setting these things from outside (and overriding corresponding config.js properties config.paths.modules and customCss)

fixes app.js which didn't respect config.paths.modules before
as modules/defaults is directly set in many places in the source code restrict config.paths.modules to foreign modules (it has never worked for default modules)
adds new /env dir in server.js for getting the new env vars in the browser
fixes TODO in server.js so test directories are now only published when running tests

These changes allow changing some main paths from outside mm with the new env vars. You now can put all user stuff into one directory, e.g. the config dir:

config.js as before
custom.css
foreign modules

This would simplify other setups e.g. the docker setup. At the moment we have to deal with 3 directories where 2 of them (modules and css) contains mixed stuff, which means mm owned files and user files. This can now simplified and leads to cleaner setups (if wanted).

hm.   this will break modules which depend on their location being two directories up from the root and others that depend on folder names.
like those that need electron-rebuild in the root node_modules folder.  and my mmm-config
will also break my upgrade script
what will be the effect on git pull if the deployed  structure is different?

the idea would be

to have modules/default/... (no change)
to have config/MMM-XY instead of modules/MMM-XY
to have config/custom.css instead of css/custom.css (which is already now possible by setting in config.js)

So can you explain what exactly would break?

what will be the effect on git pull if the deployed structure is different?

# Ignore users config file but keep the sample.
/config/*
!/config/config.js.sample

no effect because above is in .gitignore

you can already now change the modules dir and custom.css location inside config.js, this PR only introduces additional env vars for setting this from outside.

can either/both of these be outside the MagicMirror folder tree?

can either/both of these be outside the MagicMirror folder tree?

I did not test this so far, will do and report.
I think at least modules folder must reachable from the browser over https://<server>:<port>/<modules-dir> so the choice of the folder is limited.

both variables (or corresponding values in config.js) are restricted to the published directories in js/server.js:
			let directories = ["/config", "/css", "/fonts", "/modules", "/vendor", "/translations", "/env"];

ok, so one could set modules to config, but not new dir foo, same for config to modules, but not foo
and does it require leading slash?

ok, so one could set modules to config, but not new dir foo, same for config to modules, but not foo

yes.

and does it require leading slash?

no.
You can define it like
node@81ac3b4687ab:/opt/magic_mirror$ export MM_CUSTOMCSS_FILE=config/custom.css
node@81ac3b4687ab:/opt/magic_mirror$ export MM_CUSTOMCSS_FILE=/config/custom.css
node@81ac3b4687ab:/opt/magic_mirror$ export MM_CUSTOMCSS_FILE=/opt/magic_mirror/config/custom.css

All 3 variants works, the mm root dir is stripped in the code.

ok,  can cssfile be modules/foo.css
i misread cssfile, thinking config file and folder could be moved

i always assumed cssfile was name of file, not location

The default config.js key customCss is "css/custom.css". You can overwrite it in config.js or with the new env var MM_CUSTOMCSS_FILE. modules/foo.css works.

thanks, i never thought one could change the css directory, only the filename

I haven't checked the code yet,  in all cases at the time  a module is loaded are the config variables set  so I can use those as the true path? vs examining for the env myself..

in MMM-Config I launch a shell script to collect all the module defaults and generate the form object..
I have a bug now, in that I don't check for where the config.js is.. I  'assume' config/config.js and add on modules in modules
easy fix if the env variables are replicated into the config object

at the moment this is still unmerged ...

I haven't checked the code yet, in all cases at the time a module is loaded are the config variables set so I can use those as the true path? vs examining for the env myself..

if the env variables are replicated into the config object

The new variables don't touch config.js. I don't know where you need the new variables, server or browser?
Depending on this

for server you can use getEnvVarsAsObj from server_functions.js
for browser you can do something like I implemented in loader.js

		const res = await fetch(`${location.protocol}//${location.host}/env`);
		return JSON.parse(await res.text());

I don't check for where the config.js is.. I 'assume' config/config.js

this is unrelated to this PR but you should check MM_CONFIG_FILE

the config object  after parsing is available to the browser sideof the module, so I could see the config.modules.path I was just wondering  if you used the env to update that (as your PR comments say 'override' )
I don't really need the config object, and I don't care about css really.
the environment variables get passed to the node_helper in the process.env
so I can see these vars..

the config object after parsing is available to the browser sideof the module, so I could see the config.modules.path I was just wondering if you used the env to update that (as your PR comments say 'override' )

thanks, didn't thought about this, but I think it would be the better way to change the values in the config object. Will check this and change it if it works.

the config object after parsing is available to the browser sideof the module

the config object is available to the browser because of these lines in index.html:
    <script type="text/javascript" src="js/defaults.js"></script>
    <script type="text/javascript" src="#CONFIG_FILE#"></script>

(the #CONFIG_FILE# is replaced with real config.js file/path while loading)
So I stay with the current approach because otherwise I had to manipulate content of config.js or do other ugly things.

understood. i forgot they are loaded from file

@rejas from my side this is ready for review

did you change anything in the prior support config.paths.modules?

no
but there was a bug in the code so this couldn't work (because modules was hardcoded).
So what are you testing?

current develop branch
or content of this branch?

testing 2.29 prior to this PR..

but there was a bug in the code so this couldn't work (because modules was hardcoded).

so, YES, you had to change something to get this PR to work . the original code was broken
trying to test setting modules location thru config.js, before env variable..
(testing my MMM-Config module which needs to know where the modules are..and the config.js name)

so, YES, you had to change something to get this PR to work . the original code was broken

line 162 in js/app.js
		let moduleFolder = `${__dirname}/../modules/${module}`;

was not using config.paths.modules but hardcoded modules

well, I don't know exactly what you tested, AFAIS you are not testing this PR (the new env vars) but the config.js functionality (which was already there before this PR).

the paths object doesn't have the same override approach as the rest of config.js
leave out the vendor: setting amd the url loaded says undefined/node_modules....

yes, had this problem too, you must provide a full object under paths, otherwise the not provided keys are missing. Maybe objects are copied only on first level. AFAIS the property config.paths.vendor is useless (will not work with a value <> vendor) because we have vendor hardcoded in index.html. So we could "fix" this by removing config.paths.vendor and replacing config.paths.modules with e.g. config.modulesPath.
Let me know if I should update this PR with such a fix.

even tho the modules js wasn't found, it gets injected anyhow..

sorry, don't understand/don't know what you expect from me here

I tested this pr as a separate branch..
moved one module to the config folder from modules
add the paths:{ modules:"config"} to the config.js
then started MM
and looked in the browser development console, elements tab and see this

notice the 3 undefined/node_modules
for things in vendor
you confirmed its not handle the same way (Object .assign()) like the other config.js parms..
and I don't seen any other vendor/  hard coded in config.js
so it would have worked wiith the object.assign
second, on startup, the config.js references modules NOT found in the  configured 'modules' folder,
AND correctly reports that the modulename.js cannot be found during startup (the messages above)
BUT it still considers them 'loaded' as their script link is still added to the in browser index.html..
that should not be..
I understand you are 'only' proposing env vars, BUT.. the previous (undocumented)  built in support
didn't work, so you have had to fix it.. now you 'own' it for this PR. else this PR would not work either
using the env var MM_MODULES_DIR=config
(config.js paths commented out)  produces the same results for issue 2.

I'm ok with not having vendor movable.. and having only the config.modulesPath

altho.. I am sure that on all file systems we run on, there is a link capacbility for those that want to move things around physically without us having to provide config options..

@sdetweil fixed the things you mentioned above.
Tests are failing at the moment unrelated to this PR (did the tests on develop with same result).

crazy that we get these random failures...

altho.. I am sure that on all file systems we run on, there is a link capacbility for those that want to move things around physically without us having to provide config options..

the basic idea is to use one dir (config) for all user stuff so I have only to mount one volume for the docker setup. I could use ln in the container to get things in one dir but there would still be the need to mount 3 volumes. Other point could be (not officially supported) windows where are no symlinks out of the box AFAIK.

windows 11 has symlinks
https://www.howtogeek.com/16226/complete-guide-to-symbolic-links-symlinks-on-windows-or-linux/

looks good both env and config parm

I'm not willing to solve merge conflicts in the changelog over weeks/months, so can I have feedback from @rejas / @sdetweil if this can be merged?

I have tested it extensively, and I think it works well.. so yes, you can merge


Updating licensing information 7-6-21

Docs Build status updates of commit a92005c:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

licensing-information.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


add terratest executing examples/every-mt-vpe

unassigning myself as i am on vacation.

This has been released from PR https://github.com/terraform-ibm-modules/terraform-ibm-vpe-gateway/pull/443


Dump CUDA source on compilation error

Implements #190

Thank you for creating this PR.
How about merge source code to CompileException?

Do you mean like this?

Do you need to write a test?

As I told you internally, IMO unit test is not necessary because this feature is only for the case where catastrophic failure has already occurred.

Fixed.

Fixed!

LGTM


React earlier for parsing errors

Currently when the command can't be parsed, peel simple throw an Exception. The Exception should be catched earlier. Otherwise it will produce meaningless error message.

Resolved in 46acbcd2d8e0e9a3a7d9c864544bc7119f333be5.


After using spinner, cursor is not restored.

simple
    counter = Counter('waiting... ')

    for _ in range(10):
        counter.next()
        time.sleep(1)

leaves my console without cursor, so it seems that counter.finish() is not run.

In my terminal, I had to run tput cvvis afterward to restore the cursor

You need to call finish explicitly when you are using the counter this way. An alternative pattern is the following:
with Counter('waiting... ') as counter:
    for _ in range(10):
        counter.next()
        time.sleep(1)

Both are mentioned on the home page.


Debug actions pane often blocks file names

Currently the debug actions pane is always displayed at the top center position. This often blocks the file names.

It would be helpful to find a better place for the pane or add the ability to move the debug actions pane.

Yes please! It would be great to have this in its own toolbar. I'm constantly having to move it out of the way when trying to switch tabs.

Possibly something like this:

I'd love to see it be moveable (more than left to right), or in the debug pane itself. This is pretty painful as is being locked on top of the file tabs.

Assigning to april to investigate a better solution. Though not making gurantees that this gets fixed in april

@jamiedawsonyoung - would love to hear your thoughts on this.

What are those extra button in the screenshot here? How to get those buttons? Can debug buttons be placed there?

@gulshan your link is not good, however some adapter support more debug actions and when using those adapters more actions appear. For instance the chakra debugger supports step back debugging, and when debugging chakra an additional step back action will be added

The image in the link looks like-

@gulshan those buttons are from ancient times of Monaco on azure websites. That product is an ancestor to VSCode, you can google around to search for more data.

@stevencl When I looked at the debug section on the website I can see what looks like a 'grip / grabber' to the left of the controls. It would suggest that this panel has been floating in the past? Was there a reason it's now fixed?
It doesn't look (from my fairly light experience of VSC) that we have many floating windows that users can move around so I'm not sure we should introduce one here.
It might be there's a place for it in the debug view (the view I see when I click the bug icon on the left but I'd need to try out debugging something for real to see the controls work and whether they could fit.
@stevencl could you or someone send me a file I can try to debug?

Hey @jamiedawsonyoung to debug something have the following file and press F5
The screenshot in this issue is outdated, currently we allow the debug toolbar to be dragged around but only horizontally. We are thinking if we should also allow vertical dragging with some sugar on top - like the toolbar beinging docked to top and it takes extra effort to dock it out.
console.log('hey jamie')

Thanks @isidorn Got it working :-)

@jamiedawsonyoung let me know if I can help in some other way or if you would like to chat so I provide more details about the issue.

Hey @isidorn That would be great. I'll set up a wee call so we can chat about this.

To start we have decided to experiment with the default position of the debug toolbar(move it from center to all the way to the right). We have also added telemetry to track where usres usually place the debug toolbar so in the future we can make a data driven decision.

@isidorn Moving the panel to a different spot on the top definitely won't address this problem. I think telemetry is great, but I don't think it'll help as people will put the bar anywhere but over the file of interest. Why can we not move it away from the filenames altogether?

@seesemichaelj thanks for the feedback and I agree that telemetry might not be beneficial in the end.
We have some ideas on moving it from the filenames which we want to explore next milestone.
After discussing within the team we have decided to change the default location back to the center of the screen due to discoverablity.

When you say next milestone, do you mean June or July? I think we could have an initial solution for the June to see if it works out instead of waiting for July. I guess I'm still confused why the delay? Is it too difficult to solve? Do you need help implementing it?

June.

June.
This week we are in the may endgame which means we accept no new features and we are only testing. Due to that we can only do this in June.
PRs are of course always welcome, but I would prefer some UX sketches before you potentially start implementing something.

I've just published an extension for adding a debugger to the statusbar, might be a decent workaround until this gets fixed, check it out:

Being able to position it bottom left of the editor view would be my preference, or just being able to snap it to any edge along the editor view.

@robbiespeed Having it anchored at the top I believe it's a mistake, how should it behave if the there's not enough space for both the debugger and the search widget?
Having it at the bottom might cause the same problem, but for that to happen the window would need to be very weirdly resized, so it shouldn't be a concern.

I think it would be helpful if it could be optionally pinned to its own bar directly below the title bar. This would get it out of the way of everything else. This would take away a little room from the code window and sidebar, but is that a great concern?
At the least, there should be the option to get it out of the way of tabs and other stuff.
Thanks
Jordan
On Jul 18, 2017, at 8:31 AM, Fabio Spampinato<EMAIL_ADDRESS>wrote:
@robbiespeed Having it anchored at the top I believe it's a mistake, how should it behave if the there's not enough space for both the debugger and the search widget?
Having it at the bottom might cause the same problem, but for that to happen the window would need to be very weirdly resized, so it shouldn't be a concern.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Maybe you should the buttons in the activity bar. There is a lot of space for those buttons. The sessions drop down could be placed at the status bar.

I really don't know why this cant just be like an actual toolbar (not floating) like VS, or pretty much any other ide (I know.... crazy concept). (forgive me for being very opinionated on this, this is almost solely why I tend to use webstorm over vscode)
I feel like there's someone someplace that just decided idea of toolbars in vscode are evil, and while this would be the typical solution if we weren't talking vscode... suggestions to solve many of the problems toolbars would  normally solve, tend to be real round about stuff like... just remember the command, or stick a button in the status bar.

A lot of excellent ideas, pictures and comments about this issue in #9093

@ronnyek I share your thoughts. I don't mind trying to work through a better solution, but I'd love a toolbar option in the meantime.
Anyway, maybe some guidelines will help pin down a better solution. Here are my opinions:

It should be configurable. There are so many variables in how we all use Code that a single solution will cause a problem for someone.

Multiple configurations: docked in its own toolbar, positioned to top or bottom, or floated
Don't overlay the bar on top of other things, unless given the option to.

Keep its visibility independent of any sidebar or panel. It needs to be visible when viewing both the Explorer and Debug sidebars (and probably others as well). It also needs to be visible when the Debug or other panel is closed
It should not be affected by other extensions, editors, window size, etc...

Other extensions can place icons in the activity bar. Placing it here would reduce space for other extension icons, and could also be affected by the height of the window. This is the same for the status bar.

It's placement shouldn't affect other items (excluding screen real estate for a toolbar). This will allow it to become visible in debug mode without moving anything else around or other side effects.

For example, placing it in the tabs area may displace the open tabs

Don't compromise on accessibility

I'm really looking for a solution to this problem as it seems that the debugger is always in my way. I keep moving it all the time. I'm trying to exclusively use the shortcuts now (because my debugger is never in the same place) and may be you could allow us to hide it if we don't want it (because we are only using the shortcuts). Thanks.

@JeanClaudeCottier you can already hide it, just add "debug.hideActionBar": true to your settings.

@fabiospampinato I've just tried your extension. Its great, thanks a lot.

Shameless hack here jut because 😆
Open the editor web tools, paste this thing in the console and hit enter. Make sure to have the widget visible in your screen.
function getOutOfMyWay() {
  document.querySelector('.debug-actions-widget')
    .setAttribute('style', `
      bottom: 0;
      left: 0;
      position: fixed;
      height: 32px;
      display: flex;
      padding-left: 7px;
      box-shadow: none;
      transform: rotate(-90deg) translate(90px, 20px);
      transform-origin: left center 0px;
  `);
}
getOutOfMyWay();

For the last 3 months I've been using VSCode to do stuff I haven't done before: debug Python. I absolutely love it (despite the disability to debug unit tests - still can't figure out if it's a bug or a lack of some kind of massive configuration), but that constant moving of debug buttons from left to right just got on my nerve already.
Could you please let us know of your current plans on this? Thanks!!

I would be extremely okay with the thing taking up space on the left sidebar underneath the usual icons.

Hey @bgashler1 & @isidorn, is this still being picked up? For now I'm using the plugin to hide it in the statusbar. But that one does not provide a switcher between multiple running processes...
I would vote for integrating it with (or putting it below) the DEBUG panel title bar.

From #9093:

Is that what you guys mean?

This is a problem that users are having and we plan to adress it in the near future.
As to in which milestone exactly I can not specify yet, but seems like not in february.
In the meantime if somebody is willing to contribute a PR, feel free to create one where we can discuss first how to solve it and agree on the implementation.

@mattacosta I actually really like that idea. That's a pretty sensible spot for it.

@isidorn how about the next one? :)

Here's my way around it:
Open the workbench.main.js file inside vscode.app (on a mac, unsure of pc but I'm sure that file exists) and search for:
style("top",r.getTitleBarOffset()+"px")
and replace with:
style("top",(r.getTitleBarOffset()+35)+"px")
This moves the debugger down into your code area.

@fabiospampinato Your extension words fine! Thx
But not has a option to hide the native Debug actions pane.

@tiagoSAmaral the README file explains how to do that: https://github.com/fabiospampinato/vscode-statusbar-debugger#hints

Thx

I don't understand why this thing has to be locked on the y-axis... I haven't looked at the implementation but people have been complaining about this for 2 years. Wouldn't a great first step be to simply allow it to be dragged wherever the user wants? Then the telemetry results might actually be helpful.

Debugging is one of the flashiest of vscode and also one of the most critical features for users ... yet this perpetual annoyance of a debugging UI stays under discussion for years. Purporting that it is "part of a larger problem" does exactly nothing for users who would be perfectly happy with the dirtiest quick hack to allow it to float anywhere in the window instead of frozen to the top row of the window. That's gotta be what, an hour's work? Mind-blowing to read such a thread ... is there a product manager that's married to the location perhaps?

I'm sorry, but I think we need to stop complaining and go ahead and put in the work. I don't see any of you stepping up to the plate and submitting a PR as @isidorn recommended in February.
Sure, I'm not sure why this one issue has fallen through the cracks, but I feel as developers we all know what it's like to be "too busy" to work on something that isn't in our priority list, or our bosses priority list. We all probably are just going to respond "Mike, we don't have time to do this, this is Microsoft's job." Well they are busy too, and try to release a stable product monthly for the entire world to use.
I will try to put in some time to submit a PR if no one else steps up to the plate, merely because I'm tired of hearing all of the complaints.

I've submitted 2 PRs for further discussion with VSCode maintainers/@isidorn.  While #49097 is a readily available PR which allows vertical dragging, I prefer the other proposed solution in #49099 which will need further discussion in terms of implementation. I will push to have some sort of solution merged for the May or June milestones.

I've been using Fabio's extension for months (THANK YOU FABIO). Most of the time, I'm using the keyboard shortcut anyway. I switch off the original and awful (yes awful) debugger tab and never looked back. I made VSCode my main platform for development (I wouldn't have without the extension). I feel the pain for the people that can't use it. I can't believe this hasn't been fix yet. VSCode is a great product that I'm promoting all the time. But the debug tab!!!! What are you waitting for guys? No excuses, for such a critical feature.

@seesemichaelj I agree that the whinging is probably best replaced with action. My excuse is having worked into the wee hours for too many days in a row, shuffling the debug tab back and forth as I stop, start, restart, stop, start, restart a couple of micro-services ... it gets a little old ... I'll take a look at Fabio's extension, thanks for the tip @JeanClaudeCottier

@seesemichaelj I agree with you, but when I decided to try and make a PR myself I went to read about how to set up VSCode for developing... It surprisingly has quite a lot of steps, so I decided to leave it for later "when I have more time", so like never :D But I guess I should just do it today and get over with it :)

I appreciate the initiative @bakasmarius! I went ahead and created PRs (see my last comment) so at this point it would be duplicating working, and much of it relies on the outcome of UX meetings.

Thanks for the feedback and for opening PRs with great ideas.
After discussion we decided to introduce the following setting
"debug.toolbar": "float"|"dock"|"hide"
The default is like before that the debug toolbar is floating, however it can now be docked to the debug viewlet.
debug.hideActionBar is now deprecated and we will no longer support that setting in 2 milestones.
Feedback is welcome, especially on naming of settings.
What I still plan to investigate if possible is to add some action in the toolbar to dock it directly from there.
fyi @weinand @alexr00

That's better, thank you, but why not putting the commands in a secondary line/toolbar? Like this:

The current solution isn't great for compound debugging, this sidebar looks absolutely massive to me:

It can be resized, but then I loose the name fo the view ("DEBUG"), the name of the launch configuration gets cropped, and even the "more" button gets hidden.

Putting this actions on a secondary line/toolbar just seems a better solution to me.

I definitely agree with @fabiospampinato.
Further, you lose the launch configuration drop down. There are cases where you start debugging and then want to run another launch configuration. You can't press F5 because your selected configuration is already running and I believe combining the multi-process dropdown and launch configuration dropdown into one will be nonintuitive and a clutter.

Please use a separate line like fabiospampinato suggested above. I hate when the sidebar is too wide so I will have to resize it every time I switch from explorer to debuger view. I'm sure a lot of people will agree on this...


Use logging module throughout

Move from my verboseprint() to using logging module. The aim is also that when not running with --verbose there is very little output unless something fails.
None verbose output looks like:
rwellum@RWELLUM-M-34DF:[~/Desktop/Boxes]: iosxrv-x64-vbox/iosxr_iso2vbox.py iosxrv-fullk9-x64.iso
INFO:__main__:Creating Vagrant VirtualBox
INFO:__main__:Bringing up with iosxr_pexpect.py to install to disk and configure
INFO:__main__:Running basic unit tests on VirtualBox...
INFO:__main__:Passed basic test, box /Users/rwellum/Desktop/Boxes/machines/iosxrv-fullk9-x64/iosxrv-fullk9-x64.box is sane

Verbose looks like: (this is not all of it)
...Starting up
rwellum@RWELLUM-M-34DF:[~/Desktop/Boxes]: iosxrv-x64-vbox/iosxr_iso2vbox.py iosxrv-fullk9-x64.iso -v
DEBUG:__main__:Input ISO is iosxrv-fullk9-x64.iso
DEBUG:__main__:iosxrv-fullk9-x64.iso is a full image, RAM allocated is 4096 MB
INFO:__main__:Creating Vagrant VirtualBox
DEBUG:__main__:Virtual Box Manager Version: 5.0.24r108355

DEBUG:__main__:pathname: /Users/rwellum/Desktop/Boxes/iosxrv-x64-vbox
DEBUG:__main__:VM Name:  iosxrv-fullk9-x64
DEBUG:__main__:base_dir: /Users/rwellum/Desktop/Boxes/machines
DEBUG:__main__:box_dir:  /Users/rwellum/Desktop/Boxes/machines/iosxrv-fullk9-x64
DEBUG:__main__:box_out:  /Users/rwellum/Desktop/Boxes/machines/iosxrv-fullk9-x64/iosxrv-fullk9-x64.box
DEBUG:__main__:Found and deleted previous /Users/rwellum/Desktop/Boxes/machines/iosxrv-fullk9-x64/iosxrv-fullk9-x64.box
DEBUG:__main__:'iosxrv-fullk9-x64' is not running, nothing to poweroff
DEBUG:__main__:'iosxrv-fullk9-x64' is not registered, nothing to unregister and delete
DEBUG:__main__:Removing stale SSH entries
DEBUG:__main__:Create VM
DEBUG:__main__:Register VM

.... Waiting for vm to come up
INFO:__main__:logging in
2016-07-14,11:14:18 (iosxr_pexpect.py    :252 ) DEBUG (xr, port socat TCP:localhost:65000 -,raw,echo=0,escape=0x1d): Connect
DEBUG:__main__:(xr, port socat TCP:localhost:65000 -,raw,echo=0,escape=0x1d): Connect
2016-07-14,11:14:18 (iosxr_pexpect.py    :230 ) DEBUG (xr): Spawn (socat TCP:localhost:65000 -,raw,echo=0,escape=0x1d)
DEBUG:__main__:(xr): Spawn (socat TCP:localhost:65000 -,raw,echo=0,escape=0x1d)
2016-07-14,11:14:18 (iosxr_pexpect.py    :322 ) DEBUG (xr: socat TCP:localhost:65000 -,raw,echo=0,escape=0x1d): Waiting for XR login prompt
DEBUG:__main__:(xr: socat TCP:localhost:65000 -,raw,echo=0,escape=0x1d): Waiting for XR login prompt
2016-07-14,11:14:23 (iosxr_pexpect.py    :654 ) DEBUG (xr: socat TCP:localhost:65000 -,raw,echo=0,escape=0x1d): Send <enter>
DEBUG:__main__:(xr: socat TCP:localhost:65000 -,raw,echo=0,escape=0x1d): Send <enter>
2016-07-14,11:14:28 (iosxr_pexpect.py    :654 ) DEBUG (xr: socat TCP:localhost:65000 -,raw,echo=0,escape=0x1d): Send <enter>
DEBUG:__main__:(xr: socat TCP:localhost:65000 -,raw,echo=0,escape=0x1d): Send <enter>
2016-07-14,11:14:34 (iosxr_pexpect.py    :654 ) DEBUG (xr: socat TCP:localhost:65000 -,raw,echo=0,escape=0x1d): Send <enter>

..... VM is up and installing
[  116.634205] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[  116.685055] 00:04: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A
[  116.737323] Non-volatile memory driver v1.3
[  116.745204] Linux agpgart interface v0.103
[  116.760442] brd: module loaded
[  116.780167] loop: module loaded

......Configuring the VM
  Enter root-system username:
2016-07-14,11:19:46 (iosxr_pexpect.py    :376 ) DEBUG (xr: socat TCP:localhost:65000 -,raw,echo=0,escape=0x1d): Got 'Enter root-system username:'
DEBUG:__main__:(xr: socat TCP:localhost:65000 -,raw,echo=0,escape=0x1d): Got 'Enter root-system username:'
2016-07-14,11:19:46 (iosxr_pexpect.py    :654 ) DEBUG (xr: socat TCP:localhost:65000 -,raw,echo=0,escape=0x1d): Send "vagrant"
DEBUG:__main__:(xr: socat TCP:localhost:65000 -,raw,echo=0,escape=0x1d): Send "vagrant"
2016-07-14,11:19:46 (iosxr_pexpect.py    :381 ) DEBUG (xr: socat TCP:localhost:65000 -,raw,echo=0,escape=0x1d): Sent XR login 'vagrant'
DEBUG:__main__:(xr: socat TCP:localhost:65000 -,raw,echo=0,escape=0x1d): Sent XR login 'vagrant'
2016-07-14,11:19:46 (iosxr_pexpect.py    :383 ) DEBUG Wait for password request

.....Finishing up and running ut
DEBUG:__main__:Waiting for machine to shutdown
DEBUG:__main__:Successfully shut down
DEBUG:__main__:Remove serial uarts before exporting
DEBUG:__main__:Compact VDI
DEBUG:__main__:Creating Virtualbox
DEBUG:__main__:Created: /Users/rwellum/Desktop/Boxes/machines/iosxrv-fullk9-x64/iosxrv-fullk9-x64.box
INFO:__main__:Running basic unit tests on VirtualBox...
INFO:__main__:Passed basic test, box /Users/rwellum/Desktop/Boxes/machines/iosxrv-fullk9-x64/iosxrv-fullk9-x64.box is sane
DEBUG:__main__:Single node use:
DEBUG:__main__: vagrant init 'IOS XRv'
DEBUG:__main__: vagrant box add --name 'IOS XRv' /Users/rwellum/Desktop/Boxes/machines/iosxrv-fullk9-x64/iosxrv-fullk9-x64.box --force
DEBUG:__main__: vagrant up
DEBUG:__main__:Multinode use:
DEBUG:__main__: Copy './iosxrv-x64-vbox/vagrantfiles/simple-mixed-topo/Vagrantfile' to the directory running vagrant and do:
DEBUG:__main__: vagrant box add --name 'IOS XRv' /Users/rwellum/Desktop/Boxes/machines/iosxrv-fullk9-x64/iosxrv-fullk9-x64.box --force
DEBUG:__main__: vagrant up
DEBUG:__main__: Or: 'vagrant up rtr1', 'vagrant up rtr2'
DEBUG:__main__:Note that both the XR Console and the XR linux shell username and password is vagrant/vagrant
DEBUG:__main__:'iosxrv-fullk9-x64' is not running, nothing to poweroff
DEBUG:__main__:'iosxrv-fullk9-x64' is registered, unregistering and deleting

A few minor nits but otherwise looks good - 👍 Thanks for seeing this through!


Track key parameters of an organization's work setup over time

Examples include:

Where does the organization fall on the spectrum between fully in-office and fully remote work?
What is the organization's position on pay scales (some orgs like Pangea have equal pay for all workers (cf. https://www.pangealegal.org/jobs); other have multiple pay grades)?
What is the organization's reporting structure (flat, consensus-based, holacracy, almost-flat with everybody reporting to CEO, or a full complex reporting tree?)

These parameters can change over time, so it would be good to record start and end dates for specific values of the policy parameters.
I need to discuss this more with @riceissa and @sebastiansanchez123.

This seems good to track. I am wondering how the dates interact with this though in terms of storing the data in mysql. If we have a tuple like (startdate, enddate, param1, param2, param3), then if one of the parameters changes we are forced to create a new column, which results in a lot of duplicate data for the parameters that don't change. Something like (startdate, enddate, param1name, param1) would be more logical but might be annoying to query for.

Another alternative with the (startdate, enddate, param1, param2, param3) style schema is to use "sparse" rows where all non-changing rows have NULLs. That would make repetition unnecessary but having all the NULLs might be annoying or make querying difficult.


Create treatment ontology term ETL

Design
Database Schema

Create entry in scripts/init.sql for the creation of the table, pk and fks.
Create entry in etl/workflow/transformer.py with the class TransformOntologyTermTreatment
Create entry in etl/jobs/transformation/spark_transformation_job.py
Create job etl/jobs/transformation/ontology_term_treatment_transformer_job.py
Create entry in luigi.cfg
Create entry in etl/workflow/loader.py

Acceptance Criteria

[x] Table was created according with the designed database schema
[x] The transformation task can be executed and it finishes correctly

Done.


Inconsistent behaviour when filtering by metadata using $or

Hi,
i'm running into a problematic behaviour with querying chroma using $or on metadata from python:
Given a collection with a set of embeddings where the metadata contains information like this:
{
    "category": "all"
}

and a second set of embeddings with the metadata like
{
    "category": "test"
}

where the embeddings in the second set have both higher and lower distances to the search pattern used
    embeds = collection.query(
        query_texts=[search],
        where={"$or": [{"category": {"$eq": "test"}}, {"category": {"$eq": "all"}}]},
        n_results=3
    )

and that the embeddings for category "all" were added to the collection before adding category "test"
Current behaviour: the query returns only data from category "all"
Expected behaviour: the query should return values from category "all" and from category "test"
Querying individually for category "test", the resulting embeddings show both higher als lower distances to the ones returned for category "all"
Using a different, not existing, second category for the $or , the embeddings for category "test" are returned correctly
Any ideas what could be wrong?

How much data do you have and whats the distribution across categories? I attempted to repro
https://colab.research.google.com/drive/1glfVqsOOVxyLsJfmIq1GJBd9pazu1e-T#scrollTo=iWGs42guud0w
In this colab, and am seeing results for both categories. So am not able to reproduce the issue. It is possible that your data is distributed in such a way that prefiltering HNSW is causing a pathological case.

@HammadB could you please clarify what you mean by prefiltering HNSW? My understanding is that when querying with an embedding and a set of metadata filters, the filters are applied first, and approximate nearest neighbour is done on the results that match the filter only.

This is stale. Closing this issue for now. Happy to re-open.


Failures analysis: Wrong url for report from the board

Steps to reproduce

Go to '.Reporting Dashboard page'
Open Failures analysis page
Click on reporting url

Actual behavior
404 page is opened
Expected behavior (if known)
Report page is opened
Environment
UI version:1.12.0.RC2
API (back-end) version:1.12.0.RC1

@tsvrko need to find location where db-views.sql to fix views

fixed. we have to remove all reporting docker images manually!

Verified on dev. Link is wrong.

fixed in develop branch of zebrunner

Fixed.


Move tests to a separate repository

It makes it easier and better to write tests among the different languages and ensures consistency. We could use a simple syntax similar to SimpleTest. This allows the language to separately interpret the test files, use the language's code to run the test and ensure the correct stuff is matched (and correct regex is generated). The current .rule syntax may be good enough for this.

Moving tests to a separate repository is a good idea. I'll do that tomorrow. For the test syntax: It should definitely be something easy to parse no matter what language you're using, since they should be universal for all implementations. I was thinking about using YAML instead of the current selfmade syntax in the .rule files.


Time complexity of commands

Hi, usefull module. Will be great have documented time complexity of commands like in original redis documentation.

@taras-zak Hi, thanks for the suggestion. I will try to get to it ASAP.


changes to canonical and read_partition

Adding a chunk_size kwarg to canonical, this will set the size of the event_pages that are returned.
read_partition needs to be changed around to accommodate this.
canonical always returns the datum along with filled events.
canonical is a snapshot and returns only the documents present when it is called. So if canonical is called in the middle of the run, you won't get all of the run, but it can be called again to more documents.
we plan to change read_partition so that the whole unfilled run gets loaded on the first call.
the first partition will be the header(start, descriptor, resources)

Superseded by #469


Update ZIP file URL

Thanks!

did you need still this file - or is it obsolete?
https://github.com/aheze/Squirrel/blob/main/Squirrel.zip

did you need still this file - or is it obsolete? https://github.com/aheze/Squirrel/blob/main/Squirrel.zip

It's obsolete now

i have no access to the repo - someone (@aheze) should remove this zip to avoid confusion...

i have no access to the repo - someone (@aheze) should remove this zip to avoid confusion...

Make a PR that removes it


Script not selecting Start Quest

I'm running FGO LUA on my phone (Pixel 3).
I started trying to farm doors today, and this is what happens:

Script selects the 40 ap quest correctly
Script finds a support w/Mona Lisa correctly
On the quest screen, it selects the region in-between "Help" and "Total Cost" on the bottom, not Start Quest
Sits there forever until I either manually hit Start Quest or shut off script

Also:

When I hit Start Quest myself, it seems to have 0 issue finding skills / NPs / other regions to click on correctly.
I also tried messing around w/the game.lua file and manually changing the start region myself, but it kept clicking the same area.

Is the device you are using have a notch for camera on the screen? And by that, I mean does the screen have a cutout where the camera sits?

I'm pretty sure this is due to not enough delay between selecting the servant and the menu to start quest coming up.

Sorry, forgot I opened this thread.
Yes, that appears to be the issue. I put a wait cmd before it selects the start quest button and it seems to work fine lol. Guess my pixel is too slow. 😛


Publish npm package for Node

An npm package for prql-js that's usable on the backend would be fantastic. There's probably even a way to get both the bundler and nodejs builds published under one package.
For now, I threw a wasm-pack build --target nodejs build up at https://www.npmjs.com/package/prql-node, but it would be great if this was properly maintained/updated.
PS: awesome work on this.

Heya, yes I need to merge https://github.com/prql/prql/pull/732 which will bundle all 3 as one package , but glad you got it working !


Don't hijack statsd

Problem
Currently, if KubernetesDeploy is used as part of a library, it will hijack the global StatsD configuration and, among other things, will prefix all metrics with KubernetesDeploy. Ideally, KubernetesDeploy should not mutate global config like this and its StatsD configuration should be scoped more tightly to itself.
Closes #336.
Proposed solution
Instead of KubernetesDeploy::StatsD simply acting as an interface to the global StatsD configuration, it will instead extend it and become a singleton in its own right. This allows us to specify config parameters for KubernetesDeploy::StatsD without overriding the global configuration.
Issues
Unfortunately, some methods in the StatsD library make statically scoped calls to the StatsD gem itself. Thankfully this only seems to occur when initializing new metrics. After perusing the statsd-instrument repo, I noticed that we aren't the first to come across this and the proposed solution was pretty much what I had in mind. The PR is here. Testing locally with the changes to the gem yields successful test passes Edit: this has been shipped

This PR is currently blocked by statsd-instrument #148, which allows for multiple prefixes.

statsd-instrument #148 has been shipped and released as v2.3.2. This lets us use the :prefix option when creating metrics.

Rebased on master. This is ready for a final round of 👀

Looks good, the dashboards will need to be updated before we deploy this.
Perhaps it's my unfamiliarity with DataDog, but what exactly do we need to change, exactly?


upgrade packages from the @manypkg org

thanks for the feedback! looks like this will need to be reworked a bit. I'll close it for now


level0 failing when executing vm extension - script devops_runtime_baremetal.sh with follwing error...

https://github.com/aztfmod/level0/blob/71df4a427ab09ebaee711ca98c42e11ab920ca70/launchpads/launchpad_opensource/modules/blueprint_azure_devops_no_pip/scripts/devops_runtime_baremetal.sh#L22
The error message from azure vm extension:
Enable failed: failed to execute command: command terminated with exit status=1 [stdout] start install Ubuntu packages [stderr] devops_runtime_baremetal.sh: line 22: /etc/apt/apt.conf.d/90assumeyes: No such file or directory
I noticed that before was used Ubuntu as default in tfvars, now used Centos 7.6 in the latest (2005.180055). Not important but in the script is hardcoded message  - echo "install Ubuntu packages".
not sure if centos edition 7.6 contains by default apt-get which is being executed in the devops_runtime_baremetal.sh

Rover image is based on Centos 7.6. However, to run CI activities on Azure Devops self-hosted agent within the rover container the Azure Devops agent has to be deployed on a baremetal os. Docker in Docker we used in the past is not possible with the latest versions of Azure devops agent. the reason you see Ubuntu is to match with the official support of Azure Devops agent script.
2005.180055 introduced a regression we need to fix. this is why you have that error


[$500] Tread - Thread is not archived when delete WS Offline

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 1.4.32-2
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
Issue found when executing PR https://github.com/Expensify/App/pull/31900
Action Performed:

Go to https://staging.new.expensify.com/
Log in
Create a new workspace
Open the workspace/policy expense chat
Create a thread
Turn off the internet connection
Delete the workspace

Expected Result:
Thread is archived
Actual Result:
Thread is not archived when delete WS Offline
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[x] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
Add any screenshot/video evidence
https://github.com/Expensify/App/assets/93399543/b900a037-7316-473b-880a-698149600604
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~0148ea2ff5d6b35fdb
Upwork Job ID:<PHONE_NUMBER>529564160
Last Price Increase: 2024-01-25

We think that this bug might be related to #vip-vsb
CC @quinthar

Proposal
Please re-state the problem that we are trying to solve in this issue.
Chat - The task does not display in LHN that it is archived
What is the root cause of that problem?
We checked if the report is a ReportUtils.isThread(report) and removed it from the list of workspace reports that need to be archived.
https://github.com/Expensify/App/blob/889051cf6c155c225a9c602c17d95ec40d740839/src/pages/workspace/WorkspaceInitialPage.js#L103-L105
Setting the reports attribute statusNum and stateNum in the deleteWorkspace function.
What changes do you think we should make in order to solve the problem?

Remove the condition ReportUtils.isThread(report) and leave the condition checking the report, which has an Id (!report.reportID).

Proposal
Please re-state the problem that we are trying to solve in this issue.
The thread is not archived when we delete WS in Offline mode.
What is the root cause of that problem?
The thread is actually archived, but we just don't add an "archived" indicator for threads in the LHN. Here, we don't check if the parent report is archived.
What changes do you think we should make in order to solve the problem?
we can add a condition like here to check if the parent report is archived and add Localize.translateLocal('common.archived') after parentReportActionMessage to show that the thread has been archived.
What alternative solutions did you explore? (Optional)
N/A

The following reports are archived when the workspace is deleted:

admins room
announce room
workspace expense chats (the chats between each employee and the admins)
workspace rooms (the rooms created on the workspace)

Threads created from workspace chats and rooms inherit the chatType from the parent report; thus, they are archived.
Tasks do NOT inherit the chatType from the parent report; thus, they are NOT archived. The thing is that I'm not sure if it is intentional like that in the BE. I'll open an internal discussion about it.

Here's the link to the internal discussion for whoever comes here later.

According to the internal discussion, we do not want to archive the task report and the IOU/expense reports. We only archive the workspace chats and rooms and their chat threads.
@aimane-chnaif, @johncschuster, Could we confirm that the App behaves like this?

Closing since we don't want to archive tasks.

Ok then, I'll reopen for you to review the proposals

Sorry for missing this! Payment has been issued!


Use correct block size for SHA1

SHA1 has a 64 byte (512 bit) block size, not a 20 byte (160 bit) block size like you would expect.
Tests:
<?php
use Symfony\Polyfill\Php55\Php55;
require "polyfill.php";

foreach ([
    str_repeat('A', 63),
    str_repeat('A', 64),
    str_repeat('A', 65),
] as $s) {
    $x = hash_pbkdf2('sha1', $s, $s, 4096, 25);
    $y = Php55::hash_pbkdf2('sha1', $s, $s, 4096, 25);
    if (!hash_equals($x, $y)) {
         var_dump($x, $y);
         exit(1);
    }
}

@nicolas-grekas could this be tested by checking that the polyfill and the native implementation provide the same output for various length and various algorithms ?

We don't currently have a unit testing apparatus hooked up for this package. Would I be okay to add phpunit to require-dev?

Well, that PR should be submitted against the main https://github.com/symfony/polyfill repo rather than the subtree split. There, we already have a CI setup.

Here's data for a test:
hash_pbkdf2('sha1', 'passwordPASSWORDpassword', 'saltSALTsaltSALTsaltSALTsaltSALTsalt', 4096, 25);

PHP 5.5+ gives you 3d2eec4fe41c849b80c8d8366 while polyfill returns 10865a069b0b5a3f3d202c6c4.

Here's data for a test:
hash_pbkdf2('sha1', 'passwordPASSWORDpassword', 'saltSALTsaltSALTsaltSALTsaltSALTsalt', 4096, 25);

PHP 5.5+ gives you 3d2eec4fe41c849b80c8d8366 while polyfill returns 10865a069b0b5a3f3d202c6c4.

Correct. If anyone needs to reproduce a faulty hash (e.g. to re-encrypt something encrypted under a bad key):
- hash_pbkdf2('sha1', 'passwordPASSWORDpassword', 'saltSALTsaltSALTsaltSALTsaltSALTsalt', 4096, 25);
+ hash_pbkdf2('sha1', sha1('passwordPASSWORDpassword', true), 'saltSALTsaltSALTsaltSALTsaltSALTsalt', 4096, 25);

This will give you the previous result (10865a069b0b5a3f3d202c6c4) instead of the correct result (3d2eec4fe41c849b80c8d8366).


self created twitter app?

Currently, NNW app have access to my twitter account

Also, some other application support self-created twitter apps (for example https://github.com/huginn/huginn/wiki/Configuring-OAuth-applications#twitter )
So, I would like to create own twitter app and authenticate in that app

Why do you want to do this?

Its more secure
https://thehackernews.com/2013/10/buffer-hacked-twitter-facebook-flooded.html
Fortunately here only read-only rights, so possible impact is not high, but anyway after that incidents I think will be better to use personal apps

If NNW got compromised and was spamming Twitter, wouldn't we want the ability to shut it off at the server by disabling the app on Twitter? If we encouraged people to create their own Twitter apps, we wouldn't have that ability.

Anyway NNW would disable it only after some incidents, this could be a target for attackers
Fortunately currently it can only read, but still low impact (reading private posts)
And with the growing of popularity, this will only increase

We won't be adding this unnecessary complexity.