Node peer connections fail to connect when building with --enable-debug=yes

Hi,
not sure if this is wrong in the documentation but we basically just discovered that running configure while building Dynomite as per the README.md:
./configure --enable-debug=yes

results in a failure to establish inter peer connections... simply recompiling with
./configure --enable-debug=full

fixes the problem and connections between nodes are established as expected.
Cheers,
Erick

Hi,
I was able to reproduce that and it's the same behaviour for the following cases:

--enable-debug=no
--enable-debug=yes
--enable-debug=log
skip the flag (default is no, so this case is covered in the first one)

Using --enable-debug=full works fine.
Cheers,
Marcel

Thats odd! Can you send me the details ? what do you mean by failure to establish connections? I will also try it locally

Sure. Let's go..
As said above, we use the dev branch. So, how to reproduce the issue?
autoreconf -fvi cd dynomite/ ./configure --enable-debug=yes  make  sudo make install
It compiles without any errors.
Configuration files:
`
#1st node - dyno8101.yml:
dyn_o_mite:
datacenter: dc1
rack: rack1
listen: <IP_ADDRESS>:8102
dyn_listen: <IP_ADDRESS>:8101
dyn_seeds:

<IP_ADDRESS>:8113:rack1:dc1:101134286
dyn_seed_provider: simple_provider
tokens: '101134286'
servers:
<IP_ADDRESS>:11371:1
data_store: 0
stats_listen: <IP_ADDRESS>:33331
preconnect: true

2nd node dyno8103.yml:
dyn_o_mite:
datacenter: dc2
rack: rack1
listen: <IP_ADDRESS>:8114
dyn_listen: <IP_ADDRESS>:8113
dyn_seeds:

<IP_ADDRESS>:8101:rack1:dc1:101134286
dyn_seed_provider: simple_provider
tokens: '101134286'
servers:
<IP_ADDRESS>:11370:1
data_store: 0
stats_listen: <IP_ADDRESS>:33333
preconnect: true
 Then we start both nodes in the foreground:
/usr/local/sbin/dynomite -c /home/dynomite/dyno8101.yml  -v 11
/usr/local/sbin/dynomite -c /home/dynomite/dyno8103.yml  -v 11
`

tcpdump -i lo does not show any connection or connection attempts between both instances.
Example log:
 [2017-04-21 20:02:09.397] _dn_alloc:259 malloc(16) at 0x2539790 @ dyn_gossip.c:223 [2017-04-21 20:02:09.397] _dn_alloc:259 malloc(9) at 0x25397b0 @ dyn_gossip.c:226 [2017-04-21 20:02:09.397] _dn_alloc:259 malloc(32) at 0x25397d0 @ dyn_dict.c:224 [2017-04-21 20:02:09.397] _dn_alloc:259 malloc(24) at 0x2539800 @ dyn_dict.c:359 [2017-04-21 20:02:09.397] core_debug:623 =====================Peers info===================== [2017-04-21 20:02:09.397] core_debug:625 Server pool          : 'dyn_o_mite' [2017-04-21 20:02:09.397] core_debug:628 ============================================== [2017-04-21 20:02:09.397] core_debug:630 	Peer DC            : 'dc2' [2017-04-21 20:02:09.397] core_debug:631 	Peer Rack          : 'rack1' [2017-04-21 20:02:09.397] core_debug:633 	Peer name          : '<IP_ADDRESS>' [2017-04-21 20:02:09.397] core_debug:634 	Peer pname         : '<IP_ADDRESS>:8113' [2017-04-21 20:02:09.397] core_debug:636 	Peer state         : JOINING [2017-04-21 20:02:09.397] core_debug:637 	Peer port          : 8113 [2017-04-21 20:02:09.397] core_debug:638 	Peer is_local      : 1  [2017-04-21 20:02:09.397] core_debug:639 	Peer failure_count : 0  [2017-04-21 20:02:09.397] core_debug:640 	Peer num tokens    : 1 [2017-04-21 20:02:09.397] print_dyn_token:96            	Token : 1 101134286 1  [2017-04-21 20:02:09.397] core_debug:628 ============================================== [2017-04-21 20:02:09.397] core_debug:630 	Peer DC            : 'dc1' [2017-04-21 20:02:09.397] core_debug:631 	Peer Rack          : 'rack1' [2017-04-21 20:02:09.397] core_debug:633 	Peer name          : '<IP_ADDRESS>' [2017-04-21 20:02:09.397] core_debug:634 	Peer pname         : '<IP_ADDRESS>:8101:rack1:dc1:101134286' [2017-04-21 20:02:09.397] core_debug:636 	Peer state         : DOWN [2017-04-21 20:02:09.397] core_debug:637 	Peer port          : 8101 [2017-04-21 20:02:09.397] core_debug:638 	Peer is_local      : 0  [2017-04-21 20:02:09.397] core_debug:639 	Peer failure_count : 0  [2017-04-21 20:02:09.397] core_debug:640 	Peer num tokens    : 1 [2017-04-21 20:02:09.397] print_dyn_token:96            	Token : 1 101134286 1  [2017-04-21 20:02:09.397] core_debug:649 Peers Datacenters/racks/nodes ................................................. [2017-04-21 20:02:09.397] core_debug:653 Peer datacenter........'dc2' [2017-04-21 20:02:09.397] core_debug:657 	Peer rack........'rack1' [2017-04-21 20:02:09.397] core_debug:658 	Peer rack ncontinuumm    : 1 [2017-04-21 20:02:09.397] core_debug:659 	Peer rack nserver_continuum    : 1 [2017-04-21 20:02:09.397] core_debug:653 Peer datacenter........'dc1' [2017-04-21 20:02:09.397] core_debug:657 	Peer rack........'rack1' [2017-04-21 20:02:09.397] core_debug:658 	Peer rack ncontinuumm    : 1 [2017-04-21 20:02:09.397] core_debug:659 	Peer rack nserver_continuum    : 1 [2017-04-21 20:02:09.397] core_debug:662 ............................................................................... [2017-04-21 20:02:09.397] preselect_remote_rack_for_replication:1798 my rack index 0 [2017-04-21 20:02:09.397] preselect_remote_rack_for_replication:1825 Selected rack rack1 for replication to remote region dc1 [2017-04-21 20:02:09.397] _dn_alloc:259 malloc(72) at 0x2539940 @ dyn_task.c:42 [2017-04-21 20:02:09.397] core_start:335 mbuf_size not in YAML: using deprecated way  16384 [2017-04-21 20:02:09.397] mbuf_init:334 mbuf hsize 64 chunk size 16400 offset 16336 length 16336 [2017-04-21 20:02:09.397] core_start:342 max_msgs not in YAML: using deprecated way 200000 [2017-04-21 20:02:09.397] msg_init:667 msg size 448 [2017-04-21 20:02:09.397] core_process_messages:672 length of C2G_OutQ : 0 [2017-04-21 20:02:09.397] event_wait:264 epoll 0004 triggered on conn 0x25364a0 [2017-04-21 20:02:09.397] core_core:565 event FF00 on SERVER 11 [2017-04-21 20:02:09.397] server_connected:369 <SERVER 0x25364a0 11 to '<IP_ADDRESS>:11370:1'> connected  [...] [2017-04-21 20:02:39.397] core_process_messages:672 length of C2G_OutQ : 0 [2017-04-21 20:02:39.398] core_print_peer_status:63 0)0x253a6e0 dc2 rack1 <IP_ADDRESS>:8113 NORMAL [2017-04-21 20:02:39.398] core_print_peer_status:63 1)0x253a790 dc1 rack1 <IP_ADDRESS>:8101:rack1:dc1:101134286 DOWN [2017-04-21 20:02:39.398] _dn_free:305 free(0x2539940) @ dyn_task.c:106 [2017-04-21 20:02:39.407] stats_aggregate:744 skip aggregate of shadow to sum as generator is slow [...]
(Please find the complete logfiles attached.)
In the snippet above, it already says that it's peer is down ([2017-04-21 20:02:39.398] core_print_peer_status:63 1)0x253a790 dc1 rack1 <IP_ADDRESS>:8101:rack1:dc1:101134286 DOWN)
To test it:
`
root@ubuntu:/home/username# redis-cli -p 8102
<IP_ADDRESS>:8102> keys *

"mmm"
"xbx"
"x1.2"
"abc"
"xy"
<IP_ADDRESS>:8102> set foofoo bar
OK
<IP_ADDRESS>:8102> keys *
"foofoo"
"mmm"
"xbx"
"x1.2"
"abc"
"xy"
<IP_ADDRESS>:8102> get foofoo
"bar"
<IP_ADDRESS>:8102>
root@ubuntu:/home/username# redis-cli -p 8114
<IP_ADDRESS>:8114> keys *
"x"
"mmm"
"blabla"
"zert"
"x1"
"zwert"
"foo"
"zurt"
"xnm"
"xbx"
"xy"
`

So, the data was not syncronized. And again, tcpdump does not show any connection attempts to the peers. And there is nothing logged on the peer that indicates any sync or an error during the sync. (see 8101_fail.log and 8103_fail.log)
Now, we recompile dynomite:
 autoreconf -fvi ./configure --enable-debug=full  make  sudo make install
No errors when compiling and installing the new binaries.
Start both dynomite instances the same as we did it before:
 /usr/local/sbin/dynomite -c /home/dynomite/dyno8101.yml  -v 11  /usr/local/sbin/dynomite -c /home/dynomite/dyno8103.yml  -v 11
Now, the connection between both nodes works. Log message for the 8101:
[2017-04-21 20:12:51.734] dnode_peer_connected:996 <LOCAL_PEER_SERVER 0xa69ad0 12 to '<IP_ADDRESS>:8113:rack1:dc1:101134286'> connected
Finally, let's run the same test as above:
`
`
root@ubuntu:/home/username# redis-cli -p 8114
<IP_ADDRESS>:8114> keys *

"x"
"mmm"
"blabla"
"zert"
"x1"
"zwert"
"foofoo"
"foo"
"zurt"
"xnm"
"xbx"
"xy"
<IP_ADDRESS>:8114> set foo1 bar
OK
<IP_ADDRESS>:8114> keys *
"x"
"mmm"
"blabla"
"zert"
"foo1"
"x1"
"zwert"
"foofoo"
"foo"
"zurt"
"xnm"
"xbx"
"xy"
<IP_ADDRESS>:8114> get foo1
"bar"
`

Sure. Let's go..
As said above, we use the dev branch. So, how to reproduce the issue?
autoreconf -fvi cd dynomite/ ./configure --enable-debug=yes  make  sudo make install
It compiles without any errors.
Configuration files:
`
#1st node - dyno8101.yml:
dyn_o_mite:
datacenter: dc1
rack: rack1
listen: <IP_ADDRESS>:8102
dyn_listen: <IP_ADDRESS>:8101
dyn_seeds:

<IP_ADDRESS>:8113:rack1:dc1:101134286
dyn_seed_provider: simple_provider
tokens: '101134286'
servers:
<IP_ADDRESS>:11371:1
data_store: 0
stats_listen: <IP_ADDRESS>:33331
preconnect: true

2nd node dyno8103.yml:
dyn_o_mite:
datacenter: dc2
rack: rack1
listen: <IP_ADDRESS>:8114
dyn_listen: <IP_ADDRESS>:8113
dyn_seeds:

<IP_ADDRESS>:8101:rack1:dc1:101134286
dyn_seed_provider: simple_provider
tokens: '101134286'
servers:
<IP_ADDRESS>:11370:1
data_store: 0
stats_listen: <IP_ADDRESS>:33333
preconnect: true
 Then we start both nodes in the foreground:
/usr/local/sbin/dynomite -c /home/dynomite/dyno8101.yml  -v 11
/usr/local/sbin/dynomite -c /home/dynomite/dyno8103.yml  -v 11
`

tcpdump -i lo does not show any connection or connection attempts between both instances.
Example log:
 [2017-04-21 20:02:09.397] _dn_alloc:259 malloc(16) at 0x2539790 @ dyn_gossip.c:223 [2017-04-21 20:02:09.397] _dn_alloc:259 malloc(9) at 0x25397b0 @ dyn_gossip.c:226 [2017-04-21 20:02:09.397] _dn_alloc:259 malloc(32) at 0x25397d0 @ dyn_dict.c:224 [2017-04-21 20:02:09.397] _dn_alloc:259 malloc(24) at 0x2539800 @ dyn_dict.c:359 [2017-04-21 20:02:09.397] core_debug:623 =====================Peers info===================== [2017-04-21 20:02:09.397] core_debug:625 Server pool          : 'dyn_o_mite' [2017-04-21 20:02:09.397] core_debug:628 ============================================== [2017-04-21 20:02:09.397] core_debug:630 	Peer DC            : 'dc2' [2017-04-21 20:02:09.397] core_debug:631 	Peer Rack          : 'rack1' [2017-04-21 20:02:09.397] core_debug:633 	Peer name          : '<IP_ADDRESS>' [2017-04-21 20:02:09.397] core_debug:634 	Peer pname         : '<IP_ADDRESS>:8113' [2017-04-21 20:02:09.397] core_debug:636 	Peer state         : JOINING [2017-04-21 20:02:09.397] core_debug:637 	Peer port          : 8113 [2017-04-21 20:02:09.397] core_debug:638 	Peer is_local      : 1  [2017-04-21 20:02:09.397] core_debug:639 	Peer failure_count : 0  [2017-04-21 20:02:09.397] core_debug:640 	Peer num tokens    : 1 [2017-04-21 20:02:09.397] print_dyn_token:96            	Token : 1 101134286 1  [2017-04-21 20:02:09.397] core_debug:628 ============================================== [2017-04-21 20:02:09.397] core_debug:630 	Peer DC            : 'dc1' [2017-04-21 20:02:09.397] core_debug:631 	Peer Rack          : 'rack1' [2017-04-21 20:02:09.397] core_debug:633 	Peer name          : '<IP_ADDRESS>' [2017-04-21 20:02:09.397] core_debug:634 	Peer pname         : '<IP_ADDRESS>:8101:rack1:dc1:101134286' [2017-04-21 20:02:09.397] core_debug:636 	Peer state         : DOWN [2017-04-21 20:02:09.397] core_debug:637 	Peer port          : 8101 [2017-04-21 20:02:09.397] core_debug:638 	Peer is_local      : 0  [2017-04-21 20:02:09.397] core_debug:639 	Peer failure_count : 0  [2017-04-21 20:02:09.397] core_debug:640 	Peer num tokens    : 1 [2017-04-21 20:02:09.397] print_dyn_token:96            	Token : 1 101134286 1  [2017-04-21 20:02:09.397] core_debug:649 Peers Datacenters/racks/nodes ................................................. [2017-04-21 20:02:09.397] core_debug:653 Peer datacenter........'dc2' [2017-04-21 20:02:09.397] core_debug:657 	Peer rack........'rack1' [2017-04-21 20:02:09.397] core_debug:658 	Peer rack ncontinuumm    : 1 [2017-04-21 20:02:09.397] core_debug:659 	Peer rack nserver_continuum    : 1 [2017-04-21 20:02:09.397] core_debug:653 Peer datacenter........'dc1' [2017-04-21 20:02:09.397] core_debug:657 	Peer rack........'rack1' [2017-04-21 20:02:09.397] core_debug:658 	Peer rack ncontinuumm    : 1 [2017-04-21 20:02:09.397] core_debug:659 	Peer rack nserver_continuum    : 1 [2017-04-21 20:02:09.397] core_debug:662 ............................................................................... [2017-04-21 20:02:09.397] preselect_remote_rack_for_replication:1798 my rack index 0 [2017-04-21 20:02:09.397] preselect_remote_rack_for_replication:1825 Selected rack rack1 for replication to remote region dc1 [2017-04-21 20:02:09.397] _dn_alloc:259 malloc(72) at 0x2539940 @ dyn_task.c:42 [2017-04-21 20:02:09.397] core_start:335 mbuf_size not in YAML: using deprecated way  16384 [2017-04-21 20:02:09.397] mbuf_init:334 mbuf hsize 64 chunk size 16400 offset 16336 length 16336 [2017-04-21 20:02:09.397] core_start:342 max_msgs not in YAML: using deprecated way 200000 [2017-04-21 20:02:09.397] msg_init:667 msg size 448 [2017-04-21 20:02:09.397] core_process_messages:672 length of C2G_OutQ : 0 [2017-04-21 20:02:09.397] event_wait:264 epoll 0004 triggered on conn 0x25364a0 [2017-04-21 20:02:09.397] core_core:565 event FF00 on SERVER 11 [2017-04-21 20:02:09.397] server_connected:369 <SERVER 0x25364a0 11 to '<IP_ADDRESS>:11370:1'> connected  [...] [2017-04-21 20:02:39.397] core_process_messages:672 length of C2G_OutQ : 0 [2017-04-21 20:02:39.398] core_print_peer_status:63 0)0x253a6e0 dc2 rack1 <IP_ADDRESS>:8113 NORMAL [2017-04-21 20:02:39.398] core_print_peer_status:63 1)0x253a790 dc1 rack1 <IP_ADDRESS>:8101:rack1:dc1:101134286 DOWN [2017-04-21 20:02:39.398] _dn_free:305 free(0x2539940) @ dyn_task.c:106 [2017-04-21 20:02:39.407] stats_aggregate:744 skip aggregate of shadow to sum as generator is slow [...]
(Please find the complete logfiles attached.)
In the snippet above, it already says that it's peer is down ([2017-04-21 20:02:39.398] core_print_peer_status:63 1)0x253a790 dc1 rack1 <IP_ADDRESS>:8101:rack1:dc1:101134286 DOWN)
To test it:
`
root@ubuntu:/home/username# redis-cli -p 8102
<IP_ADDRESS>:8102> keys *

"mmm"
"xbx"
"x1.2"
"abc"
"xy"
<IP_ADDRESS>:8102> set foofoo bar
OK
<IP_ADDRESS>:8102> keys *
"foofoo"
"mmm"
"xbx"
"x1.2"
"abc"
"xy"
<IP_ADDRESS>:8102> get foofoo
"bar"
<IP_ADDRESS>:8102>
root@ubuntu:/home/username# redis-cli -p 8114
<IP_ADDRESS>:8114> keys *
"x"
"mmm"
"blabla"
"zert"
"x1"
"zwert"
"foo"
"zurt"
"xnm"
"xbx"
"xy"
`

So, the data was not syncronized. And again, tcpdump does not show any connection attempts to the peers. And there is nothing logged on the peer that indicates any sync or an error during the sync. (see 8101_fail.log and 8103_fail.log)
Now, we recompile dynomite:
 autoreconf -fvi ./configure --enable-debug=full  make  sudo make install
No errors when compiling and installing the new binaries.
Start both dynomite instances the same as we did it before:
 /usr/local/sbin/dynomite -c /home/dynomite/dyno8101.yml  -v 11  /usr/local/sbin/dynomite -c /home/dynomite/dyno8103.yml  -v 11
Now, the connection between both nodes works. Log message for the 8101:
[2017-04-21 20:12:51.734] dnode_peer_connected:996 <LOCAL_PEER_SERVER 0xa69ad0 12 to '<IP_ADDRESS>:8113:rack1:dc1:101134286'> connected
Finally, let's run the same test as above:
`
`
redis-cli -p 8114
<IP_ADDRESS>:8114> keys *
(empty list or set)
<IP_ADDRESS>:8114> set bar x1
OK
<IP_ADDRESS>:8114> keys *

"bar"
<IP_ADDRESS>:8114> get bar
"x1"

redis-cli -p 8102
<IP_ADDRESS>:8102> keys *

"foo1"
"foobar2"
"x1.2"
"abc"
"bar"
<IP_ADDRESS>:8102> get bar
"x1"
`

Please find the log files attached and let me know if I forgot something.
Br,
Marcel
8101_fail.txt
8101_success.txt
8103_fail.txt
8103_success.txt

Thank you for the details. I will check it

One this is the dyn_seeds in 1st node should be

<IP_ADDRESS>:8113:rack1:dc2:101134286
instead of
<IP_ADDRESS>:8113:rack1:dc1:101134286

Honestly, it works for me with dev branch as well.

I think you are trying from different nodes in the two cases above. Make the change for the dyn_seeds and let me know if that works

Hi,
regardless of whether Marcel had a configuration failure in his example, this problem is very real. I encounter the same issue using completely different configuration files, taken from the exmples under conf/ . See:
Node A (IP: <IP_ADDRESS>)
dyn_o_mite:
  datacenter: dc1
  rack: rack1
  dyn_listen: <IP_ADDRESS>:8101
  dyn_seeds:
  - <IP_ADDRESS>:8101:rack2:dc2:6543210
  listen: <IP_ADDRESS>:8102
  servers:
  - <IP_ADDRESS>:6379:1
  tokens: '12345678'
  secure_server_option: datacenter
  pem_key_file: conf/dynomite.pem
  data_store: 0
  stats_listen: <IP_ADDRESS>:22222

Node B (IP: <IP_ADDRESS>)
dyn_o_mite:
  datacenter: dc2
  rack: rack2
  dyn_listen: <IP_ADDRESS>:8101
  dyn_seeds:
  - <IP_ADDRESS>:8101:rack1:dc1:12345678
  listen: <IP_ADDRESS>:8102
  servers:
  - <IP_ADDRESS>:6379:1
  tokens: '6543210'
  secure_server_option: datacenter
  pem_key_file: conf/dynomite.pem
  data_store: 0
  stats_listen: <IP_ADDRESS>:22222

On Node A:
./src/dynomite -c ./conf/redis_dc1.yml -v 8
On Node B:
src/dynomite -c conf/redis_dc2.yml -v 8
On the first run, after compiling as mentioned in my original report:
./configure --enable-debug=yes

In the interest of brevity, just an excerpt at the end of the logs that illustrate the issue:
Node A:
[2017-04-22 10:03:22.936] server_connected:369 <SERVER 0x1f8b470 11 to '<IP_ADDRESS>:6379:1'> connected
[2017-04-22 10:03:22.936] event_del_out:181 removing conn 0x1f8b470(SERVER) from active

[2017-04-22 10:03:52.933] gossip_loop:792 Gossip is running ...
[2017-04-22 10:03:52.961] core_print_peer_status:63 0)0x1f8f710 dc1 rack1 <IP_ADDRESS>:8101 NORMAL
[2017-04-22 10:03:52.961] core_print_peer_status:63 1)0x1f8f7c0 dc2 rack2 <IP_ADDRESS>:8101:rack2:dc2:6543210 DOWN

As you can see, the peer (Node B) is seen as DOWN
Node B:
[2017-04-22 10:03:19.701] server_connected:369 <SERVER 0x11b3470 10 to '<IP_ADDRESS>:6379:1'> connected
[2017-04-22 10:03:19.701] event_del_out:181 removing conn 0x11b3470(SERVER) from active
[2017-04-22 10:03:49.702] gossip_loop:792 Gossip is running ...
[2017-04-22 10:03:49.725] core_print_peer_status:63 0)0x11b7710 dc2 rack2 <IP_ADDRESS>:8101 NORMAL
[2017-04-22 10:03:49.725] core_print_peer_status:63 1)0x11b77c0 dc1 rack1 <IP_ADDRESS>:8101:rack1:dc1:12345678 DOWN

Note the last line.. the peer (NODE A) is seen as DOWN
Ok.. now, without touching anything else , just make clean and reconfigure so:
./configure --enable-debug=full

after rebuilding with make, restart the dynomite nodes same as above.. this time:
Right away, you see in the respective logs that the connection is established successfully:
[2017-04-22 10:08:59.508] conn_connect:664 connecting on s 15 to '<IP_ADDRESS>:8101:rack1:dc1:12345678'
[2017-04-22 10:08:59.508] dnode_peer_connected:996 <REMOTE_PEER_SERVER 0x1526580 15 to '<IP_ADDRESS>:8101:rack1:dc1:12345678'> connected
[2017-04-22 10:08:59.508] event_del_out:181 removing conn 0x1526580(REMOTE_PEER_SERVER) from active

[2017-04-22 10:09:21.489] gossip_loop:792 Gossip is running ...
[2017-04-22 10:09:21.501] core_print_peer_status:63 0)0x1510750 dc2 rack2 <IP_ADDRESS>:8101 NORMAL
[2017-04-22 10:09:21.501] core_print_peer_status:63 1)0x1510800 dc1 rack1 <IP_ADDRESS>:8101:rack1:dc1:12345678 NORMAL

See the last line..  peer reaches NORMAL state
The same thing applies on the other node:
[2017-04-22 10:08:59.508] dnode_accept:168 Accepting client connection from <IP_ADDRESS>:44204 on sd 15
[2017-04-22 10:08:59.508] event_add_conn:207 adding conn 0x10bdbb0(LOCAL_PEER_CLIENT) to active
[2017-04-22 10:08:59.508] dnode_accept:210 <PEER_PROXY 0x10ba890 13 listening on '<IP_ADDRESS>:8101'> accepted <LOCAL_PEER_CLIENT 0x10bdbb0 15 from '<IP_ADDRESS>:44204'>
[2017-04-22 10:08:59.508] event_del_out:181 removing conn 0x10bdbb0(LOCAL_PEER_CLIENT) from active
[2017-04-22 10:09:27.636] gossip_loop:792 Gossip is running ...
[2017-04-22 10:09:27.666] core_print_peer_status:63 0)0x10be750 dc1 rack1 <IP_ADDRESS>:8101 NORMAL
[2017-04-22 10:09:27.666] core_print_peer_status:63 1)0x10be800 dc2 rack2 <IP_ADDRESS>:8101:rack2:dc2:6543210 NORMAL

See last line.. all is well.
Since the only change was recompiling with the changed debug option I can only conclude that this is the trigger for the problem on the first case, since no configuration file is touched between the runs.
Do you disagree folks?
Thanks
Erick

I'm sorry, the wrong DC was a relict from a former test with a 3rd node.
I set up a new Ubuntu 16.04 VM and am still able to reproduce it with Ericks configuration and my one (including the DC fix ofcourse) as well.

@erickg do not disagree at all :)
@winem I will look into this later today

sorry for the delay. Looking at it now.

Hi @erickg @winem I tried your setup locally and it worked fine with me. I then looked at your logs (thank you for sending that) and here is teh difference :
[2017-04-21 20:02:10.481] dnode_peer_init:300 init 1 seeds and peers in pool 'dyn_o_mite'
[2017-04-21 20:02:10.481] _dn_alloc:259 malloc(280) at 0x1a638f0 @ dyn_connection.c:170
[2017-04-21 20:02:10.481] dnode_peer_ref:48 dyn: ref peer conn 0x1a638f0 owner 0x1a67790 into '<IP_ADDRESS>:8113:rack1:dc1:101134286
[2017-04-21 20:02:10.482] conn_get_peer:378 get conn 0x1a638f0 LOCAL_PEER_SERVER
[2017-04-21 20:02:10.482] _dn_alloc:259 malloc(88) at 0x1a63ab0 @ dyn_dict.c:178
[2017-04-21 20:02:10.482] gossip_set_seeds_provider:860 Seed provider :::::: 'simple_provider'

And here is the success scenario:
[2017-04-21 20:21:30.715] dnode_peer_init:300 init 1 seeds and peers in pool 'dyn_o_mite'
[2017-04-21 20:21:30.715] _dn_alloc:259 malloc(408) at 0x204dc50 @ dyn_connection.c:170
[2017-04-21 20:21:30.715] dnode_peer_ref:48 dyn: ref peer conn 0x204dc50 owner 0x20517d0 into '<IP_ADDRESS>:8113:rack1:dc1:101134286
[2017-04-21 20:21:30.715] conn_get_peer:378 get conn 0x204dc50 LOCAL_PEER_SERVER
[2017-04-21 20:21:30.715] conn_connect:622 <LOCAL_PEER_SERVER 0x204dc50 14 to '<IP_ADDRESS>:8113:rack1:dc1:101134286'> connecting.....
[2017-04-21 20:21:30.715] event_add_conn:207 adding conn 0x204dc50(LOCAL_PEER_SERVER) to active
[2017-04-21 20:21:30.715] conn_connect:664 connecting on s 14 to '<IP_ADDRESS>:8113:rack1:dc1:101134286'
[2017-04-21 20:21:30.715] _dn_alloc:259 malloc(88) at 0x204de90 @ dyn_dict.c:178
[2017-04-21 20:21:30.715] gossip_set_seeds_provider:860 Seed provider :::::: 'simple_provider'

In the failure case, it gets a  connection object in conn_get_peer and then is supposed to connect by calling conn_connect.
Looking at the code for conn_connect:
`
rstatus_t
conn_connect(struct context *ctx, struct conn *conn)
{
rstatus_t status;
// Outgoing connection to another Dynomite node and admin mode is disabled
if ((conn->type == CONN_DNODE_PEER_SERVER) && (ctx->admin_opt > 0))
    return DN_OK;

// Only continue if the connection type is:
// 1. CONN_DNODE_PEER_SERVER: Outbound connection to another Dynomite node
// 2. CONN_SERVER: Outbound connection to backend datastore (Redis, ARDB)
ASSERT((conn->type == CONN_DNODE_PEER_SERVER) ||
        (conn->type == CONN_SERVER));

if (conn->sd > 0) {
    /* already connected on peer connection */
    return DN_OK;
}

conn->sd = socket(conn->family, SOCK_STREAM, 0);
if (conn->sd < 0) {
    log_error("dyn: socket for '%.*s' failed: %s", conn->pname.len,
            conn->pname.data, strerror(errno));
    status = DN_ERROR;
    goto error;
}
log_warn("%M connecting.....", conn);

`
It could silently return without connecting only if conn->sd > 0 or ctx->admin_opt > 0.
Since I am not able to repro, can you add logs reflecting values of these and see what you get?
log_warn("sd:%d admin_opt:%d", conn->sd, ctx->admin_opt);
I checked and its not an unassigned value but lets see what you get

Hi @shailesh33,
how strange it is that you guys don't run into this.. I added the extra debug statement you requested:
[2017-04-26 06:41:19.350] conn_connect:602 admin mode disabled sd:-1 admin_opt:-1

Complete logs attached:
dyno-1-log.txt
Thanks for your help!

Hi @erickg
I am myself puzzled on this. Did you by any chance dig a little deeper and find what is happening in conn_connect in your environment? Looking at the code it should print either
        log_error("dyn: socket for '%.*s' failed: %s", conn->pname.len,
or
    log_warn("%M connecting.....", conn);
do you mind running it with gdb?

let me know what you find.

Hi @shailesh33,
I am not sure if you misunderstood my last comment.. I did do as you suggested and added some extra logs to conn_connect and it is clear that the problem is that it returns silently prematurely because ctx->admin_opt > 0 as the first check at the beginning of the function says. Now the problem is IMHO that that member of the context struct is defined as an unsigned value, so the value of "-1" would be interpreted as<PHONE_NUMBER> which is of course greater than zero. I assume the intention of the code is that -1 should fail that check and proceed with the connection establishment.  Not sure why this is any different depending on the setting of the debug option given to configure.

You might be right there. But the code seem to point that we are setting it to 0.
It seems like a overrun or something.
what is the gcc version, and OS you are compiling on?

Ubuntu 16.04.1 LTS + gcc version 5.4.0

I dug around a bit more.. the problem is that conn_connect() is called before ctx->admin_opt is initialized in dn_run(). The only difference is that in the case where the code is compiled with --enable-debug=full the memory allocated appears not to have been zero'ed.. so it contains 0xFFFFFFFF (-1) ergo, the admin_opt>0 check succeeds and the conn_connect() call returns prematurely. When the code is compiled with --enable-debug=yes the memory appears to have been zero'ed out, so the check fails and thus conn_connect() succeeds even though it hasn't been officially initialized (in dyn_run())
The reason why you don't see this problem might just be that by pure chance or some other obscure reason, the uninitialized memory for you folks contains zeros.. where for us it contains all ones..
If you start gdb with some breakpoints in dyn_run() and in conn_connect() you will see that the ctx->admin_opt member is used in conn_connect before it has been initialized in dyn_run()
Thanks
Erick

Thank you. I will work on a fix for dev today.
On Thu, Apr 27, 2017 at 5:00 AM Erick Gonzalez<EMAIL_ADDRESS>wrote:

I dug around a bit more.. the problem is that conn_connect() is called
before ctx->admin_opt is initialized in dn_run(). The only difference
is that in the case where the code is compiled with --enable-debug=full
the memory allocated appears not to have been zero'ed.. so it contains
0xFFFFFFFF (-1) ergo, the admin_opt>0 check succeeds and the
conn_connect() call returns prematurely. When the code is compiled with
--enable-debug=yes the memory appears to have been zero'ed out, so the
check fails and thus conn_connect() succeeds even though it hasn't been
officially initialized (in dyn_run())
The reason why you don't see this problem might just be that by pure
chance or some other obscure reason, the uninitialized memory for you folks
contains zeros.. where for us it contains all ones..
If you start gdb with some breakpoints in dyn_run() and in conn_connect()
you will see that the ctx->admin_opt member is used in conn_connect
before it has been initialized in dyn_run()
Thanks
Erick
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Netflix/dynomite/issues/437#issuecomment-297693514,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALle53uSsdLENGptqayZpU97p03Iyawdks5r0INBgaJpZM4NEfYs
.

@erickg I couldnt assign you to the PR as a reviewer. But please feel free to have a look.


Connection broken errors

Hello,
Has anyone experienced errors like "(104) Connection reset by peer", or similar?
It seems to happen sometimes, and it stops the bot.
Would it make sense to implement requests in "try, exception" modules, and perform a new login if the connection is interrupted?
Thank you

Can you describe how to get this error? It will help to find and fix it

Sure, it appears the following text:
File "/home/maf/Desktop/BOT/../instabot/api/api.py", line 340, in getUserFollowers
return self.SendRequest('friendships/' + str(usernameId) + '/followers/?rank_token=' + self.rank_token + '&max_id=' + str(maxid))
File "/home/maf/Desktop/BOT/../instabot/api/api.py", line 109, in SendRequest
response = self.session.get(config.API_URL + endpoint) # , verify=False
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 501, in get
return self.request('GET', url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 641, in send
r.content
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 797, in content
self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 722, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: error(104, 'Connection reset by peer')", error(104, 'Connection reset by peer'))

yes, i have seen this, it because the connection are not stable. after resetting my modem, and restart the script, it gone. @ohld maybe we must implement connection lost handling. so, when connection lost, the script pause and tell error, therefore resume it.

Yes, this is what I was thinking about.
For me it is enough to restart the "run" script, without actually restarting the modem. It would be good to add an auto-reconnect option...
I also have to say that it happens much more often if likes are enabled, if I do only follow/unfollow it rarely occurs.


Bootstrap 4?

Support for bootstrap 4?  What would it take?

Hi @tblanchard , see https://github.com/ba-st/Willow-Bootstrap

That's great!
I will give it a try.
You should update this part of the Willow Readme.md to reduce confusion.

On Apr 30, 2020, at 2:38 PM, Francisco Ortiz Peñaloza<EMAIL_ADDRESS>wrote:
Hi @tblanchard https://github.com/tblanchard , see https://github.com/ba-st/Willow-Bootstrap https://github.com/ba-st/Willow-Bootstrap
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/ba-st/Willow/issues/162#issuecomment-622128594, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABIVKBMRBQNKJYENK25PLLRPHVWHANCNFSM4MWUUOSA.

I've updated the readme. I'm pushing some updates on Willow Boostrap now, but if you take the latest release you should be Ok.


Add docker-compose health checker  …

To make sure that gateway start up completely before running integrated tests
This is used to resolved the problem that integrated tests are not stable

Make sure that:

[x] You have read the contribution guidelines.
[x] You submit test cases (unit or integration tests) that back your changes.
[x] Your local test passed mvn clean install -Dmaven.javadoc.skip=true.

Codecov Report

Merging #1695 (f3752cc) into master (13663c6) will decrease coverage by 0.38%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #1695      +/-   ##
============================================
- Coverage     60.47%   60.08%   -0.39%
+ Complexity     2273     2260      -13
============================================
  Files           477      477
  Lines         10107    10107
  Branches       1028     1028
============================================
- Hits           6112     6073      -39
- Misses         3492     3538      +46
+ Partials        503      496       -7

Impacted Files
Coverage Δ

...enyu/client/tars/TarsServiceBeanPostProcessor.java
25.67% <0.00%> (-33.79%)
:arrow_down:

...henyu/plugin/grpc/resolver/ShenyuNameResolver.java
45.26% <0.00%> (-18.95%)
:arrow_down:

...java/org/apache/shenyu/common/utils/UUIDUtils.java
75.00% <0.00%> (-6.25%)
:arrow_down:

...henyu/plugin/httpclient/NettyHttpClientPlugin.java
57.14% <0.00%> (-2.05%)
:arrow_down:

...elimiter/handler/RateLimiterPluginDataHandler.java
69.23% <0.00%> (-1.54%)
:arrow_down:

.../plugin/grpc/loadbalance/AbstractLoadBalancer.java
68.47% <0.00%> (-1.09%)
:arrow_down:

...che/shenyu/sync/data/http/HttpSyncDataService.java
71.84% <0.00%> (+8.73%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 13663c6...f3752cc. Read the comment docs.


chore(deps): upgrade dependencies

Upgrades project dependencies. See details in workflow run.

Automatically created by projen via the "upgrade-2.x" workflow

⚪ Backport skipped
The pull request was not backported as there were no branches to backport to. If this is a mistake, please apply the desired version labels or run the backport tool manually.
Manual backport
To create the backport manually run:
backport --pr 458

Questions ?
Please refer to the Backport tool documentation


Variable with name "module" throws error when imported in component

Version
2.6.1
Reproduction link
https://codesandbox.io/s/zwvk4z2nl3
Steps to reproduce

Create new nuxt application.
Create test.js file in e.g. /store directory with export const module = {} line.
Add import {module} from "~/store/test" line in pages/index.vue
npm run dev and open application in browser.

What is expected ?
Page should be displayed without an error.
What is actually happening?
"Identifier 'module' has already been declared" error

This bug report is available on Nuxt community (#c9005)

It is not a bug. module is global object in node .
You can use module.exports = { variables } to export variables to another module .
So when you are using const to redefine the module, it will throw expection .

Ok, fair enough. That's a little strange however - between Nuxt versions 2.2 to 2.4 it worked flawlessly. Something changed in 2.5 or 2.6 that makes this node's "module" exposed?


405 METHOD NOT ALLOWED

I'm trying to make a call using fetch but no matter what site I try to hit I get a 450 Method not allowed.
I'm using it this way:
const options = {
  "method": "GET",
  "headers": {
     "accept":  "application/json",
     "content-type": "application/json"
   }
};

fetch('http://crossorigin.me/http://google.com', options)
        .then(res => res.text())
        .then(res => console.log(res));

output:
VM912:2 OPTIONS http://crossorigin.me/http://google.com 405 (Method Not Allowed)

(index):1 Fetch API cannot load http://crossorigin.me/http://google.com. Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:3003' is therefore not allowed access. The response had HTTP status code 405. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

Is this a bug or am I just not using crossorigin.me correctly?

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Couldn't replicate that - could you give me some more details on the context you're using that code in?

It's a frontend application using GitHub's Fetch polyfill to make the request. This happens both when trying to make the call from my script or when copying the code into the developer tool's console and triggering it from there. This is currently being run from a local development node server at localhost:3000

I'm getting the same error in response for other requests.
E.g.:
$.post("http://crossorigin.me/http://google.com");
...
POST http://crossorigin.me/http://google.com 405 (Method Not Allowed)

@Banderi That's because POST isn't implemented, so it's not allowed.

You mean $.post isn't, or the POST method altogether isn't? In any case, without 50% of the ajax working I find this service quite lacking.. don't get me wrong, I appreciate the free service, but I thought the whole point would be to proxy simple ajax requests.


Fix Text's .getBounds() returning null when text is 0

Fixes this issue: https://github.com/CreateJS/EaselJS/issues/550
Basically:

0 == "" // true
0 === "" // false

Other way it could be dealt with is by converting to string.
this.text = String(text);

Thanks for this. I've implemented this without the strict equality for null, as we want it to catch undefined as well.

Didn't thought of that, thanks!


Tracking issue for no-std compliance for dependencies in ibc-rs and tendermint-rs

This issue is part of #1158 to track the no-std support in dependencies that are used by ibc-rs and tendermint-rs. To fully support ibc-rs to be used in no-std environment like Substrate, we must also fix the dependencies by either fixing it upstream, or remove the dependencies from ibc-rs and tendermint-rs and replace them with alternatives that support no_std.
The checks are done by running no-std-check with only the specific unsupported dependency activated. Currently the checks are done from the branch at #1358.

[x] ics23 - Upstream fix yet to be merged in https://github.com/informalsystems/ics23/tree/soares/no-std.
[ ] safe-regex - Upstream fix yet to be merged in https://github.com/informalsystems/safe-regex.
[x] crossbeam-channel - Requires std and is currently used by tendermint-light-client.

Removed

[x] tonic

Fails in all checks. We will have to remove the dependency to tonic from ibc-proto and tendermint-proto
Removed in #1439

[x] socket2

Fails in all checks. It uses std and does not support Wasm targets.

[x] getrandom

Fails in check-wasm and check-substrate.

[x] ics23

Fix is merged in https://github.com/confio/ics23/pull/41 but the changes are still unreleased.

[x] thiserror

Fails in all checks. We should have already removed all use of thiserror with flex-error, but still needs to double check.
Removed in #1390.

[x] regex

Fails in check-wasm and check-substrate.
Use safe-regex as replacement in #1395

[x] sled

Fails in all checks. We may have to remove dependency to sled.

[x] tokio

Fails in all checks.

[x] toml

Fails in all checks.
Dependency removed in informalsystems/tendermint-rs#986

[x] url

Fails in all checks.
Pending PR: servo/rust-url/pull/717
Dependency removed in informalsystems/tendermint-rs#986

For Admin Use

[ ] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate milestone (priority) applied
[ ] Appropriate contributors tagged
[ ] Contributor assigned/self-assigned

Closing this with #1858 tracking the last remaining dependency fix for safe-regex.


Run provider-hosted low-trust SharePoint addin from provider side

Category

[x] Question
[ ] Typo
[ ] Bug
[ ] Additional article idea

Hello. Is it possible to run provider-hosted low-trust SharePoint addin from provider side? I already have installed my SharePoint provider-hosted addin on my site. And I want to run this addin from my side. How can I do it?

Not sure what exactly this actually means... as not understanding the "running addin from your side". You can absolutely have server-side or client-side code in your provided hosted add-in which can or not connect to SharePoint Online. If you explain your objective and scenario, we can most likely help you more easily as that would clarify what you are trying to achieve. Thx.

@VesaJuvonen Ok, I will try to explain, what I want to do. I am trying to get access to OneDrive and Sharepoint files from provider-hosted addin (in case of some limitations I can't do it with MS Graph API). What I want to do exactly: write some addin, deploy it to a tenant. Then from provider side run my program and e.g. list all files in OneDrive, which is hosted on a tenant, on which I deployed my addin. Is it possible?
As far as I understand, it's not possible, because Sharepoint addin should initiate process: Sharepoint addin goes to Azure, get token, goes to provider-hosted code with this token. Then provider-hosted code goes to Azure with token from Sharepoint addin, get another access token. And with this access token we can work with a tenant. Correct me please, if I am wrong.


[triton][tool] A CLI Tool for Tensor Layout Printing

A CLI tool to print the layout of a tensor. Currently, only triton_gpu's DistributedEncoding (no SharedEncoding) tensor layout print is supported via the exposed getLayoutStr API from the dialect library. In the future, we could also add more tensor layout print from other backend HW targets (e.g., CPU).
Example usage:
triton-tensor-layout -l "#triton_gpu.nvidia_mma<{versionMajor = 3, versionMinor = 0, warpsPerCTA = [8, 1], CTAsPerCGA = [1, 1], CTASplitNum = [1, 1], CTAOrder = [1, 0], instrShape = [16, 256, 32]}>" -t "tensor<128x256xf16>"

triton-tensor-layout -i input.mlir -t "tensor<1x128x128xf16>" -o output.txt

triton-tensor-layout -i input.mlir -t "tensor<1x128x128xf16>" -o output.txt -alias-names="blocked,mma" -use-hw-view

An input file usually looks like:
#mma = #triton_gpu.amd_mfma<{versionMajor = 2, versionMinor = 0, warpsPerCTA = [1, 1, 8], instrShape = [32, 32], isTransposed = false}>
#blocked = #triton_gpu.blocked<{sizePerThread = [1, 8, 1], threadsPerWarp = [1, 16, 4], warpsPerCTA = [1, 1, 8], order = [0, 1, 2]}>

The core Triton is a small number of people, and we receive many PRs (thank
you!).  To help us review your code more quickly, if you are a new
contributor (less than 3 PRs merged) we ask that you complete the following
tasks and include the filled-out checklist in your PR description.
Complete the following tasks before sending your PR, and replace [ ] with
[x] to indicate you have done them.

[x] I am not making a trivial change, such as fixing a typo in a comment.

[x] I have written a PR description following these
rules.

[x] I have run pre-commit run --from-ref origin/main --to-ref HEAD.

Select one of the following.

[ ] I have added tests.

/test for lit tests
/unittest for C++ tests
/python/test for end-to-end tests

[x] This PR does not need a test because FILL THIS IN.

Select one of the following.

[x] I have not added any lit tests.
[ ] The lit tests I have added follow these best practices,
including the "tests should be minimal" section. (Usually running Python code
and using the instructions it generates is not minimal.)

Sounds like a cool tool! Can we add a test for it (could be a lit test) otherwise it is likely to silently break at some point


Brought up to date

Found this really handy for the next couple of projects we're starting, so did a bit of updating.
Decided it was easier to just leave out the version numbers on most gems, since newest version of everything seemed to work fine.
Probably going to keep adding templates to it.
One thing I haven't gotten working yet, which would be good, is some way to apply it to an existing repo. Would mean introspecting the files that were going to be updated, to avoid duplicate entries. Potentially might be hard, but I'm sure there'd be a way to do it.
Couldn't figure out where the "gem" method was defined for use in the templates. How's that work?
Anyway... ta!

Oh, also, if you wanted to add me to the authors on rubygems.org I could do a gem release.


ATD-332 Change handling of OCI API private key for security purposes

It was possible to glean the OCI API private key for the cncf-ci tenancy
from the Gitlab log output. We revoked that key. This change removes the
API key from the TF template handling and moves it to the provision
script. They key should no longer be exposed now.

Looks good, thanks! @hbscharp


Avoid sitemap.xml error on Search Console

Issue detail:

Empty self-closing changefreq element is not allowed on Search Console
There is a possibility that $changefreq is a object if the attribute type is select

Thanks


Can we move PyXFormTestCase and related functionality into pyxform.test_utils

Software and hardware versions
pyxform v1.7.0 - v1.9.0, Python 3.8.13
Problem description
I have been using the functionality within pyxform_test_case in writing tests using markdown to minimise or eliminate the use of Excel files within the codebase. I believe the change introduced by the PR https://github.com/XLSForm/pyxform/pull/562 eliminated the inclusion of the package during distribution and as such the module is not available in third party code bases that rely on it.
Steps to reproduce the problem
Python 3.8.13 (main, Mar 24 2022, 04:09:24) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> from pyxform.tests.pyxform_test_case import PyxformMarkdown
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pyxform.tests'

Expected behaviour
Could we make this module available under pyxform.test_utls or any relevant path and allow the continued use by third party code bases for writing tests? The goal is the same as was for this package to reduce the use of binary XLSForm files.

@lindsay-stevens @lognaturel Thoughts?

minimise or eliminate the use of Excel files within the codebase.

The codebase you're referencing here is your external system (presumably Ona)? Are you writing additional tests that verify the conversion functionality? If so, could those be added here? Or are you using pyxform_test_case to test Ona functionality somehow? Maybe you're just using PyxformMarkdown, I guess that's what your sample import suggests? That's definitely unexpected usage to me so we should figure out just what the public API should be and make it clearly a public API.

Ah, actually, I think I just remembered that Ona uses the pyxform json representations internally instead of the actual XForm, is that right? So you use that pyxform json representation to test Ona features that interact with the form def in some way?

We use both the JSON as well as the XForm representation. Yes, in my case this is onadata, for example we use the method self.md_to_pyxform_survey(md_xlsform, kwargs=kwargs) then access the XML and JSON representation accordingly which is the goal.
survey = self.md_to_pyxform_survey(md_xlsform, kwargs=kwargs)
...
xml = survey.to_xml()
json=survey.to_json()
...

I have not done much isolation other than treating the two files as all public https://github.com/XLSForm/pyxform/pull/600/files.

Thanks for clarifying the usage! That all makes sense.
My preference would be to make PyxformMarkdown into its own module (e.g. md2pyxform) if it needs to be part of the public interface. As far as I can tell there's no reason for it to be a class and that would avoid multiple inheritance.
@lindsay-stevens I'd like to read your thoughts on what a good structure would be.

In my view tests are private code, and as a result any changes to the tests are not mentioned in the changelog and these changes don't affect versioning. So I don't think it's good practice for 3rd party code to rely on tests. I wonder if those 3rd party tests using the pyxform test code are perhaps tests that should be contributed as pyxform tests?
Another issue for onadata is that it seems to support both python 3 and 2, whereas pyxform supports only 3.7+ only.
Anyway, some suggestions:

No change: the PyxformMarkdown class and dependency function md_table_to_ss_structure are a total ~100 lines - 3rd party libs could copy this to their codebase with the pyxform license copyright notice.
No change: 3rd party libs requiring pyxform test code can clone the repo, or install from GitHub instead of PyPi e.g. pyxform<EMAIL_ADDRESS>Packaging change: pyxform publishes a separate tests package on PyPi such that 3rd party libs can have deps on e.g. "pyxform" and "pyxform_tests" (more or less same outcome as above option 2).
Code change: we polish up the test markdown parser functions (e.g. no class, support escaping pipes per GFM tables spec, upgrade parser to use scanner like in #578) and put them in the main pyxform package under md2pyxform (or similar name).

Option 4 could have other benefits beyond meeting the request in this ticket:

If markdown is sufficiently incorporated into the pyxform codebase, users could use it as a plain-text (git trackable) input option as an alternative to XLS/XLSX. This may obsolete the yxf project which for the same reason is doing this sort of thing with YAML.
The markdown code could be used in a discourse bot that converts or validates markdown forms in the ODK forum into XForms 🤖

No change: the PyxformMarkdown class and dependency function md_table_to_ss_structure are a total ~100 lines - 3rd party libs could copy this to their codebase with the pyxform license copyright notice.

Seems like a path of least effort, will consider this in the short term.

No change: 3rd party libs requiring pyxform test code can clone the repo, or install from GitHub instead of PyPi e.g. pyxform<EMAIL_ADDRESS>

I did not have much success with this option.

Code change: we polish up the test markdown parser functions (e.g. no class, support escaping pipes per GFM tables spec, upgrade parser to use scanner like in 495: Default string values with dashes are mistakenly treated as dynamic defaults #578) and put them in the main pyxform package under md2pyxform (or similar name).

I'm keen on Option 4, having it as part of the library will make things far much easier.
I will take down the PR #600. and leave the issue open hoping we pursue option 4 instead.

I like the way y'all think!

a discourse bot that converts or validates markdown forms in the ODK forum into XForms 🤖

Wild.


Moving cloudflare (hybrid node compat) to external preset

Having both unenv + cloudflare preset for wrangler hybrid node_compat in same repo/versioning has proven to be a tricky process.
The idea of this external preset is to split areas clearly:

unenv: The base/polyfill-only source for Node.js compatibility (nodeless)
unenv-preset-cloudflare: has full control over hybrid layer composition on top of unenv

Versioning
By this split, unenv versioning focuses mainly on polyfill improvements and bug fixes with both stable channels and nightly channels
I believe unenv-preset-cloudflare might need to also have two channels and stable channel releases making sure the hybrid composition, matches production tag of workerd and is also in sync with wrangler SDK.
Tests
I think the focus of these preset tests would be testing functionality with node_compat.
The focus of unenv would then be polyfill-only coverage.
porting current workerd tests is time-consuming because it was so hacky but as an alternative, since this setup has fewer constraints I think we might directly use vitest-environment-cloudflare somehow.
Ownership
With this split, I would love to have the cloudflare team having full ownership.
I would be more than glad to have cloudflare team oversights on unenv development as well, and also I would love to keep helping on this preset because we might need an extension of it for Nitro/Nuxt support to avoid duplicating efforts.
I made this POC quickly in my personal namespace and invited major contributors, if you want to move it to a better place, please do! (i would personally suggest it being part of SDK monorepo to help saving time!)
Transition
I think we can have as much as time as needed. unenv is not going to break wrangler dependency until we adopt this repository.
I would appreciate any help on making this happen faster btw since it unblocks next steps for unenv v2 release.

Notes form chat with @vicb

We should be aware of any internal/ imports (like this) to avoid breaking changes


Pop up calendar alignment issue after adding bootstrap

after adding bootstrap calendar is not aligned properly please let me how i can fix this.
i have attached the file that shows the issue.
thank you

This is intended for ionic, which doesn't use bootstrap. If you wish to override this component css, you have the classes which i mentioned in the readme.

thanks a lot.
is it possible to change colours using css?


fix(audio-player): allow localized labels for rewind/forward/caption buttons

Related Ticket(s)
#9279
Description
This change allows authors to include localized labels for the rewind, fast-forward, and closed captions buttons.
Changelog
New

props to allow user-defined labels for rewind, forward, and closed captions buttons

It sounds like the label read out is right for Forward and Rewind, but I am seeing that the Closed captioning button is announced as "open and close list of options" when using VoiceOver

It sounds like the label read out is right for Forward and Rewind, but I am seeing that the Closed captioning button is announced as "open and close list of options" when using VoiceOver.

@RichKummer this behavior is coming from the core Carbon component. The title displays correctly (on hover) but screen reader does not pick it up. We'll probably need to see if an issue is open on their end, and if not we can create one.


Sensor Status

Hi - I have just started using WeeWX with a Davis WeatherLink Live connected to a Vantage Pro 2.
Your driver works brilliantly - thank you so much - https://wx-watch.co.uk/weewx
The station is set to update every 5 minutes.
A question if I may. The Sensor Status does not always display correctly - mostly the only info is the Transmitter Battery Status, but sometimes full status is displayed - is there any fix for this.

For info this website has been moved to https://longsutton-weather.co.uk

Hi,
Battery info, signal quality are refreshed only each 15min. If you've an archive interval which is set to 5min, some values will be not shoping while is not refresh.
-Drealine

Hi
Many thanks for your reply - is the 15 minute refresh rate built into your driver or is it built into the Vantage pro?
If I alter my station refresh rate to 15 minutes does it mean it will always display correctly?
Again many thanks

Hi,
No problem. It's a limitation from the API v2 of Weatherlink.com. Nothing to do at the moment..
I've just created an issue on their Github to make an API for retrieve archive records. At the moment, it's not possible, but the WLL module can make do that. I suppose that it favorite their cloud data for historical before our own proper data... That is the choice, but no too good for peoples who doesn't want to pay an additionnal subscription to have more historical data.
-Drealine

Many thanks for your quick reply


Add type guard compatible overload for O.filter

The O.filter function doesn't have an overload that takes type guards into account. The only definition that exists looks like this:
export declare function filter<A>(predicateFn: (value: A) => boolean): (option: Option<A>) => Option<A>;

Compare this to A.filter which has an overload that allows for a type guard to be passed in:
export declare function filter<A, B extends A>(predicateFn: (value: A) => value is B): (xs: Array<A>) => Array<B>;

It would be nice to have the same for O.filter.
Here's an example that currently fails:
type WithOrWithoutYou = {type: 'with', you: string} | {type: 'without'};

declare const withOrWithout: WithOrWithoutYou;

pipe(
  O.some(withOrWithout),
  O.filter(w => w.type === 'with'),
  O.map(w => w.you), // <--- Fails with: Property 'you' does not exist on type 'WithOrWithoutYou'.
)

Thanks for a great library! 🙌

Hi. I made a PR to resolve this issue. please have a look and let me know if it's right.


IAM authentication via Workload Identity not working

Bug Description
Hi, I am trying to authenticate to my postgresql 13 instance using the cloudsql-proxy and workload identity.
I've done following things:

set cloudsql.iam_authentication=on flag on CloudSQL
create user<EMAIL_ADDRESS>      CLOUD_IAM_SERVICE_ACCOUNT  and checked with sql users --list
check the IAM policy binding

shell
% gcloud --project="xxx" iam service-accounts get-iam-policy<EMAIL_ADDRESS>bindings:
- members:
  - serviceAccount:xxx.svc.id.goog[dev-worker/dev-worker]
  role: roles/iam.workloadIdentityUser
etag: BwXAT81sc0w=
version: 1

added applicable roles to the Pod SA

% gcloud projects get-iam-policy "xxx"  \
--flatten="bindings[].members" \
--format='table(bindings.role)' \
<EMAIL_ADDRESS>ROLE
roles/cloudkms.cryptoKeyEncrypterDecrypter
roles/cloudsql.client
roles/cloudsql.instanceUser
roles/pubsub.publisher
roles/pubsub.subscriber

My cloudsql-proxy sidecar:
 imagePullPolicy: Always
        image: eu.gcr.io/cloudsql-docker/gce-proxy:1.21.0
        name: cloudsql-proxy

        command:
        - '/cloud_sql_proxy'
        - '-enable_iam_login'
        - '-instances=xxx:europe-west1:mydb=tcp:3126'

        resources:
          requests:
            cpu: 0.5
            memory: 0.5Gi
          limits:
            cpu: 0.5
            memory: 0.5Gi

        securityContext:
          readOnlyRootFilesystem: true
          privileged: false
          runAsNonRoot: true
          allowPrivilegeEscalation: false
          capabilities:
            drop: [all]
          seccompProfile:
            type: RuntimeDefault

Stacktrace
Get "https://sqladmin.googleapis.com/sql/v1beta4/projects/xxx/instances/europe-west1~mydb?alt=json&prettyPrint=false": compute: Received 403 `Unable to generate access token; IAM returned 403 Forbidden: The caller does not have permission
This error could be caused by a missing IAM policy binding on the target IAM service account.
For more information, refer to the Workload Identity documentation: https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#creating_a_relationship_between_ksas_and_gsas

Environment

OS type and version: GKE AutoPilot
Cloud SQL Proxy version (./cloud_sql_proxy -version): eu.gcr.io/cloudsql-docker/gce-proxy:1.21.0

A few things to check:

Did you enable workload identity on your cluster and your node pool?
Did you enable an IAM binding between your KSA and your SA?
Did you annotate your KSA with your SA info?

There's a full example here that you can follow.

It seems I have to provide -ip_address_types=PRIVATE when my cloudsql instance has a public endpoint for workload identity to work, is this correct?
If so, we'll need to add this into the docs.

It seems I have to provide -ip_address_types=PRIVATE when my cloudsql instance has a public endpoint for workload identity to work, is this correct?
If so, we'll need to add this into the docs.

That's not correct. You only need to pass in -ip_address_types=PRIVATE if you want to force the proxy to use the instances private IP (which is documented in the README).
I would agree w/ @enocom's guess that you do not have WI set up correctly. I'd suggest following the documentation linked  and verifying you haven't missed any steps. For example, check to make sure you have a KSA set up in the dev-worker namespace named dev-worker and that you've added it you your application.

In addition to checking the points I raised above, you might look at the example Workload identity setup if you're working with Terraform. Nonetheless, it might be worth going through the example in this repo manually, just to verify your setup.

Thank you for the documentation refs.


Double header cell when scrolling in Firefox

When scrolling in GradebookNG, the student header cell is being copied and is floating down with the scroll.

Any tips on replicating this one? I haven't had much luck.

Verified. Thanks @payten :cake:


Adds support for explicit GSI names

Following up on recent comments in https://github.com/architect/package/pull/92 related to architect/architect#974 this is a suggestion for how to add explicit GSI names to Architect

love it; will defer to @ryanblock to merge but we'll get this in for the next release

love it; will defer to @ryanblock to merge but we'll get this in for the next release

👍 Awesome, thanks everyone

👍 Awesome, thanks everyone


Add optional timestamp to all logs

Timestamps within every log, if requested on creation of Logger class
Add optional parameter to enable or disable timestamp on each logging function

Fulfilled


Unable to download subtitle for "ja-jp" from TVer

When I try to download a video subtitle from TVer, it said "WARNING: Unable to download subtitle for "ja-jp": Unable to download webpage: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.". A few days ago, ytdl easily download these subtitle from TVer, but now "unable to download subtitle for "ja-jp"". My ytdl is up to date. Can you fix this bug?

utube-dl with the --verbose

My command is : "youtube-dl.exe --all-subs URL ", I tried too many links, but still can't download these subtitles. Is my command goes wrong?


Update README.md to include more awesome channels

Added links to channels:
1)Code.org
2) Ben Eater
3)Computerphile
4)GSuite

Hey @Kush1101,
Computerphile and Ben Eater already exist.

I am sorry I didn't check that. Thanks for your consideration.


NetworkActivityIndicatorManager + Siesta stucks forever

Have anybody used NetworkActivityIndicatorManager with Siesta + Alamofire?
I have a problem where two alamofire requests are started and only one DidFinish notification is called.. and thus network indicator spinner is stuck forever...
I'll try to make a demo project in next few days to demonstrate the problem...
All what I'm doing is adding to AppDelegate NetworkActivityIndicatorManager.shared.isEnabled = true
https://github.com/Alamofire/AlamofireNetworkActivityIndicator/issues/23

Yes, please put together an example project that minimally reproduces the problem.
Does the same problem occur with Siesta’s NetworkActivityIndicator?

@pcantrell https://www.dropbox.com/sh/i8aeqd0z1o74xrh/AACFLcR3lXIqQvnaVNjBkyvZa?dl=0
No, Siesta NetworkActivityIndicator uses request decorators, and that's called once per each request. Otherwise NetworkActivityIndicatorManager is subscribing to Alamofire notifications and they are called - Resume - twice; Finished - once, thus the counter is not becoming zero.
I personally like this library, because it ensures some time before and after spinner appears/disappears...

I suggest not to move such functionality inside Siesta, maybe some extension to the Siesta...
I got it working with some hack:
        NotificationCenter.default.removeObserver(NetworkActivityIndicatorManager.shared)
        service.configure {
            $0.decorateRequests { resource, request in
                NetworkActivityIndicatorManager.shared.incrementActivityCount()
                request.onCompletion { _ in
                    NetworkActivityIndicatorManager.shared.decrementActivityCount()
                }
                return request
            }
        }

right after NetworkActivityIndicatorManager.shared.isEnabled = true.
The exact problem is in Notifications called by Alamofire... If we disable notifications by Alamofire for network activity indicator and call increment/decrement with decorate requests block then all is fine...

@pcantrell Hm... it seems the problem is inside AlamofireProvider...
    init(_ alamofireRequest: Alamofire.Request)
        {
        self.alamofireRequest = alamofireRequest
        alamofireRequest.resume()   // in case manager.startRequestsImmediately is false
        }

And what if manager.startRequestsImmediately = true? calling alamofireRequest.resume() fires new Alamofire.DidResume notification, and it increments a counter inside NetworkActivityIndicatorManager...
Where is the problem? We shouldn't call alamofireRequest.resume() on a started request, or NetworkActivityIndicatorManager should handle this situation?

Thanks for tracking that down, @piv199. I’d say that firing Alamofire.DidResume for a request that’s already started sounds like a bug in Alamofire. Someone should file that with AF.
Still, it’s reasonable to work around the issue. Does AF provide a way to test whether a request is already in progress before we call resume()?

@pcantrell
Alamofire.Request.task?.state provides the current state of the task:
        case running
        case suspended
        case canceling
        case completed

It looks like we shouldn’t call resume() unless it’s suspended. Would you be interested in taking on the fix, @piv199? You’re in a good position to test it, since you can see the bug happening.


PS#2 Download format

Hi!
I'm trying to download my Rmarkdown file in PDF but every time I try to knit I get the following error and only the graphics are saved as PDF.

Does it happen to someone else or know how to fix it?

@andreagomezp: we mentioned this in class: to compile in pdf you need to install latex (this is mentioned in the book as well). You can click on the arrow next to "Knit" and choose to compile as HTML.


Handle trailing comma at utils.getParamNames.

This was a bug at fn-args library that got fixed with https://github.com/sindresorhus/fn-args/pull/10 . Before upgrading the fn-args library the test I added fails with:
  1 failing

  1) utils
       #getParamNames
         should handle trailing comma:

      AssertionError: expected [ 'I', 'trailing', 'comma', '' ] to deeply equal [ 'I', 'trailing', 'comma' ]
      + expected - actual

       [
         "I"
         "trailing"
         "comma"
      -  ""
       ]

      at Context.it (test/unit/utils_test.js:18:21)

The upgrade fixes that issue:
✓ should handle trailing comma

@DavertMik sorry to bother. Any idea when you think you could release a new version with the fix? Thanks.


/run/fleet/units/ is not cleaned after service destruction

I noticed that /run/fleet/units/ has about 40k dead symlinks of previously ran services.
Way to reproduce:
<EMAIL_ADDRESS>[Unit]

[Service]
KillMode=none
ExecStart=/bin/bash -c 'echo  "%i $(date) ExecStart" >> /home/core/test/log'
ExecStop=/bin/bash -c 'echo "%i $(date) ExecStop" >> /home/core/test/log'

fleetctl submit<EMAIL_ADDRESS>fleetctl start ts@123
fleetctl destroy ts@123

Now there is link<EMAIL_ADDRESS>-><EMAIL_ADDRESS>which is dead.
fleetd version 0.10.2
CoreOS beta (766.3.0)

I just tried to reproduce it with current master (7fba3d13) on CoreOS stable (835).
No luck. Isn't it probably already fixed?


Switch to valuespec instead of gendecl for hardcoded credential rule

The current hardcoded credentials rule fails to handle #nosec comments in const decl blocks, like
const (
    Token = "qw12r4jjwq1rdvsp" //#nosec
)

This change allows this to be handled by the existing analyzer ignore system.

Thanks. Nice pickup.


Update voting icons and terminology

Currently the -1 icon is a block, and the hand is a -1, and it's all a bit confusing depending on how you enter the symbols. Perhaps we oculd use some others.
Maybe ✅ for a positive vote, 🛑 for a block, and ⌛ for an abstention.
This should probably be configurable somewhere in the app settings.

I think some of this is down to what the votes actually mean, not just the symbols. For 'abstain' to score -1 rather than 0 is rather unexpected, and not what abstention usually means. A sensible set of defaults probably needs to offer the four options of +1, 0, -1, and complete block, and individual projects could disable any they don't want.

Yes, that's a good point, that naming is confusing as well.
It's more like "upvote", "downvote" and "block", perhaps. I've never been convinced that formal abstention is necessary, but perhaps it could be a fourth.

yes: 🔼
no: 🔽
block: 🛑 or ⛔
abstain: difficult, but perhaps 😑

@philipjohn @openpolitics/owners I imagine you will have opinions on this

I've been giving this a lot of thought and have some ideas.
To start with I think we should make 👍 == +1 and 👎 == -1 as that seems like the most simple and straightforward way to vote.
For the "veto" I think we should do several things;

Use :stop_sign:
Only contributors with X amount of accepted proposals can veto
The veto is worth -10% of the number of contributors with voting rights (currently 4, based on 42 contributors)
The veto should never be worth more than -10% to ensure its effective with a small bunch of contributors

What do folks think?

Wikipedia has a list of consensus-building models, which look close to what we're doing here. I'm guessing that one of them is the same or very close, and we should try to match the proper terminology for what we're doing.

Interesting idea @philipjohn to limit veto to higher-level contributors, and to auto-weight by contributor pool. That would auto-scale to the long tail. But if the limit was -10%, below 10 contributors, a veto will be the same as a downvote, and basically useless.

Perhaps grant veto rights to the top x% of contributors, set per-project. Then you could adapt it to the shape of the participation tailoff and would be more effective at countering entryism.

Oh, and my only concern with reusing the thumb symbols is that it would mean that old proposals in the existing manifesto wouldn't get counted the same way, and there wouldn't be a solid record. I'm more inclined to deprecate them, but still count them the same. Reckons?

Reading wikipedia, I think we're doing exactly this one: https://en.wikipedia.org/wiki/Consensus_decision-making#Blocking

Nice find on the Wikipedia list!

below 10 contributors, a veto will be the same as a downvote, and basically useless.

It'd be -10 because of this rule, no?:

The veto should never be worth more than -10% to ensure its effective with a small bunch of contributors

my only concern with reusing the thumb symbols is that it would mean that old proposals in the existing manifesto wouldn't get counted the same way

We already have this though, right? There are proposals that had a thumbs down but went in because at that point it wasn't a veto?

I'll have to go back and look. At the moment I think it's still backwards compatible. In the early days we still tried to avoid downvotes; it was never a straight vote up or down with equal weight.

@philipjohn on the minimum veto weight, I think I've misread something... can't quite parse what you mean, but I think we're on the same wavelength.

The veto should never be worth more than -10% to ensure its effective with a small bunch of contributors

So if you had 5 contrubutors, and one vetoed, how much would that veto be worth?

The veto is worth -10% of the number of contributors with voting rights (currently 4, based on 42 contributors)

So it would be worth -0.5, and can never be worth more than that. How can it be, in fact, anything other than -0.5?

Ohhh I see what's happened - it's my mistake. Where I said

The veto should never be worth more than -10% to ensure its effective

It should have said

The veto should never be worth more than -10 to ensure its effective

So, to use your example, with 5 contributors the veto would be -0.5, which is more than -10. Therefore, the veto would be readjusted to be -10.
A better way of saying it might have been;

The veto has a maximum of -10

Or even;

With fewer than 10 contributors, the veto is fixed at -10

Here's some more examples;
Contributors = Veto value
5 = -10
10 = -10
50 = -10
150 = -15
500 = -50

Also, the veto doesn't become less than -10 until we got 100 voters

OK that makes a lot more sense :)

I'm working on a branch to renaming all this stuff.  I'm standardising on

Agree
Disagree
Block

Abstain will be removed as a term, for now, as we don't support actual abstention. We can reintroduce it later if we want to.

This will then make it easier to update to new icons, deprecating the old confusing ones.
Incidentally, abstention was cost-free for about a year - the upvote offset was added in https://github.com/openpolitics/votebot/commit/6b6f7c887b218e224ca5201047c39488fd6eb7d8

Actually. In the code, I'm going to use:

Yes
No
Block

because disagree would be moving from one meaning to another, and that's confusing as fuck. Yes and No are perfectly adequate.

I could standardise on

✅ Yes
❎ No
🚫 Block

Those would work across both the votebot app (using fontawesome) and GitHub (using emoji).


New font + better performance

seems like Inter, -apple-system, Microsoft YaHei, sans-serief will results better for hero heading?


test: basic shell of Feature file for Check Your Answers

Ticket Number
UCD-1205 / UCD-766
Description of change
Feature file showing navigation from check-your-answers -> Ts+Cs page
need for this identified while trying to sgn off 766 and realising we had no proof of this
note - i don't believe we have any validation in place yet, so the actual implementation of 'given enough data/not-enough data' is a little moot at this point..
Checklist

[ ] Requires infrastructure changes
[ ] I have updated the documentation accordingly
[ ] I have merged commits to avoid fixing things in this PR
[ ] New features have tests
[ ] Breaking change (team conversation required)

Important
Please do not merge from master (please only rebase). This keeps the history linear and easier to debug.

We're experimenting with 'picking up styling comments like this' as we do the step-defs.. taking forever to get feature files signed off otherwise..
as such; comments noted in UCD-1206 + going to merge this


invalide URL by restore of encoded characters

That is one valide URL - Click -> https://dev.hubzilla.org/page/expertmode/ru/seo-оптимированное-структурирование-содержания-многоязычных-ресурсов
But when I store this webpage item repeatedly, then is the webpage-URL invalide. I assume, that Hubzilla convert erroneously stored escaped character repeatedly. You can reproduce this behavior, when you for example in Hubzilla

create one webpage with this  Page link name-value

ru/SEO-оптимированное-структурирование-содержания-многоязычных-ресурсов

store the item from step 1
edit the item from step 1
store the item from step 1
click on the stored URL from step 4

Hubzilla-Version 2.5.2

devs: The fix to this can be cherry-picked from red 1b75e4d

@zotlabs can you please post a link to 1b75e4d -fix?


Search crashes when filtering

Issue appears in dev
Describe the bug

Open project dropdown (or e-type, m-type, type)
Select (tick) one vlue
Save
The search returns a blank screen

What part of Nexus is affected: Fusion
Expected behavior
Filtering should work without crashing the app.

https://github.com/BlueBrain/nexus-web/pull/1003


Instructions to deploy on OpenShift

Pull Request Checklist

[ ] Description: Briefly describe the changes in this pull request.
[ ] Changelog: Ensure a changelog entry following the format of Keep a Changelog is added at the bottom of the PR description.
[ ] Documentation: Have you updated relevant documentation?
[ ] Dependencies: Are there any new dependencies? Have you updated the dependency versions in the documentation?

Description
Added step by step instructions in readme to deploy open-webui on OpenShift.
Instructions can be useful for any kubernetes flavor.
open-webui needs root access to be deployed. it gives instructions on providing that access.

Changelog Entry
Added

Instructions in Readme to deploy on Openshift

Fixed

[List any fixes or corrections]

Changed

[List any changes or updates]

Removed

[List any removed features or files]

https://github.com/open-webui/docs


Neo4j OGM ClassLoader strategy prevents spring-boot-devtools from working

Hello everyone,
I am starting a new project, and decided to give a try to neo4j with spring-boot. Everything works just fine, but im having a very hard time trying to use spring boot devtools to use his automatic restart capability. When I change anything in my code, the restart just works, but when I call any query on Session, the jvm throws a ClassCastException or ignores the query results.
the classCast is always between the same class of my domain, for example
br.com.ncisaude.gr.dominio.usuario.Usuario cannot be cast to br.com.ncisaude.gr.dominio.usuario.Usuario.
Debugging neo4j-ogm code i've found that There is a class Called MetaDataClassLoader that is used by the EntityFactory. That class can instantiate my domain class, but then when GraphEntityMapper tries to build the results, it just ignores the result because of a classloader mismatch.
I'm posting the code to help clarify the question:
public <T> List<T> map(Class<T> type, GraphModel graphModel, Set<Long> nodeIds, Set<Long> edgeIds) {

        mapEntities(type, graphModel, nodeIds, edgeIds);
        List<T> results = new ArrayList<>();

        for (Long id : nodeIds) {
            Object o = mappingContext.getNodeEntity(id);
            if (o != null && type.isAssignableFrom(o.getClass())) {
                results.add(type.cast(o));
            }
        }

        // only look for REs if no node entities were found
        if (results.isEmpty()) {
            for (Long id : edgeIds) {
                Object o = mappingContext.getRelationshipEntity(id);
                if (o != null && type.isAssignableFrom(o.getClass())) {
                    results.add(type.cast(o));
                }
            }
        }

        return results;
    }

When the codeObject o = mappingContext.getNodeEntity(id);runs, my domain object is returned as expected, but in the following line o != null && type.isAssignableFrom(o.getClass()) it returns false and then the result is ignored. It hapens because the "type" object is from the current spring-boot-devtools classloader, but the o.getClass() is from a class from another classLoader.
This behavior seems to happen because of the implementation of org.neo4j.ogm.classloader.ClassLoaderResolver and org.neo4j.ogm.classloader.DefaultStrategy, but I don't now exactly how.
I hope that this kind of problem is part of neo4j-ogm scope, because i think that spring-boot-devtools can't resolve it, since it looks like a neoj-ogm classLoader related issue.
If this is your scope, tell me and I will create a public github project where you can reproduce the problem with ease.
Thanks for your time.

Hi, because of the way the Spring Boot dev tools restart functionality works, it can sometimes cause classloading issues. Please see http://docs.spring.io/spring-boot/docs/current/reference/html/using-boot-devtools.html#using-boot-devtools-customizing-classload for more details and workarounds.

Thanks for your answer Vince,
I tried using spring-devtools.properties, but it did not help.
After some more debugging i've found that after a spring-devtools restart, the neo4j ogm DefaultStrategy returns the correct classloader (Which is the org.springframework.boot.devtools.restart.classloader.RestartClassLoader). So my reloaded application and the neo4j-ogm are on the same classloader.
After some more debugging, i'm not sure it is a classloader related issue, it seems more a class cache issue.
To prove my point, I've changed the class
public abstract class MetaDataClassLoader {

    private static final ClassLoader classLoader = ClassLoaderResolver.resolve();

    public static Class loadClass(final String name) throws ClassNotFoundException {
        return Class.forName(name, false, classLoader);
    }
}

to
public abstract class MetaDataClassLoader {

    public static Class loadClass(final String name) throws ClassNotFoundException {
        return Class.forName(name, false, getClassLoader());
    }

	private static ClassLoader getClassLoader() {
		return ClassLoaderResolver.resolve();
	}
}

And after this change the GraphEntityMapper.map() are sucessfully returning my entity, but then I've started to get errors from GraphEntityMapper.writeProperty, like
java.lang.IllegalArgumentException: Can not set br.com.ncisaude.gr.dominio.instituicao.UF field br.com.ncisaude.gr.dominio.instituicao.Instituicao.uf to br.com.ncisaude.gr.dominio.instituicao.UF
And it looks like that this occurs because ClassInfo holds some references to Class on iterableGettersForType, iterableSettersForType and iterableFieldsForType, and these classes no longer exists because of the classloader change.
Dont get me wrong, loking at all this it doenst seems to me that neo4j-ogm is the problem here.
Maybe if spring-boot reloadded the neo4j-ogm context(SessionFactory and alike)  on restarts this would not occur.
I dont know how to proceed from now on. Im thinking of opening a issue on spring-data-neo4j.
Any thoughts about it?
Thank you

Hi Rafa, Spring Data Neo4j uses the neo4j-ogm under the hood, so no need to raise a new ticket. Do you have a project we can look at that reproduces this problem, so we can investigate further?

@mangrish did you sort it out back then when you switched the class-scanning-library?

I can confirm the issue on OGM 2.1.x branch - I was able to reproduce the issue from first post and fix it by using ClassLoaderResolver.resolve() or Thread.currentThread().getContextClassLoader() - reload was working fine after that.
I could not reproduce the follow up issue (stuff gets indeed cached in static Map fields in EntityAccessManager, but the keys are instances of ClassInfo, which get reloaded.
The reload works fine with Spring Boot 2.0.0.M1 and Spring Data Kay M3 so I am inclined to close this as fixed, with no solution for 2.1.x. @rafaferry would you be able to try if the reload works on your project with 2.0.0.M1?

@frant-hartm, I can confirm that OGM and spring-devtools are not working on Spring Boot 2.0.0.M1 and Spring Data Kay M3, but I've not investigated that its the same issue.
I've created a repo so you can reproduce the problem. https://github.com/rafaferry/neo4j-ogm-spring-devtools-test
The steps for reproducing are:
1 - Start a neo4j database on default ports for bolt / http (i've done this simply by starting neo4j docker image)
2 - run gradlew eclipse
3 - import eclipse project on Spring Tools Suite
4 - start the project within boot-dashboard - The project will automatically populate the database
5 - access localhost:8080/usuarios - you should receive a json response listing users from the database.
6 - Change the class UsuarioController (just add an empty line and save) - devtools will restart the application.
7 - access localhost:8080/usuarios -> java.lang.IllegalArgumentException on FieldInfo.write
I've tried another approach with OGM 2.1.x, and disabled devtools and started using spring-loaded. Worked as expected, so as a workarround I'll just stick to spring-loaded.

Found the issue, it is indeed cached static reference - https://github.com/neo4j/neo4j-ogm/blob/master/core/src/main/java/org/neo4j/ogm/utils/ClassUtils.java#L28
I will try to refactor this so we can get rid of the static reference.

Hello @frant-hartm
Are you going to merge your commit to master?
Thanks in advance.

Hello @frant-hartm
Im still facing the dev-tool reload issue.
I'm using neo4j-ogm-3.2.3 with spring-boot-2.2.2.RELEASE.jar. Every thing works fine without the dev-tools.
When injecting spring-boot-devtools
<dependency>
     <groupId>org.springframework.boot</groupId>
     <artifactId>spring-boot-devtools</
</dependency>

java.lang.IllegalArgumentException: Can not set java.lang.String field package.model.entity.Entity.id to package.model.entity.Entity.
	at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167) ~[na:1.8.0_222]
	at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171) ~[na:1.8.0_222]
	at sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:58) ~[na:1.8.0_222]
	at sun.reflect.UnsafeObjectFieldAccessorImpl.get(UnsafeObjectFieldAccessorImpl.java:36) ~[na:1.8.0_222]
	at java.lang.reflect.Field.get(Field.java:393) ~[na:1.8.0_222]
	at org.neo4j.ogm.metadata.FieldInfo.lambda$read$1(FieldInfo.java:357) ~[neo4j-ogm-core-3.2.3.jar:3.2.3]
	at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_222]
	at org.neo4j.ogm.metadata.FieldInfo.read(FieldInfo.java:354) ~[neo4j-ogm-core-3.2.3.jar:3.2.3]
	at org.neo4j.ogm.metadata.FieldInfo.read(FieldInfo.java:409) ~[neo4j-ogm-core-3.2.3.jar:3.2.3]
	at org.neo4j.ogm.metadata.ClassInfo.readPrimaryIndexValueOf(ClassInfo.java:1128) ~[neo4j-ogm-core-3.2.3.jar:3.2.3]
	at org.neo4j.ogm.context.MappingContext.generateIdIfNecessary(MappingContext.java:545) ~[neo4j-ogm-core-3.2.3.jar:3.2.3]
	at org.neo4j.ogm.context.MappingContext.nativeId(MappingContext.java:515) ~[neo4j-ogm-core-3.2.3.jar:3.2.3]
	at org.neo4j.ogm.cypher.compiler.CypherContext.getIdentity(CypherContext.java:66) ~[neo4j-ogm-core-3.2.3.jar:3.2.3]
	at org.neo4j.ogm.cypher.compiler.CypherContext.visitedNode(CypherContext.java:90) ~[neo4j-ogm-core-3.2.3.jar:3.2.3]
	at org.neo4j.ogm.context.EntityGraphMapper.mapEntity(EntityGraphMapper.java:261) ~[neo4j-ogm-core-3.2.3.jar:3.2.3]
	at org.neo4j.ogm.context.EntityGraphMapper.map(EntityGraphMapper.java:164) ~[neo4j-ogm-core-3.2.3.jar:3.2.3]
	at org.neo4j.ogm.session.delegates.SaveDelegate.lambda$save$1(SaveDelegate.java:89) ~[neo4j-ogm-core-3.2.3.jar:3.2.3]
	at java.util.Collections$SingletonList.forEach(Collections.java:4824) ~[na:1.8.0_222]
	at org.neo4j.ogm.session.delegates.SaveDelegate.save(SaveDelegate.java:89) ~[neo4j-ogm-core-3.2.3.jar:3.2.3]
	at org.neo4j.ogm.session.delegates.SaveDelegate.save(SaveDelegate.java:51) ~[neo4j-ogm-core-3.2.3.jar:3.2.3]
	at org.neo4j.ogm.session.Neo4jSession.save(Neo4jSession.java:474) ~[neo4j-ogm-core-3.2.3.jar:3.2.3]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_222]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_222]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_222]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222]
	at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282) ~[spring-core-5.2.2.RELEASE.jar:5.2.2.RELEASE]

Any workarounds to fix the devtool issue?
Thanks in advance!

This is related to #729. I can reproduce both issues.


More examples

For a documentation page that it is explaining a new feature being introduced, this page is lacking way more examples. The page currently has one example per category and this is not nearly enough to understand the capabilities of these new functionalities.
I've setup a new environment with a TSI preview to play around with the instances, hierarchies, types and I can't figure out the extent of what operations can I do with the new types.
I want to be able to introduce variables and play around with custom values and custom aggregations to assess the power of the types functionality but I haven't found the proper documentation to guide me.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 721c4d80-9266-8804-5b67-d13863912067
Version Independent ID: 66ca3e80-3ec4-9448-432a-b8be9488563a
Content: Time Series Model - Azure Time Series Insights
Content Source: articles/time-series-insights/time-series-insights-update-tsm.md
Service: time-series-insights
GitHub Login: @deepakpalled
Microsoft Alias: dpalled

@olivoarroba Thanks for the feedback! I have assigned the issue to the content author to evaluate and update as appropriate.

Hi @olivoarroba this is great feedback, while we work to add more examples to the docs we can use this thread to work through your scenarios. Please let us know your scenarios, goals, and what operations you're looking to do and we can work through it here.

Hi @lyrana: So I want to explore the aggregations, if I want to do some custom aggregations what options do I have to do. For example, I wanted to test a simple custom aggregation where I substract the first value of the period from the last value, something like:  last($value)-first($value). The TSI GUI gives me an error of 'Invalid aggregate expression(s)'. But then, I have no guidance of what would comprise a valid aggregate expression.
Same situation for the custom value. I can select from the dropdown in the GUI. But if I want to do a custom value, I have no guidance as to what options can I select.

@olivoarroba Thanks for the feedback on the documentation. We're going through a refresh right now and will update with more examples, as you've suggested!
To perform an aggregation like last($value) - first($value), you can use an Aggregate Variable. Numeric Variable kinds don't support this kind of layered aggregation method.
If you're creating one using our UX, it might look something like this:

I'll also take the feedback regarding errors back to our UX team.
Hope that helps!

@olivoarroba I hope the above helps, and see this link for the Time Series Expression (TSX) syntax with examples

@olivoarroba Do let us know if you have further questions.

@olivoarroba As mentioned, we will update the doc with more examples in the next release.
We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.

Hi @AshokPeddakotla-MSFT sorry for not answering earlier. I've continued to play around with TSI-preview APIs. And there is one thing that I have not found how to do which you might be able to help me with.
In the current TSI, if I have N number of devices reporting metrics. I can run a query to give me an aggregation of metric X over the entire N number of devices. For example, this is a typical query I use a lot:
{
    "searchSpan": {
        "from": {
            "dateTime": "2020-03-01T00:00:00.000Z"
        },
        "to": {
            "dateTime": "2020-03-10T00:00:00.000Z"
        }
    },
    "aggregates": [
        {
            "dimension": {
                "dateHistogram": {
                    "input": {
                        "builtInProperty": "$ts"
                    },
                    "breaks": {
                        "size": "1h"
                    }
                }
            },
            "measures": [
                {
                    "avg": {
                        "input": {
                            "property": "rssi",
                            "type": "Double"
                        }
                    }
                },
                {
                    "min": {
                        "input": {
                            "property": "rssi",
                            "type": "Double"
                        }
                    }
                }
            ]
        }
    ],
    "predicate": {
        "in": {
            "left": {
                "property": "iothub-connection-device-id",
                "type": "String"
            },
            "right": [
                "echotest0005",
                "echotest0001"
            ]
        }
    },
    "top": {
        "sort": [
            {
                "input": {
                    "builtInProperty": "$ts"
                },
                "order": "Asc"
            }
        ],
        "count": 1000
    }
}

By using the predicate with an 'in' clause, I can match more than one device's metrics (in this example two devices). And I can get two aggregated metrics over both of my devices at the same time, I'm getting the avg rssi and a min rssi.
These type of queries are very useful for dashboard like applications, where you want to show one chart showing the aggregated metrics for a group of devices, and then the user can drill down and look at individual devices if he/she needs to.
I don't seem to find how to perform this query on the new TSI-preview APIs. The closest I seem to get is by doing something like this:
{
    "aggregateSeries": {
        "searchSpan": {
            "from": "2020-03-01T19:34:27.657Z",
            "to": "2020-03-02T19:34:27.657Z"
        },
        "timeSeriesId": [
            "82001520101200500009"
        ],
        "interval": "PT1H",
        "inlineVariables": {
            "avg": {
                "kind": "numeric",
                "value": {
                    "tsx": "$event.[sysUpTime].Double"
                },
                "filter": null,
                "aggregation": {
                    "tsx": "avg($value)"
                }
            },
            "min": {
                "kind": "numeric",
                "value": {
                    "tsx": "$event.[sysUpTime].Double"
                },
                "filter": null,
                "aggregation": {
                    "tsx": "min($value)"
                }
            }
        },
        "projectedVariables": [
            "avg",
            "min"
        ]
    }
}

But as soon as I add more than one entry into the TimeSeriesId array, I get an error. It seems that the API only allows for a single Series to be queried at a time. So the question is:

Is there a way to have TSI-preview aggregate over multiple timeseries as the current TSI-GA does? This is a crucial part of my TSI usage since I need to have this general to specific view on my information dashboards.

Hey @olivoarroba, the PAYG version of TSI currently does not support querying over multiple TSIDs.
We've been asked for this feature often from our PAYG customers and its something we're considering for our future roadmap. If you'd like to have a further conversation about this and your use case, please feel free to send me an email -<EMAIL_ADDRESS>👍


Add a link from Girder Items

For any large image, add a button on the Girder item page that will open that image in the HistomicsTK interface.

@dgutman Do you think this should be an entry in the Actions menu or a button on, for instance, the Image Viewer header bar?


Use 'const' instead of 'constexpr'.

Eliminate 'constexpr' since it is a C++ feature.

LGTM

Looks good to me

Landed (d60bd93da337644d968c7466f9ab79e748e83b52)


Remove SketchMode commands

These aren't being used by the frontend, because of our current "client-side sketching" system Kurt made. So, instead, let's remove those endpoints. When we eventually move sketching back into the engine, we can add a new endpoint that actually works right.

covered by https://github.com/KittyCAD/modeling-api/pull/309


Make ImmDB Iterator/IteratorResult functors

Hmmmmmmmmmmmmmmmmmmmmmmmm. Now that I am actually using this..... What about the iterator IDs...? :thinking: Even ignroing these Functor instances, I want to be able to write something like
-- | Stream blocks after the given point
--
-- See also 'streamBlobsAfter'.
streamBlocksAfter :: forall m blk. (MonadCatch m, StandardHash blk, Typeable blk)
                  => ImmDB m blk
                  -> Point blk -- ^ Exclusive lower bound
                  -> m (Iterator (HeaderHash blk) m blk)
streamBlocksAfter db low = wrap <$> streamBlobsAfter db low
  where
    wrap :: Iterator (HeaderHash blk) m Strict.ByteString
         -> Iterator (HeaderHash blk) m blk
    wrap itr = Iterator {
          iteratorNext  = ...
        , iteratorClose = iteratorClose itr
        , iteratorID    = iteratorID    itr
        }

but again, that would result in different iterators with the same ID and hence reported as equal...
Not entirely sure what to do with that. Maybe the Eq instance should go?

The main reason for having the Ord instance (which we would lose if we remove the Eq instance) and in fact iteratorID is that we need to store iterators in a map (see KnownIters in the q-s-m tests).
I would be happy to drop these ugly iteratorIDs (which would make the path clear for this PR) if we have a better solution that allows us to store them in a map.
Another idea: if we somehow only expose the iteratorIDs to the testsuite and move the Eq and Ord instances to the testsuite as well, we don't need to care that an fmapped iterator shares its iteratorID with another iterator. We would only have to be careful in the testsuite.

Explored the option of removing the Eq/Ord instances. This has a number of consequences though:

The knownIters in the model now need to be map from IteratorID to mock iterators, rather from iterators to iterators.
We need some generalizations of the RefEnv, and some infrastructure for mapping over references (from Reference (Iterator ..) r to Reference IteratorID r
This then leaves us unable to close all iterators (since we'd only have their IDs), so we'd have to extend the model or fix #328 first.

None of this is impossible but also would get us no closer to a working chain DB. Instead changed iterator IDs to
-- | ID of an iterator that is not derived from another iterator
newtype BaseIteratorID = MkBaseIteratorID Int
  deriving (Show, Eq, Ord, Generic, Enum)

-- | A unique identifier for an iterator.
data IteratorID = BaseIteratorID BaseIteratorID | DerivedIteratorID IteratorID
  deriving (Show, Eq, Ord, Generic)

to avoid the problem mentioned above.

Would it be feasible to use a different iterator type for mock? Then we can put the ID in that one, and leave it out in the real one.
I have defined my own Iterator type for the Byron proxy DB abstraction because I didn't want the iterator ID (I'd have to come up with my own, which just seemed wrong).

Would it be feasible to use a different iterator type for mock? Then we can put the ID in that one, and leave it out in the real one.

No, the q-s-m tests map the real iterators to mock iterators (same type, actually); it's the from iterator (the real one) that we really need to be able to have an ID for.

I have defined my own Iterator type for the Byron proxy DB abstraction because I didn't want the iterator ID (I'd have to come up with my own, which just seemed wrong).

I don't get this. Why would you have to come up with your own IDs?

I don't get this. Why would you have to come up with your own IDs?

Because I want none :: Applicative m => Iterator m t, one :: Applicative t => t -> Iterator m t etc.

No, the q-s-m tests map the real iterators to mock iterators (same type, actually); it's the from iterator (the real one) that we really need to be able to have an ID for.

But the ID is only used for testing/simulation or similar yes?

But the ID is only used for testing/simulation or similar yes?

Currently, yes, that's the case. Not sure that will remain to be the case; possibly, though not totally sure (compare readerId in ReaderState, I suppose).

(Merged since this discussion is somewhat orthogonal to this PR.)

We could remove the iteratorID part but it would require various changes to the tests (not impossible, but a little time consuming), and I'm not entirely sure that we should. It's occasionally useful of iterators have an Eq/Ord instance (another data point: Eq instance for Handle from System.IO).
I agree that it's not optimal though.. :/

I guess we could easily move your Eq-less and iteratorID-less version to the (chain) DB itself though; that's easy enough to do.