Trying to use SSL

Trying to use ssl cert mechanism to talk to the device and its not working? did anyone get it working?
telegraf.tmpl settings for input-oc.
servers = ["<IP_ADDRESS>:50051"]
ssl_cert = "/source/jti.pem"
I was using the below mechanism to create one.
http://ipengineer.net/2018/05/configuring-ssl-grpc-junos/
Looks like it tries and fails. never attempts to connect again.
root@Jumphost2:/home/mohan/open-nti# tcpdump -i eth0 port 50051
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
13:07:49.788309 IP <IP_ADDRESS>.54626 > <IP_ADDRESS>.50051: Flags [S], seq<PHONE_NUMBER>, win 29200, options [mss 1460,sackOK,TS val 118401606 ecr 0,nop,wscale 7], length 0
13:07:49.794368 IP <IP_ADDRESS>.50051 > <IP_ADDRESS>.54626: Flags [S.], seq<PHONE_NUMBER>, ack<PHONE_NUMBER>, win 65535, options [mss 1460,nop,wscale 1,nop,nop,TS val<PHONE_NUMBER> ecr 118401606,sackOK,eol], length 0
13:07:49.794451 IP <IP_ADDRESS>.54626 > <IP_ADDRESS>.50051: Flags [.], ack 1, win 229, options [nop,nop,TS val 118401607 ecr<PHONE_NUMBER>], length 0
13:07:49.794703 IP <IP_ADDRESS>.54626 > <IP_ADDRESS>.50051: Flags [P.], seq 1:152, ack 1, win 229, options [nop,nop,TS val 118401607 ecr<PHONE_NUMBER>], length 151
13:07:49.817000 IP <IP_ADDRESS>.50051 > <IP_ADDRESS>.54626: Flags [P.], seq 1:1327, ack 152, win 33304, options [nop,nop,TS val<PHONE_NUMBER> ecr 118401607], length 1326
13:07:49.817078 IP <IP_ADDRESS>.54626 > <IP_ADDRESS>.50051: Flags [.], ack 1327, win 251, options [nop,nop,TS val 118401613 ecr<PHONE_NUMBER>], length 0
13:07:49.817496 IP <IP_ADDRESS>.54626 > <IP_ADDRESS>.50051: Flags [P.], seq 152:159, ack 1327, win 251, options [nop,nop,TS val 118401613 ecr<PHONE_NUMBER>], length 7
13:07:49.817596 IP <IP_ADDRESS>.54626 > <IP_ADDRESS>.50051: Flags [R.], seq 159, ack 1327, win 251, options [nop,nop,TS val 118401613 ecr<PHONE_NUMBER>], length 0
13:07:49.818633 IP <IP_ADDRESS>.50051 > <IP_ADDRESS>.54626: Flags [F.], seq 1327, ack 159, win 33300, options [nop,nop,TS val<PHONE_NUMBER> ecr 118401613], length 0
13:07:49.818673 IP <IP_ADDRESS>.54626 > <IP_ADDRESS>.50051: Flags [R], seq<PHONE_NUMBER>, win 0, length 0

Hi
is /source/jti.pem file inside the container ?
Regards

yes, made sure its inside the container.
On Thu, Aug 9, 2018 at 6:37 AM, psagrera<EMAIL_ADDRESS>wrote:

Hi
is /source/jti.pem file inside the container ?
Regards
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Juniper/open-nti/issues/230#issuecomment-411714621,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AcT0O8bNB3EdYB0Bw_9-0H1wQdPNgh6uks5uPBDUgaJpZM4VpbNk
.

Hi,
Try this:
openssl genrsa -out ca.key 2048

openssl req -new -x509 -key ca.key -out ca.crt

openssl genrsa -out mx1_re.key 2048

openssl req -new -key mx1_re.key -out mx1_re.csr

openssl x509 -req -in mx1_re.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out mx1_re.crt

cat mx1_re.key mx1_re.crt > mx1_re.pem

scp mx1_re.pem "user@router:/var/tmp"

set security certificates local router_mx load-key-file /var/tmp/mx1_re.pem

set system services extension-service request-response grpc ssl port 50051
set system services extension-service request-response grpc ssl local-certificate router_mx
set system services extension-service request-response grpc skip-authentication
set system services extension-service notification allow-clients address <IP_ADDRESS>/0

logs from the router:
Aug 14 00:34:11 readGrpcConfig Restarting Grpc server as there is a change in parameters Old/new: address: ::/::, port: 50051/50051, session: 5/5, SSL enabled: 1/1, skip-authentication: 0/1, buffer size: 1048576/1048576, tcp maximum segment size: 0/0, retry_count: 15/15, retry_interval: 1/1, RequestResponse grpc knob status: 1/1

logs from telegraf (running a debugging version) :
2018-08-14T08:18:25Z I! Tags enabled: host=ubuntu
2018-08-14T08:18:25Z I! Agent Config: Interval:10s, Quiet:false, Hostname:"ubuntu", Flush Interval:5s
2018-08-14T08:18:30Z I! Transport credentials &{%!s(*tls.Config=&{<nil> <nil> [] map[] <nil> <nil> <nil> <nil> 0xc420011950 [h2]  0 <nil> 	false [] false false [130 97 114 66 255 149 209 160 114 53 133 177 76 61 142 84 106 70 91 51 78 95 5 236 219 92 102 193 219 63 136 199] <nil> 	0 0 [] false 0 <nil> {{0 0} 0} {{0 0} 0 0 0 0} [{[15 236 112 160 210 126 118 101 39 196 242 85 109 29 197 239] [154 107 17 78 35 216 211 56 	19 120 89 182 107 126 131 198] [46 176 98 10 125 222 125 213 174 64 2 164 158 252 229 44]}]})}
2018-08-14T08:18:30Z I! Transport credentials set
2018-08-14T08:18:30Z D! Opened a new gRPC session to mx1_re on port 50051
2018-08-14T08:18:35Z D! Output [file] buffer fullness: 0 / 10000 metrics.
2018-08-14T08:18:36Z D! Received from mx1_re: system_id:"mx1_re" path:"sensor_1002:/junos/system/linecard/packet/usage/:/junos/system/linecard	/packet/usage/:PFE" timestamp:1534234716137 kv:<key:"__timestamp__" uint_value:1534234716140 > kv:<key:"__prefix__" str_value:"/components/	component[name='FPC0:CPU0']/" > kv:<key:"properties/property[name='lts-input-packets']/state/value" uint_value:555819 > kv:<key:"properties/	property[name='lts-output-packets']/state/value" uint_value:947824 >

telegraf file config:
[........]
[[inputs.jti_openconfig_telemetry]]

servers = ["mx1_re:50051"]
#username = "root"
#password = "Embe1mpls"
#client_id = "telegraf"

sensors = [
"junos-cpu /junos/system/cpu/memory/",
"junos-linecard-packet /junos/system/linecard/packet/usage/",
"junos-linecard-fabric /junos/system/linecard/fabric/",
"oc-bgp /bgp",
"oc-interfaces /interfaces/interface/[name='fxp0'] /interfaces/interface/[name='ge-0/0/0'] /interfaces/interface/[name='ge-0/0/1'] /	interfaces/interface/[name='gr-0/0/0']",
"oc-components /components/",
"junos-kernel-ifstate /junos/kernel-ifstate/",
"oc-bgp-neighbors /bgp/neighbors/neighbor/"
]

ssl_cert  = "mx1_re.pem"
[.......]

Ahh, let me try this out. How did you get the router log? I have this one
configured.
set system services management-grpc-api-service traceoptions file grpc.log
set system services management-grpc-api-service traceoptions flag all
Cheers,
-Mohan
On Tue, Aug 14, 2018 at 5:11 AM, psagrera<EMAIL_ADDRESS>wrote:

Hi,
Try this:
openssl genrsa -out ca.key 2048
openssl req -new -x509 -key ca.key -out ca.crt
openssl genrsa -out mx1_re.key 2048
openssl req -new -key mx1_re.key -out mx1_re.csr
openssl x509 -req -in mx1_re.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out mx1_re.crt
cat mx1_re.key mx1_re.crt > mx1_re.pem
scp mx1_re.pem "user@router:/var/tmp"
set security certificates local router_mx load-key-file /var/tmp/mx1_re.pem
set system services extension-service request-response grpc ssl port 50051
set system services extension-service request-response grpc ssl local-certificate router_mx
set system services extension-service request-response grpc skip-authentication
set system services extension-service notification allow-clients address <IP_ADDRESS>/0
logs from the router:
Aug 14 00:34:11 readGrpcConfig Restarting Grpc server as there is a change in parameters Old/new: address: ::/::, port: 50051/50051, session: 5/5, SSL enabled: 1/1, skip-authentication: 0/1, buffer size: 1048576/1048576, tcp maximum segment size: 0/0, retry_count: 15/15, retry_interval: 1/1, RequestResponse grpc knob status: 1/1
logs from telegraf (running a debugging version) :
2018-08-14T08:18:25Z I! Tags enabled: host=ubuntu
2018-08-14T08:18:25Z I! Agent Config: Interval:10s, Quiet:false, Hostname:"ubuntu", Flush Interval:5s
2018-08-14T08:18:30Z I! Transport credentials &{%!s(*tls.Config=&{  [] map[]     0xc420011950 [h2]  0  	false [] false false [130 97 114 66 255 149 209 160 114 53 133 177 76 61 142 84 106 70 91 51 78 95 5 236 219 92 102 193 219 63 136 199]  	0 0 [] false 0  {{0 0} 0} {{0 0} 0 0 0 0} [{[15 236 112 160 210 126 118 101 39 196 242 85 109 29 197 239] [154 107 17 78 35 216 211 56 	19 120 89 182 107 126 131 198] [46 176 98 10 125 222 125 213 174 64 2 164 158 252 229 44]}]})}
2018-08-14T08:18:30Z I! Transport credentials set
2018-08-14T08:18:30Z D! Opened a new gRPC session to mx1_re on port 50051
2018-08-14T08:18:35Z D! Output [file] buffer fullness: 0 / 10000 metrics.
2018-08-14T08:18:36Z D! Received from mx1_re: system_id:"mx1_re" path:"sensor_1002:/junos/system/linecard/packet/usage/:/junos/system/linecard	/packet/usage/:PFE" timestamp:1534234716137 kv:<key:"timestamp" uint_value:1534234716140 > kv:<key:"prefix" str_value:"/components/	component[name='FPC0:CPU0']/" > kv:<key:"properties/property[name='lts-input-packets']/state/value" uint_value:555819 > kv:<key:"properties/	property[name='lts-output-packets']/state/value" uint_value:947824 >
telegraf file config:
[........]
[[inputs.jti_openconfig_telemetry]]
servers = ["mx1_re:50051"]
#username = "root"
#password = "Embe1mpls"
#client_id = "telegraf"
sensors = [
"junos-cpu /junos/system/cpu/memory/",
"junos-linecard-packet /junos/system/linecard/packet/usage/",
"junos-linecard-fabric /junos/system/linecard/fabric/",
"oc-bgp /bgp",
"oc-interfaces /interfaces/interface/[name='fxp0'] /interfaces/interface/[name='ge-0/0/0'] /interfaces/interface/[name='ge-0/0/1'] /	interfaces/interface/[name='gr-0/0/0']",
"oc-components /components/",
"junos-kernel-ifstate /junos/kernel-ifstate/",
"oc-bgp-neighbors /bgp/neighbors/neighbor/"
]
ssl_cert  = "mx1_re.pem"
[.......]
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Juniper/open-nti/issues/230#issuecomment-412806903,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AcT0O3FSAtXmCdu5oHyPQiHUOEDms7bcks5uQpQngaJpZM4VpbNk
.

Like that :
set system services extension-service traceoptions file extension-service.log
set system services extension-service traceoptions file size 5m
set system services extension-service traceoptions file files 2
set system services extension-service traceoptions flag all

I am getting this, followed your procedure. Ideas?
[edit system services extension-service]
lab@vmx17# Aug 14 14:52:02 rtmDeleteEvent: topic:
"/junos/events/kernel/route/delete/inet/<IP_ADDRESS>/32"
Aug 14 14:52:27 ssl_transport_security.c:1290: No match found for server
name: mx1_re.
Aug 14 14:52:27 rtmAddEvent: topic: "/junos/events/kernel/route/add/inet/
<IP_ADDRESS>/32"
Aug 14 14:52:27 ssl_transport_security.c:201: ssl_info_callback: error
occured.
Aug 14 14:52:27 ssl_transport_security.c:947: Handshake failed with fatal
error SSL_ERROR_SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3
alert bad certificate.
Aug 14 14:52:27 handshake.c:128: Security handshake failed:
{"created":"@1534258347.582136792","description":"Handshake
failed","file":"../../../../../../../../src/dist/grpc/src/core/lib/security/transport/handshake.c","file_line":264,"tsi_code":10,"tsi_error":"TSI_PROTOCOL_FAILURE"}
Aug 14 14:52:27 server_secure_chttp2.c:119: Secure transport failed with
error 1
lab@vmx17# run show version
Aug 14 14:53:39
Hostname: vmx17
Model: vmx
Junos: 17.4R1-S2.2
request-response {
grpc {
ssl {
port 50051;
local-certificate router_mx;
}
skip-authentication;
}
}
notification {
allow-clients {
address <IP_ADDRESS>/0;
}
}
On Tue, Aug 14, 2018 at 6:42 AM, psagrera<EMAIL_ADDRESS>wrote:

Like that :
set system services extension-service traceoptions file extension-service.log
set system services extension-service traceoptions file size 5m
set system services extension-service traceoptions file files 2
set system services extension-service traceoptions flag all
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Juniper/open-nti/issues/230#issuecomment-412830177,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AcT0O-7WK-9swa8ze22u4bxHKquZef0vks5uQqmEgaJpZM4VpbNk
.

Hi,
That's what I've done :
On the server
############

1) openssl genrsa -out ca.key 2048

2) openssl req -new -x509 -key ca.key -out ca.crt (all answers in blank except FQDN:mx1_re)

3) openssl genrsa -out mx1_re.key 2048

4) openssl req -new -key mx1_re.key -out mx1_re.csr (all answers in blank except FQDN:mx1_re)

5) openssl x509 -req -in mx1_re.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out mx1_re.crt

6) cat mx1_re.key mx1_re.crt > mx1_re.pem

On the router
#############

7) delete security
   delete system services
   commit

8) file delete /var/tmp/mx1_re.pem

9) scp mx1_re.pem "user@router:/var/tmp"

10) set security certificates local router_mx load-key-file /var/tmp/mx1_re.pem

11)
set system services extension-service request-response grpc ssl port 50051
set system services extension-service request-response grpc ssl local-certificate router_mx
set system services extension-service request-response grpc skip-authentication
set system services extension-service notification allow-clients address <IP_ADDRESS>/0

Telegraf file (the hostname of the router I'm using is mx1_re junos version: 18.1R2-S1):
##############

[[inputs.jti_openconfig_telemetry]]

servers = ["mx1_re:50051"]
#username = "lab"
#password = "lab123"
#client_id = "mx1_re"

......

ssl_cert  = "mx1_re.pem"

.......

hmm,  I missed the FQDN part earlier. Now, I followed your steps on a
18.1R1.9 device again, still getting an error message.
lab@mx1_re# run show version
Hostname: mx1_re
Model: vmx
Junos: 18.1R1.9
[[inputs.jti_openconfig_telemetry]]
servers = ["mx1_re:50051"]
ssl_cert = "mx1_re.pem"
[edit]
lab@mx1_re# Aug 17 05:29:20 ssl_transport_security.c:1290: No match found
for server name: mx1_re.
Aug 17 05:29:20 ssl_transport_security.c:201: ssl_info_callback: error
occured.
Aug 17 05:29:20 ssl_transport_security.c:947: Handshake failed with fatal
error SSL_ERROR_SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3
alert bad certificate.
Aug 17 05:29:20 handshake.c:128: Security handshake failed:
{"created":"@1534483760.559614319","description":"Handshake
failed","file":"../../../../../../../../src/dist/grpc/src/core/lib/security/transport/handshake.c","file_line":264,"tsi_code":10,"tsi_error":"TSI_PROTOCOL_FAILURE"}
Aug 17 05:29:20 server_secure_chttp2.c:119: Secure transport failed with
error 1
On Thu, Aug 16, 2018 at 3:49 AM psagrera<EMAIL_ADDRESS>wrote:

Hi,
That's what I've done :
On the server
############

openssl genrsa -out ca.key 2048

openssl req -new -x509 -key ca.key -out ca.crt (all answers in blank except FQDN:mx1_re)

openssl genrsa -out mx1_re.key 2048

openssl req -new -key mx1_re.key -out mx1_re.csr (all answers in blank except FQDN:mx1_re)

openssl x509 -req -in mx1_re.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out mx1_re.crt

cat mx1_re.key mx1_re.crt > mx1_re.pem

On the router
#############

delete security
delete system services
commit

file delete /var/tmp/mx1_re.pem

scp mx1_re.pem "user@router:/var/tmp"

set security certificates local router_mx load-key-file /var/tmp/mx1_re.pem

set system services extension-service request-response grpc ssl port 50051
set system services extension-service request-response grpc ssl local-certificate router_mx
set system services extension-service request-response grpc skip-authentication
set system services extension-service notification allow-clients address <IP_ADDRESS>/0
Telegraf file (the hostname of the router I'm using is mx1_re junos version: 18.1R2-S1):
##############
[[inputs.jti_openconfig_telemetry]]
servers = ["mx1_re:50051"]
#username = "lab"
#password = "lab123"
#client_id = "mx1_re"
......
ssl_cert  = "mx1_re.pem"
.......
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Juniper/open-nti/issues/230#issuecomment-413457360,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AcT0OwAYE_F8jWl4FSvg6x-Cj49Ir0_xks5uRSQJgaJpZM4VpbNk
.

Well, another thing I did :
in the file /etc/ssl/openssl.cnf I added the following  in the v3_ca section:
[ v3_ca ]
..........
subjectAltName = IP:<IP_ADDRESS> --> mx IP
.........
If you modify that file, you'll have to recreate the certifcate again and follow the steps described above in the thread.
Regards

Please turn debug on in the telegraf config file (agent section)
..........
[agent]
...........
debug = true
and share logs
Thanks

Here's the log. No clues in it. The timeout is till I add the host
manually in /etc/hosts on the OC container.
root@Jumphost2:/home/mohan/open-nti/plugins/input-oc# more telegraf.log
2018-08-17T13:45:10Z D! Attempting connection to output: influxdb
2018-08-17T13:45:10Z D! Successfully connected to output: influxdb
2018-08-17T13:45:10Z I! Starting Telegraf v1.7.0~503fcdb
2018-08-17T13:45:10Z I! Loaded outputs: influxdb
2018-08-17T13:45:10Z I! Loaded inputs: inputs.jti_openconfig_telemetry
2018-08-17T13:45:10Z I! Tags enabled: host=234777a00ade
2018-08-17T13:45:10Z I! Agent Config: Interval:10s, Quiet:false,
Hostname:"234777a00ade", Flush Interval:5s
2018-08-17T13:45:10Z D! Opened a new gRPC session to mx1_re on port 50051
2018-08-17T13:45:25Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:45:30Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:35Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:45:40Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:45:45Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:45:50Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:45:50Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:50Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:50Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:50Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:50Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:50Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:50Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:50Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:50Z D! Retrying mx1_re with timeout 1s
...
2018-08-17T13:46:10Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:46:10Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:46:10Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:46:10Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:46:10Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:46:10Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:46:10Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:46:10Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:46:15Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:46:20Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:46:25Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:46:30Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:46:35Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:46:40Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:46:45Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:46:50Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:46:55Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:47:00Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:47:05Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:47:10Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:47:15Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:47:20Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:47:25Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:47:30Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:47:35Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
On Fri, Aug 17, 2018 at 4:19 AM psagrera<EMAIL_ADDRESS>wrote:

Please turn debug on in the telegraf config file (agent section)
..........
[agent]
...........
debug = true
and share logs
Thanks
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Hi,
I’ll try to test with the Junos release you are using, but I ‘ll be out for a couple of weeks
Regards
Enviado desde mi iPhone

El 17 ago 2018, a las 20:22, mnanduri<EMAIL_ADDRESS>escribió:
Interesting, when i tried the above procedure with FQDN on a real MX,
it worked. Still does not work on vMX.
Cheers,
-Mohan
On Fri, Aug 17, 2018 at 11:00 AM Mohan Nanduri<EMAIL_ADDRESS>wrote:

Here's the log. No clues in it. The timeout is till I add the host
manually in /etc/hosts on the OC container.
root@Jumphost2:/home/mohan/open-nti/plugins/input-oc# more telegraf.log
2018-08-17T13:45:10Z D! Attempting connection to output: influxdb
2018-08-17T13:45:10Z D! Successfully connected to output: influxdb
2018-08-17T13:45:10Z I! Starting Telegraf v1.7.0~503fcdb
2018-08-17T13:45:10Z I! Loaded outputs: influxdb
2018-08-17T13:45:10Z I! Loaded inputs: inputs.jti_openconfig_telemetry
2018-08-17T13:45:10Z I! Tags enabled: host=234777a00ade
2018-08-17T13:45:10Z I! Agent Config: Interval:10s, Quiet:false,
Hostname:"234777a00ade", Flush Interval:5s
2018-08-17T13:45:10Z D! Opened a new gRPC session to mx1_re on port 50051
2018-08-17T13:45:25Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:45:30Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:30Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:35Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:45:40Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:45:45Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:45:50Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:45:50Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:50Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:50Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:50Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:50Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:50Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:50Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:50Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:45:50Z D! Retrying mx1_re with timeout 1s
...
2018-08-17T13:46:10Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:46:10Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:46:10Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:46:10Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:46:10Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:46:10Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:46:10Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:46:10Z D! Retrying mx1_re with timeout 1s
2018-08-17T13:46:15Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:46:20Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:46:25Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:46:30Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:46:35Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:46:40Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:46:45Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:46:50Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:46:55Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:47:00Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:47:05Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:47:10Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:47:15Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:47:20Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:47:25Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:47:30Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-08-17T13:47:35Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
On Fri, Aug 17, 2018 at 4:19 AM psagrera<EMAIL_ADDRESS>wrote:

Please turn debug on in the telegraf config file (agent section)
..........
[agent]
...........
debug = true
and share logs
Thanks
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

I was trying to do mutual authentication on Junipers and open-nti. Does open-nti send cert? The router is expecting the cert and it fails with bad cert error.
When we use gnmi_client to connect  to the router, we dont see that error but a diff one.
Here's the config:
set system services ssh
set system services extension-service request-response grpc ssl port 50051
set system services extension-service request-response grpc ssl local-certificate nqa3-mx-d12-12
set system services extension-service request-response grpc ssl mutual-authentication certificate-authority JTI
set system services extension-service request-response grpc ssl mutual-authentication client-certificate-request require-certificate
set system services extension-service traceoptions file ext.log
set system services extension-service traceoptions flag all
Error with bad cert -
Aug 28 02:10:30 server_secure_chttp2.c:119: Secure transport failed with error 1
Aug 28 02:12:28 ssl_transport_security.c:947: Handshake failed with fatal error SSL_ERROR_SSL: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate.
Aug 28 02:12:28 handshake.c:128: Security handshake failed: {"created":"@1535422348.938953752","description":"Handshake failed","file":"../../../../../../../../src/dist/grpc/src/core/lib/security/transport/handshake.c","file_line":264,"tsi_code":10,"tsi_error":"TSI_PROTOCOL_FAILURE"}
When using gnmi_cli client
Aug 28 03:12:23 server_secure_chttp2.c:119: Secure transport failed with error 1
Aug 28 03:12:34 TerminateClientThreads: Number of grpc clients connected: 0
Aug 28 03:12:34 JGrpcServerStart: Proto Desriptor object not found in proto map for RPC /grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo for peer ipv6:::ffff:<IP_ADDRESS>:35556
Aug 28 03:12:34 AllocCallMem:GRPC Server Call Completion queue created successfully
Aug 28 03:12:34 AllocCallMem:GRPC Server Call Details initialized successfully
Aug 28 03:12:35 TerminateClientThreads: Number of grpc clients connected: 0
Aug 28 03:12:35 JGrpcServerStart: Proto Desriptor object not found in proto map for RPC /gnmi.gNMI/Subscribe for peer ipv6:::ffff:<IP_ADDRESS>:35556
Aug 28 03:12:35 AllocCallMem:GRPC Server Call Completion queue created successfully
Aug 28 03:12:35 AllocCallMem:GRPC Server Call Details initialized successfully
Aug 28 03:12:35 ssl_transport_security.c:439: SSL_read returned 0 unexpectedly.
Aug 28 03:12:35 secure_endpoint.c:176: Decryption error: TSI_INTERNAL_ERROR
./gnmi_cli -a mx-d12-12:50051   -qt s -q "/lldp/" --ca_crt /home/mnanduri/ca.crt --client_crt /home/mnanduri/pivo.crt --client_key /home/mnanduri/pivo.key -logtostderr
E0828 09:43:38.684796   23426 gnmi_cli.go:190] cli.QueryDisplay:
sendQueryAndDisplay(ctx, {Addrs:[nqa3-mx10003-d12-12:50051] Target: Replica:0 UpdatesOnly:false Queries:[[lldp]] Type:stream Timeout:30s NotificationHandler: ProtoHandler: Credentials: TLS:0xc7ef80 Extra:map[]}, &{PollingInterval:30s StreamingDuration:0s Count:0 countExhausted:false Delimiter:/ Display:0x83db90 DisplayPrefix: DisplayIndent:   DisplayType:group DisplayPeer:false Timestamp: DisplaySize:false Latency:false ClientTypes:[gnmi]}):
unknown response : %!s()

Hi
The current implementation of open NTI it doesn't support mutual authentication, I did a quick test modifying the code of the telegraf plugin and it works ok, so  probably mutual authentication  be supported in the future.

ah cool, thanks for the update and checking. Do you mind providing the code
that you played/modified with?
On Tue, Sep 4, 2018 at 10:04 AM psagrera<EMAIL_ADDRESS>wrote:

Hi
The current implementation of open NTI it doesn't support mutual
authentication, I did a quick test modifying the code of the telegraf
plugin and it works ok, so probably mutual authentication be supported in
the future.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Juniper/open-nti/issues/230#issuecomment-418379228,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AcT0O5CGKn14k2_Yjij_eOKFQSXwrCBgks5uXohqgaJpZM4VpbNk
.

Hi,
Follow those steps in order to test mutual-auth in open NTI (bear in mind that is only for testing/demo purposes at the moment).
1 ) clone my personal repo of openNTI
git clone https://github.com/psagrera/open-nti.git

2 ) Modify the following files:
2.1 ) Under ~/open-nti/plugins/input-oc --> telegraf.tmpl

	  [[inputs.jti_openconfig_telemetry]]

		servers = ["mx2_re:50051"] <-- your vMX / MX hostname
		username = "lab"
		password = "lab123"
		client_id = "telegraf"

	  Keep debug flag to true to verify easily if it's working or not
	  You can add/modify sensors from the original file

2.2 )  Under ~/open-nti/plugins/input-oc --> cert_files dierctory
	   Put there all files related to the certificate (follow the process you mentioned above in the thread )
	   		2.2.1 That's what I did

	   				SERVER SIDE
	   				###########
	   			openssl genrsa -out ca.key 2048
				openssl req -new -x509 -key ca.key -out ca.crt

				openssl genrsa -out mx2_re.key 2048
				openssl req -new -key mx2_re.key -out mx2_re.csr

				openssl genrsa -out oc.key 2048
				openssl req -new -key oc.key -out oc.csr

				openssl x509 -req -in mx2_re.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out mx2_re.crt
				openssl x509 -req -in oc.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out oc.crt

				cat mx2_re.key mx2_re.crt > mx2_re.pem

				     ROUTER
				     ######
				set security pki ca-profile OC ca-identity OC
                request security pki ca-certificate load ca-profile OC filename /var/tmp/ca.crt
                set security certificates certification-authority OC ca-name OC
                set security certificates local mx2_re load-key-file /var/tmp/mx2_re.pem
			    set system services extension-service request-response grpc ssl port 50051
			    set system services extension-service request-response grpc ssl local-certificate mx2_re
			    set system services extension-service request-response grpc ssl mutual-authentication 				certificate-authority OC
			    set system services extension-service request-response grpc ssl mutual-authentication 				client-certificate-request require-certificate
			    set system services extension-service notification allow-clients address <IP_ADDRESS>/0
			    set system services extension-service traceoptions file extension-service.log
			    set system services extension-service traceoptions file size 5m
			    set system services extension-service traceoptions file files 2
			    set system services extension-service traceoptions flag all

2.3 ) Under ~/open-nti  --> docker-compose.yml

		input-oc:
		  #image: telegraf:1.5
		  build: $INPUT_OC_DIR
		  extra_hosts:
		    mx2_re: <IP_ADDRESS> <-- put here your hostname/IP mapping
		  container_name: $INPUT_OC_CONTAINER_NAME
		  volumes:
		    - /etc/localtime:/etc/localtime:ro
		    - ./$INPUT_OC_DIR/telegraf.tmpl:/source/telegraf.tmpl
		  ports:
		    - "$LOCAL_PORT_OC:50051/udp"
		  links:
		    - opennti

2.4 ) make build

2.5 ) make start

2.6 ) docker logs opennti_input_oc (If it's working you will see something like:)

    [...]
	2018-09-05T11:55:18Z I! Agent Config: Interval:10s, Quiet:false, Hostname:"c7fb8b1cd83c", Flush Interval:5s
    2018-09-05T11:55:20Z D! Opened a new gRPC session to mx2_re on port 50051
    2018-09-05T11:55:25Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
    2018-09-05T11:55:27Z D! Received from mx2_re: system_id:"mx2_re" component_id:1 path:
    [...]

P.S : I'm using vMX (18.1R2-S1)

Thanks a ton, let me try it out. Will keep you posted.
On Wed, Sep 5, 2018 at 9:35 AM psagrera<EMAIL_ADDRESS>wrote:

Hi,
Follow those steps in order to test mutual-auth in open NTI (bear in mind
that is only for testing/demo purposes at the moment).
1 ) clone my personal repo of openNTI
git clone https://github.com/psagrera/open-nti.git
2 ) Modify the following files:
2.1 ) Under ~/open-nti/plugins/input-oc --> telegraf.tmpl
[[inputs.jti_openconfig_telemetry]]

  servers = ["mx2_re:50051"] <-- your vMX / MX hostname
  username = "lab"
  password = "lab123"
  client_id = "telegraf"

Keep debug flag to true to verify easily if it's working or not
You can add/modify sensors from the original file

2.2 )  Under ~/open-nti/plugins/input-oc --> cert_files dierctory
Put there all files related to the certificate (follow the process you mentioned above in the thread )
2.2.1 That's what I did
 				SERVER SIDE
 				###########
 			openssl genrsa -out ca.key 2048
  		openssl req -new -x509 -key ca.key -out ca.crt

  		openssl genrsa -out mx2_re.key 2048
  		openssl req -new -key mx2_re.key -out mx2_re.csr

  		openssl genrsa -out oc.key 2048
  		openssl req -new -key oc.key -out oc.csr

  		openssl x509 -req -in mx2_re.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out mx2_re.crt
  		openssl x509 -req -in oc.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out oc.crt

  		cat mx2_re.key mx2_re.crt > mx2_re.pem

  		     ROUTER
  		     ######
  		set security pki ca-profile OC ca-identity OC
            request security pki ca-certificate load ca-profile OC filename /var/tmp/ca.crt
            set security certificates certification-authority OC ca-name OC
            set security certificates local mx2_re load-key-file /var/tmp/mx2_re.pem
  	    set system services extension-service request-response grpc ssl port 50051
  	    set system services extension-service request-response grpc ssl local-certificate mx2_re
  	    set system services extension-service request-response grpc ssl mutual-authentication 				certificate-authority OC
  	    set system services extension-service request-response grpc ssl mutual-authentication 				client-certificate-request require-certificate
  	    set system services extension-service notification allow-clients address <IP_ADDRESS>/0
  	    set system services extension-service traceoptions file extension-service.log
  	    set system services extension-service traceoptions file size 5m
  	    set system services extension-service traceoptions file files 2
  	    set system services extension-service traceoptions flag all

2.3 ) Under ~/open-nti  --> docker-compose.yml
  input-oc:
    #image: telegraf:1.5
    build: $INPUT_OC_DIR
    extra_hosts:
      mx2_re: <IP_ADDRESS> <-- put here your hostname/IP mapping
    container_name: $INPUT_OC_CONTAINER_NAME
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./$INPUT_OC_DIR/telegraf.tmpl:/source/telegraf.tmpl
    ports:
      - "$LOCAL_PORT_OC:50051/udp"
    links:
      - opennti

2.4 ) make build
2.5 ) make start
2.6 ) docker logs opennti_input_oc (If it's working you will see something like:)
[...]

2018-09-05T11:55:18Z I! Agent Config: Interval:10s, Quiet:false, Hostname:"c7fb8b1cd83c", Flush Interval:5s
2018-09-05T11:55:20Z D! Opened a new gRPC session to mx2_re on port 50051
2018-09-05T11:55:25Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-09-05T11:55:27Z D! Received from mx2_re: system_id:"mx2_re" component_id:1 path:
[...]
P.S : I'm using vMX (18.1R2-S1)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Juniper/open-nti/issues/230#issuecomment-418732445,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AcT0Ox_tIp2CsDKe-wkdcGJyqixXMRFMks5uX9MDgaJpZM4VpbNk
.

Hi,
Did you remove old image of OC ? Maybe it's using old image with new plugin.

oh i did not. Let me remove the old image.

you are right, it was using the old image. That error is gone. I am getting a cert error, will try to regenerate again.

One question i wanted to ask and forgot earlier, i see you generated oc.key and crt but i dont see any reference in your telegraf.tmpl file. Is that expected?

Yes , its expected. In the telegraf.tmpl file only this references are needed:
client_crt = "/opt/telegraf/config/cert_files/mx2_re.crt"
client_key = "/opt/telegraf/config/cert_files/mx2_re.key"
ca_crt = "/opt/telegraf/config/cert_files/ca.crt"
ssl_cert  = "/opt/telegraf/config/cert_files/mx2_re.pem"
I generated those files following the procedure of David Gee (http://ipengineer.net/2018/05/configuring-ssl-grpc-junos/)

The files I've got in the vMX:
root@mx2_re> file list /var/tmp/
/var/tmp/:
[...]
ca.crt
ca.key
ca.srl
mx2_re.crt
mx2_re.csr
mx2_re.key
mx2_re.pem
[...]

Ok. I was thinking that OC will send its cert/key and mx has its cert/key, they will mutually authenticate each others certs.

yeah, i was trying to avoid using username password but when I use uid/password or without uid/pwd + skip-auth on router, it works.

If you use username+passw+client_id without skip config , it works ? (That's the setup I'm using)

yeap, that works.

Ok, glad to hear that. As I said , that version is only for testing/demo purposes, once is published in telegraf we will integrate that into the master branch.
Hope it's been useful
Thanks.

Yes, thanks a ton for your  help. WIll keep an eye for the merge.
On Fri, Sep 7, 2018 at 10:35 AM psagrera<EMAIL_ADDRESS>wrote:

Ok, glad to hear that. As I said , that version is only for testing/demo
purposes, once is published in telegraf we will integrate that into the
master branch.
Hope it's been useful
Thanks.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Juniper/open-nti/issues/230#issuecomment-419459082,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AcT0OxhEWuuDazPI8_-li19JrWGuh9yjks5uYoQZgaJpZM4VpbNk
.

Clarification question - if i have this below and using the normal (not your new plugin that supports SSL) instance to authenticate grpc session. I see initially auth failed but get sensor data...
[[inputs.jti_openconfig_telemetry]]
servers = ["<IP_ADDRESS>:50051"]
username = "lab"
password = "lab123"
client_id = "telegraf"
set system services extension-service request-response grpc clear-text port 50051
set system services extension-service traceoptions file extension-service.log
set system services extension-service traceoptions file size 5m
set system services extension-service traceoptions file files 2
set system services extension-service traceoptions flag all
2018-09-14T11:14:52Z E! Error in plugin [inputs.jti_openconfig_telemetry]: E! Failed to read from rpc error: code = Internal desc = invalid header field value "Authorization failed\b\xf7\xf5\b\x03": <IP_ADDRESS>
2018-09-14T11:14:54Z E! Error in plugin [inputs.jti_openconfig_telemetry]: E! Failed to read from rpc error: code = Unknown desc = Authorization failed: <IP_ADDRESS>
2018-09-14T11:14:55Z E! Error in plugin [inputs.jti_openconfig_telemetry]: E! Failed to read from rpc error: code = Unknown desc = Authorization failed: <IP_ADDRESS>
2018-09-14T11:14:55Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-09-14T11:14:55Z E! Error in plugin [inputs.jti_openconfig_telemetry]: E! Failed to read from rpc error: code = Unknown desc = Authorization failed: <IP_ADDRESS>
2018-09-14T11:14:56Z E! Error in plugin [inputs.jti_openconfig_telemetry]: E! Failed to read from rpc error: code = Unknown desc = Authorization failed: <IP_ADDRESS>
2018-09-14T11:14:56Z D! Received from <IP_ADDRESS>: system_id:"nqa3-mx10003-d12-12" path:"sensor_1000:/junos/system/linecard/firewall/:/junos/system/linecard/firewall/:PFE" timestamp:1536952162823 kv:<key:"timestamp" uint_value:1536952162893 > kv:<key:"prefix" str_value:"/junos/firewall[name='IPV4_PROTECT_RE']/state/" > kv:<key:"timestamp" uint_value:1536891839 > kv:<key:"memory-usage[name='HEAP']/allocated" uint_value:131812 > kv:<key:"prefix" str_value:"/junos/firewall[name='count-dscp-et-0/1/0.0-i']/state/" > kv:<key:"timestamp" uint_value:1536884774 > kv:<key:"memory-usage[name='HEAP']/allocated" uint_value:1396 > kv:<key:"counter[name='dscp0-et-0/1/0.0-i']/packets" uint_value:577503868 > kv:<key:"counter[name='dscp0-et-0/1/0.0-i']/bytes" uint_value:285286910792 > kv:<key:"counter[name='rest-et-0/1/0.0-i']/packets" uint_value:264198 > kv:<key:"counter[name='rest-et-0/1/0.0-i']/bytes" uint_value:15644639 > kv:<key:"prefix" str_value:"/junos/firewall[name='count-dscp-et-0/1/1.0-i']/state/" > kv:<k

Hi
Is the data being inserted properly into the database ?
Enviado desde mi iPhone

El 14 sept 2018, a las 13:17, mnanduri<EMAIL_ADDRESS>escribió:
Clarification question - if i have this below and using the normal (not your new plugin that supports SSL) instance to authenticate grpc session. I see initially auth failed but get sensor data...
[[inputs.jti_openconfig_telemetry]]
servers = ["<IP_ADDRESS>:50051"]
username = "lab"
password = "lab123"
client_id = "telegraf"
set system services extension-service request-response grpc clear-text port 50051
set system services extension-service traceoptions file extension-service.log
set system services extension-service traceoptions file size 5m
set system services extension-service traceoptions file files 2
set system services extension-service traceoptions flag all
2018-09-14T11:14:52Z E! Error in plugin [inputs.jti_openconfig_telemetry]: E! Failed to read from rpc error: code = Internal desc = invalid header field value "Authorization failed\b\xf7\xf5\b\x03": <IP_ADDRESS>
2018-09-14T11:14:54Z E! Error in plugin [inputs.jti_openconfig_telemetry]: E! Failed to read from rpc error: code = Unknown desc = Authorization failed: <IP_ADDRESS>
2018-09-14T11:14:55Z E! Error in plugin [inputs.jti_openconfig_telemetry]: E! Failed to read from rpc error: code = Unknown desc = Authorization failed: <IP_ADDRESS>
2018-09-14T11:14:55Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-09-14T11:14:55Z E! Error in plugin [inputs.jti_openconfig_telemetry]: E! Failed to read from rpc error: code = Unknown desc = Authorization failed: <IP_ADDRESS>
2018-09-14T11:14:56Z E! Error in plugin [inputs.jti_openconfig_telemetry]: E! Failed to read from rpc error: code = Unknown desc = Authorization failed: <IP_ADDRESS>
2018-09-14T11:14:56Z D! Received from <IP_ADDRESS>: system_id:"nqa3-mx10003-d12-12" path:"sensor_1000:/junos/system/linecard/firewall/:/junos/system/linecard/firewall/:PFE" timestamp:1536952162823 kv:<key:"timestamp" uint_value:1536952162893 > kv:<key:"prefix" str_value:"/junos/firewall[name='IPV4_PROTECT_RE']/state/" > kv:<key:"timestamp" uint_value:1536891839 > kv:<key:"memory-usage[name='HEAP']/allocated" uint_value:131812 > kv:<key:"prefix" str_value:"/junos/firewall[name='count-dscp-et-0/1/0.0-i']/state/" > kv:<key:"timestamp" uint_value:1536884774 > kv:<key:"memory-usage[name='HEAP']/allocated" uint_value:1396 > kv:<key:"counter[name='dscp0-et-0/1/0.0-i']/packets" uint_value:577503868 > kv:<key:"counter[name='dscp0-et-0/1/0.0-i']/bytes" uint_value:285286910792 > kv:<key:"counter[name='rest-et-0/1/0.0-i']/packets" uint_value:264198 > kv:<key:"counter[name='rest-et-0/1/0.0-i']/bytes" uint_value:15644639 > kv:<key:"prefix" str_value:"/junos/firewall[name='count-dscp-et-0/1/1.0-i']/state/" > kv:<k
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

I think so. Did not validate for sure.

Any clue on the traceoptions ?

I lost the log, let me try to do it again and will get back.

how can we have the SSL setup for multiple devices? should I use the same ssl certificate on all of the devices? I am looking to have SSL for encrypting the traffic between devices and the NTI server. also I would like to have the SSL authentication as well.

Hi @littlespace
You can define more than one input plugin and therefore attach different certificates  to each group of servers (i.e)
    ###############################################################################
    #                            INPUT PLUGINS                                    #
    ###############################################################################

    # Read OpenConfig Telemetry from listed sensors

    [[inputs.jti_openconfig_telemetry]]

      servers = ["<IP_ADDRESS>:50051"]

      ## Frequency to get data in millisecond
      sample_frequency = "5000ms"

      sensors = [
       "/network-instances/network-instance/protocols/protocol/bgp/",
      ]

    str_as_tags = false

    [[inputs.jti_openconfig_telemetry]]

      servers = ["<IP_ADDRESS>:50051"]

      ## Frequency to get data in millisecond
      sample_frequency = "5000ms"

      sensors = [
       "/network-instances/network-instance/protocols/protocol/bgp/",
      ]

    str_as_tags = false

With regard mutual authentication we are still working on the final code we will merge into the master branch.
Regards
Pablo

Hello Pablo,
When you say mutual authentication is being worked on - you mean open-nti to authenticate cert received from the router right? just want to confirm.
Cheers,
-Mohan

Yes, authenticate cert received from the router
Cheers
Pablo
Enviado desde mi iPhone

El 24 sept 2018, a las 19:02, mnanduri<EMAIL_ADDRESS>escribió:
Hello Pablo,
When you say mutual authentication is being worked on - you mean open-nti to authenticate cert received from the router right? just want to confirm.
Cheers,
-Mohan
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

got it, thanks for the confirmation.
On Mon, Sep 24, 2018 at 2:15 PM psagrera<EMAIL_ADDRESS>wrote:

Yes, authenticate cert received from the router
Cheers
Pablo
Enviado desde mi iPhone

El 24 sept 2018, a las 19:02, mnanduri<EMAIL_ADDRESS>escribió:
Hello Pablo,
When you say mutual authentication is being worked on - you mean
open-nti to authenticate cert received from the router right? just want to
confirm.
Cheers,
-Mohan
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Juniper/open-nti/issues/230#issuecomment-424072944,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AcT0O5t_-B8ZVdO9sU7f-bVlsa9A6rSPks5ueSFHgaJpZM4VpbNk
.

Any update on merging this feature with the main?

Yes , its expected. In the telegraf.tmpl file only this references are needed:
client_crt = "/opt/telegraf/config/cert_files/mx2_re.crt"
client_key = "/opt/telegraf/config/cert_files/mx2_re.key"
ca_crt = "/opt/telegraf/config/cert_files/ca.crt"
ssl_cert = "/opt/telegraf/config/cert_files/mx2_re.pem"
I generated those files following the procedure of David Gee (http://ipengineer.net/2018/05/configuring-ssl-grpc-junos/)

@psagrera Do you have the config for latest telegraf version (1.14)? TLS config params have been changed slightly. And why do you use the router certificate and key (mx2_ce.xxx) as client crt and key? Shouldn't they be the client certs and key (oc.xxx)?

@pravindamodaran check the telegraf repo. I have added the tls option to the jti plug-in almost a year ago.

@littlespace Maybe irrelevant here, but my telegraf config looks like this:
[[inputs.jti_openconfig_telemetry]]
  ## List of device addresses to collect telemetry from
  servers = ["ec2-10-10-10-10.us-east-2.compute.amazonaws.com:32767"]

  username = "test"
  password = "test123"
  client_id = "telegraf"

  sample_frequency = "50000ms"

  sensors = [
   "/interfaces"
  ]

  ## Optional TLS Config
   enable_tls = true
   tls_ca = "/etc/telegraf/certs/ca.crt"
   tls_cert = "/etc/telegraf/certs/client.crt"
   tls_key = "/etc/telegraf/certs/client.key"
  ## Use TLS but skip chain & host verification
   insecure_skip_verify = false

  retry_delay = "10000ms"

  ## To treat all string values as tags, set this to true
  str_as_tags = false

And I keep getting this error in telegraf:
telegraf_1    | 2020-05-27T15:32:12Z E! [inputs.jti_openconfig_telemetry] Error in plugin: failed to read from <IP_ADDRESS>: rpc error: code = Unknown desc = Authorization failed

Any idea? Should I open an issue in the telegraf repo?

@pravindamodaran I need to check it, what version of junos are you running on the box? Is your username super user as well?

@pravindamodaran I need to check it, what version of junos are you running on the box? Is your username super user as well?

Junos version: JUNOS 18.4R1.8 Kernel 64-bit  JNPR-11.0-20181207.6c2f68b_2_b
This is the default image available in AWS Marketplace for virtual router. And yeah, I have set the class as super-user for my username

Same question for the TLS problem
How do i modify telegraf.tmpl if i use master ???
I just add "username", "password", "client_id", "enable_tls" and the tls cert, but it fails to compose in the opennti_input_oc docker.
Thanks!


sync gateway compatibility

I use Lite-Core for C development. I ran a replicator in my code for connection to sync gateway. But i received error 403  (403 Forbidden ). i read some issues and so i realize that my problem is mismatch of lite-core version and sync gateway version. so my questions:

what is Lite-Core version?
what is Sync Gateway version for compatibility with Lite-Core

Filed against wrong repo. Also we do not support accessing LiteCore directly, only through a Couchbase Lite implementation.


Python Updates 2022-03-30

Likely the last python updates series before the upcoming zero hydra failures period.
https://hydra.nixos.org/job/nixpkgs/python-updates/tested#tabs-constituents
The previous cycle was #162538 and is currently still in staging-next due to delays caused by the latest OpenSSL update and problems around x86_64-darwin on hydra.

Rebased & dropped the superseded commits

Flask-RESTful appears broken due to https://github.com/flask-restful/flask-restful/issues/936. There's a fix, but not merged yet.

Flask-RESTful appears broken due to flask-restful/flask-restful#936. There's a fix, but not merged yet.

Adapted a change from flask-restx in ee355b0 (#166489).

@ofborg eval

Can we ask upstream about making black tests optional instead of carrying that patch?

/me considers writing a no-op dummy black package.

mitmproxy appears broken, will have to investigate tomorrow.

https://shells.darmstadt.ccc.de/~hexa/105.html
grpc fix over in #167129

https://shells.darmstadt.ccc.de/~hexa/107.html

shells.darmstadt.ccc.de/~hexa/107.html

Would be nice if we could link this in the first comment otherwise you need to expand multiple times the github thread to be able to see it.

Git currently fails to build on staging so evals are failing early.

https://malob.github.io/nix-review-tools-reports/nixpkgs:python-updates/nixpkgs_python-updates_1755257.html

Removed pyenchant from pylints check inputs because it introduces a dependency on nuspell/pandoc which is a bit heavy for my taste.


feat(hubble): number of hub peers in info http api

What is the feature you would like to implement?
Hub operators want to know the number of peers that their hubble is connected to.
Why is this feature important?
This helps for monitoring the effectiveness and health of a hub.
Will the protocol spec need to be updated??
I don't think so
How should this feature be built? (optional)
Add a peers key to the info api (https://www.thehubble.xyz/docs/httpapi/info.html#info) with a value of the number of peers or a list of peer-ids from libp2p
Additional context
Ethereum clients implement peer apis (ex. https://ethereum.github.io/beacon-APIs/#/)

Thanks for the suggestion!
We now have a dedicated endpoint for this: https://www.thehubble.xyz/docs/httpapi/currentpeers.html


saved views bug fix + e2e fixes

Fixed a bug in saved views editor caused by https://github.com/mui/material-ui/issues/10341.
The editor doesn't work without disablePortal prop. On e2e side, because of this change, MUI backdrop intercepts mouse clicks and keypresses, and tests fail. The broken tests in saved views spec have been annotated with .fixme() until we we have a fix.
Update e2e dependencies, run linter, fix type issues, etc.

Currently having difficulty with my local playwright environment. It'd be a shame to disable all these tests. I personally haven't been able to reproduce the call stack issue

Happy to defer to @manivoxel51. But maybe we just disable/fix the [saved views › searching through saved views works] spec?

@sashankaryal I'll pull and test around tomorrow. haven't had a chance to dive deep. thanks for raising this 🙌


Title overlaps text on mobile

Something's up with the CSS on the chat channel title. Desktop is not affected, only mobile. It's positioned lower than it should be and overlaps with the text area. When scrolling this frequently means overlapped text.
I don't think its a text overflow problem. Even short channel names such as "chat" are affected.
See screenshot:

cc @JSey any idea about this one? I can confirm it... wonder if there's a better way to think about styling this sidebar, which has, to this point, has been really fragile.

This one should be fixed with the markup refactor


Remove client side file watcher registration, formatting

https://github.com/bmewburn/vscode-intelephense/commit/338c983c0a2b974acbda7d92ff7b574886f561ca
https://github.com/bmewburn/vscode-intelephense/issues/414

I see bmewburn/vscode-intelephense has not released a new version, not sure the stability


iOSDisconnectNetwork() does not actually disconnect at all

Prerequisites
Check all boxes if you have done the following:

[x] Checked that your issue isn't already filed: https://github.com/tripflex/wifiwizard2/issues
[x] Make sure you fill out the Issue Type below

Issue type
Select all that apply

[x] Bug
[ ] Enhancement
[ ] Task
[ ] Question
[ ] Other

Description
I'm testing this plugin on an iPhone SE with iOS 13.2.3. Here are my observations:
        WifiWizard2.getConnectedSSID()
            .then(
                function(v) {
                    console.log('GOT SSID: '+v);
                    if (v) {
                        console.log('disconnecting from: '+v);
                        WifiWizard2
                            .iOSDisconnectNetwork(v)
                            .then(
                                function(){
                                    setTimeout(app.refreshSSID,1000);
                                }
                            );
                    }
                    else {
                        let ssid = view.getSSIDInput();
                        console.log('connecting to: '+ssid);
                        WifiWizard2
                            .iOSConnectNetwork(ssid, false)
                            .then(
                                function(){
                                    setTimeout(app.refreshSSID,1000);
                                }
                            );
                    }
                }
            )
            .catch(
                function(e){
                    console.error(e);
                    alert("can't get current SSID: "+e.message);
                }
            );

=> WifiWizard2.iOSDisconnectNetwork(v) enters in its then block but it doesn't disconnect the device from the wifi hotspot.
Steps to Reproduce

Use the code above and run it.

Expected behavior:
When entering the then blowk of the iOSDisconnectNetwork operation, the device should be disconnected from its wifi hotspot.
Actual behavior:
Nothing seems to happen when it should actually disconnect from the wifi.
Reproduces how often: 100% of the time
Versions
$ cordova -v
9.0.0<EMAIL_ADDRESS>
$ cordova platform list
Installed platforms:
  ios 5.1.1
Available platforms:
  android ^8.0.0
  browser ^6.0.0
  electron ^1.0.0
  osx ^5.0.0
  windows ^7.0.0

$ cordova plugin list
com.wizpanda.cordova.guided-access 0.0.1 "cordova-ios-guided-access"
cordova-plugin-add-swift-support 2.0.2 "AddSwiftSupport"
cordova-plugin-battery-status 2.0.5 "Battery"
cordova-plugin-brightness 0.1.5 "Brightness"
cordova-plugin-device 2.0.3 "Device"
cordova-plugin-device-name 1.3.5 "Device-Name"
cordova-plugin-exit 1.0.3 "Exit"
cordova-plugin-file 6.0.2 "File"
cordova-plugin-geolocation 4.0.2 "Geolocation"
cordova-plugin-whitelist 1.3.4 "Whitelist"
cordova-plugin-wkwebview-engine 1.2.1 "Cordova WKWebView Engine"
es6-promise-plugin 4.1.0 "Promise"
phonegap-plugin-barcodescanner 8.1.0 "BarcodeScanner"
wifiwizard2 3.1.1 "WifiWizard2"

Additional Information
n/a

I think that it probably comes from the fact that I did not connect to that hotspot with the iOSConnectNetwork method...


Make small corrections to human readable text

Remove references to task ID being a UUID
Fix task counter plural in inconsistency message
Fix typo
Fix 'Globus configuration' subheading level
This is now the same level as other globus/rsync subtopics
(using " which is one level lower than ^)
Remove whitespace in doc example
Add whitespace
Fix a(n) typo
Remove details of what the dependencies are for AWS
All the user needs to know is that they should install them.
Move misplaced comment in torque provider

@benclifford I've resolved some merge conflicts that you might want to check.

@benclifford I've resolved some merge conflicts that you might want to check.

@yadudoc conflict resolution looks fine

@yadudoc conflict resolution looks fine


Upgraded to Sql Parser 0.50

As per title, upgraded from SQL Parser 0.49 to 0.50

Pull Request Test Coverage Report for Build<PHONE_NUMBER>4
Details

7 of 7 (100.0%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.005%) to 98.357%

Totals

Change from base Build<PHONE_NUMBER>6:
0.005%

Covered Lines:
55618

Relevant Lines:
56547

💛  - Coveralls


When eval the model on amazon datasets, the result fluctuate.

Hi @rktamplayo , thank you for your great work!
I followed the README.md to reproduce the experimental results on Amazon dataset and use the check point you released to evaluate the test.json. But I didn't get the same result as you claim in the paper and the result I got is fluctuating, which is I evaluate 5 times or more but every time I got diffrent result. I wonder that whether there is some other settings I missed.

I've solved this question, the 234 line  sum_tokens[token_ids] += tokens[tindex] have potential randomness. You should change the writing style or set torch.use_deterministic_algorithms(True) to avoid this eval fluctuation.


Readme review

Adjust badges: Viewer v7, add Assets,
Description: links are pointing to developer.autodesk.com, adjust to forge.autodesk.com, Viewer link points to v6, and has a "under construction" line...
Please have at lest 1 thumbnail.png (exact name, so it appears on the list, thumbnail-1.png will not work)
Suggestion: write on the thumbnail, over the panel, a text indicating the extension name (mentioned on the text), makes easier to identify what is what
Some images have links to videos, please add a text with link
Live link not working: https://bim360assets.herokuapp.com
At the end of Work with your models section you point to other samples, please move that to "Further reading"
Running locally sections seems to be missing images
"Further reading" section contains links to bim360-private and some TDB

Updated with this PR https://github.com/Autodesk-Forge/forge-bim360-assets.viewer/pull/2


PUBD-298,319 Update EZID Plugin to work with 1.4.x

Revamp the plugin_settings file to match what Janeway 1.4.x expects
Handle null authors by looking for an account if the author object is null

Installs without complaint on the dev server. Needs testing to confirm the plugin still works.

the management/CLI commands seem to be working correctly on dev, still need to submit and accept a test preprint

It turns out that the error we observed in testing was due to a small bug in Janeway, which was fixed with this commit: https://github.com/BirkbeckCTP/janeway/commit/26767d0905cdc3f53997841f50e48c32b97bcf75
I'll do another test to confirm things are working.


Speedyspeech loss

Hi, this work is very impressive to me.
First of all, thanks to your work, paper and codes.
But i have a question about Speedyspeech loss.
According to the paper, speedyspeech use MAE and SSIM loss for logarithmic mel spectrogram
We use the sum of MAE and structural similarity index (SSIM) [12] losses for logarithmic mel spectrogram value regression

but in codes, it uses MAE and SSIM loss for just output from decoder which is not logarithmic i guess.
I wonder why it does not use loss for logarithmic output.
Can you help me for this question?
thank you

Hi, thanks for the kind words and for a good question. :) The output from the decoder is trained to match normalized log transformed mel spectrograms. Once trained, the model therefore outputs similarly transformed spectrograms. If you want to feed the result eg. to griffin-lim, you have to 1) denormalize 2) exponentiate 3) invert mel to linear. Does that answer your question?

Thank you for kind answer
i should have checked data loader but i didn't
thank you

Hi, thanks for the kind words and for a good question. :) The output from the decoder is trained to match normalized log transformed mel spectrograms. Once trained, the model therefore outputs similarly transformed spectrograms. If you want to feed the result eg. to griffin-lim, you have to 1) denormalize 2) exponentiate 3) invert mel to linear. Does that answer your question?

Hi, @janvainer. May I ask a question? when computing the SSIM between two log mel-spectorgram as metric to evaluate sound quality of the synthetic audio(vocoder output), is it ok to directly input the log mel-spectorgram to the SSIM (assume there's a function) without normalizing it?

Its probably best if you can ensure the same volume/MVN normalization of your mels.


Chore/modal class to fc

Reopened due to dependency from #4937

simplify out the close modal tree walking

Was merged in #4937


Consolidate tests in scale/crop mode

This commit focus on the problem according to issue#23. There are three
main changes:

Modify TRY_FMT to only negotiate and fill in information to user,
while S_FMT responsible for switching device format.
Initialize the expected format in create_vcam_device rather than
accomplish it in S_FMT, and open S_FMT scaling option for user to
switch from three kinds of resolution defined in vcam_sizes[]
Extract negotiate_resolution and set_crop_resolution function for
reuse in TRY_FMT, S_FMT, create_vcam_device.

The version can pass compliance test from scaling mode and cropping mode.
However the current version could not pass scaling+cropping mode in
TRY_FMT and S_FMT, which I think in current implementation is not able
to pass it. Because G_FMT return crop resolution, and in TRY_FMT will
negotiate with crop resolution which will return a new resolution type.
The test will fail when "TRY_FMT(G_FMT) != G_FMT".
The reason why we can use it normally on vlc player is that vlc will S_FMT
base on input_format but not the format return by G_FMT. I don't know
the detail how vlc player work, but it is the behavior I found.

@jserv  I have change the topic to: Modify TRY/S_FMT to pass test in scale/crop mode.

Thank @kevinshieh0225 for contributing. I amended the commit messages for consistent expressions.


Add jit compatibility

[x] remove global variables (e.g. EPS)
[x] remove @property decorator

this PR only makes changes to make Demask torch scriptable. There are probably more changes required for other models. And some won't work at all currently: https://github.com/pytorch/pytorch/pull/37749

This is a first step, thanks a lot !

Do you know what this is about?

@mpariente no, not really. But maybe we can take out a few commits to get this merged.
I think at least 3cbfb991b60bbc7353947e5e1401d0038ac2a483 should be picked

Only 5 errors left but I don't know why, you didn't touch schedulers or system's code, that seems weird.

@mhu-coder will have a look at that as well, so let's look out for merge conflicts.
We'll be three on the same branch. Please rebase (git pull --rebase) when possible and merge only when there are merge rebase conflicts.
Also, there will be some tricky decisions to make to fix jitability, take a stab at it and add a comment on the PR at the places you're not sure about, we'll iterate from there.
Thanks in advance !

I wonder if we shouldn't trace models instead of scripting them, the static typing can be more annoying than the tracing. And we would script the methods we know won't be faithful. I don't know if we have a lot of dynamic control flow in the models. Any opinion?

I wonder if we shouldn't trace models instead of scripting them, the static typing can be more annoying than the tracing. And we would script the methods we know won't be faithful. I don't know if we have a lot of dynamic control flow in the models. Any opinion?

yes, tracing would surely be favorable but I thought scripting is easier and should be done first. Also tracing requires fixed batch sizes for recurrent models. Not sure if we want to go that route?

But, you can mix scripting in tracing. Where needed, we would add the @torch.jit.script decorator (which can then handle the batch sizes in RNNs I guess), for the rest the tracing would handle the static control flow so we don't need to fix all the scripting problems.
The intermediate representation is the same between scripting and tracing so the end models can be the same if we handle it well.
I asked for opinions/feedback on twitter, just in case.

But, you can mix scripting in tracing. Where needed, we would add the @torch.jit.script decorator (which can then handle the batch sizes in RNNs I guess), for the rest the tracing would handle the static control flow so we don't need to fix all the scripting problems.

yep, thats how I handled it in open-unmix now. The core (including the LSTM) is traceable, the wiener filter is scriptable.

Torchscript assumes that every variable with undefined types are Tensors, then crashes at runtime when it discovers that it wasn't the case or when the type of the variable changed. This also includes function keyword arguments as it does not use the default value to infer the type. The doc of torch.jit advises to use a MyPy-style type annotations or type hints (like I did in param_sinc_fb.py) to force non Tensor types.
So it would be probably better to stick to one for consistency. @mpariente , do you have a preference between the two possible solutions?

yep, thats how I handled it in open-unmix now. The core (including the LSTM) is traceable, the wiener filter is scriptable.

And in the end, do you trace or script the model?

Torchscript assumes that every variable with undefined types are Tensors, then crashes at runtime when it discovers that it wasn't the case or when the type of the variable changed. This also includes function keyword arguments as it does not use the default value to infer the type. The doc of torch.jit advises to use a MyPy-style type annotations or type hints (like I did in param_sinc_fb.py) to force non Tensor types.

So it would be probably better to stick to one for consistency. @mpariente , do you have a preference between the two possible solutions?

I saw part of that, and adding typing to Asteroid's Modules wouldn't hurt.
But I still think, that both scripting and tracing can be mixed without problem, it's actually meant to be used this way, right?
I don't think it's about consistency, tracing is limited by essence but more flexible, and scripting is more general but also more rigid and torch gives us those two tools, I think we can use both of them.

I saw part of that, and adding typing to Asteroid's Modules wouldn't hurt.
I meant, would you prefer MyPy-style typing or Python type hints? Or you don't mind having a mix of both?

Oh ok sorry, didn't understand it this way.
I'll have a look now then

Python type hints please :wink:

Regarding tracing vs. scripting, since the idea is to use torch.jit.trace everywhere and to switch to script mode by decorating functions that need this mode with torch.jit.script, would it be alright to replace calls to torch.jit.script with calls to torch.jit.trace in jit_test.py?

Hey guys,
A bit late for the hackathon and I don't know if this feature is still planned but I have had another look and solved some of the issues (there only warnings left when tracing the models in the tests). Below is a summary of the issues that are left:

DPTNet expects the number of features to be a multiple of the number of heads in the multi attention layer but the __init__ method of the class gives no control to the n_heads argument of DPTransformer. This makes writing the tracing test for DPTNet with STFT rather cumbersome. A way around it would be the addition of the keyword argument n_heads to the __init__ of DPTNet, which would add another argument to a method that already has many...
When tracing a model, the jit compiler raises a warning anytime it encounters a branch in the code as only one of these branches will be taken by the traced model. In particular, all the checks written in the models are effectively bypassed

On the upside, properties do not seem problematic.

Didn't read the code in detail but that sounds great !
Which version of torch are you using btw?

On the upside, properties do not seem problematic.

Which properties in particular?

DPTNet expects the number of features to be a multiple of the number of heads in the multi attention layer but the __init__ method of the class gives no control to the n_heads argument of DPTransformer. This makes writing the tracing test for DPTNet with STFT rather cumbersome. A way around it would be the addition of the keyword argument n_heads to the __init__ of DPTNet, which would add another argument to a method that already has many...

I think it was an oversight, we should add n_heads in DPTNet. Many arguments to architectures is not a problem IMO.
Regarding the problem between number of features and n_head, I wonder if there isn't a Linear layer between the input and the masker. I'll have a look.

When tracing a model, the jit compiler raises a warning anytime it encounters a branch in the code as only one of these branches will be taken by the traced model. In particular, all the checks written in the models are effectively bypassed

Can you elaborate please?

Which version of torch are you using btw?

I am using torch 1.6.0. Are you using something else?

Which properties in particular?

I remember that jit.script had issues with class properties but I may be wrong here. When using trace, tracing a DeMask model, to which I added a shameless copy/paste sample_rate property, did not raise errors.

Can you elaborate please?

Example of warning:
tests/models/jit_test.py::test_trace_bss_model[LSTMTasNet]
  anaconda3/envs/asteroid/lib/python3.7/site-packages/torch/nn/modules/rnn.py:176: TracerWarning: Converting a
  tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python
  values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
    if self.input_size != input.size(-1):

I guess, the condition will always evaluate to the value that was found during tracing. Kind of ironic that the code that raises this warning is part of torch. But it also happens for some lines in asteroid too:
tests/models/jit_test.py::test_trace_bss_model[LSTMTasNet]
  asteroid/asteroid/filterbanks/enc_dec.py:172: TracerWarning: Converting a tensor to a Python boolean might
  cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as
  a constant in the future. This means that the trace might not generalize to other inputs!
    if channels == 1 and self.as_conv1d:

1.6.0 is fine, I was just asking.
Ok for the property. I guess it'll still break if there are graph computations inside it.
Thanks for the examples. We have to browse the code very carefully to find where the tracing isn't enough, and we need to script that part. There are graphs that'll stay static at run time (is there skip connection etc..) and others won't (number of input channels etc..).

Regarding the DPTNet, I re-checked, there is no Linear layer in the paper. We should add one if the input dim is not divisible by the number of heads, otherwise it's pretty useless. @mhu-coder, do you feel like doing it?

Regarding the DPTNet, I re-checked, there is no Linear layer in the paper. We should add one if the input dim is not divisible by the number of heads, otherwise it's pretty useless. @mhu-coder, do you feel like doing it?

Sure, will have a look in the afternoon.
FYI, I had a look at scripting the models instead of tracing them:

ConvTasNet was fine
DPRNNTasNet and LSTMTasNet: I ran into an issue in torch/nn/modules/rnn.py that read GeneratorExp aren't supported, which can be removed by converting a set comprehension into a regular loop but then ended up with a AttributeError caused by torch/jit/frontend.
DPTNet: some issues with parts of torch and asteroid but ultimately, torch.script does not support methods/functions that take callables as argument (cf https://discuss.pytorch.org/t/function-type-in-torchscript/65416). This is the case of some functions in asteroid/dsp/overlap_add.py.

Ok, what's left to do to merge this @mhu-coder ?

Just added the linear layers (one at the input, another at the output) to DPTransform when in_chan is not a multiple of n_heads.
DPTNet is still missing an argument for n_heads in its __init__ method.
Other than that, there is still the question of which model can be traced and which should be scripted instead. But I wonder if it wouldn't be better to give this responsibility the person that is going to trace/script a model.

Cool, thanks !
I actually thought that it was possible to decorate functions in nn.Module, but it's not for now (see language reference). My idea (before that) was to add a script decorator to all methods needing scripting, and that would have been satisfying. But that's not possible. So the trace method you propose is good for now.

Actually, if we make an external method out of forward, we can script automatically, which would be prettier. And for the Encoder/Decoder, it's possible.
Also, we'll use torch.jit.script_if_tracing instead of script so we avoid the start up time.

Finally scripted the forward by making it external. I guess we don't need the trace function anymore, can we remove it?

So, I merged master.
Tracing tests take quite a while. I'm wondering if it shouldn't be a different CI build altogether, so that we can really test thing rigorously. Thoughts?
Note to myself (assistance accepted): before merging, need to go through the whole masknn codebase to be sure that tracing is enough on all the modules that we don't script.

Finally scripted the forward by making it external. I guess we don't need the trace function anymore, can we remove it?

Neat! Sure, let's remove trace, no need to keep around lines that are unused

before merging, need to go through the whole masknn codebase to be sure that tracing is enough on all the modules that we don't script.

It's not as good as carefully reading the code of each mask but we can add tests to check that every model can handle input variable shapes (actually, I am facing some runtime errors but will try to fix as many as I can). This will increase the time jit_test.py takes to run though.

Just added a couple of tests to check that traced models produce the same output for n-D input tensors (with n in [1, ..., 4]). Is it normal that most of the original/non-traced models crash?

Just added a couple of tests to check that traced models produce the same output for n-D input tensors (with n in [1, ..., 4]). Is it normal that most of the original/non-traced models crash?

Yes. I don't see why not.
Conv1d expects a 3D input, and doesn't support 4d inputs, LSTM either etc..
Thanks but those tests can be removed.

I think I see what you mean.
BTW, I saw that DeMask could not handle 1-D input when traced with a 2-D input but I guess this comes from the way the forward is written and the whole class could be refactored anyway.
When a (batch, time) data (with batch > 1) is given to LSTMTasNet, there's a RuntimeError that is raised in the forward of the decoder:
E       RuntimeError: The following operation failed in the TorchScript interpreter.
E       Traceback of TorchScript (most recent call last):
E         File "asteroid/filterbanks/enc_dec.py", line 296, in multishape_conv_transpose1d
E               view_as = (-1,) + spec.shape[-2:]
E               out = F.conv_transpose1d(
E                   spec.view(view_as),
E                   ~~~~~~~~~ <--- HERE
E                   filters,
E                   stride=stride,
E       RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

I guess this could be the object of another pull request?

BTW, I saw that DeMask could not handle 1-D input when traced with a 2-D input but I guess this comes from the way the forward is written and the whole class could be refactored anyway.

I agree.

I guess this could be the object of another pull request?

This PR is already so big that we can use reshape instead, no problem.

yeah, you're right. It's time to finish this PR.
I have just reverted the res, res return in asteroid/masknn/convolutional.py to what it was before as the method output consistency in type is only needed for torch.jit.script. So, in other word, if we always trace/never script a model relying on Conv1DBlock or TDConvNet, the inconsistency is fine.
Is the thorough check of the modules in asteroid/masknn the only thing left for this PR?

I guess, yes.
Maybe we'll "fix" DeMask in another PR.

Not 100% sure

[x] masknn/norms
[x] masknn/activations
[x] masknn/attention
[x] TDConvNet, TDConvNetpp, Conv1dBlock, UConvBlock, UBlock, SuDORMRFImproved, SuDORMRF. All but # CHECK-JIT
[x] masknn/recurrent (missing   # CHECK-JIT)
[x] models: BaseModel, ConvTasNet, DPRNNTasNet, LSTMTasNet, DPTNet (+ added  # CHECK-JIT)
[x] filterbanks: all filterbanks + encoder + decoder.
[ ] filterbanks/transforms
[ ] dsp
[ ]

This PR won't assure tracability/jitability for the complex module, the DCCRN and DCUNet. We'll do that on a follow-up PR (Cc. @jonashaag). Most of it is tagged with # CHECK-JIT on the class.

Going back to norms, this doesn't work
from asteroid.masknn import norms

model = norms.GlobLN(10)
x = torch.randn(1, 10, 12)
traced = torch.jit.trace(model, x)
y = torch.randn(2, 10, 18, 12)
assert_allclose(traced(y), model(y))

If we script GlobLN instead, it's going to work.

script_if_tracing is not working on 1.6.0 because it returns a function, instead of calling the function.
So we'll make a patch for it. (Works on nightly.)

Reading for merging IMO.

Added SuDORMRF networks as well finally.
@mhu-coder @jonashaag @popcornell @faroit any feedback appreciated.


Broken Link on Chukinfo page

https://intellectualsites.gitbook.io/fastasyncworldedit/basic-commands/analysis

The link and keyboard button doesn't work.

Maybe already fixed with #75 .


Fixed Telegram support / added files deletion

-Fixed bot_token bug
-used variable for telegram id_group
-Added files deletion for uninstallation
openvpn-install.sh.txt

It's cool if you want to implement a specific feature which you need, but this will never be merged with the main branch.

Sorry dude, that was a bad move, just wanted to pull a forked branch ;)


Dynamic Host IP

Is their any way to dynamically assigning Host IP in swagger.yaml. Issue is if same swagger file is to be run on different hosts than their respective IP's have to be manually defined.
Need a way so that swagger can dynamically get Host IP.

Swag don't support this case yet. I can't thought for this case.

same case https://github.com/swaggo/swag/issues/360

How do you want to?
I thought this.
eg: swag init --host "localhost:8080"

@chiragg6 I found a solution:
if you use doc.json as the spec source, there is a hook in docs/doc.go(generated by swag init)
do not add host annotation, then the generated spec will leave a placeholder:
{
    "host": "{{.Host}}"
}

setup you dynamic host ip, before serve the swagger ui.
import (
    	"path/to/your/project/docs"
)

func init() {
        // this host will fill the placeholder.
    	docs.SwaggerInfo.Host = "your dynamic host"
}

I see.
Can you write this solution in README.md?
@webee

@pei0804
actually, this solution is in https://github.com/swaggo/swag#how-to-use-it-with-gin
        // programatically set swagger info
        // do not add corresponding declaritive comment to enable it
	docs.SwaggerInfo.Title = "Swagger Example API"
	docs.SwaggerInfo.Description = "This is a sample server Petstore server."
	docs.SwaggerInfo.Version = "1.0"
	docs.SwaggerInfo.Host = "petstore.swagger.io"
	docs.SwaggerInfo.BasePath = "/v2"

it just need to point out that do not add corresponding annotation.

@pei0804 @chiragg6
another solution:
you can start a standalone swagger-ui service(use docker, whatever) or just use https://petstore.swagger.io/
then you serve the swagger.json file, you can modify it anyway.
eg, use echo:
	e.GET("/docs/swagger.json", func(c echo.Context) error {
		f, err := os.Open(docFile)
		if err != nil {
			return err
		}
		defer f.Close()

		dec := json.NewDecoder(f)
		data := make(map[string]interface{}, 0)
		if err := dec.Decode(&data); err != nil {
			return err
		}
		data["host"] = host
		return c.JSON(http.StatusOK, data)
	})

then open https://petstore.swagger.io?url=http://you/swagger.json


Feature/notion

Modify the missing "$" in chapter_notation.

Job d2l-zh/PR-586/1 is complete.
Check the results at http://preview.d2l.ai/d2l-zh/PR-586/

Job d2l-zh/PR-586/2 is complete.
Check the results at http://preview.d2l.ai/d2l-zh/PR-586/

Can it merge?


prog: don't mutate mmap() calls too often

Due to https://github.com/google/syzkaller/issues/316 there're too many
mmap() calls in the programs, and syzkaller is spending quite a bit of
time mutating them. Most of the time changing mmap() calls won't give
us new coverage, so let's not do it too often.

LGTM


⚠ Bofrost.fr has degraded performance

In b636a2e, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 830 ms

Resolved: Bofrost.fr performance has improved in 3cd71a4 after 13 minutes.


Not installing the latest available major os update

I'm having an issue on intel / apple silicon (both compatible with Sonoma) where it appears 14.1.2 is detected, but 13.6.1 is installed as the major OS update.
My setup has 4.0.2 script deployed in a policy from Jamf with no parameters.
I have 3 config profiles with the recommended update and notification settings, along with the following settings;
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
 <dict>
   <key>InstallMacOSMajorUpgrades</key>
   <true/>
   <key>InstallMacOSMajorVersionTarget</key>
   <string>14</string>
   <key>InstallRapidSecurityResponses</key>
   <true/>
   <key>DeadlineDaysFocus</key>
   <string>3</string>
   <key>DeadlineDaysSoft</key>
   <string>10</string>
   <key>DeadlineDaysHard</key>
   <string>14</string>
   <key>DeferralTimerMenu</key>
   <string>15,60,240,1440</string>
   <key>DialogTimeoutInsufficientStorage</key>
   <string>600</string>
   <key>DialogTimeoutPowerRequired</key>
   <string>600</string>
 </dict>
</plist>

super.log
Thu Dec 07 16:52:50 super[5301]: Status: macOS update/upgrade workflows automatically authenticated via system account (root).
Thu Dec 07 16:52:50 super[5301]: **** S.U.P.E.R.M.A.N. 4.0.2 - CHECK SOFTWARE UPDATES/UPGRADE ****
Thu Dec 07 16:52:50 super[5301]: Status: No macOS beta program status cache, full update/upgrade check required.
Thu Dec 07 16:52:52 super[5301]: Status: Starting full check for all available system software...
Thu Dec 07 16:52:52 super[5301]: softwareupdate: Waiting for available Apple software updates...
Thu Dec 07 16:53:08 super[5301]: Status: A macOS minor update is available: macOS Ventura 13.6.1-22G313
Thu Dec 07 16:53:08 super[5301]: Status: 1 available non-system update(s).
Thu Dec 07 16:53:08 super[5301]: Status: Recommended (non-system) update 1: Safari17.1.2VenturaAuto-17.1.2
Thu Dec 07 16:53:08 super[5301]: mdmclient: Waiting for available macOS major upgrades...
Thu Dec 07 16:53:18 super[5301]: Status: A macOS major upgrade is available: macOS Sonoma 14.1.2
Thu Dec 07 16:53:19 super[5301]: Warning: Requested --install-macos-major-version-target=14 is greater than the currently available Apple software update version of 13.
Thu Dec 07 16:53:19 super[5301]: Status: A macOS major upgrade is available via Apple software update: macOS Ventura 13.6.1-22G313
Thu Dec 07 16:53:19 super[5301]: **** S.U.P.E.R.M.A.N. 4.0.2 - MACOS 13.6.1 MAJOR UPGRADE VIA SOFTWAREUPDATE ****
Thu Dec 07 16:53:19 super[5301]: softwareupdate: Starting macOS Ventura 13.6.1 download workflow, check /Library/Management/super/logs/asu-workflow.log for more detail.
Thu Dec 07 16:53:38 super[5301]: softwareupdate: macOS Ventura 13.6.1 is downloading...
Thu Dec 07 16:59:17 super[5301]: softwareupdate: macOS Ventura 13.6.1 download complete, now preparing...
Thu Dec 07 17:47:42 super[5301]: softwareupdate: macOS Ventura 13.6.1 download and preparation complete.
Thu Dec 07 17:47:44 super[5301]: Status: Setting new automatic zero date of: 2023-12-07:17:47.
Thu Dec 07 17:47:44 super[5301]: Status: Restarting all deadline counters.
Thu Dec 07 17:47:45 super[5301]: Status: Focus days deadline of 2023-12-10:17:47 (3 day(s) after 2023-12-07:17:47) NOT passed.
Thu Dec 07 17:47:45 super[5301]: Status: Soft days deadline of 2023-12-17:17:47 (10 day(s) after 2023-12-07:17:47) NOT passed.
Thu Dec 07 17:47:45 super[5301]: Status: Hard days deadline of 2023-12-21:17:47 (14 day(s) after 2023-12-07:17:47) NOT passed.
Thu Dec 07 17:47:45 super[5301]: IBM Notifier: Restart or defer dialog with no timeout.
Fri Dec 08 08:38:46 super[5301]: Status: User chose to restart now.
Fri Dec 08 08:38:46 super[5301]: IBM Notifier: Restart default notification.
Fri Dec 08 08:38:46 super[5301]: softwareupdate: Starting macOS Ventura 13.6.1-22G313 upgrade workflow, check /Library/Management/super/logs/asu-workflow.log for more detail.
Fri Dec 08 08:39:04 super[5301]: softwareupdate: macOS Ventura 13.6.1 is downloading...
Fri Dec 08 08:39:05 super[5301]: softwareupdate: macOS update/upgrade is prepared and ready for restart!
Fri Dec 08 08:39:05 super[5301]: Exit: Full super workflow complete! macOS update/upgrade restart is imminent and the super restart validation workflow is scheduled to automatically relaunch at next startup.
Fri Dec 08 08:39:05 super[5301]: **** S.U.P.E.R.M.A.N. 4.0.2 - CLEAN EXIT ****

Issue resolved by removing update deferrals.


Horizontal Scaling

We are currently using Certbot to create/update certificates whenever a new server is added to the cluster with the configuration shared between all the servers via an NFS. Because Certbot uses file locks, this architecture mostly works, though we've had some problems making sure edge cases are handled correctly.
How will micronaut-acme behave if it is horizontally scaled?

Perhaps the certificates—and private keys, for that matter—could be stored in AWS Systems Manager Parameter Store or whatever the equivalent service is for Azure/Google Cloud.

Please post questions in StackOverflow.
You can also try with asking in Gitter.

@RyanHoldren I missed this originally sorry about that..... the way it works right now is each server would end up requesting a new certificate which should work with in theory with no issues. Assuming you are using Lets Encrypt you can see previous responses here about doing so but main issues with this approach are 2 things. 1. would be rate limiting and 2. would be the authorizations process. (https://community.letsencrypt.org/t/aws-distributing-current-certbot-certificates-amongst-server-which-are-spun-up-with-autoscale/35739/2). So in theory you could randomize the acme.renew-within property to have them renew on different intervals to skirt the ratelimiting if you have that many servers.
But to the issue of the authorization step that would have issues if you selected http/tls since those would need to get directed to whichever server actually made the call since there is a token involved at the time of order. There is a chance it could work but unlikely due to you more than likely having a load balancer in place.
That leaves you with DNS being you only option but ATM DNS option requires manual intervention until #3 #4 #5 (or whatever DNS provider you are using that might have an API) are done and it just happens automatically.
And maybe there are other routing options or fanciness you could do to get the authorizations to work out.... id be curious to hear more about how you have certbot running on multiple servers requesting renewals or maybe you dont and you just have a "master certbot server" that does all the magic and then NFS to share the cert.
Also as far as storing the certificate in AWS Systems Manager Parameter Store or whatever the equivalent service is for Azure/Google Cloud. Not sure it solves the issue. The biggest hangup i think is the authorization process. But maybe again you just have 1 node do it, share to a storage provider and then all the other nodes check for changes on an interval. ¯_(ツ)_/¯  #spitballin. But now your down to 1 node and no redundancy.


Move non-label fields to min X (fix #983)

Resolves
#983
Proposed Changes
Before:

After:

Reason for Changes
paying penance and finishing my unfinished business with @carljbowman
Test Coverage
none, but block_render_svg is pretty light in this regard

\o/ Woo! Thanks @tmickel


Add placeholder prop

Props are merged on the TextInput so you can just pass the placeholder color directly :)


brings iterator machinery to spec, per P1878

P1878 highlights a number of problems with the readable concept,
incrementable_traits, and readable_traits; and proposed changes
necessary to fix it. The paper also prefixes the names readable,
writable, and readable_traits, with indirectly_.
Closes #26.

@CaseyCarter @ericniebler, I would appreciate your input on this one.


Missing Error Message for wrong value in OPERATOR_VULNERABILITY_SCANNER_REPORT_TTL

What steps did you take and what happened:
I ran starboard operator with the new TTL feature with  OPERATOR_VULNERABILITY_SCANNER_REPORT_TTL="7d". This makes the operator stop immediately without any message and with return code 0. So I had no clue what actually went wrong.
I was only able to fix this by try and error and fixed it with OPERATOR_VULNERABILITY_SCANNER_REPORT_TTL="168h"
I can recreate this simply with docker:
❯ docker run -it -e OPERATOR_VULNERABILITY_SCANNER_REPORT_TTL=7d  aquasec/starboard-operator:0.14.0
❯ echo $?
0
❯ docker run -it -e OPERATOR_VULNERABILITY_SCANNER_REPORT_TTL=1h aquasec/starboard-operator:0.14.0
{"level":"info","ts":1643112968.92483,"logger":"main","msg":"Starting operator","buildInfo":{"Version":"0.14.0","Commit":"4aa2a11c0c5e6362c1ce04d736ec2fd1997bdaf8","Date":"2022-01-20T11:03:24Z","Executable":""}}
{"level":"error","ts":1643112968.9249914,"logger":"main","msg":"Unable to run starboard operator","error":"resolving install mode: OPERATOR_NAMESPACE must be set"}

What did you expect to happen:
I would expect a proper error message, just like the one above for the missing OPERATOR_NAMESPACE.
Anything else you would like to add:
Environment:

Starboard version (use starboard version): 0.14.0 (as seen above)
Kubernetes version (use kubectl version):  1.22.

Thank you for reporting this issue @shaardie I think the root cause is that we are trying to use setupLog in case of configuration parser failure:
if err := run(); err != nil {
	setupLog.Error(err, "Unable to run starboard operator")
}

However, the setupLog hasn't been initialised, because we initialise in only after the configuration is parsed:
log.SetLogger(zap.New(zap.UseDevMode(operatorConfig.LogDevMode)))

Thanks to fix this so fast :+1:


Calling .collate() on a layout deletes its label

Hello,
Running this code produces a layout without a label. Is this normal?
import numpy as np
import holoviews as hv
from holoviews import opts

hv.extension('bokeh')

x = np.random.normal(0, 1, 10000)
y = np.random.normal(0, 1, 10000)
data1 = {'x' : x, 'y' : y}
data2 = {'x' : x+2, 'y' : y*2}

sctr1 = hv.Scatter(data1).opts(tools=['hover'])
sctr2 = hv.Scatter(data2).opts(tools=['hover'])
ovr = hv.Overlay([sctr1, sctr2], label='Overlay')

ovr.collate()

I know that in this case there is no need to use collate(), but the point is that simply calling ovr will work and will produce a plot with a label, so it's clear that collate() removes the label.
Is this intended? Am i doing something wrong?
Thank you.

Could you elaborate on your use case? collate does not specifically delete anything, it just creates a new Overlay:
def collate(self):
    """
    Collates any objects in the Overlay resolving any issues
    the recommended nesting structure.
    """
    return reduce(lambda x,y: x*y, self.values())

I'm trying to overlay a datashaded image and a decimated scatter plot to get some hover info out of the plot. This is then part of a layout. In essence, something like:
hv.Layout([ovr, hv.Overlay([datashade(sctr2), sctr1])])
Yes, there is probably a correct way of doing that. But the point still stands: Why is .collate()  removing the label when it's called?


update flat field calculation code to account for random noise in master bias

master bias is currently noisy because it's only based on the four available bias frames...

This doesn't just impact the flat field calculation. For example, it also increases uncertainties on the bias-subtracted pixel values during reductions.

This has become too outdated.


Improve Donation creation form @wip

Addresses #158
I wasn't able to claim the issue without contributor status, so I'm opening this PR as a placeholder.
More details to come!

@laurenball Awesome, you rule! Just post here again when you're ready to be pulled in :)

Hey @laurenball we recognize that this issue requires a bit of creative thinking as the there are just some recommendations/ideas from the stakeholder at the diaper bank. We're definitely happy to talk this through and creative problem solve with you if that would help. If you join our slack (grab an invite at https://rubyforgood.herokuapp.com) you can find us in the #diaper channel.

hey @seanmarcia! I might end up joining there so I can run some ideas past you guys. I've been so quiet on this because I just happened to get a project at work (also a rails dev full time) that requires some non-standard form creation, so I'm working on that to get a feel for what I should do here. (I'm leaning towards making a DonationForm model, if you have initial thoughts on that)
I'm hoping to spend some time on this in the next week and will probably come in and share some thoughts/screenshots while I'm doing this.
I'm glad you checked in because I was worried you guys thought I'd abandoned this. I'll be in touch soonish :)

@laurenball Awesome! Just post here when you feel your PR is ready for review (or request a review from @seanmarcia or I :) )

Hey @laurenball its been over 2 months since an update. We're going to go ahead an close this PR and re-open the issue. Get in touch if you are still interested in working on this and we'll happily re-open the PR!