Nginx Restart with loaded module cause segfaults (maybe problem for ModSec v3 and not connector itself)

Hi Devs, i noticed that latest master of connector (maybe issue not in connector exactly) causing nginx segfaults during its restart.
Reload or Stop then Start do not cause such segfaults.
Also by default files for shared collection dropped to root folder '/'
ls -l / | grep shared
-rw-r--r--   1 root root 1048576 Jan 13 17:03 modsec-shared-collections
-rw-r--r--   1 root root    8192 Jan 13 17:05 modsec-shared-collections-lock

But, during restart Nginx with init/init.d scripts those files created in directory where cli command was used.
Jan 13 17:03:07 abyss kernel: [15515838.077384] nginx[7190]: segfault at 7f9548807000 ip 00007fbe74c5b9da sp 00007ffcde6b9208 error 4
Jan 13 17:03:07 abyss kernel: [15515838.077395] nginx[7192]: segfault at 7f9548807000 ip 00007fbe74c5b9da sp 00007ffcde6b9208 error 4
Jan 13 17:03:07 abyss kernel: [15515838.077398] nginx[7186]: segfault at 7f9548807000 ip 00007fbe74c5b9da sp 00007ffcde6b9208 error 4
Jan 13 17:03:07 abyss kernel: [15515838.077400] nginx[7191]: segfault at 7f9548807000 ip 00007fbe74c5b9da sp 00007ffcde6b9208 error 4
Jan 13 17:03:07 abyss kernel: [15515838.077401]  in libc-2.19.so[7fbe74bd3000+1ba000]
Jan 13 17:03:07 abyss kernel: [15515838.077402] nginx[7188]: segfault at 7f9548807000 ip 00007fbe74c5b9da sp 00007ffcde6b9208 error 4
Jan 13 17:03:07 abyss kernel: [15515838.077403]  in libc-2.19.so[7fbe74bd3000+1ba000]
Jan 13 17:03:07 abyss kernel: [15515838.077404]
Jan 13 17:03:07 abyss kernel: [15515838.077405]  in libc-2.19.so[7fbe74bd3000+1ba000]
Jan 13 17:03:07 abyss kernel: [15515838.077405]
Jan 13 17:03:07 abyss kernel: [15515838.077406]
Jan 13 17:03:07 abyss kernel: [15515838.077407]  in libc-2.19.so[7fbe74bd3000+1ba000]
Jan 13 17:03:07 abyss kernel: [15515838.077414]  in libc-2.19.so[7fbe74bd3000+1ba000]
Jan 13 17:03:07 abyss kernel: [15515838.080151] nginx[7185]: segfault at 7f9548807000 ip 00007fbe74c5b9da sp 00007ffcde6b92e8 error 4 in libc-2.19.so[7fbe74bd3000+1ba000]

Jan 13 17:03:24 abyss kernel: [15515854.689932] nginx[7689]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954ecc8 error 4
Jan 13 17:03:24 abyss kernel: [15515854.689934] nginx[7687]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954ecc8 error 4 in libc-2.19.so[7fea863e5000+1ba000]
Jan 13 17:03:24 abyss kernel: [15515854.689936]  in libc-2.19.so[7fea863e5000+1ba000]
Jan 13 17:03:24 abyss kernel: [15515854.689960] nginx[7686]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954ecc8 error 4 in libc-2.19.so[7fea863e5000+1ba000]
Jan 13 17:03:24 abyss kernel: [15515854.689978] nginx[7690]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954ecc8 error 4 in libc-2.19.so[7fea863e5000+1ba000]
Jan 13 17:03:24 abyss kernel: [15515854.689983] nginx[7691]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954ecc8 error 4 in libc-2.19.so[7fea863e5000+1ba000]
Jan 13 17:03:24 abyss kernel: [15515854.690068] nginx[7688]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954ecc8 error 4 in libc-2.19.so[7fea863e5000+1ba000]
Jan 13 17:03:24 abyss kernel: [15515854.692350] nginx[7685]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954eda8 error 4 in libc-2.19.so[7fea863e5000+1ba000]

Hello @md2k, could you please provide full nginx configuration (in recent versions it can be obtained by running nginx -T), full modsecurity configuration, and backtraces?
Thanks.

I can semi-confirm this bug! Have already posted it in the modsec_dev list.
And also after updating the source from 12.12.16 - 27.01.17 the shared-collections are in /
root@waf-1-a-02:~# ls -la /etc/nginx/modsec/
-rw-r--r--  1 root root 1048576 Jan 27 13:59 modsec-shared-collections
-rw-r--r--  1 root root    8192 Jan 27 14:00 modsec-shared-collections-lock
root@waf-1-a-02:~# ls -la /
-rw-r--r--   1 root root 1048576 Jan 27 14:34 modsec-shared-collections
-rw-r--r--   1 root root    8192 Jan 27 14:34 modsec-shared-collections-lock
I'm running latest N+ and build MS3 from source, not the N+ module.

I could also send you a nginx -T directly if you like cause it's very huge and sensible

@defanator  i can send it, but it also big for me and and obfuscation will take some time (for nginx).

@md2k we can start from backtraces then.

@mimugmail if you're using N+, I would suggest to address this via N+ support channel. You can share backtraces here though.

No i not use N+, too pricy for own use :)
It will be helpful if you can tell me how i can get backtrace from modsec/nginx

@defanator Since I'm not using the commercial WAF module I don't think that we can expect much support from N+, also I don't want to keep these guys rotating (was a promise to OwenGarret@Nginx) :)
I'll try to make a detailed bug report

@md2k you need to obtain core file and then use GDB to get output of the full bt command. These links could be useful for further details:
https://www.nginx.com/resources/admin-guide/debug/
https://github.com/spiderlabs/modsecurity/tree/v3/master#debugging

Thank, will check them later, going to prepare virtual box for this to not mess with my production server

OS: Debian 8
Nginx: 1.11.9
MS3 source: 04.02.2017
Nginx connector: 04.02.2017
Backtrace full:
root@nginx:~# gdb /opt/nginx/sbin/nginx /tmp/core
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /opt/nginx/sbin/nginx...done.
[New LWP 16984]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `nginx: worker process'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  strlen () at ../sysdeps/x86_64/strlen.S:106
106     ../sysdeps/x86_64/strlen.S: No such file or directory.
(gdb) backtrace full
#0  strlen () at ../sysdeps/x86_64/strlen.S:106
No locals.
#1  0x00007fe95e437c28 in std::string::compare(char const*) const () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#2  0x00007fe95f5d36cc in operator==<char, std::char_traits, std::allocator > (__rhs="/var/log/modsec_audit.log", __lhs=)
at /usr/include/c++/4.9/bits/basic_string.h:2528
No locals.
#3  modsecurity::utils::SharedFiles::find_handler (this=this@entry=0x7fe95f857ce8 modsecurity::utils::SharedFiles::getInstance()::instance,
fileName="/var/log/modsec_audit.log") at utils/shared_files.cc:42
current = 0x7fe960912000
#4  0x00007fe95f5d3710 in modsecurity::utils::SharedFiles::close (this=0x7fe95f857ce8 modsecurity::utils::SharedFiles::getInstance()::instance, fileName=...)
at utils/shared_files.cc:181
a =
#5  0x00007fe95f56d119 in modsecurity::audit_log::writer::Serial::~Serial (this=0x2443220, __in_chrg=) at audit_log/writer/serial.cc:28
No locals.
#6  0x00007fe95f56b148 in modsecurity::audit_log::AuditLog::~AuditLog (this=0x2443100, __in_chrg=) at audit_log/audit_log.cc:69
No locals.
#7  0x00007fe95f56fc16 in ~RulesProperties (this=0x24471a0, __in_chrg=) at ../headers/modsecurity/rules_properties.h:106
No locals.
#8  modsecurity::Rules::~Rules (this=0x24471a0, __in_chrg=) at rules.cc:80
No locals.
#9  0x00007fe95f56ff0e in modsecurity::msc_rules_cleanup (rules=0x24471a0) at rules.cc:335
No locals.
#10 0x00007fe95f85a4d4 in ngx_http_modsecurity_config_cleanup (data=0x2454298) at /opt/ModSecurity-nginx/src/ngx_http_modsecurity_module.c:595
t = 0x2454298
#11 0x000000000040df31 in ngx_destroy_pool (pool=0x23e6d30) at src/core/ngx_palloc.c:57
p =
n =
l =
c = 0x24542f8
#12 0x00000000004330a0 in ngx_worker_process_exit (cycle=cycle@entry=0x23e6d80) at src/os/unix/ngx_process_cycle.c:1001
i =
c =
#13 0x00000000004331af in ngx_worker_process_cycle (cycle=cycle@entry=0x23e6d80, data=data@entry=0x0) at src/os/unix/ngx_process_cycle.c:758
worker = 0
#14 0x000000000043185a in ngx_spawn_process (cycle=cycle@entry=0x23e6d80, proc=proc@entry=0x4330d8 <ngx_worker_process_cycle>, data=data@entry=0x0,
---Type  to continue, or q  to quit---
name=name@entry=0x48ad7e "worker process", respawn=respawn@entry=-3) at src/os/unix/ngx_process.c:198
on = 1
pid = 0
s = 0
#15 0x000000000043241d in ngx_start_worker_processes (cycle=cycle@entry=0x23e6d80, n=1, type=type@entry=-3) at src/os/unix/ngx_process_cycle.c:358
i = 0
ch = {command = 1, pid = 0, slot = 0, fd = 0}
#16 0x0000000000433a7b in ngx_master_process_cycle (cycle=cycle@entry=0x23e6d80) at src/os/unix/ngx_process_cycle.c:130
title = 0x2458ab4 "master process /opt/nginx/sbin/nginx"
p =
size =
i =
n =
sigio =
set = {__val = {0 <repeats 16 times>}}
itv = {it_interval = {tv_sec = 38111800, tv_usec = 0}, it_value = {tv_sec = 0, tv_usec = 0}}
live =
delay =
ls =
ccf = 0x23e8878
#17 0x000000000040c7b7 in main (argc=, argv=) at src/core/nginx.c:368
b =
log = 0x6b9880 <ngx_log>
i =
cycle = 0x23e6d80
init_cycle = {conf_ctx = 0x0, pool = 0x23e6920, log = 0x6b9880 <ngx_log>, new_log = {log_level = 0, file = 0x0, connection = 0, disk_full_time = 0,
handler = 0x0, data = 0x0, writer = 0x0, wdata = 0x0, action = 0x0, next = 0x0}, log_use_stderr = 0, files = 0x0, free_connections = 0x0,
free_connection_n = 0, modules = 0x0, modules_n = 0, modules_used = 0, reusable_connections_queue = {prev = 0x0, next = 0x0}, reusable_connections_n = 0,
listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, paths = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, config_dump = {
elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, config_dump_rbtree = {root = 0x0, sentinel = 0x0, insert = 0x0}, config_dump_sentinel = {
key = 0, left = 0x0, right = 0x0, parent = 0x0, color = 0 '\000', data = 0 '\000'}, open_files = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0},
size = 0, nalloc = 0, pool = 0x0}, shared_memory = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0},
connection_n = 0, files_n = 0, connections = 0x0, read_events = 0x0, write_events = 0x0, old_cycle = 0x0, conf_file = {len = 27, data = 0x23e6970 ""},
conf_param = {len = 0, data = 0x0}, conf_prefix = {len = 17, data = 0x23e6970 ""}, prefix = {len = 12, data = 0x485bb0 "/opt/nginx//"}, lock_file = {
len = 0, data = 0x0}, hostname = {len = 0, data = 0x0}}
cd =
---Type  to continue, or q  to quit---
ccf =
nginx -T:
root@nginx:~# /opt/nginx/sbin/nginx -T
nginx: the configuration file /opt/nginx//conf/nginx.conf syntax is ok
nginx: configuration file /opt/nginx//conf/nginx.conf test is successful
configuration file /opt/nginx//conf/nginx.conf:
#user  nobody;
worker_processes  1;
worker_rlimit_core  500M;
working_directory   /tmp/;
#error_log  logs/error.log;
#error_log  logs/error.log  notice;
error_log  logs/error.log  debug;
pid        logs/nginx.pid;
load_module "modules/ngx_http_modsecurity_module.so";
events {
worker_connections  1024;
}
http {
include       mime.types;
default_type  application/octet-stream;
#log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
#                  '$status $body_bytes_sent "$http_referer" '
#                  '"$http_user_agent" "$http_x_forwarded_for"';

access_log  logs/access.log;

sendfile        on;
#tcp_nopush     on;

#keepalive_timeout  0;
keepalive_timeout  65;

#gzip  on;

server {
    listen       80;
    server_name  localhost;

modsecurity on;
modsecurity_rules_file /opt/nginx/modsec/main.conf;

    #charset koi8-r;

    #access_log  logs/host.access.log  main;

    location / {
        root   html;
        index  index.html index.htm;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }

    # proxy the PHP scripts to Apache listening on <IP_ADDRESS>:80
    #
    #location ~ \.php$ {
    #    proxy_pass   http://<IP_ADDRESS>;
    #}

    # pass the PHP scripts to FastCGI server listening on <IP_ADDRESS>:9000
    #
    #location ~ \.php$ {
    #    root           html;
    #    fastcgi_pass   <IP_ADDRESS>:9000;
    #    fastcgi_index  index.php;
    #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
    #    include        fastcgi_params;
    #}

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    #location ~ /\.ht {
    #    deny  all;
    #}
}

# another virtual host using mix of IP-, name-, and port-based configuration
#
#server {
#    listen       8000;
#    listen       somename:8080;
#    server_name  somename  alias  another.alias;

#    location / {
#        root   html;
#        index  index.html index.htm;
#    }
#}

# HTTPS server
#
#server {
#    listen       443 ssl;
#    server_name  localhost;

#    ssl_certificate      cert.pem;
#    ssl_certificate_key  cert.key;

#    ssl_session_cache    shared:SSL:1m;
#    ssl_session_timeout  5m;

#    ssl_ciphers  HIGH:!aNULL:!MD5;
#    ssl_prefer_server_ciphers  on;

#    location / {
#        root   html;
#        index  index.html index.htm;
#    }
#}

}
configuration file /opt/nginx//conf/mime.types:
types {
text/html                             html htm shtml;
text/css                              css;
text/xml                              xml;
image/gif                             gif;
image/jpeg                            jpeg jpg;
application/javascript                js;
application/atom+xml                  atom;
application/rss+xml                   rss;
text/mathml                           mml;
text/plain                            txt;
text/vnd.sun.j2me.app-descriptor      jad;
text/vnd.wap.wml                      wml;
text/x-component                      htc;

image/png                             png;
image/tiff                            tif tiff;
image/vnd.wap.wbmp                    wbmp;
image/x-icon                          ico;
image/x-jng                           jng;
image/x-ms-bmp                        bmp;
image/svg+xml                         svg svgz;
image/webp                            webp;

application/font-woff                 woff;
application/java-archive              jar war ear;
application/json                      json;
application/mac-binhex40              hqx;
application/msword                    doc;
application/pdf                       pdf;
application/postscript                ps eps ai;
application/rtf                       rtf;
application/vnd.apple.mpegurl         m3u8;
application/vnd.ms-excel              xls;
application/vnd.ms-fontobject         eot;
application/vnd.ms-powerpoint         ppt;
application/vnd.wap.wmlc              wmlc;
application/vnd.google-earth.kml+xml  kml;
application/vnd.google-earth.kmz      kmz;
application/x-7z-compressed           7z;
application/x-cocoa                   cco;
application/x-java-archive-diff       jardiff;
application/x-java-jnlp-file          jnlp;
application/x-makeself                run;
application/x-perl                    pl pm;
application/x-pilot                   prc pdb;
application/x-rar-compressed          rar;
application/x-redhat-package-manager  rpm;
application/x-sea                     sea;
application/x-shockwave-flash         swf;
application/x-stuffit                 sit;
application/x-tcl                     tcl tk;
application/x-x509-ca-cert            der pem crt;
application/x-xpinstall               xpi;
application/xhtml+xml                 xhtml;
application/xspf+xml                  xspf;
application/zip                       zip;

application/octet-stream              bin exe dll;
application/octet-stream              deb;
application/octet-stream              dmg;
application/octet-stream              iso img;
application/octet-stream              msi msp msm;

application/vnd.openxmlformats-officedocument.wordprocessingml.document    docx;
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet          xlsx;
application/vnd.openxmlformats-officedocument.presentationml.presentation  pptx;

audio/midi                            mid midi kar;
audio/mpeg                            mp3;
audio/ogg                             ogg;
audio/x-m4a                           m4a;
audio/x-realaudio                     ra;

video/3gpp                            3gpp 3gp;
video/mp2t                            ts;
video/mp4                             mp4;
video/mpeg                            mpeg mpg;
video/quicktime                       mov;
video/webm                            webm;
video/x-flv                           flv;
video/x-m4v                           m4v;
video/x-mng                           mng;
video/x-ms-asf                        asx asf;
video/x-ms-wmv                        wmv;
video/x-msvideo                       avi;

}
This is just a development machine, non-productive!

This is my last one ... also installed it with CentOS7 and also got the segfault.

@mimugmail, thanks for the backtrace!
It seems like the issue you're observing is related to a number of other issues we've been also facing. Latest attempt to provide a fix was made here: https://github.com/SpiderLabs/ModSecurity/pull/1306 (though there's a number of questions to that PR).
Tagging @zimmerle here - this is related to the libmodsecurity itself, not just the connector.
@md2k, were you able to grab a core and obtain a backtrace from it?

@mimugmail thanks.
@defanator most probably i will have time to deal with it next week due my main project workload.

As @zimmerle told me to disable lmdb in another issue I tried it on this one, but --without-lmdb I also get a segfault here.

Hi @defanator , as you asked, in Gist output of gdb from core file after process crashed during nginx restart operation.
https://gist.github.com/md2k/4e18cc10649601bb93eed6a17bffc106

Thanks @md2k.

@defanator  i added 2 more backtraces to same gist as Another GDB (modsec compiled there with CFLAGS=-g -O0)

Hi,
The problem was related to shared memory. The first design was meant to be used by forked processes. In the forked process, the address space and file descriptor will be common in every forked-process (worker). The reload (in that case ./nginx -t) creates a new process, leading to different file descriptors and consequently a segfault. The segfault seems to be resolved, although I have faced `zombie' processes while stressing the implementation.
Do you guys mind to test? Those are the commits:
SharedFile class:

https://github.com/SpiderLabs/ModSecurity/commit/37bf69356db438267fdf671d86944ffae660cb7c
https://github.com/SpiderLabs/ModSecurity/commit/55d28aff9835dae00b236e899c772a376c6b2479

To use the SharedFile class within the DebugLogs:

https://github.com/SpiderLabs/ModSecurity/commit/1225076d3a81732ec7ba6ed553335387d4110024

The are available on the branch dev/parser:

https://github.com/SpiderLabs/ModSecurity/tree/v3/dev/parser

Thank you guys, and sorry for the huge delay.

Thanks @zimmerle for fixing this! I can confirm that with dev parser branch I can cleanly compile and reload the process. BUT, now when I start nginx it looks like this:
root@nginx:~# /opt/nginx/sbin/nginx
**********************************************************************************root@nginx:~#
Also when reloading there are these asterisks.
When I disable ModSec they are gone.
The performance like in the other issue is also not better than before, will post my results there.
Thank you!

Hello,
Same that @mimugmail with the last release (thanks for the update @zimmerle ) on check of status of nginx
Feb 24 12:29:59 ip-10-65-3-219 systemd[1]: Starting A high performance web server and a reverse proxy server...
Feb 24 12:30:03 ip-10-65-3-219 nginx[1062]: ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Feb 24 12:30:03 ip-10-65-3-219 nginx[1062]: **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Feb 24 12:30:07 ip-10-65-3-219 nginx[1067]: ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Feb 24 12:30:07 ip-10-65-3-219 systemd[1]: Started A high performance web server and a reverse proxy server.
Feb 24 12:30:07 ip-10-65-3-219 nginx[1067]: **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

@zimmerle I've just tested libmodsecurity from the current head of v3/dev/parser, and can confirm that nginx is not segfaulting anymore.
Then I realized that I've been running nginx with older connector module, built with libmodsecurity v3/master. I tried to rebuild it with v3/dev/parser code and got the following:
cc -c -fPIC -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include -I objs -I src/http -I src/http/modules \
    -o objs/addon/src/ngx_http_modsecurity_module.o \
    ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c
../ModSecurity-nginx/src/ngx_http_modsecurity_module.c: In function 'ngx_http_modsecurity_create_main_conf':
../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:473:34: error: passing argument 2 of 'msc_set_log_cb' from incompatible pointer type [-Werror=incompatible-pointer-types]
     msc_set_log_cb(conf->modsec, ngx_http_modsecurity_log);
                                  ^~~~~~~~~~~~~~~~~~~~~~~~
In file included from ../ModSecurity-nginx/src/ngx_http_modsecurity_common.h:25:0,
                 from ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:21:
/usr/include/modsecurity/modsecurity.h:325:6: note: expected 'ModSecLogCb {aka void (*)(void *, const void *)}' but argument is of type 'void (*)(void *, const char *)'
 void msc_set_log_cb(ModSecurity *msc, ModSecLogCb cb);
      ^~~~~~~~~~~~~~
cc1: all warnings being treated as errors

Are there any plans to merge v3/dev/parser to v3/master, or cherry pick a set of changes affecting the logging part, so we could adjust nginx connector code here?

v3/dev/parser is now part of v3/master ;)
Thank you for the reports ;)

I got this error too
../ModSecurity-nginx/src/ngx_http_modsecurity_module.c: In function 'ngx_http_modsecurity_create_main_conf':
../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:426:5: error: passing argument 2 of 'msc_set_log_cb' from incompatible pointer type [-Werror]
     msc_set_log_cb(conf->modsec, ngx_http_modsecurity_log);

What I am doing wrong or how do I fix it? Thanks.

Hi @hernandanielg,
Please upgrade both: ModSecurity and ModSecurity-nginx connector

https://github.com/SpiderLabs/ModSecurity/tree/v3/master
https://github.com/SpiderLabs/ModSecurity-nginx

Worked like a charm :) thanks!

I am having this issue now; details here:
https://github.com/SpiderLabs/ModSecurity/issues/1318#issuecomment-295539544
any thoughts?


release a 1.0.0 version

[ ] make sure the changelog is up to date
[ ] setup a release job on github
[ ] hope the CI of the people who use the plugin wont break :crossed_fingers:

@herrboyer do you know when version 1.0.0 would be released?

I just noticed that the last version still point to gitlab for the homepage but the setup.cfg is up to date here so releasing would fix this (minor) issue.


dataDecoded=null

Describe the bug
call cgw url , dataDecoded=null
To Reproduce
Steps to reproduce the behavior:

Call url ip:port/v1/chains/<chain_id>/transactions/<transaction_id>

Provide json you are submitting to the service (if it applies)

Links to issues in other repos (if possible)
https://github.com/safe-global/safe-infrastructure/issues/77

Expected behavior
A clear and concise description of what you expected to happen.
I queried through txs api/v1/data-decoder/api that dataDecoded is valuable.I hope to find out through the CGW URL that it also has a value.

Environment (please complete the following information):

Browser [ chrome]
Version [ 3.51.0 ]

Additional context
Add any other context about the problem here.
cgw.env FEATURE_FLAG_NESTED_DECODING=true; is set

Closing this as it's duplicated by https://github.com/safe-global/safe-infrastructure/issues/77, we can continue there 🙂


Fix 1556 Toggle events fire before toggled is changed

Fixes #1556
Toggle events change, keyup and click will stay with the old value instead of the new when firing. By changing the order of  execution of change and keyup, it seems to fix the problem. click stayed the same because the user may wants to execute something before the toggled change status.

Thank you. Fixed in v0.70.13.


support appending boxes without access to source code (#47)

Thanks for opening this PR @boyvinall
This PR adds a lot of changes unrelated to the summary. I'm going through the PR to cherry pick some small stuff (typo fixes, etc.). Would you be willing to afterwards rebase this PR on master, and leave out unrelated changes such as dep and editor config files?


Update performance_guide.md

Can one of the admins verify this patch?

Jenkins, test this please.

fixes #7551


50% of fees redirected to a third party wallet

Hey,
I'm looking at your deployed contract at https://basescan.org/address/0x70d4511b04eb995d00066af35416a9ababa27e81
and it looks like the party DAO takes 50% of eth from LP fees.
This was not disclosed in the docs or any other source I could find.
Can you explain the fee structure and reasoning behind it?

This is a parameter specified in the constructor of the FeeCollector. https://github.com/PartyDAO/erc20-creator/blob/a24465781ce89aa2edec8791ceb29517ce56d0ea/src/FeeCollector.sol#L40-L45
The fee is disclosed on the frontend.


Implement renameCollection for SQLite

What should be done?
SQLite backend and handler should support renameCollection command.
Collection renaming should not rename the underlying table; only the metadata should be updated.
CollectionCreate method should also be updated to treat a hash as a number and increment it until the selected table name is available.
Where?
See references to this issue.
Definition of Done

handler updated;
integration/compatibility tests enabled and pass;
no references to this issue are left.

Hi @AlekSi,
Can I work on this one? :smiley:

@kropidlowsky Sure! Please start with a separate small PR for the CollectionCreate method

I'm starting to work on this


Use GitHub as source

Instead of querying Package Catalog, just use the GitHub API to search for swiftlang repos that include a Package.swift. Would be cool to explore how well this works out.

After some playing around I came up with this GraphQL query for GitHub's API.
{
  rateLimit {
    remaining
    resetAt
  }
  search(query: "promisekit", type: REPOSITORY, first: 100) {
    repositoryCount
    edges {
      node {
        ... on Repository {
          name
          nameWithOwner
          description
          url
          isFork
          isPrivate
          pushedAt
          languages(first: 3) {
            edges {
              node {
                name
              }
            }
          }
          object(expression: "master:Package.swift") {
            ... on Blob {
              abbreviatedOid
            }
          }
        }
      }
    }
  }
}

The problem here is obviously that the query will return quite a lot of non-swift repos and since the list will have to be filtered client-side for the existence of a Package.swift id, it will also only work for a branch called master. Definitely not ideal. But it's a start.

closed with #15


search_dates not recognizing dd.mm.yyyy

search_dates('test 13.07.2016 test')
output:
[('2016', datetime.datetime(2016, 2, 1, 0, 0))]
How to make search_dates recognize this format?
How to pass some date formats to search_dates?
Please help.

This fix would be very important to me. This format is very common in german texts.


Update Dockerfile: change base image to SLE BCI

Related issue: longhorn/longhorn#2939
Reference build result with manually built test image: https://ci.longhorn.io/job/private/job/longhorn-tests-regression/1116/

The BCI branch will be deleted because it's for a temporary testing purpose. Please PR to the master branch and fix the DCO issue. Thanks.

I was syncing my longhorn-tests forked repo via Github UI, which result in unwanted and  unsigned commit.
Had to re-pull and commit the code to fix the commit history, leave note here(might be better way):
$ git clone longhorn-test
$ git checkout create-a-feature-branch
$ `# apply the change from current commit`
$ git commit --signoff
$ git remote rename origin upstream
$ git remote add origin https://github.com/kaxing/longhorn-tests.git
$ git push origin bci --force


Refactor: Rename Address to Venue

Fixes #47
Files not included in this refactor:
docs/tutorials

Looks :thumbs: to me


Add js messaging framework

Task/Issue URL:
Description
Steps to test this PR
Feature 1

[ ]
[ ]

UI changes

Before
After

!(Upload before screenshot)
(Upload after screenshot)

Current dependencies on/for this PR:

develop

PR #3545   👈

This comment was auto-generated by Graphite.


[Fixes #5463] JsonPatchDocument.ApplyTo With ModelState Throws Exceptions

Related PR: https://github.com/aspnet/JsonPatch/pull/46

what about this statement from the bug:

ArgumentException when the patch document is invalid e.g. empty op property or typo in the op property.

Are there other kinds of things we should test?

Are there other kinds of things we should test?

I do not think so as there are basically 3 areas for exceptions here from what I have investigated:

Deserialization into JsonPatchDocument or its generic version. These errors should already be in the ModelState.
User calling ApplyTo on the JsonPatchDocument. (This should be fixed with the current PR)
Exceptions that happen due to invalid data (ex: Invalid 'path' in the operation). [This is already handled by ObjectAdapter]

Ok thanks, these tests seem ready to go :shipit:

047def1aaf097b080b899fd829f4f47d53bf16c8


More optimizations for ebarlas submission

Implementation class: CalculateAverage_ebarlas
Runtime on c5.2xlarge EC2 instance:
real	0m7.920s
user	1m0.896s
sys	0m0.664s

I added the following line in an attempt to smoothen the sdk selection process. Please be sure GraalVM is selected as it provides a noticeable speed-up over base OpenJDK.
source "$HOME/.sdkman/bin/sdkman-init.sh"

Could you squash all that into a single commit, based off of the latest main? Will make it a bit simpler for me to handle. Thx!

Sure, I'll do that now.

Done, @gunnarmorling !

Wow, seeing a massive improvement with this one over your version on main. Do you see that in your env too? What makes the difference?

Yes! Absolutely massive, as you can see in my timing: 7.x seconds.
I actually provisioned a CX33 VM and it reliably runs at 6.5 seconds there.

The loop unrolling was the latest leap forward.

But also, I'm still not quite able to reconcile the prior timing with your reports on my equivalently size EC2 instance. That is, in part, why I provisioned a CX33 VM of my own.

I actually provisioned a CX33 VM and it reliably runs at 6.5 seconds there.

Oh, wow, that's weird. I see 10-11 sec as of this PR 🤔 . To be sure, I don't think there's CX33, only CCX33. Is that what you have? I definitely see stable results on my machine (just ran yours from main again, it's the same value as currently is on the leaderboard). I could see some variance between machines of the same spec, but 40% seems crazy.

Darn! I was hoping for sub-10s!
Yes, CCX33. I chalked it us to huge variable in the underlying physical machines. My VM is in US. I may try VMs in other geos to try to approach your results.
Here's my last run on CXX33 VM:
[root@fedora-32gb-hil-1 ~]# for i in {1..5}; do time java $JAVA_OPTS --class-path average-1.0.0-SNAPSHOT.jar dev.morling.onebrc.CalculateAverage_ebarlas measurements.txt 8 > out.txt; done

real	0m6.587s
user	0m47.744s
sys	0m1.336s

real	0m6.577s
user	0m47.831s
sys	0m1.278s

real	0m6.550s
user	0m47.730s
sys	0m1.251s

real	0m6.568s
user	0m47.673s
sys	0m1.336s

real	0m6.544s
user	0m47.709s
sys	0m1.320s

Darn! I was hoping for sub-10s!

Pretty sure we'll get there :)

I may try VMs in other geos to try to approach your results.

Mine is in nbg1-dc3. They might well have newer boxes in US, as those DCs are newer. It might make sense, once dust has settled a bit, and there's a clear top ten or so, to re-run them again in a fresh box. Let's see. What unnerves me now a little bit, seeing this stark difference, is the risk to lose my current box as it'd mean all existing results are moot. While I have automated the eval pipeline quite a bit by now, I'm not looking forward to doing all that again. Hoping for the best :)
Btw. I have some Terraform set-up here, just in case it's interesting to you.

Exactly. The instability really came into focus when I got wildly different results on the same VM class. I immediately wondered what your setup is. And also if they can even migrate VMs to different physical hardware.

And also if Hetzner can migrate VMs to different physical hardware!

At least not that easily I suppose, as there's a local disk attached.
Coming in at 00:10.870. First place! Let's see what @royvanrijn's latest one has to offer :)


Create or Insert a model without touching relations

Hi,
Is there any way to create or insert a model without affecting its relations. My use case here is let's say I have a collection of User models. When click on a User I'll fetch its relation such as Member.
So now when I run (assuming I defined relationship between User and Member)
store.dispatch['entities/member/create', { data: member }]
It does create a Member model but also affect my User collection. Only one User associated with the created Member left.
Thanks for your helps in advance.

Hi! So the member contains user record too right? In that case, you can pass create / insert / update / insertOrUpdate option to each preset methods. In your case you could do something like this.
// `create` member, but `insert` user.
store.dispatch('entities/member/create', {
  data: member,
  insert: ['user']
})

Hi @kiaking ,
Since I fetched users already so when fetching member from the server, I don't want to include the associated user (to save a query to database).
That means there's no userId in my member model.
Hope it makes sense. Thanks for your reply!

Ahah. Could you paste the member data that you're trying to save to the state? I'm wondering if there is a bug where user would get created even if there aren't users data in member's record that you're trying to save.

@kiaking Just checked the code. I do have userId but no related model from the Server. So I don't think any bug on your great plugin. I ended up creating via new Member(data) and store it different than Vuex-ORM data collection. By doing this I can use it on specific Vue component, without affecting the whole data.
Thanks for your prompt reply!

Oh OK! Glad you found a fix : )


Doesn't follow search rules to find .p4config

If .p4config is in the top-level directory everything seems to be OK.  However, if its in the Parent, the extension doesn't honor it.  Or, it may be similar to the issues #166 and #176
Request is to look for .p4config based on the P4CONFIG environment variable and the rules defined by Helix/Perforce: https://www.perforce.com/manuals/cmdref/Content/CmdRef/P4CONFIG.html

Hi
You might want to try my fork
I reworked this in v4.0.0 so it should detect all p4config files using the built in logic from p4


Update pom.xml

When using the AEM Eclipse plugin wizard, server creation complains about the servlet specification (2.4) which is too old and per Adobe documentation, it should be at least 2.5. Same case with JDK version, this pom.xml uses 1.6 when the minimum for AEM 6.x is 1.7
See: https://docs.adobe.com/docs/en/aem/6-0/deploy/technical-requirements.html

Is it possible to use your PR with the AEM Eclipse plugin wizard? At the end of Wizard I am getting
'''
The server does not support version 2.4 of the J2EE Web module specification
'''
AEM 6.1, Eclipse Mars 4.5.1, Maven 3.3.9, Java 1.8, Windows

@HaroldPutman Yes, for sure, you can use it, before you configure the AEM server, appy this PR, then the message should not appear. Let me know, regards.

Also see https://github.com/Adobe-Marketing-Cloud/aem-eclipse-developer-tools/issues/67

Manually applied the Servlet API version bump in 65b91b61e3fc4a38369773bd28fa302f4f5edb8d, thanks for the pull request!


ReactPHP Multi-Core Env

Hi I am currently exploring ReactPHP and would like to know if there is a way for it to run multiple server instances on different cores of a multi-core machine. Same as  Nodejs cluster. I tried searching but was not able to find anything regarding this. Same goes for Amphp's Aerys server.
Thanks!

ReactPHP is designed to allow async I/O within a single thread/process. This means that by default it will not spawn multiple processes.
This means that you can spawn any number of processes to fully utilize all your CPU cores. There are a number of different strategies to do so, so this is currently left up to you.
If you know a project (or feel like creating one) that offers a reusable API, I would love to hear about this, so make sure to let us know if this is something you're working on this :+1:
I hope this helps :+1:


Instructions to build the go application

Can you add the Instructions to build the go application of openshifter (Linux, MacOs) or to generate them as binary under release section ?

Yes, and in fact I'm working already in the wip-mh9 branch on it, see https://github.com/openshift-evangelists/openshifter/blob/wip-mh9/contrib.md

Tracking in #24 as well.

Closing this as it's not relevant to Python, the recommended and supported way to use the tool is using the provided container.


Bug in OnValidate trigger of Base Unit of Measure field

Excerpt from the OnValidate trigger of the Base Unit of Measure field in the Item table:
"Sales Unit of Measure" := "Base Unit of Measure";
"Purch. Unit of Measure" := "Base Unit of Measure";
if "Base Unit of Measure" <> '' then begin
  // If we can't find a Unit of Measure with a GET,
  // then try with International Standard Code, as some times it's used as Code
  if not UnitOfMeasure.Get("Base Unit of Measure") then begin
      UnitOfMeasure.SetRange("International Standard Code", "Base Unit of Measure");
      if not UnitOfMeasure.FindFirst then
          Error(UnitOfMeasureNotExistErr, "Base Unit of Measure");
      "Base Unit of Measure" := UnitOfMeasure.Code;
  end;

If the Base Unit of Measure code is changed because it was found with the International Standard Code, then you must change Sales Unit of Measure and Purch. Unit of Measure as well.

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


Is there support for multiple servers?

Hi! :wave:
I looking for some way to connect to a redis sentinel cluster in a laravel project and ran into this package.
There is something I don't understand though.
Is there no way to specify multiple servers? The documentation doesn't hint on how to set that up and looking through the source code I can't really figure it out either.
We are expecting to have three servers running redis sentinel and if one of these are down, we would still want the applications to be able to connect.
Do I understand this right? With this package, you can only specify one server, and if that server is down (but all other servers in the sentinel cluster is up), you won't be able to reach the redis sentinel cluster?

This package was written with the intention to be used behind a load balancer for Redis Sentinel, i.e. where all Redis Sentinel instances share one IP and failed instances are removed automatically by the load balancer.
If you allow configuring multiple Redis Sentinel instances in the application, you move the responsibility to detect failed instances to the application. In case you have multiple applications with a similar architecture, this means you have the same problem and responsibility over and over again. Therefore it is preferrable to have a load balancer do this for you.
In case of Kubernetes (which is where we have been using this library, but are not anymore due to architectural changes), we've simply used a Service in front of our Redis Sentinel StatefulSet in combination with health checks.
There are many other suitable tools for load balancing and/or virtual IPs though, and you can certainly do the same on a VM or with Docker.

@Namoshek Ok. Thanks for the insights. We were considering running sentinel without a load balancer and just letting the application try the other instances, in case one doesn't answer. I see your point, however. We will look into it and consider it. Thanks.
Are you not using the library at all anymore? Are you planning to continue maintain it, adding support for new Laravel and php-versions?

I think the library is still a dependency of our application and would probably work just fine, but we have configured a different Redis driver at the moment. That's also why the library never made its way to a v1 - I've not enough production-level experience with it so far.
As long as there are no major rewrites required (i.e. when Laravel or the phpredis extension change completely), I will probably support it, yes. The whole library isn't really that big and complex, so this should not be an issue anway.

So if we are talking for a setup in Kubernetes with 3 sentinel pods , the REDIS_SENTINEL_HOST will be the SENTINEL service while the
REDIS_SENTINEL_SERVICE will be the redis service?
'default' => [
    'sentinel_host' => env('REDIS_SENTINEL_HOST', '<IP_ADDRESS>'),
    'sentinel_port' => env('REDIS_SENTINEL_PORT', 26379),
    'sentinel_service' => env('REDIS_SENTINEL_SERVICE', 'mymaster'),

With redis service I mean the service that maps to the  redis pods and with sentinel service the one that maps to sentinel pods.


kubectl delete starrocks-fe-and-be.yaml

Running the delete command in OpenShift gets me stuck and won't return cursor to me.
atwong@Alberts-MBP ~ % kubectl delete -f https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/examples/starrocks/starrocks-fe-and-be.yaml
starrockscluster.starrocks.com "starrockscluster-sample" deleted

are you cleaned the operator deployment first?

atwong@Alberts-MBP ~ % export APIURL=https://api.cluster-nvm6k.nvm6k.sandbox1844.opentlc.com:6443
atwong@Alberts-MBP ~ % for i in $( kubectl get ns | grep Terminating | awk '{print $1}'); do echo $i; kubectl get ns $i -o json| jq "del(.spec.finalizers[0])"> "$i.json"; curl -k -H "Authorization: Bearer $(oc whoami -t)" -H "Content-Type: application/json" -X PUT --data-binary @"$i.json" "$APIURL/api/v1/namespaces/$i/finalize"; done
starrocks
{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "starrocks",
    "uid": "78be41e4-9978-4168-8b6e-fa178b82dec2",
    "resourceVersion": "93623",
    "creationTimestamp": "2023-05-05T23:03:42Z",
    "deletionTimestamp": "2023-05-05T23:25:45Z",
    "labels": {
      "control-plane": "cn-controller-manager",
      "kubernetes.io/metadata.name": "starrocks"
    },
    "annotations": {
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"labels\":{\"control-plane\":\"cn-controller-manager\"},\"name\":\"starrocks\"}}\n",
      "openshift.io/sa.scc.mcs": "s0:c26,c0",
      "openshift.io/sa.scc.supplemental-groups": "1000650000/10000",
      "openshift.io/sa.scc.uid-range": "1000650000/10000"
    },
    "managedFields": [
      {
        "manager": "cluster-policy-controller",
        "operation": "Update",
        "apiVersion": "v1",
        "time": "2023-05-05T23:03:42Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {"f:metadata":{"f:annotations":{"f:openshift.io/sa.scc.mcs":{},"f:openshift.io/sa.scc.supplemental-groups":{},"f:openshift.io/sa.scc.uid-range":{}}}}
      },
      {
        "manager": "kubectl-client-side-apply",
        "operation": "Update",
        "apiVersion": "v1",
        "time": "2023-05-05T23:03:42Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}},"f:labels":{".":{},"f:control-plane":{},"f:kubernetes.io/metadata.name":{}}}}
      },
      {
        "manager": "kube-controller-manager",
        "operation": "Update",
        "apiVersion": "v1",
        "time": "2023-05-05T23:25:51Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"NamespaceContentRemaining\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionContentFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceFinalizersRemaining\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}},
        "subresource": "status"
      }
    ]
  },
  "spec": {

  },
  "status": {
    "phase": "Terminating",
    "conditions": [
      {
        "type": "NamespaceDeletionDiscoveryFailure",
        "status": "False",
        "lastTransitionTime": "2023-05-05T23:25:51Z",
        "reason": "ResourcesDiscovered",
        "message": "All resources successfully discovered"
      },
      {
        "type": "NamespaceDeletionGroupVersionParsingFailure",
        "status": "False",
        "lastTransitionTime": "2023-05-05T23:25:51Z",
        "reason": "ParsedGroupVersions",
        "message": "All legacy kube types successfully parsed"
      },
      {
        "type": "NamespaceDeletionContentFailure",
        "status": "False",
        "lastTransitionTime": "2023-05-05T23:25:51Z",
        "reason": "ContentDeleted",
        "message": "All content successfully deleted, may be waiting on finalization"
      },
      {
        "type": "NamespaceContentRemaining",
        "status": "True",
        "lastTransitionTime": "2023-05-05T23:25:51Z",
        "reason": "SomeResourcesRemain",
        "message": "Some resources are remaining: services. has 2 resource instances, starrocksclusters.starrocks.com has 1 resource instances, statefulsets.apps has 1 resource instances"
      },
      {
        "type": "NamespaceFinalizersRemaining",
        "status": "True",
        "lastTransitionTime": "2023-05-05T23:25:51Z",
        "reason": "SomeFinalizersRemain",
        "message": "Some content in the namespace has finalizers remaining: starrocks.com.service/protection in 2 resource instances, starrocks.com.starrockscluster/protection in 1 resource instances, starrocks.com.statefulset/protection in 1 resource instances"
      }
    ]
  }
}%

I can't seem to delete the statefulset.

all the crd related resources have finalizer protection setting, need to clean the finalizer first if deleting them from kubectl. those resources can be deleted by operator, if your operator pod is still there and you just need to issue a kubectl delete starrockscluster <clustername>

try for i in $(oc get starrockscluster | grep Terminating| awk '{print $1}'); do oc patch pvc $i --type='json' -p='[{"op": "replace", "path": "/metadata/finalizers", "value":[]}]'; done

thanks guys!


Add tests for checking the index file

Add tests for checking the index file
I've been bitten a few times in another my lodash-fp plugin by the fact that my index.js file had some typos in it (either in the exported rule, or the recommended config).
I've thus added a test to verify those.

Good idea!

👍

I'll probably add the same test file to eslint-plugin-xo (I have added it to eslint-plugin-lodash-fp too). Wondering if this is an "okay" copy/paste code, or if this should be a module (doesn't seem generic enough to me)

Yeah, sure, doesn't seem generic enough for a module.


Code analysis highlight error

From<EMAIL_ADDRESS>on 2012-02-22T13:19:13Z
Spyder Version:  2.2.0dev
Python Version:  2.7.2
Qt Version:      4.7.4, PyQt4 (API v1) 4.8.6 on Windows
What steps will reproduce the problem?
Paste this code into the editor:
def func():
dog=str()
bark = dog.is_bark_ing()
print dog
The local variable bark is unused, so Spyder underlines it.  It also erroneously underlines the bark in is_bark_ing().  See attached screenshot. What is the expected output? What do you see instead? Only the variable name should be underlined.
Attachment: syntax highlighting error.png
Original issue: http://code.google.com/p/spyderlib/issues/detail?id=948

From<EMAIL_ADDRESS>on 2012-02-28T23:41:33Z
I wonder who is responsible for code analysis, for highlighting and how it works together?

From pierre.raybaut on 2012-03-18T08:14:36Z
The highlighting is done here: https://code.google.com/p/spyderlib/source/browse/spyderlib/widgets/sourcecode/codeeditor.py#1519
Status: Duplicate
Mergedinto: 709

From<EMAIL_ADDRESS>on 2012-02-29T09:01:49Z
My impression was this analysis was done by pyflakes, but I'm not 100% certain.


[fix][broker] Set the ServiceUnitStateChannel topic compaction threshold explicitly and other bug fixes

Motivation
We better set the compaction threshold of the serviceUnitStateChannel topic when initializing itself instead of relying on other components' init.
Also, I found the lookup and unload request often timeout  by
Modifications

Set the compaction threshold of the serviceUnitStateChannel topic in the monitor thread
return timeout completed future from deferGetOwnerRequest
Release metadataAndPayload buffer in RawBatchMessageContainerImpl.
Automatically restart load data store producer or table view if inactive.
Updated getActiveOwnerAsync logic for better synchronization of ownership checks and deferred lookup handling
added minor retry logic in the ExtensibleLoadManager tests

Verifying this change

[x] Make sure that the change passes the CI checks.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

[ ] Dependencies (add or upgrade a dependency)
[ ] The public API
[ ] The schema
[ ] The default values of configurations
[ ] The threading model
[ ] The binary protocol
[ ] The REST endpoints
[ ] The admin CLI options
[ ] The metrics
[ ] Anything that affects deployment

Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository: https://github.com/heesung-sn/pulsar/pull/61/files

re-raised the pr here. https://github.com/apache/pulsar/pull/22064


feat(Algebra/Homology/Embedding): the canonical truncation functor trunGEFunctor

Given an embedding e : Embedding c c' of complex shapes which satisfy e.IsTruncGE, we construct the canonical truncation functor e.truncGEFunctor C : HomologicalComplex C c' ⥤ HomologicalComplex C c'.

Thanks!
maintainer merge


slider plugins animation issues

I tried slick.js and swiperjs.
In both there is issue in animation after second slide.
slick.js:
https://github.com/Vinlic/WebVideoCreator/assets/1027651/579c97d2-cd1c-481f-9674-abd7ad67e789
swiperjs:
https://github.com/Vinlic/WebVideoCreator/assets/1027651/6d8365f0-9c87-4c3d-abd5-33a04710a634

@ivangusev Can you provide a reproducible URL？

@Vinlic , here it is https://www.normad.systems/AnimToVideoSlick

@ivangusev
I have reproduced this issue. Currently, 0.0.17 has fixed the issue where Swiperjs can only play transition animations once, but it still cannot solve the problem of automatically playing from the first slide. I am thinking about the reason...  :D

I have resolved this issue and now it works well in version 0.0.18. Here is the rendering result:
slick.js:
https://github.com/Vinlic/WebVideoCreator/assets/20235341/841125e6-cf43-4765-86e4-7debbd59a487
swiperjs:
https://github.com/Vinlic/WebVideoCreator/assets/20235341/2b91e2c9-0241-4368-9015-c8b6352c60c9

@ivangusev
I have resolved this issue and now it works well in version 0.0.18. Here is the rendering result:
slick.js:
https://github.com/Vinlic/WebVideoCreator/assets/20235341/841125e6-cf43-4765-86e4-7debbd59a487
swiperjs:
https://github.com/Vinlic/WebVideoCreator/assets/20235341/2b91e2c9-0241-4368-9015-c8b6352c60c9


Why box_loss / labels.numel() instead of box_loss / labels_pos.numel()?

❓ Questions and Help
For the box_loss here:
https://github.com/facebookresearch/maskrcnn-benchmark/blob/master/maskrcnn_benchmark/modeling/roi_heads/box_head/loss.py#L165
According to my understanding, the box_loss is only defined on postive samples.
So why not box_loss = box_loss / labels_pos.numel() ?

This is to maintain consistency with the implementation in Detectron, which used their custom SmoothL1Loss with inside_weights and outside_weights.
They divide the loss by labels.numel(), see here. We could probably change it to labels_pos.numel(), but it would involve changing the scaling of the losses.
For completeness, here is how it is implemented in Detectron: the inside_weights and outside_weights and the smooth l1 loss computation


Fix/keep old models

There were problems using the new object with the Dockstore CLI, reverting to old Workflow model with a bunch of nulls. Slightly larger reponse but oh well.

aww crud, i just realized MyWorkflows is still in the swagger.yaml and openapi.yaml


[Candidate 1v1 Arizona Airfield ZH v1] Oil Derricks, Oil Refineries and Artillery Platforms have custom rotations that make their flag less visible

Issue Description
Oil Derricks, Oil Refineries and Artillery Platforms have custom rotations that make their flag less visible.
Expected behavior
Oil Derricks, Oil Refineries and Artillery Platforms preferably have default rotation or a rotation that makes their flag highly visible.
Screenshots

Fixed. New version v2 will be on GR.org topic


Fortune Recording Oracle - Get data from manifest

Description
Develop a client to retrieve data from the manifest.
Motivation
We want to develop a client that would be able to receive data from the manifests presented in the escrow in the manifestUrl parameter.
Implementation details

[ ] Develop a client to retrieve data from the manifest

In current implementation event IntermediateStorage(url, hash) has required hash parameter. Should we send non-encrypted data there? Anyway events cannot be edit, does we need to store hash there? Maybe it makes more sense to hash the data itself to preserve its integrity? @posix4e

The hash is a hash of the unecrypted data. The url can point to encrypted / unencrypted


Add sample Electron app that uses msal-node

Coverage remained the same at 82.219% when pulling d4748be5df7081741cd54d6271dde1617f4891a6 on sso-hackathon-electron-sample into fc150c85612660d20f14e823ca32495a75b0b312 on dev.


Demo generated Data wrong status

The generated demo data got the 'changed' status (4), due to that the mobile app freezes trying to upload the not existing demo images, when snding to server.

The Status of the generated data is now STATUS.NONE.


Running on Raspberry Pi 3

What did I do wrong?
pi@raspberrypi:~ $ plantgateway
INFO:root:loaded config file from /home/pi/.plantgw.yaml
Traceback (most recent call last):
  File "/usr/local/bin/plantgateway", line 20, in <module>
    main()
  File "/usr/local/bin/plantgateway", line 14, in main
    pg = PlantGateway()
  File "/usr/local/lib/python3.4/dist-packages/plantgw/__init__.py", line 76, in __init__
    self._start_client()
  File "/usr/local/lib/python3.4/dist-packages/plantgw/__init__.py", line 90, in _start_client
    self.mqtt_client.connect(self.config.mqtt_server, self.config.mqtt_port, 60)
  File "/usr/local/lib/python3.4/dist-packages/paho/mqtt/client.py", line 686, in connect
    return self.reconnect()
  File "/usr/local/lib/python3.4/dist-packages/paho/mqtt/client.py", line 808, in reconnect
    sock = socket.create_connection((self._host, self._port), source_address=(self._bind_address, 0))
  File "/usr/lib/python3.4/socket.py", line 491, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/lib/python3.4/socket.py", line 530, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -5] No address associated with hostname

The error occurs during the connection to the MQTT server. Did you check if the address of the mqtt server is correct? Can you ping the MQTT server?

did not hear anything -> closing issue


[PA-1017] Add ClusterController

What this PR does / why we need it:
This PR introduces a library to simplify context handling and eliminate direct usage of scheduler.contexts, improving code quality and reducing user errors in the test case.
Which issue(s) this PR fixes (optional)
Closes #PA-1017
Special notes for your reviewer:
Please note that this PR is currently lacking comprehensive testing.
Please review the Jenkins build for the "ClusterControllerDemo" test case using the link below
https://jenkins.pwx.dev.purestorage.com/job/Users/job/Sumit/job/Custom-Pipelines/job/px-backup-on-demand-system-test-byoc/1447

There are so many changes to be made.


Error: connect ECONNREFUSED <IP_ADDRESS>:8081

I have been testing tunnelmole with the NPM module tunnelmole
OS - Ubuntu
NodeJs - 18.16.1
Code
import { tunnelmole } from 'tunnelmole';

const url = await tunnelmole({
    port: 3000
});

Error
Error: connect ECONNREFUSED <IP_ADDRESS>:8081
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '<IP_ADDRESS>',
  port: 8081
}

Looks like you've grabbed the old version that had this bug.
Upgrade to the latest version 2.2.10 and this should solve the issue.
Sent from Proton Mail mobile
-------- Original Message --------
On 26 Nov 2023, 01:19, umeshtyagitvz wrote:

I have been testing tunnelmole with the NPM module tunnelmole
OS - Ubuntu
NodeJs - 18.16.1
Code
import { tunnelmole } from 'tunnelmole';
const url = await tunnelmole({
port: 3000
});
Error
Error: connect ECONNREFUSED <IP_ADDRESS>:8081
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '<IP_ADDRESS>',
port: 8081
}
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: @.***>

I have been using the latest version only.

Thats the one behind. You'll need 2.2.11. Your screenshot shows 2.2.10.

Thanks for help


Consider removing ParquetFilters

What is the problem the feature request solves?
The ParquetFilters were added originally so we could shade Parquet in Comet. However the shading was removed later as it caused a lot of trouble on the Iceberg side. In addition, we ported several Parquet classes from parquet-mr into Comet, so now the boundary between Comet and parquet-mr is fairly thin. Therefore, we could consider removing ParquetFilters and directly use the one from Spark.
The advantage of this is we are able to absorb changes & improvements in the newer version of Spark. For instance, https://github.com/apache/spark/pull/36696 added Parquet In/NotIn pushdown from Spark side, which is only available since Spark 3.4. At the moment, as Comet keeps a copy of Spark's ParquetFilters, the feature is not added in order to be backward compatible with Spark 3.2 and 3.3.
Describe the potential solution
Evaluate whether we can remove ParquetFilters from Comet.
Additional context
No response

I will work on this

I have tried this out. Everything works fine except this fix https://issues.apache.org/jira/browse/SPARK-46092. The fix is in Spark 3.4.3. I will wait for Spark 3.4.3 is released, and then remove the ParquetFilters


Add entries for ref_assembly and star_index to scpca-meta.json files

In working on processing projects through scpca-nf, I noticed that the libraries with multiplexing were not skipping genetic demuxing, even though the results files already exist and they should be skipped. When I went to look at why, it was because the scpca-meta.json for multiplexed samples did not contain a ref_assembly version. That's what we use to check to ensure we skip demultiplexing. Here, I'm updating the script we use to add in fields to the scpca-meta.json files to include the ref_assembly field.
When I was poking around, I also noticed that we don't store the star_index in scpca-meta.json for the multiplexed libraries, but we do for all other libraries. I am also adding that as a field to include in the json files here.
Note that I updated one of the json files and then re-ran the workflow and things were now skipping genetic demultiplexing as they should be.

Just to check: are we setting this in the multiplexing workflow as we should?

Yes, we write out the meta object in the vireo process and the meta object should contain both of those items.


Prepare for the demo

Please prepare for the demo:

figure out what to say and in which order
anticipate questions and do a revamp of bacis knowledge from CBRS domain
conduct the demo a few times alone

Additionally, figure out how to hide sensitive data from the recording, mainly SAS User Id.

All terminal logs will be passed through a sed filter which will cut out the known SAS User ID that FreedomFi is using.
All UI elements will be hidden using OBS recording/streaming software which will be used as the video source during the Demo - specific scenes prepared for each of the NMS UI sections will blur out parts of the screen statically, that contain sensitive data


Ensure variable names are intuitive to prep for v2.1 macros documentation

Mostly a user transparent change for now.  I will review the variable names and roll names to ensure they make sense for general documentation and use once they are publicly exposed in v2.1

Decided this will coincide with converting to sheet worker calculations


Working on auto generating go from memory or unix pipes

Signed-off-by: Kris Nóva<EMAIL_ADDRESS>

Leaving some notes:
I want to be able to do the following
cat deployment.yaml | naml codify > deployment.go
kubectl get po <name> -oyaml | naml codify > pod.go

Closes #40


PyTorch Lightning: Support saving model to model_file

To be consistent with other integrations.

Copy here from PR conversation:
LightningModule implements method log, which user call to store everything he wants, and then LightningModule calls logger. Checkpoints are saved at the end of batch/epoch via hooks like on_train_batch_end, etc. Logger doesn't has access to these hooks.
To save models we should write callback and encourage users to use it with our logger. But users can use native callbacks and control saving everything himself, and track it with DVC.
We can implement after_save_checkpoint method and save model_file here. But it will duplicate files if user won't disable native checkpointing at all, so we should encourage users to do so.

Hi @daavoo @sirily - I've just been bashing my head against this one and would be happy to implement a solution.
Maybe I've got the idea totally backwards, but I'd like to use DVC to version the checkpoint file and recover state from earlier experiments. So far I have:
dvc.yaml
stages:
  train:
    cmd: python train.py
    deps:
    - data/
    - train.py
    outs:
    - ckpts/model.ckpt:
        checkpoint: true
    live:
      logs:
        summary: true
        html: false

train.py
from pytorch_lightning import Trainer
from pytorch_lightning.callbacks import ModelCheckpoint
from dvclive.lightning import DvcLiveLogger

checkpoint_callback = ModelCheckpoint(filename="model",
                                      dirpath='ckpts',
                                      save_top_k = 1)

dvclive_logger = DvcLiveLogger(
    path='logs',
    resume=True,
    summary=True
)

m = MyModel()

trainer = Trainer(logger=dvclive_logger, callbacks = checkpoint_callback)

trainer.fit(m, ckpt_path='ckpts/model.ckpt') # loads from checkpoint

I encounter the problem that PL doesn't like overwriting files in the ckpt directory, and instead saves new checkpoint files as ckpts/model-v1.ckpt *-v2.ckpt etc., meaning that the new model state is not reflected in dvc.lock

We can implement after_save_checkpoint method and save model_file here. But it will duplicate files if user won't disable native checkpointing at all, so we should encourage users to do so.

@sirily - is your suggestion to use after_save_checkpoint to persist PL's checkpoints into a file that DVC is tracking? Would it fire as intended with PL's enable_checkpointing=False to suppress the native checkpointing?

Would it fire as intended with PL's enable_checkpointing=False to suppress the native checkpointing?

@a-orn afaik enable_checkpointing just creates a ModelCheckpoint callback with default parameters. If you use ModelCheckpoint callback on your own or some custom callback, you don't have to set this flag.
But I can't check it right now, I need a couple of days to return home

I encounter the problem that PL doesn't like overwriting files in the ckpt directory, and instead saves new checkpoint files as ckpts/model-v1.ckpt *-v2.ckpt etc

If you set a metric to monitor, PL will save only one checkpoint.
Something like this:
cp = pl.callbacks.model_checkpoint.ModelCheckpoint(
    monitor="train_loss_epoch",
    save_top_k=1,
    dirpath=ckpt_path,
    filename='checkpoint_{epoch}')

Thanks, that's good to know. I think it still results in a PL checkpoint named differently to model.ckpt in dvc.yaml, so it might require a more complex stage than the one I've proposed. I think, elsewhere, you proposed using a foreach stage to checkpoint at regular intervals?

I think it still results in a PL checkpoint named differently to model.ckpt

With filename='checkpoint' it will be the only one name, wouldn't it? I can't check it right now.

I think, elsewhere, you proposed using a foreach stage to checkpoint at regular intervals?

Yep, I think it's the easiest way. If you have in mind something more beautiful, you are welcome to propose it

I recently discovered the experimental 'Checkpointing IO API' for Pytorch Lightning:
import torch, os

from pytorch_lightning import Trainer
from pytorch_lightning.plugins import CheckpointIO
from pytorch_lightning.callbacks import ModelCheckpoint

from dvclive.lightning import DvcLiveLogger

# Override default naming pattern
class CustomCheckpointIO(CheckpointIO):
    def save_checkpoint(self, checkpoint, path):
        if(os.path.exists(model_file)):
            filename = os.path.basename(model_file)
            os.rename(model_file, model_file.replace(filename, 'old.ckpt'))
        torch.save(checkpoint, model_file)

    def load_checkpoint(self, path):
        return torch.load(model_file)

    def remove_checkpoint(self, path):
        filename = os.path.basename(model_file)
        os.remove(model_file.replace(filename, 'old.ckpt'))

# Set path to file checkpointed in dvc.yaml
model_file = 'model.ckpt'

custom_checkpoint_io = CustomCheckpointIO()

dvclive_logger = DvcLiveLogger(
    path='logs',
    resume=True,
    summary=True
)

m = MyModel()

trainer = Trainer(
    logger=dvclive_logger,
    plugins=custom_checkpoint_io,
    callbacks=ModelCheckpoint()
)

trainer.fit(m, model_file ') # loads from checkpoint

This is starting to get to the workflow I expected. I call dvc exp run and it starts from the last checkpointed epoch until completion or is killed. The checkpoints are overwritten and dvc.lock is updated. When I've finished training, I commit the experiment and the checksum of the final checkpoint file is tracked in git for use in downstream tasks.
@daavoo is this the pattern you had in mind for #140 and #174?
Docs at: https://pytorch-lightning.readthedocs.io/en/stable/advanced/checkpoint_io.html

I recently discovered the experimental 'Checkpointing IO API' for Pytorch Lightning:
import torch, os

from pytorch_lightning import Trainer
from pytorch_lightning.plugins import CheckpointIO
from pytorch_lightning.callbacks import ModelCheckpoint

from dvclive.lightning import DvcLiveLogger

# Override default naming pattern
class CustomCheckpointIO(CheckpointIO):
    def save_checkpoint(self, checkpoint, path):
        if(os.path.exists(model_file)):
            filename = os.path.basename(model_file)
            os.rename(model_file, model_file.replace(filename, 'old.ckpt'))
        torch.save(checkpoint, model_file)

    def load_checkpoint(self, path):
        return torch.load(model_file)

    def remove_checkpoint(self, path):
        filename = os.path.basename(model_file)
        os.remove(model_file.replace(filename, 'old.ckpt'))

# Set path to file checkpointed in dvc.yaml
model_file = 'model.ckpt'

dvclive_logger = DvcLiveLogger(
    path='logs',
    resume=True,
    summary=True
)

m = MyModel()

trainer = Trainer(
    logger=dvclive_logger,
    plugins=CustomCheckpointIO(),
    callbacks=ModelCheckpoint()
)

trainer.fit(m, ckpt_path=model_file) # loads from checkpoint

This is starting to get to the workflow I expected. I call dvc exp run and it starts from the last checkpointed epoch until completion or is killed. The checkpoints are overwritten and dvc.lock is updated. When I've finished training, I commit the experiment and the checksum of the final checkpoint file is tracked in git for use in downstream tasks.

Thanks for the code sample @a-orn ! I need to try it.

@daavoo is this the pattern you had in mind for #140 and #174?

The pattern is to overwrite the model on each save call, like #174 does, and rely on DVC Chekpoints functionality to resume/restart.
Overwritten models are not lost because of DVC checkpoints mechanism so there is no need to store the latest one under old.ckpt.
I think that we should implement what @sirily suggested:

We can implement after_save_checkpoint method and save model_file here. But it will duplicate files if user won't disable native checkpointing at all, so we should encourage users to do so.

And emphasize on docs how to disable native checkpointing in order to prevent duplication
The checkpoint_io plugin makes sense, but it has the downside of requiring the user to pass 2 different things to the trainer instead of just the callback. I think that

@daavoo

I think that we should implement what @sirily suggested:
I've tried it, but PL doesn't call after_save_checkpoint if there is no checkpoint callback

I think to do things the most dvc-way, we don't even have to use plugin
import os
import pytorch_lightning as pl
import torch
import argparse
from dvclive.lightning import DvcLiveLogger
from tests.test_lightning import LitMNIST

if __name__ == '__main__':
    model = LitMNIST()
    # set checkpoint path
    ckpt_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "checkpoints"))
    # checkpoints will be saved to checkpoints/model.ckpt
    cp = pl.callbacks.model_checkpoint.ModelCheckpoint(
        monitor="train_loss_epoch",
        save_top_k=1,
        dirpath=ckpt_path,
        filename='model')

    # Set path to file checkpointed in dvc.yaml
    model_file = os.path.join(ckpt_path, "model.ckpt")

    # init logger
    dvclive_logger = DvcLiveLogger()
    # set None if it is the first run
    resume_from_checkpoint = None if not os.path.exists(model_file) else model_file
    trainer = pl.Trainer(
        logger=dvclive_logger, max_epochs=10,
        callbacks=[cp]
    )
    trainer.fit(model, ckpt_path=resume_from_checkpoint)

with dvc.yaml like this:
stages:
  train:
    cmd: python train.py
    outs:
      - checkpoints/model.ckpt:
          checkpoint: true

This is the workflow user expects, I guess. dvc exp run begins an experiment and resumes it if possible. But we still have to use ModelCheckpoint callback.
I don't see any ways to use only logger for saving checkpoints properly.
There is one problem with this way: DVC always tries to resume training, even if you change some parameter or track .py file as dependency and change it. If I use plugin as @a-orn suggested, I have this behavior either. And I have to delete old experiments or clear dvc.lock file. The way from December community gems seems more convinient to me.

@sirily If I understand you correctly, your solution from the community gems then solves two different issues you have:

Unlike other frameworks, PyTorch Lightning logging does not expose hooks to methods like on_train_batch_end, making it challenging to save the model in a dvclive integration.
Checkpoints in DVC always resume training from the previous checkpoint. There is an ongoing discussion and proposed behavior change in https://github.com/iterative/dvc/discussions/6104#discussioncomment-1838965. Would that proposal simplify your workflow? Is that the behavior you would expect?

@dberenbaum yes, I expect, that DVC resume an experiment only after fail. If it is interrupted by Ctrl+C or completed, I expect, that DVC start a new experiment (when smth changed of course). And if the experiment was failed, but some parameter has changed, I expect, that DVC start a new experiment either.
With such workflow it will be possible to use simple code from my previous post in this thread with native PL checkpoint callback.
But one still has to use PL callback with dvclive logger, and we can't implement saving inside the logger.


labels/annotations not propagated to child secret if using both spec.dataFrom and spec.target.template.data unless worked around

Describe the bug
For an ExternalSecret, if spec.target.template.data is used with spec.dataFrom, the annotations/values on the ExternalSecret aren't propagated down to the child secret. I raise this as a bug because https://external-secrets.io/v0.9.1/api/externalsecret/#example notes that labels/annotations are expected to be propagated down. Adding the labels annotations under spec.target.template.metadata will propagate the secret down however, so this might just be the expected behaviour, please let me know if this is the case. https://external-secrets.io/v0.9.1/api/spec/#external-secrets.io/v1beta1.ExternalSecretTemplate
Workaround/hack: I found that if only spec.dataFrom exists when the secret is created, then if the spec.target.template.data field is added and applied after, the labels/annotations are propagated down to the child secret. It has to be done in this order to work.
To Reproduce
Steps to reproduce the behavior:
k8s version: v1.25.11-eks-a5565ad
ESO app version: 0.9.1
ESO chart version: v0.9.1

provision /some/key/path into SSM and setup secretstore etc.
apply external-secret.yaml, observe labels/annotations not propagating to secret
remove spec.target.template.data from external-secret.yaml and apply again. Observe annotations/data propagating to child secret
Re-add spec.target.template.data and apply. Observe annotations/data propagating to child secret.

# external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: "hello-world"
  labels:
    foo: bar
  annotations:
    foo: bar
spec:
  secretStoreRef:
    name: ssm
    kind: SecretStore
  refreshInterval: "2m"

  # the target describes the secret that shall be created
  # there can only be one target per ExternalSecret
  target:
    name: a-secret
    creationPolicy: 'Owner'
    # Specify a blueprint for the resulting Kind=Secret
    template:
      engineVersion: v2
      mergePolicy: "Merge"
      # if the manifest is applied without data first, then the secrets from dataFrom will be propagated into the secret
      data:
        foo: bar
        biz: baz

  dataFrom:
  - extract:
      key: /some/key/path

/some/key/path
{
  "frog": "dog"
}

Expected behavior
Based on https://external-secrets.io/v0.9.1/api/externalsecret/#example, I'd expect labels/annotations on the ExternalSecret to be propagated down to the child secret. But perhaps https://external-secrets.io/v0.9.1/api/spec/#external-secrets.io/v1beta1.ExternalSecretTemplate isn't expected to work this way and will always take precedence
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Thanks for this project, it's awesome

Hi @thetombombadil . Have you tried using template.MergePolicy=Merge? 😁

That worked a treat, thanks @gusfcarvalho 🙌
For what it's worth I was referencing https://external-secrets.io/v0.8.5/api/externalsecret/#example so I didn't stumble upon mergepolicy, but on reflection I should have been looking at https://external-secrets.io/v0.8.5/api/spec/#external-secrets.io/v1beta1.TemplateMergePolicy like a normal person
Thanks again!

Actually, the labels/annotations aren't propagated down in this scenario if they're under metadata - they're only propagated to the child secret if they're placed under spec.target.template.metadata. The mergePolicy merges the values fine however.
So that may be a bug in this instance

Actually, the labels/annotations aren't propagated down in this scenario if they're under metadata - they're only propagated to the child secret if they're placed under spec.target.template.metadata. The mergePolicy merges the values fine however.

This is expected behaviour. When you define a template under spec.target.template this will define how the actual secret will look like, ignoring the labels/annotations on the ExternalSecret resource.
Imagine, someone who doesn't want the labels/annotations to be propagated to the resulting secrets (that's desirable when using argocd for instance, it uses it to track resources)

Closing since it seems to have been already resolved 🙂


Background image and outlines

Adds some settings to enhance prettiness of videos.
New settings:

Video output example:

Maybe I'll pick this up later. I may be rewriting Git history soon, so this PR probably won't apply cleanly.


Enabling SSR

Removes the global client-only and tries to solve the various upcoming issues.
To do

[ ] Sparc logo does not work
[ ] pages/data/index.vue: useHead had to be commented out (no clue what's the issue)
[ ] Twitter feed widget (bring back)

Removes the global client-only and tries to solve the various upcoming issues.
To do

[ ]  Sparc logo does not work
[ ]  pages/data/index.vue: useHead had to be commented out (no clue what's the issue)
[ ]  Twitter feed widget (bring back)

Is this ready for review? It says it is still in Draft

Removes the global client-only and tries to solve the various upcoming issues.
To do

[ ]    Sparc logo does not work
[ ]    pages/data/index.vue: useHead had to be commented out (no clue what's the issue)
[ ]    Twitter feed widget (bring back)

Is this ready for review? It says it is still in Draft

Some items are still undone.. can we discuss about those sometime this week?

Yeah. Lets discuss either today or tomorrow morning since the CCB is tomorrow


The set-env command is disabled.

As of the update this afternoon to restrict usage of set-env, this task now fails. https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/
Full error:
Run Azure/webapps-deploy@v1
  with:
    app-name: featureflags-data-eu-service
    package: serviceapp
    slot-name: staging
  env:
    AZURE_HTTP_USER_AGENT:
    AZUREPS_HOST_ENVIRONMENT:
Error: Unable to process command '::set-env name=AZURE_HTTP_USER_AGENT,::GITHUBACTIONS_DeployWebAppToAzure_6a3d8fbc4fa9b0a8568f4b02dcb98ca6aa6248e064d52b6d666423fb24ff6a2c' successfully.
Error: The `set-env` command is disabled. Please upgrade to using Environment Files or opt into unsecure command execution by setting the `ACTIONS_ALLOW_UNSECURE_COMMANDS` environment variable to `true`. For more information see: https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/
Package deployment using ZIP Deploy initiated.
Deploy logs can be viewed at https://featureflags-data-eu-service-staging.scm.azurewebsites.net/api/deployments/00271bf5bba7486a9203f66263ce5bbd/log
Successfully deployed web package to App Service.
Successfully updated deployment History at https://featureflags-data-eu-service-staging.scm.azurewebsites.net/api/deployments/ff45d2b9a986fc4f1e7209c19943be7f0dce44f81605577784550
App Service Application URL: http://featureflags-data-eu-service-staging.azurewebsites.net
Error: Unable to process command '::set-env name=AZURE_HTTP_USER_AGENT,::' successfully.
Error: The `set-env` command is disabled. Please upgrade to using Environment Files or opt into unsecure command execution by setting the `ACTIONS_ALLOW_UNSECURE_COMMANDS` environment variable to `true`. For more information see: https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/

Facing same issue. Any resolution?

Any update on this?

v1 tag is updated now. Please verify.

@aksm-ms Fixed! Thank you for fixing so quickly!


Error restarting sed couldn't open temporary file permission denied

Checklist

Have you pulled and found the error with jc21/nginx-proxy-manager:latest docker image?

Yes

Are you sure you're not using someone else's docker image?

Yes

Have you searched for similar issues (both open and closed)?

Yes

Describe the bug
Error restarting sed couldn't open temporary file permission denied
Nginx Proxy Manager Version

2.10.3 (via Docker Compose)
To Reproduce
Steps to reproduce the behavior:

Go to main proxy page and login with the admin user
change the user and create new password, etc..
Select Hosts | Add Proxy Host
Create a proxy host with your domain to an machine in your network.
Click Save
Restart nginx-proxy-manager
Error

Expected behavior

To allow a restart and to create a proxy hosts successfully.
Screenshots

filerun8-proxy-1  | ❯ Configuring npm user ...
filerun8-proxy-1  | 0
filerun8-proxy-1  | usermod: no changes
filerun8-proxy-1  | ❯ Configuring npm group ...
filerun8-proxy-1  | ❯ Checking paths ...
filerun8-proxy-1  | ❯ Setting ownership ...
filerun8-proxy-1  | ❯ Dynamic resolvers ...
filerun8-proxy-1  | ❯ IPv6 ...
filerun8-proxy-1  | Enabling IPV6 in hosts in: /etc/nginx/conf.d
filerun8-proxy-1  | - /etc/nginx/conf.d/include/force-ssl.conf
filerun8-proxy-1  | - /etc/nginx/conf.d/include/proxy.conf
filerun8-proxy-1  | - /etc/nginx/conf.d/include/assets.conf
filerun8-proxy-1  | - /etc/nginx/conf.d/include/ssl-ciphers.conf
filerun8-proxy-1  | - /etc/nginx/conf.d/include/ip_ranges.conf
filerun8-proxy-1  | - /etc/nginx/conf.d/include/block-exploits.conf
filerun8-proxy-1  | - /etc/nginx/conf.d/include/letsencrypt-acme-challenge.conf
filerun8-proxy-1  | - /etc/nginx/conf.d/include/resolvers.conf
filerun8-proxy-1  | - /etc/nginx/conf.d/default.conf
filerun8-proxy-1  | - /etc/nginx/conf.d/production.conf
filerun8-proxy-1  | Enabling IPV6 in hosts in: /data/nginx
filerun8-proxy-1  | - /data/nginx/proxy_host/1.conf
filerun8-proxy-1  | sed: couldn't open temporary file /data/nginx/proxy_host/sedN25JQv: Permission denied
filerun8-proxy-1  | s6-rc: warning: unable to start service prepare: command exited 4
filerun8-proxy-1  | /run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more information.

Operating System

Mac M1, Colima, Docker
Additional context

Getting the same error when I restart.

Any idea what it could be? I have the exact same thing happening. When I get lucky it runs but made the mistake to stop the container and now the issue is back and I can’t get it to start anymore

@Antergosgeek a temporary solution is change the file sharing from VirtioFS to gRPC FUSE in Docker Desktop for Mac. Details here: https://github.com/NginxProxyManager/nginx-proxy-manager/issues/3343

Thank you! Sorry been a while since I looked!


update imports for fastext - needed when using binary embeddings

This PR fixes the imports for fasttext.
Although the change is minimal, I wonder whether to include fasttext directly in the requirements (the installation seems to be very brittle, especially on centos, whose ships an older version of GCC) or in the documentation, probably installed from the sources only if needed..
Any preference?

Hi Luca !
The readme points to fasttextmirror (0.8.22) which uses import fastText (so not a bug with fasttextmirror, but the commented import in the requirement files disappeared at some point!). I guess moving to fasttext 0.9.2 instead is better because more recent?

I wonder whether to include fasttext directly in the requirements (the installation seems to be very brittle, especially on centos, whose ships an older version of GCC) or in the documentation, probably installed from the sources only if needed..

So we would add (commented) fasttext 0.9.1 in the requirement files because easier to compile than version 0.9.2?
I think using the fasttext binary embeddings in general is not a super idea because it slows down everything (no embeddings cache), it was added more for reproducibility purposes. So I would keep it commented/outside the default requirement as before.

Hi Luca !
The readme points to fasttextmirror (0.8.22) which uses import fastText (so not a bug with fasttextmirror, but the commented import in the requirement files disappeared at some point!). I guess moving to fasttext 0.9.2 instead is better because more recent?

ahh I did not notice that.. fasttextmirror seems not updated since version 0.8.. I guess fasttext 0.9.2 is the most recent.

So we would add (commented) fasttext 0.9.1 in the requirement files because easier to compile than version 0.9.2?

I would go for the version 0.9.2 - but they are needed only if someone uses the bin version of the vectors...

I think using the fasttext binary embeddings in general is not a super idea because it slows down everything (no embeddings cache), it was added more for reproducibility purposes. So I would keep it commented/outside the default requirement as before.

The bin offers the advantage to being able to produce embeddings for words not seen before. I've notices a slight improvements certain times. Indeed, the management is inconvenient, more because they are loaded in memory at startup. I guess from the pont of view of the speed after a long startup will not impact the performances..
I wonder if we could load the vec with the cache and being able to load the from the bin at the same time - looks a bit science fiction

The bin offers the advantage to being able to produce embeddings for words not seen before. I've notices a slight improvements certain times.

Yes it was added to DeLFT (with the help of Pedro!) for this reason. However I observed that, for labeling general texts in English, glove embeddings were always better than the fastext bin, and pmc embeddings always better for scientific text (with random embeddings for unseen words). For text classification, I always got better results with word2vec (.vec) if I remember well. So i think it's only relevant for non-English content?

I wonder if we could load the vec with the cache and being able to load the from the bin at the same time - looks a bit science fiction

I think it's not that complicated, using the bin library only for unknown tokens, but honestly probably not worth because RNN are outdated by the transformers which are better with small sequences (except when using ELMo maybe, but we have then other problems).

Ok, as early spring cleaning, we can forget about this, since the binary does not improve the scoring, it's not portable and is loading the whole blob in memory...

absolutely !


Removal of a unused function in redirectResponse

Removal unused function getOriginalContent in redirectResponse.

You can't remove public methods on patch releases.

Moreover, this function is not unused. It's there to provide compatibility with Response.


Merchant should be able to select local body on registration

List of local bodies as available here: https://github.com/coronasafe/datastore/tree/master/JSON
Merchant can select one local body off the list that is saved to the database at the time of registration

We can have a select field during merchant registration. The user can select the local body.
We can pass in lb_code which is unique in the api the call @agneym
No api and db support for local bodies @unnitallman

@edwinbbu I have done this. Please go through https://github.com/coronasafe/journal/pull/77#issue-454325993
/cc @agneym


rel.Text should be NVARCHAR(MAX)

Same disclaimer:  Just reviewing code, not an actual user at this point.
The "text" type is deprecated in SQL Server and will be removed in a future version.
https://docs.microsoft.com/en-us/sql/t-sql/data-types/ntext-text-and-image-transact-sql?view=sql-server-ver15
Microsoft suggests replacing with NVACHAR(MAX) or VARCHAR(MAX).  I'd prefer NVARCHAR(MAX) to support Unicode.
https://github.com/go-rel/mssql/blob/e7d7d121f21fde19a9e6d66c8a9f946631b9d868/mssql.go#L143-L145

Thanks for your feedback, it's very helpful 🙏
I've created a pr to address this issue, please take a look if you have time 😄


Add e2e test for apply-delete-apply bug

What this PR does / why we need it:
This covers regression in #1596
It had to be e2e because in integration, the deletion actually never happens because the GC controller is not running.

@nfnt oh right, there could be multiple resources in an assert, correct? Yeah, let's change that


BadRequestKeyError  werkzeug.exceptions.BadRequestKeyError: 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. KeyError: 'code'

I get to the login screen, but after putting in my user name to sign in I get this in the browser session...
Traceback (most recent call last)
File "/opt/homebrew/lib/python3.9/site-packages/flask/app.py", line 2095, in __call__

return self.wsgi_app(environ, start_response)

File "/opt/homebrew/lib/python3.9/site-packages/flask/app.py", line 2080, in wsgi_app

response = self.handle_exception(e)

File "/opt/homebrew/lib/python3.9/site-packages/flask/app.py", line 2077, in wsgi_app

response = self.full_dispatch_request()

File "/opt/homebrew/lib/python3.9/site-packages/flask/app.py", line 1525, in full_dispatch_request

rv = self.handle_user_exception(e)

File "/opt/homebrew/lib/python3.9/site-packages/flask/app.py", line 1523, in full_dispatch_request

rv = self.dispatch_request()

File "/opt/homebrew/lib/python3.9/site-packages/flask/app.py", line 1509, in dispatch_request

return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) [Open an interactive python shell in this frame]

File "/Users/mthibode/onenote_export/onenote_export.py", line 232, in main_logic

code = flask.request.args['code']

File "/opt/homebrew/lib/python3.9/site-packages/werkzeug/datastructures.py", line 375, in __getitem__

raise exceptions.BadRequestKeyError(key)

werkzeug.exceptions.BadRequestKeyError: 400 Bad Request: The browser (or proxy) sent a request that this server could not understand.
KeyError: 'code'

Ran through procedures again and it worked well. Not sure what went wrong.


Modifying UNIX connection pooling tests to not require root permissions

Instead of modifying odbcinst.ini in /etc folder, the test now copies odbcinst.ini into the current folder and sets an environment variable to make unixODBC use the custom odbcinst.ini. This way the test does not requiere root permissions to modify that file.

Closing and re-opening the pull request, because Appveyor died instantly

Coverage remained the same at 74.168% when pulling 64c585dd4b02cef5f9b113b6a4b83f5941424c10 on ulvii:UnixPoolingTests into e959679aa664670a22013eb1811e2a123d47149a on Microsoft:dev.

Coverage remained the same at 74.168% when pulling 64c585dd4b02cef5f9b113b6a4b83f5941424c10 on ulvii:UnixPoolingTests into e959679aa664670a22013eb1811e2a123d47149a on Microsoft:dev.

Coverage remained the same at 74.168% when pulling 5236ee279be3807b5ce76b53f64aa61197fb1895 on ulvii:UnixPoolingTests into e959679aa664670a22013eb1811e2a123d47149a on Microsoft:dev.

Coverage remained the same at 74.168% when pulling 5236ee279be3807b5ce76b53f64aa61197fb1895 on ulvii:UnixPoolingTests into e959679aa664670a22013eb1811e2a123d47149a on Microsoft:dev.


[LTD-5504] CLE cache invalidation

Aim
This change ensures that our CLE cache will refresh on frontend deployments by using GIT_COMMIT as a key to vary the cache on.
This assumes that GIT_COMMIT will be available.  This appears to be the case for gov-PAAS environments.  For local environments this will not be the case (although we could potentially pass this through in docker-compose env vars).  We will also need to think about maintaining this for DBT platform environments when we make the switch.
In the worst case, where this env var is not available, we will have a lag of up to 24H for CLEs to refresh on the frontend.
Additionally, this PR ensures that the python manage.py invalidate command provided by django-cacheops can run and that things cached by cacheops are cached under redis DB 2.  This means if we needed to manually invalidate the cache on an environment, we would only invalidate cacheops records and not all redis records.
LTD-5504

Just want to triple check this on a real env before merging - so converting to draft in the meantime


Feature Request: "Copy, Clean, and Open in New Tab" in the context menu

First, GREAT extension, thank you!!! I like the context menu approach, but it would be a real click saver if there was a second option that just open the cleaned link in a new tab. :0)

Flattery will get you everywhere. This is done and will be out in the next release.


detection precision problem

I use the code 'wR2.py' and the dataset you provided to train a detection model with 25 epochs.
Because I can not find the code to evaluate the precision of detection model,so I write it by myself.
But I find the license plate detection results are very poor.Why?
@detectRecog
wR2.out as follow:

I am also getting very poor results. I found a couple bugs that I fixed but the results are still poor (Roughly 82.8% on Val set). Did you manage to resolve?

I am also getting very poor results.23% in ccpd_db