OpenSSL 3.2.0-alpha1 04-test_bio_dgram fails on NonStop

A significant number of tests are failing on NonStop. I have not included the detailed test log because of the size. Configuration is at the bottom. I can split this if they have different root causes:
Portion of the log:
04-test_bio_dgram.t .....................
        # INFO:  @ /home/randall/openssl/test/bio_dgram_test.c:125
        # # Testing with AF_INET6, local=0
        #
        # ERROR: (int) 'fd1 >= 0' failed @ /home/randall/openssl/test/bio_dgram_test.c:171
        # [-1] compared to [0]
        # 00000000:error:80001011:system library:BIO_socket:Address family not supported:/home/randall/openssl/crypto/bio/bio_sock2.c:51:calling socket()
        # 00000000:error:10000076:BIO routines:BIO_socket:unable to create socket:/home/randall/openssl/crypto/bio/bio_sock2.c:53:
        # OPENSSL_TEST_RAND_ORDER=1694107046
        not ok 2 - iteration 2
# ------------------------------------------------------------------------------
        # INFO:  @ /home/randall/openssl/test/bio_dgram_test.c:125
        # # Testing with AF_INET6, local=1
        #
        # ERROR: (int) 'fd1 >= 0' failed @ /home/randall/openssl/test/bio_dgram_test.c:171
        # [-1] compared to [0]
        # 00000000:error:80001011:system library:BIO_socket:Address family not supported:/home/randall/openssl/crypto/bio/bio_sock2.c:51:calling socket()
        # 00000000:error:10000076:BIO routines:BIO_socket:unable to create socket:/home/randall/openssl/crypto/bio/bio_sock2.c:53:
        # OPENSSL_TEST_RAND_ORDER=1694107046
        not ok 4 - iteration 4
# ------------------------------------------------------------------------------
    # OPENSSL_TEST_RAND_ORDER=1694107046
    not ok 1 - test_bio_dgram
# ------------------------------------------------------------------------------
        # ERROR: (int) 'i >= 9' failed @ /home/randall/openssl/test/bio_dgram_test.c:565
        # [6] compared to [9]
        # 00000000:error:10000070:BIO routines:dgram_pair_write:non-fatal or transient error:/home/randall/openssl/crypto/bio/bss_dgram_pair.c:1271:
        # OPENSSL_TEST_RAND_ORDER=1694107046
        not ok 5 - iteration 1
# ------------------------------------------------------------------------------
        # ERROR: (int) 'i >= 9' failed @ /home/randall/openssl/test/bio_dgram_test.c:565
        # [6] compared to [9]
        # 00000000:error:10000070:BIO routines:dgram_pair_write:non-fatal or transient error:/home/randall/openssl/crypto/bio/bss_dgram_pair.c:1271:
        # OPENSSL_TEST_RAND_ORDER=1694107046
        not ok 6 - iteration 2
# ------------------------------------------------------------------------------
    # OPENSSL_TEST_RAND_ORDER=1694107046
    not ok 2 - test_bio_dgram_pair
# ------------------------------------------------------------------------------
../../util/wrap.pl ../../test/bio_dgram_test => 255
not ok 1 - running bio_dgram_test
04-test_bio_dgram.t ..................... 1/? ----------------------------------
#   Failed test 'running bio_dgram_test'
#   at /home/randall/openssl/util/perl/OpenSSL/Test/Simple.pm line 77.
04-test_bio_dgram.t ..................... Dubious, test returned 1 (wstat 256, 0

Configuration:
Command line (with current working directory = .):

    perl ./Configure nonstop-nsx_64 --with-rand-seed=rdcpu enable-fips

Perl information:

    perl
    5.30.3 for NSV-D-nonstop_kernel

Enabled features:

    acvp-tests
    apps
    argon2
    aria
    async
    autoalginit
    autoerrinit
    autoload-config
    bf
    blake2
    bulk
    cached-fetch
    camellia
    capieng
    cast
    chacha
    cmac
    cmp
    cms
    comp
    ct
    deprecated
    des
    dgram
    dh
    docs
    dsa
    dso
    dtls
    dynamic-engine
    ec
    ec2m
    ecdh
    ecdsa
    ecx
    egd
    engine
    err
    filenames
    fips
    fips-securitychecks
    gost
    http
    idea
    legacy
    loadereng
    md4
    mdc2
    module
    multiblock
    nextprotoneg
    ocb
    ocsp
    padlockeng
    pic
    pinshared
    poly1305
    posix-io
    psk
    quic
    rc2
    rc4
    rdrand
    rfc3779
    rmd160
    scrypt
    secure-memory
    seed
    shared
    siphash
    siv
    sm2
    sm2-precomp
    sm3
    sm4
    sock
    srp
    srtp
    sse2
    ssl
    ssl-trace
    static-engine
    stdio
    tests
    tls
    ts
    ui-console
    whirlpool
    tls1
    tls1-method
    tls1_1
    tls1_1-method
    tls1_2
    tls1_2-method
    tls1_3
    dtls1
    dtls1-method
    dtls1_2
    dtls1_2-method

Disabled features:

    afalgeng            [not-linux]      OPENSSL_NO_AFALGENG
    asan                [default]        OPENSSL_NO_ASAN
    asm                 [no asm_arch]    OPENSSL_NO_ASM
    brotli              [default]        OPENSSL_NO_BROTLI
    brotli-dynamic      [default]        OPENSSL_NO_BROTLI_DYNAMIC
    buildtest-c++       [default]
    winstore            [not-windows]    OPENSSL_NO_WINSTORE
    crypto-mdebug       [default]        OPENSSL_NO_CRYPTO_MDEBUG
    default-thread-pool [cascade]        OPENSSL_NO_DEFAULT_THREAD_POOL
    devcryptoeng        [default]        OPENSSL_NO_DEVCRYPTOENG
    ec_nistp_64_gcc_128 [default]        OPENSSL_NO_EC_NISTP_64_GCC_128
    external-tests      [default]        OPENSSL_NO_EXTERNAL_TESTS
    fuzz-afl            [default]        OPENSSL_NO_FUZZ_AFL
    fuzz-libfuzzer      [default]        OPENSSL_NO_FUZZ_LIBFUZZER
    ktls                [default]        OPENSSL_NO_KTLS
    makedepend          [unavailable]
    md2                 [default]        OPENSSL_NO_MD2 (skip crypto/md2)
    msan                [default]        OPENSSL_NO_MSAN
    rc5                 [default]        OPENSSL_NO_RC5 (skip crypto/rc5)
    sctp                [default]        OPENSSL_NO_SCTP
    tfo                 [default]        OPENSSL_NO_TFO
    thread-pool         [cascade]        OPENSSL_NO_THREAD_POOL
    threads             [config]
    trace               [default]        OPENSSL_NO_TRACE
    ubsan               [default]        OPENSSL_NO_UBSAN
    unit-test           [default]        OPENSSL_NO_UNIT_TEST
    uplink              [no uplink_arch] OPENSSL_NO_UPLINK
    weak-ssl-ciphers    [default]        OPENSSL_NO_WEAK_SSL_CIPHERS
    zlib                [default]        OPENSSL_NO_ZLIB
    zlib-dynamic        [default]        OPENSSL_NO_ZLIB_DYNAMIC
    zstd                [default]        OPENSSL_NO_ZSTD
    zstd-dynamic        [default]        OPENSSL_NO_ZSTD_DYNAMIC
    ssl3                [default]        OPENSSL_NO_SSL3
    ssl3-method         [default]        OPENSSL_NO_SSL3_METHOD

Config target attributes:

    AR => "ar",
    ARFLAGS => "qc",
    CC => "c99",
    HASHBANGPERL => "/usr/bin/env perl",
    RANLIB => "ranlib",
    RC => "windres",
    bn_ops => "SIXTY_FOUR_BIT",
    build_file => "Makefile",
    build_scheme => [ "unified", "unix" ],
    cflags => "-g -O2 -Wextensions -Wnowarn=203,220,272,734,770,1506 -Wbuild_neutral_library -Wverbose -Wtarget=tns/x -Wsystype=oss -Wlp64 -WIEEE_float",
    cppflags => "",
    defines => [ "OPENSSL_BUILDING_OPENSSL", "OPENSSL_VPROC=\$(OPENSSL_VPROC)", "_XOPEN_SOURCE", "_XOPEN_SOURCE_EXTENDED=1", "_TANDEM_SOURCE", "__NSK_OPTIONAL_TYPES__", "B_ENDIAN" ],
    disable => [ "threads" ],
    dso_scheme => "DLFCN",
    enable => [ "egd" ],
    ex_libs => "-lrld",
    includes => [  ],
    lflags => "-Wxld=\"-set systype oss\" -Wxld=\"-set data_model lp64\" -Wxld=\"-set floattype ieee_float\"",
    lib_cflags => "",
    lib_cppflags => "",
    lib_defines => [  ],
    module_cflags => "",
    module_cxxflags => undef,
    module_ldflags => "-Wshared",
    multibin => "64",
    multilib => "64",
    perl => "/usr/bin/perl",
    perl_platform => "Unix",
    shared_argfileflag => "-Wxld_obey=",
    shared_cflag => "",
    shared_defflag => "-Wxld_obey=",
    shared_defines => [  ],
    shared_extension => ".so",
    shared_ldflag => "-Wshared",
    shared_rcflag => "",
    shared_target => "nonstop-shared",
    sys_id => "TANDEM",
    thread_defines => [  ],
    thread_scheme => "(unknown)",
    unistd => "<unistd.h>",

Recorded environment:

    AR =
    ARFLAGS =
    AS =
    ASFLAGS =
    BUILDFILE =
    CC =
    CFLAGS =
    CPP =
    CPPDEFINES =
    CPPFLAGS =
    CPPINCLUDES =
    CROSS_COMPILE =
    CXX =
    CXXFLAGS =
    HASHBANGPERL =
    LD =
    LDFLAGS =
    LDLIBS =
    MT =
    MTFLAGS =
    OPENSSL_LOCAL_CONFIG_DIR =
    PERL =
    RANLIB =
    RC =
    RCFLAGS =
    RM =
    WINDRES =
    __CNF_CFLAGS =
    __CNF_CPPDEFINES =
    __CNF_CPPFLAGS =
    __CNF_CPPINCLUDES =
    __CNF_CXXFLAGS =
    __CNF_LDFLAGS =
    __CNF_LDLIBS =

Makevars:

    AR              = ar
    ARFLAGS         = qc
    ASFLAGS         =
    CC              = c99
    CFLAGS          =
    CPPDEFINES      =
    CPPFLAGS        =
    CPPINCLUDES     =
    CXXFLAGS        =
    HASHBANGPERL    = /usr/bin/env perl
    LDFLAGS         =
    LDLIBS          =
    PERL            = perl
    RANLIB          = ranlib
    RC              = windres
    RCFLAGS         =

NOTE: These variables only represent the configuration view.  The build file
template may have processed these variables further, please have a look at the
build file for more exact data:
    Makefile

build file:

    Makefile

build file templates:

    Configurations/common0.tmpl
    Configurations/unix-Makefile.tmpl

The 80-test_cmp_http.t test seems to have a problem:
# The results of this test will end up in test-runs/test_cmp_http
Starting1..6
Current directory is /home/randall/openssl/test/recipes/80-test_cmp_http_data/Mock
Launching Mock server: ../../../../util/wrap.pl ../../../../apps/openssl cmp -config server.cnf
Mock server PID=1409286238
cmp: error setting accept on port 0
00000000:error:80001011:system library:BIO_socket:Address family not supported:/home/randall/openssl/crypto/bio/bio_sock2.c:51:calling socket()
00000000:error:10000076:BIO routines:BIO_socket:unable to create socket:/home/randall/openssl/crypto/bio/bio_sock2.c:53:
00000000:error:80001011:system library:acpt_state:Address family not supported:/home/randall/openssl/crypto/bio/bss_acpt.c:233:calling socket(::, 0)
00000000:error:10000076:BIO routines:acpt_state:unable to create socket:/home/randall/openssl/crypto/bio/bss_acpt.c:236:
Mock server output: cmp_main:/home/randall/openssl/apps/cmp.c:3080:CMP info: using section(s) 'cmp' of OpenSSL configuration file 'server.cnf'
Killing Mock server with PID=0Killed

The server cannot have a PID=0 (that is the process itself), so appears to be causing the perl test wrapper process to terminate.

Moved 80-test_cmp_http to #22014

There are some considerations on the platform that may apply here. Binding to specific sockets from 0-1024 are restricted to root only. I cannot change that. This applies to both UDP and TCP. I am wondering whether that is the cause of both errors above.

There are some considerations on the platform that may apply here. Binding to specific sockets from 0-1024 are restricted to root only. I cannot change that. This applies to both UDP and TCP. I am wondering whether that is the cause of both errors above.

It seems unlikely since the same restrictions apply to Linux.
The failing of the dgram_pair test is curious since that is an in-memory BIO and doesn't do anything with OS sockets.

There are some considerations on the platform that may apply here. Binding to specific sockets from 0-1024 are restricted to root only. I cannot change that. This applies to both UDP and TCP. I am wondering whether that is the cause of both errors above.

It seems unlikely since the same restrictions apply to Linux.
The failing of the dgram_pair test is curious since that is an in-memory BIO and doesn't do anything with OS sockets.

I can rerun the tests with V=1 if that would help. Now that I'm past 80-test_cmp_http (still failing but I have a fix for it killing perl), there might be other tests that fail but I can see the output.

I threw this into gdb at BIO_socket. The call is resolved to socket(AF_INET6, SOCK_DGRAM, 17), which normally should work. It appears that IPv6 is not configured on the box I am testing, so we are getting EAFNOSUPPORT. Perhaps we need to add a check and bypass the test if IPv6 is not enabled. I am urgently working with the sysadmin to get this changed as it seems to be hitting multiple tests.
Note: It may take a week to get this changed because this is a shared system and I have to give lots of notice for this type of change. Would that cause problems for other pull requests that might result to get this into 3.2.0-beta?

Nonetheless — I think this is a bug if our code is breaking without IPv6.

Yeah, we absolutely must support non-IPv6 boxes. They are still common.

Wasn't @levitte looking at similar IPv6 issues on VMS? Is this the same as those?

Wasn't @levitte looking at similar IPv6 issues on VMS? Is this the same as those?

I do not think it is the same problem - in @levitte's case the IPv6 support was AFAIK completely non-present on the VMS builder, but there was still some attempt to resolve IPv6 addresses. However here the IPv6 support is enabled in headers, etc. so this builds with IPv6 support, but IPv6 is not configured, so the tests using IPv6 will fail.
So IMO completely different issue. We might want to try to configure a VM without IPv6 to test this.

We reconfigured to have IPv6 running on the box. This moves the problem somewhat.  As of now, 80-test_cmp_http and 79-test_http work. 04-test_bio_dgram still fails, with the full log below. I am renaming this case to focus on that.
04-test_bio_dgram.t ..
# The results of this test will end up in test-runs/test_bio_dgram
1..1
    # Subtest: ../../test/bio_dgram_test
    1..2
        # Subtest: test_bio_dgram
        1..4
        # INFO:  @ /home/randall/openssl/test/bio_dgram_test.c:119
        # # Testing with AF_INET, local=0
        #
        ok 1 - iteration 1
        # INFO:  @ /home/randall/openssl/test/bio_dgram_test.c:125
        # # Testing with AF_INET6, local=0
        #
        ok 2 - iteration 2
        # INFO:  @ /home/randall/openssl/test/bio_dgram_test.c:119
        # # Testing with AF_INET, local=1
        #
        ok 3 - iteration 3
        # INFO:  @ /home/randall/openssl/test/bio_dgram_test.c:125
        # # Testing with AF_INET6, local=1
        #
        ok 4 - iteration 4
    ok 1 - test_bio_dgram
        # Subtest: test_bio_dgram_pair
        1..3
        # ERROR: (int) 'i >= 9' failed @ /home/randall/openssl/test/bio_dgram_test.c:565
        # [6] compared to [9]
        # 00000000:error:10000070:BIO routines:dgram_pair_write:non-fatal or transient error:/home/randall/openssl/crypto/bio/bss_dgram_pair.c:1271:
        # OPENSSL_TEST_RAND_ORDER=1694181873
        not ok 5 - iteration 1
        # ERROR: (int) 'i >= 9' failed @ /home/randall/openssl/test/bio_dgram_test.c:565
        # [6] compared to [9]
        # 00000000:error:10000070:BIO routines:dgram_pair_write:non-fatal or transient error:/home/randall/openssl/crypto/bio/bss_dgram_pair.c:1271:
        # OPENSSL_TEST_RAND_ORDER=1694181873
        not ok 6 - iteration 2
        ok 7 - iteration 3
    # OPENSSL_TEST_RAND_ORDER=1694181873
    not ok 2 - test_bio_dgram_pair
../../util/wrap.pl ../../test/bio_dgram_test => 255
not ok 1 - running bio_dgram_test

#   Failed test 'running bio_dgram_test'
#   at /home/randall/openssl/util/perl/OpenSSL/Test/Simple.pm line 77.
# Looks like you failed 1 test of 1.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests

The test seems to be failing at
https://github.com/openssl/openssl/blob/ebce766bb82b472eb3b796c5ee5ee2c19beb71f5/crypto/bio/bss_dgram_pair.c#L1246
with
 b->rbuf
$12 = {
  start = 0x10001b310 "",
  len = 17408,
  count = 17126,
  idx = {17126, 0}
}

Inside, at ring_buf_head_tail, we hit a line on the 6th iteration, where max_len starts at 17408, then hits max_len = r->len - r->count;, where r->len=17408 and r->count=17408 causing max_len to be 0. This causes the test to report the above transient error. This does not seem correct.

I should point out that I think the DGRAM size limit is configured to be 8K on the IP stack I am using.

I should point out that I think the DGRAM size limit is configured to be 65K on the IP stack I am using.

That is not the default MTU used by the dgram_pair BIO here. - it should be set to 1472. What do you see if you display the contents of the bio_dgram_pair_st inside the BIO?

To help debug this further could you print out the values of max_len and r->count each time we enter ring_bug_head_tail for the first test that fails?
Also what is the value of mtu1 in test_bio_dgram_pair (test/bio_dgram_test.c)

In test_bio_dgram_pair, mtu1=1472
Iteration 0: max_len = 17408, r={ start = 0x10001b310 "", len = 17408, count = 0, idx = {0, 0}}
Iteration 1: max_len = 15344, r={ start = 0x10001b310 "", len = 17408, count = 2064, idx = {2064, 0}}
Iteration 2: max_len = 14901, r={ start = 0x10001b310 "", len = 17408, count = 2507, idx = {2507, 0}}
Iteration 3: max_len = 12837, r={ start = 0x10001b310 "", len = 17408, count = 4571, idx = {4571, 0}}
Iteration 4: max_len = 11777, r={ start = 0x10001b310 "", len = 17408, count = 5631, idx = {5631, 0}}
Iteration 5: max_len = 9713, r={ start = 0x10001b310 "", len = 17408, count = 7695, idx = {7695, 0}}
Iteration 6: max_len = 8620, r={ start = 0x10001b310 "", len = 17408, count = 8788, idx = {8788, 0}}
Iteration 7: max_len = 6556, r={ start = 0x10001b310 "", len = 17408, count = 10852, idx = {10852, 0}}
Iteration 8: max_len = 6392, r={ start = 0x10001b310 "", len = 17408, count = 11016, idx = {11016, 0}}
Iteration 9: max_len = 4328, r={ start = 0x10001b310 "", len = 17408, count = 13080, idx = {13080, 0}}
Iteration 10: max_len = 3176, r={ start = 0x10001b310 "", len = 17408, count = 14232, idx = {14232, 0}}
Iteration 11: max_len = 1112, r={ start = 0x10001b310 "", len = 17408, count = 16296, idx = {16296, 0}}
Iteration 12: max_len = 237, r={ start = 0x10001b310 "", len = 17408, count = 17171, idx = {17171, 0}}
Iteration 13: max_len = 17408, r={ start = 0x10001b310 "", len = 17408, count = 17408, idx = {17408, 0}}
max_len=0 at the end of this call.
In dgram_pair_write_inner, total_written=237, which seems wrong because sizeof(hdr)=2064 and that drops through to the failure code. saved_count=17171. Is it possible that handling count is incorrect?

The following lines are not intuitive to me (probably my lack of understanding of the intent of the code):
https://github.com/openssl/openssl/blob/d2873946dfaff5537ea3d1adf3890e33a3f276ff/crypto/bio/bss_dgram_pair.c#L1181-L1182
Is this saying that if sz > 0 then dst_len = sz but what happened to the rest of the contents of the buffer? Or is sz the remaining size? It might help to have that variable descriptive of what it is.
The lines following:
https://github.com/openssl/openssl/blob/d2873946dfaff5537ea3d1adf3890e33a3f276ff/crypto/bio/bss_dgram_pair.c#L1187-L1189
Seem to make the above assignment strange because sz will always be set to 0 if dst_len > sz for the next iteration of the loop.

I should point out that I think the DGRAM size limit is configured to be 65K on the IP stack I am using.

That is not the default MTU used by the dgram_pair BIO here. - it should be set to 1472. What do you see if you display the contents of the bio_dgram_pair_st inside the BIO?

@t8m I verified that the MTU is 1472. This situation is perplexing. If there is any help you can provide so I can investigate more/fix this, I would appreciate it. (see previous comments).

sizeof(hdr)=2064

!!!! Oh wow! That's huge! This is the cause of the problem. On my system this value is 232.
This is the line in the test that is failing:
https://github.com/openssl/openssl/blob/84f371a130dbe7a46595fbabd274f152a0e6385f/test/bio_dgram_test.c#L561-L566
The default write buffer size is 17408. Each datagram consists of a random length up to the max mtu of 1472 plus the datagram header (2064 bytes in your case).
On my system, in the worst case scenario where every datagram has the maximum length, this means we could fit 17408 / (1472 + 232) == 10.2 datagrams into the default buffer size. This test conservatively assumes that we should be able to fit at least 9 in.
But on your system the test is failing with:
        # ERROR: (int) 'i >= 9' failed @ /home/randall/openssl/test/bio_dgram_test.c:565
        # [6] compared to [9]

i.e. it only managed to fit 6 datagrams in. This seems about right. If the average datagram length is half the mtu (736). Then each datagram plus the header is 736+2064 == 2800. With the buffer size of 17408 we can fit 17408 / 2064 == 6.2.

The hdr struct is defined here:
https://github.com/openssl/openssl/blob/e22ebb893e2f44bd08f69f9ce4ccfc5e4d2990e2/crypto/bio/bss_dgram_pair.c#L243-L246
I assume it is sizeof(BIO_ADDR) which is larger than expected (perhaps 1000 bytes assuming sizeof(size_t) is 64??)
a BIO_ADDR is simply a union containing these things:
https://github.com/openssl/openssl/blob/e22ebb893e2f44bd08f69f9ce4ccfc5e4d2990e2/include/internal/bio_addr.h#L16-L27
Where all of those types are defined by the system. It's very strange that they are so large on Non-stop!

Possible fix for this in #22058.
@rsbeckerca - can you test the fix?

My guess is what is going on here is that struct sockaddr_un is massive due to how NonStop decided to implement it.
TLDR: struct sockaddr (well, struct sockaddr_storage) is a tagged union structure. This means it needs to have a fixed maximum size. This is a problem with UNIX domain sockets because they are identified by paths. As I understand it Linux basically cheesed a solution to this problem by defining a fixed maximum length for UNIX domain socket paths:
struct sockaddr_un {
	sa_family_t sun_family;
	char sun_path[108];
};

So while Linux supports long path names in most cases, for UNIX domains sockets specifically you have a special limitation in order to keep the size of struct sockaddr_storage reasonable.
My guess is NonStop decided they didn't care about memory usage and to choose some massive size for sockaddr_un, say 2k.
The obvious solution here is to base our testing on the platform's sizeof(BIO_ADDR).
Another option is to avoid using BIO_ADDR here as we never actually use UNIX domain sockets. We could just define a union of struct sockaddr_in and struct sockaddr_in6. Probably a better option but more work and more duplication. Probably not worth us doing this for Nonstop but a PR would probably be accepted if the memory wastage is considered an issue.

(You could also work around this temporarily by building with -DOPENSSL_NO_UNIX_SOCK, of course)

Possible fix for this in #22058.
@rsbeckerca - can you test the fix?

@mattcaswell Running a build/test cycle now. Will run the individual test then a full regression, just to make sure.

I literally just updated the commit with the latest suggestion from @t8m...:-)

I literally just updated the commit with the latest suggestion from @t8m...:-)

I just picked up this change and am restarting. No worries.

@mattcaswell I'm sorry to say that we only got on more iteration through the loop:
04-test_bio_dgram.t ..
# The results of this test will end up in test-runs/test_bio_dgram
1..1
    # Subtest: ../../test/bio_dgram_test
    1..2
        # Subtest: test_bio_dgram
        1..4
        # INFO:  @ /home/randall/openssl/test/bio_dgram_test.c:119
        # # Testing with AF_INET, local=0
        #
        ok 1 - iteration 1
        # INFO:  @ /home/randall/openssl/test/bio_dgram_test.c:125
        # # Testing with AF_INET6, local=0
        #
        ok 2 - iteration 2
        # INFO:  @ /home/randall/openssl/test/bio_dgram_test.c:119
        # # Testing with AF_INET, local=1
        #
        ok 3 - iteration 3
        # INFO:  @ /home/randall/openssl/test/bio_dgram_test.c:125
        # # Testing with AF_INET6, local=1
        #
        ok 4 - iteration 4
    ok 1 - test_bio_dgram
        # Subtest: test_bio_dgram_pair
        1..3
        ok 5 - iteration 1
        # ERROR: (int) 'i >= 9' failed @ /home/randall/openssl/test/bio_dgram_test.c:568
        # [7] compared to [9]
        # 00000000:error:10000070:BIO routines:dgram_pair_write:non-fatal or transient error:/home/randall/openssl/crypto/bio/bss_dgram_pair.c:1272:
        # OPENSSL_TEST_RAND_ORDER=1694446579
        not ok 6 - iteration 2
        ok 7 - iteration 3
    # OPENSSL_TEST_RAND_ORDER=1694446579
    not ok 2 - test_bio_dgram_pair

Well, at least we got one more iteration further :-)
Will take another look tomorrow.

Well, at least we got one more iteration further :-)
Will take another look tomorrow.

Sounds good to me. I'm probably not gong to be able to do much about it tomorrow - at the NonStop TBC Conference talking about Ransomware Attack Vectors tomorrow.

@mattcaswell I can't decipher what, but my gut is telling me that there is a missing loop termination condition causing the lengths to reset, which then causes the error. I also cannot figure out why the header appears to large on NonStop. There is one pad in the old sockaddr (not used) that is 64 bytes. This is perplexing to me also. It should not be 2062.

NO_UNIX_SOCK

I think this might break access to PRNGD, which is accessed via UNIX sockets. Going through TCP is an option, but a painful one. We only use PRNGD on ia64, not x86, though.

I have added another commit to #22058 to address the failure in the other iteration. I was actually able to recreate this on my local machine by artificially increasing the size of the BIO_ADDR structure.

@rsbeckerca - please could you retest?

@mattcaswell Hurrah!
04-test_bio_dgram.t ..
# The results of this test will end up in test-runs/test_bio_dgram
1..1
    # Subtest: ../../test/bio_dgram_test
    1..2
        # Subtest: test_bio_dgram
        1..4
        # INFO:  @ /home/randall/openssl/test/bio_dgram_test.c:120
        # # Testing with AF_INET, local=0
        #
        ok 1 - iteration 1
        # INFO:  @ /home/randall/openssl/test/bio_dgram_test.c:126
        # # Testing with AF_INET6, local=0
        #
        ok 2 - iteration 2
        # INFO:  @ /home/randall/openssl/test/bio_dgram_test.c:120
        # # Testing with AF_INET, local=1
        #
        ok 3 - iteration 3
        # INFO:  @ /home/randall/openssl/test/bio_dgram_test.c:126
        # # Testing with AF_INET6, local=1
        #
        ok 4 - iteration 4
    ok 1 - test_bio_dgram
        # Subtest: test_bio_dgram_pair
        1..3
        ok 5 - iteration 1
        ok 6 - iteration 2
        ok 7 - iteration 3
    ok 2 - test_bio_dgram_pair
../../util/wrap.pl ../../test/bio_dgram_test => 0
ok 1 - running bio_dgram_test
ok
All tests successful.
Files=1, Tests=1,  1 wallclock secs ( 0.01 usr  0.00 sys +  0.63 cusr  0.00 csys =  0.64 CPU)
Result: PASS

Great! Are there any other outstanding NonStop test failures - or do you get a clean run now?

Great! Are there any other outstanding NonStop test failures - or do you get a clean run now?

Restarting the CI stream. Need a couple of hours but it should be clean.

Great! Are there any other outstanding NonStop test failures - or do you get a clean run now?

Restarting the CI stream. Need a couple of hours but it should be clean.

@mattcaswell I am pulling master, merging your fix, and running the NonStop CI.

@mattcaswell Clean run!

Great! Thanks for confirming.

@rsbeckerca / @mattcaswell
I am.also.facing same issue , can you please provide the fix to resolve this issue.
Thanks ,
Veeranji

I am trying to install the same version and  facing the same error . If you could remember the fix ,kindly share with me.
If anyone is aware of the fix . Kindly share

@rsbeckerca I am trying to install the same version and facing the same error . If you could remember the fix ,kindly share with me.
If anyone is aware of the fix . Kindly share

Is there a reason you need the openssl-3.2-alpha1 version instead of openssl-3.2 ?

@rsbeckerca
I am not installing alpha1, where as same issue appeared even for 3.2 sha .

@rsbeckerca I am not installing alpha1, where as same issue appeared even for 3.2 sha .

Can you open a new issue? This is working correctly on NonStop. We would need to see your configuration and the test output.

@veeranji11 to clarify, this particular issue is closed as resolved. In order to work on your situation, we need a new case opened, with details about how you built the code and the errors you are seeing - they may be the same or different, but the configuration is crucial. Use the following command at the highest directory in your clone (probably openssl):
perl configdata.pm --dump

That will tell the team how you built the code. Also, please send the output of the following command:
git show --oneline -s

which will show the commit you are using for the build. Also include the errors.
Please do this using the New Issue button from the Issues tab in the openssl/openssl repository.


sysctl: Add complete sysctl support

We support most of the namespaced sysctls today as we pass them to
libcontainer as part of the OCI spec. libcontainer then applies them
for the container after veryfing they can be applied.
However, the verification fails for network related sysctls as
libcontainer expects a separate network namespace for network
sysctls. This check fails for us as we create network namspace on
the host side. To fix these, apply the network sysctls manually
and purge them from the spec, leaving other sysctls to be applied
by libcontainer.
Fixes 472
Signed-off-by: Archana Shinde<EMAIL_ADDRESS>

cc @egernst @mcastelino @sboeuf

/test

Hi @amshinde - thanks for raising. Could you add a unit test or two for this change?

/test

@jodh-intel Done.

/test
but we got a bunch of CI fails - so, let's nudge them all again. If they fail again, we'd better go stare at them @amshinde

/test

Thats right @bergwolf

@grahamwhaley CI is failing! I looked through the logs and I am seeing this which looks like the point of error, unless I missed something else.
•Build timed out (after 5 minutes). Marking the build as aborted.
Build was aborted
Performing Post build task...
Match found for :.* : True
Logical operation result is TRUE
Running script  : #!/bin/bash

cc @chavafg

@amshinde - yeah, I checked two different CI fails, and they both timed out, but at different places. It is feeling like this PR may be disrupting something somewhere, but hard to tell what. Are you able to run the tests locally in a loop and see if you can reproduce at all?
and then @nitkon - looks like the power8 CI has gotten upset by a file it cannot remove (my guess, a hung test is keeping the library file open). You may have to hand fix that...
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete '/home/jenkins/workspace/kata-containers-agent-Power8-ubuntu-16-04-PR-initrd/go/pkg/linux_ppc64le/github.com/kata-containers/tests/vendor/github.com/BurntSushi/toml.a'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
ERROR: Cannot delete workspace: Unable to delete '/home/jenkins/workspace/kata-containers-agent-Power8-ubuntu-16-04-PR-initrd/go/pkg/linux_ppc64le/github.com/kata-containers/tests/vendor/github.com/BurntSushi/toml.a'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.

the timeouts shouldn't happen anymore.
There is an issue with latest image, see https://github.com/kata-containers/tests/issues/1270
for the moment, we are skipping the test that was timing out.
/test

All CIs have passed except Power8, which has been failing for the agent.
Merging this.

Power8 slave looks like it has been failing to pass anything for 11 days. /cc @nitkon
http://jenkins.katacontainers.io/computer/Power8_slave01/builds


⚠ Bofrost.be has degraded performance

In 0d9bc01, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1116 ms

Resolved: Bofrost.be performance has improved in 58e29fd.


Change globally unique to persistent for identifier bullet

Based on the work from creating the best practice companion document to the primer suggest changing "globally unique" to "persistent" for the identifier bullet such that it reads "Use persistent identifiers"

https://github.com/ESIPFed/bds-primer-best-practices/issues/8

Done


#750 fix: docs build fails due to outdated links

Related issue
Closes #750 Docs build on develop fails
Scope of work

Update some SCSS imports in Vue files for docs
Update a link in a MD file to a MD file which was moved
Use colors as a named import for colorsValues. This is already declared in other files and hence expected in the auto-generation script.

@EduardDopler sorry for my overlooking, thanks for your vigilance 🚀


IOrchardHost.GetShellContext - Creating and setting up tenants

I'm trying to create and setup tenants by code.
When I write this code:
_tenantService.CreateTenant(tenantSettings);
var shellContext = _orchardHost.GetShellContext(newTenantShellSettings);

shellContext  is null. Then I changed the code to this:
_tenantService.CreateTenant(tenantSettings);
_orchardHost.EndRequest();
_orchardHost.BeginRequest();
var shellContext = _orchardHost.GetShellContext(newTenantShellSettings);

shellContext is valid but it invalidates the login state (I mean log out me).
What's wrong all with this?

Please don't open issues for questions.


Default value for not null fields in migration?

Hey! Really love this package.
I have a question to the following scenario:
The user table needs to be extended with a new column. This new column must not be null.
In my example we add a bool field like that:
@Model()
abstract class User {
  // EXISTING SCHEMA
  @PrimaryKey()
  String get id;
  String get email;
  String get passwordHash;
  int get registerDate;

  // PENDING MIGRATION
  bool get emailConfirmed;
}

The build will succeed but the migration will fail when applying the changes:
Select a database to update:
Database: connecting to  at <IP_ADDRESS>...
Database: connected
Getting schema changes of
=========================
++ COLUMN users.email_confirmed
=========================
Do you want to apply these changes? (yes/no): yes
Database schema changed, applying updates now:
---
ALTER TABLE "users"
ADD COLUMN "email_confirmed" bool NOT NULL
Transaction finished with error: PostgreSQLSeverity.unknown 23502: Spalte Â»email_confirmedÂ« von Relation Â»usersÂ« enthÃ¤lt NULL-Werte Table: users Column: email_confirmed
========================
---
ALL CHANGES REVERTED, EXITING

This is to be expected because stormberry would need to update every existing entry with a default value.
Therefore my question now, would it be possible to define default values for fields so that they:

are optional in the constructor
the migration could automaticly update the database if we add a new not null collumn to a table with existing entries.

Hope i got this right :)
Cheers

Thanks, thats a cool idea.


fix(cardinalities): enable the required toggle programmatically (DEV-1790)

…for newly assigned properties
resolves DEV-1790

Comment is changed. Tx!


[tool_crash] FileSystemException: Failed to decode data using encoding 'utf-8', null

Command
flutter --quiet assemble --no-version-check --dep

I tried to build the appbundle from the working source.
Debug mode is working as well.
It was building for appbundle in the past.
After install system, I got the error.
Hoping to get help for this error.
Thank you.

Closing this as a duplicate of https://github.com/flutter/flutter/issues/65021.


v2.6.5 Release Task - Merge PR for KDM, Feature-Charts and System-Charts

Rancher Backend Engineering Task
This is a Rancher backend engineering task for (https://github.com/rancher/rancher/issues/37315) for the rancher engineering team 3 & the Release captain to:

[ ] Merge the PRs for KDM, Feature-Charts, and System-Charts

Finished 11 May 2022


Code formatting and consistency

@fsbraun I really love this project, and it looks like it has great potential!
There is some lack of consistency in indentation, imports, use of f-strings where not needed, etc. Would a PR that addresses all of this be appreciated?
My recommendations would be to use the following tools:

isort

In several places imports can be optimized and cleaned up. For instance, cms_plugins.py imports models, flow, and views on separate lines, but they can be combined to a single import.

black

black has become the defacto standard in the python world. We would need to use the # fmt: on/off comments to keep the visual style (multiple spaces) you use for demonstrating the workflows, which is very effective.

flake8

There are some minor issues such as f-strings used with no variables, and a local variable that is assigned-to, but never used. While these don't cause any problems, they could be cleaned up.

djhtml

Provides a nice consistent indentation style for django templates.

I would love your thoughts on this. If this all sounds good, please let me know and I will submit a PR to implement these changes.

I very much appreciate the idea! I have made some adjustments but have not automated the process. I would appreciate a PR very much..

Looks like you added the pre-commit hooks before I was able to get around to it :smile: Nice!
Should we consider c1b8894 to fully close this issue out?

Let me know if you come across issues with the pre-commits. In the mean time: Closed with commit c1b8894.


Add Jlama support

Issue

https://github.com/langchain4j/langchain4j/issues/1350
Change
Adds support for Jlama engine
General checklist

[X] There are no breaking changes
[X] I have added unit and integration tests for my change
[X] I have manually run all the unit and integration tests in the module I have added/changed, and they are all green
[X] I have manually run all the unit and integration tests in the core and main modules, and they are all green

[X] I have added/updated the documentation
[ ] I have added an example in the examples repo (only for "big" features)
[ ] I have added/updated Spring Boot starter(s) (if applicable)

Checklist for adding new model integration

[X] I have added my new module in the BOM

Should be all set

I also noticed langchain4j uses tokenizers directly, is that something you'd want native?  Jlama supports sentence-piece and bpe tokenizers for the HF models.  Let me know I'll do a separate PR

I also noticed langchain4j uses tokenizers directly, is that something you'd want native? Jlama supports sentence-piece and bpe tokenizers for the HF models. Let me know I'll do a separate PR

I am not sure this is required at this point, but feel free to open a PR if you think it will be useful.
Tokenizers are usually used in LC4j for a few reasons:

to estimate token usage before sending the request to chat/language models (to control the cost)
to split documents into smaller segments (for RAG)
In TokenWindowChatMemory to make sure ChatMemory keeps only N tokens worth of chat messages (to control the cost)

@tjake could you please add Jlama example to https://github.com/langchain4j/langchain4j-examples?
Would be great if there were instructions on how to properly run it with --enable-preview and other preconditions

Sure, will do tonight

Added https://github.com/langchain4j/langchain4j-examples/pull/89


Will this framework works in iOS

Hi All,
I wanted to use Object detection API with Unity3D and support both Android and iOS platform.
I understood it works in Android, what about iOS?

Hi @Sharathmk99,
Unfortunately, I don't have an iOS device to test it, but from first glance I don't see why it wouldn't work. If you can try it and provide feedback that'd be great.

I'm gonna close this issue for now, feel free to reopen if you have any specific problem with deploying on iOS.


Loss values during training

Thank you for the awesome work. I am currently training the FOMM model with 20 keypoints. Can you please let me know what should be the optimal value for each of the losses:
currently it is like: perceptual - 116.30898; equivariance_value - 0.28680; equivariance_jacobian - 0.64459
So after when I can start trying out the models for testing. Thank you

perceptual should be around 82. Rest is not that meaningfull.


Use goreleaser for releases to quay.io

Thanks for the PR @friedrichg. I was also about to push the Dockerfile for the project.
I have encountered an error while trying to run the image. I built and run the image but I got this error message:
exec /usr/bin/auth-gateway: no such file or directory
The commands I run:

docker build -t auth-gateway .
docker run -p 8080:80 auth-gateway gateway/testdata/configuration.yaml

@dogukanteber Install Goreleaser https://goreleaser.com/install/?h=ins
and then use
goreleaser release --snapshot

Thank you @friedrichg :heart:


PDF.JS Upgrade questions

I just downloaded the newest version of pdf.js on my local machine (I used to have version 1.0.1040). What is new in this latest version? I also noticed that the printer icon from this one is gone, was it taken out? Thanks.

Also I noticed that in the demo version (http://mozilla.github.io/pdf.js/web/viewer.html) it seems to have  fullscreen capabilities. Is there any way I can include that in my version?

I also noticed that the printer icon from this one is gone
I noticed that in the demo version (http://mozilla.github.io/pdf.js/web/viewer.html) it seems to have fullscreen capabilities

Downloaded from http://mozilla.github.io/pdf.js/getting_started/#download viewer shall have all capabilities of http://mozilla.github.io/pdf.js/web/viewer.html. See also https://github.com/mozilla/pdf.js/wiki/Frequently-Asked-Questions .
Printing and presentation mode capabilities functions are tested against a web browser capabilities (e.g. mozPrintCallback or fullscreenEnabled), so they may be disabled if deployed not as http://mozilla.github.io/pdf.js/web/viewer.html or different browser used.
Closing as answered.

OK thank you, but do you recommend the stable or beta version?

For production we always recommend the stable version. The beta version is usually for testing purposes. For development of PDF.js you can use the master branch on GitHub.


Add possibility to monitor a list of URLs instead of single URL

It would be nice, if you could add a file with a list of URLs that get checked instead of a single URL.

Good idea! How would you want to proceed with tolerance and XPath parameters? Would you suggest to configure them separate for each URL (see below) or globally for all? What would be appropriate for your use case?
# (url, tolerance, xpath)
https://url1.com, 75, //body
https://url2.com, 0, //div[name="article-container"]
...

Good idea! How would you want to proceed with tolerance and XPath parameters? Would you suggest to configure them separate for each URL (see below) or globally for all? What would be appropriate for your use case?
# (url, tolerance, xpath)
https://url1.com, 75, //body
https://url2.com, 0, //div[name="article-container"]
...

Separate for each URL would be nice of course :) What do you think of json as input?
{
  "url_list": [
    {
      "url": "https://url1.com",
      "tolerance": 75,
      "xpath": "body"
    },
    {
      "url": "https://url2.com",
      "tolerance": 0,
      "xpath": "div[name="article-container"]"
    }
  ]
}

Separate for each URL would be nice of course :) What do you think of json as input?
{
  "url_list": [
    {
      "url": "https://url1.com",
      "tolerance": 75,
      "xpath": "body"
    },
    {
      "url": "https://url2.com",
      "tolerance": 0,
      "xpath": "div[name="article-container"]"
    }
  ]
}

A JSON-formatted scrape config like the one you suggested seems to be a good option.
I'm thinking about keeping the logic for performing multiple scrapes outside of the actual watcher and provide a Bash script instead. It would read JSON config, iterate the array of URLs and parameters and call watcher.py for each entry. Doing so wouldn't add additional complexity to the actual code and have control logic separated from business logic. The only drawback would be the fact that Windows users wouldn't be able to use that functionality unless we also provide a Batch- or PowerShell script.

A JSON-formatted scrape config like the one you suggested seems to be a good option.
I'm thinking about keeping the logic for performing multiple scrapes outside of the actual watcher and provide a Bash script instead. It would read JSON config, iterate the array of URLs and parameters and call watcher.py for each entry. Doing so wouldn't add additional complexity to the actual code and have control logic separated from business logic. The only drawback would be the fact that Windows users wouldn't be able to use that functionality unless we also provide a Batch- or PowerShell script.

Yeah that's a good point, might be better to do it this way.

Yeah that's a good point, might be better to do it this way.


[WFCORE-4251] Provide ability to list module dependencies for a deployment and subdeployment

A feature that enables a command to list the module dependencies of a specific deployment and subdeployments.
It is included only a test case for non ear deployments. Ear deployment processors are in wildfly-full, so a new PR will be added by @zhantemirov to cover this section of the test plan.
Jira issue: https://issues.jboss.org/browse/WFCORE-4251
Analysis doc: https://github.com/wildfly/wildfly-proposals/pull/159
Community doc: https://github.com/wildfly/wildfly/pull/12041

Core - Full Integration Build 8327 outcome was UNKNOWN using a merge of dc7b81f2761dd24310fe7415f3ac1a7da5999aba
Summary: Canceled (Tests failed: 1 (1 new), passed: 4636, ignored: 128) Build time: 02:43:35
Failed tests
org.jboss.as.test.multinode.transaction.TransactionInvocationTestCase.testSameTransactionEachCall: javax.ejb.EJBException: EJBCLIENT000082: Cannot outflow the remote transaction "Local transaction (delegate=TransactionImple < ac, BasicAction: 0:ffff7f000101:-3af69f57:5c5c1cfb:10 status: ActionStatus.RUNNING >, owner=Local transaction context for provider JBoss JTA transaction provider)" as its timeout elapsed
	at org.jboss.ejb.protocol.remote.EJBClientChannel.processInvocation(EJBClientChannel.java:471)
	at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.lambda$handleDone$0(RemoteEJBReceiver.java:91)
	at org.xnio.FinishedIoFuture.addNotifier(FinishedIoFuture.java:79)
	at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.handleDone(RemoteEJBReceiver.java:76)
	at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.handleDone(RemoteEJBReceiver.java:74)
	at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208)
	at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720)
	at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:949)
	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
	at java.lang.Thread.run(Thread.java:748)
Caused by: javax.transaction.SystemException: EJBCLIENT000082: Cannot outflow the remote transaction "Local transaction (delegate=TransactionImple < ac, BasicAction: 0:ffff7f000101:-3af69f57:5c5c1cfb:10 status: ActionStatus.RUNNING >, owner=Local transaction context for provider JBoss JTA transaction provider)" as its timeout elapsed
	at org.jboss.ejb.protocol.remote.EJBClientChannel.writeTransaction(EJBClientChannel.java:581)
	at org.jboss.ejb.protocol.remote.EJBClientChannel.processInvocation(EJBClientChannel.java:367)
	... 12 more
------- Stdout: -------
04:56:37,095 INFO  [org.jboss.threads] (main) JBoss Threads version 2.3.3.Final
04:56:37,269 INFO  [org.jboss.as.arquillian.container.managed.ManagedDeployableContainer] (main) Starting container with: [/usr/java/jdk1.8.0_152/bin/java, -D[Standalone], -Dorg.jboss.ejb.client.wildfly-testsuite-hack=true, -Xmx512m, -XX:MetaspaceSize=128m, -Djboss.dist=/store/work/tc-work/37b47ae8b9c60325/full/testsuite/integration/multinode/../../../build/target/wildfly-16.0.0.Beta1-SNAPSHOT, -Dmaven.repo.local=/store/repository, -Djava.net.preferIPv4Stack=false, -Djava.net.preferIPv6Addresses=true, -Djboss.bind.address=::1, -Djboss.bind.address.unsecure=::1, -Djboss.bind.address.private=::1, -Djboss.bind.address.management=::1, -Djboss.default.multicast.address=ff13::1, -server, -Dts.timeout.factor=100, -Dnode0=::1, -Dnode1=::1, -Dmcast=ff13::1, -Dmcast.ttl=0, -Djbossas.ts.submodule.dir=/store/work/tc-work/37b47ae8b9c60325/full/testsuite/integration/multinode, -Djbossas.ts.integ.dir=/store/work/tc-work/37b47ae8b9c60325/full/testsuite/integration/multinode/.., -Djbossas.ts.dir=/store/work/tc-work/37b47ae8b9c60325/full/testsuite/integration/multinode/../.., -Djbossas.project.dir=/store/work/tc-work/37b47ae8b9c60325/full/testsuite/integration/multinode/../../.., -Djava.io.tmpdir=/store/work/tc-work/37b47ae8b9c60325/full/testsuite/integration/multinode/target, -Djboss.node.name=multinode-client, -ea, -Djboss.home.dir=/store/work/tc-work/37b47ae8b9c60325/full/testsuite/integration/multinode/target/jbossas-multinode-client, -Dorg.jboss.boot.log.file=/store/work/tc-work/37b47ae8b9c60325/full/testsuite/integration/multinode/target/jbossas-multinode-client/standalone/log/server.log, -Dlogging.configuration=file:/store/work/tc-work/37b47ae8b9c60325/full/testsuite/integration/multinode/target/jbossas-multinode-client/standalone/configuration/logging.properties, -jar, /store/work/tc-work/37b47ae8b9c60325/full/testsuite/integration/multinode/target/jbossas-multinode-client/jboss-modules.jar, -mp, /store/work/tc-work/37b47ae8b9c60325/full/testsuite/integration/multinode/target/jbossas-multinode-client/modules:/store/work/tc-work/37b47ae8b9c60325/full/build/target/wildfly-16.0.0.Beta1-SNAPSHOT/modules:/store/work/tc-work/37b47ae8b9c60325/full/testsuite/integration/multinode/target/modules, org.jboss.as.standalone, -Djboss.home.dir=/store/work/tc-work/37b47ae8b9c60325/full/testsuite/integration/multinode/target/jbossas-multinode-client, -Djboss.server.base.dir=/store/work/tc-work/37b47ae8b9c60325/full/testsuite/integration/multinode/target/jbossas-multinode-client/standalone, -Djboss.server.log.dir=/store/work/tc-work/37b47ae8b9c60325/full/testsuite/integration/multinode/target/jbossas-multinode-client/standalone/log, -Djboss.server.config.dir=/store/work/tc-work/37b47ae8b9c60325/full/testsuite/integration/multinode/target/jbossas-multinode-client/standalone/configuration, -Dts.wildfly.version=16.0.0.Beta1-SNAPSHOT, -c=standalone.xml]
04:56:37,318 INFO  [org.jboss.remoting] (main) JBoss Remoting version 5.0.8.Final
04:56:37,377 INFO  [org.xnio] (main) XNIO version 3.6.5.Final
04:56:37,397 INFO  [org.xnio.nio] (main) XNIO NIO Implementation Version 3.6.5.Final
04:56:37,542 INFO  [org.wildfly.security] (main) ELY00001: WildFly Elytron version 1.8.0.CR2
 [0m04:56:38,281 INFO  [org.jboss.modules] (main) JBoss Modules version 1.9.0.Final
 [0m [0m04:56:38,883 INFO  [org.jboss.msc] (main) JBoss MSC version 1.4.5.Final
 [0m [0m04:56:38,905 INFO  [org.jboss.threads] (main) JBoss Threads version 2.3.3.Final
 [0m [0m04:56:39,196 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: WildFly Full 16.0.0.Beta1-SNAPSHOT (WildFly Core 8.0.0.Beta5-SNAPSHOT) starting
 [0m [0m04:56:40,824 INFO  [org.wildfly.security] (ServerService Thread Pool -- 26) ELY00001: WildFly Elytron version 1.8.0.CR2
 [0m [0m04:56:41,918 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
 [0m [0m04:56:41,977 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 32) WFLYCTL0028: Attribute 'username' in the resource at address '/subsystem=remoting/remote-outbound-connection=remote-ejb-connection' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
 [0m [0m04:56:41,977 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 32) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=remoting/remote-outbound-connection=remote-ejb-connection' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
 [0m [0m04:56:41,978 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 32) WFLYCTL0028: Attribute 'protocol' in the resource at address '/subsystem=remoting/remote-outbound-connection=remote-ejb-connection' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
 [0m [0m04:56:41,999 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 35) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
 [0m [0m04:56:42,104 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
 [0m [0m04:56:42,143 INFO  [org.xnio] (MSC service thread 1-6) XNIO version 3.6.5.Final
 [0m [0m04:56:42,169 INFO  [org.xnio.nio] (MSC service thread 1-6) XNIO NIO Implementation Version 3.6.5.Final
 [0m [0m04:56:42,299 INFO  [org.wildfly.extension.microprofile.config.smallrye._private] (ServerService Thread Pool -- 58) WFLYCONF0001: Activating WildFly MicroProfile Config Subsystem
 [0m [0m04:56:42,351 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 49) WFLYCLINF0001: Activating Infinispan subsystem.
 [0m [0m04:56:42,356 INFO  [org.jboss.as.jaxrs] (ServerService Thread Pool -- 51) WFLYRS0016: RESTEasy version 3.6.2.Final
 [0m [0m04:56:42,379 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 72) WFLYWS0002: Activating WebServices Extension
 [0m [33m04:56:42,382 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 70) WFLYTX0013: The node-identifier attribute on the /subsystem=transactions is set to the default value. This is a danger for environments running multiple servers. Please make sure the attribute value is unique.
 [0m [0m04:56:42,391 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 68) WFLYSEC0002: Activating Security Subsystem
 [0m [0m04:56:42,404 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 62) WFLYNAM0001: Activating Naming Subsystem
 [0m [0m04:56:42,408 INFO  [org.wildfly.extension.microprofile.opentracing] (ServerService Thread Pool -- 61) WFLYTRACEXT0001: Activating MicroProfile OpenTracing Subsystem
 [0m [0m04:56:42,418 INFO  [org.wildfly.extension.microprofile.health.smallrye] (ServerService Thread Pool -- 59) WFLYHEALTH0001: Activating Eclipse MicroProfile Health Subsystem
 [0m [0m04:56:42,422 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 56) WFLYJSF0007: Activated the following JSF Implementations: [main]
 [0m [0m04:56:42,429 INFO  [org.wildfly.extension.microprofile.metrics.smallrye] (ServerService Thread Pool -- 60) WFLYMETRICS0001: Activating Eclipse MicroProfile Metrics Subsystem
 [0m [0m04:56:42,448 INFO  [org.wildfly.extension.io] (ServerService Thread Pool -- 50) WFLYIO001: Worker 'default' has auto-configured to 8 core threads with 64 task threads based on your 4 available processors
 [0m [0m04:56:42,538 INFO  [org.jboss.remoting] (MSC service thread 1-2) JBoss Remoting version 5.0.8.Final

@jmesnil the analysis, test plan and precheck for this are done, and the community docs PR is basically fine (minor polish). So this is good to go from the non-PR-review POV.


debugger: fix debugging of examples

Previously, invocating the action #debug on an SBExample failed because SBSmallError did not understand #focus.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 0.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
0

Relevant Lines:
0

💛  - Coveralls


Revert commits from 9fcd40c to HEAD

The goal with this change is to get the codebase back to a known-good state (0.1.27-26), and then start re-applying changes in later PRs.
I had to make some changes to get the build to pass:

Update Rush to latest, with latest pnpm 5.x
Add a bin file to @misk/cli that always exists, so that Rush's linking step doesn't break
Add prebuild to the build step, and miskweb prebuild to the examples' prebuild steps

I've also applied these changes:

Remove .vscode/settings.json ( c1b5247 , #1335)

Ok, now that this is a green build, I can try and apply some changes back on this PR


overlapping path delays not respected

If one set_max_delay constraint terminates (-to) on combinational cell where another one starts (-from) one or both of these constraints are lost. I'm attaching a simple example. Two first SDC lines should create two timing paths:
set_max_delay -from [get_pins {_1_/A}] -to [get_ports out0] 0.01000
set_max_delay -from [get_ports in0] -to [get_pins {_1_/X}] 0.02000
#set_max_delay -from [get_ports in0] -to [get_pins {_1_/A}] 0.02000

But none are reported:
openroad -exit sdc_ignored.tcl
OpenROAD v2.0-3265-gc1c315118
This program is licensed under the BSD-3 license. See the LICENSE file for details.
Components of this program may be licensed under more restrictive licenses which must be honored.
[INFO ODB-0222] Reading LEF file: sky130_fd_sc_hd__nom.tlef
[INFO ODB-0223]     Created 13 technology layers
[INFO ODB-0224]     Created 25 technology vias
[INFO ODB-0226] Finished LEF file:  sky130_fd_sc_hd__nom.tlef
[INFO ODB-0222] Reading LEF file: sky130_fd_sc_hd.lef
[WARNING ODB-0220] WARNING (LEFPARS-2008): NOWIREEXTENSIONATPIN statement is obsolete in version 5.6 or later.
The NOWIREEXTENSIONATPIN statement will be ignored. See file sky130_fd_sc_hd.lef at line 2.

[INFO ODB-0225]     Created 441 library cells
[INFO ODB-0226] Finished LEF file:  sky130_fd_sc_hd.lef
No paths found.

If we comment out any one of these two first SDC lines the remaining path will be reported as normal:
openroad -exit sdc_ignored.tcl
OpenROAD v2.0-3265-gc1c315118
...
[INFO ODB-0226] Finished LEF file:  sky130_fd_sc_hd.lef
Startpoint: in0 (input port)
Endpoint: _1_/X (internal path endpoint)
Path Group: **default**
Path Type: max

  Delay    Time   Description
---------------------------------------------------------
   0.00    0.00 v input external delay
   0.00    0.00 v in0 (in)
   0.07    0.07 v _1_/X (sky130_fd_sc_hd__buf_1)
           0.07   data arrival time

   0.02    0.02   max_delay
   0.00    0.02   output external delay
           0.02   data required time
---------------------------------------------------------
           0.02   data required time
          -0.07   data arrival time
---------------------------------------------------------
          -0.05   slack (VIOLATED)

Startpoint: in0 (input port)
Endpoint: _1_/X (internal path endpoint)
Path Group: **default**
Path Type: max

  Delay    Time   Description
---------------------------------------------------------
   0.00    0.00 ^ input external delay
   0.00    0.00 ^ in0 (in)
   0.07    0.07 ^ _1_/X (sky130_fd_sc_hd__buf_1)
           0.07   data arrival time

   0.02    0.02   max_delay
   0.00    0.02   output external delay
           0.02   data required time
---------------------------------------------------------
           0.02   data required time
          -0.07   data arrival time
---------------------------------------------------------
          -0.05   slack (VIOLATED)

Interestingly if we comment out the second SDC line and uncomment initially commented third, which terminates on another pin of the same cell, then only the third path will be reported:
openroad -exit sdc_ignored.tcl
OpenROAD v2.0-3265-gc1c315118
...

This is actually an OpenSTA issue.
It is fixed by commit c7debe5 but not integrated into openroad (yet)


Update dust icon

weather-dust icon just got added into material icons https://github.com/Templarian/MaterialDesign/issues/6658. That means it will be available around HA 2022.09. When it will be added dust icon needs to be changed to this icon using methods like tsunami https://github.com/MrBartusek/MeteoalarmCard/blob/master/src/data.ts#L34-L56
The estimated time of implementing this is ≈  Septeber 2022

The estimated time of implementing this is ≈ Septeber 2022

sike


ci: pin the NimSkull version

Summary
Pin the NimSkull version used to the most recent working one
(0.1.0-dev.21431), making it the official minimum supported version of
NimSkull to compile the various Phy program with.

Notes For Reviewers

this fixes the CI failures caused by the most-recent broken NimSkull release
skully also needs a pinned version, as it imports compiler modules directly

@saem: The change should be uncontroversial, and I'm merging this eagerly in order to unblock the #97.


Remove legacy context

Follow-up #631

any update on this?

any update on this?

Maybe @oliviertassinari could take a look and merge it if it's suitable as it seems to be.

@ykzts merge it?

@marklai1998  I don't have permission to merge into this repository.

@oliviertassinari can you merge this one and release a new version with the recent updates? Thanks in advance.


no report with clustering precision modified

f4d4887
Description
If the precision parameter in the OTU-clustering command is changed, the preprocessing part runs just fine, but the analysis part does not.
Steps to reproduce

Dry-run preprocessing part with agc clustering method.
Modify the precision parameter in the cluster command to cluster.split(fasta=current, count=current, taxonomy=current, cutoff=0.03, large=T, method=agc, precision=1000);
Run the modified script.

Expected behaviour
Once the preprocessing step is done, mothulity should run the analysis steps and output the final report in the HTML file.
Actual behaviour
The final report HTML is not generated. The shared (both full and subsampled) and tax.summary files are in the analysis/OTU/ directory. The alpha biodiversity part runs as expected, though the beta directory contains only the mothur's log files.

This is caused by the subsampled shared file name mismatch. mothulity expects the subsampled file to be named <analysis_name>.0.03.subsample.shared while it is actually <analysis_name>.0.030.subsample.shared so it cannot be copied.
The solution should be to detect the file name, similarly as it is with the initial shared file here.

Not really. The label is detected correctly in a numerical sense but the trailing zero is present in the raw shared file but truncated during CSV read-in. So it is a bug in mothulity.read_info_shared

@Setnea timing :1st_place_medal:


getAccessTokenWithPopup results in blank popup

Describe the problem
Trying to use getAccessTokenWithPopup to authenticate to an API. When that function runs, a popup with a blank screen appears.

This is the link that popup is trying to resolve:
https://inclusivecareco.us.auth0.com/authorize?scope=openid%20profile%20email%20read%3Ausers%20admin%3Aall&client_id=KbacVudgFLdeqHDDfyxXYVkb6flNvFNl&redirect_uri=http%3A%2F%2Flocalhost%3A8000&audience=https%3A%2F%2Fapi.inclusivecareco.org&display=popup&response_type=code&response_mode=web_message&state=M01Pb0w1UkFkT0dXVThtN1pnWDF1Uk5ORkpHYjhBWjhXY21nVm1UMjUtMg%3D%3D&nonce=SGlyTmNDRHJxdGlGcWtfbEhqSGwyT1lhUEN%2BSWJiZXdMSDJ0QlNUaGJfdQ%3D%3D&code_challenge=GsrjhS0M04HNLyuBqI_xDlik5IgvTfpua6_NQv37W8I&code_challenge_method=S256&auth0Client=eyJuYW1lIjoiYXV0aDAtcmVhY3QiLCJ2ZXJzaW9uIjoiMS44LjAifQ%3D%3D

I don't see any errors in the console for that popup.
What was the expected behavior?
I expected to see a popup with content asking for authorization.
Reproduction

Detail the steps taken to reproduce this error, and whether this issue can be reproduced consistently or if it is intermittent.
Note: If clear, reproducable steps or the smallest sample app demonstrating misbehavior cannot be provided, we may not be able to follow up on this bug report.

Have credentials for the user that has sufficient permissions to access the API layer ('admin:all' must be present on the token in this case)
Run gatsby develop after cloning this repo: https://github.com/OutBoulderCounty/icc-ui
After logging in at http://localhost:8000/admin, attempt to access http://localhost:8000/admin/forms
The blank popup appears

Auth0 Client Details:

Client ID: KbacVudgFLdeqHDDfyxXYVkb6flNvFNl
Domain: inclusivecareco.us.auth0.com

Environment

Please provide the following:

Version of auth0-react used: 1.8.0
Which browsers have you tested in? Chrome and Firefox
Which framework are you using, if applicable (Angular, React, etc): React & Gatsby
Other modules/plugins/libraries that might be involved: React Query

Here is the relevant location in the code: https://github.com/OutBoulderCounty/icc-ui/blob/main/src/components/admin/forms.tsx#L50-L53
I have tried changing the display attribute to "popup", "page", and "touch" and none of these are working.

Closing this due to no activity. Feel free to reply and we can reopen if needed.


Decouple individual improvements/features from each other

Select all the relevant HTML elements first, then have the features modify them. This paves the way for features to be turned on/off.

Essentially complete.


Improve introduction (better cover historical context)

As a referee to our f1000 article notes:

There is a historical context that Ankenbrand et al. miss in discussing biom-format and subsequently imply that the biom-format is more widely adopted than being field specific format. If the authors leave the introduction more general, then I would suggest they include more background on the history of high-throughput data storage and reproducibility in programmatic languages, perhaps starting with the Minimum Information About a Microarray Experiment - MIAME format [1] and exprSet classes developed in R about 15 years ago before the genomics standards consortium (formed in 2005), for which biom-format is a member.

Paulson J. Referee Report For: biojs-io-biom, a BioJS component for handling data in Biological Observation Matrix (BIOM) format [version 1; referees: 3 approved with reservations]. F1000Research 2016, 5:2348 (doi: 10.5256/f1000research.10362.r16545)
[1] Brazma A, Hingamp P, Quackenbush J, Sherlock G, Spellman P, Stoeckert C, Aach J, Ansorge W, Ball CA, Causton HC, Gaasterland T, Glenisson P, Holstege FC, Kim IF, Markowitz V, Matese JC, Parkinson H, Robinson A, Sarkans U, Schulze-Kremer S, Stewart J, Taylor R, Vilo J, Vingron M: Minimum information about a microarray experiment (MIAME)-toward standards for microarray data.Nat Genet. 2001; 29 (4): 365-71

you are first author :) I dont have a clue about the history as well

Added a little historical background as suggested by the reviewer.


pull request for e2e test at 01/12/2017 05:15:11

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


fix(typings): clean up typings

Fix Nuxt's Context/App typings
Add Vuex Store types

@graberzz
Thank you! I'm not familiar with type definition well. So a contribution like this PR is very helpful.


make name of Compiler.Scanning.Method accessible from Python code

Current master branch of Cython does not work when Plex.Scanner.trace is enabled. For example, the following code
import Cython.Compiler.Scanning
Cython.Compiler.Scanning.trace_scanner = 1
import cython
cython.inline("return 2")

generates the following output and error messages
State 3, 1/0:'bol' -->
State 48
State 48, 1/0:u'p' -->
blocked
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-0f503002be7a> in <module>()
----> 1 cython.inline("return 2")
[omitted]
/Plex/Actions.c:1545)()

AttributeError: 'Cython.Compiler.Scanning.Method' object has no attribute '__name__'

Marking the name as readonly can fix this.

Thanks.
https://github.com/cython/cython/commit/da06b146d0dfbe6f65d22174d0187175f3c88a3b


fibonacci is recursive

Due to the fact that
EM.fibonacci(100);

is recursively implemented, the page will hang on large numbers as input.
A possible solution would be to put the function on a thread so the main thread doesn't get blocked.

It's looped now


traefik: init at 1.3.8

Motivation for this change
Traefik is used as a reverse proxy alternative to other tools such as nginx/caddy.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] Linux

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

I would prefer if traefik would be build from source instead. In the past I opened a pull request so that traefik would vendorize all their source: https://github.com/containous/traefik/pull/969,
so it should be easy to build it from the release source tarball.

Of course, I'll look into it.

@Mic92 I have updated to building traefik from source. There are a couple of things which feel a little hacky, but I couldn't find any docs on how to go about it.
I would appreciate any feedback on changes to make it more idiomatic.
Thanks

I reworked the package. Should be less hacky now.

Thanks @Mic92 & @vyp for the help.
Mic92, it looks considerably better now, and more robust!
What needs to be done next to move this forward?

Maybe a nixos module?

That is my next change, I'll start working on it


alasql USE without callback parameter gives error

Using alasql v0.3.9.
This works (in browser):
alasql('CREATE INDEXEDDB DATABASE IF NOT EXISTS geo;\
        ATTACH INDEXEDDB DATABASE geo; \
        USE geo;', function(){} );

This gives "Error: Database 'geo' does not exist" in the console:
alasql('CREATE INDEXEDDB DATABASE IF NOT EXISTS geo;\
        ATTACH INDEXEDDB DATABASE geo; \
        USE geo;');

I agree that it is not really awesome - but the reason is that you are trying to do a sync call to each statement in the second one. At the moment this assumes each statement is sync, but attatching the indexxed DB is async, so the geo table is not created yet.
I suggest using the lazy promise notation as follows:
alasql(['CREATE INDEXEDDB DATABASE IF NOT EXISTS geo',
           'ATTACH INDEXEDDB DATABASE geo',
           'USE geo'].then(function(){
                 // now geo is ready to be used
          }).catch(console.log)
´´´

Thank you for your reply. I can see that asynchronization execution might be the problem in my original code. However, this code, which I believe should be synchronous, also fails and prints "Error: Database 'geo2' does not exist" to the console:
alasql(['CREATE INDEXEDDB DATABASE IF NOT EXISTS geo2',
        'ATTACH INDEXEDDB DATABASE geo2',
        'USE geo2'])
       .then( function() { console.log("ok"); })
       .catch( function(err) { console.log(err); })

hmmmmm...

If I remove "IF NOT EXISTS" I get the same error message "Error: Database 'geo2' does not exist", but also:
Error: IndexedDB: Cannot create new database "geo2" because it already exists       alasql.js:16314:11
So it does not exist and it already exists... seems like alasql is having an existential crisis! :)


[REQUEST] View stashed stashes

Is it possible to view all stashed items?
I know PyCharm can do this, and I would really love to be able to do it in Atom too.

v7.12.0 has a new command Manage Stashes that you can use


simplify metadata workflow

ensure topics are lower case
delineate dataset topic and discovery metadata identifier
pin Shapely requirement to <2.0 (pywis-pubsub safeguard)
update discovery metadata encoding
s/center_id/centre_id/

Addresses #343, #344

Update: pending outcome of https://github.com/wmo-im/wis2-topic-hierarchy/issues/16

Validated in https://github.com/wmo-im/wis2-topic-hierarchy/issues/16 (going with center_id).


Design Thinking Guide

We need to create the basic guidance for preparing and submitting a DT Track project.

Develop the design thinking artifacts guide

[ ]   Develop the Mural Board for the Theme Builder
[ ]   Details for Plenary Sessions from participant Guide
[ ]   Sample user Story
[ ]   Sample materials
[ ]   Confirm Exit Critera
[ ]    PowerPoint Playback Brief

All changes are reflected via this PR: https://github.com/finos-labs/gaad-hackathon-template/pull/15
@vinomaster

@mdacic01 @mattmills097  Please update the DT Guide to point to our sample Board in Action to reflect a how to do example.

sample DT board added to DT guide in this PR: https://github.com/finos-labs/gaad-hackathon-template/pull/21


OpenSource : 2. Getting Started With Open Source

Description :

Getting Started With Open Source

Build a good Tech-Related background

Software products are built with different kinds of tech stacks and
also by following different architectural patterns. Therefore it is
better to study architectural patterns, design patterns, and
well-known technologies before you begin anything.

Note :

Changes should be made inside the Open_Source/ directory & Opensource branch.
Issue only for GWOC'21 contributors.
Issue will be assigned on a first come first serve basis, 1 Issue  ==  1 PR.

Must Follow : Contributing Guidelines & Code of Conduct before start Contributing.
Happy Contributing 🙌

please assign me

please assign me

Assigning this to you @priyanshuc4423
Happy Documenting 🙌

PR merged #164 , Closing this issue


Add strikethrough

Any plans to add a <s> strikethrough button?

Example: I am not sure certain this feature is used often.

@amdelamar you can easily create your own custom module for this.
See https://github.com/chmln/vue-wysiwyg/blob/master/src/editor/examples/customModule_simple.js for example

Ok, so its extensible... but do you have plans to add more buttons/features to this wysiwyg?
Because I have other buttons like img src="//" urls and separating <code> and <pre><code> for code snippets and blocks.
What are your plans for vue-wysiwyg?


Read me Corrections

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

😂 you only added a '.' at the end of the Line.


typo in calculation

This is actually still not accurate because my example is fundamentally incorrect.  I caught and fixed it in my article that I stole all the examples from, so I will have to port over the updated examples from there.

#2

Just fixed this


weapon cfg

please help me how to identify which weapon is for Ts and which is for CTs.

https://yapb.readthedocs.io/en/latest/configuring.html#weapon-priorities


[doc] pipeline doc typos/improvements

This PR fixes a few typos, rewrites a few sentences to hopefully be easier to understand.
And after reading this doc I have a questions:

It's not clear from the tutorial why TransformerBlockPipe needs to return mask.  And if there is some hidden reason, shouldn't it then return hidden too?

Why do you need the vars to be a single tensor or a tuple? This surely could be a problematic limitation since some layers could have forward args which aren't tensors. Perhaps a more dynamic approach could be used instead? e.g. with the automatic data remapper I posted here: https://github.com/pytorch/pytorch/issues/49961#issuecomment-753441248

wrt Tied Layers - could you please give an example? or at least a pointer to where it can be seen in action?

Please let me know if it'd be better to separate the questions into an Issue and not mix it with the PR. I was just thinking that perhaps the answers could be integrated into the tutorials, hence posted them here.
Thank you!

Thanks so much for your ongoing contributions to DeepSpeed!
Good questions!

It's not clear from the tutorial why TransformerBlockPipe needs to return mask. And if there is some hidden reason, shouldn't it then return hidden too?

Pipeline parallelism is somewhat rigid in that the input to each layer's forward() is always the output of the previous layer. When you have a stack of transformers, we need to return the mask in order to provide it to the next layer.  We don't return hidden because it's not needed by the next layer. The next layer uses output as its hidden input.
From a usability perspective, I'd love to be able to specify that some inputs are static and globally available (e.g., mask) so that we wouldn't need to forward it down the pipeline. No concrete plans for it yet, though.

Why do you need the vars to be a single tensor or a tuple?

Agreed that this is a limitation. As a starting place, we wanted to match torch.nn.Sequential functionality and that container has the same restriction. I'd like to be more flexible. The primary limitation is the communication code at stage boundaries. I've been working on a more flexible communication manager that would allow something like this and also remove some hacky code in the pipeline engine.

wrt Tied Layers - could you please give an example?

Definitely. I'm trying to release our Megatron GPT implementation asap. The word embeddings are tied because they are used on both the first and last stages.

Thanks so much for your ongoing contributions to DeepSpeed!
Good questions!

It's not clear from the tutorial why TransformerBlockPipe needs to return mask. And if there is some hidden reason, shouldn't it then return hidden too?

Pipeline parallelism is somewhat rigid in that the input to each layer's forward() is always the output of the previous layer. When you have a stack of transformers, we need to return the mask in order to provide it to the next layer.  We don't return hidden because it's not needed by the next layer. The next layer uses output as its hidden input.
From a usability perspective, I'd love to be able to specify that some inputs are static and globally available (e.g., mask) so that we wouldn't need to forward it down the pipeline. No concrete plans for it yet, though.

Why do you need the vars to be a single tensor or a tuple?

Agreed that this is a limitation. As a starting place, we wanted to match torch.nn.Sequential functionality and that container has the same restriction. I'd like to be more flexible. The primary limitation is the communication code at stage boundaries. I've been working on a more flexible communication manager that would allow something like this and also remove some hacky code in the pipeline engine.

wrt Tied Layers - could you please give an example?

Definitely. I'm trying to release our Megatron GPT implementation asap. The word embeddings are tied because they are used on both the first and last stages.

Thank you for all the other answers in the previous comment, @ShadenSmith.

It's not clear from the tutorial why TransformerBlockPipe needs to return mask. And if there is some hidden reason, shouldn't it then return hidden too?

Pipeline parallelism is somewhat rigid in that the input to each layer's forward() is always the output of the previous layer. When you have a stack of transformers, we need to return the mask in order to provide it to the next layer. We don't return hidden because it's not needed by the next layer. The next layer uses output as its hidden input.

OK, can then this example be expanded to show how the mask is fed to forward on subsequent calls in the pipeline?
https://www.deepspeed.ai/tutorials/pipeline/#inputs-and-outputs
The issue here is that there could be other inputs and it's not clear at all how the user should handle those. i.e. you're saying "The next layer uses output as its hidden input." but this is not generic enough.
I think all that code should be explicit in the open so that the user can see what's going on and then expand it to support other inputs.
I hope you are seeing how the provided example isn't helping with a general case.

Thank you for all the other answers in the previous comment, @ShadenSmith.

It's not clear from the tutorial why TransformerBlockPipe needs to return mask. And if there is some hidden reason, shouldn't it then return hidden too?

Pipeline parallelism is somewhat rigid in that the input to each layer's forward() is always the output of the previous layer. When you have a stack of transformers, we need to return the mask in order to provide it to the next layer. We don't return hidden because it's not needed by the next layer. The next layer uses output as its hidden input.

OK, can then this example be expanded to show how the mask is fed to forward on subsequent calls in the pipeline?
https://www.deepspeed.ai/tutorials/pipeline/#inputs-and-outputs
The issue here is that there could be other inputs and it's not clear at all how the user should handle those. i.e. you're saying "The next layer uses output as its hidden input." but this is not generic enough.
I think all that code should be explicit in the open so that the user can see what's going on and then expand it to support other inputs.
I hope you are seeing how the provided example isn't helping with a general case.

After studying this more I finally wrapped my head around this and I propose the following to make it crystal clear from the get going:

Rewrite the code as:

before:
class TransformerBlock(nn.Module)
    ...
    def forward(self, hidden, mask):
        hidden = self.compute(hidden, mask)
        return hidden

after:
class TransformerBlockPipe(TransformerBlock)
    def forward(self, inputs):
        hidden, mask = inputs
        hidden = super().forward(hidden, mask)
        return (hidden, mask)

and adding a note that:
"It's clear to see how an output of one stage becomes an input to the next stage, since the previous stage in the pipeline now returns a tuple of (hidden, mask) and the next stage expects besides self, a tuple of (hidden, mask).

One more piece is missing from that section of the tutorial. Since the original code expected just one non-tuple variable in return from TransformerBlock.forward you have to modify the caller from:

output = self.transformer_block(hidden, mask)

to:
output, _ = self.transformer_block_pipe(hidden, mask)

because otherwise output will contain a tuple, and not output
Please let me know if you feel this makes things easier to understand and I will use that in this PR.

After studying this more I finally wrapped my head around this and I propose the following to make it crystal clear from the get going:

Rewrite the code as:

before:
class TransformerBlock(nn.Module)
    ...
    def forward(self, hidden, mask):
        hidden = self.compute(hidden, mask)
        return hidden

after:
class TransformerBlockPipe(TransformerBlock)
    def forward(self, inputs):
        hidden, mask = inputs
        hidden = super().forward(hidden, mask)
        return (hidden, mask)

and adding a note that:
"It's clear to see how an output of one stage becomes an input to the next stage, since the previous stage in the pipeline now returns a tuple of (hidden, mask) and the next stage expects besides self, a tuple of (hidden, mask).

One more piece is missing from that section of the tutorial. Since the original code expected just one non-tuple variable in return from TransformerBlock.forward you have to modify the caller from:

output = self.transformer_block(hidden, mask)

to:
output, _ = self.transformer_block_pipe(hidden, mask)

because otherwise output will contain a tuple, and not output
Please let me know if you feel this makes things easier to understand and I will use that in this PR.

And as I'm trying to port t5 transformers to Pipe I see why there is a requirement of all inputs to be Tensors - because they have to be spliced into micro-batches.

If I got it right - they shouldn't just be tensors, they should be tensors with first dimension of batch size. I actually had a tensor that was tracking things, but it's a no-go, since it's of the wrong shape. So this pre-requisite can and probably should be clarified.

Surely, there should be a way to handle booleans here. So the input parts that are of batch size will be spliced into micro-batches, but parts that are on/off switches can be probably left alone - hmm, but then it might be difficult to put them back into outputs correctly - perhaps the inputs should have an optional leave_me_alone_inputs which should be replicated as is? So the sig would be:

def forward(self, inputs, leave_me_alone_inputs):
    ....
    return (inputs, leave_me_alone_inputs)

?
This would also allow passing random other inputs that don't need to be micro-batch-sliced.

Same issue happens to optional input parts, which could be None or a tensor. The Pipe needs to slice the tensor, but leave None alone.

I think I have an idea how to work around this by creating a Pipe wrapper at run time and initialize it with a late __init__ so overcome this. I will keep you posted.
Currently in transformers t5 I have 15 distinct input parts, some tensors of BS, some None, some Bool, some tensors that aren't of BS.

And as I'm trying to port t5 transformers to Pipe I see why there is a requirement of all inputs to be Tensors - because they have to be spliced into micro-batches.

If I got it right - they shouldn't just be tensors, they should be tensors with first dimension of batch size. I actually had a tensor that was tracking things, but it's a no-go, since it's of the wrong shape. So this pre-requisite can and probably should be clarified.

Surely, there should be a way to handle booleans here. So the input parts that are of batch size will be spliced into micro-batches, but parts that are on/off switches can be probably left alone - hmm, but then it might be difficult to put them back into outputs correctly - perhaps the inputs should have an optional leave_me_alone_inputs which should be replicated as is? So the sig would be:

def forward(self, inputs, leave_me_alone_inputs):
    ....
    return (inputs, leave_me_alone_inputs)

?
This would also allow passing random other inputs that don't need to be micro-batch-sliced.

Same issue happens to optional input parts, which could be None or a tensor. The Pipe needs to slice the tensor, but leave None alone.

I think I have an idea how to work around this by creating a Pipe wrapper at run time and initialize it with a late __init__ so overcome this. I will keep you posted.
Currently in transformers t5 I have 15 distinct input parts, some tensors of BS, some None, some Bool, some tensors that aren't of BS.

Hi @stas00 , thanks a ton for your contibutions and comments. Unfortunately I lost my first reply in a power outage. Here are a few quick comments and clarifications and I'll double back over the day.
DeepSpeed does not slice inputs into micro-batches, we just rely on the data loader to do that. We already have the concept of micro-batches for gradient accumulation and build off of that instead. Instead, we just next() the provided data iterator gradient_accumulation_steps number of times.
The forward() input limitations are partly a trade-off of pipeline parallelism, and partly engineering limitations. Pipeline modules use this forward implementation implicitly:
def forward(self, inputs):
    x = inputs
    for layer in self.layers:
        x = layer(x)
    return x

This contract lets us split self.layers and place them on different machines. There are two consequences here:

Inputs like booleans, etc. still need to "flow" into the layer unless they are default valued keyward args. That's where something like the TransformerBlockPipe subclass comes in, we can wrap the functionality of an existing module and just specialize the call into forward with any additional args. Within the pipeline, there's no call site since it's done implicitly by the engine.
Successive layers could reside on different machines, and so any inputs must be communicated by NCCL. This is why we are limited to tensor inputs/outputs today. I agree that your recursive to() would help in a non-distributed environment, but we need a bit more in order to send/recv (also, the receiving processes do not know the shape/type of data to receive without an initial handshake).  Maybe we could something like pickle more general data structures and send them as ByteTensors ?

Hi @stas00 , thanks a ton for your contibutions and comments. Unfortunately I lost my first reply in a power outage. Here are a few quick comments and clarifications and I'll double back over the day.
DeepSpeed does not slice inputs into micro-batches, we just rely on the data loader to do that. We already have the concept of micro-batches for gradient accumulation and build off of that instead. Instead, we just next() the provided data iterator gradient_accumulation_steps number of times.
The forward() input limitations are partly a trade-off of pipeline parallelism, and partly engineering limitations. Pipeline modules use this forward implementation implicitly:
def forward(self, inputs):
    x = inputs
    for layer in self.layers:
        x = layer(x)
    return x

This contract lets us split self.layers and place them on different machines. There are two consequences here:

Inputs like booleans, etc. still need to "flow" into the layer unless they are default valued keyward args. That's where something like the TransformerBlockPipe subclass comes in, we can wrap the functionality of an existing module and just specialize the call into forward with any additional args. Within the pipeline, there's no call site since it's done implicitly by the engine.
Successive layers could reside on different machines, and so any inputs must be communicated by NCCL. This is why we are limited to tensor inputs/outputs today. I agree that your recursive to() would help in a non-distributed environment, but we need a bit more in order to send/recv (also, the receiving processes do not know the shape/type of data to receive without an initial handshake).  Maybe we could something like pickle more general data structures and send them as ByteTensors ?

Inputs like booleans, etc. still need to "flow" into the layer unless they are default valued keyward args. That's where something like the TransformerBlockPipe subclass comes in, we can wrap the functionality of an existing module and just specialize the call into forward with any additional args. Within the pipeline, there's no call site since it's done implicitly by the engine.

Yes, this one I have already figured out and basically created a wrapper class to which I pass the flow True/False args via its __init__. So we are good here.

Successive layers could reside on different machines, and so any inputs must be communicated by NCCL. This is why we are limited to tensor inputs/outputs today. I agree that your recursive to() would help in a non-distributed environment, but we need a bit more in order to send/recv (also, the receiving processes do not know the shape/type of data to receive without an initial handshake).  Maybe we could something like pickle more general data structures and send them as ByteTensors ?

I think what would help you to help me is to show you exactly what I'm dealing with:

I have a few simple things like:
None, which may become a tensor - but it shouldn't be sliced or reconstructed on the way out.
There really should be a None type of tensor

Then I have a medium complexity things like:
(). which gets filled out by each stage - e.g. it accumulates all hidden_states, attentions, etc - I have 5 of these in t5
This one is difficult since now instead of accumulating say 6 values (for a stack of 6 blocks), I end up with 12, 18 or more depending on the number of chunks and so I'm attempting to re-construct them once we are out of the pipeline. Not simple but doable.

Same here - it shouldn't be sliced or reconstructed on the way out.
What I'm attempting at the moment is a closure, that I pass to the wrapper's __init__ and grow these accumulators via the closure.

And the one I'm fighting with at the moment is this:
past_key_values which is tuple of tuples of tensors, and the batch dimension is not even on the top level.
This one needs to be sliced but on the inside of a tuple, but since it's a tuple of a tuple I can't pass it with input - I'm trying to pass it to wrapper's __init__ instead and to do my own slicing from inside the wrapper, by deriving the size of the microbatch and counting each invocation (and hoping they are not async).
I'm thinking of adding some encoding logic where I convert the tuple of a tuple into a new stack of tensors, pushing the batch dimension to the front so that it get sliced correctly and then inside forward pick out the section that is relevant to that block. pretty nuts.

The original logic runs a simple loop, picks the right segment for each block in the loop from the first tuple, then takes chunks of that segment via the nested tuple and eventually gets a 2 tensor slice which it adds to hidden_state in self-attention. here is the structure I need to pass around:

Hope this helps to understand what I'm dealing with and I hope you have some practical ideas.
Thank you!

Inputs like booleans, etc. still need to "flow" into the layer unless they are default valued keyward args. That's where something like the TransformerBlockPipe subclass comes in, we can wrap the functionality of an existing module and just specialize the call into forward with any additional args. Within the pipeline, there's no call site since it's done implicitly by the engine.

Yes, this one I have already figured out and basically created a wrapper class to which I pass the flow True/False args via its __init__. So we are good here.

Successive layers could reside on different machines, and so any inputs must be communicated by NCCL. This is why we are limited to tensor inputs/outputs today. I agree that your recursive to() would help in a non-distributed environment, but we need a bit more in order to send/recv (also, the receiving processes do not know the shape/type of data to receive without an initial handshake).  Maybe we could something like pickle more general data structures and send them as ByteTensors ?

I think what would help you to help me is to show you exactly what I'm dealing with:

I have a few simple things like:
None, which may become a tensor - but it shouldn't be sliced or reconstructed on the way out.
There really should be a None type of tensor

Then I have a medium complexity things like:
(). which gets filled out by each stage - e.g. it accumulates all hidden_states, attentions, etc - I have 5 of these in t5
This one is difficult since now instead of accumulating say 6 values (for a stack of 6 blocks), I end up with 12, 18 or more depending on the number of chunks and so I'm attempting to re-construct them once we are out of the pipeline. Not simple but doable.

Same here - it shouldn't be sliced or reconstructed on the way out.
What I'm attempting at the moment is a closure, that I pass to the wrapper's __init__ and grow these accumulators via the closure.

And the one I'm fighting with at the moment is this:
past_key_values which is tuple of tuples of tensors, and the batch dimension is not even on the top level.
This one needs to be sliced but on the inside of a tuple, but since it's a tuple of a tuple I can't pass it with input - I'm trying to pass it to wrapper's __init__ instead and to do my own slicing from inside the wrapper, by deriving the size of the microbatch and counting each invocation (and hoping they are not async).
I'm thinking of adding some encoding logic where I convert the tuple of a tuple into a new stack of tensors, pushing the batch dimension to the front so that it get sliced correctly and then inside forward pick out the section that is relevant to that block. pretty nuts.

The original logic runs a simple loop, picks the right segment for each block in the loop from the first tuple, then takes chunks of that segment via the nested tuple and eventually gets a 2 tensor slice which it adds to hidden_state in self-attention. here is the structure I need to pass around:

Hope this helps to understand what I'm dealing with and I hope you have some practical ideas.
Thank you!

let's merge this as is?


longevity test fail when using lyvecloud storage

Describe the bug
https://buildkite.com/risingwave-test/longevity-kubebench/builds/238#01875ada-8dbc-4ff2-8625-876beeaa50b3
log:

To Reproduce
No response
Expected behavior
No response
Additional context
No response

@wcy-fdu rerun the test,
https://buildkite.com/risingwave-test/longevity-kubebench/builds/244#01876ede-19a1-4b9d-b977-d66ba35f9035
this time, it works fine.

This error is because compactor read a deleted sst/object:

"log": "2023-04-07T08:53:22.053740256Z stdout F 2023-04-07T08:53:22.053493Z ERROR risingwave_meta::barrier::recovery: inject_barrier failed err=Rpc error: gRPC error (Internal error): Actor 2 exit unexpectedly: Executor error: Storage error: Hummock error: ObjectStore failed with IO error Internal error: read "hummock_001/245/194.data" in block Some(BlockLocation { offset: 23848589, size: 469971 }) failed, error: NoSuchKey: The specified key does not exist."

However, after searching log, this object is not deleted by RisingWave(compactor), there is no log about Finish vacuuming SSTs 194. So I believe this is because this object is mistakenly deleted by others manually.
Just rerun the longevity test for lyvecloud storage for an hour, and it works fine, so let's close this issue.


Color correction?

Im not sure what it does exactly but seems like it doesn't do anything.

Try increasing your graphics settings in-game

Try increasing your graphics settings in-game

I play on high settings.

Then you have old version of Osiris, with outdated offsets, they have recently been changed

Then you have old version of Osiris, with outdated offsets, they have recently been changed

Like honestly, I downloaded new version 2 days ago and updated it this morning. eksdee

Update: I just realised it isn't working when Disable post processing is enabled.