Avoid undefined behavior with unaligned accesses

Fixes: #4983

Isn't this still a strict aliasing violation? (Or do we not care about those? Strict aliasing is rather obnoxious.)

Isn't this still a strict aliasing violation? (Or do we not care about those? Strict aliasing is rather obnoxious.)

No, it is no strict aliasing violation, because c points to char and that can alias anything.
Yes, I do care.

I compiled everything with my new -Wcast-align=strict warning,
and there are a few more places where unaligned accesses may happen.
Some look like real errors.  For instance the ctr128.c uses GETU32 to access
the CTR field even on strict alignment, but who says that ivec is 4-byte aligned?
crypto/aes/aes_ige.c: In function 'AES_ige_encrypt':
crypto/aes/aes_ige.c:58:32: warning: cast increases required alignment of target type [-Wcast-align]
             aes_block_t *ivp = (aes_block_t *) ivec;
                                ^
crypto/aes/aes_ige.c:59:33: warning: cast increases required alignment of target type [-Wcast-align]
             aes_block_t *iv2p = (aes_block_t *) (ivec + AES_BLOCK_SIZE);
                                 ^
crypto/aes/aes_ige.c:62:36: warning: cast increases required alignment of target type [-Wcast-align]
                 aes_block_t *inp = (aes_block_t *) in;
                                    ^
crypto/aes/aes_ige.c:63:37: warning: cast increases required alignment of target type [-Wcast-align]
                 aes_block_t *outp = (aes_block_t *) out;
                                     ^
crypto/aes/aes_ige.c:28:41: warning: cast increases required alignment of target type [-Wcast-align]
 # define load_block(d, s)        (d) = *(const aes_block_t *)(s)
                                         ^
.....

In file included from crypto/evp/e_aes_cbc_hmac_sha1.c:20:
crypto/modes/modes_lcl.h:90:27: warning: cast increases required alignment of target type [-Wcast-align]
 # define PUTU32(p,v)     *(u32 *)(p) = BSWAP4(v)
                           ^
crypto/evp/e_aes_cbc_hmac_sha1.c:365:9: note: in expansion of macro 'PUTU32'
         PUTU32(out + 0, ctx->A[i]);
         ^~~~~~
crypto/modes/modes_lcl.h:90:27: warning: cast increases required alignment of target type [-Wcast-align]
 # define PUTU32(p,v)     *(u32 *)(p) = BSWAP4(v)
                           ^
crypto/evp/e_aes_cbc_hmac_sha1.c:366:9: note: in expansion of macro 'PUTU32'
         PUTU32(out + 4, ctx->B[i]);
         ^~~~~~
crypto/modes/modes_lcl.h:90:27: warning: cast increases required alignment of target type [-Wcast-align]
 # define PUTU32(p,v)     *(u32 *)(p) = BSWAP4(v)
                           ^
crypto/evp/e_aes_cbc_hmac_sha1.c:367:9: note: in expansion of macro 'PUTU32'
         PUTU32(out + 8, ctx->C[i]);
         ^~~~~~
crypto/modes/modes_lcl.h:90:27: warning: cast increases required alignment of target type [-Wcast-align]
 # define PUTU32(p,v)     *(u32 *)(p) = BSWAP4(v)
                           ^
crypto/evp/e_aes_cbc_hmac_sha1.c:368:9: note: in expansion of macro 'PUTU32'
         PUTU32(out + 12, ctx->D[i]);
         ^~~~~~
crypto/modes/modes_lcl.h:90:27: warning: cast increases required alignment of target type [-Wcast-align]
 # define PUTU32(p,v)     *(u32 *)(p) = BSWAP4(v)
                           ^
crypto/evp/e_aes_cbc_hmac_sha1.c:369:9: note: in expansion of macro 'PUTU32'
         PUTU32(out + 16, ctx->E[i]);
         ^~~~~~

In file included from crypto/evp/e_aes_cbc_hmac_sha256.c:21:
crypto/modes/modes_lcl.h:90:27: warning: cast increases required alignment of target type [-Wcast-align]
 # define PUTU32(p,v)     *(u32 *)(p) = BSWAP4(v)
                           ^
crypto/evp/e_aes_cbc_hmac_sha256.c:377:9: note: in expansion of macro 'PUTU32'
         PUTU32(out + 0, ctx->A[i]);
         ^~~~~~
crypto/modes/modes_lcl.h:90:27: warning: cast increases required alignment of target type [-Wcast-align]
 # define PUTU32(p,v)     *(u32 *)(p) = BSWAP4(v)
                           ^
crypto/evp/e_aes_cbc_hmac_sha256.c:378:9: note: in expansion of macro 'PUTU32'
         PUTU32(out + 4, ctx->B[i]);
         ^~~~~~
crypto/modes/modes_lcl.h:90:27: warning: cast increases required alignment of target type [-Wcast-align]
 # define PUTU32(p,v)     *(u32 *)(p) = BSWAP4(v)
                           ^
crypto/evp/e_aes_cbc_hmac_sha256.c:379:9: note: in expansion of macro 'PUTU32'
         PUTU32(out + 8, ctx->C[i]);
         ^~~~~~
crypto/modes/modes_lcl.h:90:27: warning: cast increases required alignment of target type [-Wcast-align]
 # define PUTU32(p,v)     *(u32 *)(p) = BSWAP4(v)
                           ^
crypto/evp/e_aes_cbc_hmac_sha256.c:380:9: note: in expansion of macro 'PUTU32'
         PUTU32(out + 12, ctx->D[i]);
         ^~~~~~
crypto/modes/modes_lcl.h:90:27: warning: cast increases required alignment of target type [-Wcast-align]
 # define PUTU32(p,v)     *(u32 *)(p) = BSWAP4(v)
                           ^
crypto/evp/e_aes_cbc_hmac_sha256.c:381:9: note: in expansion of macro 'PUTU32'
         PUTU32(out + 16, ctx->E[i]);
         ^~~~~~
crypto/modes/modes_lcl.h:90:27: warning: cast increases required alignment of target type [-Wcast-align]
 # define PUTU32(p,v)     *(u32 *)(p) = BSWAP4(v)
                           ^
crypto/evp/e_aes_cbc_hmac_sha256.c:382:9: note: in expansion of macro 'PUTU32'
         PUTU32(out + 20, ctx->F[i]);
         ^~~~~~
crypto/modes/modes_lcl.h:90:27: warning: cast increases required alignment of target type [-Wcast-align]
 # define PUTU32(p,v)     *(u32 *)(p) = BSWAP4(v)
                           ^
crypto/evp/e_aes_cbc_hmac_sha256.c:383:9: note: in expansion of macro 'PUTU32'
         PUTU32(out + 24, ctx->G[i]);
         ^~~~~~

crypto/modes/ctr128.c: In function 'ctr128_inc_aligned':
crypto/modes/ctr128.c:54:12: warning: cast increases required alignment of target type [-Wcast-align]
     data = (size_t *)counter;
            ^
In file included from crypto/modes/ctr128.c:11:
crypto/modes/ctr128.c: In function 'CRYPTO_ctr128_encrypt_ctr32':
crypto/modes/modes_lcl.h:89:34: warning: cast increases required alignment of target type [-Wcast-align]
 # define GETU32(p)       BSWAP4(*(const u32 *)(p))
                                  ^
crypto/modes/modes_lcl.h:46:35: note: in definition of macro 'BSWAP4'
 #   define BSWAP4(x) ({ u32 ret_=(x);                   \
                                   ^
crypto/modes/ctr128.c:170:13: note: in expansion of macro 'GETU32'
     ctr32 = GETU32(ivec + 12);
             ^~~~~~
crypto/modes/modes_lcl.h:90:27: warning: cast increases required alignment of target type [-Wcast-align]
 # define PUTU32(p,v)     *(u32 *)(p) = BSWAP4(v)
                           ^
crypto/modes/ctr128.c:193:9: note: in expansion of macro 'PUTU32'
         PUTU32(ivec + 12, ctr32);
         ^~~~~~
crypto/modes/modes_lcl.h:90:27: warning: cast increases required alignment of target type [-Wcast-align]
 # define PUTU32(p,v)     *(u32 *)(p) = BSWAP4(v)
                           ^
crypto/modes/ctr128.c:206:9: note: in expansion of macro 'PUTU32'
         PUTU32(ivec + 12, ctr32);
         ^~~~~~

crypto/modes/gcm128.c: In function 'CRYPTO_gcm128_encrypt':
crypto/modes/gcm128.c:1055:37: warning: cast increases required alignment of target type [-Wcast-align]
                     size_t *out_t = (size_t *)out;
                                     ^
crypto/modes/gcm128.c:1056:42: warning: cast increases required alignment of target type [-Wcast-align]
                     const size_t *in_t = (const size_t *)in;
                                          ^
crypto/modes/gcm128.c: In function 'CRYPTO_gcm128_decrypt':
crypto/modes/gcm128.c:1242:37: warning: cast increases required alignment of target type [-Wcast-align]
                     size_t *out_t = (size_t *)out;
                                     ^
crypto/modes/gcm128.c:1243:42: warning: cast increases required alignment of target type [-Wcast-align]
                     const size_t *in_t = (const size_t *)in;
                                          ^

crypto/modes/xts128.c: In function 'CRYPTO_xts128_encrypt':
crypto/modes/xts128.c:48:25: warning: cast increases required alignment of target type [-Wcast-align]
         scratch.u[0] = ((u64 *)inp)[0] ^ tweak.u[0];
                         ^
crypto/modes/xts128.c:49:25: warning: cast increases required alignment of target type [-Wcast-align]
         scratch.u[1] = ((u64 *)inp)[1] ^ tweak.u[1];
                         ^
crypto/modes/xts128.c:57:10: warning: cast increases required alignment of target type [-Wcast-align]
         ((u64 *)out)[0] = scratch.u[0] ^= tweak.u[0];
          ^
crypto/modes/xts128.c:58:10: warning: cast increases required alignment of target type [-Wcast-align]
         ((u64 *)out)[1] = scratch.u[1] ^= tweak.u[1];
          ^
crypto/modes/xts128.c:131:25: warning: cast increases required alignment of target type [-Wcast-align]
         scratch.u[0] = ((u64 *)inp)[0] ^ tweak1.u[0];
                         ^
crypto/modes/xts128.c:132:25: warning: cast increases required alignment of target type [-Wcast-align]
         scratch.u[1] = ((u64 *)inp)[1] ^ tweak1.u[1];
                         ^
crypto/modes/xts128.c:151:10: warning: cast increases required alignment of target type [-Wcast-align]
         ((u64 *)out)[0] = scratch.u[0] ^ tweak.u[0];
          ^
crypto/modes/xts128.c:152:10: warning: cast increases required alignment of target type [-Wcast-align]
         ((u64 *)out)[1] = scratch.u[1] ^ tweak.u[1];
          ^

ssl/s3_cbc.c: In function 'ssl3_cbc_digest_record':
ssl/s3_cbc.c:174:22: warning: cast increases required alignment of target type [-Wcast-align]
         if (MD5_Init((MD5_CTX *)md_state.c) <= 0)
                      ^
ssl/s3_cbc.c:184:23: warning: cast increases required alignment of target type [-Wcast-align]
         if (SHA1_Init((SHA_CTX *)md_state.c) <= 0)
                       ^
ssl/s3_cbc.c:192:25: warning: cast increases required alignment of target type [-Wcast-align]
         if (SHA224_Init((SHA256_CTX *)md_state.c) <= 0)
                         ^
ssl/s3_cbc.c:200:25: warning: cast increases required alignment of target type [-Wcast-align]
         if (SHA256_Init((SHA256_CTX *)md_state.c) <= 0)
                         ^
ssl/s3_cbc.c:208:25: warning: cast increases required alignment of target type [-Wcast-align]
         if (SHA384_Init((SHA512_CTX *)md_state.c) <= 0)
                         ^
ssl/s3_cbc.c:218:25: warning: cast increases required alignment of target type [-Wcast-align]
         if (SHA512_Init((SHA512_CTX *)md_state.c) <= 0)
                         ^

Fixed everything from the above list, except ssl/s3_cbc.c which is a false positive.

If this allows to run ubsan tests without additional flags (as can be seen in .travis.yml), then we should remove those flags. I've just attempted enable-ubsan without those additional flags, test_evp is the only one failing and it's failing in wp_block.c:599. So that one would need adjustment too. In other words suggestion also is to modify .travis.yml and add [extended tests] to last commit.

I've just attempted enable-ubsan without those additional flags. ... wp_block.c:599.

Just in case, enable-ubsan but with no-asm. Additional flags to be removed are -fno-sanitize=alignment and -DPEDANTIC.

I'm categorically against "unal", it just doesn't sound right.

No, it is no strict aliasing violation, because c points to char and that can alias anything.

Huh. I've always understood strict aliasing differently. Given you work on a compiler, I'm sure I'm just misunderstanding things, but maybe you could help find the fault in my reasoning:
The wording is:

The effective type of an object for an access to its stored value is the declared type of the
object, if any. [various rules for what to do if the object was allocated and has no declared type.]
An object shall have its stored value accessed only by an lvalue expression that has one of
the following types:

a type compatible with the effective type of the object,
[various other flavors of "basically the same type"]
a character type

Notably, whether an access is allowed is not symmetric between the effective type of the object and the type of the lvalue expression. So if the object itself was declared as an unaligned_size_t and I access it by dereferencing a char*, that is legal. However, if the object was declared as a bunch of chars and I access it by dereferencing an unaligned_size_t*, that is not legal.
So, in particular, this is not legal:
unsigned_size_t read_unaligned_size_t(const char *c) {
  return *((const unsigned_size_t *)c);
}

void foo(void) {
  char c[sizeof(unaligned_size_t)];
  read_unaligned_size_t(c);
}

Of course, if the compiler only sees read_unaligned_size_t, as in here, it cannot mangle anything because, for all it knows, read_unaligned_size_t is only ever called as:
unsigned_size_t s;
read_unaligned_size_t((const char *)&s);

Is the reasoning then that the global violation is not a concern, or have I misunderstood something about how this works?

@davidben
the not-legal example has turned out to be legal C++, gcc-7 had to be changed to make all
arrays of unsigned char (or std::byte) class-less storage because of that.
Yes the standard can evolve.
I must admit that my knowledge of the GCC implementation of the TBAA may be better
than my understanding of the exact wording of the standard.  GCC does not have
a concept of the declared type of an object, it just looks at the different read and
write accesses, and if one object may alias another object.   In this regard,
writing a value changes the effective type.
IMHO as long as the in and out data are declared as array of char and the only other
access is via ua_u32 that is fine.  As least for GCC.
What would violate the strict aliasing rules, would be declaring in / out as array of float,
casting that to char ,  or allocating with malloc and assigning values as float,
and when we access one of these float values as ua_u32 that would be a potential problem.

However, if the object was declared as a bunch of chars and I access it by dereferencing an unaligned_size_t*, that is not legal.

Does standard actually prohibits implementation to implement a type with sizeof(type) > sizeof(char) that would be compatible with char[]? I mean standard speaks of types defined in the standard as being incompatible with char[], but does it say that there may not be non-standard type compatible with char[]? Suggestion is obviously that additional attribute makes it non-standard and compatible. Related question is what constitutes "a type compatible with the effective type of the object". Thing is that cast are not actually prohibited, only outcome[!] of actual dereferencing by incompatible[!] type is claimed undefined. But what makes a type compatible? Intuitively you want to say "alignment", i.e. if pointer is correspondingly aligned, then you can count on dereference work as expected. Naturally it can also mean things like endianness and floating-point number representation, in sense that you need to know these things to handle them correctly. But you should be entitled for a chance to handle them correctly and in predictable manner. I mean if standard was as categorical as "original type or literally nothing is guaranteed [to degree that you can get unpredictable results, i.e. different at different times on same system]" it would probably mean that you'd have no chance of writing a program that let's say reads VAX-y floating-point data on any other system but VAX...
[On related note I sometimes wonder if "undefined" rather mens "undefined here" and consequently "look elsewhere for answers", and not "there is no answer anywhere else."]

The wording is:

An object shall have its stored value accessed only by an lvalue expression that has one of
the following types:

On additional note it should be noted that this sentence has a footnote that reads "The intent of this list is to specify those circumstances in which an object may or may not be aliased." Would it be appropriate to interpret this fact as it's not really about sheer illegality of derefencing a pointer casted to [allegedly] incompatible type? I mean it's not like "you shall not cast pointer to char to anything but char", but rather "compiler shall apply these rules when you do"?

"compiler shall apply these rules when you do"

Or rather "compiler shall make these assumptions about aliasing when you do".

To summarize. Strict aliasing rules are not about making char * to say int * cast illegal, but about what can happen when you do so. On practical level they mean that compiler is not obliged to preserve the program order of loads and stores with incompatible types. Which can have unexpected effect when they actually overlap, but not otherwise. And here actually lies answer to the question "Isn't this still a strict aliasing violation?" No, it isn't, as it wasn't. It would have been if there were actually overlapping mixed-cast references. But there are none here, as casted pointers are used consistently. [I get kind of a deja-vu feeling, as if this was discussed already, maybe not in this specific context, but very similar.] Once again, you're free to cast, it's just that you have to know what you are doing. So that correct question is "do we know what we're doing here?"

I think in the foreseeable future we will have no issue with strict aliasing here.
Things could of course change when for instance LTO starts to inline openssl code
side by side with user code.
If we need we could use the GCC may_alias type attribute to let the compiler know
what we are doing here.  I don't know if clang has a similar construct.

LTO starts to inline openssl code side by side with user code.

Customarily there will be at least one indirect call between application code and the subroutines in question, at EVP layer, so that it's bound to remain out of LTO's reach. I mean indirect call will effectively prevent that kind of optimization. Well, one can still wonder if compiler can manage to resolve indirect call at compile time... But it would have to be custom application [with custom non-shared OpenSSL build] likely featuring custom crypto....

Well, one can still wonder if compiler can manage to resolve indirect call at compile time...

Compiler can do that. GCC has so called de-virtualization that is replacing
C++ virtual function calls with inline code.
That is not much different than resolving indirect calls at compile time, if not exactly the same.

That said, I also believe it is not likely to happen in this context.

Well, one can still wonder if compiler can manage to resolve indirect call at compile time...

Compiler can do that.

Yes, of course they can do that. Question was rather what would it take to resolve it specifically at EVP level. I.e. when you pass pointer to [dynamically allocated] EVP_CIPHER_CTX that has pointer to EVP_CIPHER...

Rebased.

Ping @mattcaswell ?

Once again, "I suggest safe_as_<type>, because it describes the intention most adequately. "unaligned" is a bit misleading, because it suggests that pointer has to be treated as misaligned, which is not actual intent." I'd even make it stronger by removing "a bit" from "a bit misleading". I mean as in ""unaligned" [prefix], and of course any variant of it such as "ua", is [just] misleading." Indeed, consider what's going on platforms that don't tolerate unaligned load/store.
if (ptr % sizeof(type) == 0)
    (type *)ptr;

I.e. check if pointer is properly aligned[!], then cast. So what would be more readable as replacement for (type *)? (unaligned_type *) or (safe_as_type *)? Even on platforms that tolerate unaligned load/store safe_as_type would capture the situation better. I.e. it's not that we expect data to be unaligned, but that it's safe to cast even if it is [misaligned.]

safe_as_size_t does not say at all, what is safe and why.
I think when it becomes so difficult to find a proper name for something
simple like this it shows there is something wrong with the design.
Maybe the code should be re-factored to use unaligned types for unaligned
or potentially unaligned data and aligned types otherwise.
Except for the ua_size_t, which is unaligned on non-strict-aligned-platform,
there are many more places where I simply have:
#ifdef __GNUC__
typedef u64 ua_u64 __attribute((__aligned__(1)));
#else
typedef u64 ua_u64;
#endif

These are unaligned types in the true sense, which implies that
each access is a true bug.

What I mean with my previous comment is changing the program logic of
all places where ua_size_t is used like the following.
diff --git a/crypto/modes/cbc128.c b/crypto/modes/cbc128.c
index 4ce5eb2..90afb04 100644
--- a/crypto/modes/cbc128.c
+++ b/crypto/modes/cbc128.c
@@ -11,10 +11,16 @@
 #include "modes_lcl.h"
 #include <string.h>

-#if !defined(STRICT_ALIGNMENT) && !defined(PEDANTIC)
+#ifndef STRICT_ALIGNMENT
 # define STRICT_ALIGNMENT 0
 #endif

+#ifdef __GNUC__
+typedef size_t ua_size_t __attribute((__aligned__(1)));
+#else
+typedef size_t ua_size_t;
+#endif
+
 void CRYPTO_cbc128_encrypt(const unsigned char *in, unsigned char *out,
                            size_t len, const void *key,
                            unsigned char ivec[16], block128_f block)
@@ -26,11 +32,12 @@ void CRYPTO_cbc128_encrypt(const unsigned char *in, unsigned char *out,
         return;

 #if !defined(OPENSSL_SMALL_FOOTPRINT)
-    if (STRICT_ALIGNMENT &&
+    if (!STRICT_ALIGNMENT ||
         ((size_t)in | (size_t)out | (size_t)ivec) % sizeof(size_t) != 0) {
         while (len >= 16) {
-            for (n = 0; n < 16; ++n)
-                out[n] = in[n] ^ iv[n];
+            for (n = 0; n < 16; n += sizeof(size_t))
+                *(ua_size_t *)(out + n) =
+                    *(ua_size_t *)(in + n) ^ *(ua_size_t *)(iv + n);
             (*block) (out, out, key);
             iv = out;
             len -= 16;
@@ -83,18 +90,21 @@ void CRYPTO_cbc128_decrypt(const unsigned char *in, unsigned char *out,
     if (in != out) {
         const unsigned char *iv = ivec;

-        if (STRICT_ALIGNMENT &&
+        if (!STRICT_ALIGNMENT ||
             ((size_t)in | (size_t)out | (size_t)ivec) % sizeof(size_t) != 0) {
             while (len >= 16) {
+                ua_size_t *out_t = (ua_size_t *)out;
+                ua_size_t *iv_t = (ua_size_t *)iv;
+
                 (*block) (in, out, key);
-                for (n = 0; n < 16; ++n)
-                    out[n] ^= iv[n];
+                for (n = 0; n < 16 / sizeof(size_t); n++)
+                    out_t[n] ^= iv_t[n];
                 iv = in;
                 len -= 16;
                 in += 16;
                 out += 16;
             }
-        } else if (16 % sizeof(size_t) == 0) { /* always true */
+        } else {
             while (len >= 16) {
                 size_t *out_t = (size_t *)out, *iv_t = (size_t *)iv;

@@ -109,10 +119,14 @@ void CRYPTO_cbc128_decrypt(const unsigned char *in, unsigned char *out,
         }
         memcpy(ivec, iv, 16);
     } else {
-        if (STRICT_ALIGNMENT &&
+        if (!STRICT_ALIGNMENT ||
             ((size_t)in | (size_t)out | (size_t)ivec) % sizeof(size_t) != 0) {
-            unsigned char c;
             while (len >= 16) {
+                size_t c;
+                ua_size_t *out_t = (ua_size_t *)out;
+                ua_size_t *ivec_t = (ua_size_t *)ivec;
+                const ua_size_t *in_t = (const ua_size_t *)in;
+
                 (*block) (in, tmp.c, key);
                 for (n = 0; n < 16; ++n) {
                     c = in[n];
@@ -123,7 +137,7 @@ void CRYPTO_cbc128_decrypt(const unsigned char *in, unsigned char *out,
                 in += 16;
                 out += 16;
             }
-        } else if (16 % sizeof(size_t) == 0) { /* always true */
+        } else {
             while (len >= 16) {
                 size_t c, *out_t = (size_t *)out, *ivec_t = (size_t *)ivec;
                 const size_t *in_t = (const size_t *)in;

Defining a type with a name that ends in _t is also undefined behavior, FWIW.

What I mean with my previous comment is changing the program logic of all places where ua_size_t is used like the following.

Why would you want to do that? The question is legitimate because by doing so you effectively make code gcc-dependent, in sense that it's will be guaranteed to crash on system that can't tolerate misaligned load/store, which is not equipped with gcc or gcc-alike. This would be unacceptable. I would even be inclined to believe that gcc would have to be recent enough in some cases. At least I can't confirm that attribute(align(1)) makes older gcc for ARM generate by-byte load.

If the align attribute does not work reliably with all gcc versions at least then this will not work.
Which gcc version did you check?

Which gcc version did you check?

4.8. But this is hardly relevant, because it won't work with any non-gcc compiler. This is sufficient.

To summarize. Base pattern is
if (ptr % sizeof(type) != 0)
    byte-oriented processing
else
    *(type *)-based processing

This is perfectly legitimate code(*). For which safe_as_type tells right story. The pattern is then modified for platforms that can tolerate misaligned load/store as following
if (STRICT_ALIGNMENT && ptr % sizeof(type) != 0)
    byte-oriented processing
else
    *(type *)-based processing

So that on platforms where STRICT_ALIGNMENT is 0, e.g. on x86[_64], byte-oriented code is rendered dead, and *(type *)-based processing is used in all cases, more specifically both aligned and misaligned. And this last part, that it's used in both cases, makes safe_as_type [more] suitable. It reads "we tell compiler that if's just safe to cast."
(*) Except for highly unlike possibility that LTO pass (if enabled) will manage to "transcend" all indirect calls, inline this code so that it can consider interleaving instructions with caller's, and actually reorder some loads and stores.

(*) Except for highly unlike possibility that LTO pass (if enabled) will manage to "transcend" all indirect calls, inline this code so that it can consider interleaving instructions with caller's, and actually reorder some loads and stores.

Actually caller would have to write data with non-character stores. I mean if caller does things like char_buffer[i] = value & 0xff, char_buffer[i] = (value >> 8) & 0xff, then no reorder should take place. It's only problem if caller does short_buffer[i] = value & 0xffff. Once again, compiler would also have to recognize that indirect calls at EVP are actually constant.

Regarding the gcc code generation, I tried 4.8.3, and the code generation seems to work fine.
In fact gcc is much smarter than the STRICT_ALIGNMENT macro.
It generates unaligned word loads dependent on the CPU arch.
For armv7 load handle unaligned addresses, and for armv6 it does not,
so byte loads are generated:
$ cat align.c
typedef int ua_int_t __attribute((__aligned__(1)));
int test(char *x)
{
  return * (ua_int_t *) x;
}
$ arm-linux-gnueabihf-gcc -march=armv7 -msoft-float -S -O3 align.c
$ cat align.s
[...]
        .thumb
        .thumb_func
        .type   test, %function
test:
        @ args = 0, pretend = 0, frame = 0
        @ frame_needed = 0, uses_anonymous_args = 0
        @ link register save eliminated.
        ldr     r0, [r0]        @ unaligned
        bx      lr
        .size   test, .-test
        .ident  "GCC: (crosstool-NG linaro-1.13.1-4.8-2014.04 - Linaro GCC 4.8-2014.04) 4.8.3 20140401 (prerelease)"
        .section        .note.GNU-stack,"",%progbits
$ arm-linux-gnueabihf-gcc -march=armv6 -msoft-float -S -O3 align.c
$ cat align.s
[...]
        .thumb_func
        .type   test, %function
test:
        ldrb    r3, [r0, #1]
        @ sp needed
        ldrb    r2, [r0]
        lsl     r3, r3, #8
        orr     r3, r2
        ldrb    r2, [r0, #2]
        ldrb    r0, [r0, #3]
        lsl     r2, r2, #16
        orr     r3, r2
        lsl     r0, r0, #24
        orr     r0, r3
        bx      lr
        .size   test, .-test
        .ident  "GCC: (crosstool-NG linaro-1.13.1-4.8-2014.04 - Linaro GCC 4.8-2014.04) 4.8.3 20140401 (prerelease)"
        .section        .note.GNU-stack,"",%progbits

Regarding the gcc code generation, I tried 4.8.3, and the code generation seems to work fine.

It seems that I've misinformed you. I have a whole bunch of different compilers and cross-compilers on my computer, I must have lost myself. The version I can't confirm this for is gcc 4.6[.3]. I apologize for confusion. But once again, it's essentially irrelevant, because our worlds does not consist of gcc compiler. I mean we can't have code that works only with [recent enough] gcc and is effectively guaranteed to crash with another compiler.

This is perfectly legitimate code. For which safe_as_type tells right story.

Naturally provided that it's just typedef type safe_as_type on platform that don't tolerate misaligned load/store. In the essence it's rather a message to human reader of the code that says "we know what we are doing".

Well, you are right, 4.6 does not seem to honour these type attributes in the current
form.
I think it will need another attempt to find a syntax that works on all relevant platforms.

So, the syntax with the typedef ... aligned(1) starts to work with gcc-4.8.0 and newer
before that version it is ignored.
An alternative would be using a packed structure with a single data member, this seems
work with all gcc versions at least, but it is a too ugly thing to do in my opinion.
Regarding the name, I would say fast_u32 would be my favourite in the moment.

How about "spry" as a synonym for "fast" ?

All examples of "spry" in sentence are about old people being more agile than expected for their age. No, I for one don't feel for it... Can't speak for others. But just in case, "not bad" doesn't makes "fast" my preference, I'm is still for safe_as_type.

I don't like safe_as because safe would imply using something that is
always safe because it does not make assumptions about the alignment.
That would be possible for instance with
typedef struct { u32 safe_value; } __attribute((__packed__)) safe_u32;
#defined GETU32(x) (((safe_u32*)(x))->safe_value)

With gcc this is clumsy but works with all versions I have checked.
However it does not take advantage of the possible alignment
of the value, and on strict alignment platforms this generates code
that is equivalent to single byte accesses.  So there would be
needed for an aligned_u32 type that creates better code
on strict alignment platforms, when the pointer happens to be
sufficiently aligned.
How about fa_ prefix, as acronym for fast access?

With gcc this ... works with all versions I have checked.

Once again, our code has to work with non-gcc compilers, so that assertion "works with all gcc versions" has limited value. And once again, you are looking at it from wrong angle. You aim to replace byte-oriented processing(*), while the goal is to make *(type *)-oriented processing(*) alignment-agnostic specifically on platforms that can tolerate misaligned load/store. There is no point to touch byte-oriented processing for following reason. Let's imagine for a second that we do so. Since our code has to work with non-gcc compilers, tailored code would have to go to #ifdef __GNUC__, and original code to corresponding #else. This alone is a turn-down, because why have two segments to maintain when one is sufficient? Why is one sufficient? Would new segment provide significant performance improvement on platforms that can't tolerate misaligned load/store? No, as it would still be same amount of load/store instructions (actually larger amount of other instructions, shifts to be more specific). But would it provide significant improvement on platform that tolerate misaligned load/store? Also no, because resulting binary code would be if(condition) {code} else {**duplicate of previous code**}. I.e. double as large with potentially mispredicted branch. So that performance would be same in benchmark, but it would have overall negative impact on real-life applications, because branch prediction and code cache are shared resources.
(*)
if (ptr % sizeof(type) != 0)
    byte-oriented processing
else
    *(type *)-based processing

How about fa_ prefix, as acronym for fast access?

This is question to others. I already verbalized my preferred choice.

would it provide significant improvement on platform that tolerate misaligned load/store?

It is just a vague idea, but one thing is sure, the current code for armv7 is not optimal,
since this platform would allow misaligned load/store, dependent on the presence of
the -munaligned-access, and yes, the code would branch between one section that uses
unaligned type but the compiler knows that and generates byte or misaligned load/store
and a possibly identical section that uses aligned load/store.
While now one section uses byte load/store whatsoever, and another section that uses
aligned load/store.  So this would give an improvement for armv7 and would not change
anything for armv5.

current code for armv7 is not optimal,

This is actually a "trust" issue. At some earlier point STRICT_ALIGNMENT was set to 0 on armv7, but then it turned out that compiler used ldrd, instruction that loads pair of values. And trouble is that unlike ldr, ldrd does not tolerate misaligned pointer. So what do you do? Engage in elaborated #if that-version-but-not-this-one that would haunt you forever, or just do something that guarantees fault-free operation? Besides, it's only suboptimal if data is misaligned. So all application has to do is keep data aligned. TLS layer does...

And trouble is that unlike ldr, ldrd does not tolerate misaligned pointer.

Yes, and this is the reason why compiler needs to know that the type is aligned(1).
In this case we would have one section that uses ldr but not ldrd because the
compiler knows the value is unaligned, and an almost identical section that uses
ldr and ldrd because the compiler knows the type is aligned(4).

In this case we would have one section that uses ldr but not ldrd because the compiler knows the value is unaligned,  and an almost identical section that uses ldrd

While optimal is one section that uses ldr. Because difference between using ldr and ldrd would be comparable to mispredicted branch and cache miss penalties. In other words it's hardly a point to have two sections, one is more than sufficient. BTW, what would it take to make compiler generate one section in this request as it stands now? It would take #if that would cover minimum gcc compiler version and apparently __ARM_FEATURE_UNALIGNED that drop STRICT_ALIGNMENT to 0. This is provided that it's established that compiler won't use ldrd.
On more general note. There always will be corner cases and possibilities to make improvement in specific contexts. But if context is overly specific, such possibility by itself doesn't automatically qualify for implementation. Code tailored for overly specific context has to provide significant or notable improvement.

Yes. __ARM_FEATURE_UNALIGNED sounds like a good idea.

I think I have a better solution for the type name problem now.
Most types are always aligned(1) and these will be named _a1.
But other types are sometimes aligned(1) or aligned(4) or aligned(8)
these will be named _aX, X = 1 on non-strict-alignment platforms,
X=sizeof(type) on strict-alignment platforms.
Using a postfix keeps this away from the _t name scope.

Most types are always aligned(1) and these will be named _a1.

Recall that they are aligned(1) only in gcc's "mind". In other words suggestion is effectively gcc-specific. This makes it unfit to me, sorry...

No.  It creates a type that is intended to be used aligned(1) and even if some compilers
will not understand it, the use is limited to non-strict alignment platforms.

No. It creates a type that is intended to be used aligned(1)

You are still looking at it from wrong angle, from byte-oriented case perspective. Once again, reference pattern to consider is
if (ptr % sizeof(type) != 0)
    byte-oriented processing
else
    *(type *)-based processing

"Reference pattern" means that it's the one that has to make perfect sense [to unprepared reader if you wish].

Allow @sanijajoy too "reques. changes" too "openssl/Configure"-"Fixes #2462 openssl/.gitignore" ). Via "comment"

This PR is primarily aiming at formal correctness:
Using an a1 type is always formally correct whether or not the pointer is aligned.
Using an a1 type on a non-strict-alignment platform should be avoided, because not
all compilers will understand it correctly, that can be ensured by only defining the type
on non-strict-alignment platforms, nevertheless the type is conceptually always align(1).
Using an aX type is always formally correct when the pointer is aligned.
When it is not aligned it is formally correct to do so on a non-strict-alignment platform.
Conceptually this is something completely different than a1, because the type is not always align(1).
Neither a1 nor aX types are safe or efficient by itself.

 Why? Because formal correctness is not something that can be specific to one compiler (of subset of them). In other words in order to call a piece of code "formally correct" it may not rely on any compiler-specific feature/extension. Formal correctness is also about making least assumptions about underlying hardware. In other words formally correct code is expected to produce exactly same effect with literally any compliant compiler on literally any hardware. The goal here is rather to make sanitizer consider formally incorrect[!] code suitable for the platform sanitizer is executing on. In other words goal is actually to keep code formally incorrect[!], by still permitting unaligned access with single load/store instruction, but without sanitizer complaining about it.

In other words goal is actually to keep code formally incorrect[!], by still permitting unaligned access with single load/store instruction, but without sanitizer complaining about it.

To put it slightly differently. Formally correct code shall never perform unaligned access with single load/store instruction. But our goal here is still to keep unaligned access with single load/store instruction on platforms that can tolerate it. Which is effectively opposite to formal correctness.

This PR is about formal correctness.

Hm.  Have we had any bug reports that the current code fails?  Maybe this is too much effort for no practical gain?

This PR isn't strictly for 1.1.1, since it covers code that wasn't touch in 1.1.1  So I am removing the 1.1.1 milestone.

I closed this PR because it had no progress almost 2 years.
AND it has of course merge conflicts all the time, so this soaks up my energy.
Nevertheless I will update this PR once more to reflect the latest update of #11781
the patch against 1.1.1 LTS branch.

I closed this PR because it had no progress almost 2 years.

I hope the project will try to do something about this kind of thing.

@bernd-edlinger do you think it makes sense to change also
https://github.com/openssl/openssl/blob/9c47a3386d6733512b72f5fab43bffba6a1fe72b/include/crypto/modes.h#L30-L39
removing line 31 and 39, and de-indenting by one space the block at lines L32-L38?
Seems like L31 was added explicitly to appease clang ubsan, so it might be masking some other items that need to be addressed in this PR? (I did not try this, I am just wondering).
(for the backport this would be crypto/modes/modes_local.h)

Travis is now failing with
test/ossl_shim/ossl_shim.cc: In function 'int bssl::TicketKeyCallback(SSL*, uint8_t*, uint8_t*, EVP_CIPHER_CTX*, HMAC_CTX*, int)':
test/ossl_shim/ossl_shim.cc:394:73: warning: 'int HMAC_Init_ex(HMAC_CTX*, const void*, int, const EVP_MD*, ENGINE*)' is deprecated [-Wdeprecated-declarations]
   if (!HMAC_Init_ex(hmac_ctx, kZeros, sizeof(kZeros), EVP_sha256(), NULL) ||
                                                                         ^
In file included from include/openssl/opensslv.h:109:0,
                 from include/openssl/macros.h:11,
                 from include/openssl/opensslconf.h:14,
                 from include/openssl/macros.h:10,
                 from include/openssl/bio.h:14,
                 from test/ossl_shim/include/openssl/base.h:14,
                 from test/ossl_shim/packeted_bio.h:13,
                 from test/ossl_shim/ossl_shim.cc:14:
include/openssl/hmac.h:39:22: note: declared here
 DEPRECATEDIN_3_0(int HMAC_Init_ex(HMAC_CTX *ctx, const void *key, int len,
                      ^
include/openssl/macros.h:38:39: note: in definition of macro 'DECLARE_DEPRECATED'
 #     define DECLARE_DEPRECATED(f)    f __attribute__ ((deprecated));
                                       ^
include/openssl/hmac.h:39:1: note: in expansion of macro 'DEPRECATEDIN_3_0'
 DEPRECATEDIN_3_0(int HMAC_Init_ex(HMAC_CTX *ctx, const void *key, int len,
 ^~~~~~~~~~~~~~~~

I'm still checking if this is unrelated to this PR

I'm still checking if this is unrelated to this PR

The failure seems to be in the test/ossl_shim/ossl_shim.cc file, which hasn't been updated since the batch of deprecations, and calls
https://github.com/openssl/openssl/blob/057452a06f02eb24d8008b975cde32d51361a2cf/test/ossl_shim/ossl_shim.cc#L394
It is not related to this PR:  I opened #11942 to address it.

removing line 31 and 39, and de-indenting by one space the block at lines L32-L38?

Well I would not remove PEDANTIC here, as it may be used to force strict C code.
When PEDANTIC is defined all code (except ec_nistp which does only compile
when the compiler has GNU C-extensions).
So despite the original intention to only help with ubsan I see still some value here.

removing line 31 and 39, and de-indenting by one space the block at lines L32-L38?

Well I would not remove PEDANTIC here, as it may be used to force strict C code.
When PEDANTIC is defined all code (except ec_nistp which does only compile
when the compiler has GNU C-extensions).
So despite the original intention to only help with ubsan I see still some value here.

Makes sense, thanks!

As soon as #11944 is merged we could rebase this to trigger the CI past and verify everything passes, other than that and assuming everything will pass, I would approve this PR.
Given @mattcaswell is flagged as having requested changes we would need his approval, or at least he would have to consent to dismiss his previous review.

FYI: There is still a failure with the external gost_tests after rebasing: https://travis-ci.org/github/romen/openssl/jobs/691216006
Looking into that now

Hmmm. This failure is new for me

LGTM. (caveat: I still think this requires an acknowledgment by @mattcaswell before merging)

no, not at all.
initially the code looked like that
typedef unsigned int ossl_uint
#ifdef __GNUC__
  __attribute((__aligned__(1)))
#endif
;

Matt asked me to change it like this
#ifdef __GNUC__
typedef unsigned int ossl_uint __attribute((__aligned__(1)));
#else
typedef unsigned int ossl_uint;
#endif

I agreed that it looks better this way, and took his advice.
Why bother him to review the whole PR again?
Anyway, it would be way better when Matt would just
have added a comment, but that may as well be
just a wrong mouse click, maybe that is not so
significant any more, after 2 years, 1 month and 1 days ?

LGTM. (caveat: I still think this requires an acknowledgment by @mattcaswell before merging)

I have not reviewed and I don't think I need to. But just to be clear I have no objections to this being merged.

Then, after all this time, this PR is finally approved!
Thanks @bernd-edlinger for working on this!

24 hours has passed since 'approval: done' was set, but as this PR has been updated in that time the label 'approval: ready to merge' is not being automatically set.  Please review the updates and set the label manually.

Finally merged as 77286fe3ec6b9777934e67e35f3b7007143b0734.
Thanks!

Just for the record for future reference #11781 is the (merged) PR backporting these changes to 1.1.1.


autoUpdater does not work properly.

Update error: Code signature at URL file:///Users/lamqu01/Library/Caches/com.webcatalog.app.ShipIt/update.zxHBM1P/WebCatalog.app/ did not pass validation: code failed to satisfy specified code requirement(s)

electron-userland/electron-builder#1098

Related to afterPack script.
https://developer.apple.com/library/content/technotes/tn2206/_index.html#//apple_ref/doc/uid/DTS40007919-CH1-TNTAG311
Store Python, Perl, shell, and other script files and other non-Mach-O executables in your app's Contents/Resources directory. While it's possible to sign such executables and store them in Contents/MacOS, this is not recommended.

thats fixable by using binary instead of bash script
ie. go binary: https://gist.github.com/natrim/04a0324ca6898619fad5579ef1facb68
https://www.dropbox.com/s/yogkelfvkvrea65/Webcatalog?dl=0

@natrim I tried a similar method. It fixed the signing issue but it still doesn't work somehow. I'll test again with your code soon. It'd be better if there's an API or something to modify the behavior with code instead of dealing with a file hack like this.

@natrim Better solution: Simply duplicate the executable file, but all the apps have to update to link to the duplicated executable. I don't remember why I did not do this (maybe to keep the apps' compatibility)

hmm, better idea - just have duplicate Webcatalog binary
seems to work for me,
i just duplicated the WebCatalog_Real as Webcatalog
rename the old WebCatalog_Real as WebCatalogLauncher
and resigne the app
and it works

@natrim
@natrim The reason why I do it is because when you launch an app from Launchpad, if macOS detects that the executable is running, it won't start a new instance. In the case of WebCatalog, all the apps installed from it is actually just a script to open a new instance of WebCatalog executable file with some parameters. Because of this design, for example, if you are opening Messenger and try to open WebCatalog from Launchpad, it won't start because Apple thinks the executable is already running.
With the current solution, using a script file as main executable, WebCatalog is able to trick macOS. macOS will think the executable file of the app is "WebCatalog" while the real one is "WebCatalog_Real". And because the script file only runs to call the real one and then exit, macOS will think WebCatalog is not running and start the app.
This is what we need to resolve.

@natrim Haha, whatever. Duplicating the executable seems working for me.

#43 3.1.1 is up with auto updater :)


Feature request: 'age' output visually

Similar to #3519, when watching logs, or build processes, similar lines being shown repeatedly can often make it hard to determine if anythings happening (as each new page of data looks the same as the previous one)
#3519 mentions smooth scrolling, but it would also be useful to apply some visual effect to old output. This is a bit unusual, I don't know of any other terminal that does this, but it would also be very useful
Eg, remove brightness by X percent for output older than 5 minutes, y% for output older than 15 minutes.

I doubt that this would be something that we'd ship as a part of the Terminal, though this is something we'll consider while designing the extensibility model. If the terminal were to support this, it would need to track when each line was put into the scrollback, and then be able to render each of those lines differently based on that amount of time. I don't necessarily want the Terminal to be directly in the business of tracking all that info and configuring that itself, but maybe an extension could.


exe does not exist

hi
d2dsy_gui.exe is nowhere to be found
the link in the zip file will certainly not work since I have no corvus user
C:\Users\corvus\Documents\GitHub\pd2dsy\dist\pd2dsyGUI\data\pd2dsy_gui.exe

do you even check what you upload ?

Unfortunately, Windows shortcuts cannot be relative, so there isn't simple, portable way to make the exe easy to find. The obvious solution to this problem is to simply remove the shortcut, but the inelegance of that solution has us hesitating a bit. In the meantime, you can find the exe in the data folder. Of course, you can create your own shortcut and place it wherever you like.

For clarity, the shortcut has been removed. If you're unsure of how to run the program, the README in its place should help out!


[#4360] Add akka-http plugin

Change how acquire some variables.(rpcName, endPoint)

Codecov Report

Merging #4380 into master will increase coverage by <.01%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master    #4380      +/-   ##
==========================================
+ Coverage   40.89%   40.89%   +<.01%
==========================================
  Files        2447     2447
  Lines       75328    75347      +19
  Branches    10212    10217       +5
==========================================
+ Hits        30803    30812       +9
- Misses      41699    41711      +12
+ Partials     2826     2824       -2

Impacted Files
Coverage Δ

.../pinpoint/plugin/akka/http/HttpRequestAdaptor.java
0% <0%> (ø)
:arrow_up:

...pinpoint/profiler/instrument/AccessorAnalyzer.java
57.14% <0%> (-8.58%)
:arrow_down:

...m/navercorp/pinpoint/web/util/ThreadDumpUtils.java
67.3% <0%> (-3.85%)
:arrow_down:

...navercorp/pinpoint/rpc/common/SocketStateCode.java
80.48% <0%> (-2.44%)
:arrow_down:

...corp/pinpoint/rpc/stream/StreamChannelManager.java
70.68% <0%> (-1.73%)
:arrow_down:

...ercorp/pinpoint/profiler/util/ThreadDumpUtils.java
62.22% <0%> (-1.12%)
:arrow_down:

...point/rpc/client/DefaultPinpointClientHandler.java
71.65% <0%> (+0.31%)
:arrow_up:

...orp/pinpoint/rpc/server/DefaultPinpointServer.java
80.55% <0%> (+1.04%)
:arrow_up:

...om/navercorp/pinpoint/rpc/codec/PacketDecoder.java
63.82% <0%> (+2.12%)
:arrow_up:

...ofiler/sender/StandbySpanStreamDataSendWorker.java
83.58% <0%> (+2.98%)
:arrow_up:

... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d703e76...cfa82f9. Read the comment docs.


[bug]: postgres error: could not serialize access due to read/write dependencies among transactions (SQLSTATE 40001)

Background
Been observing an issue with PostgreSQL DB since version 0.17 regarding serialization of reads/writes (probably because LND now exposes SQL errors to logs?).
Before 0.17 I used to compact on-the-fly while running lnd in background. With 0.17 I ran across this serialization error when compacting (vaccuum full). Today this error also led to a force close of a channel (no compaction happened at this time):
2023-10-01 13:21:03.430 [ERR] HSWC: ChannelLink(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): failing link: unable to accept revocation: ERROR: could not serialize access due to read/write dependencies among transactions (SQLSTATE 40001) with error: invalid revocation

followed by multiple
2023-10-01 13:21:38.721 [INF] CNCT: Attempting to update ContractSignals for ChannelPoint(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3)
2023-10-01 13:21:39.055 [INF] PEER: Peer(020d1617e27ac022395352f2b3774969593d3d6ddff6fb117d820a9dda8da45217): unable to read message from peer: read next header: EOF
2023-10-01 13:21:39.055 [WRN] HSWC: ChannelLink(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): error when syncing channel states: first message sent to sync should be ChannelReestablish, instead received: *lnwire.Error
2023-10-01 13:21:39.055 [INF] PEER: Peer(020d1617e27ac022395352f2b3774969593d3d6ddff6fb117d820a9dda8da45217): disconnecting 020d1617e27ac022395352f2b3774969593d3d6ddff6fb117d820a9dda8da45217@<IP_ADDRESS>:40009, reason: read handler closed
2023-10-01 13:21:39.055 [ERR] HSWC: ChannelLink(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): failing link: unable to synchronize channel states: first message sent to sync should be ChannelReestablish, instead received: *lnwire.Error with error: unable to resume channel, recovery required
2023-10-01 13:21:39.055 [INF] HSWC: ChannelLink(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): exited
2023-10-01 13:21:39.155 [INF] HSWC: ChannelLink(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): stopping

resulting in
2023-10-01 13:44:00.128 [DBG] NTFN: Found spend of outpoint=61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3, script=0 f42feb87b57ea3d00cf87591efeb58fd9dbc8ec913f35ed3f003780bf93d20b3: spend_tx=ef66de761c1253ae85d7d3deb4f3791c52d44484b9a51084c5da3d7262574891, block_height=810154
2023-10-01 13:44:00.385 [DBG] NTFN: Found spend of outpoint=61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3, script=0 f42feb87b57ea3d00cf87591efeb58fd9dbc8ec913f35ed3f003780bf93d20b3: spend_tx=ef66de761c1253ae85d7d3deb4f3791c52d44484b9a51084c5da3d7262574891, block_height=810154
2023-10-01 13:44:00.385 [DBG] NTFN: Spend request outpoint=61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3, script=0 f42feb87b57ea3d00cf87591efeb58fd9dbc8ec913f35ed3f003780bf93d20b3 spent at tip=810154
2023-10-01 13:44:10.989 [INF] NTFN: Dispatching confirmed spend notification for outpoint=61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3, script=0 f42feb87b57ea3d00cf87591efeb58fd9dbc8ec913f35ed3f003780bf93d20b3 at current height=810154: ef66de761c1253ae85d7d3deb4f3791c52d44484b9a51084c5da3d7262574891[0] spending 61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3 at height=810154
2023-10-01 13:44:10.989 [DBG] CNCT: ChannelArbitrator(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): attempting state step with trigger=chainTrigger from state=StateDefault
2023-10-01 13:44:10.990 [DBG] CNCT: ChannelArbitrator(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): new block (height=810154) examining active HTLC's
2023-10-01 13:44:10.990 [DBG] CNCT: ChannelArbitrator(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): checking commit chain actions at height=810154, in_htlc_count=0, out_htlc_count=0
2023-10-01 13:44:10.991 [DBG] CNCT: ChannelArbitrator(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): no actions for chain trigger, terminating
2023-10-01 13:44:10.993 [DBG] CNCT: ChannelArbitrator(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): terminating at state=StateDefault
2023-10-01 13:44:11.257 [INF] CNCT: Unilateral close of ChannelPoint(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3) detected
2023-10-01 13:44:11.258 [INF] CNCT: ChannelArbitrator(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): remote party has closed channel out on-chain
2023-10-01 13:44:22.062 [DBG] CNCT: ChannelArbitrator(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): attempting state step with trigger=remoteCloseTrigger from state=StateDefault
2023-10-01 13:44:22.062 [DBG] CNCT: ChannelArbitrator(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): new block (height=810154) examining active HTLC's
2023-10-01 13:44:22.062 [DBG] CNCT: ChannelArbitrator(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): checking commit chain actions at height=810154, in_htlc_count=0, out_htlc_count=0
2023-10-01 13:44:22.428 [DBG] CNCT: ChannelArbitrator(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): attempting state step with trigger=remoteCloseTrigger from state=StateContractClosed
2023-10-01 13:44:22.463 [DBG] CNCT: ChannelArbitrator(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): checking commit chain actions at height=810154, in_htlc_count=0, out_htlc_count=0
2023-10-01 13:44:22.621 [DBG] CNCT: ChannelArbitrator(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): inserting 1 contract resolvers
2023-10-01 13:44:22.847 [DBG] CNCT: ChannelArbitrator(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): attempting to resolve *contractcourt.commitSweepResolver
2023-10-01 13:44:22.847 [DBG] CNCT: ChannelArbitrator(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): contract *contractcourt.commitSweepResolver not yet resolved
2023-10-01 13:44:22.978 [DBG] CNCT: ChannelArbitrator(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): attempting state step with trigger=remoteCloseTrigger from state=StateWaitingFullResolution
2023-10-01 13:44:22.978 [INF] CNCT: ChannelArbitrator(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): still awaiting contract resolution
2023-10-01 13:44:22.997 [DBG] CNCT: ChannelArbitrator(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): still have unresolved contract: *contractcourt.commitSweepResolver
2023-10-01 13:44:22.997 [DBG] CNCT: ChannelArbitrator(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): terminating at state=StateWaitingFullResolution
2023-10-01 13:44:23.136 [DBG] CNCT: *contractcourt.commitSweepResolver(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): commit conf_height=810154, unlock_height=810154
2023-10-01 13:44:23.136 [DBG] CNCT: *contractcourt.commitSweepResolver(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): isDelayedOutput=false, isLocalCommitTx=false
2023-10-01 13:44:23.136 [INF] CNCT: *contractcourt.commitSweepResolver(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): Sweeping with witness type: CommitmentNoDelayTweakless
2023-10-01 13:44:23.136 [INF] CNCT: *contractcourt.commitSweepResolver(61430ab592efc91b2bc0241867c2d204d001f2f733b9f237ebf5f56d1996cbf6:3): sweeping commit output

Your environment

version of lnd: 0.17-rc6
which operating system (uname -a on *Nix): Linux 5.15.0-84-generic 93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023 x86_64
version of btcd, bitcoind, or other backend: bitcoind v25
any other relevant environment details:

Postgresql 14
peer in question is running CLN v23.08.1

lnd.conf:
db.postgres.timeout=0
db.postgres.maxconnections=50

Steps to reproduce
Reproduction method unknown.
Expected behaviour
No SQL error.
Actual behaviour
Serialization error in database.
Logs:
FC Log: fc.log
SQLSTATE 40001 while compacting: compacting.log
tl;dr:
I think this is the equivalent of #7869 for postgres. I'll try to set db.postgres.timeout to 10m. Although not sure why lnd is not retrying transactions as specified in #7960 (running latest 0.17-rc6).

Since https://github.com/lightningnetwork/lnd/pull/7927 we do re-try on certain SQL failures. Perhaps we just need to catch SQLSTATE 40001 as a re-tryable error.
Setting the DB timeout probably isn't a bad idea, though I wouldn't set the value that high. Maybe try with 1 or 5 minutes first?
Things should also get better with https://github.com/lightningnetwork/lnd/pull/7992, which hopefully will make it in for v0.17.1-beta.

Since #7927 we do re-try on certain SQL failures. Perhaps we just need to catch SQLSTATE 40001 as a re-tryable error.

Reading https://github.com/lightningnetwork/lnd/blob/master/sqldb/sqlerrors.go#L70-L74 with pgerr.SerializationFailure referencing https://github.com/jackc/pgerrcode/blob/master/errcode.go#L218 - which actually is SQLSTATE 40001 - I assumed this already is the case.

Setting the DB timeout probably isn't a bad idea, though I wouldn't set the value that high. Maybe try with 1 or 5 minutes first?

Okay, I'll try 5 minutes and see how it goes. Usually compacting takes less than that.

Things should also get better with #7992, which hopefully will make it in for v0.17.1-beta.

Yeah, can't wait to try out #7992! Although a huge key-value table still is a restriction regarding blocking techniques.

So https://github.com/lightningnetwork/lnd/pull/7992 also fixes issues with the retry logic: before certain errors weren't properly wrapped, so they weren't detected as serialization errors.
Interesting that you're running into it as is though, since we have an in process mutex that should limit to just a single writer.

Okay, this explains why vacuuming the database with the same database user as lnd is set up with throws the error. I can reproduce the error by background vaccuuming regardless of timeout setting.
Then I'm not sure what caused the serialization error that led to the force close. Could it be heavy usage due to parallelization of rebalancing and forwarding?
I'll keep an eye on it.

Today I compacted the database manually with user "postgres" instead. This correctly pauses all db actions for lnd and resumes them after compaction has finished. No errors in lnd's log. Still not sure what caused the force close now but I guess a concurrent database access with the same lnd user is the closest I could think of.

Tried another compaction run which caused LND to shutdown again.
With config:
db.postgres.timeout=2m

Compacting duration:
200.74 sec

Compaction Starting Time: 2023-10-08 16:41:50
First Error Sighting: 2023-10-08 16:42:10
2023-10-08 16:42:10.893 [CRT] CHDB: Caught unhandled error: ERROR: could not serialize access due to read/write dependencies among transactions (SQLSTATE 40001)
2023-10-08 16:42:10.894 [INF] CHDB: Sending request for shutdown
2023-10-08 16:42:10.894 [INF] LTND: Received shutdown request.
2023-10-08 16:42:10.894 [INF] LTND: Shutting down...

This issue will be addressed with https://github.com/lightningnetwork/lnd/pull/7992


Updating install file for dosomething_northstar module.

Refs #5778
What's this PR do?
This PR updates the dosomething_northstar.install file to modify how the function to create the caching table is created. The drupal_get_schema_unprocessed() function relies on being passed the name of an existing table to grab the schema from it and copy to create a new table with the same schema. The previous code referred to a table that does not exist so the function was likely silently failing.
Where should the reviewer start?
Just check the codez changes and make sure it looks correct :)
Any background context you want to provide?
While this updates and successfully creates a new cache_dosomething_northstar table in the database, it might not really matter since we're using Redis for our caching. However, the redis module overriding caching to put all caching in Redis seems to be broken on local development so it is difficult to confirm. @sena and I have chatted a bit about this and need to confirm it is still properly working on our test and prod versions and need to troubleshoot why local is broken.
What are the relevant tickets?
#5778 reference to caching related things

@angaither
cc: @chloealee @sheyd

:+1:


link for MmGetSystemAddressForMdlSafe  is broken

All the links for MmGetSystemAddressForMdlSafe point to drivers/kernel/mm-bad-pointer - and indeed the documentation for this function appears to have entirely disappeared,

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 29cf2a30-3b4d-4cec-33ff-1ed18e05c1e5
Version Independent ID: 4c5683f8-157c-0a26-19d4-c65556e0e26b
Content: Using MDLs - Windows drivers
Content Source: windows-driver-docs-pr/kernel/using-mdls.md
Product: windows-hardware
Technology: kernel
GitHub Login: @tedhudek
Microsoft Alias: tedhudek

OIC that page is for all the kernel macros-  you need to thumb down it to find the one you care about.
So the info is there but buried.
I leave it to you to decide whether this matters but it confused me.  And, BTW, it stops  and most search engines from finding it.

It's a great point Rod, thanks for reminding me.  I think we can make some improvement here.  I'll keep this issue open for a bit to keep you apprised of progress.

@tedhudek , Sir i think the below link is perfect
https://docs.microsoft.com/windows-hardware/drivers/kernel/mm-bad-pointer#mmgetsystemaddressformdlsafe

I am editing this article now.

Hi @rodwiddowson,
I went ahead and creating new, standalone pages for many of the macros that had been documented on the mm-bad-pointer page.  I put links to the new pages in a See Also section at the bottom of https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/mm-bad-pointer (and a Note explaining this at the top).
Also note the new TOC in the left pane when you are on that page.
Search engines should also be picking up the new pages as we speak, so hopefully soon you'll get these in search results (which I saw, as you did, that the old setup failed miserably on that score).
Really appreciate your opening this - it was of a tip of the iceberg sort of thing once I got into it!
And thanks to @RAJU2529 for the PR -- hopefully this overhaul will serve you much better.
Feedback welcome :)

@tedhudek   Thanks for updating this article. Still exact location link must be update  .  I have found fee topic  links.
Witg your permission can i edit this article again.

@tedhudek . The following topics links are redirected to  this page https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/mm-bad-pointer
**
MmGetMdlVirtualAddress
MmGetMdlByteOffset
MmInitializeMdl
MmGetSystemAddressForMdlSafe**

Thanks @RAJU2529 I am planning on fixing these links this week.  I'll keep you posted!

Links should all be updated now @RAJU2529 - please test a couple and verify.  Thanks again to both of you for the sharp eye!

@tedhudek. all links are fixed. Thanks


Need time codes on every frame

The python code does not generate time codes, so the frontend relies on parsing the title field, which feels brittle.

@yellowcap not urgent, but would you mind regenerating the geojson files used by the frontend? Can't see the dates field on them.


TransactionScope in 2.1 throwing exception only during debugging in VS2017 15.7.0 Preview 6.0

While testing the .net core 2.1 implementation of TransactionScope I ran into the following issue.
We have a library that configures settings for DB Connections and Transaction scope (Data.Core). Other libraries call this library to execute queries using dapper.
When I execute the code without the debugger, everything runs fine. When I set a breakpoint in the unit test AND also in the method where the DB connection is opened. I then get the exception below.
Any ideas?
System.PlatformNotSupportedException
  HResult=0x80131539
  Message=This platform does not support distributed transactions.
  Source=System.Transactions.Local
  StackTrace:
   at System.Transactions.Distributed.DistributedTransactionManager.GetDistributedTransactionFromTransmitterPropagationToken(Byte[] propagationToken)
   at System.Transactions.TransactionInterop.GetDistributedTransactionFromTransmitterPropagationToken(Byte[] propagationToken)
   at System.Transactions.TransactionStatePSPEOperation.PSPEPromote(InternalTransaction tx)
   at System.Transactions.TransactionStateDelegatedBase.EnterState(InternalTransaction tx)
   at System.Transactions.EnlistableStates.Promote(InternalTransaction tx)
   at System.Transactions.Transaction.Promote()
   at System.Transactions.TransactionInterop.ConvertToDistributedTransaction(Transaction transaction)
   at System.Transactions.TransactionInterop.GetExportCookie(Transaction transaction, Byte[] whereabouts)
   at System.Data.SqlClient.SqlInternalConnection.GetTransactionCookie(Transaction transaction, Byte[] whereAbouts)
   at System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx)
   at System.Data.ProviderBase.DbConnectionPool.PrepareConnection(DbConnection owningObject, DbConnectionInternal obj, Transaction transaction)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Data.Core.DataStore.get_Connection() in D:\TestApps\Repro\Debugger.Issue\Data.Core\DataStore.cs:line 37
   at Data.Core.Tests.ReproTest1.TestMethod1() in D:\TestApps\Repro\Debugger.Issue\Data.Core.Tests\ReproTest1.cs:line 14

This is the test method in Data.Core.Tests:
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace Data.Core.Tests
{
    [TestClass]
    public class ReproTest1
    {
        [TestMethod]
        public void TestMethod1()
        {
            DataStore ds = new DataStore();
            using (var scope = ds.BeginTransactionScope())
            {
                // Set breakpoint on the following line. 1 of 2 breakpoints.
                var conn = ds.Connection;

                // The following will never be executed in debug

                // Run raw ADO.NET command in the transaction
                var command = conn.CreateCommand();
                command.CommandText = "DELETE FROM dbo.SomeTable";
                command.ExecuteNonQuery();
            }

        }
    }
}

This is DataStore class:
using System;
using System.Data;
using System.Data.Common;
using System.Data.SqlClient;
using System.Transactions;

namespace Data.Core
{
    using IsolationLevel = System.Transactions.IsolationLevel;

    public class DataStore
    {

        public DataStore()
        {
            this.DatabaseConnectionString = "Data Source=.;Initial Catalog=SomeDB;Integrated Security=True";
        }

        /// <summary>
        /// Gets or sets the database connection string used to create the database connection.
        /// This cannot be null.
        /// </summary>
        private string DatabaseConnectionString { get; set; }

        /// <summary>
        /// Gets the connection.
        /// </summary>
        public SqlConnection Connection
        {
            get
            {
                // Set breakpoint on the following line. 2 of 2 breakpoints.
                var connection =  new SqlConnection(this.DatabaseConnectionString);

                if (connection.State != ConnectionState.Open)
                {
                    // This line will throw exception
                    connection.Open();
                }

                // See if we need to join an existing transaction scope
                if (Transaction.Current != null)
                {
                    ((DbConnection) connection).EnlistTransaction(Transaction.Current);
                }

                return connection;
            }
        }

        /// <inheritdoc />
        public TransactionScope BeginTransactionScope()
        {
            return new TransactionScope(
                TransactionScopeOption.Required,
                new TransactionOptions { IsolationLevel = IsolationLevel.ReadCommitted });
        }
    }
}

I have attached a project to reproduce the error.
Debugger.Issue.zip
[EDIT] Add C# syntax highlighting by @karelz

do we have any solution
I am running in same issue
with .net core 2.2

Hi @BrotherOdin
I tested this issue today with provided repro application and I was able to reproduce issue with EF Core v2.1.0-rc1-final
But I tried again with EF Core v3.1.9 (netcoreapp3.1) that references Microsoft.Data.SqlClient now and this issue seems to be resolved.
Could someone verify and confirm so we can close this issue?
Thanks!

@David-Engel @DavoudEshtehari
I wonder if this can be closed since there is no activity or reports of the same?

Closing due to inactivity. Feel free to comment if still experiencing the issue.


Memory violation access, crash on start if OS hardened.

Info :

v3.3.rc6.official
Windows 10x64

Issue :
After loading WINSTA.dll, Access violation - code c0000005

( was working fine before RC 6 )

Replication :
Run it on windows 10 with hardened memory protection

Windows Security > App & browser control > Exploit protection settings

Commits between 3.2.4rc5 and 3.3rc6: https://github.com/godotengine/godot/compare/b169a16cb51b7203a171245acb5b4193c9d4bca4...15ff752737c53a1727cbc011068afa15683509be

Might be related to #46900, CC @Ev1lbl0w.
@NotsOverflow Could you confirm that if you try 3.2.4 RC 5 now it doesn't crash? Just to rule out a potential issue linked to a system update.

@akien-mga yep, it always worked and work with any RC and stable. only the RC6 is killed by windows security for memory access violation.

I'm trying to reproduce the issue on Windows 10 but I see somewhat different things.

Replication :
Run it on windows 10 with hardened memory protection
Windows Security > App & browser control > Exploit protection settings

Which setting is that exactly? The only one I had turn off by default is "Force randomization for images (Mandatory ASLR)". Is that the one you're referring to?
If so, after enabling it, I do get several issues:

Godot 3.3 RC 6 closes without error.
Godot 3.2.4 RC 5 also closes without error. Can you confirm again that 3.2.4 RC 5 works fine for you?
Godot 3.2.3 stable works fine.

After loading WINSTA.dll, Access violation - code c0000005

Where do you see the errors? For me Godot just stops without any error message, no feedback given to the user.

So I tested more builds and:

Godot 3.2.4 RC 3 and RC 4 work fine.
Godot 3.2.4 RC 5 and 3.3 RC 6 raise an exception c0000005, found some details in Event Viewer.
A self-compiled build with MSVC from latest 3.x branch doesn't seem to have the issue.

So it's a regression from RC 5, not RC 6, and it affects MinGW builds specifically.
Here's the diff between RC 4 and RC 5:
https://github.com/godotengine/godot/compare/dc99f04d51d6556e5ba4d9cfcce8117d168ac6f1...b169a16cb51b7203a171245acb5b4193c9d4bca4
Potential culprit: https://github.com/godotengine/godot/commit/8feb53e509672b9208abbd56936f6e80e65b3a6e
Makes it seem related to #47219.
CC @RandomShaper @bruvzg @hpvb

@akien-mga I go the info from running it with Windbg preview. I don't have the debug symbols so it doesn't tell me much beside the last loaded dll, register states and exception code.
Settings :
I turned all option "on" in "Exploit protection settings" so it's forced on PE files
also I have ( not sure if related ) :

Windows Security > Device security > Core isolation > On

Findings :

Godot 3.3 RC 6 closes without error.
Godot 3.2.4 RC 5 closes without error.
Godot 3.2.4 RC 4 works fine.
Godot 3.2.3 stable works fine.

( Maybe the windows version should only be compiled with Visual C++ build tools ? )

Didn't have time to test them tonight but I made two binaries from 8feb53e509672b9208abbd56936f6e80e65b3a6e and its parent commit, to see if my hunch is correct that this change caused the regression:
https://downloads.tuxfamily.org/godotengine/testing/windows-debug-47256.zip
There are two .exe in this zip to test both versions.

Didn't have time to test them tonight but I made two binaries from 8feb53e and its parent commit, to see if my hunch is correct that this change caused the regression:

I confirm that 8feb53e (cherry-pick of #46813) is causing the regression.

Fixed by #47368.


Wip/wd/vector editor3

Pull Request Description

Important Notes

Checklist
Please ensure that the following checklist has been satisfied before submitting the PR:

[ ] The documentation has been updated, if necessary.
[ ] Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
[ ] All code follows the
Scala,
Java,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
All code has been tested:

[ ] Unit tests have been written where possible.
[ ] If GUI codebase was changed, the GUI was tested when built using ./run ide build.

Bug for the WASM test failure: #6315


Training with NLG + NLU error

Hello
I am trying to train your DDQ model with NLG + NLU on.
I ran the following command:
python run.py --agt 9 --usr 1 --max_turn 40
	      --movie_kb_path ./deep_dialog/data/movie_kb.1k.p
	      --dqn_hidden_size 80 --experience_replay_pool_size 5000
	      --episodes 500
	      --simulation_epoch_size 100
	      --run_mode 3
	      --act_level 1
	      --slot_err_prob 0.0
	      --intent_err_prob 0.00
	      --batch_size 16
	      --goal_file_path ./deep_dialog/data/user_goals_first_turn_template.part.movie.v1.p
	      --warm_start 1 --warm_start_epochs 100
	      --planning_steps 9
	      --write_model_dir ./deep_dialog/checkpoints/DDQAgent
	      --torch_seed 100
	      --grounded 0
	      --boosted 1
	      --train_world_model 1

But I received this error during the warm_start phase:

Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1668, in <module>
    main()
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1662, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1072, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "run.py", line 535, in <module>
    run_episodes(num_episodes, status)
  File "run.py", line 458, in run_episodes
    warm_start_simulation()
  File "run.py", line 408, in warm_start_simulation
    episode_over, reward = dialog_manager.next_turn()
  File "deep_dialog/dialog_system/dialog_manager.py", line 117, in next_turn
    self.user_action)
  File "deep_dialog/usersims/usersim_model.py", line 304, in register_experience_replay_tuple
    action_idx = self.action_index(copy.deepcopy(user_a_t))
  File "deep_dialog/usersims/usersim_model.py", line 292, in action_index
    raise Exception("action index not found")
Exception: action index not found

It looks like when the user says the following, it is not inside the self.feasible_actions_users, do you know what is going on?

after: {'request_slots': {}, 'nl': 'I want to watch avengers.', 'turn': 2, 'diaact': 'request', 'inform_slots': {'moviename': 'avengers'}}
Turn 2 usr: request, inform_slots: {'moviename': 'avengers'}, request_slots: {}
Turn 2 usr: I want to watch avengers.

Thanks for this feedback.
The intent of NLU is wrong here, it should be an inform action. Just fixed it. You can check out the latest code.
Thanks

Hello @xjli, thanks for your reply.
I just git pull the latest code, and re-ran the command above, I received this another error:
{'request_slots': {'ticket': 'UNK'}, 'inform_slots': {'numberofpeople': 'PLACEHOLDER'}, 'diaact': 'request'}

Thanks, let me check NLU part. The NLU is quite brittle.
You can run the code without NLU (--act_level 0) first.
I will let you know when we fix it.
Thanks

@randyhouse , I fixed the bugs in act_level = 1 mode.
You can check out the latest code.
Thanks so much,


Allow reading/writing text envelopes from pipes

Previously the functions to read text envelopes used readFile from
Data.ByteString. Unfortunately, that doesn't work with pipes, so we've
switched to a function that does.
Additionally, on unix based systems, we've changed how writing to an
owned file works. Previously a new file was created (and since we
created it, we must own it) and then copied into place. Unfortunately
this doesn't work with symlinks, fifos, unix sockets or anything else
that isn't a "normal" file. What we do now is grab the file descriptor,
call fchown on it, which sets us as the owner (or errors out if we can't
do that), and then write to the file.
Fixes https://github.com/input-output-hk/cardano-node/issues/4235

bors r+


Add command-line interface for codegen

Essentially it will simply call regenerate_dir_to_all_versions. Use argparse to limit the number of dependencies our user needs to install. We could call it cadwyn generate-versioned-schemas.
Note to whoever is working on this: the name regenerate_dir_to_all_versions is not great. Please, change it.

I'll do it this weekend.


Add a way to remove images from the database

Currently images stay in the database forever. This is problematic, since images take up by far the largest amount of storage in the app. I can think of the following cases but there is a good chance I'm missing some:

[ ] Deletion of a Person: When deleting a person with an avatar assigned, that avatar should also be removed from the database
[ ] Deletion of a Tree: When deleting a tree, the cover image of the tree needs to be removed, along with each person assigned to the tree (which in turn should delete all their avatars)
[ ] Replacing an existing image by uploading a new image: This is the trickiest case, as the reference to the image is only replaced on pressing the update tree / person button. This is good, as it gives users the option to undo an image change before committing it, but this also means that we can't immediately delete the old image. Worse yet, if a user uploads an image but then closes their browser (or uploads multiple images in a row) these images will never be attached to any tree or person and thus never deleted.

Somewhat related, adding a delete button to remove an image might be useful as well, so a user can decide to remove an image without needing to provide a replacement.


ServusTv plugin question

Hello and thank you so much for your cool plugins.

I tried to adapt servusTVplugin for my local machine yt-dlp build.
Downloaded servustv.py, moved it to yt-dlp/extactors, edited extractors.py and the actual servustv.py file a bit to match paths for .common and ..utils, basically made all the edits needed including a couple methods to utils.py...
However, when trying to compile I'm getting this error below:
/usr/bin/env python3 devscripts/make_lazy_extractors.py yt_dlp/extractor/lazy_extractors.py
Traceback (most recent call last):
File "/yt-dlp/devscripts/make_lazy_extractors.py", line 125, in
main()
File "/yt-dlp/devscripts/make_lazy_extractors.py", line 38, in main
module_src = '\n'.join((
File "/yt-dlp/devscripts/make_lazy_extractors.py", line 79, in build_ies
yield build_lazy_ie(ie, ie.name, attr_base)
File "/yt-dlp/devscripts/make_lazy_extractors.py", line 121, in build_lazy_ie
return s + '\n'.join(extra_ie_code(ie, attr_base))
File "/yt-dlp/devscripts/make_lazy_extractors.py", line 65, in extra_ie_code
val = getattr(ie, var)
File "/yt_dlp/utils.py", line 5348, in get
return self.func(cls)
File "/yt-dlp/yt_dlp/extractor/common.py", line 3649, in age_limit
return max(traverse_obj(
TypeError: '>' not supported between instances of 'tuple' and 'int'
make: *** [yt_dlp/extractor/lazy_extractors.py] Error 1

As your problem is actually related to yt-dlp, I cannot answer it out of the box. Maybe you post your issue on their project.


Deletes most unnecessary ansible dependencies

postgresql
pgweb
geospatial libs
server side app config stuff (directories, upstart jobs, etc)

+1


Prepare 0.25.2 release

Summary
This commit prepares the 0.25.2 release, this consists of bumping the version number to 0.25.2 in all the appropriate files. It also adds a prelude release note to provide a brief introduction to the release. When this commit merges it should be tagged as the 0.25.2 release.
Details and comments

[ ] Wait for last milestone PRs to merge
[ ] Move the 0.25.2 milestone pending PR/issues to 0.25.3
[x] Add prelude release note
[x] Update release notes

edit: I now realise you just copied Matt's commit message from the 0.25.1 release haha. I did idly think it didn't sound like your voice.

lol. You got me :P Fixed in the original description.

Luciano: in case you look at this at all over the weekend, I think there are a few release notes merged into 0.25.2 that may have broken cross-references and typos, etc.  We'll want to fix those, and also because we're backporting to a situation that still has a static release notes file (legacy compat), we'll need to manually run reno report as described in the release procedures for the metapackage and manually update doc/release_notes.rst.


Nextcloud add extraTrustedProxies and/or extraConfig option

Issue description
In the nextcloud module, add an option to set the trusted_proxies configuration option. It is needed when running the nextcloud service behind a reverse proxy.
Alternatively or additional add an extraConfig option to the module.
I'm a nix learner; once I feel comfortable I'll implement this myself. Until then, feel free to implement it or just leave the issue as a reminder.
Technical details

system: "x86_64-linux"
host os: Linux 4.14.76, NixOS, 18.09.1379.04005e5f71a (Jellyfish)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.1.3
channels(root): "nixos-18.09.1379.04005e5f71a, unstable-19.03pre155123.32bcd72bf28"
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

This has already been implemented in #77480, closing here :tada:


Circuit Board sprites

[ ] Autolathe board
[ ] P.A.C.M.A.N.-type portable Generator circuit board
[ ] Power control module
[ ] Fire alarm electronics
[ ] S.U.P.E.R.P.A.C.M.A.N.-type portable Generator circuit board
[ ] Airlock Electronics.
[ ] Computer circuit

#16


adding docker policies

Heyy @pavniii  I have made the changes for resource type following is the new format of config
 {
  "docker_add": [
    {
      "id": "docker_add.9c83e001e6a5018c8cb8431b9c25d0e0",
      "name": "Dockerfile",
      "source": "Dockerfile",
      "line": 4,
      "type": "docker_add",
      "config": "setup.sh terrascan.sh launch-atlantis.sh entrypoint.sh /usr/local/bin/",
      "skip_rules": null,
      "max_severity": "",
      "min_severity": ""
    },
    {
      "id": "docker_add.1e9374244a7e9316f9f83b640ace542e",
      "name": "Dockerfile",
      "source": "Dockerfile",
      "line": 8,
      "type": "docker_add",
      "config": "terrascan-workflow.yaml /etc/atlantis/workflow.yaml",
      "skip_rules": null,
      "max_severity": "",
      "min_severity": ""
    }
  ]
}

I have modified all the policies merged earlier, please pull the latest changes and modify your policies accordingly.
Thank you


Only generates testnet addresses

Current implementation appears to be only generating addresses regardless of the -test flag

Fixed in https://github.com/blackkeyboard/zgenerate/commit/ba44320fff607c7cbd95da301b6676298f676063


RFP - Ledger Live

Checklist

[x] The package I am requesting does not already exist on https://chocolatey.org/packages;
[x] There is no open issue for this package;
[x] The issue title starts 'RFP - ';
[x] The download URL is public and not locked behind a paywall / login;

Package Details
Software project URL : https://shop.ledger.com/pages/ledger-live
Direct download URL for the software / installer : https://github.com/LedgerHQ/ledger-live-desktop/releases/download/v1.18.2/ledger-live-desktop-1.18.2-win.exe
Software summary / short description:
Ledger Live is a new generation wallet desktop application providing a unique interface to maintain multiple cryptocurrencies for your Ledger Nano S / Blue. Manage your device, create accounts, receive and send cryptoassets, ...and many more.

Initial version has just been pushed to the repository, and now awaiting feedback.
https://community.chocolatey.org/packages/ledger-live-desktop/2.35.1

facepalm
Turns out I should've double-checked someone else didn't beat me to it. The requested package is already available here:
https://community.chocolatey.org/packages/ledger-live
This issue can be closed.


GetVideoContext: can't bypass age restriction: login required to confirm your age

Using library
import ytdl "github.com/kkdai/youtube/v2"
few days ago ytdl.Client.GetVideoContext started to return error:
can't bypass age restriction: login required to confirm your age
for literally all videos.

I'm seeing the same problem.

Related to https://github.com/yt-dlp/yt-dlp/issues/10128
TL;DR
Seems like YouTube has blacklisted your personal/virtual machine's IP to disallow use of downloaders. You'll have to change the virtual machine or wait for your IP to be changed/refreshed by your Internet service provider. The only solution the community has found is using cookies (obtained from an existing YouTube account) but it can lead to a temporary or a permanent ban.
Other solutions:

Use VPN or proxy server

@ppalone Are you certain it's related? That's a different error message.

@ppalone thanks a lot for this!

tried from a different server from a very different subnet, different datacenter, different country - still getting the same error message, but youtubedr still works from my laptop.

but youtubedr gives the same error message
can't bypass age restriction: login required to confirm your age
from the same server host

so i guess it is indeed the ban thing

Hello @shoce,
Were you able to find a workaround to this problem?
I'm facing the same issue and can't understand why it works on my computer but doesn't on my EC2 instance.

Hello @shoce, Were you able to find a workaround to this problem? I'm facing the same issue and can't understand why it works on my computer but doesn't on my EC2 instance.

Hey, I still haven't found a solution. The reason it works on your/my personal computer and doesn't work on EC2 instances or VMs in other hosting data centers is the IP address. Seems like google just banned most of IP addresses that belong to servers in data centers. I guess if you set up a VPN server on your EC2 instance and connect your computer to that VPN then you will see youtubedr does not work too. I still do not know what to do. Most probably going to host on my laptop for some time :)


Prepare E-charts in backend

We want to start preparing charts in backend.
For charts we typically use e-charts
We want to prepare classes or functions for following types of charts (see @ https://reenact-prototype.vercel.app/questionsDefault/regionToday):

This means data, title, units, colors are given to function and chart data is created from it.
Following chart types shall be prepared (from upper left to lower chart):

[ ] bar chart,
[ ] bar chart with multiple bars per x-tick,
[ ] stacked bar chart

Result of this chart classes or functions shall be a dict whcih can be used in e-charts to render chart.
We did this already in other projects by preparing a default E-charts JSON which is enriched with data in function.
Afterwards this dict is given to Frontend via JsonResponse and gets rendered.
See Charts class here: https://github.com/empowerplan/epp-app/blob/41538037f8dae40aead48778e9baf85ffde18b33/digiplan/map/charts.py#L15
which takes a lookup (default JSON template for e-chart) and data and has a function render() which returns ready chart.
You could adapt/reuse this if you want.

plotly instead of e-charts


WSAPatch has broken in v2304 of WSA

@cinit WSAPatch is no longer working with WSA v2304.40000.3.0 and higher, giving the following error:

Please can you have a look at the issue on hand.
Many thanks,
MustardChef

I have corresponded with @cinit and have been  told that they are busy with their studies and will work on the patch in the coming weeks after they have some free time. I would revert back to 2303.40000.5.0 in the meantime by either getting WSA from here https://github.com/MustardChef/WSABuilds/releases/tag/Windows_10_2303.40000.5.0

@MustardChef Hi mate. Apparently WSA_2305.40000.3.0_x64 seems to work smooth enough for me. I am running window 10 ltsc (21h2).
How did I build it? Downloaded the corresponding release from WSABuilds which is by default meant for win11. Downloaded patches from WSAPatch and placed them at the respective place, then edited appxmanifest.xml and ran run.bat.
At this stage, I did encounter an issue where the resource language and resource uap nodes in appxmanifest.xml got repeated twice after running run.bat and then install.ps1 would throw error, so simply edited out the extra instances, don't know what triggered that error but it's surely something related to run.bat, happened twice.
Later, also changed the location of the folder in which resides userdata.vhdx, and works completely fine as well.
Rest, it simply works good, and that for apps so for it's only play store that might close once in a while usually during initiating wsa, later on it's fine, and anyways wsa is still up in bg if play store crashes, just reopen it. Pretty stable for daily usage imho. Note it was a fresh install, no amazon store and had gapps. Cpu is i5-3230m if that matters.
Lastly, thanks a lot @MustardChef and @cinit for the awesome work, you didn't leave a single reason for me to switch to windows 11, I still feel windows 10 is much better and has better ui, not to mention it's customizable enough.

@Ansh4011 Can you explain more about your way please

@Ansh4011 Can you explain more about your way please. I can't understand this line

Later, also changed the location of the folder in which resides userdata.vhdx, and works completely fine as well.

I was referring to step 3 of this guide on MustardChef's repo WSABuild. https://github.com/MustardChef/WSABuilds/blob/master/Guides/MoveToExtDrive.md#step-3-moving-the-appdata-vhdx--data-from-your-windows-drive-c-to-the-desired-partitiondrive

So you mean we must install WSA at other drive then use mklink to make symlink for working with Windows 10 ?

@Ansh4011 So you mean we must install WSA at other drive then use mklink to make symlink for working with Windows 10 ?

@Ansh4011 And can you please tell me which DLL version you are using for patching ?

@Ansh4011 So you mean we must install WSA at other drive then use mklink to make symlink for working with Windows 10 ?

Oh no, what I actually meant was that it works that way as well, as I said, 'later', so its not a requirement whatsoever and rather completely optional, just saves me a hefty amount of space in c drive.

@Ansh4011 And can you please tell me which DLL version you are using for patching ?

20230119

@Ansh4011 Wait what. Nothing different. Can you test with this build (It have auto patch, so no need to patch DLL mannually)
https://github.com/YT-Advanced/WSA-Script/releases/latest

@Ansh4011 Wait what. Nothing different between your way with my patch. Can you test with this build (It have auto patch, so no need to patch DLL mannually) https://github.com/YT-Advanced/WSA-Script/releases/latest

Was about to install, but just checked the patch files are not present there neither is appxmanifest.xml patched.

@Ansh4011 Haha, I have added patcher to powershell script, just install and all of them will be patched, include DLLs and appxmanifest

@YT-Advanced

Oh wait browser didn't refresh lol, that's smart haha. Will try in a while, in class rn.

@Ansh4011 You can check these line

@Ansh4011 You can check these line

@YT-Advanced Yea I tried v2304, doesn't seem to work, tried your repo and wsabuilds too. But then out of curiosity I ran v2305, the pre-release from your repo, modified install.sh for 21h2 and it worked, worked smooth af. Seems like the issue was only with v2304.

I can't understand the line below. You mean v2305 from my repos worked smooth without any modify from you ?

the pre-release from your repo, modified install.sh for 21h2 and it worked

@Ansh4011 I can't understand the line below. You mean v2305 from my repos worked smooth without any modify from you ?

the pre-release from your repo, modified install.sh for 21h2 and it worked

I was referring to this 22h2, bcz I run 21h2 so I had to edit it to make it work.

Oh, thanks. I will modify it :)))

So does this mean we can install the latest version on Windows 10 as is or not? Any step-by-step example otherwise?

So does this mean we can install the latest version on Windows 10 as is or not? Any step-by-step example otherwise?

I think we can.

Oh, thanks. I will modify it :)))

Welcome 😄

So does this mean we can install the latest version on Windows 10 as is or not? Any step-by-step example otherwise?

https://github.com/cinit/WSAPatch/issues/33#issuecomment-1583364007

P/s: I'm building with new Powershell script
https://github.com/YT-Advanced/WSA-Script/actions/runs/5217274339

@YT-Advanced Good job man! Need me to test it? Can do today, cause I'll be busier tmrw onwards.

@YT-Advanced Good job man! Need me to test it? Can do today, cause I'll be busier tmrw onwards.

@Ansh4011 Yes, you can test it

You can get WSA Builds v2305 for Windows 10 from here: https://github.com/MustardChef/WSABuilds/releases/tag/Windows_10_2305.40000.3.0


Refactor and fix container streaming.

This PR:

Cleanup and refactor container streaming code.
Add our own IO pipe logic for exec. The new exec io pipe logic has similar behavior with attach. (Actually in docker, exec and attach uses the same piece of io pipe code, https://github.com/moby/moby/blob/master/container/stream/attach.go#L60).

This fixes the e2e test timeout. https://k8s-testgrid.appspot.com/sig-node-containerd#e2e-gci
The test Kubectl should handle in-cluster config timed out, because it assumes that after input ends with EOF, exec stdin should be closed, and tar xvf - XXX should stop.
However, in default containerd io pipe, even if stdio finishes, it still holds the container stdio open. So the test blocks on the following line forever:
INFO: Running '/workspace/github.com/kubernetes-incubator/cri-containerd/kubernetes/platforms/linux/amd64/kubectl --server=https://<IP_ADDRESS> --kubeconfig=/workspace/.kube/config cp /workspace/github.com/kubernetes-incubator/cri-containerd/kubernetes/platforms/linux/amd64/kubectl e2e-tests-kubectl-hzzv7/nginx:/tmp/'

With this PR, we'll close exec stdin if input finishes. It should fix the test timeout.
Signed-off-by: Lantao Liu<EMAIL_ADDRESS>

@mikebrow Addressed comments. Thanks for reviewing! Very good cleanup suggestion. :)


Enabled single example for example tables in HTML report

I'm currently using BDDfy for my UI testing project and used the example table to increase readability. Eventhough example tables might not be designed for single examples, doing so was beneficial at least in my case.

Interesting. Yes, I found it odd that if you create a single example it displays in the normal way rather than as a single example. Will be interested to hear @JakeGinnivan and @MehdiK take on this.

Honestly i forget why I did that. This makes sense to me

Thanks @VanDesu . Nice addition!


Use scheduled_at on filtered hours report

Can you modify the travisci config to make sure it runs on ruby 2.2 or so?


Update Turbopack CSS docs

Description
Updates the Turbopack CSS documentation to reflect reality.

Testing Instructions

#7925  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @timneutkens and the rest of your teammates on  Graphite