make error: vfork: resource temporarily unavailable

Hi,
I'm trying to install OpenBLAS 0.2.20 on a node in a local directory over which I have permissions (I don't have root access). I seem to be encountering an error when attempting the build process. Trying make with or without any flags is resulting in a long string of errors that look like:
make[1]: vfork: Resource temporarily unavailable
I've posted the whole output in a text file. Could anyone give any suggestions on how to troubleshoot the problem?
make_error.txt
Many thanks. Tim
EDIT: In case this is useful, here is also the output of a few server parameters.
uname - or
<IP_ADDRESS>-0.3-default GNU/Linux
lsb_release -a
LSB Version:	core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:desktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch
Distributor ID:	SUSE LINUX
Description:	SUSE Linux Enterprise Server 11 (x86_64)
Release:	11
Codename:	n/a
cat /etc/*-release
LSB_VERSION="core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64"
SGI Accelerate 1.3, Build 705r10.sles11-1110192111
SGI Foundation Software 2.5, Build 705r10.sles11-1110192111
SGI MPI 1.3, Build 705r10.sles11-1110192111
SGI Performance Suite 1.3, Build 705r10.sles11-1110192111
SGI UPC 1.3, Build 705r10.sles11-1110192111
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 1
lscpu:
Architecture:          x86_64
CPU(s):                64
Thread(s) per core:    1
Core(s) per socket:    8
CPU socket(s):         8
NUMA node(s):          8
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 46
Stepping:              6
CPU MHz:               2266.424
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              24576K

Looks like you are running out of resources - be it available memory or number of concurrent processes.
Try reducing the number of subprocesses started by make - either set MAKE_NB_JOBS to a low number (it defaults to the number of cores detected, which may be inappropriate for your virtual system), or run with NO_PARALLEL_MAKE=1 (see Makefile.rule for available make options)

Thanks for the comment. I tried setting the NO_PARALLEL_MAKE option in Makefile.rule but I seem to have stumbled into another problem. I've attached the last few lines below. I'm fairly new to compiling these programs so I may have missed something obvious!
I had also set the prefix options and the gfortran thread safe options.
make[2]: Leaving directory /home/FIa/FIa164/programs/openblas/OpenBLAS-0.2.20_src/lapack/trtri' make[1]: Leaving directory /home/FIa/FIa164/programs/openblas/OpenBLAS-0.2.20_src/lapack'
make[1]: Entering directory /home/FIa/FIa164/programs/openblas/OpenBLAS-0.2.20_src/lapack-netlib' ( cd INSTALL; make ) make[2]: Entering directory /home/FIa/FIa164/programs/openblas/OpenBLAS-0.2.20_src/lapack-netlib/INSTALL'
gfortran -openmp -fp-model precise -I/ap/vni/imsl/fnl700/lnxsg111e64/include -O2 -frecursive -Wall -m64 -fPIC -c -o lsame.o lsame.f
gfortran: precise: No such file or directory
f951: error: unrecognized command line option "-fp-model"
make[2]: *** [lsame.o] Error 1
make[2]: Leaving directory /home/FIa/FIa164/programs/openblas/OpenBLAS-0.2.20_src/lapack-netlib/INSTALL' make[1]: *** [lapack_install] Error 2 make[1]: Leaving directory /home/FIa/FIa164/programs/openblas/OpenBLAS-0.2.20_src/lapack-netlib'
make: *** [netlib] Error 2

Can you share custom "gfortran thread safe" parameters you are using?
fp-model is Intel compiler flag.
Does it finish compilation by simple 'make clean ; make MAKE_NB_JOBS=1' ?

Your fortran compiler does not understand the option "-fp-model precise" (which it probably picked up from some FLAGS variable in your environment - the option appears to be specific to the Intel compiler
while you are using gfortran)

Thanks for the thoughts. This was the Makefile.rule I was using.
Makefile.rule.txt
I will try again with 'make clean ; make MAKE_NB_JOBS=1'

It got a little closer! Using the above (only adding a prefix and MAKE_NB_JOBS=1), it seems that it has defaulted to using ifort, which seems to understand the '-fp-model precise' flag but has the following issue. The last few lines are posted below. Did it fail attempting to compile a fortran code?
...
ifort -openmp -fp-model precise -I/ap/vni/imsl/fnl700/lnxsg111e64/include -O2 -fPIC -c ssytrd_sb2st.F -o ssytrd_sb2st.o
ssytrd_sb2st.F(486): error #5082: Syntax error, found IDENTIFIER 'DEPEND' when expecting one of: UNTIED PRIVATE FIRSTPRIVATE SHARED IF DEFAULT  ;
!$OMP TASK DEPEND(in:WORK(MYID+SHIFT-1))
-----------^
ssytrd_sb2st.F(487): error #5082: Syntax error, found IDENTIFIER 'DEPEND' when expecting one of: ( % . = =>
!$OMP$     DEPEND(in:WORK(MYID-1))
-----------^
ssytrd_sb2st.F(488): error #5082: Syntax error, found IDENTIFIER 'DEPEND' when expecting one of: ( * ) :: ,  ; + . - % (/ [ : ] /) . ' ** / ...
!$OMP$     DEPEND(out:WORK(MYID))
-----------^
ssytrd_sb2st.F(488): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :
!$OMP$     DEPEND(out:WORK(MYID))
---------------------------------^
ssytrd_sb2st.F(497): error #5082: Syntax error, found IDENTIFIER 'DEPEND' when expecting one of: UNTIED PRIVATE FIRSTPRIVATE SHARED IF DEFAULT  ;
!$OMP TASK DEPEND(in:WORK(MYID+SHIFT-1))
-----------^
ssytrd_sb2st.F(498): error #5082: Syntax error, found IDENTIFIER 'DEPEND' when expecting one of: ( % . = =>
!$OMP$     DEPEND(out:WORK(MYID))
-----------^
ssytrd_sb2st.F(497): error #6404: This name does not have a type, and must have an explicit type.   [DEPEND]
!$OMP TASK DEPEND(in:WORK(MYID+SHIFT-1))
-----------^
ssytrd_sb2st.F(497): error #6514: A substring must be of type CHARACTER.   [DEPEND]
!$OMP TASK DEPEND(in:WORK(MYID+SHIFT-1))
-----------^
ssytrd_sb2st.F(497): error #6404: This name does not have a type, and must have an explicit type.   [IN]
!$OMP TASK DEPEND(in:WORK(MYID+SHIFT-1))
------------------^
ssytrd_sb2st.F(498): error #6514: A substring must be of type CHARACTER.   [DEPEND]
!$OMP$     DEPEND(out:WORK(MYID))
-----------^
ssytrd_sb2st.F(498): error #6404: This name does not have a type, and must have an explicit type.   [OUT]
!$OMP$     DEPEND(out:WORK(MYID))
------------------^
compilation aborted for ssytrd_sb2st.F (code 1)
make[2]: *** [ssytrd_sb2st.o] Error 1
make[2]: Leaving directory /home/FIa/FIa164/programs/openblas/OpenBLAS-0.2.20_src/lapack-netlib/SRC' make[1]: *** [lapacklib] Error 2 make[1]: Leaving directory /home/FIa/FIa164/programs/openblas/OpenBLAS-0.2.20_src/lapack-netlib'
make: *** [netlib] Error 2

You can try recording output from compilation with "script"

Can you run plain "make" in freshly cloned/unzipped OpenBLAS 0.2.20 or git develop branch?
You inherit some *_FLAGS from unrelated products, probably built next door. i dare not to guess what could be
vni/imsl/fnl700/lnxsg111e64

Which version of ifort is that ? Apparently it does not cope with some of the OPENMP directives, which would suggest a rather old version of the Intel compiler. (Having what appears to be the include file path of the IMSL library in the compiler options probably does not hurt, but is not useful either - you may want to check with whoever administers this system which compiler and options to use there)

Hi, I had a chat with one of the admins and they suggested switching to a more modern node - you were correct, the previous was an old one! I tried a fresh build using make PREFIX=/home/FIa/FIa164/programs/openblas/OpenBLAS-0.2.20. (I assume adding the prefix is ok? Since I don't have permissions for the other directories).
Please see attached for the output script.
make_output.txt
There still seems to be some kind of issue with the compile? Thanks for the help.
If you nmeed the info, here are the new node parameters:
uname -or
3.0.101-108.13-default GNU/Linux
lsb_release -a
LSB Version:	core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:desktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch
Distributor ID:	SUSE LINUX
Description:	SUSE Linux Enterprise Server 11 (x86_64)
Release:	11
Codename:	n/a
lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                640
On-line CPU(s) list:   0-639
Thread(s) per core:    1
Core(s) per socket:    10
Socket(s):             64
NUMA node(s):          64
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 62
Stepping:              4
CPU MHz:               2399.982
BogoMIPS:              4802.71
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              25600K
NUMA node0 CPU(s):     0-9
... and a lot of NUMA nodes here ...

Compiler problems cannot be solved in absence of versions, and probably you need to approach intel support website if you want to interlink their modules with gcc ones.
I wonder if this is relevant?
https://github.com/xianyi/OpenBLAS/wiki/Precompiled-installation-packages#opensusesle
Can you cut last CPU core ID from /proc/cpuinfo, it has all the sibling numbers, and most notably lets identify CPU generation you have.

The Intel fortran compiler on that node still seems unable to handle OpenMP4.0 directives. Do you want to use it anyway, or is it just picked as first choice because of something in your environment ? (You could try with make FC=gfortran after a make clean, assuming that the GNU fortran compiler is installed on that node as well)

Actually "make clean" is absent in all build logs....

What are compiler versions? gcc --version ; gfortran --version
btw LSB does not define any compiler standard interface, no need to post it at all.

Here is the output using gfortran make PREFIX=/home/FIa/FIa164/programs/openblas/OpenBLAS-0.2.20 FC=gfortran: make_output.txt
It seems even is it still not recognising some of the commands. Do I have to setup the server in any other way? Thanks.

After a quick google, FFLAGS are fortran flags is that correct? I did a quick echo $FFLAGS and it returned the following:
-openmp -axAVX -fp-model source -I/ap/vni/imsl/fnl701/lnxsg121e64/include
I don't know if they are necessary for anything on the node, but I suppose that these are the ones causing issues? How should I remove them temporarily? Is it safe to do so?

unset FFLAGS should do the trick. (You will need them only when you want to build something with the Intel fortran compiler, and depending on where and when these defaults were added to your user environment they may not even be appropriate for the current compute node)

I see, so they could be remnants of an admin's setup? After running unset FFLAGS, should I try the default FC or go with gfortran again? Thanks @martin-frbg

Go with gfortran (or try to get an admin to look into why your intel compiler installation does not handle openmp4 directives - as far as I can find out, these should be supported by ifort since the 2015 version. Maybe yours is just too old - your gfortran is quite old as well)

can you attach what "set" outputs so we can help you to tidy it up?
The problem is caused by openblas build system using all sorts of CFLAGS etc temporarily internally and not making good friends if they are set ahead of time.

Hello, after a chat with the admins, I'll try to build the latest gfortran for this (and other builds I need) before trying again.
I don't know if this will still be an issue, but I tried building with a slightly newer version of gcc packaged with openfoam but received an error regarding max_nodes (NUMA) is too small? I've attached the output here: make.txt
Also here is the result of set: set.txt
Thanks for the install support.

You must use BIGNUMA=1 build flag for >256 CPU cores (see Makefile.rule)
For variables:
$unset FC F90 F90FLAGS
$make clean ; make BIGNUMA=1

I just built gfortran 7.2.0 in my local home directory and attempted to run the make process again, specifically running:
make clean
unset FC F90 F90FLAGS FFLAGS
make PREFIX=/home/FIa/FIa164/programs/openblas/OpenBLAS-0.2.20 FC=gfortran BIGNUMA=1
It now seems to reach the test stage before failing with GotoBLAS : Can't open shared memory. Terminated. The full output is attached here: make.txt
And to confirm versions: versions.txt

Strange duplicity as I have just committed a change that will show a more detailed error message when shared memory allocation fails. The background for that was #1351 where use of SELinux apparently blocked access to shared memory segments, though in your case it happens even earlier when OpenBLAS tries to allocate a shared memory segment of size 32768.
Perhaps this information gives your local sysadmin a clue already ? There is nothing in the manpage for shmget() that suggests we are trying something naughty, though for smaller systems without BIGNUMA=1 set the requested size is only 4096 here. Unfortunately I have no personal experience with your class of hardware.

From #1351, you should be able to get around this by setting setenforce 0 if it happens to be SELinux-related, provided you have sufficient rights to use the setenforce command. Interventions by this kind of security-hardening software will probably also be logged somewhere on the system.

@martin-frbg it seems like the setenforce command does not exist on my distro - at least I was presented with: bash: setenforce: command not found. I think the supercomputer is running SUSE. Should I try to run a single core compile? Or will this not help?

You would probably need to be root to use setenforce, but it can also be that SELinux is not installed.
A single core compile would certainly work but it depends on your use case if a single-threaded OpenBLAS library is suitable for you - if you want to use it from a program that already creates multiple threads for its calculations it would probably be fine even in terms of performance.
If everything is singlethreaded, you would obviously lose most of the advantages of the multicore system, but at least you would be able to run programs that make use of BLAS or LAPACK functions (and still get some speedup compared to the netlib reference implementation).

Do I have many troubleshooting iodine left for the multi threaded install?
On 10 Nov 2017 19:02, "Martin Kroeker"<EMAIL_ADDRESS>wrote:

You would probably need to be root to use setenforce, but it can also be
that SELinux is not installed.
A single core compile would certainly work but it depends on your use case
if a single-threaded OpenBLAS library is suitable for you - if you want to
use it from a program that already creates multiple threads for its
calculations it would probably be fine even in terms of performance.
If everything is singlethreaded, you would obviously lose most of the
advantages of the multicore system, but at least you would be able to run
programs that make use of BLAS or LAPACK functions (and still get some
speedup compared to the netlib reference implementation).
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/xianyi/OpenBLAS/issues/1348#issuecomment-343430610,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQD-Etan2NhprrRpr0RdowkpCJ9bhIPnks5s1B9KgaJpZM4QSyL9
.

Should be fine as long as the trouble does not start to shoot back (though with that big supercomputer, you may be venturing where few have gone before).
You'd need to get the current development snapshot of OpenBLAS from github to get the better error messages (use the green "Clone or Download" button to the right of the table heading on the "Code" tab here to either follow the instructions for a "git clone" call or to download a zip archive). And if possible talk to some local admin about what could cause accesses to shared memory to fail (maybe you just need to get appropriate user permissions on that supercomputer)

Hi, I have spoken to the admin and they said they will investigate. In the meantime, I downloaded a new copy of develop and attempted a fresh install. I have attached a copy of the output in the following file. It appears to finish compiling but fails on the tests with a 'Segmentation fault' - does the new install present any new information? Many thanks.
make_output.txt

Strange that it segfaults now without printing the cause of the shmget error, maybe I did something wrong in that recent change after all. (Could be the last error number (errno) is already undefined when I try to print it, as the shmget is retried in a loop.)

If you have the ipcs command available, you could run ipcs -l to see what if any shared memory resources are available to you.

You can run your test program withing gdb (or admin can do)
As minimum show output of:
gdb ./testprog
gdb> run
Signal crash whatever
gdb> t a a bt
(this is the interesting thing, might be long for 1000 threads)

@brada4 I think he (still) does not get far enough to have more than one thread - the fundamental problem is still the same (unable to obtain shared memory, hence unable to set up two threads in the tests), only now there is a segfault without any error message instead of the more verbose message I had committed for this case. The only reason for this (I think) is that errno has become undefined by
the time the code reaches the perror() call and it is the perror printout itself now that is segfaulting due to this.

There should be some grain in bt-s

Methinks driver/others/init.c should be updated to this here (rename back from .txt to .c)
init.txt but I am having trouble updating my fork for a regular PR at the moment. @timjim333 perhaps you can just replace the file and recompile to see if this brings some more useful info ?

I'll give the recompile a go. Meanwhile, it seems that the server has ipcs, here is the output:
FIa164@afispb07:~/programs/openblas/OpenBLAS-0.2.20_src$ ipcs -l

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) =<PHONE_NUMBER>9481983
max total shared memory (kbytes) =<PHONE_NUMBER>427386880
min seg size (bytes) = 1

------ Semaphore Limits --------
max number of arrays = 1024
max semaphores per array = 250
max semaphores system wide = 256000
max ops per semop call = 32
semaphore max value = 32767

------ Messages Limits --------
max queues system wide = 32768
max size of message (bytes) = 65536
default max size of queue (bytes) = 65536

Nothing obvious from that, looks similar to my laptop except perhaps for the memory size ...

@brada4 apologies, I'm not sure I follow regarding your post on gdb. I could start gdb but I'm not sure what testprog file to test in particular - what would be a sufficient? Is signal crash a ctrl-c? Thanks.

And here is the ouput from a recompile with the new init.c (I ran make clean before this).
make_output.txt

Should be gdb ./sblat1 in tests/

I attempted to run the test but not a lot came out of it: gdb_test.txt
Was this the expected output?

Can you try OPENBLAS_NUM_THREADS=2 gdb ./sblat1 please ?

Please see attached. Not much seems to have changed!
gdb_test2.txt

Nothing could have changed, please put that on one line

You need to repeat "r" maybe multiple times until it crashes with same message as during build.

@brada4 Sorry, I'm not sure what you mean? Should I enter r after calling run?

Seems he wanted you to repeat the run (from within gdb) as it did not crash the first time.
My suspicion is that you need to have the OPENBLAS_NUM_THREADS=2 and gdb ./sblat1 on the same command line to ensure the test actually runs with two threads (but that depends on the shell you use, in bash that is usually the default on SuSE your approach is fine).

Is this what you mean? I called run quite a few times but it did not crash. Should I keep calling run?gdb_test3.txt

Can you try with spaces where you put the ampersands, to make sure the THREADS setting ends up in the same environment  that sblat1 sees ?
OPENBLAS_NUM_THREADS=2 gdb ./sblat1

Something is different between when the test runs and under GDB but I dont understand what.

Maybe there is a memory problem somewhere in the BIGNUMA code, but at the very least I would expect it to get as far as previous runs, i.e. to the point where the code printed the "cannot open shared memory" message and should be printing a more detailed message including a reason now. So my bet is still on the OPENBLAS_NUM_THREADS setting not getting seen by the code.

That did the trick and triggered the seg fault. gdb_test4.txt

I seem to have had a successful compile - under the suggestion of the admin, I added the NO_AFFINITY flag in the make call: make PREFIX=/home/FIa/FIa164/programs/openblas/OpenBLAS-0.2.20 FC=gfortran BIGNUMA=1 NO_AFFINITY=1. The output can be seen here: make_output.txt
This is still with the replaced init.c that you sent earlier. Has this produced the expected result and will installing this build produce a working copy of OpenBLAS?

Yes, you have usable OpenBLAS, and you pointed to issue that BIGNUMA=1+NO_AFFINITY=0 causes failure initialising thread affinity.

Yes, looks like this produced a usable build (though performance may be decreased by threads getting rescheduled to a different processor occasionally). I will need to look at the code path leading to the shared memory allocation again, seems the informational message I added to the latest init.c may be wrong and the code will just lose cpu affinity but not multithreading when the shmget fails.
Unfortunately the gdb backtrace does not show the failing call, OpenBLAS would need to be build with
DEBUG=1 to add the necessary symbols for the debugger. (Apologies for not mentioning this earlier).
So far all that can be learned is that it segfaults in the routine that initiates the previously failing calls
to shmget/shmat - maybe now that these failures are handled, it staggers on a bit beyond them.

I see, so in the meantime I can link to this build then. Meanwhile, I'm
happy to rebuild with a debug flag if that helps get to the root of the
problem.
On 17 Nov 2017 18:21, "Martin Kroeker"<EMAIL_ADDRESS>wrote:

Yes, looks like this produced a usable build (though performance may be
decreased by threads getting rescheduled to a different processor
occasionally). I will need to look at the code path leading to the shared
memory allocation again, seems the informational message I added to the
latest init.c may be wrong and the code will just lose cpu affinity but not
multithreading when the shmget fails.
Unfortunately the gdb backtrace does not show the failing call, OpenBLAS
would need to be build with
DEBUG=1 to add the necessary symbols for the debugger. (Apologies for not
mentioning this earlier).
So far all that can be learned is that it segfaults in the routine that
initiates the previously failing calls
to shmget/shmat - maybe now that these failures are handled, it staggers
on a bit beyond them.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/xianyi/OpenBLAS/issues/1348#issuecomment-345188965,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQD-Evr8VL3JLk0cuOU11j_d_dE9-V19ks5s3U__gaJpZM4QSyL9
.

... and once permanent code fix is in place ... just replace library file with new one...
i.e dont close the issue, your help with big system testing will be handy.

Right, so I will rebuild with make PREFIX=/home/FIa/FIa164/programs/openblas/OpenBLAS-0.2.20 FC=gfortran BIGNUMA=1 DEBUG=1 and attempt torun the gdb tet again.

I built a a debug build using the following and ran the gdb test. I've attached the results below - I hope it helps.
unset FC F90 F90FLAGS FFLAGS
make PREFIX=/home/FIa/FIa164/programs/openblas/TEST_OpenBLAS-0.2.20 FC=gfortran BIGNUMA=1 DEBUG=1

make_debug.txt
debug_out.txt

Not sure yet what to make of this - the node_info array is declared to hold MAX_NODES (128) and MAX_BITMASK_LEN members so I would not expect it to overflow in this initialization loop (unless the calculation of MAX_BITMASK_LEN went wrong much earlier - this is the number of cpus as returned by the CPU_SETSIZE macro divided by 64).
Can you do
print j
print  node
print MAX_BITMASK_LEN

where you currently have the t a a bt please ?

Here is the output. There was no MAX_BITMASK_LEN: debug_out2.txt

Hmm. Seems one would need to build with a higher debug level flag to get access to #defined constants.
But checking on my system, CPU_SETSIZE appears to be 4096 (although the manpage claims 1024) on all Linux platforms at the moment, and 8*sizeof(unsigned long) should be either 32 or 64 so the loop should still do fine on element 59 of 64.
Guess you could try replacing the single occurence of CPU_SETSIZE near the top of driver/others/init.c by some number that is only somewhat bigger than your actual 640 cpus, say 896 just to see if
this changes anything. On the other hand I think it should be possible to trick a BIGNUMA build into running into the problematic code on my small system so maybe I can track this down myself.

Well, replacing the CPU_SETSIZE by 896 works for me here, while with CPU_SETSIZE it happens to fail at j=59 (though different nodes number) as well. My current thinking is that the struct just gets too big to fit on the stack.

Turns out there appears to be (or have been) some disagreement between glibc maintainers, in particular from SuSE about raising the value (as set in /usr/include/bits/sched.h) for CPU_SETSIZE from 1024 to 4096. (As far as I understood the discussion threads, this was to reflect increased capability of the Linux kernel when built with "maximum NUMA nodes" option, but the criticism was that it broke the API). The last discussion appears to have taken place here, which contains some
hints for correct usage of sched_getaffinity() on big systems that may be relevant for OpenBLAS:  https://sourceware.org/ml/libc-alpha/2016-03/msg00043.html
At the very least it is conceivable wernsaar's (experimental) code for BIGNUMA support was never tested, nor expected to work beyond the "traditional" value of 1024. In my absolutely unscientifc tests,
2048 appeared to still work (although I currently do not own a laptop capable of traversing all affected code paths  :-) ) so my suggestion is to replace the current use of CPU_SETSIZE with a constant 2048 or 1024 as a quick fix.

3232 appears to be about the limit that still survives the compile tests for me, 3264 is already crashing.

@martin-frbg in init.c, can I confirm, you mean that I should try replacing the below block:
#if defined(BIGNUMA)
// max number of nodes as defined in numa.h
// max cpus as defined in sched.h
#define MAX_NODES	128
#define MAX_CPUS	CPU_SETSIZE
#else
#define MAX_NODES	16
#define MAX_CPUS	256
#endif

with:
#if defined(BIGNUMA)
// max number of nodes as defined in numa.h
// max cpus as defined in sched.h
#define MAX_NODES	128
#define MAX_CPUS	1024
#else
#define MAX_NODES	16
#define MAX_CPUS	256
#endif

then make clean, recompile and run the gdb test, is that correct?

Yes, exactly. If it works, you should already see the build pass all tests.

@martin-frbg Is there an alternative debug level I should set, or should I go with make PREFIX=/home/FIa/FIa164/programs/openblas/TEST_OpenBLAS-0.2.20 FC=gfortran BIGNUMA=1 DEBUG=1 again?

You can add NO_LAPACK=1 NO_CBLAS=1 , it will not make complete library usable for you, but just bare minimum to get fastest to the test that fails. Sure 'make clean' between tries.

I'd just do a normal build, assuming that with such a big system build time should not be an issue.

Alright, so sticking with DEBUG=1 then.

In principle there is 'ar' invoked after blaS, then cblas, then lapack, then lapacke. In principle rebuilding last 3 is not necessary for rapid repeater.

@timjim333 , did you get around to testing this ?

Yes, sorry for the delay. It appears to build without errors: debug_out3.txt

Great, thanks for testing. I'll prepare the corresponding PR to fix this in the develop branch later.


Define an execution model

Currently jobs are run manually (with a helper method to run all currently pending jobs). We should define an execution model.

Does it make sense to have more than one execution model (sync vs async)?

How does the API look?

my_agenda.start()

How does the agenda, itself, execute (in a thread or coordinated with an event loop)?

How does each job execute (threadpool) ?

https://github.com/frewsxcv/rust-threadpool
http://alexcrichton.com/futures-rs/futures_cpupool/

Are Jobs re-entrant?


Planned deprecation

Current status (Update: Nov 14)
Docs-Mode is now available in Storybook v5.2.0!
If you want well-integrated, SB native looks, and easy to use documentation, please try it!
(documentation for events and slots are available from v5.3.0)

Storybook Docs: Technical Preview(Guide docs)
More on #docs-mode channel on Storybook Discord

This is a remind issue.
It seems that Storybook team plans to add docs-view mode, which has almost the same features to info addon.
I think when it comes this addon is no longer necessary. It should go to maintenance mode then.

@pocka any new about docs-view mode?

@pribilinskiy Nothing new. At least they probably will not ship docs-view mode with v5.

FYI There is the #docs-mode channel in Storybook Discord.

@pocka I'm prototyping it now that v5 is shipped, and will be requesting feedback once it's in basic shape. Would you be available to give feedback?
Or even help develop it? I'm making it view layer independent, like the rest of storybook, but will need help on each view layer. As you can imagine, I'm looking at this addon to prototype Vue, and am even using your docgen-loader unmodified.

@shilman
Thank you for pinging. I'm looking forward to it!

I put the link to the Docs-mode on top of the README.
And noticed storybook depends on this package... maybe for the docgen loader?
Probably I need to separate the loader before deprecating this addon...

Deprecated.
I'll deprecate the npm package once SB6 was released.


Having all the configurations set in the Apache fashion.

The Apache configuration directives are available in v2 here:
https://github.com/SpiderLabs/ModSecurity/blob/v2/master/apache2/apache2_config.c#L3180-L3923
This functionality have to be mimic-ed into something related on version 3. In case of an error, the error message could differ a little bit but to logic should prevail. Run time errors should be treated in run time. Startup error should be handled into startup phase.
A feature that is not supported in v3 any longer, should generate a warning but it should not block the web server startup.
A special note on memory consumption. Memory consumption to load the configurations was proved to be a critical point in may use case. Therefore it should be treated with special attention.

Hi @misilot
We added some updates to documentation as of: https://github.com/SpiderLabs/ModSecurity-apache/commit/7a8c78f3d20a4848bdd7e6eb056e808a184b164e
If you have any comments or suggestions regarding documentation, please add them at https://github.com/SpiderLabs/ModSecurity-apache/issues/53 or some other related issue.
Thanks!


Update README.md

From the requirements:
README.md file that gives an overview of the project

https://github.com/nguyenha1910/opentowork/pull/138


Improve MongoDB 3.x support

This improves support for MongoDB 3.x, which currently doesn't work very well:

The output format of mongo --version has changed.
The behaviour of the localhost exemption mode has changed.
There is a new password hashing scheme based on SCRAM-SHA1.

See individual commits for details and rationale on each change.

This is very important change! please merge!

I'll look at fixing the test failures here when I'm back at work in the new year.  However, we've been running this branch in production with no problems so far.

I had to do some additional fixing to make this actually work for us, but with the fixes included in https://github.com/cybercom-finland/puppetlabs-mongodb/tree/0.17-fixes everything seems to work as it should, with authentication enabled, admin creation and a 3-node replica set.
The code should probably force the creation of the admin user for MongoDB >= 3.0 when authentication is not explicitly disabled.
The is_master facts and ruby code contain some pretty ugly regexp hacks. Is there really no better way to handle this? It also seems that every provider implements their own logic, which doesn't seem optimal either.
For example, I changed db_ismaster to only return the one field it's interested in (still wrapped in a json object and parsed so that it errors out if mongo returns garbage) and it seems to work much better. I don't know why the code tried to remove the non-parseable parts with regexps instead.

This module is essentially broken for 3.x versions as of now, this contains some good code to get started with adding proper 3.x support. Same as the user management the management of databases is also broken.
Debug: Request failed: 'Execution of '/usr/bin/mongo admin --quiet --host :27017 --eval printjson(db.getMongo().getDBs())' returned 1:
2017-01-16T13:32:20.348+0100 E QUERY    [main] Error: Empty host component parsing HostAndPort from ":27017" :

Has anyone looked at the test failures (relating to create_admin among other things)? I have not tested myself yet, but a colleague mentioned that the admin user is not created correctly even with this PR.
I'm guessing it wouldn't be merged until at least the tests can be improved and / or test failures corrected?

@wyardley most of the problems seem to be due to facts not working properly I think. Look at my  branch (link above) for a slightly fixed version that works. (caveat: at least it does for me)

@oranenj: have you made a PR of your fixes?

@wyardley: no. They're mostly just this branch + another PR plus a couple additional fixes that were enough to get my 3.4 replica set installed without errors. I don't have the time to massage them into a proper pull request unfortunately, but anyone is free to draw inspiration from them if it helps :)
But the gist of it seems to be that with 3.x you need to

assume that the admin db exists. which it should, unless your installation upgraded from 2.x and is broken, I guess.
Always enable auth, and when mongodb starts for the first time, connect via localhost to the admin db, ignore any errors, and create the admin user. This is the only operation allowed when there is no preconfigured user, and after you've added a user, you must reauthenticate with the user you created.

The above seems to work just fine even with replica sets. With the changes in this PR, There are some warnings when bringing up the first nodes of the set, but it stabilizes pretty nicely.

Any possibility of getting this merged soon?

@unixwitch are you willing to rework this after we do a Voxpupuli modulesync? This module is now in the Voxpupuli namespace.

ps - We can drop Mongodb 2.x support entirely if it makes things any easier.

#405 has what I'm thinking should be in 2.0.0 release, which would drop support for 2.x entirely.
If folks on this PR are interested in helping, please connect with me on IRC or Slack (in #voxpupuli); I think the plan is to rip out 2.x support entirely, so it may end up being easier to just rework some of this stuff and resubmit, but open to suggestions.
Unfortunately, the data types conversion will probably create some additional hassles in terms of rebasing.

@unixwitch are you willing to rework this after we do a Voxpupuli modulesync?

Sorry, I can't commit any time to this at the moment as we're very busy with some other internal work and we only use MongoDB in a couple of minor places.  I'm happy to test any proposed changes, though.
If you're removing 2.x support though (and presumably breaking backward compatibility at the same time), I think you can probably do this in a much cleaner way than I did here anyway.

@unixwitch no worries! It's been a long time since this was created. I will leave the PR open for now in case someone wants to rework it, and also just for reference.

Why is this still waiting after over a year? Is the puppet-mongodb module a dead project at this point?

@diranged, it is not dead. This PR as merge conflicts that must be resolved. You can cherry-pick the commits and rebase them if you want. Afterwards we can review and merge.

81ac1d7ff7267255f3de4653bd26c90feb0b829c took care of the version fact. I'm not entirely sure about the admin user creation and I'd like to see an updated PR with acceptance tests.


Can't step a zero degree polynomial

When running:
import control as ctrl
G = ctrl.tf([1],[1])
(y, t) = ctrl.step(G)

I get this error (I get the same in python 2.7 and 3.5):
Traceback (most recent call last):
File "./prueba.py", line 10, in
(y, t) = ctrl.step(G)
File "/usr/local/lib/python3.5/dist-packages/control/matlab.py", line 1328, in step
transpose = True, **keywords)
File "/usr/local/lib/python3.5/dist-packages/control/timeresp.py", line 483, in step_response
sys = _get_ss_simo(sys, input, output)
File "/usr/local/lib/python3.5/dist-packages/control/timeresp.py", line 409, in _get_ss_simo
sys_ss = _convertToStateSpace(sys)
File "/usr/local/lib/python3.5/dist-packages/control/statesp.py", line 656, in _convertToStateSpace
ssout = td04ad('R',sys.inputs, sys.outputs, index, den, num,tol=0.0)
File "/usr/local/lib/python3.5/dist-packages/slycot/transform.py", line 423, in td04ad
out = _wrapper.td04ad_r(m,p,index,dcoeff,ucoeff,n,tol,ldwork)
ValueError: failed to create intent(cache|hide)|optional array-- must have defined dimensions but got (1,0,)

Confirmed that this is a bug in the latest versions of the code (including several fixes).  The time response functions are not set up to handle static systems properly.


HealEntityEvent not working for players base health regeneration?

Dear Sponge-Team,
I am currently migrating my Spigot plugin to Sponge and figured out that the HealEntityEvent didn't get fired, when a player regenerates health. Is this event currently not working or isn't it triggering for the base player health regeneration?
---EDIT 1---
The event also get not triggered when a player consumes a health potion
SpongeForge Version: 1.12.2-2838-7.1.7-RC3884
SpongeApi Version: 7.1.0
Regards
Sascha

@Sago92 correct, it hasn't been implemented. An oversight on our end! I'll see what I can whip up to implement it in a rough manner.

Any time schedule for the implementation?


Add trigger for go-ovirt-client-tests suite

Adds a trigger file that will trigger execution in OST of the
go-ovirt-client-tests-suite-master suite
Please describe the change you are making
...
Are you the owner of the code you are sending in, or do you have permission of the owner?
...
The code will be published under the BSD 3 clause license. Have you read and understood this license?
...

/ost

/ost

/ost

/ost

obsoleted by #120


Loop fail with horizontal reverse

I have a question, i trying to use $PlayOrientation=5 (horizontal reverse) but with this config the $loop var is not like a carousel (option1), instead $loop look like rewind (option=2) i need a especial config? or just is not supported?
my config is
        var options2 = {
            $AutoPlay: true,
            $Loop: 1,
            $AutoPlaySteps: 1,
            $AutoPlayInterval: 0,
            $PauseOnHover: 4,
            $SlideEasing: $JssorEasing$.$EaseLinear,
            $SlideDuration: 1600,
            $MinDragOffsetToSlide: 20,
            $SlideWidth: 140,
            $SlideSpacing: 50,
            $DisplayPieces: 7,
            $ParkingPosition: 0,
            $UISearchMode: 1,
            $PlayOrientation: 5,
            $DragOrientation: 5
        };

$PlayOrientation works only with value 1 or 2.
Not sure if the value 5 stopped it from working properly.

I have just improved $AutoPlaySteps handling. Please download the latest version.
And then you can set $AutoPlaySteps: -1, $PlayOrientation: 1 and $DragOrientation: 1.


InitialRouteName doesn't load correctly

Actual behaviour
I have got the following configuration:
screen: TabNavigator({
	tab1: {
	     screen: Tab1
        },
        tab2: {
             screen: Tab2
        },
        tab3: {
             screen: Tab3
        }
},{
        initialRouteName: 'tab2',
        animationEnabled: true,
        swipeEnabled: true,
        tabBarComponent: CustomTabBar,
        tabBarPosition: 'bottom',
        tabBarOptions: {
	     showIcon: true,
             showLabel: false,
             pressColor: 'transparent'
	}
})

What is actually happening: The first screen tab1 will be shown and is immediately flickering over to the initialRouteName tab2 instead of just loading the initial route.
This is obviously related to https://github.com/react-navigation/react-navigation/issues/694.
But the commit of @satya164 doesn't seem to fix the issue for me.
Expected behaviour
I am expecting the TabView to render the initial route smoothly without the pre-rendering of the fist route defined in the navigator.

initialRouteName is just ignored in my case, I solved this by switching tab order but it's not the result I want, this is part of my main render method
Consider the initialRouteName of the TabNavigator
let tabFeed = {}

if (AUTH.user.sellerLevel > 0)
    tabFeed = {
        MyShop: {
            screen: StackNavigator({
            SellerProfile: {
                screen: SellerProfile
            },
            Upgrade: {
                screen: MembershipUpgrade
            },
            FieldEditor: {
                screen: FieldEditor
            },
            AddressFieldEditor: {
                screen: AddressFieldEditor
            },
            ExpiredWishes: {
                screen: ExpiredWishes
            },
            ActiveWishes: {
                screen: ActiveWishes
            },
            WishEditor: {
                screen: WishEditor
            },
            ProductsDashboard: {
                screen: ProductsDashboard
            },
            ProductEditor: {
                screen: ProductEditor
            }
            }),

            navigationOptions: {
            tabBarIcon: ({focused}) => getTabBarIcon('shopSettings', focused)
            }
        }
    }

tabFeed = {
    ...tabFeed,
    Home: {
        screen: StackNavigator({
            Tutorial: {
                screen: HomeTutorial
            },
            UpgradeNow: {
                screen: MembershipUpgrade
            },
            Home: {
                screen: Home
            },
            LocationEditor: {
                screen: LocationEditor
            },
            WonWishesSummary: {
                screen: WonWishesSummary
            },
            WishDetail: {
                screen: WishDetail
            },
            ProductDetail: {
                screen: ProductDetail
            },
            SellerDetail: {
                screen: SellerDetail
            }
        }, {
            navigationOptions: {
                headerTitle: <Image source={require('./assets/logo.png')} style={{height: 35, alignSelf: 'center'}} resizeMode='contain' />,
                headerStyle: {
                    backgroundColor: '#112233',
                    elevation: 0,
                },
                headerTintColor: 'white'
            },
            initialRouteName: AUTH.user.isFirstLogin && !AUTH.user.sellerLevel ? 'Tutorial' : 'Home'
        }),
        navigationOptions: {
            tabBarIcon: ({focused}) => getTabBarIcon('home', focused)
        }
    }
}

const Main = new TabNavigator(tabFeed, {
    tabBarPosition: 'bottom',
    swipeEnabled: false,
    animationEnabled: false,
    initialRouteName: AUTH.user.sellerLevel ? 'MyShop' : 'Home',
    tabBarOptions: {
        showIcon: true,
        showLabel: false,
        style: {
        backgroundColor: '#eee'
        },
        indicatorStyle: {
        backgroundColor: PALETTE.secondary
        }
    }
})

return (<Main />)

Hey, I just released a new alpha 2.0.0-alpha.0 of the library. It's rewritten using react-native-gesture-handler and react-native-reanimated addresses a many platform specific bugs and performance problems. The documentation is updated as well.
Please try the new version and see if it addresses your issue. If not, please open a new issue following the issue template.


Uncaught TypeError after switching to new builder: @angular-devkit/build-angular:browser-esbuild

Which @angular/* package(s) are the source of the bug?
Don't known / other
Is this a regression?
Yes
Description
I tried to migrate our app to use the new build system, but we get the following errors during runtime:
"Uncaught TypeError: Class extends value undefined is not a constructor or null"  please see the image below:

I just changed the builder in angular.json to this
"architect": {
  "build": {
    "builder": "@angular-devkit/build-angular:browser-esbuild"
   }
}

The application builds as expected but when running with ng serve I get a blank page and the error from above.
Please provide a link to a minimal reproduction of the bug
No response
Please provide the exception or error you saw
form-service-base.ts:11 Uncaught TypeError: Class extends value undefined is not a constructor or null

Please provide the environment you discovered this bug in (run ng version)
Angular CLI: 17.0.7
Node: 20.10.0
Package Manager: npm 9.3.0
OS: win32 x64

Angular: 17.0.7
... animations, cli, common, compiler, compiler-cli, core, forms
... language-service, localize, platform-browser
... platform-browser-dynamic, router

Package                         Version
---------------------------------------------------------
@angular-devkit/architect       0.1700.7
@angular-devkit/build-angular   17.0.7
@angular-devkit/core            17.0.7
@angular-devkit/schematics      17.0.7
@angular/cdk                    17.0.4
@angular/material               17.0.4
@schematics/angular             17.0.7
rxjs                            7.8.1
typescript                      5.2.2
zone.js                         0.14.2

Anything else?
No response

This is very likely due to an import cycle.

Can you provide a minimal reproduction demonstrating the problem?
Unfortunately, without a reproduction it will be difficult to further investigate the issue.

Thanks @JoostK @clydin for the quick response. I'm currently investigating the issue on our side and try to figure out whether there are some import cycles.
I'll let you know if this solves the issue already. Otherwise I'll try to create a small stackblitz demo.
Best regards :-)

Thanks for reporting this issue. However, you didn't provide sufficient information for us to understand and reproduce the problem. Please check out our submission guidelines to understand why we can't act on issues that are lacking important information.
If the problem persists, please file a new issue and ensure you provide all of the required information when filling out the issue template.


Reduce memory consumption in batched_forward_pass

This PR reduces memory consumption in batched_forward_pass of PPOTrainer, by avoiding the storage of logits when they are not necessary.
Before this PR, batched_forward_pass stored all of the model's logits all the time like other tensors such as values and logprobs. Here, logits tensors have a much larger size (batch_size * tokens * vocabulary_size) compared to logprobs and values tensors (batch_size × tokens), consuming a significant amount of cuda memory.
I have modified batched_forward_pass to avoid unnecessary storage of logits, which is only required when calculating entropy in the loss method.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


DO NOT MERGE: experimenting with primer

If I get it to work, I will setup a separate playground repo for remix + primer
Right now i cannot get it to work. The error I'm getting is this
npm install
npm run dev

gives me
 💿  remix dev

 info  building...
 info  built (329ms)
{
  default: {
    colors: {
      light: [Object],
      light_high_contrast: [Object],
      light_colorblind: [Object],
      light_tritanopia: [Object],
      dark: [Object],
      dark_dimmed: [Object],
      dark_high_contrast: [Object],
      dark_colorblind: [Object],
      dark_tritanopia: [Object]
    },
    spacing: { normal: [Object] },
    typography: { normal: [Object] }
  }
}
file:///Users/gregor/code/gr2m/remix-playground/node_modules/@primer/react/lib-esm/theme.js:8
} = primitives.typography.normal;
                          ^

TypeError: Cannot read properties of undefined (reading 'normal')
    at file:///Users/gregor/code/gr2m/remix-playground/node_modules/@primer/react/lib-esm/theme.js:8:27
    at ModuleJob.run (node:internal/modules/esm/module_job:194:25)

Node.js v18.18.2

I'm trying to avoid TypeScript but if that extra build step is the only way to resolve the problem I'll oblige 🤷🏼

We can try the 0.0.0-20231009163439 pre-release from this PR:
https://github.com/primer/react/pull/3787

That didn't help. I tried a few things with @colebemis's help but we couldn't get it to work unfortunately. I guess this has to wait until Primer switches over to ESM

this might be a hack to make it work: https://www.npmjs.com/package/remix-esbuild-override. See also https://mortenbarklund.com/blog/styled-components-remix/


Bug with TransformerDecoderLayer

Version of tools.
torch.__version__ == '1.12.1'

I tried to construct an arbitrarily simple standard decoder, I didn't include an embedding layer to remove complexity so I just generated data of the expected shape in the forward pass. (I included a linear at the end mostly because I forgot to remove it - my original model had it.)
class Transformer(torch.nn.Module):
    def __init__(self, num_embeddings: int, d_model: int):
        super().__init__()
        self.d_model = d_model
        self.num_embeddings = num_embeddings

        self.decoder_layer = torch.nn.TransformerDecoderLayer(self.d_model, nhead=1,
                                        dropout=0.1, dim_feedforward=64)

        self.linear = torch.nn.Linear(
            self.d_model, self.num_embeddings, bias=False)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.decoder_layer.forward(x,x) # sending x to both is wrong, just for simplicity
        return self.linear(x)

And called the model as follows:
batch_size = 100
seq_len = 3
embedding_dim = 128
num_embeddings = 97

sample_input = torch.rand((batch_size, seq_len, embedding_dim)).float()
sample_output = torch.rand((batch_size, seq_len)).long()
model = Transformer(num_embeddings, 128)
patch_model(model)
optim = TGAdamW(model.parameters())

out = model(sample_input).transpose(-2, -1)
loss = F.cross_entropy(out, sample_output)
loss.backward()
optim.step()

This resulted in the following error:
File "<local file path>/TrueGrad/truegrad/optim.py", line 30, in step
    raise ValueError(f"Parameter of type {type(p)}, shape {list(p.size())} doesn't have `square_grad` attribute. "
ValueError: Parameter of shape [384, 128] doesn't have `square_grad` attribute. Make sure to use truegrad.utils.patch_model() or truegrad.nn for all optimized parameters.

We can confirm that this parameter is a truegrad parameter by modifying the ValueError to emit the type of the object getting the following error message.
raise ValueError(f"Parameter of type {type(p)}, shape {list(p.size())} doesn't have `square_grad` attribute. "
ValueError: Parameter of type <class 'truegrad.nn.TrueGradParameter'>, shape [384, 128] doesn't have `square_grad` attribute. Make sure to use truegrad.utils.patch_model() or truegrad.nn for all optimized parameters.

Possibly explanations?
Playing around a little with the following function
def check(model: torch.nn.Module):
    good_params = []
    print("BROKEN PARAMS:\n")
    for name, param in model.named_parameters():
        if not hasattr(param, "square_grad") and hasattr(param, "grad")  :
            print(f"\t {name} {param.shape}")
        else:
            good_params.append((name, param))
    print(f"\n\nGOOD PARAMS: \n")
    for name, param in good_params:
        print(f"\t{name} {param.shape}")

We can run this after loss.backward() but before optim.step() and we get the following results:
BROKEN PARAMS:

	 decoder_layer.self_attn.in_proj_weight torch.Size([384, 128])
	 decoder_layer.self_attn.in_proj_bias torch.Size([384])
	 decoder_layer.self_attn.out_proj.weight torch.Size([128, 128])
	 decoder_layer.self_attn.out_proj.bias torch.Size([128])
	 decoder_layer.multihead_attn.in_proj_weight torch.Size([384, 128])
	 decoder_layer.multihead_attn.in_proj_bias torch.Size([384])
	 decoder_layer.multihead_attn.out_proj.weight torch.Size([128, 128])
	 decoder_layer.multihead_attn.out_proj.bias torch.Size([128])

GOOD PARAMS:

	decoder_layer.linear1.weight torch.Size([64, 128])
	decoder_layer.linear1.bias torch.Size([64])
	decoder_layer.linear2.weight torch.Size([128, 64])
	decoder_layer.linear2.bias torch.Size([128])
	decoder_layer.norm1.weight torch.Size([128])
	decoder_layer.norm1.bias torch.Size([128])
	decoder_layer.norm2.weight torch.Size([128])
	decoder_layer.norm2.bias torch.Size([128])
	decoder_layer.norm3.weight torch.Size([128])
	decoder_layer.norm3.bias torch.Size([128])
	linear.weight torch.Size([97, 128])

Both multihead_attn and self_attn are instances of MultiheadAttention, its also not really clear to me why attributes of both objects. i.e. "k_proj_weight" appear in neither but they are Nones. I'm probably going to spend more time looking into this but I hope this helps you debug as well. This is all the information I have at the moment.

Thank you for the detailed bug report!
It seems the issue stems from multi_head_attention_forward being a fused function that returns a tuple of outputs. As tuples of tensors aren't recognized as tensors by TrueGradParameter, it doesn't modify the backward pass to add the square_grad.
While it'd be possible to solve the tuple issue, I'm afraid that would only uncover more problems.
To compute the square_grad attribute, truegrad does a second (inner) backward pass using torch.autograd.backward(f(x^2), dy^2). While that works well for simple operations that directly access the parameter, like linear or convolution, it works horribly for fused layers. For example, LayerNorm(input^2, scale, shift), or in this case, attention(x^2), gives very different results from ( (x-x.mean())/x.std() )^2 * scale + shift (which is what we need).
The best way I see of tackling this issue is to

extend the readme with an explicit warning about fused functions
fix the tuple issue
port more of torch.nn to truegrad.nn

What do you think?

I 100% agree with the steps. Technically, you can "fix" the tuple issue by having the forward pass for wrap emit x not torch.zeros_like(x) but I don't think this solution will do backprop correctly and failing silently seems like a worse solution. I do think porting as much of torch.nn to truegrad.nn seems like the best way forward. If this just involves looking at the torch C++ implementation and copying the backprop code and squaring when necessary I can start doing this as well.
It might be worth while to also have patch_model do a check that looks for Modules that do fused functions and fail loudly if any are encountered.

If you can port all torch.nn by the weekend thats insane, uh but yes I can look into how to detect fused functions. I'll try to think about it in a programmatic/mathematical way and see if I can think of/find anything clever. The fallback would be sort of a blacklist of fused functions in torch.nn.functional that do fused ops and then hooking into a python LSP to inspect the forward methods for use of blacklist operations.
If you port all of torch.nn though it seems like patch_model would be pointless? You could just swap torch.nn for truegrad.nn, do you really plan to implement the entirety of torch.nn?

We'll still need something like patch_model for off-the-shelf models like torchvision and huggingface transformers. However, it seems like patching the parameters is incredibly difficult. For example, TrueGradParameter(x).reshape(-1) * input would link reshape as torch function instead of the multiplication. It seems like we have to find the last operations with a data axis like batch, sequence, and width.
Additionally, inspecting the python code won't work for a general case, as many projects (such as xformers) depend on fused cuda kernels.
The most straightforward way forward would be to provide a ported version of torch.nn, write a patch_torch_nn function, and add a fall_back_to_adam flag in TGAdamW so mixed models can exist.
However, if we have to patch and reimplement all of torch.nn anyway, we can fall back to backpack. Backpack contains reimplemented torch modules and has functions to compute the square_grad attribute.

Added backpack support in https://github.com/ClashLuke/TrueGrad/commit/0a14e3877db30b25f89b6506bbd365386f7a70e4 / 0.1.0. We'll probably need a more extensive truegrad.nn.functional rather than a larger truegrad.nn, so I'll work on that tomorrow.

Truegrad 2.0.0 gives the following output using the truegrad.utils.patch_torch backend:
BROKEN PARAMS:

GOOD PARAMS:

	decoder_layer.self_attn.in_proj_weight torch.Size([384, 128])
	decoder_layer.self_attn.in_proj_bias torch.Size([384])
	decoder_layer.self_attn.out_proj.weight torch.Size([128, 128])
	decoder_layer.self_attn.out_proj.bias torch.Size([128])
	decoder_layer.multihead_attn.in_proj_weight torch.Size([384, 128])
	decoder_layer.multihead_attn.in_proj_bias torch.Size([384])
	decoder_layer.multihead_attn.out_proj.weight torch.Size([128, 128])
	decoder_layer.multihead_attn.out_proj.bias torch.Size([128])
	decoder_layer.linear1.weight torch.Size([64, 128])
	decoder_layer.linear1.bias torch.Size([64])
	decoder_layer.linear2.weight torch.Size([128, 64])
	decoder_layer.linear2.bias torch.Size([128])
	decoder_layer.norm1.weight torch.Size([128])
	decoder_layer.norm1.bias torch.Size([128])
	decoder_layer.norm2.weight torch.Size([128])
	decoder_layer.norm2.bias torch.Size([128])
	decoder_layer.norm3.weight torch.Size([128])
	decoder_layer.norm3.bias torch.Size([128])
	linear.weight torch.Size([97, 128])

Complete code:
import torch
from torch.nn import functional as F

from truegrad.optim import TGAdamW
from truegrad.utils import patch_torch

patch_torch()

class Transformer(torch.nn.Module):
    def __init__(self, num_embeddings: int, d_model: int):
        super().__init__()
        self.d_model = d_model
        self.num_embeddings = num_embeddings

        self.decoder_layer = torch.nn.TransformerDecoderLayer(self.d_model, nhead=1,
                                                              dropout=0.1, dim_feedforward=64)

        self.linear = torch.nn.Linear(
                self.d_model, self.num_embeddings, bias=False)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.decoder_layer.forward(x, x)  # sending x to both is wrong, just for simplicity
        return self.linear(x)

batch_size = 100
seq_len = 3
embedding_dim = 128
num_embeddings = 97

sample_input = torch.rand((batch_size, seq_len, embedding_dim)).float()
sample_output = torch.rand((batch_size, seq_len)).long()
model = Transformer(num_embeddings, 128)
optim = TGAdamW(model.parameters())

out = model(sample_input).transpose(-2, -1)
loss = F.cross_entropy(out, sample_output)
loss.backward()

good_params = []
print("BROKEN PARAMS:\n")
for name, param in model.named_parameters():
    if not hasattr(param, "sum_grad_squared") and hasattr(param, "grad"):
        print(f"\t {name} {param.shape}")
    else:
        good_params.append((name, param))
print(f"\n\nGOOD PARAMS: \n")
for name, param in good_params:
    print(f"\t{name} {param.shape}")

Wow, sorry I haven't looked at anything yet, busy with family stuff, this is super impressive. Thank you so much for solving this so fast though, wow.

All good. Take care, and let me know if there are any other issues.


Martketplace and Sendgrid

Hello Team,
After Sendgrid was moved to Microsoft.SaaS, Sendgrid resource can move between resource groups and subscriptions.
However, due to limitation, Sendgrid resource cannot be moved in or out from CSP subscription.
It's hard to know this information from this document.
Also, I'm not sure whether other Marketplace SaaS resources has the same limitation.
Would it be possible to have a list of Marketplace services and show what action they can do (support or not support to move) and what special scenarios we will encounter (such as CSP offer has some limitations)?
Thank you!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 23df5354-6833-a5da-0eb3-d860150c8b78
Version Independent ID: 682ec206-0fda-3e0b-5841-543860a87e34
Content: Move operation support by resource type - Azure Resource Manager
Content Source: articles/azure-resource-manager/management/move-support-resources.md
Service: azure-resource-manager
Sub-service: management
GitHub Login: @tfitzmac
Microsoft Alias: tomfitz

Thank you for your feedback! We have assigned this issue to the author to review further and take the next course of action.

@LingYi-Wu - thanks for this feedback. I have added clarification for Saas resources. It should get published tomorrow.
#please-close


the source-map-support is an old version, causes [DEP0005] DeprecationWarning: Buffer()

hey,
please update the "source-map-support": "~0.4.0"  package.
it causes: "[DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead."
when used with typescript and its sourcemap Feature.

node -v v11.9.0
npm -v 6.5.0
protractor --version Version 5.4.2

We are not updating 5.4.2 and it looks like we should update it on selenium4.
https://github.com/angular/protractor/blob/selenium4/package.json#L23

Okay closing this issue. I did an audit on the release for Protractor 6. This should be fixed.


client always uploads the layer image even if the layer image (blob with the SHA) exists

Noticed that when push a new repository to the registry, even if a referenced layer image exists, client always upload. Only in the final commit, the registry will check if the SHA layer image exists or not; if yes, it skips copying the upload over to final blob location.
In such case, client only needs to add the link file to the real layer image, but does not need to upload the image itself at all. Is it possible to do so?

This is possible using a cross-repository blob mount: https://github.com/docker/distribution/blob/master/docs/spec/api.md#cross-repository-blob-mount
Docker has supported this since 1.10, so generally recent versions of Docker (when paired with a recent enough version of the registry) won't have to upload layers that already exist on the registry.

@aaronlehmann Thank you! Will take a look at the cross-repository mount.


Samsung Tablet-Firefox: Loading

server: https://spartacus.c39j2-walkersde1-d4-public.model-t.cc.commerce.ondemand.com/
Observed Results
Loading takes like (~20 seconds) when

you try to access the web page
when you refresh

First time loading some components, you can see it loading slowly when

you add to cart and the modal open, the image is loaded very slowly. You can see text instead of the image. But after 2-5 seconds, the image is shown.
Similar to category page with all the images
Similar to the search suggestions

Expected Results

We should not see the text placeholder for the images for 5 seconds before it shows the image
We should be able to access the web page under 5 seconds or so.

would like to know why this is happening on these devices, on my ipad and iphone, loading is reasonable.

Dupe with #3492


RBD: Failed to mount cloned PVC when parent PVC and snapshot is deleted

With kernel v3.10.x failed to mount the PVC when the parent PVC and snapshot is deleted
  Normal   Scheduled               10s              default-scheduler        Successfully assigned default/csirbd-demo-pod to minikube
  Normal   SuccessfulAttachVolume  10s              attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-ebfbcb08-41ee-4042-bcb9-24b4d7b096da"
  Warning  FailedMount             3s (x4 over 7s)  kubelet                  MountVolume.MountDevice failed for volume "pvc-ebfbcb08-41ee-4042-bcb9-24b4d7b096da" : rpc error: code = Internal desc = image not found: RBD image not found

steps to reproduce the issue

Create PVC
Create Snapshot from PVC
Delete PVC
Create Clone from snapshot
Delete snapshot
Mount PVC clone to application pod
Delete PVC clone and application pod

@Madhu-1 if the kernel does not have deep-flatten support, Isnt  'mount' supposed to fail ? or we have some other bug which allowed other operations to proceed like snapshot creation , parent PVC deletion, restore but ONLY fails at "mount" of restored PVC which is not correct?

@Madhu-1 if the kernel does not have deep-flatten support, Isnt 'mount' supposed to fail ?

the mount should not fail if the mount fails it's problematic for user applications. cephcsi should handle this problem by flattening the image before mounting.

or we have some other bug which allowed other operations to proceed like snapshot creation , parent PVC deletion, restore but ONLY fails at "mount" of restored PVC which is not correct?

can you please elaborate on this one?

steps to reproduce the issue

    Create PVC
    Create Snapshot from PVC
    Delete PVC
    Create Clone from snapshot
    Delete snapshot
    Mount PVC clone to application pod
    Delete PVC clone and application pod

@Madhu-1 even if we do first 4 steps, ie Create PVC-> Snapshot-> Delete PVC -> Create Clone-> Mount should also fail, Isnt it ?

Yes, there are multiple ways to reproduce it. the main problem is that snapshot or parent pvc should be deleted


Couchbase Image Spesific Version Support

When pulling Couchbase images, the latest tag is always used. We should also be able to pull the specific requested tag.

That would be a nice addition!

Implemented with #297


Use of GraphQL API, show commits difference and additions/deletions difference

Users have to paste their GitHub token in the dedicated field. Token is then stored in localStorage for later usage


JavaSDK is wrong on capabilities

Agent version and platform
Version of your agent? Hosted
Current agent version: '2.112.0'

OS of the machine running the agent? Windows

VSTS type and version
VisualStudio.com
http://universalplatform.visualstudio.com

What's not working?
I need to specify the JDK directory for my build, i look at the JDK on capabilities and it shows
C:\Program Files\Java\jdk1.8.0_40
But that's not the actually path for the jdk 1.8 installed on the machine..
If i run a dir of Java folder i get:
2017-03-08T19:09:53.0366026Z Task         : Command Line
2017-03-08T19:09:53.0366026Z Description  : Run a command line with arguments
2017-03-08T19:09:53.0366026Z Version      : 1.1.2
2017-03-08T19:09:53.0366026Z Author       : Microsoft Corporation
2017-03-08T19:09:53.0376023Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkID=613735)
2017-03-08T19:09:53.0376023Z ==============================================================================
2017-03-08T19:09:53.0406023Z ##[command]dir "C:\Program Files\Java\"
2017-03-08T19:09:53.0536056Z  Volume in drive C has no label.
2017-03-08T19:09:53.0536056Z  Volume Serial Number is 78C2-210A
2017-03-08T19:09:53.0536056Z
2017-03-08T19:09:53.0536056Z  Directory of C:\Program Files\Java
2017-03-08T19:09:53.0536056Z
2017-03-08T19:09:53.0606019Z 12/02/2016  02:25 AM    <DIR>          .
2017-03-08T19:09:53.0606019Z 12/02/2016  02:25 AM    <DIR>          ..
2017-03-08T19:09:53.0606019Z 12/02/2016  02:25 AM    <DIR>          jdk1.6.0_45
2017-03-08T19:09:53.0606019Z 12/02/2016  02:24 AM    <DIR>          jdk1.7.0_75
2017-03-08T19:09:53.0606019Z 12/02/2016  02:23 AM    <DIR>          jdk1.8.0_102
2017-03-08T19:09:53.0606019Z 12/02/2016  02:23 AM    <DIR>          jre1.8.0_102
2017-03-08T19:09:53.0606019Z 12/02/2016  02:25 AM    <DIR>          jre6
2017-03-08T19:09:53.0606019Z 12/02/2016  02:24 AM    <DIR>          jre7
```<br>

@rmarinho
base on https://github.com/Microsoft/vsts-agent/blob/master/src/Misc/layoutbin/powershell/Add-JavaCapabilities.ps1
Add-CapabilityFromRegistry -Name 'jdk_8_x64' -Hive 'LocalMachine' -View 'Registry64' -KeyName $jdk8KeyName -ValueName 'JavaHome' -Value ([ref]$latestJdk)

We are reading from your registry
LocalMachine\Software\JavaSoft\Java Development Kit\1.8\JavaHome
Can you check it?

how do i check the registry on a remote machine? remember this is the hosted agent

@rmarinho sorry, didn't realize it's hosted.
@thechrisjohnson how do we install Java SDK on Hosted image?

Also the "java" capability is missing in Hosted 2017 agents, it was there in the old Hosted agents. When I try to use the SonarQube tasks from the Marketplace they are failing on Hosted 2017 agents because Java is missing. Don't know if fixing the JDK path is sufficient. Perhaps also the "java" capability needs to be added.
Can somebody confirm that this is a bug and needs to be fixed? We are currently blocked a bit because we cannot run our static code analysis in CI. Don't want to setup private agents for that now.

@madhurig @thechrisjohnson
Is the Java installed on VS2017 Hosted Image? do we have plan to fix it?

pinging offline for an update

@SnoopyCoder you can add a user capability to enable routing to the agent

@SnoopyCoder you can add a user capability to enable routing to the agent

@rmarinho as a workaround you can use powershell to check the registry values on the hosted agent

The hosted vs 2017 was updated and should have rolled out everywhere now.
I'm somewhat lost on the current state of the issue. Is this resolved? Or is there any remaining problem (e.g. incorrect capabilities)?

AFAIK the "java" capability is still missing. When trying yo use the SonarQube extension on Hosted VS2017 it fails with:

No agent found in pool Hosted VS2017 which satisfies the specified demands:
msbuild
java
DotNetFramework
visualstudio
vstest
Agent.Version -gtVersion 1.102

@thechrisjohnson @madhurig

@henryju thanks for the update (as a workaround you can add as a user capability)

@ericsciple thanks for the workaround, that work fine. We'll document that to our users, but it would be great to have the capability added out of the box.

Agreed. @thechrisjohnson is working on updating the capabilities on the VS2017 agent. Also I believe we have some work in progress that will keep the capabilities up to date and accurate in the future (right now it's a static list).

A fix for this was pushed on Friday, we're just waiting on our deployment to go out to all Scale Units.


Add pointer to ScalaDoc.

Should we move this to github? It was suggested the firrtl equivalent should live on github.

If we can do it as a github page or whatever I don't see why not. I think there is also a tool that generates Markdown from Scaladoc but I think it's less powerful than the html version

Check out sbt-ghpages. I believe we already load the required plugins.


Fix the multiarch image building

Related issue: https://github.com/docker/buildx/issues/493
Fix for the GitHub actions  is very simple - we should add the line
- uses: docker/setup-qemu-action@v1

before the docker/setup-buildx-action action execution

@jetexe please, squash the commits and re-create the latest release after this PR merging

:warning: IMPORTANT
Please, modify the line
https://github.com/jetexe/cookiecutter-docker/blob/1b440662de29b16776f05be1977fc985cbbd393a/.github/workflows/release.yml#L32
To
          platforms: linux/amd64,linux/386,linux/arm64,linux/arm/v6,linux/arm/v7

By yourself. The fork has already been deleted on my side, and I cannot make the required changes by myself :(

ping @jetexe


pex fails to run on windows: "No module named 'fcntl'"

PS C:\Users\mgrandi> py -3 -m venv .\Temp\testvenv
PS C:\Users\mgrandi> .\Temp\testvenv\Scripts\Activate.ps1
(testvenv) PS C:\Users\mgrandi> pip3 install pex
Collecting pex
  Using cached pex-2.1.24-py2.py3-none-any.whl (2.5 MB)
Installing collected packages: pex
Successfully installed pex-2.1.24
WARNING: You are using pip version 20.2.3; however, version 20.3.3 is available.
You should consider upgrading via the 'c:\users\mgrandi\temp\testvenv\scripts\python.exe -m pip install --upgrade pip' command.
(testvenv) PS C:\Users\mgrandi> pex --help
Traceback (most recent call last):
  File "C:\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\mgrandi\Temp\testvenv\Scripts\pex.exe\__main__.py", line 4, in <module>
  File "c:\users\mgrandi\temp\testvenv\lib\site-packages\pex\bin\pex.py", line 19, in <module>
    from pex.common import die, safe_delete, safe_mkdtemp
  File "c:\users\mgrandi\temp\testvenv\lib\site-packages\pex\common.py", line 9, in <module>
    import fcntl
ModuleNotFoundError: No module named 'fcntl'
(testvenv) PS C:\Users\mgrandi> python --version
Python 3.9.0
(testvenv) PS C:\Users\mgrandi> pip3 list
Package    Version
---------- -------
pex        2.1.24
pip        20.2.3
setuptools 49.2.1
WARNING: You are using pip version 20.2.3; however, version 20.3.3 is available.
You should consider upgrading via the 'c:\users\mgrandi\temp\testvenv\scripts\python.exe -m pip install --upgrade pip' command.
(testvenv) PS C:\Users\mgrandi> $PSVersionTable.PSVersion

Major  Minor  Build  Revision
-----  -----  -----  --------
5      1      19041  610

(testvenv) PS C:\Users\mgrandi> [environment]::OSVersion.Version

Major  Minor  Build  Revision
-----  -----  -----  --------
10     0      19042  0

@mgrandi unfortunately, this is expected:
https://github.com/pantsbuild/pex/blob/46adfb48ba53c8663dc34644dc8e04686f7b83a3/pyproject.toml#L13-L17
We currently have no contributors that use Windows which is likely the boost we need to get Windows CI set up to give us a green baseline to develop against and ensure we don't start providing Windows support while continually degrading. If you're willing to step in and champion a Windows support effort, that would be lovely. Besides CI setup, at a minimum work will be needed where you hit this error - which is file locking code and likely various bits of code that assume creating symlinks will work (IIUC they require special privileges on Windows).

I have the same issue as OP. I had built my toolchain around pex==1.6.12, which worked on Windows, but then I ran into this issue when I tried updating to the latest version. My proposal would be to remove the import of fcntl and instead use portalocker, which is a cross-platform equivalent for locking (which from a quick search, looks like all that fcntl is used for in this package). I am happy to take a crack at implementing that in a new PR, but wanted to run it by you to make sure that it would be ok to introduce a new dependency.

Thanks for the anecdote @chrisjbremner and the fcntl alternative suggestion. Note though that Pex can never support Windows in good faith until it grows Windows CI jobs to ensure things stay green. If either of you wants to pitch in and get this all set up then we can do the 1st official release that adds Operating System :: Microsoft :: Windows ... classifiers.

Sorry @chrisjbremner missed your last sentences... Something like portalocker would be fine as long as its small, focused and well tested. It would be added and accessed via the vendoring system. So that means:

Add a line here:
https://github.com/pantsbuild/pex/blob/46adfb48ba53c8663dc34644dc8e04686f7b83a3/pex/vendor/__init__.py#L102-L129
Run tox -evendor
Use the code via from pex.third_party.portalocker import ...

The vendoring does throw a wrench in things a little bit as portalocker has a dependency of pywin32 (on Windows) -- if this were included as a vendored package, would that mean that all of the pywin32 source would be included in the generated .pex file, even in those built on Linux/Mac?

Yeah, that's not awesome. Maybe https://github.com/harlowja/fasteners ? That also has the pywin32 dependency, but opts to vendor in the small slices it needs.

I'll try out a few as suggested here and report back

Alternatively - and I think preferrably - just add the windows-specific code. Pex only uses an exlusive write lock and that impl in fasteners does not require pywin32:
https://github.com/harlowja/fasteners/blob/a39ce891ca9c833b554cd806f2395ec43f017349/fasteners/process_lock.py#L413-L436

Do you know how long files are locked for? Would using LK_LOCK (which blocks for 10s) work? Otherwise, I imagine we could make it nonblocking and add a loop that checks the lock periodically (which is what filelock does), which would presumably more closely match the behavior of LOCK_EX from fcntl

The lock is forever - its a blocking write lock:
https://github.com/pantsbuild/pex/blob/46adfb48ba53c8663dc34644dc8e04686f7b83a3/pex/common.py#L404-L413
So some sort of loop over the lock attempt that allows for OSError will be needed it seems. It would be great if that OSError coud be narrowed down though via an errno - that's a disturbingly exception catch on the surface.

It seems to manifest itself as a PermissionError when tested on my machine, which is slightly more specific than an OSError, although I cannot be confident that it'd always raise that. Anyways, I made a PR with the change here, I suppose we should continue discussion there?


correct dependency

From what I understand the behaviour of peerDependencies will change from npm@3, not sure if this is what we want, but then again I could be wrong.


Add a type button to the month navigation elements

This will prevent HTML forms to assume the navigation buttons as being the ones who submit the form in which they are used. :)

Great working again, thank you so much! I will get both merged in a few hours. 😃

Done — published 0.6.0! Thanks once again! 😃

Awesome! Thanks to you!


Wrongfully showing "Something went wrong" when the problem is that no tests were ran

Describe the bug
The Test Results and Test Runs tabs on the report show "Something went wrong." whenever a user installed Elementary for the first time without running any tests. In the past, the CLI would continue executing whenever an error occurred and therefore we displayed this error, however, current the CLI itself would raise an error if there's a problem and this page is no longer accurate.
Expected behavior
The page should say "No tests were ran." instead.

Internal issue, will be handled in linear. Closing this public issue


Enable MultilineBlockLayout cop

For discussion.
Followup to https://github.com/capistrano/capistrano/pull/1594

This seems reasonable. :+1:


Requests not evenly balanced across ingessgateway

Requests are not being properly load balanced across ingressgateway pods
istio-ingressgateway-56d9549884-rkdw8 -c istio-proxy -- curl localhost:15000/stats | grep conn
server.total_connections: 57

istio-ingressgateway-56d9549884-xqlcr -c istio-proxy -- curl localhost:15000/stats | grep conn
server.total_connections: 16

Leading to mismatched CPU usage
Namespace     Name                        CPU    CPU Req    CPU Lim    Mem     Mem Req    Mem Lim
------------  --------------------------  -----  ---------  ---------  ------  ---------  ---------
istio-system  istio-ingressgateway-rkdw8  207m   4000m      6000m      88Mi    512Mi      512Mi
istio-system  istio-ingressgateway-xqlcr  88m    4000m      6000m      81Mi    512Mi      512M

Scaling up new ingress pods does not make it any better...
Running the perf load tests
On GKE, Istio 1.2
I think this should be handled in the GCP LoadBalancer, so might not be istio related

My pilot was broken due to a bug fix that hadn't landed in the version I was using yet. Once I fixed that all was working


Simplify most of flatten

Motivation

Test plan

See included automated tests.

Most of this PR is removing special-casing that shouldn't have existed in the first place

These changes landed in #2642


[Communication] - Chat - Replaced ChatUserCredentialPolicy with BearerTokenAuthenticationPolicy

TODO:

[ ] CommunicationTokenCredential tests

Everything looks good but I would recommend removing ChatUserCredentialPolicy I don't think there are any more instances of it being used


feat(es-dev-server): add module dedupe option

This adds a dedupe option to go along with https://github.com/open-wc/open-wc/pull/1000. The implementation is actually identical to the rollup implementation, because we're both using the same resolve package underneath. There are some limitations I'd like to see different, but I'd rather keep both options in sync and try to fix them on rollup's side too.
TODO:

[x] Add tests
[x] Update docs
[x] Figure out how to handle duplicates not in root
[x] Figure out relative imports

This is ready for review now

Will take a look today

LGTM, seems to run as intended locally as well 👍


Consider adding WAF to this reference architecture

The premise of this reference architecture is to be actionable guidance on secure connectivity, but Traffic Manager is resolving directly to App Service instances rather than using an Azure App Gateway in WAF mode or replacing Traffic Manager with Azure Front Door (WAF enabled). To better reflect how this scenario should be realized in the real world, please consider extending to include a WAF.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d0461064-5b0f-2722-c228-cc04aef150c9
Version Independent ID: 705b8a1a-787e-c09f-dff3-c25373743d1d
Content: Multi-region web app with private connectivity to database - Azure Example Scenarios
Content Source: docs/example-scenario/sql-failover/app-service-private-sql-multi-region.yml
Service: architecture-center
Sub-service: example-scenario
GitHub Login: @jelledruyts
Microsoft Alias: edprice

Thanks for reporting - this issue is under review. This is a Microsoft Internal DevOps Bug ID # 395507

This was closed with the following PR: https://github.com/MicrosoftDocs/architecture-center-pr/pull/8779.


Portainerstack was created outside of Portainer!

How can I deploy the portainer Stack inside portainer?
I start portainer from the CLI with
sudo docker stack deploy -c ./portainer/docker-compose.yml portainer
When I open portainer and go to "stack" the control of the stack "portainer" is "Limited"
How can i control the stack from portainer?
Thanks rob

You cant.
A limited stack is one created outside portainer, and portainer must always be started outside of Portainer.
Rgds,
Neil Cresswell
On 2/01/2020, at 5:51 AM, rootmessages BPMspace<EMAIL_ADDRESS>wrote:

How can I deploy the portainer Stack inside portainer?
I start portainer from the CLI with
sudo docker stack deploy -c ./portainer/docker-compose.yml portainer
When I open portainer and go to "stack" the control of the stack "portainer" is "Limited"
How can i control the stack from portainer?
Thanks rob
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_portainer_portainer_issues_3486-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DAFQ2XFM7UUVV5OEGQKEXVR3Q3TCXPA5CNFSM4KB34DUKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IDSRCEA&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=V3gjBjEewlvX3VHYLaMye_5dAeeTvTB_cLWfawjP1yc&s=wKtWPXkU1UW3HGktrygjcjwvTouH45gILPzMhUffwe4&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AFQ2XFPPWJNNRBWP2YMAIS3Q3TCXPANCNFSM4KB34DUA&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=V3gjBjEewlvX3VHYLaMye_5dAeeTvTB_cLWfawjP1yc&s=xXpVPUS5HFGmofmxmORSUO06WhWgmMp1vHVo_QXtQgI&e=.

Hello, Neil,
I could! I have done the following:
Do you think there are any disadvantages here? Thanks for the help

Rob

Oh! The many and varied ways to completely mess up your Portainer install.
Given that a redeployment of Portainer that fails will break Portainer unless your stack is completely bulletproof, you would bootstrap:

Do a cli container based deployment of Portainer, using a temporary port mapping. Make sure the map the data volume to something well known.

Use that to deploy a Portainer stack, using the final port mapping / traefik binding / whatever to expose it. Point the data volume to the external volume you defined above.

Stop the Container.

--
NB. In step 2 I am assuming that it is safe for 2 instances of Portainer to access the same DB volume at the same time. If this is not the case then simply let the container and stack instance have separate DBs. Given that you could accidentally do something bad to Portainer via the GUI which would render Portainer unusable, having the container Portainer around to re-invoke from the command line is probably a good idea as a fallback.

Thanks - i will give it a try ...

I would like to point out this is not the recommended approach though, given that you could easily mess up the Portainer stack and then not have access to Portainer anymore to revert changes.
I am going to close this now as it has been answered.

I am sorry for the possibility very dumb question, but is there a way to manage Portainer stack from within Portainer? I am especially referring to upgradability and the automagic hooks on git-repository + docker-compose.yaml for Portainer.
I do understand, that one has to be cautious, of course.


CS title update

👷 Deploy Preview for rayko-research processing.

Name
Link

🔨 Latest commit
12ccbed315d980b513061bb614781f3f62632e2f

🔍 Latest deploy log
https://app.netlify.com/sites/rayko-research/deploys/66696878b68ad30008734b52


User Story #2 - semantic HTML elements

GIVEN a webpage meets accessibility standards
WHEN I view the source code
THEN I find semantic HTML elements

 replaced with  where applicable.


[Question]: Need help on dapr retries.

I want to draw your attention on the existing github issue: 7698
I have seen your dapr side-car service to service invocation video on youtube. Really great explanation and great flow of the lecture👍.
I want to know the content of component folders, and dapr-config,yaml files.
Or basically, I want to know that I have written the resiliency.yaml file as shown below:
apiVersion: dapr.io/v1alpha1
kind: Resiliency
metadata:
  name: myresiliency
scopes:
  - order-service
  - order-validation-service
spec:
  policies:
    timeouts:
    - general: 8s

    retries:
    - important:
        policy: exponential
        duration: 5s
        maxRetries: 10

    circuitBreakers:
    - simpleCB:
        maxRequests: 1
        interval: 60s
        timeout: 8s
        trip: consecutiveFailures > 3

  targets:
    apps:
    - order-service:
        timeout: general
        retry: important
        circuitBreaker: simpleCB
    - order-validation-service:
        timeout: general
        retry: important
        circuitBreaker: simpleCB

Where the order-service(written in dotnet) will communicate with the order-validation-service(node API, same as your item-service). I want to apply the retry policy to the dotnet api, which in turn at the end communicate with node api, if the node API is down then dotnet API should retry. And Want to show the logs for the same as you have shown in your youtube lecture.
I doubt, I am missing dapr-config.yaml file or I might be wrong with side car invocation. Is it compulsory call order-service with the dapr port, or we can simply call it with the normal port of dotnet api which is assigned by dotnet in launchsettings. Just to inform: I have made the order-validation-service(Node API) down to test the retry policy.

Greetings @vishipayyallore, Can you please look into the issue I've mentioned above? I am doing the same thing as shown in lecture but couldn't get retry logs. If I've set retry to 5 times after interval of 3 seconds, Then It should print retry logs after each 3 seconds for atleast 2 or 3 times. But just got one retry log and that was also "retrying in 1 sec". And that was not expected. Should be "retrying in 3 sec".
Many thanks.


Loading loop for tables in database.

When click on database, the tables list load is in an endless loop.
(Using MariaDB over SSH tunnel)

Do you have any idea how many schemas and tables there are in that database? If there is a huge amount of schemas could you try to limit the loading by filtering schemas with include/ignore filter:

It could an exception occurring during the database loading as well. In that case, you would need to open the DevTools window on View/Toggle DevTools to see the error properly.


Add support for security setups to logbook-servlet

In our common setup we would like to have a filter chain setup like this:

logging, 1st pass
security
logging, 2nd pass

The first logging filter should log both, request and response, if and only if the response is a 401 (or 403?) and it should ignore the request body. This is desired because unauthorized requests can potentially be DoS attacks with very large requests to take our service down.
The second logging filter logs requests as they come in and responses as they go out - business as usual. The only important thing is, that any request/response is only logged exactly once.

consuming the request / response should be configurable per filter instance (i.e. body is optional)
optional communication channel between filter instances to have "exactly once" semantic when logging request or response

Just for clarification: The proposed setup will lead to log files that are not 100% correct in terms of chronological order. Any request to an external service inside the security filter will be logged before the original request (e.g. to verify OAuth token):
OAuth request
OAuth response
Original request
Original response

while the real order (as it actually happened) is more like:
Original request
OAuth request
OAuth response
Original response

I don't think this is an issue and it will even be consistent between authenticated/authorized requests and unauthenticated/unauthorized requests.
Basically: Every request made before the actual logging will be done out of order.


[react-core] init

Description:
This PR creates a new thirdweb-dev/react-core package that extracts the core functionality of the thirdweb-dev/react package and eliminates all dependencies on React-specific components and packages. The goal is to reuse as much code as possible between our React and future React Native SDKs.
Steps taken:
1- Removed all UI components and the react-dom dependency
2- Removed all wagmi and wallet connectors dependencies
3- Removed ThirdwebProvider. The React and React Native SDKs will implement their own providers
How I tested it?
yalc'ed the new react-core package into a new React Native app and tested some of the core hooks. Will run a TestJam after the react package adds the react-core package as a dependency
Follow-ups

Test in Solana
Remove the window reference with the new auth implementation
Integrate the new thirdweb-dev/chains package once it's ready

@ikethirdweb make sure you update turbo.json at the root of the repo to configure the cache behavior (copy the pattern from the other packages)