CI: Add make testp

Validates that "make testp" does not error out.

Any thoughts on whether P=1 or P=2 is preferable?

why run this for buildold?

Any thoughts on whether P=1 or P=2 is preferable?

I would say P=1 if we're just going to validate that the testp target works (to minimize the execution time).

why run this for buildold?

I picked the build with the shortest execution time, currently.

Any thoughts on whether P=1 or P=2 is preferable?

I would say P=1 if we're just going to validate that the testp target works (to minimize the execution time).

Do the tests increase in size as P increases or are they divided up into smaller sections? If the latter then maybe higher values of P will go faster than lower.

Github CI (and all others) run with 2 smt cores per VM.

I'd go for P=2 if that's the case.

2020-07-23T22:12:15.3515170Z make -C partitions testp OPTS='-optimize -production  -g -optimize -production  -g' TESTOPTS='++local'
2020-07-23T22:12:15.3536949Z make[3]: Entering directory '/home/runner/work/charm/charm/netlrts-linux-x86_64/tests/charm++/partitions'
2020-07-23T22:12:15.3537825Z ../../../bin/testrun  ./hello +p2 $(( 2 * 4 )) $(( 2 / 2 )) +partitions $(( 2 / 2 ))  ++local
2020-07-23T22:12:15.3676817Z Charmrun> scalable start enabled.
2020-07-23T22:12:15.3677241Z Charmrun> started all node programs in 0.007 seconds.
2020-07-23T22:12:15.3678627Z Charm++> Running in non-SMP mode: 2 processes (PEs)
2020-07-23T22:12:15.3678930Z Converse/Charm++ Commit ID: v6.11.0-devel-328-gf89f53f6d
2020-07-23T22:12:15.3728662Z Isomalloc> Synchronized global address space.
2020-07-23T22:12:15.3733109Z Charm++> scheduler running in netpoll mode.
2020-07-23T22:12:15.3733468Z CharmLB> Load balancer assumes all CPUs are same.
2020-07-23T22:12:15.3746012Z Charm++> Running on 1 hosts (1 sockets x 2 cores x 1 PUs = 2-way SMP)
2020-07-23T22:12:15.3746597Z Charm++> cpu topology info is gathered in 0.001 seconds.
2020-07-23T22:12:15.3748234Z hello: hello.C:26: Main::Main(CkArgMsg*): Assertion `CkNumPes() == expectedNumPes' failed.
2020-07-23T22:12:15.3749425Z ------------- Processor 0 Exiting: Caught Signal ------------
2020-07-23T22:12:15.3749968Z Reason: unknown signal
2020-07-23T22:12:15.3761655Z [0] Stack Traceback:
2020-07-23T22:12:15.3762807Z   [0:0] libc.so.6 0x7ffff6c77fd0
2020-07-23T22:12:15.3764388Z   [0:1] libc.so.6 0x7ffff6c77f47 gsignal
2020-07-23T22:12:15.3765711Z   [0:2] libc.so.6 0x7ffff6c798b1 abort
2020-07-23T22:12:15.3766707Z   [0:3] libc.so.6 0x7ffff6c6942a
2020-07-23T22:12:15.3767688Z   [0:4] libc.so.6 0x7ffff6c694a2
2020-07-23T22:12:15.3769099Z   [0:5] hello 0x5555555fdc54 CkIndex_Main::_call_Main_CkArgMsg(void*, void*)
2020-07-23T22:12:15.3770231Z   [0:6] hello 0x5555556011ae _initCharm(int, char**)
2020-07-23T22:12:15.3771318Z   [0:7] hello 0x55555570476e ConverseInit
2020-07-23T22:12:15.3772374Z   [0:8] hello 0x555555600069 charm_main
2020-07-23T22:12:15.3773467Z   [0:9] libc.so.6 0x7ffff6c5ab97 __libc_start_main
2020-07-23T22:12:15.3774927Z   [0:10] hello 0x5555555fbf6a _start
2020-07-23T22:12:15.3776744Z Fatal error on PE 0> unknown signal

2020-07-23T22:12:15.3515170Z make -C partitions testp OPTS='-optimize -production  -g -optimize -production  -g' TESTOPTS='++local'
2020-07-23T22:12:15.3536949Z make[3]: Entering directory '/home/runner/work/charm/charm/netlrts-linux-x86_64/tests/charm++/partitions'
2020-07-23T22:12:15.3537825Z ../../../bin/testrun  ./hello +p2 $(( 2 * 4 )) $(( 2 / 2 )) +partitions $(( 2 / 2 ))  ++local
2020-07-23T22:12:15.3676817Z Charmrun> scalable start enabled.
2020-07-23T22:12:15.3677241Z Charmrun> started all node programs in 0.007 seconds.
2020-07-23T22:12:15.3678627Z Charm++> Running in non-SMP mode: 2 processes (PEs)
2020-07-23T22:12:15.3678930Z Converse/Charm++ Commit ID: v6.11.0-devel-328-gf89f53f6d
2020-07-23T22:12:15.3728662Z Isomalloc> Synchronized global address space.
2020-07-23T22:12:15.3733109Z Charm++> scheduler running in netpoll mode.
2020-07-23T22:12:15.3733468Z CharmLB> Load balancer assumes all CPUs are same.
2020-07-23T22:12:15.3746012Z Charm++> Running on 1 hosts (1 sockets x 2 cores x 1 PUs = 2-way SMP)
2020-07-23T22:12:15.3746597Z Charm++> cpu topology info is gathered in 0.001 seconds.
2020-07-23T22:12:15.3748234Z hello: hello.C:26: Main::Main(CkArgMsg*): Assertion `CkNumPes() == expectedNumPes' failed.
2020-07-23T22:12:15.3749425Z ------------- Processor 0 Exiting: Caught Signal ------------
2020-07-23T22:12:15.3749968Z Reason: unknown signal
2020-07-23T22:12:15.3761655Z [0] Stack Traceback:
2020-07-23T22:12:15.3762807Z   [0:0] libc.so.6 0x7ffff6c77fd0
2020-07-23T22:12:15.3764388Z   [0:1] libc.so.6 0x7ffff6c77f47 gsignal
2020-07-23T22:12:15.3765711Z   [0:2] libc.so.6 0x7ffff6c798b1 abort
2020-07-23T22:12:15.3766707Z   [0:3] libc.so.6 0x7ffff6c6942a
2020-07-23T22:12:15.3767688Z   [0:4] libc.so.6 0x7ffff6c694a2
2020-07-23T22:12:15.3769099Z   [0:5] hello 0x5555555fdc54 CkIndex_Main::_call_Main_CkArgMsg(void*, void*)
2020-07-23T22:12:15.3770231Z   [0:6] hello 0x5555556011ae _initCharm(int, char**)
2020-07-23T22:12:15.3771318Z   [0:7] hello 0x55555570476e ConverseInit
2020-07-23T22:12:15.3772374Z   [0:8] hello 0x555555600069 charm_main
2020-07-23T22:12:15.3773467Z   [0:9] libc.so.6 0x7ffff6c5ab97 __libc_start_main
2020-07-23T22:12:15.3774927Z   [0:10] hello 0x5555555fbf6a _start
2020-07-23T22:12:15.3776744Z Fatal error on PE 0> unknown signal

I'll look into this issue.

I'll look into this issue.

Sounds good. Feel free to push to this branch.

I'll look into this issue.

Sounds good. Feel free to push to this branch.

The previously added command was incorrect breaking an assert in the test. I fixed it in the latest commit.

@nitbhat Looks like examples/ampi/Cjacobi3D is now failing due to parameters being incorrect.

@nitbhat Looks like examples/ampi/Cjacobi3D is now failing due to parameters being incorrect.

The error was because there was no testp target inside benchmarks. I fixed it in the latest commit.

2020-07-27T19:54:42.0374002Z make[1]: Leaving directory '/home/runner/work/charm/charm/netlrts-linux-x86_64/examples'
2020-07-27T19:54:42.0374417Z make -C ../benchmarks testp
2020-07-27T19:54:42.0476437Z make[1]: *** No rule to make target 'testp'.  Stop.
2020-07-27T19:54:42.0476743Z make[1]: Entering directory '/home/runner/work/charm/charm/netlrts-linux-x86_64/benchmarks'
2020-07-27T19:54:42.0480297Z make: *** [testp] Error 2
2020-07-27T19:54:42.0480709Z make[1]: Leaving directory '/home/runner/work/charm/charm/netlrts-linux-x86_64/benchmarks'

Most recent failure was:

make[3]: Leaving directory '/home/runner/work/charm/charm/netlrts-linux-x86_64/benchmarks/ampi/alltoall'
make -C isomalloc testp OPTS='-optimize -production  -g -optimize -production  -g' TESTOPTS='++local' P='2'
make[3]: Entering directory '/home/runner/work/charm/charm/netlrts-linux-x86_64/benchmarks/ampi/isomalloc'
../../../bin/testrun  ./gnu-isomalloc +p2 +vp2 +balancer RandCentLB  ++local
Charmrun> scalable start enabled.
Charmrun> started all node programs in 0.019 seconds.
Charm++> Running in non-SMP mode: 2 processes (PEs)
Converse/Charm++ Commit ID: v6.11.0-devel-337-g4ee763eff
Isomalloc> Synchronized global address space.
Charm++> scheduler running in netpoll mode.
CharmLB> Load balancer assumes all CPUs are same.
Charm++> Running on 1 hosts (1 sockets x 2 cores x 1 PUs = 2-way SMP)
Charm++> cpu topology info is gathered in 0.001 seconds.
[0] TreeLB in LEGACY MODE support
[0] TreeLB: Using PE_Root tree with strategy Random
malloc stress test: ./gnu-isomalloc (...) +vp2 randomseed 171084924
Charmrun> error on request socket to node 0 '<IP_ADDRESS>'--
Socket closed before recv.

real	1m25.849s
user	0m0.004s
sys	0m0.025s
Makefile:65: recipe for target 'testp-gnu-isomalloc' failed

The error was because there was no testp target inside benchmarks. I fixed it in the latest commit.

Ah, I see. I guess the jacobi tests don't call MPI_Abort or return a non-zero exit code when parameters are wrong, so that they fail silently. That is worth noting but we don't strictly need to address it as part of this PR.
2020-07-27T21:09:43.2531055Z make -C Cjacobi3D testp OPTS='-optimize -production  -g' TESTOPTS='++local' P='2'
2020-07-27T21:09:43.2573436Z make[3]: Entering directory '/home/runner/work/charm/charm/netlrts-linux-x86_64/examples/ampi/Cjacobi3D'
2020-07-27T21:09:43.2573873Z ../../../bin/testrun  +p2 ./jacobi.pup 2 2 2 40 +vp2 +balancer RotateLB +LBDebug 1  ++local
2020-07-27T21:09:43.2839599Z Charmrun> scalable start enabled.
2020-07-27T21:09:43.2840044Z Charmrun> started all node programs in 0.019 seconds.
2020-07-27T21:09:43.2841033Z Charm++> Running in non-SMP mode: 2 processes (PEs)
2020-07-27T21:09:43.2841596Z Converse/Charm++ Commit ID: v6.11.0-devel-337-g4ee763eff
2020-07-27T21:09:43.2854173Z Isomalloc> Synchronized global address space.
2020-07-27T21:09:43.2854600Z Charm++> scheduler running in netpoll mode.
2020-07-27T21:09:43.2869814Z CharmLB> Verbose level 1, load balancing period: -1 seconds
2020-07-27T21:09:43.2870252Z CharmLB> Load balancer assumes all CPUs are same.
2020-07-27T21:09:43.2919470Z Charm++> Running on 1 hosts (1 sockets x 2 cores x 1 PUs = 2-way SMP)
2020-07-27T21:09:43.2921654Z Charm++> cpu topology info is gathered in 0.001 seconds.
2020-07-27T21:09:43.2924153Z [0] TreeLB in LEGACY MODE support
2020-07-27T21:09:43.2925636Z [0] TreeLB: Using PE_Root tree with strategy Rotate
2020-07-27T21:09:43.3129097Z 2 x 2 x 2 != 2
2020-07-27T21:09:43.3131717Z [Partition 0][Node 0] End of program
2020-07-27T21:09:43.3140607Z
2020-07-27T21:09:43.3140955Z ../../../bin/testrun  +p2 ./jacobi.pup 2 2 2 40 +vp$(( 2 * 2 )) +balancer RotateLB +LBDebug 1  ++local
2020-07-27T21:09:43.3141270Z real	0m0.052s
2020-07-27T21:09:43.3141416Z user	0m0.004s
2020-07-27T21:09:43.3141550Z sys	0m0.000s
2020-07-27T21:09:43.3287253Z Charmrun> scalable start enabled.
2020-07-27T21:09:43.3288302Z Charmrun> started all node programs in 0.007 seconds.
2020-07-27T21:09:43.3289162Z Charm++> Running in non-SMP mode: 2 processes (PEs)
2020-07-27T21:09:43.3289636Z Converse/Charm++ Commit ID: v6.11.0-devel-337-g4ee763eff
2020-07-27T21:09:43.3310763Z Isomalloc> Synchronized global address space.
2020-07-27T21:09:43.3314070Z Charm++> scheduler running in netpoll mode.
2020-07-27T21:09:43.3314684Z CharmLB> Verbose level 1, load balancing period: -1 seconds
2020-07-27T21:09:43.3315155Z CharmLB> Load balancer assumes all CPUs are same.
2020-07-27T21:09:43.3342955Z Charm++> Running on 1 hosts (1 sockets x 2 cores x 1 PUs = 2-way SMP)
2020-07-27T21:09:43.3344227Z Charm++> cpu topology info is gathered in 0.001 seconds.
2020-07-27T21:09:43.3345263Z [0] TreeLB in LEGACY MODE support
2020-07-27T21:09:43.3346498Z [0] TreeLB: Using PE_Root tree with strategy Rotate
2020-07-27T21:09:43.3573805Z 2 x 2 x 2 != 4
2020-07-27T21:09:43.3839164Z [Partition 0][Node 0] End of program
2020-07-27T21:09:43.3844789Z
2020-07-27T21:09:43.3848045Z real	0m0.065s
2020-07-27T21:09:43.3848502Z user	0m0.003s
2020-07-27T21:09:43.3848653Z sys	0m0.000s

Most recent failure was:
../../../bin/testrun  ./gnu-isomalloc +p2 +vp2 +balancer RandCentLB  ++local

I'll look at this one.

The error was because there was no testp target inside benchmarks. I fixed it in the latest commit.

Ah, I see. I guess the jacobi tests don't call MPI_Abort or return a non-zero exit code when parameters are wrong, so that they fail silently. That is worth noting but we don't strictly need to address it as part of this PR.
2020-07-27T21:09:43.2531055Z make -C Cjacobi3D testp OPTS='-optimize -production  -g' TESTOPTS='++local' P='2'
2020-07-27T21:09:43.2573436Z make[3]: Entering directory '/home/runner/work/charm/charm/netlrts-linux-x86_64/examples/ampi/Cjacobi3D'
2020-07-27T21:09:43.2573873Z ../../../bin/testrun  +p2 ./jacobi.pup 2 2 2 40 +vp2 +balancer RotateLB +LBDebug 1  ++local
2020-07-27T21:09:43.2839599Z Charmrun> scalable start enabled.
2020-07-27T21:09:43.2840044Z Charmrun> started all node programs in 0.019 seconds.
2020-07-27T21:09:43.2841033Z Charm++> Running in non-SMP mode: 2 processes (PEs)
2020-07-27T21:09:43.2841596Z Converse/Charm++ Commit ID: v6.11.0-devel-337-g4ee763eff
2020-07-27T21:09:43.2854173Z Isomalloc> Synchronized global address space.
2020-07-27T21:09:43.2854600Z Charm++> scheduler running in netpoll mode.
2020-07-27T21:09:43.2869814Z CharmLB> Verbose level 1, load balancing period: -1 seconds
2020-07-27T21:09:43.2870252Z CharmLB> Load balancer assumes all CPUs are same.
2020-07-27T21:09:43.2919470Z Charm++> Running on 1 hosts (1 sockets x 2 cores x 1 PUs = 2-way SMP)
2020-07-27T21:09:43.2921654Z Charm++> cpu topology info is gathered in 0.001 seconds.
2020-07-27T21:09:43.2924153Z [0] TreeLB in LEGACY MODE support
2020-07-27T21:09:43.2925636Z [0] TreeLB: Using PE_Root tree with strategy Rotate
2020-07-27T21:09:43.3129097Z 2 x 2 x 2 != 2
2020-07-27T21:09:43.3131717Z [Partition 0][Node 0] End of program
2020-07-27T21:09:43.3140607Z
2020-07-27T21:09:43.3140955Z ../../../bin/testrun  +p2 ./jacobi.pup 2 2 2 40 +vp$(( 2 * 2 )) +balancer RotateLB +LBDebug 1  ++local
2020-07-27T21:09:43.3141270Z real	0m0.052s
2020-07-27T21:09:43.3141416Z user	0m0.004s
2020-07-27T21:09:43.3141550Z sys	0m0.000s
2020-07-27T21:09:43.3287253Z Charmrun> scalable start enabled.
2020-07-27T21:09:43.3288302Z Charmrun> started all node programs in 0.007 seconds.
2020-07-27T21:09:43.3289162Z Charm++> Running in non-SMP mode: 2 processes (PEs)
2020-07-27T21:09:43.3289636Z Converse/Charm++ Commit ID: v6.11.0-devel-337-g4ee763eff
2020-07-27T21:09:43.3310763Z Isomalloc> Synchronized global address space.
2020-07-27T21:09:43.3314070Z Charm++> scheduler running in netpoll mode.
2020-07-27T21:09:43.3314684Z CharmLB> Verbose level 1, load balancing period: -1 seconds
2020-07-27T21:09:43.3315155Z CharmLB> Load balancer assumes all CPUs are same.
2020-07-27T21:09:43.3342955Z Charm++> Running on 1 hosts (1 sockets x 2 cores x 1 PUs = 2-way SMP)
2020-07-27T21:09:43.3344227Z Charm++> cpu topology info is gathered in 0.001 seconds.
2020-07-27T21:09:43.3345263Z [0] TreeLB in LEGACY MODE support
2020-07-27T21:09:43.3346498Z [0] TreeLB: Using PE_Root tree with strategy Rotate
2020-07-27T21:09:43.3573805Z 2 x 2 x 2 != 4
2020-07-27T21:09:43.3839164Z [Partition 0][Node 0] End of program
2020-07-27T21:09:43.3844789Z
2020-07-27T21:09:43.3848045Z real	0m0.065s
2020-07-27T21:09:43.3848502Z user	0m0.003s
2020-07-27T21:09:43.3848653Z sys	0m0.000s

Most recent failure was:
../../../bin/testrun  ./gnu-isomalloc +p2 +vp2 +balancer RandCentLB  ++local

I'll look at this one.

I missed the Jacobi silent failure. I'll fix that.
For the gnu-isomalloc, when I locally tried it, there was an issue with CMA (https://github.com/UIUC-PPL/charm/issues/2962). But I think CMA is not enabled in the ci test environment, so this issue seems to be different.
On trying locally with +noCMAForZC, I am able to reproduce the issue.
../../../bin/testrun  ./gnu-isomalloc +p2 +vp2 +balancer RandCentLB  +noCMAForZC
Charmrun> scalable start enabled.
Charmrun> started all node programs in 3.976 seconds.
Charm++> Running in non-SMP mode: 2 processes (PEs)
Converse/Charm++ Commit ID: v6.11.0-devel-335-g7a48f012c
Charm++ built without optimization.
Do not use for performance benchmarking (build with --with-production to do so).
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
Charm++> scheduler running in netpoll mode.
CharmLB> Load balancer assumes all CPUs are same.
Charm++> Running on 1 hosts (1 sockets x 4 cores x 2 PUs = 8-way SMP)
Charm++> cpu topology info is gathered in 0.000 seconds.
[0] TreeLB in LEGACY MODE support
[0] TreeLB: Using PE_Root tree with strategy Random
malloc stress test: ./gnu-isomalloc (...) +vp2 randomseed 398988888
Charmrun> error on request socket to node 1 '<IP_ADDRESS>'--
Socket closed before recv.

real	3m57.642s
user	0m0.027s
sys	0m0.013s
Makefile:65: recipe for target 'test-gnu-isomalloc' failed
make: *** [test-gnu-isomalloc] Error 1

I think the test takes really long because it uses the copy based approach for transferring (and maybe charmrun times out with the usual socket closed for recv error).

../../../bin/testrun  ./gnu-isomalloc +p2 +vp2 +balancer RandCentLB  +noCMAForZC ++timeout 500
Even with a large timeout, I see this error. So, I'm wondering if this is a segfault.

../../../bin/testrun ./gnu-isomalloc +p2 +vp2 +balancer RandCentLB +noCMAForZC ++timeout 500
Even with a large timeout, I see this error. So, I'm wondering if this is a segfault.

Autobuild is blown up with crashes in the isomalloc benchmark and they look like segfaults.

Running locally reveals the issue: Charm++> [1] Isomalloc tried to mmap(0x2aaaa8000000,<PHONE_NUMBER>), but encountered error 12, which means ENOMEM. In a two-process, two-rank run, each process consumes more than 10 GB of memory and eventually uses all of the machine's RAM and swap space.
I'm not sure why this would be the case, because IIRC I designed this benchmark to only use about 1 GB of memory total across the entire distributed job.

@nitbhat I've found that there is a memory leak in the pup_buffer API, at least with +noCMAForZC. Here is a unit test case that demonstrates the problem by migrating in a loop with a large stack size. If you observe memory usage in a tool like htop it will continually increase with each migration until the system runs out of space. If the test calls AMPI_Register_pup to disable pup_buffer, the leak does not occur and memory usage stays constant.
// ampiCC simple-migrate.cpp -o simple-migrate -module TreeLB -memory default -g
// ./charmrun ./simple-migrate +p2 +vp2 ++local +balancer RotateLB +noCMAForZC +tcharm_stacksize 256M
// Then, observe memory usage in `htop`.
// The bug does not occur when compiled with -DTEST_PUP, which disables pup_buffer.

// With valgrind:
// ./charmrun `which valgrind` --leak-check=full --show-leak-kinds=all -- ./simple-migrate +p2 +vp2 ++local +balancer RotateLB +tcharm_stacksize 256M +noCMAForZC ++in-xterm

#include <stdio.h>
#include "mpi.h"

#ifdef TEST_PUP
void null_pup(pup_er p, void *d)
{
}
#endif

int main(int argc, char** argv)
{
  MPI_Init(&argc, &argv);

  int rank, flag;
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);

#ifdef TEST_PUP
  void * ptr;
  int idx;
  AMPI_Register_pup(null_pup, &ptr, &idx);
#endif

  for (int i = 0; i < 1000; ++i)
  {
    {
      int wth, process;
      MPI_Comm_get_attr(MPI_COMM_WORLD, AMPI_MY_WTH, &wth, &flag);
      MPI_Comm_get_attr(MPI_COMM_WORLD, AMPI_MY_PROCESS, &process, &flag);
      printf("[%d](%d)(%d) Migrating...\n", rank, process, wth);
    }

    AMPI_Migrate(AMPI_INFO_LB_SYNC);

    {
      int wth, process;
      MPI_Comm_get_attr(MPI_COMM_WORLD, AMPI_MY_WTH, &wth, &flag);
      MPI_Comm_get_attr(MPI_COMM_WORLD, AMPI_MY_PROCESS, &process, &flag);
      printf("[%d](%d)(%d) Migrated.\n", rank, process, wth);
    }
  }

  MPI_Finalize();

  return 0;
}

With the valgrind command and the loop changed to 10 iterations I get the following:
==20526== 1,671,544 bytes in 29,849 blocks are still reachable in loss record 631 of 633
==20526==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==20526==    by 0x5BEA3A: EnqueueOutgoingDgram(OutgoingMsgStruct*, char*, int, OtherNodeStruct*, int, int) (machine-eth.C:312)
==20526==    by 0x5BEBD1: DeliverViaNetwork(OutgoingMsgStruct*, OtherNodeStruct*, int, unsigned int, int) (machine-eth.C:352)
==20526==    by 0x5BFDC6: DeliverOutgoingMessage(OutgoingMsgStruct*) (machine.C:1640)
==20526==    by 0x5BFE93: LrtsSendFunc(int, int, int, char*, int) (machine.C:1688)
==20526==    by 0x5B9927: CmiInterSendNetworkFunc(int, int, int, char*, int) (machine-common-core.C:692)
==20526==    by 0x5B99E2: CmiInterFreeSendFn (machine-common-core.C:729)
==20526==    by 0x5B9959: CmiFreeSendFn (machine-common-core.C:698)
==20526==    by 0x5C234B: CmiIssueRput(ncpystruct*) (conv-rdma.C:119)
==20526==    by 0x5C2153: getRequestHandler(_converseRdmaMsg*) (conv-rdma.C:50)
==20526==    by 0x5C585C: CmiHandleMessage (convcore.C:1702)
==20526==    by 0x5C5B13: CsdScheduleForever (convcore.C:1953)
==20526==
==20526== 1,342,526,280 bytes in 5 blocks are possibly lost in loss record 632 of 633
==20526==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==20526==    by 0x5B68E8: malloc_nomigrate (memory.C:1083)
==20526==    by 0x5C8CD1: CmiAlloc (convcore.C:3209)
==20526==    by 0x5BECE3: AssembleDatagram(OtherNodeStruct*, ExplicitDgramStruct*) (machine-eth.C:392)
==20526==    by 0x5BEF1E: AssembleReceivedDatagrams(OtherNodeStruct*) (machine-eth.C:443)
==20526==    by 0x5BF0BF: IntegrateMessageDatagram(ExplicitDgramStruct*) (machine-eth.C:487)
==20526==    by 0x5BF694: ReceiveDatagram() (machine-eth.C:672)
==20526==    by 0x5BF81B: CommunicationServerNet(int, int) (machine-eth.C:725)
==20526==    by 0x5BFEDA: LrtsAdvanceCommunication(int) (machine.C:1760)
==20526==    by 0x5BAF48: AdvanceCommunication(int) (machine-common-core.C:1629)
==20526==    by 0x5BB1E1: CmiGetNonLocal (machine-common-core.C:1816)
==20526==    by 0x5C5976: CsdNextMessage (convcore.C:1814)
==20526==
==20526== 1,342,526,280 bytes in 5 blocks are possibly lost in loss record 633 of 633
==20526==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==20526==    by 0x5B68E8: malloc_nomigrate (memory.C:1083)
==20526==    by 0x5C8CD1: CmiAlloc (convcore.C:3209)
==20526==    by 0x5BECE3: AssembleDatagram(OtherNodeStruct*, ExplicitDgramStruct*) (machine-eth.C:392)
==20526==    by 0x5BEF1E: AssembleReceivedDatagrams(OtherNodeStruct*) (machine-eth.C:443)
==20526==    by 0x5BF0BF: IntegrateMessageDatagram(ExplicitDgramStruct*) (machine-eth.C:487)
==20526==    by 0x5BF694: ReceiveDatagram() (machine-eth.C:672)
==20526==    by 0x5BF81B: CommunicationServerNet(int, int) (machine-eth.C:725)
==20526==    by 0x5BFEDA: LrtsAdvanceCommunication(int) (machine.C:1760)
==20526==    by 0x5BAF48: AdvanceCommunication(int) (machine-common-core.C:1629)
==20526==    by 0x5BB2A8: CmiNotifyStillIdle(CmiIdleState*) (machine-common-core.C:1903)
==20526==    by 0x5C0F07: call_cblist_keep(_ccd_cblist*, double) (conv-conds.C:166)
==20526==
==20526== LEAK SUMMARY:
==20526==    definitely lost: 22,636 bytes in 70 blocks
==20526==    indirectly lost: 0 bytes in 0 blocks
==20526==      possibly lost: 2,685,055,224 bytes in 49 blocks
==20526==    still reachable: 1,994,223 bytes in 31,587 blocks
==20526==                       of which reachable via heuristic:
==20526==                         newarray           : 36,008 bytes in 1 blocks
==20526==         suppressed: 0 bytes in 0 blocks

With -DTEST_PUP the multi-GB leaks disappear but there is still this leak of about 1.5 MB:
==20682==
==20682== 1,672,048 bytes in 29,858 blocks are still reachable in loss record 598 of 598
==20682==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==20682==    by 0x5BEA9A: EnqueueOutgoingDgram(OutgoingMsgStruct*, char*, int, OtherNodeStruct*, int, int) (machine-eth.C:312)
==20682==    by 0x5BEC31: DeliverViaNetwork(OutgoingMsgStruct*, OtherNodeStruct*, int, unsigned int, int) (machine-eth.C:352)
==20682==    by 0x5BFE26: DeliverOutgoingMessage(OutgoingMsgStruct*) (machine.C:1640)
==20682==    by 0x5BFEF3: LrtsSendFunc(int, int, int, char*, int) (machine.C:1688)
==20682==    by 0x5B9987: CmiInterSendNetworkFunc(int, int, int, char*, int) (machine-common-core.C:692)
==20682==    by 0x5B9A42: CmiInterFreeSendFn (machine-common-core.C:729)
==20682==    by 0x5B99B9: CmiFreeSendFn (machine-common-core.C:698)
==20682==    by 0x4E5EFC: Converse::CmiSyncSendAndFree(int, int, char*) (middle-conv.h:65)
==20682==    by 0x4EB5BB: _noCldEnqueue(int, envelope*) (ck.C:1508)
==20682==    by 0x4EBCE8: _sendMsgBranch(int, void*, _ckGroupID, int, int) (ck.C:1745)
==20682==    by 0x4EC494: CkSendMsgBranch (ck.C:1848)
==20682==
==20682== LEAK SUMMARY:
==20682==    definitely lost: 20,608 bytes in 51 blocks
==20682==    indirectly lost: 0 bytes in 0 blocks
==20682==      possibly lost: 2,664 bytes in 39 blocks
==20682==    still reachable: 1,994,811 bytes in 31,560 blocks
==20682==                       of which reachable via heuristic:
==20682==                         newarray           : 36,008 bytes in 1 blocks
==20682==         suppressed: 0 bytes in 0 blocks

Thanks @evan-charmworks for the test-case and the diagnosis. The bug was caused because of not freeing the received message in the putDataHandler (in the non onesided implementation of the ZC API, which was exercised only on netlrts). I've posted a fix for it in this PR (https://github.com/UIUC-PPL/charm/pull/2998).

Ready for review.
The Isomalloc benchmark will require #2188 to work correctly so I've opted to disable the test for now.

Does this not work with the new build system?

It does. Which CI would you suggest we run make testp on? I don't think we can run both make test and make testp without it timing out.

It seems make testp takes only 4 minutes, or am I seeing it wrong? Imho, we could also add this to at least one other cmake build, say Verbs Linux SMP.

Thanks for pointing that out. I think the Isomalloc benchmark was taking a disproportionate amount of time. I've added testp to various CIs that complete in <= 40 minutes, including some +p4 ++ppn 2 variants in SMP builds.
I'm not sure if we can enable tests for the Verbs CI if there is no Infiniband hardware.

Maximum test time on GitHub AFAIR is 6 hours, so we should be able to run it even if it takes a bit longer.
Edit:
See https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#usage-limits for maximum job time.

Good to know. We should still avoid increasing test length too much so the CI is not a burden on the merge master.

netlrts smp +p4 ++ppn 2 failure in lb_test:
make -C lb_test testp OPTS='-g -Werror=vla -optimize -production' TESTOPTS='++local +setcpuaffinity +CmiSleepOnIdle ++ppn 2'
make[3]: Entering directory '/home/circleci/project/netlrts-linux-x86_64-smp/tests/charm++/load_balancing/lb_test'
../../../../bin/testrun  +p4 ./lb_test $(( 25 * 4)) 100 10 40 10 1000 ring +balancer GreedyLB +LBDebug 1  ++local +setcpuaffinity +CmiSleepOnIdle ++ppn 2
Charmrun> scalable start enabled.
Charmrun> started all node programs in 0.008 seconds.
Charm++> Running in SMP mode: 2 processes, 2 worker threads (PEs) + 1 comm threads per process, 4 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: v6.11.0-devel-366-gc2f0768cb
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
Charm++> scheduler running in netpoll mode.
CharmLB> Verbose level 1, load balancing period: -1 seconds
CharmLB> Load balancer assumes all CPUs are same.
traceprojections was off at initial time.
Charm++> cpu affinity enabled.
Charm++> Running on 1 hosts (1 sockets x 1 cores x 2 PUs = 2-way SMP)
Charm++> cpu topology info is gathered in 0.106 seconds.
WARNING: Multiple PEs assigned to same core, recommend adjusting processor affinity or passing +CmiSleepOnIdle to reduce interference.
[0] TreeLB in LEGACY MODE support
[0] TreeLB: Using PE_Root tree with strategy Greedy
Running lb_test on 4 processors with 100 elements
Print every 10 steps
Sync every 40 steps
First node busywaits 10 usec; last node busywaits 1000 usec

Selecting Topology Ring
Generating topology 0 for 100 elements
[0] Total work/step = 0.020908 sec
calibrated iterations 12721987
TIME PER STEP	10	0.639974	0.104232
TIME PER STEP	20	0.744706	0.104731
TIME PER STEP	30	0.839714	0.095009
--------- Started LB step 0 ---------
[0] num_procs=4 num_objs=100
[0] obj loads: min=0.000222 mean=0.013601 max=0.175276
[0] proc loads: min=0.233311 mean=0.365779 max=0.509561
[0] strategy Greedy time=0.000012 secs, maxLoad after strategy=0.365835, num_migrations=73 migrations_sum_hops=0
--------- Finished LB step 0 ---------
------------- Processor 0 Exiting: Called CmiAbort ------------
Reason: [0] Assertion "numLevels > 0 && awaitingLB[level]" failed in file /home/circleci/project/src/ck-ldb/TreeLB.C line 465.

[0] Stack Traceback:
  [0:0] lb_test 0x5d5f37
  [0:1] lb_test 0x5e99fa
  [0:2] lb_test 0x4cfcac TreeLB::objMovedIn(bool)
------------- Processor 2 Exiting: Called CmiAbort ------------
Reason: [2] Assertion "numLevels > 0 && awaitingLB[level]" failed in file /home/circleci/project/src/ck-ldb/TreeLB.C line 465.

------------- Processor 3 Exiting: Called CmiAbort ------------
Reason: [3] Assertion "numLevels > 0 && awaitingLB[level]" failed in file /home/circleci/project/src/ck-ldb/TreeLB.C line 465.

[2] Stack Traceback:
[3] Stack Traceback:
  [2:0] lb_test 0x5d5f37
  [0:3] lb_test 0x55e5dd CkLocRec::CkLocRec(CkLocMgr*, bool, bool, CkArrayIndex const&, unsigned long)
  [0:4] lb_test 0x56d224 CkLocMgr::createLocal(CkArrayIndex const&, bool, bool, bool)
  [0:5] lb_test 0x56d6e2 CkLocMgr::immigrate(CkArrayElementMigrateMessage*)
  [0:6] lb_test 0x52d7f4 CkDeliverMessageFree
  [0:7] lb_test 0x530638 _processHandler(void*, CkCoreState*)
  [0:8] lb_test 0x5e613b CsdScheduleForever
  [0:9] lb_test 0x5e691d CsdScheduler
  [0:10] lb_test 0x5df8c9
  [0:11] lb_test 0x5dfe64 ConverseInit
  [0:12] lb_test 0x5ba017 charm_main
  [3:0] lb_test 0x5d5f37
  [3:1] lb_test 0x5e99fa
  [3:2] lb_test 0x4cfcac TreeLB::objMovedIn(bool)
  [3:3] lb_test 0x55e5dd CkLocRec::CkLocRec(CkLocMgr*, bool, bool, CkArrayIndex const&, unsigned long)
  [3:4] lb_test 0x56d224 CkLocMgr::createLocal(CkArrayIndex const&, bool, bool, bool)
  [3:5] lb_test 0x56d6e2 CkLocMgr::immigrate(CkArrayElementMigrateMessage*)
  [3:6] lb_test 0x52d7f4 CkDeliverMessageFree
  [3:7] lb_test 0x530638 _processHandler(void*, CkCoreState*)
  [3:8] lb_test 0x5e613b CsdScheduleForever
  [3:9] lb_test 0x5e691d CsdScheduler
  [0:13] libc.so.6 0x7ffff6b51830 __libc_start_main
  [0:14] lb_test 0x4b7409 _start
Fatal error on PE 0> [0] Assertion "numLevels > 0 && awaitingLB[level]" failed in file /home/circleci/project/src/ck-ldb/TreeLB.C line 465.

real	0m0.888s
user	0m0.005s
sys	0m0.000s
Makefile:34: recipe for target 'testp' failed
make[3]: *** [testp] Error 1
make[3]: Leaving directory '/home/circleci/project/netlrts-linux-x86_64-smp/tests/charm++/load_balancing/lb_test'

UTH times out in benchmarks/converse/cthtest with +p2.

buildold make test failed with the same hang in benchmarks/ampi/alltoall exhibited by other CIs.

netlrts-linux, netlrts-darwin, and mpi-linux-smp are all hanging in the pingpong-cachemiss test of benchmarks/converse/commbench. #2076

The Isomalloc benchmark will require #2188 to work correctly so I've opted to disable the test for now.

Nitin mentioned in #2962 (comment) that CMA has a 2 GB limit per read() call, so #2188 may be overkill to solve this problem if we have to subdivide transfers into blocks anyway.

Besides the above SMP issue with lb_test, the only remaining failure is a timeout in mpi-linux-smp because pingpong_multipairs took 36 minutes to complete:
../../../bin/testrun  ./pingpong_multipairs +p2  +setcpuaffinity

Running on 2 processors:  ./pingpong_multipairs +setcpuaffinity
charmrun>  /usr/bin/setarch x86_64 -R  mpirun -np 2  ./pingpong_multipairs +setcpuaffinity
Charm++> Running on MPI version: 3.1
Charm++> level of thread support used: -1 (desired: 0)
Charm++> Running in SMP mode: 2 processes, 1 worker threads (PEs) + 1 comm threads per process, 2 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: fe0fa19
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
Charm++> cpu affinity enabled.
Charm++> Running on 1 hosts (1 sockets x 1 cores x 2 PUs = 2-way SMP)
Charm++> cpu topology info is gathered in 0.080 seconds.
Multiple pair send/recv
 bytes 		 latency(us)	 bandwidth(MBytes/sec)
8		  9464.50   0.00
16		  18884.01   0.00
32		  28413.01   0.00
64		  38033.01   0.00
128		  47511.01   0.00
256		  56820.51   0.00
512		  66266.51   0.01
1024		  75828.51   0.01
2048		  85187.51   0.02
4096		  94754.01   0.04
8192		  112328.51   0.07
16384		  128587.51   0.13
32768		  138194.51   0.24
65536		  155832.51   0.42
131072		  173531.01   0.76
262144		  192566.01   1.36
524288		  212303.01   2.47
1048576		  232655.51   4.51
2097152		  253255.51   8.28
4194304		  273867.01   15.32
[Partition 0][Node 0] End of program

real	36m33.207s
user	61m1.942s
sys	11m54.468s
make[2]: Leaving directory '/home/travis/build/UIUC-PPL/charm/mpi-linux-x86_64-smp/benchmarks/converse/pingpong'

@rbuch The same lb_test issue happened with MPI Linux SMP +p4 ++ppn 2
make -C lb_test testp OPTS='-g -Werror=vla -optimize -production' TESTOPTS='+setcpuaffinity ++ppn 2'
make[3]: Entering directory '/home/travis/build/UIUC-PPL/charm/mpi-linux-x86_64-smp/tests/charm++/load_balancing/lb_test'
../../../../bin/testrun  +p4 ./lb_test $(( 25 * 4)) 100 10 40 10 1000 ring +balancer GreedyLB +LBDebug 1  +setcpuaffinity ++ppn 2

Running on 2 processors:  ./lb_test 100 100 10 40 10 1000 ring +balancer GreedyLB +LBDebug 1 +setcpuaffinity +ppn 2
charmrun>  /usr/bin/setarch x86_64 -R  mpirun -np 2  ./lb_test 100 100 10 40 10 1000 ring +balancer GreedyLB +LBDebug 1 +setcpuaffinity +ppn 2
Charm++> Running on MPI version: 3.1
Charm++> level of thread support used: -1 (desired: 0)
Charm++> Running in SMP mode: 2 processes, 2 worker threads (PEs) + 1 comm threads per process, 4 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: 3024712
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
CharmLB> Verbose level 1, load balancing period: -1 seconds
CharmLB> Load balancer assumes all CPUs are same.
traceprojections was off at initial time.
Charm++> cpu affinity enabled.
Charm++> Running on 1 hosts (1 sockets x 1 cores x 2 PUs = 2-way SMP)
Charm++> cpu topology info is gathered in 0.072 seconds.
Charm++> Warning: Running with more SMP threads (6) than physical cores (2).
         Use +CmiSleepOnIdle (default behavior) or +CmiSpinOnIdle to silence this message.
WARNING: Multiple PEs assigned to same core, recommend adjusting processor affinity or passing +CmiSleepOnIdle to reduce interference.
[0] TreeLB in LEGACY MODE support
[0] TreeLB: Using PE_Root tree with strategy Greedy
Running lb_test on 4 processors with 100 elements
Print every 10 steps
Sync every 40 steps
First node busywaits 10 usec; last node busywaits 1000 usec

Selecting Topology Ring
Generating topology 0 for 100 elements
[0] Total work/step = 0.020908 sec
calibrated iterations 12163733
TIME PER STEP	10	0.580004	0.076805
TIME PER STEP	20	0.666661	0.086656
TIME PER STEP	30	0.768079	0.101418
--------- Started LB step 0 ---------
[0] num_procs=4 num_objs=100
[0] obj loads: min=0.000226 mean=0.014686 max=0.181409
[0] proc loads: min=0.262121 mean=0.372752 max=0.484037
[0] strategy Greedy time=0.000079 secs, maxLoad after strategy=0.372875, num_migrations=69 migrations_sum_hops=0
[2] Stack Traceback:
------------- Processor 2 Exiting: Called CmiAbort ------------
Reason: [2] Assertion "numLevels > 0 && awaitingLB[level]" failed in file /home/travis/build/UIUC-PPL/charm/src/ck-ldb/TreeLB.C line 465.

--------- Finished LB step 0 ---------
[0] lb time = 0.008794
  [2:0] lb_test 0x5c861a CmiAbortHelper(char const*, char const*, char const*, int, int)
  [2:1] lb_test 0x5c873b
  [2:2] lb_test 0x5d285c
  [2:3] lb_test 0x4d368a TreeLB::objMovedIn(bool)
  [2:4] lb_test 0x553afd CkLocRec::CkLocRec(CkLocMgr*, bool, bool, CkArrayIndex const&, unsigned long)
  [2:5] lb_test 0x563bea CkLocMgr::createLocal(CkArrayIndex const&, bool, bool, bool)
  [2:6] lb_test 0x563e08 CkLocMgr::immigrate(CkArrayElementMigrateMessage*)
  [2:7] lb_test 0x5280c2 CkDeliverMessageFree
  [2:8] lb_test 0x52a9ce _processHandler(void*, CkCoreState*)
  [2:9] lb_test 0x5cefc8 CsdScheduleForever
  [2:10] lb_test 0x5cf2bd CsdScheduler
  [2:11] lb_test 0x5cb3c2
  [2:12] lb_test 0x5cbd7c
  [2:13] libpthread.so.0 0x7ffff7bc16ba
  [2:14] libc.so.6 0x7ffff67ad41d clone
application called MPI_Abort(comm=0x84000002, 1) - process 1

real	0m0.891s
user	0m0.012s
sys	0m0.004s
Makefile:34: recipe for target 'testp' failed
make[3]: *** [testp] Error 1
make[3]: Leaving directory '/home/travis/build/UIUC-PPL/charm/mpi-linux-x86_64-smp/tests/charm++/load_balancing/lb_test'
Makefile:26: recipe for target 'testp-lb_test' failed
make[2]: *** [testp-lb_test] Error 2

There is also a segfault in examples/charm++/load_balancing/hello:
make -C hello testp OPTS='-g -Werror=vla -optimize -production' TESTOPTS='+setcpuaffinity ++ppn 2' P='4'
make[3]: Entering directory '/home/travis/build/UIUC-PPL/charm/mpi-linux-x86_64-smp/examples/charm++/load_balancing/hello'
../../../../bin/testrun  +p4 ./hello +balancer GreedyRefineLB +LBDebug 1 +setcpuaffinity ++ppn 2

Running on 2 processors:  ./hello +balancer GreedyRefineLB +LBDebug 1 +setcpuaffinity +ppn 2
charmrun>  /usr/bin/setarch x86_64 -R  mpirun -np 2  ./hello +balancer GreedyRefineLB +LBDebug 1 +setcpuaffinity +ppn 2
Charm++> Running on MPI version: 3.1
Charm++> level of thread support used: -1 (desired: 0)
Charm++> Running in SMP mode: 2 processes, 2 worker threads (PEs) + 1 comm threads per process, 4 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: 3024712
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
CharmLB> Verbose level 1, load balancing period: -1 seconds
CharmLB> Load balancer assumes all CPUs are same.
Charm++> cpu affinity enabled.
Charm++> Running on 1 hosts (1 sockets x 1 cores x 2 PUs = 2-way SMP)
Charm++> cpu topology info is gathered in 0.047 seconds.
Charm++> Warning: Running with more SMP threads (6) than physical cores (2).
         Use +CmiSleepOnIdle (default behavior) or +CmiSpinOnIdle to silence this message.
WARNING: Multiple PEs assigned to same core, recommend adjusting processor affinity or passing +CmiSleepOnIdle to reduce interference.
[0] TreeLB in LEGACY MODE support
[0] TreeLB: Using PE_Root tree with strategy GreedyRefine
Hello, I'm chare 0 on PE 0
Hello, I'm chare 1 on PE 0
Hello, I'm chare 2 on PE 1
Hello, I'm chare 3 on PE 1
--------- Started LB step 0 ---------
Hello, I'm chare 4 on PE 2
Hello, I'm chare 6 on PE 3
Hello, I'm chare 7 on PE 3
Hello, I'm chare 5 on PE 2
[0] num_procs=4 num_objs=8
[0] obj loads: min=0.000008 mean=0.500797 max=1.004737
[0] proc loads: min=0.000088 mean=1.001966 max=2.006403
[0] GreedyRefine: greedy maxload is 1.004796, tolerance set to 1.000000
[0] strategy GreedyRefine time=0.000011 secs, maxLoad after strategy=1.004796, num_migrations=4 migrations_sum_hops=0
------------- Processor 2 Exiting: Caught Signal ------------
Reason: Segmentation fault
[2] Stack Traceback:
  [2:0] hello 0x5c32f8
  [2:1] libpthread.so.0 0x7ffff7bcb390
  [2:2] hello 0x4cfd19 PELevel::processDecision(TreeLBMessage*, int&, int&)
  [2:3] hello 0x4cb73d TreeLB::receiveDecision(TreeLBMessage*, int)
  [2:4] hello 0x4cb94b TreeLB::sendDecisionDown(CkMessage*)
  [2:5] hello 0x51c042 CkDeliverMessageFree
  [2:6] hello 0x51e94e _processHandler(void*, CkCoreState*)
  [2:7] hello 0x5c762e CsdScheduleForever
  [2:8] hello 0x5c789d CsdScheduler
  [2:9] hello 0x5c39a2
  [2:10] hello 0x5c435c
  [2:11] libpthread.so.0 0x7ffff7bc16ba
  [2:12] libc.so.6 0x7ffff67ad41d clone
application called MPI_Abort(comm=0x84000002, 1) - process 1

real	0m2.179s
user	0m0.015s
sys	0m0.001s
Makefile:21: recipe for target 'testp' failed
make[3]: *** [testp] Error 1
make[3]: Leaving directory '/home/travis/build/UIUC-PPL/charm/mpi-linux-x86_64-smp/examples/charm++/load_balancing/hello'
Makefile:25: recipe for target 'testp-hello' failed
make[2]: *** [testp-hello] Error 2

@rbuch The same lb_test issue happened with MPI Linux SMP +p4 ++ppn 2
make -C lb_test testp OPTS='-g -Werror=vla -optimize -production' TESTOPTS='+setcpuaffinity ++ppn 2'
make[3]: Entering directory '/home/travis/build/UIUC-PPL/charm/mpi-linux-x86_64-smp/tests/charm++/load_balancing/lb_test'
../../../../bin/testrun  +p4 ./lb_test $(( 25 * 4)) 100 10 40 10 1000 ring +balancer GreedyLB +LBDebug 1  +setcpuaffinity ++ppn 2

Running on 2 processors:  ./lb_test 100 100 10 40 10 1000 ring +balancer GreedyLB +LBDebug 1 +setcpuaffinity +ppn 2
charmrun>  /usr/bin/setarch x86_64 -R  mpirun -np 2  ./lb_test 100 100 10 40 10 1000 ring +balancer GreedyLB +LBDebug 1 +setcpuaffinity +ppn 2
Charm++> Running on MPI version: 3.1
Charm++> level of thread support used: -1 (desired: 0)
Charm++> Running in SMP mode: 2 processes, 2 worker threads (PEs) + 1 comm threads per process, 4 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: 3024712
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
CharmLB> Verbose level 1, load balancing period: -1 seconds
CharmLB> Load balancer assumes all CPUs are same.
traceprojections was off at initial time.
Charm++> cpu affinity enabled.
Charm++> Running on 1 hosts (1 sockets x 1 cores x 2 PUs = 2-way SMP)
Charm++> cpu topology info is gathered in 0.072 seconds.
Charm++> Warning: Running with more SMP threads (6) than physical cores (2).
         Use +CmiSleepOnIdle (default behavior) or +CmiSpinOnIdle to silence this message.
WARNING: Multiple PEs assigned to same core, recommend adjusting processor affinity or passing +CmiSleepOnIdle to reduce interference.
[0] TreeLB in LEGACY MODE support
[0] TreeLB: Using PE_Root tree with strategy Greedy
Running lb_test on 4 processors with 100 elements
Print every 10 steps
Sync every 40 steps
First node busywaits 10 usec; last node busywaits 1000 usec

Selecting Topology Ring
Generating topology 0 for 100 elements
[0] Total work/step = 0.020908 sec
calibrated iterations 12163733
TIME PER STEP	10	0.580004	0.076805
TIME PER STEP	20	0.666661	0.086656
TIME PER STEP	30	0.768079	0.101418
--------- Started LB step 0 ---------
[0] num_procs=4 num_objs=100
[0] obj loads: min=0.000226 mean=0.014686 max=0.181409
[0] proc loads: min=0.262121 mean=0.372752 max=0.484037
[0] strategy Greedy time=0.000079 secs, maxLoad after strategy=0.372875, num_migrations=69 migrations_sum_hops=0
[2] Stack Traceback:
------------- Processor 2 Exiting: Called CmiAbort ------------
Reason: [2] Assertion "numLevels > 0 && awaitingLB[level]" failed in file /home/travis/build/UIUC-PPL/charm/src/ck-ldb/TreeLB.C line 465.

--------- Finished LB step 0 ---------
[0] lb time = 0.008794
  [2:0] lb_test 0x5c861a CmiAbortHelper(char const*, char const*, char const*, int, int)
  [2:1] lb_test 0x5c873b
  [2:2] lb_test 0x5d285c
  [2:3] lb_test 0x4d368a TreeLB::objMovedIn(bool)
  [2:4] lb_test 0x553afd CkLocRec::CkLocRec(CkLocMgr*, bool, bool, CkArrayIndex const&, unsigned long)
  [2:5] lb_test 0x563bea CkLocMgr::createLocal(CkArrayIndex const&, bool, bool, bool)
  [2:6] lb_test 0x563e08 CkLocMgr::immigrate(CkArrayElementMigrateMessage*)
  [2:7] lb_test 0x5280c2 CkDeliverMessageFree
  [2:8] lb_test 0x52a9ce _processHandler(void*, CkCoreState*)
  [2:9] lb_test 0x5cefc8 CsdScheduleForever
  [2:10] lb_test 0x5cf2bd CsdScheduler
  [2:11] lb_test 0x5cb3c2
  [2:12] lb_test 0x5cbd7c
  [2:13] libpthread.so.0 0x7ffff7bc16ba
  [2:14] libc.so.6 0x7ffff67ad41d clone
application called MPI_Abort(comm=0x84000002, 1) - process 1

real	0m0.891s
user	0m0.012s
sys	0m0.004s
Makefile:34: recipe for target 'testp' failed
make[3]: *** [testp] Error 1
make[3]: Leaving directory '/home/travis/build/UIUC-PPL/charm/mpi-linux-x86_64-smp/tests/charm++/load_balancing/lb_test'
Makefile:26: recipe for target 'testp-lb_test' failed
make[2]: *** [testp-lb_test] Error 2

There is also a segfault in examples/charm++/load_balancing/hello:
make -C hello testp OPTS='-g -Werror=vla -optimize -production' TESTOPTS='+setcpuaffinity ++ppn 2' P='4'
make[3]: Entering directory '/home/travis/build/UIUC-PPL/charm/mpi-linux-x86_64-smp/examples/charm++/load_balancing/hello'
../../../../bin/testrun  +p4 ./hello +balancer GreedyRefineLB +LBDebug 1 +setcpuaffinity ++ppn 2

Running on 2 processors:  ./hello +balancer GreedyRefineLB +LBDebug 1 +setcpuaffinity +ppn 2
charmrun>  /usr/bin/setarch x86_64 -R  mpirun -np 2  ./hello +balancer GreedyRefineLB +LBDebug 1 +setcpuaffinity +ppn 2
Charm++> Running on MPI version: 3.1
Charm++> level of thread support used: -1 (desired: 0)
Charm++> Running in SMP mode: 2 processes, 2 worker threads (PEs) + 1 comm threads per process, 4 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: 3024712
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
CharmLB> Verbose level 1, load balancing period: -1 seconds
CharmLB> Load balancer assumes all CPUs are same.
Charm++> cpu affinity enabled.
Charm++> Running on 1 hosts (1 sockets x 1 cores x 2 PUs = 2-way SMP)
Charm++> cpu topology info is gathered in 0.047 seconds.
Charm++> Warning: Running with more SMP threads (6) than physical cores (2).
         Use +CmiSleepOnIdle (default behavior) or +CmiSpinOnIdle to silence this message.
WARNING: Multiple PEs assigned to same core, recommend adjusting processor affinity or passing +CmiSleepOnIdle to reduce interference.
[0] TreeLB in LEGACY MODE support
[0] TreeLB: Using PE_Root tree with strategy GreedyRefine
Hello, I'm chare 0 on PE 0
Hello, I'm chare 1 on PE 0
Hello, I'm chare 2 on PE 1
Hello, I'm chare 3 on PE 1
--------- Started LB step 0 ---------
Hello, I'm chare 4 on PE 2
Hello, I'm chare 6 on PE 3
Hello, I'm chare 7 on PE 3
Hello, I'm chare 5 on PE 2
[0] num_procs=4 num_objs=8
[0] obj loads: min=0.000008 mean=0.500797 max=1.004737
[0] proc loads: min=0.000088 mean=1.001966 max=2.006403
[0] GreedyRefine: greedy maxload is 1.004796, tolerance set to 1.000000
[0] strategy GreedyRefine time=0.000011 secs, maxLoad after strategy=1.004796, num_migrations=4 migrations_sum_hops=0
------------- Processor 2 Exiting: Caught Signal ------------
Reason: Segmentation fault
[2] Stack Traceback:
  [2:0] hello 0x5c32f8
  [2:1] libpthread.so.0 0x7ffff7bcb390
  [2:2] hello 0x4cfd19 PELevel::processDecision(TreeLBMessage*, int&, int&)
  [2:3] hello 0x4cb73d TreeLB::receiveDecision(TreeLBMessage*, int)
  [2:4] hello 0x4cb94b TreeLB::sendDecisionDown(CkMessage*)
  [2:5] hello 0x51c042 CkDeliverMessageFree
  [2:6] hello 0x51e94e _processHandler(void*, CkCoreState*)
  [2:7] hello 0x5c762e CsdScheduleForever
  [2:8] hello 0x5c789d CsdScheduler
  [2:9] hello 0x5c39a2
  [2:10] hello 0x5c435c
  [2:11] libpthread.so.0 0x7ffff7bc16ba
  [2:12] libc.so.6 0x7ffff67ad41d clone
application called MPI_Abort(comm=0x84000002, 1) - process 1

real	0m2.179s
user	0m0.015s
sys	0m0.001s
Makefile:21: recipe for target 'testp' failed
make[3]: *** [testp] Error 1
make[3]: Leaving directory '/home/travis/build/UIUC-PPL/charm/mpi-linux-x86_64-smp/examples/charm++/load_balancing/hello'
Makefile:25: recipe for target 'testp-hello' failed
make[2]: *** [testp-hello] Error 2

Yeah, I've been debugging it since yesterday, I'll make a separate issue. It seems like sometimes PEs get an LB decision that says that no objects are incoming even though there are incoming objects, but I've only seen this in cases with p > 1 and ppn > 1. I'll make a separate issue for this, thanks for pointing it out.

Our MPI Linux SMP build in particular appears to be buggy and I'm leaning toward disabling the +p4 ++ppn 2 part of its CI for this PR to unblock it. NetLRTS SMP does not have the same issues and can keep the PPN CI.

Something about the MPI machine layer with SMP is causing hangs in examples/charm++/zerocopy/entry_method_api/{reg,prereg,unreg}/stencil3d and tests/ampi/intercomm_coll.

I once got this failure in examples/charm++/zerocopy/entry_method_post_api/unreg/simpleZeroCopy, but could not reproduce it:

../../../../../../bin/testrun  +p4 ./simpleZeroCopy $(( 4 * 10 )) +balancer GreedyLB ++ppn 2 +setcpuaffinity

Running on 2 processors:  ./simpleZeroCopy 40 +balancer GreedyLB +ppn 2 +setcpuaffinity
charmrun>  /usr/bin/setarch x86_64 -R  mpirun -np 2  ./simpleZeroCopy 40 +balancer GreedyLB +ppn 2 +setcpuaffinity
Charm++> Running on MPI version: 3.1
Charm++> level of thread support used: -1 (desired: 0)
Charm++> Running in SMP mode: 2 processes, 2 worker threads (PEs) + 1 comm threads per process, 4 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: v6.11.0-devel-384-g86fa91abd
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
CharmLB> Load balancer assumes all CPUs are same.
Charm++> cpu affinity enabled.
Charm++> Running on 1 hosts (1 sockets x 4 cores x 2 PUs = 8-way SMP)
Charm++> cpu topology info is gathered in 0.024 seconds.
[0] TreeLB in LEGACY MODE support
[0] TreeLB: Using PE_Root tree with strategy Greedy
send: completed
[NemeanLion:20932] *** An error occurred in MPI_Irecv
[NemeanLion:20932] *** reported by process [2524905473,1]
[NemeanLion:20932] *** on communicator MPI COMMUNICATOR 3 DUP FROM 0
[NemeanLion:20932] *** MPI_ERR_BUFFER: invalid buffer pointer
[NemeanLion:20932] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[NemeanLion:20932] ***    and potentially your MPI job)
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI COMMUNICATOR 3 DUP FROM 0
with errorcode 1.

You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
------------- Processor 4 Exiting: Caught Signal ------------
Reason: Terminated
[4] Stack Traceback:
  [4:0] simpleZeroCopy 0x5555557425fb
  [4:1] libpthread.so.0 0x7ffff7bc88a0
  [4:2] mca_btl_sm.so 0x7fffe5b03d22 mca_btl_sm_component_progress
  [4:3] libopen-pal.so.20 0x7ffff625fabc opal_progress
  [4:4] mca_pml_ob1.so 0x7fffe56da903 mca_pml_ob1_iprobe
  [4:5] libmpi.so.20 0x7ffff716e31a PMPI_Iprobe
  [4:6] simpleZeroCopy 0x555555740d7b
  [4:7] simpleZeroCopy 0x555555741519 LrtsAdvanceCommunication(int)
  [4:8] simpleZeroCopy 0x5555557417ce CommunicationServerThread(int)
  [4:9] simpleZeroCopy 0x555555741b8a
  [4:10] simpleZeroCopy 0x555555742194 ConverseInit
  [4:11] simpleZeroCopy 0x555555722efc charm_main
  [4:12] libc.so.6 0x7ffff6794b97 __libc_start_main
  [4:13] simpleZeroCopy 0x55555562800a _start

real  0m0.311s
user  0m0.365s
sys 0m0.082s

Also this failure in megatest:

make[3]: Entering directory '/home/evan/Charmworks/charm/mpi-linux-x86_64-smp/tests/charm++/megatest'
../../../bin/testrun  ./pgm +p4  ++ppn 2 +setcpuaffinity

Running on 2 processors:  ./pgm +ppn 2 +setcpuaffinity
charmrun>  /usr/bin/setarch x86_64 -R  mpirun -np 2  ./pgm +ppn 2 +setcpuaffinity
Charm++> Running on MPI version: 3.1
Charm++> level of thread support used: -1 (desired: 0)
Charm++> Running in SMP mode: 2 processes, 2 worker threads (PEs) + 1 comm threads per process, 4 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: v6.11.0-devel-384-g86fa91abd
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
CharmLB> Load balancer assumes all CPUs are same.
Charm++> cpu affinity enabled.
Charm++> Running on 1 hosts (1 sockets x 4 cores x 2 PUs = 8-way SMP)
Charm++> cpu topology info is gathered in 0.020 seconds.
Megatest is running on 2 nodes 4 processors.
test 0: initiated [groupring (milind)]
test 0: completed (0.12 sec)
test 1: initiated [nodering (milind)]
test 1: completed (0.00 sec)
test 2: initiated [varsizetest (mjlang)]
test 2: completed (0.00 sec)
test 3: initiated [varsizetest2 (phil)]
test 3: completed (0.00 sec)
test 4: initiated [varraystest (milind)]
test 4: completed (0.00 sec)
test 5: initiated [groupcast (mjlang)]
test 5: completed (0.00 sec)
test 6: initiated [groupmulti (gengbin)]
test 6: completed (0.00 sec)
test 7: initiated [groupsectiontest (ebohm)]
test 7: completed (0.00 sec)
test 8: initiated [multisectiontest (ebohm)]
test 8: completed (0.03 sec)
test 9: initiated [nodecast (milind)]
test 9: completed (0.00 sec)
test 10: initiated [synctest (mjlang)]
test 10: completed (0.01 sec)
test 11: initiated [fib (jackie)]
test 11: completed (0.01 sec)
test 12: initiated [arrayring (fang)]
test 12: completed (0.00 sec)
test 13: initiated [packtest (fang)]
test 13: completed (0.00 sec)
test 14: initiated [queens (jackie)]
test 14: completed (0.01 sec)
test 15: initiated [migration (jackie)]
test 15: completed (0.00 sec)
test 16: initiated [marshall (olawlor)]
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI COMMUNICATOR 3 DUP FROM 0
with errorcode 1.

You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
test 16: completed (0.32 sec)
test 17: initiated [priomsg (fang)]
test 17: completed (0.00 sec)
test 18: initiated [priotest (mlind)]
test 18: completed (0.03 sec)
test 19: initiated [rotest (milind)]
test 19: completed (0.01 sec)
test 20: initiated [statistics (olawlor)]
test 20: completed (0.00 sec)
test 21: initiated [templates (milind)]
test 21: completed (0.00 sec)
test 22: initiated [inherit (olawlor)]
test 22: completed (0.00 sec)
test 23: initiated [reduction (olawlor)]
test 23: completed (0.00 sec)
test 24: initiated [bitvector (jbooth)]
test 24: completed (0.00 sec)
test 25: initiated [immediatering (gengbin)]
test 25: completed (0.00 sec)
test 26: initiated [callback (olawlor)]
test 26: completed (0.00 sec)
test 27: initiated [inlineem (phil)]
test 27: completed (0.00 sec)
test 28: initiated [completion_test (phil)]
Starting test
Created detector, starting first detection
Started first test
Finished second test
Started third test
test 28: completed (0.00 sec)
test 29: initiated [groupdependence (nbhat4)]
test 29: completed (0.00 sec)
test 30: initiated [multi groupring (milind)]
test 30: completed (0.00 sec)
test 31: initiated [multi nodering (milind)]
test 31: completed (0.00 sec)
test 32: initiated [multi varsizetest (mjlang)]
test 32: completed (0.00 sec)
test 33: initiated [multi varsizetest2 (phil)]
test 33: completed (0.00 sec)
test 34: initiated [multi varraystest (milind)]
test 34: completed (0.00 sec)
test 35: initiated [multi groupcast (mjlang)]
test 35: completed (0.00 sec)
test 36: initiated [multi groupmulti (gengbin)]
test 36: completed (0.00 sec)
test 37: initiated [multi groupsectiontest (ebohm)]
test 37: completed (0.00 sec)
test 38: initiated [multi multisectiontest (ebohm)]
test 38: completed (0.11 sec)
test 39: initiated [multi nodecast (milind)]
test 39: completed (0.05 sec)
test 40: initiated [multi synctest (mjlang)]
[4] Stack Traceback:
  [4:0] pgm 0x555555812acb
  [4:1] libpthread.so.0 0x7ffff7bc88a0
  [4:2] mca_pml_ob1.so 0x7fffe56ebab4 mca_pml_ob1_recv_req_start
  [4:3] mca_pml_ob1.so 0x7fffe56da8e5 mca_pml_ob1_iprobe
  [4:4] libmpi.so.20 0x7ffff716e31a PMPI_Iprobe
  [4:5] pgm 0x55555581124b
  [4:6] pgm 0x5555558119e9 LrtsAdvanceCommunication(int)
  [4:7] pgm 0x555555811c9e CommunicationServerThread(int)
  [4:8] pgm 0x55555581205a
  [4:9] pgm 0x555555812664 ConverseInit
  [4:10] pgm 0x5555557a2bfc charm_main
  [4:11] libc.so.6 0x7ffff6794b97 __libc_start_main
  [4:12] pgm 0x55555568bb6a _start
[5] Stack Traceback:
  [5:0] pgm 0x555555812acb
  [5:1] libpthread.so.0 0x7ffff7bc88a0
  [5:2] libopen-pal.so.20 0x7ffff62bb093
  [5:3] libopen-pal.so.20 0x7ffff625f9a9 opal_progress
  [5:4] mca_pml_ob1.so 0x7fffe56da903 mca_pml_ob1_iprobe
  [5:5] libmpi.so.20 0x7ffff716e31a PMPI_Iprobe
  [5:6] pgm 0x55555581124b
  [5:7] pgm 0x5555558119e9 LrtsAdvanceCommunication(int)
  [5:8] pgm 0x555555811c9e CommunicationServerThread(int)
  [5:9] pgm 0x55555581205a
  [5:10] pgm 0x555555812664 ConverseInit
  [5:11] pgm 0x5555557a2bfc charm_main
  [5:12] libc.so.6 0x7ffff6794b97 __libc_start_main
  [5:13] pgm 0x55555568bb6a _start
------------- Processor 4 Exiting: Caught Signal ------------
Reason: Terminated
------------- Processor 5 Exiting: Caught Signal ------------
Reason: Terminated
[NemeanLion:22516] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[NemeanLion:22516] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

real	0m1.630s
user	0m3.254s
sys	0m0.046s

As an additional note on performance, pingpong_multipairs has an aberration in speed between iterations of size <= 8192 and >= 16384.

make[3]: Entering directory '/home/evan/Charmworks/charm/mpi-linux-x86_64-smp/benchmarks/converse/pingpong'
../../../bin/testrun  ./pingpong_multipairs +p4 ++ppn 2 +setcpuaffinity

Running on 2 processors:  ./pingpong_multipairs +ppn 2 +setcpuaffinity
charmrun>  /usr/bin/setarch x86_64 -R  mpirun -np 2  ./pingpong_multipairs +ppn 2 +setcpuaffinity
Charm++> Running on MPI version: 3.1
Charm++> level of thread support used: -1 (desired: 0)
Charm++> Running in SMP mode: 2 processes, 2 worker threads (PEs) + 1 comm threads per process, 4 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: v6.11.0-devel-384-g86fa91abd
Converse/Charm++ Commit ID: v6.11.0-devel-384-g86fa91abd
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
Charm++> cpu affinity enabled.
Charm++> Running on 1 hosts (1 sockets x 4 cores x 2 PUs = 8-way SMP)
Charm++> cpu topology info is gathered in 0.028 seconds.
Multiple pair send/recv
 bytes     latency(us)   bandwidth(MBytes/sec)
8     2.65   3.01
16      6.00   2.67
32      11.59   2.76
64      18.26   3.50
128     23.41   5.47
256     28.73   8.91
512     32.51   15.75
1024      37.44   27.35
2048      42.51   48.18
4096      60.71   67.47
8192      80.76   101.44
16384     2512.18   6.52
32768     6642.26   4.93
65536     6679.92   9.81
131072      6749.01   19.42
262144      6810.27   38.49
524288      7113.52   73.70
1048576     7505.03   139.72
2097152     8142.08   257.57
4194304     9566.39   438.44
[Partition 0][Node 0] End of program

real  2m16.499s
user  11m10.740s
sys 0m15.628s

Ready for review.

Should we create an issue for the tests that fail with oversubscription?

They don't fail, they just take a long time (20-35 minutes).
I will create follow-up PRs to track some of the tests that are entirely disabled.

intercomm_coll timed out. I can reproduce the same thing on multicore.

Needs one more review


Frequently starts swapping until process gets killed

Influxdb regularly starts swapping until all swap is used and then gets killed by the kernel. This happens regularly (every day) when used in conjunction with Grafana. It perhaps runs 4 queries, all time constrained with a where-clause to 1 or 2 days of secondly data. There's no heavy write activity.
System info:

InfluxDB v0.8.3 (git: fbf9a47) (leveldb: 1.15) from apt repo
Linux 3.13.0-32 64bit VPS Ubuntu
1gb of ram and 1gb of swap

I already constrained the settings to
[storage.engines.rocksdb]
max-open-files = 300
lru-cache-size = "200m"

Are there any other resources/ulimit or similar that I could constrain so this does not happen? Can I make Influxdb log more so I could attach useful information here? And should this happen at all given that I'm not querying a huge amount of data points?
Thanks

Not sure what this is, but we won't be making any more releases to the 0.8 line. This almost certainly isn't a problem in the upcoming 0.9 release.


Remove responsibility from the WG Technical Lead role

The responsibility "Succession - identifying next steps for members of the working group" isn't very clear and seems to be covered by triaging issues and roadmap alignment. I believe it makes sense to remove it from the table.
Authored-by: Angela Chin<EMAIL_ADDRESS>

Thanks, @angelachin! I think this language came in on https://github.com/knative/community/pull/56, but I don't see a corresponding explanation there either, so LGTM to remove it from our role descriptions.

Merging PR, as it has approval from a quorum of the TOC and has been open for a week without objection from any other TOC members.


[Draft] SSE implementation using AsyncContext

This is a suggestion for an implementation using AsyncContext instead of Atmosphere, as discussed with Andryi Redko on the dev cxf list.
This implementation seems to be working correctly for Aries JAX-RS Whiteboard implementation, but some tests under systests/rs-sse in CXF tests fail.
There are probably many things wrong so a review would be very much appreciated.
Thx in advance.
Carlos.

any preliminary comments on this anyone?

In general, I really like the idea.  But I'm concerned about the systest failures.  Do you have more information on why they are failing?  If they are failing because they are coded to Atmosphere-specific behaviors, then I'm less concerned (but then we should fix the tests).  I'm more concerned if this change is removing/altering the SSE behavior.

hey @andymc12 @reta,
I was not very acquainted with the test infra. I made some debugging and I found Jackson ObjectWriter to generate this expection:
	at org.apache.coyote.http11.InternalNioOutputBuffer.flushBuffer(InternalNioOutputBuffer.java:234)
	at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(InternalNioOutputBuffer.java:189)
	at org.apache.coyote.http11.InternalNioOutputBuffer.commit(InternalNioOutputBuffer.java:177)
	at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:740)
	at org.apache.coyote.Response.action(Response.java:179)
	at org.apache.coyote.Response.sendHeaders(Response.java:350)
	at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:335)
	at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:317)
	at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:110)
	at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1100)
	at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:915)
	at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:625)
	at org.apache.cxf.jaxrs.sse.OutboundSseEventBodyWriter.writePayloadTo(OutboundSseEventBodyWriter.java:139)
	at org.apache.cxf.jaxrs.sse.OutboundSseEventBodyWriter.writeTo(OutboundSseEventBodyWriter.java:112)
	at org.apache.cxf.jaxrs.sse.OutboundSseEventBodyWriter.writeTo(OutboundSseEventBodyWriter.java:40)
	at org.apache.cxf.jaxrs.sse.SseEventSinkImpl.dequeue(SseEventSinkImpl.java:124)
	at org.apache.catalina.core.AsyncContextImpl$RunnableWrapper.run(AsyncContextImpl.java:571)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)```

this seems to be what is preventing tests from passing 😕

nope... jetty tests fail in a similar way

it looks like only one test fails in TomcatEmbeddedTest and it is testBooksStreamIsBroadcasted because the OutputStream is closed when JacksonJsonProvider tries to flush. Is is possible that the test took some assumptions about the threading that were valid for atmosphere only? I am a bit in the dark here.

@csierra Thanks a lot for looking into it. No, the tests do not make any assumptions regarding threading, they are 100% black box in this regards. However, server-side is quite sensitive to threading (and thread-local proxies), we would have to look into that for sure.

hey @reta @andymc12,
tests are green on my machine now! \o/
I needed to remove the transportId from the war file for those tests to work (🤦♂ ) and I found that testBooksStreamIsBroadcasted was taking for granted that broadcaster.broadcast is synchronous, which is not anymore (I guess it was).
what should I do to have the tests run again in the pull?

hi @reta,
thank you guys for looking into this. I am happy to see that the PR is useful.

Closing in favor of https://github.com/apache/cxf/pull/415, as discussed


Best way to handle clicks on hashless URLs?

I'd like to achieve the following:

use navigation to handle SPA
use "normal" urls using history api
still have URL hints and being able to open them in new page using ctrl-click

Currently, I should be able to achieve that using:
a
  [ onPreventDefaultClick NewPage
  , href "/newpage" ]
  [ text "foobar" ]

where onNonPropagatedClick would set preventDefault to true.
Should navigation support this pattern better?
Refs https://github.com/domenkozar/hydra-frontend/issues/5

I've implemented first draft in https://github.com/domenkozar/hydra-frontend/pull/7, but it's still not perfect.

@domenkozar I made this about 3 months ago. I haven't done any elm since then but I think it's probably still mostly up-to-date.

I'm currently using the implementation suggested by @sircharleswatson above. It uses the following function:
clickTo : String -> List (Attribute Msg)
clickTo path =
  [ href path
  , onWithOptions
      "click"
      { stopPropagation = True, preventDefault = True }
      (Json.map (\_ -> Go path) Json.value)
  ]

Links can then be written as:
a (clickTo "/my/path") [ text "whatever" ]

However, ctrl-click resp. cmd-click still don't work with this implementation. Does anybody know how to make this work properly? An eventual package level solution without having to define clickTo would be even better.

I found a piece of code and transformed it using the parser.
Normal URL works and clicking as well.
Using Elm 0.18
OnClickPage.elm
onPreventDefaultClick : msg -> Attribute msg
onPreventDefaultClick message =
    onWithOptions
        "click"
        { defaultOptions | preventDefault = True }
        (Json.Decode.succeed message)

onClickPage : Page -> List (Attribute Msg)
onClickPage page =
    [ style [ ( "pointer", "cursor" ) ]
    , href (pageToUrl page)
    , onPreventDefaultClick (NewPage page)
    ]

Routing
type Page
    = NotFound
    | Home
    | Register

matchers : Parser (Page -> a) a
matchers =
    oneOf
        [ map Home top
        , map Register (s (pageToUrl Register))
        ]

parseLocation : Location -> Page
parseLocation location =
    case (parsePath matchers location) of
        Just route ->
            route

        Nothing ->
            NotFound

pageToUrl : Page -> String
pageToUrl page =
    case page of
        Home ->
            ""

        Register ->
            "register"

        NotFound ->
            "not-found"

Msg.elm
type Msg
    = NewPage Page
    | NewLocation Location

Main.elm
main : Program Never Model Msg
main =
    Navigation.program NewLocation
        { init = init
        , view = view
        , update = update
        , subscriptions = subscriptions
        }

init : Location -> ( Model, Cmd Msg )
init location =
    let
        currentRoute =
            Routing.parseLocation location
    in
        initialModel currentRoute ! []

initialModel : Routing.Page -> Model
initialModel route =
    { windowSize = { width = 0, height = 0 }
    , page = route
    }

type alias Model =
    { windowSize : Window.Size
    , page : Routing.Page
    }

update : Msg -> Model -> ( Model, Cmd Msg )
update msg model =
    case msg of
        NewLocation location ->
            let
                newRoute =
                    parseLocation location
            in
                { model | page = newRoute } ! []

        NewPage page ->
            model ! [ Navigation.newUrl (Routing.pageToUrl page) ]

@jvdvleuten does middle clicking work (meaning it opens a new tab)?

Yes it does. Middle clicking, right click and open in new tab.
The way it works is with a "normal" click we send a NewPage message which will issue a Navigation.newUrl (Routing.pageToUrl page) command. This updates the browser history and provides a new NewLocation message. See http://package.elm-lang.org/packages/elm-lang/navigation/2.0.1/Navigation#newUrl.
With a new tab click, parsing works normally as it would have been without the NewPage message.

Wonderful, I'll test this out!

That's basically the same as the solution that I posted before, except for cosmetics. So unfortunately, it still doesn't work with Cmd-click on a Mac.

I overlooked the fact you do not want to prevent the link to be opened when ctrl or command key is pressed.
The following change works at my computer using the "ctrl" key. Can you check it works using the "command" key at your mac as well?
Passing a Json.Decode.fail will open a new tab when control clicking.
onClickPage : Page -> List (Attribute Msg)
onClickPage page =
    [ style [ ( "pointer", "cursor" ) ]
    , href (pageToUrl page)
    , onPreventDefaultClick (NewPage page)
    ]

onPreventDefaultClick : msg -> Attribute msg
onPreventDefaultClick message =
    onWithOptions "click"
        { defaultOptions | preventDefault = True }
        (preventDefault2
            |> Json.Decode.andThen (maybePreventDefault message)
        )

preventDefault2 : Decoder Bool
preventDefault2 =
    Json.Decode.map2
        (invertedOr)
        (Json.Decode.field "ctrlKey" Json.Decode.bool)
        (Json.Decode.field "metaKey" Json.Decode.bool)

maybePreventDefault : msg -> Bool -> Decoder msg
maybePreventDefault msg preventDefault =
    case preventDefault of
        True ->
            Json.Decode.succeed msg

        False ->
            Json.Decode.fail "Normal link"

invertedOr : Bool -> Bool -> Bool
invertedOr x y =
    not (x || y)

Great, that works. Finally! Thanks!

Can we include that in the Navigation package? It seems like a very common requirement.

@woylie What part and how would you like this to include it in the Navigation package? I think we can include some like the onPreventDefaultClick in the Html.Events package and include onClickPage in the Html.Attribute package if it is widely used? The problem is all the interaction with routing, navigation, clicking and the message generating a command to change the URL.
Any ideas?

You're right, the Navigation package probably isn't the right place for this.
The onClickPage function isn't really that important. It also makes some assumptions (Page type, Newpage Page msg, pageToUrl Page function). If you strip it down, you'd end up with something like this:
onClickPage : Attribute msg -> String -> List (Attribute Msg)
onClickPage msg url =
    [ href url
    , onPreventDefaultClick msg
    ]

This doesn't seem very useful to me as a library function. And if you want to add some more attributes, you would have to write your links like this:
a (onClickPage (NewPage "somevalue") "somestring") ++ [target "_blank", style "btn-solid"] [text "abc"]

That's kind of ugly. I can do without that.
The onPreventDefaultClick function on the other hand would be a great addition to Html.Events. Let's move the discussion there.

I like where this ended up, great work!
I'll add a note in the README of this project pointing to the solution in https://github.com/elm-lang/html/issues/110 which seems like the way to go.
I'm working on lots of SPA related things now, so this is in the path of "batched work" for now. It may be a bit before I can merge it into HTML, but it's on the path!

Okay, note added to the readme. Thanks again for sorting through this!

I'm a bit of a noob with urls, I've almost implemented #-less urls in elm but when I copy/paste or open in a new tab I'm still having some issues when there is a path parameter.  Any ideas? Thanks!


Create the "all tickets" page

Create a page will all the tickets, without order for now with DataTables.
It needs to be loaded dynamically as to not slow down the page if the user have a great number of tickets.

On this page I want to be able do order DESC / ASC on each fields.
I want to be able to search in each fields too


Backport "HBASE-25482 Improve SimpleRegionNormalizer#getAverageRegionSizeMb (#2858)" to branch-2

Signed-off-by: Nick Dimiduk<EMAIL_ADDRESS>

This is on master and branch-2.4, but missed for branch-2.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
2m 16s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ branch-2 Compile Tests _

+1 :green_heart:
mvninstall
4m  6s
branch-2 passed

+1 :green_heart:
compile
3m 30s
branch-2 passed

+1 :green_heart:
checkstyle
1m 12s
branch-2 passed

+1 :green_heart:
spotbugs
2m  9s
branch-2 passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 36s
the patch passed

+1 :green_heart:
compile
3m 26s
the patch passed

+1 :green_heart:
javac
3m 26s
the patch passed

+1 :green_heart:
checkstyle
1m 12s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
12m 35s
Patch does not cause any errors with Hadoop 3.1.2 3.2.1.

+1 :green_heart:
spotbugs
2m 26s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 16s
The patch does not generate ASF License warnings.

44m 52s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3153/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3153

Optional Tests
dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile

uname
Linux 76abbd484e02 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 / 702220a135

Default Java
AdoptOpenJDK-1.8.0_282-b08

Max. process+thread count
95 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3153/1/console

versions
git=2.17.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
3m 59s
Docker mode activated.

-0 :warning:
yetus
0m  6s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ branch-2 Compile Tests _

+1 :green_heart:
mvninstall
4m 19s
branch-2 passed

+1 :green_heart:
compile
1m  7s
branch-2 passed

+1 :green_heart:
shadedjars
7m  0s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 43s
branch-2 passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
4m  0s
the patch passed

+1 :green_heart:
compile
1m  7s
the patch passed

+1 :green_heart:
javac
1m  7s
the patch passed

+1 :green_heart:
shadedjars
6m 46s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 40s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
132m 38s
hbase-server in the patch passed.

164m 42s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3153/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3153

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux c3cc3229c4cd 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 / 702220a135

Default Java
AdoptOpenJDK-11.0.10+9

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3153/1/testReport/

Max. process+thread count
3950 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3153/1/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 34s
Docker mode activated.

-0 :warning:
yetus
0m  8s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ branch-2 Compile Tests _

+1 :green_heart:
mvninstall
3m 44s
branch-2 passed

+1 :green_heart:
compile
0m 56s
branch-2 passed

+1 :green_heart:
shadedjars
6m  2s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 38s
branch-2 passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 24s
the patch passed

+1 :green_heart:
compile
0m 57s
the patch passed

+1 :green_heart:
javac
0m 57s
the patch passed

+1 :green_heart:
shadedjars
6m  1s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 37s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
141m  9s
hbase-server in the patch passed.

166m 27s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3153/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3153

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux f1e414b2b966 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 / 702220a135

Default Java
AdoptOpenJDK-1.8.0_282-b08

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3153/1/testReport/

Max. process+thread count
4512 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3153/1/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.


[New Feature]: Ops team executing E2E testing for R1-RC5

Checked for duplicates
Yes - I've already checked
Alternatives considered
Yes - and alternatives don't suffice
Related problems
No response
Describe the feature request
I suggest that we make sure that the ops team is able to execute the Release 1 processing with the latest stable release. To this point, sometimes in the next couple of weeks:
o Can Phil deploy R1-RC5 to the I&T venue?
o Can Lala and Sri execute the standard regression test suite, documenting the operations and result?

R1-RC5 has been deployed to I&T FWD and the system has been smoke tested. Lala will clear up the system and run regression - I will help as needed.

Sri executed the test using R1-RC5 and captured the results in this wiki page:
https://wiki.jpl.nasa.gov/pages/viewpage.action?pageId=671764746
Closing.


BUG: k-build silently doesn't work if k.config.mjs is incorrect

If the k.config.mjs file is incorrect (e.g contains a require(), which seems not to work in .mjs files), then k-build does nothing without raising any error.
Version 2.2.2 also stopped working with k.config.js (note the extension), where a warning is printed:
(node:8141 Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
(Use `node --trace-warnings ...` to show where the warning was created)

and then k-build also does nothing. The build documentation language seems to indicate this should be supported.

I'll look into why it fails silently, but the require in .mjs (or .js with a "type": "module" in your package.json) is part of the spec for es6 modules.
I'll update the documentation to be more specific about npm modules vs. commonjs packages.

Yeah I figured I was doing something incorrect with JS. For the silence, I suppose k.All() gets called with undefined options, or at least doesn't have the source directory, and thus does nothing?
If so, a simple warning that no directory to process was passed should be sufficient


fix cloud-init lock root login issue

这个 PR 实现什么功能/修复什么问题:
避免root账号被锁
是否需要 backport 到之前的 release 分支:

release/2.11.0
release/2.10.0

/area region
/cc @swordqiu @Zexi

Codecov Report

Merging #1503 into master will decrease coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master   #1503      +/-   ##
=========================================
- Coverage     7.7%   7.68%   -0.02%
=========================================
  Files         411     411
  Lines       62834   62834
=========================================
- Hits         4839    4829      -10
- Misses      57586   57595       +9
- Partials      409     410       +1

Flag
Coverage Δ

#aFlag
7.68% <100%> (-0.02%)
:arrow_down:

Impacted Files
Coverage Δ

pkg/util/cloudinit/cloudconfig.go
57.62% <100%> (ø)
:arrow_up:

pkg/appsrv/response.go
66.15% <0%> (-3.08%)
:arrow_down:

pkg/util/rbacutils/rbac.go
71.33% <0%> (-2.5%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2f7086e...d3c43fd. Read the comment docs.

/lgtm
/approve


Fix/exports icons

:tada: This PR is included in version 1.0.3 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Nagarjuna33 patch 1

@nagarjuna33 Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.


hud watermark

i was trying to understad the watermark, but cannot find a way to customized by image..  i mean.. set a image using only text.. do you know a way?
its nice to see your mark but i guess an image will be more hard to change.. also less propense to change
if you can set this issue to private i guess is better, this mod is open source but not nessesary open to lammers, you work hard in this and me try to work hard on understand you

how you want to change watermark? give me an concept

i men .. the watermark only is pure text.. much better is a forced image..

i men .. the watermark only is pure text.. much better is a forced image..

if you know a lot about the player's api, you can change the watermark at your choice.


Font Support P10K

For users with ZSH and themes like p10k it would be great to include compatible shell fonts like meslo, patched for fontawesome.
https://github.com/romkatv/powerlevel10k/blob/master/font.md

Hello @g4challenge 👋
Thank you a lot for your issue.
I'm referencing issue #60 to track progress.

I'm now working on additional font support

Hello @g4challenge 👋
Please try the latest WebSSH 15.0 Beta Release :
https://testflight.apple.com/join/QSrBK59z
P10K font have been included.
Have a nice day ☀

Many Thanks @isontheline  - works like a charm!

Hello @g4challenge 👋
Happy to hear that 😉
Thank you a lot for your feedback!


feat: add ssh key and userdata

Current dependencies on/for this PR:

main

PR #19

PR #20

PR #21   👈

This stack of pull requests is managed by Graphite.

Merge activity

Nov 9, 10:20 AM: @richardcase started a stack merge that includes this pull request via Graphite.


Opening a new window shows a prompt that GitHub extension cannot be activated

Steps to Reproduce:

Open a new window. Following prompt is shown

This is because Git extension is disabled because of workspace trust and GitHub extension is enabled (as it supported). Actual fix is to disable GitHub extension which will be taken care here - https://github.com/microsoft/vscode/issues/122448.
Did a pragmatic fix here:

Fixed the prompt in this case by providing appropriate message and no actions are shown.
In order to provide better experience OOB ie., not to show these error prompts, made GitHub extension to not to support untrusted workspaces as its dependency Git extensions does not.

@sandy081, in case that we detect that the dependency is not loaded due to workspace trust, could we be more specific in the message - Cannot activate the X extension because it depends on the Y extensions which is not supported in Restricted Mode. Could we also add an action called Manage Workspace Trust which would open the workspace trust editor?

Done

Cloning a Repository Container now fails in a new empty window (and untrusted workspaces in general): https://github.com/microsoft/vscode-remote-release/issues/5134. Is there a way to make the repository picker available when the workspace is not trusted yet?

It seems @eamodio changed back to support git & GitHub extensions to be supported in untrusted workspaces. So @chrmarti your scenario shall work now.


arch/riscv: Minor style change and text correction

Summary

Add Private Data comment block
Remove unneeded #include
Update description text of irq 10

Impact
No
Testing
CI

Conflict resolved.


GSD Request

--- GSD JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from v5.4.29 to before v5.4.168",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=86137342fd4cf52882842aa8d1318b2661f08e8a",
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=027a13973dadb64ef4f19db56c9b619ee82c3375"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "86137342fd4cf52882842aa8d1318b2661f08e8a",
      "note": "introduced"
    },
    {
      "type": "commit",
      "value": "027a13973dadb64ef4f19db56c9b619ee82c3375",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "net/packet: rx_owner_map depends on pg_vec\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v5.4.168 by commit 027a13973dadb64ef4f19db56c9b619ee82c3375, it was introduced in version v5.4.29 by commit 86137342fd4cf52882842aa8d1318b2661f08e8a. For more details please see the references link."
}
--- GSD JSON ---

This issue has been assigned GSD-2021-1002676


v3 programmatic interface

This issue is to propose a programmatic interface that enables complementary services to the use of annotations (JAX-RS 2.0, OpenAPI v3, or both).
We'll have a single interface (for simplicity) that encapsulates a few different scenarios.  Here's a mock of what we're thinking about:
interface OpenAPIController

bootstrap(OpenAPI)
beforeScan(OpenAPI, Config)
afterScan (OpenAPI)

Where:

bootstrap would be used for populating the entire OpenAPI model, and avoiding annotating scanning
beforeScan would be used for partially populating a OpenAPI model before an annotation scan starts.  The optional Config obj would encapsulate things like a filter, and scanning options such as which packages / classes to scan, etc.
afterScan would be used for manipulating the OpenAPI model after it was populated via annotations.

After more discussion, we think it's better to further simplify the interface.
interface OpenAPIController

OpenAPIConfig bootstrap();

where bootstrap() can be used to return an OpenAPIConfig object which encapsulates all configurations (for Reader, Scanner, OpenAPI, etc) along with a predefined OpenAPI definition (optional) to start-off (similar to beforeScan in v2).
OpenAPIConfig class will have getters and setters for configurable items:
getOpenAPI()
setOpenAPI(OpenAPI)
getClasses()
setClasses(Set<Class<?>>)
isScan()
setScan(boolean)
getFilterClass()
setFilterClass(String)
isPrettyPrint()
setPrettyPrint(boolean)
getIgnoredRoutes()
setIgnoredRoutes(Collection)
.....
.....
All setters will return the OpenAPIConfig back to allow users to set more configurations easily.
OpenAPIConfig should ideally only use exposed classes / interfaces and avoid using internal / heavy-lift classes (more dependency),  so it can be exposed to users easily.
Any other methods that needs to be invoked by Reader/Scanner can be added to OpenAPIController. Dropped 'afterScan' method, since it seems users can instead use the OpenAPIConfig to control the generated OpenAPI definition (by setting which packages / classes to scan / ignore) instead of overwriting the generated definition. @fehguy your thoughts?

A sample implementation of OpenAPIController :
public class OpenAPIControllerImpl implements OpenAPIController {

	@Override
	public OpenAPIConfig bootstrap() {
		return new OpenAPIConfig()
				.setOpenAPI(new OpenAPI()
						.info(new Info()
							.contact(new Contact()
<EMAIL_ADDRESS>			                        	.name("Sample Name")
			                        	.url("https://foo.bar")))
						.externalDocs(new ExternalDocumentation()
							.description("read more here")
							.url("http://swagger.io"))
						.addTagsItem(new Tag()
							.name("sample tag")
							.description("all about neat things")))
				.setFilterClass("com.sample.FilterClassImpl")
				.setScan(true)
				.setPrettyPrint(true);
	}
}

Hi, looks good to me.  For scanning, would you expect that the setClasses method would be used to provide a list of classes which should be scanned, and some "other helper" would determine that list of classes independently?  So, a web.xml reader, for example, would scan for the appropriate classes (either by Swagger annotations or other), then call this configuration utility?
I think the only thing missing to me is how we trigger such logic.  In the programmatic case, this should be sufficient, as long as we have a mechanism to provide packages or classes, by names (strings) for scanning.

The OpenAPIConfig will consolidate all configurations that can be specified by users. It's sole purpose would be to get the user settings to control the generated OpenAPI definition. We'll use default values for configurations not set by users.
The list of methods mentioned above are from existing interfaces (Scanner, ReaderConfig, SwaggerConfig, etc). We can modify the methods to make it easier for users to pass in the configuration values. I was thinking that the 'setClasses' method would be used by users to provide the classes to scan, so instead of list of Classes, we can accept packages and/or classes by names (strings) for scanning. If user didn't specify the classes, then we'll scan all.
Similar to ReaderListener in v2, we will find the user's implementation of OpenAPIController and invoke the bootstrap() method to get the OpenAPIConfig and then use its settings to scan and generate OpenAPI definition. In v2, I believe the implementations of ReaderListener had to be annotated in order for it to be found and its methods to be invoked (I could be wrong), but we should find the implementation of OpenAPIController even if it doesn't have any annotations. Unlike ReaderListener, which could have multiple implementations, we should only accept one implementation of OpenAPIController.

Update:
interface OpenAPIController
OpenAPIConfig bootstrap(Map<String,Object> environment);
interface OpenAPIConfig
getOpenAPI()
setOpenAPI(OpenAPI)
getClasses()
setClasses(Set<Class<?>>)
isScan()
setScan(boolean)
getFilterClass()
setFilterClass(String)
getUserDefinedOptions()
setUserDefinedOptions(Map<String,Object>
interface OpenAPIFilter
-> OAS version of https://github.com/swagger-api/swagger-core/blob/master/modules/swagger-core/src/main/java/io/swagger/core/filter/SwaggerSpecFilter.java


Please make a Django 4.0 compatible release

Apologies if this isn't the appropriate communication channel - I know as a maintainer that this sort of issue can be unwanted noise...
Please would you consider putting out a 2.1 release with the Django 4.0 compatibility fixes that are now in master? I'm really hoping that Wagtail 2.16 (due for release early February) will be able to offer formal Django 4.0 support, and currently django-taggit is the last remaining blocker for that.
Looking at issues / PRs here it's now been two weeks with no further compatibility issues reported, so I think we're confident as we'll ever be that a new release is good to go. Happy to assist if there's any way I can.

@gasman that shouldn't be a problem, I understand the pain of waiting around for this kind of stuff. I'll see to prepping a release on Monday morning (JST), please reach out to me if this didn't happen by Tuesday

Thanks, much appreciated!

@gasman release is now out, 2.1.0 on pypi, hope that resolves stuff for you


WebApplicationFactory and IWebHostBuilder are causing deadlocks

We have a lot of tests which are written in a way described here:
https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-3.1#basic-tests-with-the-default-webapplicationfactory
Sometimes WebApplicationFactory cause a deadlock while running under xUnit.
Is your feature request related to a problem? Please describe.
Following the example in the documentation here and all my interfaces are async. Hence, I have to do this:
 protected override void ConfigureWebHost(IWebHostBuilder builder)
        {
            builder.ConfigureTestServices(services =>
            {
                SeedTestData(services).GetAwaiter().GetResult();
            });
        }

Describe the solution you'd like
builder.ConfigureTestServices could be async friendly.
Additional context
xunit integration testing
asp.net core 2.2.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 77889d43-fedb-f8b4-8fea-379ec946555c
Version Independent ID: f499b131-8b14-7e10-1728-543fdd5a5656
Content: Integration tests in ASP.NET Core
Content Source: aspnetcore/test/integration-tests.md
Product: aspnet-core
Technology: aspnetcore-test
GitHub Login: @guardrex
Microsoft Alias: riande

This is a doc issue that we need to clean up.

Ok, now it clear - docs are not up to date.
Thank you.


Fix retain and Home Assistant battery icon

Issue #1:
Just noticed that if my mailbox is closed (payload "empty") and i restart Home Assistant, the MQTT Broker apparently resends retained message again, which was "open".

Quick fix is to include retain: true into your automation, but maybe there is a way to not retain messages at all, so that after restart the status would not change. I'm thinking of trying to remove the retain flag in the Gateway's client.publish method, because now after restart it's resent to HA, which is admittedly much better than before:

Issue #2:
So i got a list of batteries to control with default entities card. Noticed mailbox with 77% showed 100% icon.

Compared with other sensor's autodiscovery payload and removed the icon. That fixes auto-icon from HA.

I did something similar but i put a delay of 30 seconds before publishing the empty payload. otherwise, i got a load of notifications in rapid succession.

I did something similar but i put a delay of 30 seconds before publishing the empty payload. otherwise, i got a load of notifications in rapid succession.

you mean the retain issue... i knew i messed two PRs up 🤣 but let's discuss it. yes, i'm with reed switch, so i got only 2 messages (because board sends them twice), so doing the 3 second trigger is enough for me.
from: "off"
to: "on"
for:
  seconds: 3

Motion Detection may trigger more often, i suppose.

I did something similar but i put a delay of 30 seconds before publishing the empty payload. otherwise, i got a load of notifications in rapid succession.

yes, i'm with reed switch, so i got only 2 messages (because board sends them twice), so doing the 3 second trigger is enough for me.
from: "off"
to: "on"
for:
  seconds: 3

Motion Detection may trigger more often, i suppose.

Ah yes because the PIR will trigger while motion continues unlike read switch which will only signal once I guess.

Thanks


update Soccer Mens : add Gold Cup, rename CC Champ League

Description
Add CONCACAF Gold Cup tournament
rename CONCACAF Champions League to Champions Cup (announced June 15 2023)
Copilot

🤖 Generated by Copilot at ffaf8d3
Summary
🏆🌎📝

The soccermens app adds support for two new CONCACAF competitions and fixes a naming inconsistency. It updates the VERSION, leagueShortNames, and leagueOptions variables in soccermens.star.

Sing, O Muse, of the skillful coder who updated
The soccermens app, the source of joy for many fans,
Who added two new contests of the brave CONCACAF
And fixed the league names, pleasing the eagle-eyed clans.

Walkthrough

Update app version to reflect new features (link)
Add support for CONCACAF Gold Cup and CONCACAF Champions Cup competitions (link, link)
Rename CONCACAF Champions League to CONCACAF Champions Cup to match official name (link)

:shipit:    baby...

:shipit: