Otter failure on PSycloneBench example with taskloop/taskwait.

Hi again,
I'm testing otter on some generated example code for PSyclone, and feel like I must have done something wrong.
The execution is handled by:
OMP_NUM_THREADS=4 OMP_TOOL_LIBRARIES=/home/achalk/otter/lib/libotter.so ./nemolite2d.exe

The main bulk of the code (Fortran, though this shouldn't cause issues based on previous experience) is covered by:
FUNCTION invoke(...)
!$omp parallel private(i, j)
!$omp single
!$omp taskloop grainsize(32)
 ....double loop
!$omp end taskloop
...repeat taskloops & occasional taskwaits (though currently these aren't needed but groups should be added soon)
!$omp end single
!$omp end parallel
END FUNCTION

this function is just called every timestep.
When I run Otter on the resultant trace:
python3 -i -m otter -i trace/otter_trace.2917929/otter_trace.2917929.otf2

I get some errors in the OTF2_GlobalEvtReaderCallback function, but I'm not clear why.
loading OTF2 anchor file: trace/otter_trace.2917929/otter_trace.2917929.otf2
generating chunks from event stream...
yielding chunks: An unhandled python exception has occurred in an OTF2_GlobalEvtReaderCallback:
Traceback (most recent call last):
  File "/home/achalk/.INSTALL/otf2/lib/python3.6/site-packages/_otf2/GlobalEvtReaderCallbacks.py", line 50, in wrapper
    ret = func(location, time, py_userData, attribute_list, *args)
  File "/home/achalk/.INSTALL/otf2/lib/python3.6/site-packages/otf2/event_reader.py", line 143, in _enter
    self._append(events.Enter, *args)
  File "/home/achalk/.INSTALL/otf2/lib/python3.6/site-packages/otf2/event_reader.py", line 18, in _append
    event = event_type._construct(self.reader.definitions, time, *args)
  File "/home/achalk/.INSTALL/otf2/lib/python3.6/site-packages/otf2/events.py", line 106, in _construct
    return cls(time, registry.regions[region])
  File "/home/achalk/.INSTALL/otf2/lib/python3.6/site-packages/otf2/registry.py", line 120, in __getitem__
    return self._elements_by_ref[ref]
KeyError: 19
Traceback (most recent call last):
  File "/home/achalk/.INSTALL/python-3.9.0/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/achalk/.INSTALL/python-3.9.0/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/achalk/otter/src/python/otter/__main__.py", line 296, in <module>
    main()
  File "/home/achalk/otter/src/python/otter/__main__.py", line 44, in main
    items = zip(*results)
  File "/home/achalk/otter/src/python/otter/__main__.py", line 43, in <genexpr>
    results = (process_chunk(chunk, verbose=args.verbose) for chunk in yield_chunks(tr))
  File "/home/achalk/otter/src/python/otter/trace.py", line 135, in yield_chunks
    for location, event in tr.events:
  File "/home/achalk/.INSTALL/otf2/lib/python3.6/site-packages/otf2/event_reader.py", line 46, in __iter__
    read_events = _otf2.GlobalEvtReader_ReadEvents(self._handle, self.batch_events)
  File "/home/achalk/.INSTALL/otf2/lib/python3.6/site-packages/_otf2/GlobalEvtReader.py", line 40, in GlobalEvtReader_ReadEvents
    c_ReadEvents(reader, recordsToRead, ctypes.byref(recordsRead))
  File "/home/achalk/.INSTALL/otf2/lib/python3.6/site-packages/_otf2/ErrorCodes.py", line 166, in HandleErrorCode
    raise Error(errcode)
_otf2.ErrorCodes.Error: INTERRUPTED_BY_CALLBACK (92): Record reading interrupted by reader callback

When I just use  otf2-print trace/otter_trace.2917929/otter_trace.2917929.otf2 > dump I get some warnings from otf2-print:
otf2-print: warning: out of order Region definition: "Parallel Region 0" <1>
otf2-print: warning: out of order Region definition: "implicit task 1" <2>
otf2-print: warning: out of order Region definition: "barrier_implicit" <4>
otf2-print: warning: out of order Region definition: "implicit task 4" <7>
otf2-print: warning: out of order Region definition: "barrier_implicit" <9>
otf2-print: warning: out of order Region definition: "implicit task 2" <3>
otf2-print: warning: out of order Region definition: "barrier_implicit" <5>
otf2-print: warning: out of order Region definition: "implicit task 3" <6>
otf2-print: warning: out of order Region definition: "Parallel Region 1" <10>
....

Is this again me failing on something? The trace is available here: https://drive.google.com/file/d/1RmYyiXXC0GOkUIYiIOwjHHBkNea-NYX2/view?usp=sharing

Hi Aidan,
Thanks for letting me know about this. I wonder if there is a region definition missing from the trace somehow. I think the warning about out of order region definitions is a red herring - I think that happens because OTF2 wants the region definitions to be recorded in order of their monotonically increasing integer ID (this is a separate and less critical bug to that of possibly missing definitions).
It looks like you only sent me part of the trace - I will need the contents of the folder which contains the .otf2 file please, as that's just the anchor file. The events and defnitions are kept in a subfolder, 1 file of each per thread. The folder structure looks like this:
❯ l -R
.:
total 12K
drwxrwxr-x 2 adam adam 4.0K Sep  3 17:04 otter_trace.16331
-rw-rw-r-- 1 adam adam 2.6K Sep  3 17:04 otter_trace.16331.def    <-- global definitions
-rw-rw-r-- 1 adam adam   72 Sep  3 17:04 otter_trace.16331.otf2   <-- anchor file

./otter_trace.16331:
total 16K
-rw-rw-r-- 1 adam adam  20 Sep  3 17:04 0.def   <-- thread-local definitions
-rw-rw-r-- 1 adam adam 688 Sep  3 17:04 0.evt   <-- thread-local events
-rw-rw-r-- 1 adam adam  20 Sep  3 17:04 1.def
-rw-rw-r-- 1 adam adam 493 Sep  3 17:04 1.evt

I've tar.gzd the whole trace folder now, so we should get everything now: https://drive.google.com/file/d/1RmYyiXXC0GOkUIYiIOwjHHBkNea-NYX2/view?usp=sharing - it might be the same link (as google drive replaced the file) - if anything is still missing let me know :)

Looks like there are a lot of missing definitions (the 2nd column should increase by 1 for each new definition):
❯ /opt/otf2/bin/otf2-print -A otter_trace.2917929/otter_trace.2917929.otf2 2>&1 | grep "Region: INVALID" | wc -l
30393

❯ /opt/otf2/bin/otf2-print -G otter_trace.2917929/otter_trace.2917929.otf2 2>/dev/null | grep -P "^REGION" | sort -n -k 2
REGION                                 0  Name: "initial task 0" <115> (Aka. "" <0>), Descr.: "" <0>, Role: TASK, Paradigm: OPENMP, Flags: NONE, File: "" <0>, Begin: 0, End: 0
REGION                                 1  Name: "Parallel Region 0" <105> (Aka. "" <0>), Descr.: "" <0>, Role: PARALLEL, Paradigm: OPENMP, Flags: NONE, File: "" <0>, Begin: 0, End: 0
REGION                                 2  Name: "implicit task 1" <106> (Aka. "" <0>), Descr.: "" <0>, Role: TASK, Paradigm: OPENMP, Flags: NONE, File: "" <0>, Begin: 0, End: 0I don't think this is your fault, it's almost certainly something I've overlooked in how the region definitions are handled. To put it briefly, each thread gathers up definitions as it goes and should hand them off to a parallel region at a _parallel-end_ event. The last thread out of the parallel region is supposed to write them to the trace. Something is going wrong with this in the specific case of your program, probably there's some combination of regions I didn't account for.
 Each thread gathers up definitions as it goes and should hand them off to a parallel region at a parallel-end event. The last thread out of the parallel region is supposed to write them to the trace. Something is going wrong with this in the specific case of your program, probably there's some combination of regions I didn't account for.
It is odd to me that there appears to be missing parallel-end events (I'd expect the number of parallel-begin and parallel-end to always match):
❯ /opt/otf2/bin/otf2-print -A otter_trace.2917929/otter_trace.2917929.otf2 2>&1 | grep "\"parallel\" <77>" | cut -d, -f 4 | sort | uniq -c
    408  ("event_type" <10>; STRING; "parallel_begin" <56>)
    309  ("event_type" <10>; STRING; "parallel_end" <57>)

Would you be able to do the following for me please:

see if the issue is replicated with the smallest possible problem e.g. a single timestep.
re-compile otter with make clean && DEBUG_OTTER=3 make and capture the output to stderr when you re-run it - you should get lots of messages about the callbacks that are triggered

This should help me get some insight into what is happening.

Number of parallel-begin events for each unique parallel region:
❯ /opt/otf2/bin/otf2-print -A otter_trace.2917929/otter_trace.2917929.otf2 2>&1 | grep "\"parallel_begin\"" | cut -d, -f 6 | sort | uniq -c
      1 STRING                                56  "parallel_begin"
      4  ("unique_id" <0>; UINT64; 0)
      4  ("unique_id" <0>; UINT64; 1)
      4  ("unique_id" <0>; UINT64; 10)
      4  ("unique_id" <0>; UINT64; 100)
      4  ("unique_id" <0>; UINT64; 101)
      4  ("unique_id" <0>; UINT64; 11)
      4  ("unique_id" <0>; UINT64; 12)
      4  ("unique_id" <0>; UINT64; 13)
      4  ("unique_id" <0>; UINT64; 14)
      4  ("unique_id" <0>; UINT64; 15)
      4  ("unique_id" <0>; UINT64; 16)
      4  ("unique_id" <0>; UINT64; 17)
      4  ("unique_id" <0>; UINT64; 18)
      4  ("unique_id" <0>; UINT64; 19)
      4  ("unique_id" <0>; UINT64; 2)
      4  ("unique_id" <0>; UINT64; 20)
      4  ("unique_id" <0>; UINT64; 21)
      4  ("unique_id" <0>; UINT64; 22)
      4  ("unique_id" <0>; UINT64; 23)
      4  ("unique_id" <0>; UINT64; 24)
      4  ("unique_id" <0>; UINT64; 25)
      4  ("unique_id" <0>; UINT64; 26)
      4  ("unique_id" <0>; UINT64; 27)
      4  ("unique_id" <0>; UINT64; 28)
      4  ("unique_id" <0>; UINT64; 29)
      4  ("unique_id" <0>; UINT64; 3)
      4  ("unique_id" <0>; UINT64; 30)
      4  ("unique_id" <0>; UINT64; 31)
      4  ("unique_id" <0>; UINT64; 32)
      4  ("unique_id" <0>; UINT64; 33)
      4  ("unique_id" <0>; UINT64; 34)
      4  ("unique_id" <0>; UINT64; 35)
      4  ("unique_id" <0>; UINT64; 36)
      4  ("unique_id" <0>; UINT64; 37)
      4  ("unique_id" <0>; UINT64; 38)
      4  ("unique_id" <0>; UINT64; 39)
      4  ("unique_id" <0>; UINT64; 4)
      4  ("unique_id" <0>; UINT64; 40)
      4  ("unique_id" <0>; UINT64; 41)
      4  ("unique_id" <0>; UINT64; 42)
      4  ("unique_id" <0>; UINT64; 43)
      4  ("unique_id" <0>; UINT64; 44)
      4  ("unique_id" <0>; UINT64; 45)
      4  ("unique_id" <0>; UINT64; 46)
      4  ("unique_id" <0>; UINT64; 47)
      4  ("unique_id" <0>; UINT64; 48)
      4  ("unique_id" <0>; UINT64; 49)
      4  ("unique_id" <0>; UINT64; 5)
      4  ("unique_id" <0>; UINT64; 50)
      4  ("unique_id" <0>; UINT64; 51)
      4  ("unique_id" <0>; UINT64; 52)
      4  ("unique_id" <0>; UINT64; 53)
      4  ("unique_id" <0>; UINT64; 54)
      4  ("unique_id" <0>; UINT64; 55)
      4  ("unique_id" <0>; UINT64; 56)
      4  ("unique_id" <0>; UINT64; 57)
      4  ("unique_id" <0>; UINT64; 58)
      4  ("unique_id" <0>; UINT64; 59)
      4  ("unique_id" <0>; UINT64; 6)
      4  ("unique_id" <0>; UINT64; 60)
      4  ("unique_id" <0>; UINT64; 61)
      4  ("unique_id" <0>; UINT64; 62)
      4  ("unique_id" <0>; UINT64; 63)
      4  ("unique_id" <0>; UINT64; 64)
      4  ("unique_id" <0>; UINT64; 65)
      4  ("unique_id" <0>; UINT64; 66)
      4  ("unique_id" <0>; UINT64; 67)
      4  ("unique_id" <0>; UINT64; 68)
      4  ("unique_id" <0>; UINT64; 69)
      4  ("unique_id" <0>; UINT64; 7)
      4  ("unique_id" <0>; UINT64; 70)
      4  ("unique_id" <0>; UINT64; 71)
      4  ("unique_id" <0>; UINT64; 72)
      4  ("unique_id" <0>; UINT64; 73)
      4  ("unique_id" <0>; UINT64; 74)
      4  ("unique_id" <0>; UINT64; 75)
      4  ("unique_id" <0>; UINT64; 76)
      4  ("unique_id" <0>; UINT64; 77)
      4  ("unique_id" <0>; UINT64; 78)
      4  ("unique_id" <0>; UINT64; 79)
      4  ("unique_id" <0>; UINT64; 8)
      4  ("unique_id" <0>; UINT64; 80)
      4  ("unique_id" <0>; UINT64; 81)
      4  ("unique_id" <0>; UINT64; 82)
      4  ("unique_id" <0>; UINT64; 83)
      4  ("unique_id" <0>; UINT64; 84)
      4  ("unique_id" <0>; UINT64; 85)
      4  ("unique_id" <0>; UINT64; 86)
      4  ("unique_id" <0>; UINT64; 87)
      4  ("unique_id" <0>; UINT64; 88)
      4  ("unique_id" <0>; UINT64; 89)
      4  ("unique_id" <0>; UINT64; 9)
      4  ("unique_id" <0>; UINT64; 90)
      4  ("unique_id" <0>; UINT64; 91)
      4  ("unique_id" <0>; UINT64; 92)
      4  ("unique_id" <0>; UINT64; 93)
      4  ("unique_id" <0>; UINT64; 94)
      4  ("unique_id" <0>; UINT64; 95)
      4  ("unique_id" <0>; UINT64; 96)
      4  ("unique_id" <0>; UINT64; 97)
      4  ("unique_id" <0>; UINT64; 98)
      4  ("unique_id" <0>; UINT64; 99)

And for parallel-end:
❯ /opt/otf2/bin/otf2-print -A otter_trace.2917929/otter_trace.2917929.otf2 2>&1 | grep "\"parallel_end\"" | cut -d, -f 6 | sort | uniq -c
      1 STRING                                57  "parallel_end"
      4  ("unique_id" <0>; UINT64; 0)
      4  ("unique_id" <0>; UINT64; 1)
      3  ("unique_id" <0>; UINT64; 10)
      3  ("unique_id" <0>; UINT64; 100)
      4  ("unique_id" <0>; UINT64; 101)
      3  ("unique_id" <0>; UINT64; 11)
      3  ("unique_id" <0>; UINT64; 12)
      3  ("unique_id" <0>; UINT64; 13)
      3  ("unique_id" <0>; UINT64; 14)
      3  ("unique_id" <0>; UINT64; 15)
      3  ("unique_id" <0>; UINT64; 16)
      3  ("unique_id" <0>; UINT64; 17)
      3  ("unique_id" <0>; UINT64; 18)
      3  ("unique_id" <0>; UINT64; 19)
      3  ("unique_id" <0>; UINT64; 2)
      3  ("unique_id" <0>; UINT64; 20)
      3  ("unique_id" <0>; UINT64; 21)
      3  ("unique_id" <0>; UINT64; 22)
      3  ("unique_id" <0>; UINT64; 23)
      3  ("unique_id" <0>; UINT64; 24)
      3  ("unique_id" <0>; UINT64; 25)
      3  ("unique_id" <0>; UINT64; 26)
      3  ("unique_id" <0>; UINT64; 27)
      3  ("unique_id" <0>; UINT64; 28)
      3  ("unique_id" <0>; UINT64; 29)
      3  ("unique_id" <0>; UINT64; 3)
      3  ("unique_id" <0>; UINT64; 30)
      3  ("unique_id" <0>; UINT64; 31)
      3  ("unique_id" <0>; UINT64; 32)
      3  ("unique_id" <0>; UINT64; 33)
      3  ("unique_id" <0>; UINT64; 34)
      3  ("unique_id" <0>; UINT64; 35)
      3  ("unique_id" <0>; UINT64; 36)
      3  ("unique_id" <0>; UINT64; 37)
      3  ("unique_id" <0>; UINT64; 38)
      3  ("unique_id" <0>; UINT64; 39)
      3  ("unique_id" <0>; UINT64; 4)
      3  ("unique_id" <0>; UINT64; 40)
      3  ("unique_id" <0>; UINT64; 41)
      3  ("unique_id" <0>; UINT64; 42)
      3  ("unique_id" <0>; UINT64; 43)
      3  ("unique_id" <0>; UINT64; 44)
      3  ("unique_id" <0>; UINT64; 45)
      3  ("unique_id" <0>; UINT64; 46)
      3  ("unique_id" <0>; UINT64; 47)
      3  ("unique_id" <0>; UINT64; 48)
      3  ("unique_id" <0>; UINT64; 49)
      3  ("unique_id" <0>; UINT64; 5)
      3  ("unique_id" <0>; UINT64; 50)
      3  ("unique_id" <0>; UINT64; 51)
      3  ("unique_id" <0>; UINT64; 52)
      3  ("unique_id" <0>; UINT64; 53)
      3  ("unique_id" <0>; UINT64; 54)
      3  ("unique_id" <0>; UINT64; 55)
      3  ("unique_id" <0>; UINT64; 56)
      3  ("unique_id" <0>; UINT64; 57)
      3  ("unique_id" <0>; UINT64; 58)
      3  ("unique_id" <0>; UINT64; 59)
      3  ("unique_id" <0>; UINT64; 6)
      3  ("unique_id" <0>; UINT64; 60)
      3  ("unique_id" <0>; UINT64; 61)
      3  ("unique_id" <0>; UINT64; 62)
      3  ("unique_id" <0>; UINT64; 63)
      3  ("unique_id" <0>; UINT64; 64)
      3  ("unique_id" <0>; UINT64; 65)
      3  ("unique_id" <0>; UINT64; 66)
      3  ("unique_id" <0>; UINT64; 67)
      3  ("unique_id" <0>; UINT64; 68)
      3  ("unique_id" <0>; UINT64; 69)
      3  ("unique_id" <0>; UINT64; 7)
      3  ("unique_id" <0>; UINT64; 70)
      3  ("unique_id" <0>; UINT64; 71)
      3  ("unique_id" <0>; UINT64; 72)
      3  ("unique_id" <0>; UINT64; 73)
      3  ("unique_id" <0>; UINT64; 74)
      3  ("unique_id" <0>; UINT64; 75)
      3  ("unique_id" <0>; UINT64; 76)
      3  ("unique_id" <0>; UINT64; 77)
      3  ("unique_id" <0>; UINT64; 78)
      3  ("unique_id" <0>; UINT64; 79)
      3  ("unique_id" <0>; UINT64; 8)
      3  ("unique_id" <0>; UINT64; 80)
      3  ("unique_id" <0>; UINT64; 81)
      3  ("unique_id" <0>; UINT64; 82)
      3  ("unique_id" <0>; UINT64; 83)
      3  ("unique_id" <0>; UINT64; 84)
      3  ("unique_id" <0>; UINT64; 85)
      3  ("unique_id" <0>; UINT64; 86)
      3  ("unique_id" <0>; UINT64; 87)
      3  ("unique_id" <0>; UINT64; 88)
      3  ("unique_id" <0>; UINT64; 89)
      3  ("unique_id" <0>; UINT64; 9)
      3  ("unique_id" <0>; UINT64; 90)
      3  ("unique_id" <0>; UINT64; 91)
      3  ("unique_id" <0>; UINT64; 92)
      3  ("unique_id" <0>; UINT64; 93)
      3  ("unique_id" <0>; UINT64; 94)
      3  ("unique_id" <0>; UINT64; 95)
      3  ("unique_id" <0>; UINT64; 96)
      3  ("unique_id" <0>; UINT64; 97)
      3  ("unique_id" <0>; UINT64; 98)
      3  ("unique_id" <0>; UINT64; 99)

Observation: Most parallel regions have 4 enter events but only 3 leave events - somehow there seems to always be a thread that fails to record a parallel-end event, which would mean the definitions accumulated within that parallel region would never get written out.

I'm not sure if this is even possible, but based on what I've seen in the trace I suspect you might have a sequence of nested parallel regions inside your main parallel region which are missing their !$omp end parallel statement... but I'd be very surprised if this could be compiled as a legal OpenMP program.

Do you by any chance have this combined construct in your code: https://www.openmp.org/spec-html/5.0/openmpsu63.html#x90-3040002.13.1
It looks like you are actually allowed to omit the !$omp end parallel do when using this construct... which I suspect means that the parallel-end event is never actually triggered in the runtime, despite the fact that a parallel region does infact end.
To which runtime do you link your Fortran code?

No, the only constructs are parallel, single, taskloop and taskwait, and all contain the closing statements. I also don't think omitting the closing statements should stop the runtime end events triggering.
I've done all that, this time its a 64x64 (the minimum multi-task loop-size per loop) and runs 2 iterations (i think? It might be 1, i'm not totally familiar with the input file), Its still failing here. I've placed the file output in the .tar.gz which contains stderr and stdout (including the programs stdout), and is here: https://drive.google.com/file/d/1myQp_CRjeRfUCbZiVv_NlSPSVNTGXjhY/view?usp=sharing
The same issue is here:
[achalk@glados trace]$ otf2-print -A otter_trace.3061254/otter_trace.3061254.otf2 2>&1 | grep "\"parallel\" <77>" | cut -d, -f 4 | sort | uniq -c
     16  ("event_type" <10>; STRING; "parallel_begin" <56>)
     15  ("event_type" <10>; STRING; "parallel_end" <57>)

The runtime is linked to the llvm openmp runtime:
[achalk@glados nemolite2d_ompt]$ ldd nemolite2d.exe
        linux-vdso.so.1 (0x00007ffe3677a000)
        libomp.so => /home/achalk/.INSTALL/llvm/lib/libomp.so (0x00007f10843af000)
        libgfortran.so.4 => /apps/packages/compilers/gcc/7.3.0/lib64/libgfortran.so.4 (0x00007f10846ff000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f108402d000)
        libgcc_s.so.1 => /apps/packages/compilers/gcc/7.3.0/lib64/libgcc_s.so.1 (0x00007f1084014000)
        libquadmath.so.0 => /apps/packages/compilers/gcc/7.3.0/lib64/libquadmath.so.0 (0x00007f1083fd2000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1083db2000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f10839f0000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f10837e7000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f10835e3000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f10846af000)

I also don't think omitting the closing statements should stop the runtime end events triggering.

I agree, I expect they should still be triggered. I was just considering the possibility that they might not. Regardless, the error definitely appears to be mine. The parallel-end callback is being triggered, but for whatever reason no thread is writing the definitions as intended.
What would I need to do to run this on my own machine? I'd like to step through with gdb as something very fishy is going on.

If it's not possible or feasible for me to run this myself, more debug output could help:
make clean && DEBUG_TRACE=3 DEBUG_OTTER=3 make

Ok - so the code is generated by (so needs installed) PSyclone https://github.com/stfc/PSyclone - the current master branch should be sufficient.
The actual code is available through https://github.com/stfc/PSycloneBench - you'd need the psykal_taskloop branch.
Once PSyclone is installed, and you've downloaded the benchmark repository you'll need to setup the environment which is not included in the repository at the moment (because it doesn't quite fit into things). The code doesn't build with flang, so I use gcc and link to the llvm runtime, so I'll post that from my laptop in a second.

Thanks! I shall give this a try and report back.

Progress update:
Stepping through in gdb I've observed that the stack which tracks a thread's active regions becomes corrupted. This is causing certain leave events to be recorded against the wrong region definitions - in particular, this means parallel-end events are not recognised correctly and so region definitions are not getting written to the trace. I'm 80% confident this is down to the way I handle task-switching.
While I'm still looking for the exact cause & fix, I've added asserts to all region-leave events to make sure the correct type of region is being used for the event. At least that way the error will be immediately obvious, which is preferable to a silent failure and corrupted trace.
The asserts are added in c530896. Use make NDEBUG="" to enable them, they're disabled by default (I will at some point get round to a proper CMake-based build system to avoid hacks with variables like this).
I'll let you know when I have a fix.

What fortran compiler+version do you use to compile the nemolite2d.exe benchmark?

GCC 7.3 - i could probably use a later on if that would help?
I think there is a C version but without the task stuff - I can see if its possible to generate a C version or not tomorrow.

A C version without the task stuff would be very helpful, hopefully it has all the other OMP constructs (e.g. the single).
See my updated post above - I think I've found an underlying problem that causes the region stack to become corrupted.

I also intend to check whether I observe missing events with Flang (my guess is the results will be the same as they were with Flang i.e. no missing events)

This script contains a MWE: https://gist.github.com/adamtuft/630b13fb5db2810c115b3591535350a6
The two trace files produced by this example are different: the one from the C executable looks good:
❯ /opt/otf2/bin/otf2-print -L 0 -A trace/otter_trace.62370/otter_trace.62370.otf2 2>&1 | grep "ENTER\|LEAVE"
ENTER                                      0<PHONE_NUMBER>56588  Region: "initial task 0" <0>
ENTER                                      0<PHONE_NUMBER>01916  Region: "Parallel Region 0" <1>
ENTER                                      0<PHONE_NUMBER>46444  Region: "implicit task 1" <2>
ENTER                                      0<PHONE_NUMBER>71505  Region: "single_executor" <3>
LEAVE                                      0<PHONE_NUMBER>95309  Region: "single_executor" <3>
LEAVE                                      0<PHONE_NUMBER>15357  Region: "implicit task 1" <2>
LEAVE                                      0<PHONE_NUMBER>29652  Region: "Parallel Region 0" <1>
LEAVE                                      0<PHONE_NUMBER>91351  Region: "initial task 0" <0>

The one from the Fortran executable is missing an event:
❯ /opt/otf2/bin/otf2-print -L 0 -A trace/otter_trace.62378/otter_trace.62378.otf2 2>&1 | grep "ENTER\|LEAVE"
ENTER                                      0<PHONE_NUMBER>24254  Region: "initial task 0" <0>
ENTER                                      0<PHONE_NUMBER>02210  Region: "Parallel Region 0" <1>
ENTER                                      0<PHONE_NUMBER>39400  Region: "implicit task 1" <2>
ENTER                                      0<PHONE_NUMBER>63453  Region: "single_executor" <3>
LEAVE                                      0<PHONE_NUMBER>87769  Region: "single_executor" <3>  <--- this is the implicit-task-end event mislabelled due to corrupted region stack data
LEAVE                                      0<PHONE_NUMBER>07394  Region: "implicit task 1" <2>
LEAVE                                      0<PHONE_NUMBER>36041  Region: "Parallel Region 0" <1>

The leave events in the Fortran trace are incorrectly labelled - they attribute events to the region definitions on the stack, but as the single-executor event didn't happen, the region wasn't popped during the correct event and so there's an off-by-one error among the leave events.

There is a not a fully-C version - there are C implementations of the openMP calls and loops contained inside, but the main function & kernels only have Fortran implementations. If that is of use (though since you have a MWE it seems unlikely) the code is in benchmarks/nemo/nemolite2d/manual_versions/psykal_cpp - the moment its all done with parallel for but I can make a new version with the appropriate structure.

I suspect that wouldn't help. I think something wonky is happening when compiling with GCC while linking to LLVM. If you gave me a mixed Fortran/C source, I expect the default would be to build it all under GCC, in which case I think the error would still be present. Replacing Clang with GCC in the MWE gives the same error as for the Fortran source:
gcc -g -fopenmp -lomp
events-parallel-single-c.txt:
[d] [on_ompt_callback_thread_begin   ] [t=0] (event) thread-begin
[d] [on_ompt_callback_implicit_task  ] [t=0] (event) initial-task-begin
[d] [on_ompt_callback_parallel_begin ] [t=0] (event) parallel-begin
[d] [on_ompt_callback_implicit_task  ] [t=0] (event) implicit-task-begin
[d] [on_ompt_callback_work           ] [t=0] (event) workshare-region-begin (single 1)
[d] [on_ompt_callback_implicit_task  ] [t=0] (event) implicit-task-end
[d] [on_ompt_callback_parallel_end   ] [t=0] (event) parallel-end
[d] [on_ompt_callback_implicit_task  ] [t=0] (event) initial-task-end
[d] [on_ompt_callback_thread_end     ] [t=0] (event) thread-end

gfortran -g -fopenmp -lomp
events-parallel-single-f90.txt:
[d] [on_ompt_callback_thread_begin   ] [t=0] (event) thread-begin
[d] [on_ompt_callback_implicit_task  ] [t=0] (event) initial-task-begin
[d] [on_ompt_callback_parallel_begin ] [t=0] (event) parallel-begin
[d] [on_ompt_callback_implicit_task  ] [t=0] (event) implicit-task-begin
[d] [on_ompt_callback_work           ] [t=0] (event) workshare-region-begin (single 1)
[d] [on_ompt_callback_implicit_task  ] [t=0] (event) implicit-task-end
[d] [on_ompt_callback_parallel_end   ] [t=0] (event) parallel-end
[d] [on_ompt_callback_implicit_task  ] [t=0] (event) initial-task-end
[d] [on_ompt_callback_thread_end     ] [t=0] (event) thread-end

It would be interesting to see whether the following build steps resolve this issue:

Separate all OMP calls into C source and compile to .o with clang -fopenmp
Compile Fortran (which accesses OMP only through the C source) to .o with gfortran
Link the above .o files into an executable with -lomp to link the LLVM runtime

I think this might work, as all OpenMP code would be compiled by and refer to LLVM, meaning it should be safe to use the LLVM runtime when linking.

Updated the MWE to make changing compilers and flags easier.

Does this happen with all versions of gcc? I can probably test with a newer version if you think that might make a difference?

I'm using 9.3.0. You could try it with a newer version but I don't think that will make a difference. I'm currently building Flang to see if compiling everything under LLVM helps. If this issue isn't seen under Flang, I would conclude that combining gfortran and libomp is not avdisable when using Otter.

Do you know whether the benchmark can be compiled with the Intel Fortran compiler?
❯ ifort --version
ifort (IFORT) 2021.3.0 20210609
Copyright (C) 1985-2021 Intel Corporation.  All rights reserved.

I tried replacing gfortran with ifort and -fopenmp with -qopenmp in env.sh but I seem to get lots of compiler errors.

I think it should, though you'll need -liomp instead of -lomp I think. I'm not sure if I have access to a machine with intel on right now but I can try and work it out tomorrow.

I've dug further and have found why some OMPT events are missing when linking to libomp. It is because libomp (and libiomp5) implements the interface to libgomp as a shim layer. The libgomp interface is different to the kmp interface used by libomp and libiomp5. Specifically, while the kmp interface defines both __kmpc_single[1] and __kmpc_end_single [2] the gomp interface only defines GOMP_single_start and no equivalent _end [3]. This means there is no way for the libomp runtime to know a single-region has ended when linked by a gcc/gfortran binary, and no way for the single-end event to fire.
This means that an OMPT tool used to record events in an application compiled under GNU and linked to libomp or libiomp5 will only receive callbacks for events defined in the gomp interface. AFAICT, the only way to get all the events you expect from an OMPT tool is to use it on appplications which link to the kmp interface (i.e. compiled under clang/intel), not the gomp interface (i.e. compiled under gnu)

I've updated the label as this doesn't appear to be a bug in Otter, but rather a limitation of the events available to an OMPT tool when used with an application compiled under gnu but linked to libomp/libiomp5.

Ok - I'll try to find a way to use ifort in future, thanks for looking into this. Hopefully GCC eventually has a OMPT implementation (or flang can compile PSyclone code).


Make sticky footer more efficient

Noticed while using an instance of this building block that it was jittery - the reason is it was re-traversing the DOM to recapture items in ever scroll event.  This PR resolves this by "cashing them outside" (h/t @tdhartwick).

Thanks @kball! It's looking good to me!


Array field row labels cannot be customized

Link to reproduction
No response
Payload Version
3.0.0-beta.63
Node Version
v22.4.1
Next.js Version
15.0.0-canary.53
Describe the Bug
Using docs example:
https://payloadcms.com/docs/fields/array#example
This example of customizing an array field row label doesn't seem to work
It displays Slide undefined
Reproduction Steps
Just try this example
Adapters and Plugins
No response

In 3.0 you need to provide a custom component for it, like this https://github.com/payloadcms/payload/blob/f1b043ba57ee1ea04189185600b8db7f2cfd7015/templates/ecommerce-3/src/payload/collections/Products/ui/RowLabels/VariantLabel.tsx
Which you can then include in your field

{
  name: 'variants',
  type: 'array',
  admin: {
    components: {
      RowLabel: VariantLabel,
    },
  },
  fields: []
}


MudTextField label read after input

Bug type
Component
Component name
MudTextField
What happened?
Currently the label is generated below the input field. This has the effect that when using a screenreader, the user is not aware of what they have to fill in the field.
Would it be possible to reverse the positions of these two elements?
Expected behavior
The user should be able to know what they have to type into the field before entering into it.
Reproduction link
https://try.mudblazor.com/snippet/cYmQOCaGzrdPGoei
Reproduction steps

Make a <MudTextField with a label
Open Narrator or NVDA
Navigate to the element. It looks correct, but you have to pass the input field to get the screenreader to read the label.
...

Relevant log output
No response
Version (bug)
current
Version (working)
6.0.14
What browsers are you seeing the problem on?
Microsoft Edge
On what operating system are you experiencing the issue?
Windows
Pull Request

[ ] I would like to do a Pull Request

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Do you think we should change this @igotinfected ?

@danielchalmers I think this is already fixed, both for the Label and the HelperText. On a macbook using VoiceOver it announces the label and the helper text on this snippet:

Pretty sure the for on the label and the new aria-describedby on the input values ensure this works as expected.

Fantastic, thanks @igotinfected !
@chripa85 Please let me know if it's still a problem for you and I can re-open the issue


@paddyroddy to improve Renovate set up

Either at @UCL-ARC level, or repo level. Taking what I've learnt in https://github.com/paddyroddy/.github

Fixed in UCL-ARC/.github#6 & UCL-ARC/.github#7


Create a system to throttle the ingress of integration tests into the cluster

Create a control system to stagger the ingress of new integration test jobs into the cluster by measuring its capacity and current resources available to take on new jobs.

The PR is in progress


Documentation clarification - app level token

Is there an existing issue for this?

[X] I have searched for any related issues and avoided creating a duplicate issue.

Have you read our code of conduct?

[X] I have read and agree to the code of conduct

Current Behavior
Setting this up for the first time.
In the node.js documention (setup.md), Step#2>

Click Socket Mode from the left hand menu, toggle the Enable Socket Mode option, and follow the steps to create an app-level token. Copy that token into your .env as SLACK_APP_TOKEN.

Socket Mode is already on (from the app manifest), so no need to toggle it.  When I attempt to create the app token, you don't specify the scopes to add.
Expected Behavior
Would like to know which tokens to enable
Steps To Reproduce

Following the guide for setup for Node.JS
Import app from manifest.yml per the setup doc
Step 2 from the setup - socket mode is already enabled in the app.

Environment
Win10
Node 16.4.1

Anything else?
Thanks for developing a great app!  Looking forward to using it in the future!

Good spot, thanks for letting us know!


caf: 0.19.6 -> 1.0.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/actor-framework/actor-framework/releases.
meta.description for caf is: Open source implementation of the actor model in C++
meta.homepage for caf is: http://actor-framework.org/
meta.changelog for caf is: https://github.com/actor-framework/actor-framework/raw/1.0.0/CHANGELOG.md
Updates performed

Version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.0.0 with grep in /nix/store/ciwxrnvb5cx066q6vv27xgaz8w9f2x95-actor-framework-1.0.0
found 1.0.0 in filename of file in /nix/store/ciwxrnvb5cx066q6vv27xgaz8w9f2x95-actor-framework-1.0.0

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

caf

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/ciwxrnvb5cx066q6vv27xgaz8w9f2x95-actor-framework-1.0.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A caf https://github.com/r-ryantm/nixpkgs/archive/09d1cc0d1a66d9fcd3d6b03002ca71d5d77550b1.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/09d1cc0d1a66d9fcd3d6b03002ca71d5d77550b1#caf

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/ciwxrnvb5cx066q6vv27xgaz8w9f2x95-actor-framework-1.0.0
ls -la /nix/store/ciwxrnvb5cx066q6vv27xgaz8w9f2x95-actor-framework-1.0.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

1 package built:

caf

Maintainer pings
cc @bobakker @tobim for testing.

Result of nixpkgs-review pr 322658 run on x86_64-linux 1

1 package built:

caf

@NixOS/nixpkgs-merge-bot merge

Result of nixpkgs-review pr 322658 run on x86_64-linux 1

1 package built:

caf


Added tests for setting the loggedAxis for primary series.

Coverage remained the same at 99.034% when pulling b7d920d7768addf8cf19fcbe955d8b7b2bf5d188 on thongsav-usgs:master into eeed8c7f7de4ea045d44f4ca60030128fa8a2b4f on USGS-R:master.


Remove zio-json dependency

Hi, thanks for creating this plugin.
As per the comment in sbt-ci-realese, adding a zio-json as a dependency adds some controversy.
Would it be possible to use e.g. sjon-new, since it is already the go to json library in sbt?
Original comment (https://github.com/sbt/sbt-ci-release/pull/298#pullrequestreview-2169049028):

Does this transitively introduce Zio dependency just for JSON decoding? Could we use Contraband / sjson-new that are already in use for sbt or uPickle instead?

@KacperFKorban, closing this now that https://github.com/lumidion/sonatype-central-client/pull/6 and https://github.com/lumidion/sbt-sonatype-central/pull/10 are merged with fresh releases.

Thanks! :heart:


multiple ports support?

can add multiple ports function?

it's still not suport multiple ports ?

Ouch. I like zmaps randomization much more, but the multiple port thing is a killer.

@dkran What's your use case for multiple-port scanning?

Any service can conceivably be run on any port. So to get an accurate census of the internet, zmap's randomization and collection of tools is required, but service discovery on "1-65,535" would give a way more accurate result.  I can scan the ipv4 address space from home in a short time on one port, but my accuracy would suffer due to this.  Plus services running on odd ports is what makes the internet interesting.  Like the botnets that constantly try to brute-force me. I understand that's a massive, massive increase on the time it would take, but with enough shards and processing we would have a complete picture rather than a "guess" that everyone is running certain services on certain ports.  And if they're secure.

Any service can conceivably be run on any port. So to get an accurate census of the internet, zmap's randomization and collection of tools is required, but service discovery on "1-65,535" would give a way more accurate result. I can scan the ipv4 address space from home in a short time on one port, but my accuracy would suffer due to this. Plus services running on odd ports is what makes the internet interesting. Like the botnets that constantly try to brute-force me. I understand that's a massive, massive increase on the time it would take, but with enough shards and processing we would have a complete picture rather than a "guess" that everyone is running certain services on certain ports. And if they're secure. @dadrian
I originally started a github repo using masscan to discover ips, then had multiple nmap nodes that could be run from any computer in the world to process all the ports, using nodejs and rethinkdb, along with those command line tools. Although much slower, it had much higher accuracy on a single IP. I even wrote a parser for nmap xml to json since nothing existed in the JS ecosystem.
edit: Did a test last night, and most of the ports that came up were not standard HTTP ports but entirely odd ports (used masscan at a rate of 60MB / sec on ports 1-65535 and was getting thousands of results on odd ports. I've stored them in the rethinkdb queue and I'll use nmap service discovery on it, passing back and forth with websockets. Unfortunately rethinkdb didn't handle that rate of input well, I need more RAM or servers, but I'll solve this problem. I guess there's no perfect tool for this job yet.

How many minutes took to scan the ipv4 using one port : ex port 80 ? (better to add the pc spec's also)

Any service can conceivably be run on any port. So to get an accurate census of the internet, zmap's randomization and collection of tools is required, but service discovery on "1-65,535" would give a way more accurate result. I can scan the ipv4 address space from home in a short time on one port, but my accuracy would suffer due to this. Plus services running on odd ports is what makes the internet interesting. Like the botnets that constantly try to brute-force me. I understand that's a massive, massive increase on the time it would take, but with enough shards and processing we would have a complete picture rather than a "guess" that everyone is running certain services on certain ports. And if they're secure. @dadrian
I originally started a github repo using masscan to discover ips, then had multiple nmap nodes that could be run from any computer in the world to process all the ports, using nodejs and rethinkdb, along with those command line tools. Although much slower, it had much higher accuracy on a single IP. I even wrote a parser for nmap xml to json since nothing existed in the JS ecosystem.
edit: Did a test last night, and most of the ports that came up were not standard HTTP ports but entirely odd ports (used masscan at a rate of 60MB / sec on ports 1-65535 and was getting thousands of results on odd ports. I've stored them in the rethinkdb queue and I'll use nmap service discovery on it, passing back and forth with websockets. Unfortunately rethinkdb didn't handle that rate of input well, I need more RAM or servers, but I'll solve this problem. I guess there's no perfect tool for this job yet.

I agree with you that the distribution scan mode can find more alive IPs and ports in my test data. Meanwhile，I want to say that the scan results of Zmap  are effected by upstream routers. Last 2 years I rent some VPS to scan global IP addresses, only 30Mbps can get property response data.  So if you want to scan global IPv4 addresses and 65535 ports, you'd better rent lots of nice servers or hosts. :)

sry, change property to properly. :)

any update on the multi-port scan support?

Still waiting for multiport support :)

We will implement multiple ports sometime after the pandemic, when the ZMap team can meet up in person again.


chore: added facebook icon on navbar

Closes #124
Description of changes

added facebook icon for dark/light mode on the navbar

Additional context

thanks for contributing @amelia2802 🙂 we just added an all-contributors section to the README - can you let me know if you prefer not to be recognized in this way?
@all-contributors please add @amelia2802 for code

Hi @keyserj, thank you for giving me the opportunity to contribute to this amazing project, also
sharing the tips as I am unaware of the squash/merge button. It's really helpful.
And I'd like to add myself as a contributor and hope to make more useful contributions! 😊


Maya: Hide animation instance in creator

Changelog Description

Hide animation instance in creator
Add inventory action to recreate animation publish instance for loaded rigs

Additional info
The animation family was often 'misused' by new users as a pointcache output but was originally designed for rigs only. This hides the animation instance but preserves the rig -> animation instance behavior.
It also adds an inventory action to recreate the animation instance if it was accidentally deleted.

Testing notes:

Load a rig - animation instance should be created
Recreate animation instance through scene inventory / manager actions.

I like this! can't test right now, but it's a good idea


Resource Type filter includes hidden resource types

By default, Parameter resources are hidden from the dashboard, yet this type of resource shows up in the resource type filter.
Adjusting the filter to show only Parameter types, shows "no Resources"

It feels a bit odd to be given the option to select Parameter resources, and then be given no results when choosing that resource type.

@kvenkatrajan PTAL


zed: small fixes

CI: Only sync new images after build
swap: Note that Kayobe supports swap in Zed
tempest: update max microversions for Cinder & Nova
Update stackhpc.pulp to 0.5.2
ceph: remove unnecesary Jinja braces
aufn-ceph: Use stackhpc/zed branch in script
requirements.txt: drop version specifier
pulp: Stop syncing prometheus-jiralert image
prometheus: Drop some version overrides
aufn-ceph: Drop cephadm_image override
aufn-ceph: Sync inventory with kolla-ansible

Might also be a good time to bump the pulp container a bit?

cephadm_image in aufn-ceph/cephadm.yml needs updating

cephadm_image in aufn-ceph/cephadm.yml needs updating

I removed it since the default should now work.

Do you want more suggestions for things that need to change but are not included on this PR, or should we just try to get this through with the changes that are in at the moment?

Do you want more suggestions for things that need to change but are not included on this PR, or should we just try to get this through with the changes that are in at the moment?

i.e. we need to change https://github.com/stackhpc/stackhpc-kayobe-config/blob/stackhpc/zed/.github/workflows/stackhpc-build-kayobe-image.yml#L70C12-L70C12 at some point to include BASE_IMAGE: rockylinux:9. There's also some references to virtualenv in the aufn deployment script that should be swapped for python3 -m venv

Do you want more suggestions for things that need to change but are not included on this PR, or should we just try to get this through with the changes that are in at the moment?

i.e. we need to change https://github.com/stackhpc/stackhpc-kayobe-config/blob/stackhpc/zed/.github/workflows/stackhpc-build-kayobe-image.yml#L70C12-L70C12 at some point to include BASE_IMAGE: rockylinux:9. There's also some references to virtualenv in the aufn deployment script that should be swapped for python3 -m venv

Mostly looking for a +1/-1 on these changes. Unless there are issues with these or related things I'd prefer to get this one merged and make tickets in Jira for outstanding issues.


New-AdminPowerApp Support

Any planned support for a New-AdminPowerApp? Would be happy to test if there's a beta in the CLI for this functionality.
Would like to see this for both Canvas and Model-drive apps.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 02f1be79-2b3a-31da-7a7d-ace1eac5d1f1
Version Independent ID: fa7c7450-15a6-edc8-5671-f45763267ca6
Content: PowerShell support (preview) - Power Platform Admin center
Content Source: power-platform/admin/powerapps-powershell.md
Service: power-platform
GitHub Login: @jimholtz
Microsoft Alias: jimholtz

@gregdegruy
Thanks for your feedback. For product-related feedback, visit https://ideas.powerapps.com.


Added --http_address option.

xref: https://github.com/mesosphere/marathon/issues/1261

:shipit:

looks good to me


Setting page size of PDF

Hello, I looked through the documentation, is there a way to set a page size, so that no matter what is being converted to PDF the elements are 8.5x11 ?
Thanks

FYI - The 2 links above are broken while I'm searching for this solution


Update blade templates to bootstrap-5 / backpack-6

PR that updates the blade templates + generators:

The command to create the menu items now uses the <x-backpack::menu-item> components, to retain compatability with all the core Backpack themes.
The views have been updated to bootstrap 5 (minor formatting tweaks; fixes to the modal etc), and now extend the correct Backpack view.

Just added a quick update to improve the usability of the autogenerated menu items in different layouts. Promise I'm done tinkering now!


Test

agree merge

:tada: This PR is included in version 1.15.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Turning off authentication?

Is there any way to turn off the login screen? I'd prefer to use my own identity provider, and while something like OIDC is quite a big effort, I was hoping there might be a stick way to disable auth completely?

Hey, a separate identity provider will (probably) not work, as the password is required to derive a password (PBKDF) for the encryption. And the encryption will always be mandatory.

Ahh, shame. I was hoping maybe the encryption key might be separate from the login flow itself, but if they're tied then I guess it'll be too much work to split. Thanks anyway!


Isiod is missing profiles

Is this the right place to submit this?

[X] This is not a security vulnerability or a crashing bug
[X] This is not a question about how to use Istio

Bug Description
Based on documentation found here https://istio.io/latest/docs/setup/additional-setup/config-profiles/ and https://artifacthub.io/packages/helm/istio-official/istiod#profiles we are able to use --set profile=<profile>. Using this feature will allow only some components to be installed using HELM. The problem is that while using this option we are having issue with:
Error: UPGRADE FAILED: execution error at (istiod/templates/zzz_profile.yaml:25:3): unknown profile minimal
Upon investigation we found that problem is comming from zzz_profile.yaml line 25 which is in this case else https://github.com/istio/istio/blob/master/manifests/charts/istiod-remote/templates/zzz_profile.yaml#L25
Based on code from line 23 the if is looking for file in files/profile-<profile>.yaml
https://github.com/istio/istio/blob/master/manifests/charts/istiod-remote/templates/zzz_profile.yaml#L23
But minimal profile as described in the official docs are missing there (no profile-minimal.yaml file)
https://github.com/istio/istio/tree/master/manifests/charts/istiod-remote/files
Version
HELM ls
istio-base 1.22.2
istiod  1.22.2

kubectl version
Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.7

Additional Information
No response

This is intentional but a documentation issue.
minimal doesn't make sense for Helm, as in Helm you explicitly install each component individually. The minimal profiles only role was to NOT enable components, as istioctl install bundles them all together

This is intentional but a documentation issue.
minimal doesn't make sense for Helm, as in Helm you explicitly install each component individually. The minimal profiles only role was to NOT enable components, as istioctl install bundles them all together

That make perfect sense. I was also wondering what is the meaning for it in HELM and now I know. Thanks for clarification!


Some Questions about SAFA

you said you reimplemented the SAFA with Pytorch in your paper, but I only found the reference of SAFA in your code like these.

Could you give me the definition code of SAFA?

SAFA+ours is re-implemented using Tensorflow. I will publish the relative code in the later.

Thank you very much for replying to me so quickly, but you said that you re-implemented the SAFA model in TensorFlow. Why does its reference appear in the Pytorch code?

At that time, I tried to reproduce SAFA using pytorch, but the result I got was not as good as the source code. You can ignore these commented codes


CarouselCards issue with attachmentLayout.carousel

Below code only show the hero card listing but not the attachmentLayout.carousel.

function CreateHeroCard(session, builder, title, subtitle, text, url, buttons){
    var card = new builder.HeroCard(session)
        .title(title)
        .subtitle(subtitle)
        .text(text)
        .images([builder.CardImage.create(session, url)])
        .buttons(buttons);
    return card;
}

function SendHeroCard(session, builder){
     var request = require("request");
    var url = 'http://example.com/fetch/' + session.userData.profile.userToken + '/' + session.userData.profile.Level1;
    request(url, function (error, response, body)
    {
        if (!error && response.statusCode == 200) {
            var attachmentsArr = [];
            var attachmentsChoice = [];
            var jsondata = parseJson(body);
            var nextLevel = treeRoot.first(function (node) {
                return node.model.attributes.id === session.userData.profile.treeSelectdNodeId; // 1.1
            });
            for (var i = 0; i < jsondata.length; i++) {
                var secondListChild = [];
                if(nextLevel.model.children.length > 0) {
                    for (var j = 0; j < nextLevel.model.children.length; j++) {
                        if (nextLevel.model.children[j].attributes.title != '' && nextLevel.model.children[j].attributes.title != undefined) {
                            secondListChild.push(builder.CardAction.imBack(session, nextLevel.model.children[j].attributes.id+":"+ jsondata[i].HierarchyLevel2, nextLevel.model.children[j].attributes.title));
                        }
                        attachmentsChoice.push(nextLevel.model.children[j].attributes.id+":"+ jsondata[i].HierarchyLevel2);
                    }
                    secondListChild.push(builder.CardAction.imBack(session, "previous menu", "previous menu"));
                }
                var card = CreateHeroCard(session, builder, jsondata[i].HierarchyLevel2, "RITS Builder Result", "", "http://hsplbot.headfitted.in/images/RockwellLogo3.png",
                secondListChild);
                attachmentsArr.push(card);

            }
            var reply = new builder.Message(session)
                    .textFormat(builder.TextFormat.xml)
                    //.attachmentLayout(builder.AttachmentLayout.carousel)
                    .attachments(attachmentsArr);
            //session.send(reply);
            builder.Prompts.choice(session, reply, attachmentsChoice);
        }
    });
};

If you review my code. I commented
//.attachmentLayout(builder.AttachmentLayout.carousel)
If you remove the comment of this line below error received in bot node
Error: Request to 'https://smba.trafficmanager.net/v3/conversations/29%3A1z7akAj7gyQq-8N4O-hkjHHJHGKJsdsd/activities' failed: [400] Bad Request
    at Request._callback (/home/node_modules/botbuilder/lib/bots/ChatConnector.js:413:46)
    at Request.self.callback (/home/node_modules/request/request.js:186:22)
    at emitTwo (events.js:106:13)
    at Request.emit (events.js:191:7)
    at Request.<anonymous> (/home/node_modules/request/request.js:1081:10)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:188:7)
    at IncomingMessage.<anonymous> (/home/node_modules/request/request.js:1001:12)
    at IncomingMessage.g (events.js:291:16)
    at emitNone (events.js:91:20)

One of the CarouselCards Example BotBuilder-Samples there is one note is mention.
Note: At the time of writing this sample, there is a limit on the amount of cards that can be stacked in a carousel. For Facebook, attachments carousel is mapped to the Generic Template which has a limit of 10 elements. For Skype, there's also a limit of 10 elements in the carousel.

@mukeshzt  can we use CarouselCards in Slack??

CarouselCards are displayed as list of cards in slack. You can look at channel inspector for different capabilities in different channels: https://docs.botframework.com/en-us/channel-inspector/channels/Slack/#navtitle

Thank You so much :)

Answer for my above error is " For Skype, there's also a limit of 10 elements in the carousel."


GSD Request

--- GSD JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from  to before v5.15.18",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=145407e54fd15fd2a87ee7d6c66077e614036a9b"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "145407e54fd15fd2a87ee7d6c66077e614036a9b",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "select: Fix indefinitely sleeping task in poll_schedule_timeout()\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v5.15.18 by commit 145407e54fd15fd2a87ee7d6c66077e614036a9b. For more details please see the references link."
}
--- GSD JSON ---

This issue has been assigned GSD-2022-1000238


Pathing with new StripPrefixPath is still missing paths (1.3 branch)

I'm trying to identify why, but for all paths: https://pack.io and https://pack.io/api; I'm still seeing all 404's with the 1.3 branch.

the two paths that are configured are:
/ -> packwebsite
/api -> packapi
both are returning 404's although it doesn't seem to make sense why

Could you provide your configuration?

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: pack-web-app
  namespace: pack
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.frontend.rule.type: pathprefixstrip
spec:
  rules:
  - host: api.pack.io
    http:
      paths:
        - backend:
            serviceName: packapi
            servicePort: 80

  - host: pack.io
    http:
      paths:
       - path: /api
         backend:
           serviceName: packapi
           servicePort: 80

       - path: /
         backend:
           serviceName: pack-maint-site
           servicePort: 80

Problem is the normal paths work fine, the ones under pack.io; via pathprefixstrip, are just 404'ing.
I'm still digging into why, from the router, it seems like it should work.

https://api.pack.io has no issue, for example

Oh, is this maybe an issue on my end?
time="2017-05-26T22:17:55Z" level=debug msg="Wiring frontend onboarding.packio.net to entryPoint http"
time="2017-05-26T22:17:55Z" level=debug msg="Creating route onboarding.packio.net Host:onboarding.packio.net"
time="2017-05-26T22:17:55Z" level=debug msg="Creating backend onboarding.packio.net"
time="2017-05-26T22:17:55Z" level=debug msg="Creating load-balancer wrr"
time="2017-05-26T22:17:55Z" level=debug msg="Creating server onboarding-2041587088-2x0d3 at http://<IP_ADDRESS>:80 with weight 1"
time="2017-05-26T22:17:55Z" level=debug msg="Creating server onboarding-2041587088-5cbqj at http://<IP_ADDRESS>:80 with weight 1"
time="2017-05-26T22:17:55Z" level=debug msg="Creating server onboarding-2041587088-6gxlr at http://<IP_ADDRESS>:80 with weight 1"
time="2017-05-26T22:17:55Z" level=debug msg="Creating circuit breaker NetworkErrorRatio() > 0.5"
time="2017-05-26T22:17:55Z" level=debug msg="Creating frontend pack.io/"
time="2017-05-26T22:17:55Z" level=debug msg="Wiring frontend pack.io/ to entryPoint http"
time="2017-05-26T22:17:55Z" level=error msg="Error creating route for frontend pack.io/: Error parsing rule: Error parsing rule: 'pathprefixstrip:/'. Unknown function: 'pathprefixstrip'"
time="2017-05-26T22:17:55Z" level=error msg="Skipping frontend pack.io/..."
time="2017-05-26T22:17:55Z" level=debug msg="Creating frontend pack.io/api"
time="2017-05-26T22:17:55Z" level=debug msg="Wiring frontend pack.io/api to entryPoint http"
time="2017-05-26T22:17:55Z" level=error msg="Error creating route for frontend pack.io/api: Error parsing rule: Error parsing rule: 'pathprefixstrip:/api'. Unknown function: 'pathprefixstrip'"
time="2017-05-26T22:17:55Z" level=error msg="Skipping frontend pack.io/api..."
time="2017-05-26T22:17:55Z" level=info msg="Server configuration reloaded on :80"

Wow. We can ignore this, this was totally my fault; it was working with the prior configuration before.

Difference was:
# Invalid
traefik.frontend.rule.type: pathprefixstrip

vs
traefik.frontend.rule.type: PathPrefixStrip

Seems like this should be something that's documented; are there plans to update the Traefik documentation with a caveat about capitalization?

@gaballard the related PR #1695 was merged 2 hours ago.


Refactor CMakeLists.txt

Removed redundancy and adds more consistency especially for the wasm build

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 81.83%. Comparing base (f678e0d) to head (34f8335).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #164   +/-   ##
=======================================
  Coverage   81.83%   81.83%
=======================================
  Files          19       19
  Lines         837      837
  Branches       91       91
=======================================
  Hits          685      685
  Misses        152      152

@vgvassilev this should be ready (P.S i made the exact same change on xeus-r few days ago and I realized they would be applicable on xeus-cpp too so pretty confident :) )


Support for coming back from link jumps

So I've been reading a book that is full of footnotes. The footnotes themselves link to appendices. And appendices themselves have links to other parts of the book.
In such kind of book, as Quality of Life improvement to the reader, is keeping a mark for the return of each jump, as the current code allow with  the '  mark, but that allows only for one jump.
The backspace could be bound to this move.
It is similar to what Ctrl + O does on vim.

Would be great if we could somehow select footnotes with the keyboard, too.
E.g., to have a dedicated key to go the first available footnote counting from the top of the screen.


Circular build warning during building from source

Circular build warning
During building soletta from source
after make command
make: Circular build/stage/modules/flow/converter/string-format.o <- build/stage/modules/flow/converter/string-format.o.dep dependency dropped.
is displayed
but the installation continues
is this a general warning?
Steps to reproduce

cd soletta
2.make thirdparty-update
3.make alldefconfig
4.make

Host OS and architecture: -Ububtu 14.04 x64

There's a PR (https://github.com/solettaproject/soletta/pull/1940) fixing that.


[Network] Bump api version from '2020-11-01' to '2021-02-01'

Description

Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change.
[Component Name 2] az command b: Add some customer-facing feature.

This checklist is used to make sure that common guidelines for a pull request are followed.

[ ] The PR title and description has followed the guideline in Submitting Pull Requests.

[ ] I adhere to the Command Guidelines.

[ ] I adhere to the Error Handling Guidelines.

Network

For my code owned part, recommend to run test in live mode not just replacing api verison. Replacing api version is accepted only when there is service issue or hard-coded resource.

It is a good advice. However, some test of other service is very hard to run with live mode. And when bump version for network, run live test of network is enough since network has the most test scenarios. For other modules, their usage scenario is just a very small part of network test scenarios. The bump experience also proves that: only network test ever found useful breaking change. And as for other part, their live test for network could not find problems except wasting time in waiting for live result.


Backup with pg_dump fails when timestamp column is used

Hi Alex @staticlibs,
Can the windows authentication request be added for back up tool wdb_backup?
also when we try to take backup for our database, backup failed with below error

I would like to remind you about the same problem which we faced earlier and you added the fix for the same, you may need to add same fix in back up tool. could you please check on it?

@psrinivasa2

Hi Alex @staticlibs, Can the windows authentication request be added for back up tool wdb_backup?

No, this is not planned. pg_dump works over Postgres protocol, and using SSPI with Postgres connections is much less convenient than with TDS ones (some details are in this comment).
Some other Postgres auth feature can be used to handle passwords in this case, for example a password file.

also when we try to take backup for our database, backup failed with below error
I would like to remind you about the same problem which we faced earlier and you added the fix for the same, you may need to add same fix in back up tool. could you please check on it?

Thanks for the details! I also think that the problem is related to #24. But for some reason I cannot reproduce it with pg_dump. Tried both binary(8) and rowversion columns and pg_dump (called from wdb_backup) dumps it successfully for me.
Would you mind simplifying your table (the one with scn column) DDL to reproduce the problem with minimal number of columns and data records? Alternatively, if you run wdb_backup on debug version of WiltonDB MSI and collect the backtrace for this error - it also may be helpful.

The problem happens here, when rowversion column is requested to be dumped by pg_dump. There seems to be a discrepancy in detecting when column is rowversion and needs to be excluded by pg_dump.
Would you mind checking the registered type for the problematic column with the following query in PgAdmin (replacing table and column names):
select att.attname, typ.typname, typ.oid from pg_attribute att
join pg_type typ on typ.oid = att.atttypid
where att.attrelid = 'test1_dbo.tab1'::regclass::oid
and att.attname = 'col1'

And the OIDs for the registered types:
select typname, oid from pg_type
where typname in ('binary', 'rowversion', 'timestamp')

Can you also try to drop and re-create the table on the latest MSI version and check the type again?

Hi @staticlibs ,
I have tried drop and recreate table with latest MSI below is the result of the query before and after .
wilton - <IP_ADDRESS>

wilton-<IP_ADDRESS>

OIDs for the registered types:

Hi @staticlibs ,
we have tried password file option for wilton backup tool but it did not credentials from there.
Below is the path i have the passconfig file.

Below is the details of pass file.

Any idea of what we missing there.

@Aravind2601 , thanks for details! Assuming scn column must be binary(8), that means that column type was confused with timestamp at table creation time. If you can reproduce this - can you share a standalone (simplified) T-SQL DDL that creates the table - so I can reproduce this locally?
PS: I will comment on pgpass file separately in a few hours.

@staticlibs  I have sent in mail.

@Aravind2601 , on pgpass file, please provide more details on the error you are getting. Is it something like the following, or a different one:
> psql -h <IP_ADDRESS> -p 5432 -U wilton
psql: error: connection to server at "<IP_ADDRESS>", port 5432 failed: FATAL:  password authentication failed for user "wilton"
password retrieved from file "C:\Users\alexk\AppData\Roaming/postgresql/pgpass.conf"
connection to server at "<IP_ADDRESS>", port 5432 failed: FATAL:  password authentication failed for user "wilton"
password retrieved from file "C:\Users\alexk\AppData\Roaming/postgresql/pgpass.conf"

I suppose your problem is in localhost part, depending how local address is resolved. I suggest to use something like this:
*:*:*:wilton:wilton

@Aravind2601 , on rowversion/timestamp problem - thanks for providing the DDL! The problem is reproducible with the following table:
create table tab1(col1 int, col2 timestamp)

Curiously enough, it is NOT reproducible with:
create table tab1(col1 int, col2 rowversion)

despite rowversion and timestamp being synonyms. It appeared that typename is returned incorrectly quoted for T-SQL timestamp columns and this breaks COPY command called by pg_dump. I was unable to quickly find a root cause on server, instead added a workaround to pg_dump.  Updated wdb_backup to version 1.0.5 including fixed pg_dump.exe. Please try this version for GUI backups, and also use pg_dump.exe from there (instead of one in C:\Program Files\WiltonDB Software\wiltondb3.3\bin) for command-line backup calls.
I will keep this issue open for a proper fix in DB backend and potential upstreaming to Babelfish.

Given the way Babelfish works with PostgreSQL and MVCC transactions under the cover, I recommend to all customers to remove timestamp and rowversion columns from their tables. Under the covers, Babelfish implements the columns using the "hidden" system columns in pg_attributes. The actually show down transactions when present. The values are completely useless when you transfer them with pg_dump / restore. So, delete them!
Here are the specifics from Claude.ai you can follow up on.
PostgreSQL system columns, including ctid, play important roles in transaction management and other database functions. Let's explore some key system columns:

ctid (Cursor Tuple ID):

Format: (block_number, tuple_index_within_block)
Functions:

Unique identifier for each row version within a table
Used for row-level locking
Enables efficient updates by creating new row versions

Transaction management: Helps implement Multi-Version Concurrency Control (MVCC)

xmin (Transaction ID that inserted the row):

Indicates which transaction created this row version
Used for determining row visibility to concurrent transactions

xmax (Transaction ID that deleted or updated the row):

Null if row is still valid
Set to the transaction ID that deleted or updated this row version
Used for determining row visibility and cleanup of old versions

cmin (Command identifier within the inserting transaction):

Helps order operations within a single transaction

cmax (Command identifier within the deleting transaction):

Similar to cmin, but for the deleting transaction

tableoid:

Contains the OID of the table containing this row
Useful in inheritance hierarchies to identify which table a row belongs to

oid (Object Identifier):

Deprecated in modern PostgreSQL versions
Previously used as a unique row identifier

These system columns contribute to PostgreSQL's MVCC implementation, enabling:

Concurrent access without read locks
Consistent snapshots of the database
Efficient implementation of transactions and isolation levels

@staticlibs,  thanks for the fix in version 1.0.5 . its working fine with this exe.
Regarding the password file pgadmin is taking the credentials from there. But backup tool is not taking those values it still ask for password . Could you please help use on this?

@Aravind2601 , yes, I will add support for pgpass file to wdb_backup and wdb_config.

Hi Alex@staticlibs ,
Our plan is to use OS user in all the place, as you mentioned in different ticket, we will generate random unknown password for wilton user. Please advise us in terms of OS user in back tool, configuration tool and other required places

@psrinivasa2 , it may be problematic to use OS users with current version. With sspi auth set up in pg_hba.conf only Postgres logins will work. TDS logins currently can only be done when password (md5) auth is used in pg_hba.conf. If you intend to use SSPI logins on both TDS and Postgres ports it makes sense to add support for sspi in pg_hba.conf and also for pg_ident.conf maps for TDS logins (to have the same setup effective for both Postgres and TDS ports). This is possible, but this is not a trivial task.

Just for the record, submitted bug report upstream about the timestamp problem with pg_dump. Modified pg_dump from wdb_backup-1.0.5 (with a workaround to this problem) is going to be included into upcoming MSI installer update.

@Aravind2601 , about the "Cannot determine schema name" error: I've filed #46 for it and updated wdb_backup with the fix.

@staticlibs thanks alex, will check

@staticlibs, we checked with above latest backup tool but the restore was failed. I have shared the log details in mail. Please check and let us know if you got any details.

@Aravind2601
Thanks for the logs! First 2 errors there are related to schema name rewriting (for DB renaming during restore) in views/procedures. It seems that SQL tokenizer library I've used does not support some of the naming used in T-SQL (like $ characters in names). I will follow-up on this.
I wonder if you can check the details about DB3.txt on your side? DB rename is performed successfully there, but then it fails on constraint violation. I wonder if you can check what is this constraint doing and if it is actually being violated by the restored records?

@staticlibs Any update on above requests

@Aravind2601
Working on DB1 and DB2 problems (different problems in the same area), and DB3 behaviour is assumed to work as expected (unless something is wrong with constraint handling - cannot tell without seeing the constraint and corresponding matching record).

@staticlibs  we will check the DB3 from our end and then let you know Alex, you can analyze and help to resolve the issue in DB1 and DB2.

@Aravind2601
I've filed #55 to track fixes for DB1 and DB2 - please see the details (and the updated version of wdb_backup) there.

The fix is included in update 12.16.1.


KubeCon Docs Sprint: Update page weights for content/en/docs/concepts…

contributes to https://github.com/kubernetes/website/issues/35093
Updates page weights in content/en/docs/concepts/cluster-administration
Before my changes:

After applyed my changes:

thanks @haosdent!
/lgtm
/approve


(maint) Generate do-see-do

I attempted to manage the git history with a commit of just git mv and updating the references to that file in external requires (no constant updates), then a commit that doesn't move any files but updates all the constants/references in docs, then a third commit that moves specs that should be named similar to the files they test.
However that doesn't seem to have preserved the git history like I had wanted, not sure if there's a better way to do this or if this is just as good as it gets with Github's blame.

ping @Magisus when you get a minute

I can do a little poking, but off the top of my head I don't know of a way.

I'm fine leaving the history as is.


WIP Run a two-steps tobiko workflow

The test-operator periodic job podified-multinode-edpm-deployment-crc will run a tobiko workflow with two test steps.

@eduolivares, FYI, I've tested the workflows here -> https://github.com/openstack-k8s-operators/test-operator/pull/58
It seems to be working but additional change is needed on the ci-framework side. Right now we collect stdout only from one pod [1][2].
[1] https://logserver.rdoproject.org/58/58/c5175fd3777cff2e6393ccaa8241b740ee51454b/github-check/podified-multinode-edpm-deployment-crc/3793439/controller/ci-framework-data/tests/test_operator/
[2] https://github.com/openstack-k8s-operators/ci-framework/blob/319bce54c09867cd73155c0f95ef0d81c4d5b78f/roles/test_operator/tasks/run-test-operator-job.yml#L127C1-L127C49

Also, the question is whether we want to collect the stdout using the test_operator role at all because the stdout is already being collected by must-gather -> https://logserver.rdoproject.org/58/58/c5175fd3777cff2e6393ccaa8241b740ee51454b/github-check/podified-multinode-edpm-deployment-crc/3793439/controller/ci-framework-data/logs/openstack-k8s-operators-openstack-must-gather/namespaces/openstack/pods/tobiko-tests-step1-workflow-step-0-mkvw5/logs/tobiko-tests.log
Previously, the tempest / tobiko role needed to collect the logs because the execution was not part of the openshift deployment.

Also, the question is whether we want to collect the stdout using the test_operator role at all because the stdout is already being collected by must-gather -> https://logserver.rdoproject.org/58/58/c5175fd3777cff2e6393ccaa8241b740ee51454b/github-check/podified-multinode-edpm-deployment-crc/3793439/controller/ci-framework-data/logs/openstack-k8s-operators-openstack-must-gather/namespaces/openstack/pods/tobiko-tests-step1-workflow-step-0-mkvw5/logs/tobiko-tests.log
Previously, the tempest / tobiko role needed to collect the stdout because the execution was not part of the openshift deployment.

Ok, thank you. I didn't know you were already working on this.
It looks good to me to remove the collection of the container output file from the test_operator role, considering it's included in the must-gather artifacts.

recheck

@eduolivares, can you please remove the WIP flag if the running job passes and change the patch so that it is mergable, when you have time? I think it would be good we merge this and have the workflow covered in the periodic job.

@eduolivares, can you please remove the WIP flag if the running job passes and change the patch so that it is mergable, when you have time? I think it would be good we merge this and have the workflow covered in the periodic job.

OK!
If the job passes, I will fix the patch (enable tempest again, remove the check pipeline), so that it can be approved.

@lpiwowar
The results were good: https://review.rdoproject.org/zuul/build/8e14e2e46fe14eed92cbbdcd274fdc32
This PR is ready to review.


Connection timed out after Setup.sh

I just run ./Setup.sh and it always says connect time out:
➜  carla git:(master) ✗ ./Setup.sh
Backing up existing Content...
'/media/jintian/Netac/CodeSpace/ng/auto_car/system/carla/Unreal/CarlaUE4/Content' -> '/media/jintian/Netac/CodeSpace/ng/auto_car/system/carla/Unreal/CarlaUE4/Content_20171213111344'
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/util/connection.py", line 83, in create_connection
    raise err
  File "/usr/local/lib/python3.5/dist-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connection.py", line 150, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f0c7e805eb8>: Failed to establish a new connection: [Errno 110] Connection timed out

What is this step wrong?

If you are in China,I think this problem occur because of "wall".

@coolKeen Well, Can I manully download from some link and import it into where it should to be.

To be honest,I try downloading the content from  this link: https://zhuanlan.zhihu.com/p/30979943.But I failed to combine these content part.
So I just download the compiled project from https://github.com/carla-simulator/carla/releases/tag/0.7.0 and it runs well on my computer.For specific explanation,see #20 .

Closing this, I created #84 to have a single place to discuss possible solutions to the download problem.


Allow to keep many runs available

For now only the data from the last available run is usually kept in the DB. It would be interesting to configure how many previous runs have to be kept in DB in order to possibly choose among different runs for raw data visualization or probe results search. Indeed, it appears that some runs are more accurate depending on the providers, eg the 0h run is usually more accurate at Météo France.

In addition to extend the ttl we should also change the update code that removes previous run data if any.

Should be possible now


Fix: cleaning invalid task with a wrong task name

Func removeTask uses task name without prefix "T_".

Looks reasonable. Have you tested this?

@brndnmtthws  Tested.

@depay, Could you put val taskId = f.substring(taskPrefix.size) above the if statement so taskId can be used here to make the code cleaner?

:+1:


Added swarm support in embarkjs, isAvailable for messages/storage, swarm/ipfs checks

added EmbarkJS.Storage.isAvailable and EmbarkJS.Messages.isAvailable() and underlying provider functions for Whisper, Orbit, IPFS, and Swarm so service availability can be checked from the DApp (@richard-ramos)
Finished swarm implementation in embarkjs plus cleanup
updated test app storage config to swarm to show swarm config option
added eth-lib to parse result of swarm text
added protocol to storage config (defaulting to "http")
selectively starts storage service depending on which one is configured in the storage config
run service check for ipfs/swarm prior to uploaded
added swarm methods for embarkjs
removed unneeded provider property.
add 'swarm' as a provider in the storage.config
update method for swarm service check

on the test app, run ../../bin/embark upload ipfs or ../../bin/embark upload swarm causes the following error:
/Users/emizzle/Code/__Github/embark/lib/core/engine.js:149
          self.watch.restart(); // Necessary because changing a file while it is writing can stop it from being watched
                     ^

TypeError: Cannot read property 'restart' of undefined
    at /Users/emizzle/Code/__Github/embark/lib/core/engine.js:149:22
    at /Users/emizzle/Code/__Github/embark/lib/pipeline/pipeline.js:155:11
    at /Users/emizzle/Code/__Github/embark/node_modules/async/dist/async.js:473:16
    at replenish (/Users/emizzle/Code/__Github/embark/node_modules/async/dist/async.js:993:25)
    at iterateeCallback (/Users/emizzle/Code/__Github/embark/node_modules/async/dist/async.js:983:17)
    at /Users/emizzle/Code/__Github/embark/node_modules/async/dist/async.js:958:16
    at /Users/emizzle/Code/__Github/embark/lib/pipeline/pipeline.js:150:13
    at /Users/emizzle/Code/__Github/embark/node_modules/async/dist/async.js:1126:9
    at /Users/emizzle/Code/__Github/embark/node_modules/async/dist/async.js:473:16
    at replenish (/Users/emizzle/Code/__Github/embark/node_modules/async/dist/async.js:993:25)
    at iterateeCallback (/Users/emizzle/Code/__Github/embark/node_modules/async/dist/async.js:983:17)
    at /Users/emizzle/Code/__Github/embark/node_modules/async/dist/async.js:958:16
    at /Users/emizzle/Code/__Github/embark/node_modules/async/dist/async.js:1123:13
    at /Users/emizzle/Code/__Github/embark/lib/pipeline/pipeline.js:180:16
    at /Users/emizzle/Code/__Github/embark/node_modules/async/dist/async.js:473:16
    at replenish (/Users/emizzle/Code/__Github/embark/node_modules/async/dist/async.js:993:25)
    at iterateeCallback (/Users/emizzle/Code/__Github/embark/node_modules/async/dist/async.js:983:17)
    at /Users/emizzle/Code/__Github/embark/node_modules/async/dist/async.js:958:16
    at /Users/emizzle/Code/__Github/embark/lib/pipeline/pipeline.js:174:9
    at /Users/emizzle/Code/__Github/embark/node_modules/async/dist/async.js:3096:16
    at replenish (/Users/emizzle/Code/__Github/embark/node_modules/async/dist/async.js:998:17)
    at /Users/emizzle/Code/__Github/embark/node_modules/async/dist/async.js:1002:9

@jrainville could you add the missing early returns please and then merge? @emizzle is currently traveling

I went back to IPFS after testing, but I had what you put in the test_app:
{
  "default": {
    "enabled": true,
    "available_providers": ["ipfs", "swarm"],
    "ipfs_bin": "ipfs",
    "provider": "ipfs",
    "host": "localhost",
    "port": 5001
  },
  "development": {
    "enabled": true,
    "provider": "swarm",
    "host": "swarm-gateways.net",
    "port": 80,
    "getUrl": "http://swarm-gateways.net/bzzr:/"
  },
  "livenet": {
    "enabled": true,
    "provider": "ipfs",
    "host": "ipfs.infura.io",
    "protocol": "https",
    "port": false,
    "getUrl": "https://ipfs.infura.io/ipfs/"
  }
}

The config looks good. I just feel like Swarm is strangely unreliable. Mostly on localhost, getting a resource seemed to return a 200 but with 0 bytes. While swarm-gateways.net seems to 404 most often (and sometimes the resource will eventually become available). And other times, I'll get a CORS error when trying to get a resource. It's definitely flaky.
In saying that, I have never hit an issue with not getting a hash.
And even weirder, I swear sometimes it would seem to work better when running a local node, but using swarm-gateways.net. But that would never be 100% reliable either. Witch craft.


[5.3] Type strict validation and adding a "Nullable" modifier

Currently these cases will pass the validation:
Validator::make(['roles' => ''], ['roles' => 'array']); // same for numeric, integer, boolean

Validator::make(['price' => ''], ['price' => 'numeric|min:2']);

Validator::make(['price' => []], ['price' => 'numeric']);

Validator::make(['name' => null], ['name' => 'string']);

I understand that there's a debate going on around this matter as in here https://github.com/laravel/framework/issues/11452#, while null and empty string is not an integer for example, some people believe that validation should pass in such case.
In this PR I made type validation strict, so that an integer only accepts an integer, not null and not an empty string (same for numeric, boolean, array, and string in case of null), however I also introduced a nullable modifier.
Validator::make(['middle_name' => null], ['middle_name' => 'string']); //fails

Validator::make(['middle_name' => null], ['middle_name' => 'string|nullable']); //passes

So for those who use nullable/string, nullable/integer, nullable/* fields and expect a type check to pass on null they may use the nullable modifier.
For those expecting an empty string to pass validation I think they need to array_filter or we shall add a modifier for that, but I believe it makes sense to fail when validating an empty string is an integer.

We can't strict validate integers like that. HTML forms will always post strings and it won't be reasonable to force a strict integer there for applications that aren't using AJAX calls from a JS framework for example.


Rename Server Name in Settings.cfg aka SEGSAdmin not registering Server Name

Summary
Server Name in settings.cfg is not currently used by the server. Further, the client will not accept custom server names. Currently server_name setting is useless.
Server Name's purpose is to eventually provide a customizable and unique identifier to facilitate promotion of private servers in the Segs Launcher and Verifier (SLAV).
Proposal
I'm prepurposing @HeraldOfOmega's original bug report to propose we rename server name to something to make it clear that it's use is internal ONLY, and that it will not change the selectable server name in the Client's server select screen.
This is a simple tasks of refactoring and updating any relevant files + SEGSAdmin. As such I'm marking good-first-issue
Steps to complete

[ ] Rename all instances of server_name something more clear: perhaps internal_server_name
[ ] Update Settings.cpp, Settings.h and settings_template.cfg to reflect this change
[ ] Update SEGSAdmin to reflect this change. Perhaps add a note to further clarify that this is an internally used server identifier, and will not change the client's server selection screen.

edited by broxen on Tues, Sept 25
@HeraldOfOmega's original Bug Submission:
Expected behavior
SEGSAdmin.exe would create a server of the name provided when you click [Generate Config File]
Actual behavior/symptoms
Server is named "Freedom"
Steps to reproduce behavior/symptoms
Create a new install of SEGS, run SEGSAdmin.exe
SEGS version
0.5.0
Operating system
Windows 7
C++ compiler and version
unknown
Other relevant information
Virtue was better than Freedom! :1st_place_medal:
jk

Thank you so much for submitting this Bug Report!
Unfortunately this is a limitation of the client (the server names are hardcoded)
Since fixing this isn't a possibility at this time, I'm marking this wontfix, but perhaps we can revise the documentation and/or SEGSAdmin to make it clear that servername is used internally ONLY.

Yeah, that echos my thoughts. In a possible custom-client future, we can actually leverage this in the server selection screen, but for now it's just confusing.

OK Updated.
I could be compelled to suggest removing server_name altogether if there's no foreseeable use for it internally.

Did we decide on renaming versus removing server_name?

The only utility for this is for a future SLAV server list. I'd say we just remove it until there's an actual need. It's not complex to add again later.

Closed in #658


How do I test blog localizations?

No information about testing blog localizations. Not entirely applicable as well.


How to start appium server thru code (AppiumServiceBuilder) and assign DesiredCapabilities during run time

Hi @asolntsev We love using selenide used for our Web Automation and it works like a charm, thinking to use it for our newly designed Android and iOS native apps.
Started creating demo project to get approved thru POC.
Have some challenges if you can please answer.

We want appium server to start during run time using AppiumServiceBuilder
Want to assign DesiredCapabilities caps during run time because we want to run test cases in parallel on multiple devices so we need caps to assign during run time

Here is the demo Appium Android project we have created please advice how we can integrate Selenide-appium in our project in order to use awesome locator strategy, ajax call that selenide provide.
https://github.com/prashant-ramcharan/courgette-jvm-appium-android

@LearningPur I realized that we started using AppiumServiceBuilder in Selenide own tests.
Here you can look how it's used:
https://github.com/selenide/selenide/blob/main/modules/appium/src/test/java/it/mobile/ITTest.java#L21


Patents

Do any implementations here have patents on them?

With software patents you can never be sure.

LZW patents expired on 2004 world wide.
Arithmetic coding patents (as used in these old implementations) have been expired as well.  I've not implemented any algorithm that I know that has know issues.
But until someone and claims (a valid) patent on one of them, we can never be 100% sure


Update VERSION before building pushing to ecr

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 69.555%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
12869

Relevant Lines:
18502

💛  - Coveralls


variable font-family

Hopefully this issue hasn't come up before, but would it make sense to switch to using a variable for milligram's font-family? That way people who don't design around Roboto could use for example the native font stack as described here and make milligram even lighter. Native stack is also a default in bootstrap.

like you say I think that font-family  should be adjusted to the value.

@whodidthis make sense to me. I have plans to implement this feature in the next version.
Thank you for your idea. For now, I will close this issue.
Please, feel free to reopen or open a new one.


Support non sequential label images and unsorted measurements

Hi Laura @lazigu , hi Mara @marabuuu ,
similar to recent changes in napari-skimage-regionprops, I'm adding support for unsorted measurements and non-sequential label images here. The trick is to use skimage.utils.map_array and this requires to have a list of label IDs for the corresponding predictions.
A python-test was added to see if it works on normal images and on timelapse-data. I also checked that it works from the GUI, but only tested the normal-image case with blobs.gif. Would you mind testing it again, also with timelapse data, to be sure that I didn't screw anything up?
Big thanks!
Best,
Robert

The failing test (dimensionality reduction / test_bad_measurements) seems unrelated to my changes 👼

Hi Robert @haesleinhuepf,
I investigated the timelapse compatibility on the Platynereis dataset and it seems to work all fine.
Steps I did in Detail:

used from stardist.matching group_matching_labels to get non sequential label images

opened nuclei and membrane channel as well as grouped labels in napari GUI

measured size, intensity and shape using Regionprops of all frames (nsr)

Performed DR using UMAP

when using Clustering Widget (ncp) to use the GMM I get the following error:
ModuleNotFoundError: No module named 'deprecated'
but the GMM is still applied

using the Plotter Widget (ncp) works as well

visualisation was tested by plotting indexv against frame` and checking label numbers using lasso tool -> works fine

Best,
Mara


upgrade on requests with body

Status Quo
mod_h[ttp]2 supports HTTP/1 protocol upgrades by default auf h2c, if configured, also for h2 upgrades.
One severe restriction to the upgrade is that the request triggering the Upgrade does not have a body, not even a small one. This forces clients to either continue using HTTP/1 (curlfor example) or having to prepend an OPTIONS * request as first on a new connection. This enforces an additional roundtrip to the server, even though the client may have a good idea that HTTP/2 is supported.
The only alternative at the moment is for the client to use h2c direct mode. However, unless support for that is really certain, it lacks a graceful degradation/fallback and will lead to failures and re-attempts.
Additionally, the Apache Serf client, used for Subversion, already sends an OPTIONS request on a new connection. However the subversion protocol requires that request to have a body, forcing it to stay on HTTP/1.1 with the current mod_http2.
Proposal
Implement support for Upgrades with a request body, ideally for bodies of arbitrary length. This would allow clients to Upgrade with certainty on any HTTP/1.1 request, if the servers configuration supports HTTP/2.
If the implementation of arbitrary length Upgrades hits road blocks, such as limitations of existing nghttp2 implementation (unlikely) or exposes the server to unacceptable DOS threats, provide a configurable maximum length setting, up to which upgrade request bodies are accepted.

libnghttp2 does not have any limitation to support HTTP upgrade with POST.
Actually, nghttpx already supports HTTP Upgrade to h2 with POST in any size of body (with some minor limitation, but this is a limitation of nghttpx and not libnghttp2).

Good to know! With limitations I was more concerned about some internal timeouts that maybe would bite me. But am looking forward to implement this...

Abandoned. There is really no one using h2c.


Installation of Build Tools for Visual Studio 2019 failed

Windows 11, Version 21H2 Os Build 22000.493

It helped me to run this as administrator, but then extensions are not visible... Does not work for me either.

windows 11 is having problems,working on windows 10

Try the new release available from Raspberry Pi:
https://github.com/raspberrypi/pico-setup-windows/releases/latest
The Visual Studio dependency has been eliminated now, so this should not be a problem anymore.


Fix for not showing random floor data

Earlier implementation was making all the floor levels visible when clearing map filter, which is not the right thing to do as it leads to unintended results.
The fix here removes clearMapFilter logic to match javascript api implementation

Thanks @eri9000 and @shubham7109. Merging..


how can I reproduce your imageNet AUROC?

First and Foremost , I really appreciate you for sharing code.
Now I try to reproduce ImageNet AUROC 85.7%, but I don't know exact parameter setting(eg  learning rate, epoch etc..)
and also ImageNet dataset downloaded from your github is enough for training? plz let me know~ thanks!

Download the one class ImageNet test set here. The one class ImageNet training set is here.

The training set should be enough for training. I think https://github.com/hendrycks/ss-ood/blob/master/train.py should have reasonable hyperparameters. The code does not have resize prediction, but that only slightly helped performance.


Update Node SDK

Made it a little thinner, still missing a few pieces (built-in signing/OAP/metadata) but in serviceable shape, and tested every call individually.
Oh, and currently, it needs to be installed locally (npm pack the repo followed by installing the tarball). Would be nice to publish it to npm.

Great PullRequest!


Looking for an example of allowNullable()

Trying to use allowNullable() on a Integer query param like this :
Integer foo = context.queryParamAsClass("foo", Integer.class).allowNullable().check((f) -> f >= 1 && f <= 100, "invalid foo").get();
and it's throwing on a null foo, what is the right way to use it?

The nullpointer makes sense (to me at least):
Integer foo = context.queryParamAsClass("foo", Integer.class)
    .allowNullable() // this removes the built in null-checking in the validator
    .check((f) -> f >= 1 && f <= 100, "invalid foo") // f might be null here, so you will get a nullpointer when comparing
    .get();

What are you trying to do?

I added a null check, was just trying to see if there was a more simple usage that I missed.
Integer foo = context.queryParamAsClass("foo", Integer.class)
    .allowNullable() // this removes the built in null-checking in the validator
    .check((f) -> f >= f == null || (f >= 1 && f <= 100, "invalid foo")).get();

You could skip nullable and do getOrDefault? (if I remember the api correctly)

You could skip nullable and do getOrDefault? (if I remember the api correctly)

getOrDefault doesn't exist. https://javadoc.io/doc/io.javalin/javalin/latest/io/javalin/validation/BodyValidator.html
it should be deleted from javalin's web. https://javalin.io/documentation#validation