service_2/evaluate_activation_test/* tests went from flaky to RuntimeError on dartk-mac-debug-x64

The tests
service_2/evaluate_activation_test/instance/dds RuntimeError (expected Pass)
service_2/evaluate_activation_test/instance/service RuntimeError (expected Pass)
service_2/evaluate_activation_test/scope/dds RuntimeError (expected Pass)
service_2/evaluate_activation_test/scope/service RuntimeError (expected Pass)

are failing on configurations
dartk-mac-debug-x64

From the log:
00:14 +0 -1: evaluate_activation_test_instance.dart (VM Service) [E]

  NoSuchMethodError: Class 'DartError' has no instance getter 'valueAsString'.
  Receiver: Instance of 'DartError'
  Tried calling: valueAsString

  dart:core-patch/object_patch.dart 54:5                                                      Object.noSuchMethod
  xcodebuild/DebugX64/generated_tests/service_2/evaluate_activation_test_instance.dart 92:16  testMethod.<fn>
  ===== asynchronous gap ===========================
  dart:async/zone.dart 1286:19                                                                _CustomZone.registerUnaryCallback
  dart:async-patch/async_patch.dart 45:22                                                     _asyncThenWrapperHelper
  xcodebuild/DebugX64/generated_tests/service_2/evaluate_activation_test_instance.dart        testMethod.<fn>
  package:stack_trace/src/stack_zone_specification.dart 126:26                                StackZoneSpecification._registerUnaryCallback.<fn>.<fn>
  package:stack_trace/src/stack_zone_specification.dart 208:15                                StackZoneSpecification._run
  package:stack_trace/src/stack_zone_specification.dart 126:14                                StackZoneSpecification._registerUnaryCallback.<fn>
  dart:async/zone.dart 1362:47                                                                _rootRunUnary
  dart:async/zone.dart 1265:19                                                                _CustomZone.runUnary
  dart:async/zone.dart 1170:7                                                                 _CustomZone.runUnaryGuarded
  dart:async/stream_impl.dart 341:11                                                          _BufferingStreamSubscription._sendData
  dart:async/stream_impl.dart 591:14                                                          _DelayedData.perform
  dart:async/stream_impl.dart 706:11                                                          _StreamImplEvents.handleNext
  dart:async/stream_impl.dart 663:7                                                           _PendingEvents.schedule.<fn>
  package:stack_trace/src/stack_zone_specification.dart 208:15                                StackZoneSpecification._run
  package:stack_trace/src/stack_zone_specification.dart 116:48                                StackZoneSpecification._registerCallback.<fn>
  dart:async/zone.dart 1346:47                                                                _rootRun
  dart:async/zone.dart 1258:19                                                                _CustomZone.run
  dart:async/zone.dart 1162:7                                                                 _CustomZone.runGuarded
  dart:async/zone.dart 1202:23                                                                _CustomZone.bindCallbackGuarded.<fn>
  package:stack_trace/src/stack_zone_specification.dart 208:15                                StackZoneSpecification._run
  package:stack_trace/src/stack_zone_specification.dart 116:48                                StackZoneSpecification._registerCallback.<fn>
  dart:async/zone.dart 1354:13                                                                _rootRun
  dart:async/zone.dart 1258:19                                                                _CustomZone.run
  dart:async/zone.dart 1162:7                                                                 _CustomZone.runGuarded
  dart:async/zone.dart 1202:23                                                                _CustomZone.bindCallbackGuarded.<fn>
  dart:async/schedule_microtask.dart 40:21                                                    _microtaskLoop
  dart:async/schedule_microtask.dart 49:5                                                     _startMicrotaskLoop
  dart:isolate-patch/isolate_patch.dart 120:13                                                _runPendingImmediateCallback
  dart:isolate-patch/isolate_patch.dart 185:5                                                 _RawReceivePortImpl._handleMessage

Logs:
https://dart-ci.appspot.com/log/vm-kernel-mac-debug-x64/dartk-mac-debug-x64/11311/service_2/evaluate_activation_test/instance/dds
https://dart-ci.appspot.com/log/vm-kernel-mac-debug-x64/dartk-mac-debug-x64/11311/service_2/evaluate_activation_test/instance/service
https://dart-ci.appspot.com/log/vm-kernel-mac-debug-x64/dartk-mac-debug-x64/11311/service_2/evaluate_activation_test/scope/dds
https://dart-ci.appspot.com/log/vm-kernel-mac-debug-x64/dartk-mac-debug-x64/11311/service_2/evaluate_activation_test/scope/service
@bkonyi @rmacnak-google Could you please take a look?

Same for dartk-mac-release-x64 today.
https://dart-ci.appspot.com/log/vm-kernel-mac-release-x64/dartk-mac-release-x64/14869/service_2/evaluate_activation_test/instance/dds
  NoSuchMethodError: Class 'DartError' has no instance getter 'valueAsString'.
  Receiver: Instance of 'DartError'
  Tried calling: valueAsString

It looks like these tests don't pass on any configurations. I'm thinking some behavior changed around the Dart 2 migration and that these tests aren't correct anymore. I'm going to skip these for now and reference this issue.

Same on dartk-linux-release-simarm today.

Same happened on dartk-linux-release-arm64.
Log: https://dart-ci.appspot.com/log/cross-vm-linux-release-arm64/dartk-linux-release-arm64/9827/service_2/evaluate_activation_test/scope/dds

Same on dartk-win-debug-x64c:
https://dart-ci.appspot.com/log/vm-kernel-win-debug-x64c/dartk-win-debug-x64c/488/service_2/evaluate_activation_test/scope/service

flaky -> runtime error: https://dart-ci.appspot.com/log/vm-kernel-checked-linux-release-x64/dartk-checked-linux-release-x64/14967/service_2/evaluate_activation_test/instance/service

Those tests are testing that the debugger can - when breaking at a function - access the state of all variables that are in the lexical scope (including in outer scopes from inner closures).
For example the test has this:
breakHere() {}
use(dynamic v) => v;

class C {
  var instVar = 1;
  static var classVar = 2;

  method(methodParam) {
    var methodTemp = 4;
    use(methodTemp);
    [5].forEach((outerParam) {
      var outerTemp = 6;
      use(outerTemp);
      [7].forEach((innerParam) {
        var innerTemp = 8;
        use(innerTemp);
        breakHere();                          // <-- Assumes we can access `this`, even the Dart code doesn't need it.
      });
    });
  }

It seems to me that we're not willing to actually to keep everything alive - just for the possibility of a debugger later attaching & examining the objects. i.e. we prefer to keep less memory alive and have faster code, even in JIT, even in unoptimized code.
If you all agree (/cc @mraleph @alexmarkov @a-siva @rmacnak-google), then I think the test should be re-written to expect the behavior we actually have implemented or want to implement.
(Right now the relevant subtests are failing and are therefore of little use. But we spent SWE time on gardening, looking at flakes, ...)
Related: https://github.com/dart-lang/sdk/issues/20047

I tried this test under devtools interactively and I noticed that when stopped at breakHere in method the 'Variables' window only shows innerTemp and innerParam I would have expected it to at least state that this, methodParam, outerTemp and methodTemp are 'optimized out'.
I agree that we should change the test to reflect this fact that certain values are optimized out.

The test should probably be adjusted to reflect the current implementation, however there is a deeper question of developer experience here and how to balance it with our desire for good performance (and predictability) of JIT.
It is hard to argue with the fact that the current behaviour is unfriendly - as a user I would expect to be able to inspect more variables than compiler lets me.
Maybe there is some improvements we could make in this area - though it is not entirely clear to me how. Maybe an adaptive strategy is possible where if you put a breakpoint into a closure then we recompile things in such a way that instances of the closure which are allocated after the breakpoint was put there capture more information?
We can also probably have a mode in which we disable some of JIT optimizations (like dead variable pruning).
We could even consider reifying / capturing the whole stack frame all the time - though that would introduce considerable difference in GC behavior between JIT and AOT.

My 2 cents: if a variable isn't used in a scope and is being optimized out by the compiler as a result, is there any benefit in making it inspectable? If we don't currently report variables that are optimized out as optimized out we should at least make sure that we do that, but I doubt it's worth the effort to try and display their values in the debugger.

The test service/evaluate_activation_test/instance/service went flaky -> RuntimeError on vm-linux-release-x64 configuration:
00:13 +0 -1: evaluate_activation_test_instance.dart (VM Service) [E]

  Expected: '123'
    Actual: '<optimized out>'
     Which: is different.
            Expected: 123
              Actual: <optimized ...
                      ^
             Differ at offset 0

  package:matcher/src/expect/expect.dart 149:31                                       fail
  package:matcher/src/expect/expect.dart 144:3                                        _expect
  package:matcher/src/expect/expect.dart 56:3                                         expect
  out/ReleaseX64/generated_tests/service/evaluate_activation_test_instance.dart 87:5  testMethod.<fn>

Result Instance(null)

  Expected: true
    Actual: <false>

  package:matcher/src/expect/expect.dart 149:31                                        fail
  package:matcher/src/expect/expect.dart 144:3                                         _expect
  package:matcher/src/expect/expect.dart 56:3                                          expect
  out/ReleaseX64/generated_tests/service/evaluate_activation_test_instance.dart 114:3  testMethod
  ===== asynchronous gap ===========================
  out/ReleaseX64/generated_tests/service/test_helper.dart 489:17                       _ServiceTesterRunner.run.runTest.<fn>.<fn>
  ===== asynchronous gap ===========================
  out/ReleaseX64/generated_tests/service/test_helper.dart 369:9                        _ServiceTesterRunner.run.ignoreLateException
  ===== asynchronous gap ===========================
  package:test_api/src/backend/declarer.dart 215:9                                     Declarer.test.<fn>.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/declarer.dart 213:7                                     Declarer.test.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/invoker.dart 258:9                                      Invoker._waitForOutstandingCallbacks.<fn>

https://dart-ci.appspot.com/log/vm-linux-release-x64/vm-linux-release-x64/530/service/evaluate_activation_test/instance/service

This happened again today. @aam would you be interested in looking at these expression evaluation tests?

Flakiness caused by these tests timing out once in a while.  For example,
https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8780081974082306833/+/u/test_results/ignored_flaky_test_failure_logs
The test never reaches https://github.com/dart-lang/sdk/blob/d083c636850a7ecc9a5f84ad2d1db3593742a532/runtime/observatory/tests/service/evaluate_activation_test.dart#L113, doesn't exit from 'rootLib.evaluate'.
Might be the case of an overloaded bot.
/=====================================================================================================\
| service/evaluate_activation_test/instance/dds failed again (RuntimeError -> Timeout, expected Pass) |
\=====================================================================================================/

--- Command "vm" (took 01:01.000386s):
DART_CONFIGURATION=ReleaseX64 out/ReleaseX64/dart -DUSE_DDS=true --sound-null-safety -Dtest_runner.configuration=vm-linux-release-x64 --ignore-unrecognized-flags --packages=/b/s/w/ir/.dart_tool/package_config.json /b/s/w/ir/out/ReleaseX64/generated_tests/service/evaluate_activation_test_instance.dart

exit code:
1

diagnostics:
Process list including children: [251745, 251830]
Trying to capture stack trace for pid 251745
PID 251745 - process
TID 251745:
#0  0x00007f5525b4d376 pthread_cond_wait@@GLIBC_2.3.2
#1  0x00005634f3c1eb77 dart::Monitor::WaitMicros(long)
#2  0x00005634f408b14a Dart_RunLoop
#3  0x00005634f39d39c2 dart::bin::RunMainIsolate(char const*, char const*, bool, dart::bin::CommandLineOptions*)
#4  0x00005634f39d4720 dart::bin::main(int, char**)
#5  0x00005634f39d371d main
#6  0x00007f5525821083 __libc_start_main
#7  0x00005634f39cd76a _start
TID 251751:
#0  0x00007f552591c46e epoll_wait
#1  0x00005634f39e2ed8 dart::bin::EventHandlerImplementation::Poll(unsigned long)
#2  0x00005634f3add4a2 dart::bin::ThreadStart(void*)
#3  0x00007f5525b46609 start_thread
#4  0x00007f552591c133 __clone
TID 251831:
#0  0x00007f55258dfc7f wait4
#1  0x00005634f39ea118 dart::bin::ExitCodeHandler::ExitCodeHandlerEntry(unsigned long)
#2  0x00005634f3add4a2 dart::bin::ThreadStart(void*)
#3  0x00007f5525b46609 start_thread
#4  0x00007f552591c133 __clone
TID 251933:
#0  0x00007f5525b4d7d1 pthread_cond_timedwait@@GLIBC_2.3.2
#1  0x00005634f3c1eb5f dart::Monitor::WaitMicros(long)
#2  0x00005634f3b661dd dart::MutatorThreadPool::OnEnterIdleLocked(dart::MonitorLocker*)
#3  0x00005634f3c9b15d dart::ThreadPool::WorkerLoop(dart::ThreadPool::Worker*)
#4  0x00005634f3c9b688 dart::ThreadPool::Worker::Main(unsigned long)
#5  0x00005634f3c1e406 dart::ThreadStart(void*)
#6  0x00007f5525b46609 start_thread
#7  0x00007f552591c133 __clone
TID 251934:
#0  0x00007f5525b4d7d1 pthread_cond_timedwait@@GLIBC_2.3.2
#1  0x00005634f3c1eb5f dart::Monitor::WaitMicros(long)
#2  0x00005634f3c9b2ff dart::ThreadPool::WorkerLoop(dart::ThreadPool::Worker*)
#3  0x00005634f3c9b688 dart::ThreadPool::Worker::Main(unsigned long)
#4  0x00005634f3c1e406 dart::ThreadStart(void*)
#5  0x00007f5525b46609 start_thread
#6  0x00007f552591c133 __clone
TID 251950:
#0  0x00007f5525b4d7d1 pthread_cond_timedwait@@GLIBC_2.3.2
#1  0x00005634f3c1eb5f dart::Monitor::WaitMicros(long)
#2  0x00005634f3c9b2ff dart::ThreadPool::WorkerLoop(dart::ThreadPool::Worker*)
#3  0x00005634f3c9b688 dart::ThreadPool::Worker::Main(unsigned long)
#4  0x00005634f3c1e406 dart::ThreadStart(void*)
#5  0x00007f5525b46609 start_thread
#6  0x00007f552591c133 __clone
TID 251985:
#0  0x00007f5525b4d7d1 pthread_cond_timedwait@@GLIBC_2.3.2
#1  0x00005634f3c1eb5f dart::Monitor::WaitMicros(long)
#2  0x00005634f3c9b2ff dart::ThreadPool::WorkerLoop(dart::ThreadPool::Worker*)
#3  0x00005634f3c9b688 dart::ThreadPool::Worker::Main(unsigned long)
#4  0x00005634f3c1e406 dart::ThreadStart(void*)
#5  0x00007f5525b46609 start_thread
#6  0x00007f552591c133 __clone

Trying to capture stack trace for pid 251830
PID 251830 - process
TID 251830:
#0  0x00007f5e8d68a376 pthread_cond_wait@@GLIBC_2.3.2
#1  0x0000559f52ecbb77 dart::Monitor::WaitMicros(long)
#2  0x0000559f5333814a Dart_RunLoop
#3  0x0000559f52c809c2 dart::bin::RunMainIsolate(char const*, char const*, bool, dart::bin::CommandLineOptions*)
#4  0x0000559f52c81720 dart::bin::main(int, char**)
#5  0x0000559f52c8071d main
#6  0x00007f5e8d35e083 __libc_start_main
#7  0x0000559f52c7a76a _start
TID 251832:
#0  0x00007f5e8d45946e epoll_wait
#1  0x0000559f52c8fed8 dart::bin::EventHandlerImplementation::Poll(unsigned long)
#2  0x0000559f52d8a4a2 dart::bin::ThreadStart(void*)
#3  0x00007f5e8d683609 start_thread
#4  0x00007f5e8d459133 __clone
TID 251833:
#0  0x00007f5e8d68a376 pthread_cond_wait@@GLIBC_2.3.2
#1  0x0000559f52ecbb77 dart::Monitor::WaitMicros(long)
#2  0x0000559f52f4760f dart::ThreadInterrupter::ThreadMain(unsigned long)
#3  0x0000559f52ecb406 dart::ThreadStart(void*)
#4  0x00007f5e8d683609 start_thread
#5  0x00007f5e8d459133 __clone
TID 251834:
#0  0x00007f5e8d68a7d1 pthread_cond_timedwait@@GLIBC_2.3.2
#1  0x0000559f52ecbb5f dart::Monitor::WaitMicros(long)
#2  0x0000559f52ed1d15 dart::SampleBlockProcessor::ThreadMain(unsigned long)
#3  0x0000559f52ecb406 dart::ThreadStart(void*)
#4  0x00007f5e8d683609 start_thread
#5  0x00007f5e8d459133 __clone
TID 251835:
#0  0x00007f5e8d68a7d1 pthread_cond_timedwait@@GLIBC_2.3.2
#1  0x0000559f52ecbb5f dart::Monitor::WaitMicros(long)
#2  0x0000559f52f482ff dart::ThreadPool::WorkerLoop(dart::ThreadPool::Worker*)
#3  0x0000559f52f48688 dart::ThreadPool::Worker::Main(unsigned long)
#4  0x0000559f52ecb406 dart::ThreadStart(void*)
#5  0x00007f5e8d683609 start_thread
#6  0x00007f5e8d459133 __clone
TID 251844:
#0  0x00007f5e8d68a7d1 pthread_cond_timedwait@@GLIBC_2.3.2
#1  0x0000559f52ecbb5f dart::Monitor::WaitMicros(long)
#2  0x0000559f52f482ff dart::ThreadPool::WorkerLoop(dart::ThreadPool::Worker*)
#3  0x0000559f52f48688 dart::ThreadPool::Worker::Main(unsigned long)
#4  0x0000559f52ecb406 dart::ThreadStart(void*)
#5  0x00007f5e8d683609 start_thread
#6  0x00007f5e8d459133 __clone
TID 251845:
#0  0x00007f5e8d68a7d1 pthread_cond_timedwait@@GLIBC_2.3.2
#1  0x0000559f52ecbb5f dart::Monitor::WaitMicros(long)
#2  0x0000559f52f482ff dart::ThreadPool::WorkerLoop(dart::ThreadPool::Worker*)
#3  0x0000559f52f48688 dart::ThreadPool::Worker::Main(unsigned long)
#4  0x0000559f52ecb406 dart::ThreadStart(void*)
#5  0x00007f5e8d683609 start_thread
#6  0x00007f5e8d459133 __clone
TID 251847:
#0  0x00007f5e8d68a7d1 pthread_cond_timedwait@@GLIBC_2.3.2
#1  0x0000559f52ecbb5f dart::Monitor::WaitMicros(long)
#2  0x0000559f52f482ff dart::ThreadPool::WorkerLoop(dart::ThreadPool::Worker*)
#3  0x0000559f52f48688 dart::ThreadPool::Worker::Main(unsigned long)
#4  0x0000559f52ecb406 dart::ThreadStart(void*)
#5  0x00007f5e8d683609 start_thread
#6  0x00007f5e8d459133 __clone
TID 251848:
#0  0x00007f5e8d68a7d1 pthread_cond_timedwait@@GLIBC_2.3.2
#1  0x0000559f52ecbb5f dart::Monitor::WaitMicros(long)
#2  0x0000559f52f482ff dart::ThreadPool::WorkerLoop(dart::ThreadPool::Worker*)
#3  0x0000559f52f48688 dart::ThreadPool::Worker::Main(unsigned long)
#4  0x0000559f52ecb406 dart::ThreadStart(void*)
#5  0x00007f5e8d683609 start_thread
#6  0x00007f5e8d459133 __clone
TID 251899:
#0  0x00007f5e8d68a7d1 pthread_cond_timedwait@@GLIBC_2.3.2
#1  0x0000559f52ecbb5f dart::Monitor::WaitMicros(long)
#2  0x0000559f52e131dd dart::MutatorThreadPool::OnEnterIdleLocked(dart::MonitorLocker*)
#3  0x0000559f52f4815d dart::ThreadPool::WorkerLoop(dart::ThreadPool::Worker*)
#4  0x0000559f52f48688 dart::ThreadPool::Worker::Main(unsigned long)
#5  0x0000559f52ecb406 dart::ThreadStart(void*)
#6  0x00007f5e8d683609 start_thread
#7  0x00007f5e8d459133 __clone
TID 251930:
#0  0x00007f5e8d68a7d1 pthread_cond_timedwait@@GLIBC_2.3.2
#1  0x0000559f52ecbb5f dart::Monitor::WaitMicros(long)
#2  0x0000559f52e131dd dart::MutatorThreadPool::OnEnterIdleLocked(dart::MonitorLocker*)
#3  0x0000559f52f4815d dart::ThreadPool::WorkerLoop(dart::ThreadPool::Worker*)
#4  0x0000559f52f48688 dart::ThreadPool::Worker::Main(unsigned long)
#5  0x0000559f52ecb406 dart::ThreadStart(void*)
#6  0x00007f5e8d683609 start_thread
#7  0x00007f5e8d459133 __clone
TID 251940:
#0  0x00007f5e8d68a7d1 pthread_cond_timedwait@@GLIBC_2.3.2
#1  0x0000559f52ecbb5f dart::Monitor::WaitMicros(long)
#2  0x0000559f52e131dd dart::MutatorThreadPool::OnEnterIdleLocked(dart::MonitorLocker*)
#3  0x0000559f52f4815d dart::ThreadPool::WorkerLoop(dart::ThreadPool::Worker*)
#4  0x0000559f52f48688 dart::ThreadPool::Worker::Main(unsigned long)
#5  0x0000559f52ecb406 dart::ThreadStart(void*)
#6  0x00007f5e8d683609 start_thread
#7  0x00007f5e8d459133 __clone

stdout:
00:00 +0: evaluate_activation_test_instance.dart (DDS)

** Launching SERVICE_TEST_TESTEE=true out/ReleaseX64/dart --disable-dart-dev --write-service-info=file:///b/s/w/itia2w5c01/dart_serviceCRVJLR/service_info.json --profiler -DUSE_DDS=true --sound-null-safety -Dtest_runner.configuration=vm-linux-release-x64 --ignore-unrecognized-flags --packages=/b/s/w/ir/.dart_tool/package_config.json --enable-vm-service:0 /b/s/w/ir/out/ReleaseX64/generated_tests/service/evaluate_activation_test_instance.dart

>testee>out> The Dart VM service is listening on http://<IP_ADDRESS>:38003/MgBCEZVA8TM=/

>testee>out>

** Signaled to run test queries on http://<IP_ADDRESS>:41239/ae3awYWXKh0=/ (DDS)

Loading VM...

Done loading VM

Running evaluate_activation_test_instance.dart [1/4]

Breakpoint: Breakpoint 1 at evaluate_activation_test_instance.dart:token(374)[unresolved]

Event ServiceEvent(owner='isolates/830363021983679', kind='BreakpointResolved', time=2023-05-25 16:52:46.989)

Event ServiceEvent(owner='isolates/830363021983679', kind='PauseBreakpoint', time=2023-05-25 16:52:46.990)

00:36 +0 -1: evaluate_activation_test_instance.dart (DDS) [E]

  Expected: '123'
    Actual: '<optimized out>'
     Which: is different.
            Expected: 123
              Actual: <optimized ...
                      ^
             Differ at offset 0

--- Re-run this test:
python3 tools/test.py -n vm-linux-release-x64 service/evaluate_activation_test/instance/dds

Reapproved again today.


Add support for node debugger

When node-debug flag is set to true in server config, the js module will start with --inspect command line flag (debugger)

I will do some refactoring to how the CLI args are passed, but ty for the PR


implict import a module named test

after I dive to py3 using futurize, I find that my module name "test.py" will automaitcally be imported. I found out it's because the below code:
from future import standard_library
standard_library.install_aliases()

which write:
try:
    import test
except ImportError:
    pass

as implict relative import is allowed in py2, it imports my "test.py" instead of the standard test module

I'm hitting this problem too.   Is there no way to fix it?

Is anybody working on a fix for this. We just encountered this problem too.

Duplicate #268


Doc update

Coverage remained the same at 56.477% when pulling<PHONE_NUMBER>5a57545ee3236adf1b65189cbe6db4 on zhicaizack:DocUpdate into 1f66994b79d9158994bfb53939d3dc12ed80c2c5 on CS2113-AY1819S1-T16-1:master.


Guys need help fastlane

Hello all. Need help, tell me please how I can write password for match deliver pem and etc from Fastfile without keychain?

Hey @GandjaFuzz could you describe the setup you're using and what you're trying to accomplish?
In most cases, you should be able to create a temporary keychain: https://docs.fastlane.tools/actions/create_keychain/

I receive from server login and password for release apps. and me need in cycle setup login and pass for fast lane.

if I create keychain how I can automatic programmatically add in keychain login with password for fast lane delivery etc


Add nn.temporalConvolution and nn.Linear correction.

Just want to include nn.TemporalConvolution() for your wonderful initialization implementation, plus a little correction on the nn.Linear layer (since weight.size(1) is the fanout and weight.size(2) is the fanin, am I right?). See if this is ok for use and merge.

yes great, thanks!


Packages with same names across different buckets.

I have the same package across 2 different buckets. Is there any way to choose which bucket to install the package from, and if not could it be implemented. The reason I can't just remove the first bucket and install it that way is because I need that repo to learn how to write manifests for scoop but I also need to be able to install the applications I need from the 'extras' bucket.

scoop install extras/reaper


Deployment Issue- I have used the code and deploy on the server, where internet is not running

I have down loaded the below JS add in our source code-

But now extracting the SCORM file it is sowing below error

Please help me out from this issue.

Your server code is missing the Altova.dll file. This file is part of the .net core solution. You have to build the solution with all the projects included. See the screenshot that shows all the projects in the solution:


Advanced configurations | External users

Hello,
We try to apply Conditional Access policy for SharePoint, this is done in tenant level and applied "Allow-limited, web-only access" and then we run this cmdlet: Set-SPOTenant -ApplyAppEnforcedRestrictionsToAdHocRecipients $false
Still external users after 48 hours not able to download files.
Can u please confirm how this cmdlet should work?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 2f2abda5-9273-1a56-91b6-e4e77d5706b0
Version Independent ID: 6a85f2a5-b892-55f7-b49a-4ab15f1e8b73
Content: Control access from unmanaged devices
Content Source: SharePoint/SharePointOnline/control-access-from-unmanaged-devices.md
Service: sharepoint-online
GitHub Login: @kaarins
Microsoft Alias: kaarins

Have you configured the Conditional Access Policy in Azure?

Yes, it is created automatically if we enable limited access in SharePoint admin center.
Sent from mobile phone

On 8 Nov 2019, at 17:41, Trevor Seward<EMAIL_ADDRESS>wrote:

Have you configured the Conditional Access Policy in Azure?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

Hello,
I think that besides the PS command, external users must be also excluded from Conditional Access policies.
Just tested this and then my test external and guest users start to work for download:

If this is a case, the article should be updated accordingly.
So in short:

If scenario apply to SPO level and Device access enabled: "Allow-limited, web-only access"
Run PS: Set-SPOTenant -ApplyAppEnforcedRestrictionsToAdHocRecipients $false
Exclude Gues/External from Conditional Access policies

Hello,
Small update, after playing with Conditional Access policies I got this now:

Existing in AAD Guest users - limited to download
Shared files to Ad-Hoc users - allowed to download

So like now I got exactly how it should work: "All users limited download, but Ad-Hoc can download".
What I did is:

Changed Conditional Access policy from Include All users -> to only my user (with which was testing)
Changed Conditional Access policy back to Include All users

Just in case will test tomorrow, but I have a feeling that CA stuck somehow and should be like "re-changed"......
Still, this will provde that it is working like it should, but maybe an FAQ could be handy for this behavior :)

Thank you for the additional details. It is important to make this feature set clearer and for there to be more testing by the product group to ensure the configuration is consistent with the intent. For you to have to change it from a group with one user, then back just sounds like this feature is flawed.

@lightupdifire Thank you for submitting feedback. We understand that this issue has been resolved.
Please feel free to re-open this issue if there is a specific area of the docs that we can improve or make better. Thank you.

Hello Manuela,
Can you take a closer look at this issue?
I understand that you may want to close it, maybe you get credit for closures, but if you haven't fully considered @lightupdifire's comments he is basically explaining that the article does not work as described. There are bugs that he was want to work through. The article needs more work.

From: Manuela Scanu<EMAIL_ADDRESS>Sent: Wednesday, November 13, 2019 3:40 PM
To: MicrosoftDocs/OfficeDocs-SharePoint<EMAIL_ADDRESS>Cc: Jmartinez589<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/OfficeDocs-SharePoint] Advanced configurations | External users (#1505)
Closed #1505https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2FOfficeDocs-SharePoint%2Fissues%2F1505&data=02|01||854edc6ed4554d4c7ffb08d76892df94|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637092852326083796&sdata=uuULm10ZKuESH3ZdPObCEIJpz4935HTWEbnju43q8fA%3D&reserved=0.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2FOfficeDocs-SharePoint%2Fissues%2F1505%3Femail_source%3Dnotifications%26email_token%3DAHCLP4DEWYGQIZVK6PBDZG3QTSF67A5CNFSM4JKFAXD2YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOU25ALFI%23event-2797208981&data=02|01||854edc6ed4554d4c7ffb08d76892df94|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637092852326093801&sdata=k9ybyoSBMPlmmziZMpDu2iF0KPk2FWuxxrngrcbZt%2FY%3D&reserved=0, or unsubscribehttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAHCLP4FEN42UJMH4C77PQSLQTSF67ANCNFSM4JKFAXDQ&data=02|01||854edc6ed4554d4c7ffb08d76892df94|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637092852326113829&sdata=QI5EdDF%2BWZZRSffS80EE9554tYkZzVhmqEuXShI9uNs%3D&reserved=0.


we have a issue in azure devops release pipeline for bigcommerce cli application(java)

Note
Issues in this repo are for tracking bugs, feature requests and questions for the tasks in this repo
For a list:
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks
If you have an issue or request for the Azure Pipelines service, use developer community instead:
https://developercommunity.visualstudio.com/spaces/21/index.html )
Required Information
Entering this information will route you directly to the right team and expedite traction.
Question, Bug, or Feature?
Type: here
Enter Task Name: here
list here (V# not needed):
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks
Environment

Server - Azure Pipelines or TFS on-premises?

If using TFS on-premises, provide the version:

If using Azure Pipelines, provide the account name, team project name, build definition name/build number:

Agent - Hosted or Private:

If using Hosted agent, provide agent queue name:

If using private agent, provide the OS of the machine running the agent and the agent version:

Issue Description
[Include task name(s), screenshots and any other relevant details]
Task logs
[Enable debug logging and please provide the zip file containing all the logs for a speedy resolution]
Troubleshooting
Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting
Error logs
[Insert error from the logs here for a quick overview]

getting below error while release pipline(upload the theme to big commerce site)
2019-05-20T10:51:21.7604467Z SCSS Parsing Started...
2019-05-20T10:51:21.7742756Z Template Parsing Started...
2019-05-20T10:51:21.7743791Z Language Files Parsing Started...
2019-05-20T10:51:21.7744448Z Building Theme Schema File...
2019-05-20T10:51:21.9155495Z Theme task Started...
2019-05-20T10:51:21.9545215Z /home/testadmin/vstsagent/_work_mount/_tool/node/8.16.0/x64/lib/node_modules/@bigcommerce/stencil-cli/lib/stencil-bundle.js:113
2019-05-20T10:51:21.9566881Z             const filterFiles = files.filter((file) => {
2019-05-20T10:51:21.9567912Z                                       ^
2019-05-20T10:51:21.9568394Z
2019-05-20T10:51:21.9569371Z TypeError: Cannot read property 'filter' of undefined
2019-05-20T10:51:21.9571253Z     at Fs.readdir (/home/testadmin/vstsagent/_work_mount/_tool/node/8.16.0/x64/lib/node_modules/@bigcommerce/stencil-cli/lib/stencil-bundle.js:113:39)
2019-05-20T10:51:21.9572018Z     at FSReqWrap.oncomplete (fs.js:135:15)
2019-05-20T10:51:23.6202082Z ##[error]Bash exited with code '1'.
please help me on this issuse

The issue template wasn't filled out with any details and the small snippet is not enough to go on.  It also appears to be a custom bash script.   If the bug is in a task in this repo, log an issue with the template filled out and we will route.


Release: hwc-buildpack (Jul)

Release reminder for hwc-buildpack

See diff from latest version and validate if a release is required.
Make sure the latest commit on  has passed tests on the CI
Refer release instructions. (private link)

Nothing to release.


DocsArchive-<IP_ADDRESS>Auto archive content

Auto archive content from https://github.com/OPS-E2E-PPE/docs-archive-test-source

Docs Build status updates of commit 94d680e:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error]

Cannot sync git repo to specified commit because branch Release_Archive_master_2020-04-20-06-17-02 has been deleted or has been force pushed: fatal: couldn't find remote ref refs/heads/Release_Archive_master_2020-04-20-06-17-02

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Unsure about GetAllState

Hi there, correct me if I'm wrong, but GetAllState wont actually 'copy' the state if it contains any reference types, thus you gain no immutability.
// GetAllState return a full copy of the current state.
func (s *Store) GetAllState() interface{} {
    ret := map[string]interface{}{}

    s.storeStateLock.RLock()
    for k, v := range s.storeState.state {
        ret[k] = v  // <- This will only copy the address of v if v is a map, slice, or interface.
    }
    s.storeStateLock.RUnlock()

    return ret  // <- If someone mutates this variable later on it can potentially mutate the original.
}

Perhaps use reflect.DeepCopy or similar for true immutability.
Otherwise it would be good to document this potential side effect.
What are your thoughts?

Good point! The concept about immutability is very important concerning Redux. In Golang you also need to account for race conditions.


Update to support Gammapy v1.2

Fixes #152
Changes proposed in this pull request:

Update various dependency versions
Remove redundant features in asgardpy

TBD:

Check pydantic's custom serializer to replace json_encoders in the BaseConfig definition.
Wait for autodoc_pydantic's new support or use alternative for the Sphinx docs
Try to have better definition for Time Intervals
Look into unifying the astropy Quantity types
Usage of Gammapy Metadata and other new features as per the Release Notes
Remove changes made in #160
Revisit the changes made in #132

It is better to update the validation and serialization effort on the usage of the new pydantic dependency in a separate PR.


Ease running tests in development with different db motors using docker-compose

Currently, running tests with a specific db it's a bit unconfortable as to deploy a db using the scripts, one needs to remember which environment variables needs to be set.
This can be easily solved by adding a docker-compose that has the same set up as the scripts.

Fixed by #55


Allow for n satellites propogation

The code base currently support propagation of 4 satellites.
Expand the capability to provide results for n many satellites.

This is closed by #8 .


Finalize experience calculations

Need to finalize the calculation of experience and levels before Mainnet deployment

Added this in a7cd30


What is x1,y1,x2,y2 in sample_parser.py?

Is x1=xmin,y1=ymin,x2=xmax,y2=ymax as given in pascal voc format? Is it possible to train on own images using pascal voc format for the annotations rather than the .txt file?

yes, you are right about x1,y1,x2,y2. To put it simply those are the coordinates of the leading diagonal of the bbox.
And it is possible to train own images using pascal voc format for annotations . You must modify pascal_voc_parser accordingly. But I don't see why you would do that. Using the text file is easier compared to that

@Abhijit-2592 , Thanks for your reply!


⚠ SceneTime has degraded performance

In 5c117e8, SceneTime ($ST) experienced degraded performance:

HTTP code: 200
Response time: 2094 ms

Resolved: SceneTime performance has improved in b040e69 after 7 minutes.


Allow optional content on completed transaction pg

https://trello.com/c/caDndyIX
As a Service Manager
I want to be able to indicate whether Organ Donor Registration promotion
should be incorporated into my "Completed transaction" page
so that I can offer users of my service to register for Organ Donation
In order to avoid adding more fields to Edition for such requirements, we're storing information on one Edition field: presentation_toggles. It is a Hash containing a boolean which toggles display of this field, and a URL which the promotion links to. Going forward if we have more such requirements, we can dump it into the presentation_toggles Hash.
PresentationToggles includes that new field and adds getter/setter methods on the Edition, to make it behave as if PresentationToggles are multiple fields on the Edition. This helps with form submissions.

I would perhaps shorten organ_donor_registration_promotion to organ_donor_promotion, but that is probably a lot of change to make at this stage, so I'll leave it up to you.

we've shortened the name by using an active voice as @fofr suggested: promote_organ_donor_registration.


Spring-Boot camel quickstart does not run locally (not deployed to local f8)

The command mvn spring-boot:run fails with the stack trace as can be seen here -> http://pastebin.com/UmvGB9fF
There is a need for a Hibernate RI Validator reported by the stack trace.

Thanks for reporting, I get the same error


polyOsc vscode example project

Setup right now for linux, we can add different configurations for OSX and Windows.

Since we have vscode support built into everything I'm going to close this PR.

Since we have vscode support built into everything I'm going to close this PR.


Typescript 2.4 breaks support of enum

With 2.4, Typescript add some check about weak types.
See the related changelog.
If we look at the deserializeAs annotation signature
export declare function deserializeAs(keyNameOrType: string | Function | ISerializable, keyName?: string): any;

And its usage with enums

// in my.enum.ts
export enum MyEnum { foo, bar }

SerializableEnumeration(MyEnum);

// in my.model.ts
export class MyModel {

    @deserializeAs(MyEnum)
    typeEnum: MyEnum;
}

MyEnum is supposed to be an ISerializable. With SerializableEnumeration(), it is, at runtime. But at compilation type, Typescript has no idea about this extension and the new weak type checking throws.
Typescript proposes a workaround at the end of the section.

Since this is a breaking change, you may need to know about the workarounds which are the same as those for strict object literal checks:

Declare the properties if they really do exist.
Add an index signature to the weak type (i.e. [propName: string]: {}).
Use a type assertion (i.e. opts as Options).

HI, I'm new to typescript and all. I've got this issue now with my enum annotated with serializeauto. Is there a workaround or we have to wait for an updated version of the lib for that issue?
Thanks!

@ThePainnn
You can go back to TypeScript 2.3 until @weichx looks at the problem.

I'll have to think about how to handle this, I'm not sure off the top of my head

I think the only solution is have SerializableEnum return an indexable mapped type. I forget exactly how mapped types work but it would something like this:
function SerializableEnum<T extends Indexable<number|string>>(_enum : T) : {[K in keyof T] : number|string}

const EnumType = SerializableEnum<MyEnum>(enum Thing { ... });

export class MyModel {
    @deserializeAs(EnumType) typeEnum: EnumType;
}

Thoughts?

+1 Look forward to seeing how this is resolved ;-)

Any resolution on this? Thanks.

Should be fixed by #71


Duplicated generations

I think you should only show unique generations in such cases.

Nice suggestion!
https://github.com/ricklamers/shell-ai/commit/b78eccabbae6e1188774ff24987db904d39f687d
PS don't do something crazy


JS version of int.Parse fails on some input data which are parsed fine in C#

Hi, guys.
I'm find that the int.Parse() method can not parse some input data in JS code that considering as valid in C# code.
This is a case where we pass string containing trailing or leading space characters.
Steps To Reproduce
Actual - Deck.net: https://deck.net/4bd94206f415b22c297e7e5d511bf20f
Expected - .NET Fiddle: https://dotnetfiddle.net/bd1mAM
public class Program
{
    public static void Main()
    {
        Console.WriteLine(int.Parse(" 1"));
	Console.WriteLine(int.Parse("2 "));
	Console.WriteLine(int.Parse(" 3 "));

	Console.WriteLine(int.Parse("\t4"));
        Console.WriteLine(int.Parse("5\t"));
	Console.WriteLine(int.Parse("\t6\t"));

	Console.WriteLine(int.Parse("\r\n7"));
	Console.WriteLine(int.Parse("8\n"));
	Console.WriteLine(int.Parse("\r\n9\n"));
    }
}

Expected Result
1
2
3
4
5
6
7
8
9

Actual Result
System.Exception: Uncaught System.FormatException: Input string was not in a correct format.
Error: Input string was not in a correct format.
    at $ctor1.ctor (https://deck.net/resources/js/bridge/bridge.min.js?17.2.0:7:91540)
    at $ctor1 (https://deck.net/resources/js/bridge/bridge.min.js?17.2.0:7:92638)
    at new $ctor1 (https://deck.net/resources/js/bridge/bridge.min.js?17.2.0:7:579885)
    at Function.parseInt (https://deck.net/resources/js/bridge/bridge.min.js?17.2.0:7:121516)
    at Function.parse (https://deck.net/resources/js/bridge/bridge.min.js?17.2.0:7:113509)
    at Function.Main (https://deck.net/RunHandler.ashx?h=1987598395:9:51)
    at https://deck.net/resources/js/bridge/bridge.min.js?17.2.0:7:42576
    at HTMLDocument.i (https://deck.net/resources/js/bridge/bridge.min.js?17.2.0:7:6515)
    at i (https://cdnjs.cloudflare.com/ajax/libs/jquery/2.2.4/jquery.min.js:2:27151)
    at Object.add [as done] (https://cdnjs.cloudflare.com/ajax/libs/jquery/2.2.4/jquery.min.js:2:27450)

Hi @serge-shaldanov – Thanks for the report and excellent samples demonstrating how to reproduce the issue. We will review and attempt to fix right away.

Fixed! The fix will make it to the next Bridge release! Check milestone for the version it will effectively be available in.


Multilanguage option.

Add an option for multilanguage, description of  SA-MP methods will be translated by Google Translate using API.

Do you want to translate description for 400+ native functions from SA:MP?

I can.

@Malganys WOW.
@WopsS Why don't you just get the translations from the wiki itself.

@JohnyMac I do that, but in english.


Invocable Actions REST API endpoint support

From Winter'15
https://developer.salesforce.com/docs/atlas.en-us.192.0.api_rest.meta/api_rest/resources_actions_invocable.htm

Changed milestone because of no such good usage senario until 33.0 API version, in which custom action will be available.

Any Updates on this?

we are also using this on a node.js service to call into a InvocableMethod as a rest web service. Any update on when this feature will be arriving?

Until there is an official solution here is an example to create a custom notification but keep using jsforce connection to handle most of the tedious work of calling the API:
conn.requestPost('/actions/standard/customNotificationAction', { "inputs" :[{
title: "JSForce Custom Notification",
body: "The notifications are coming from JSForce!",
customNotifTypeId: '0ML1F0000004IOYWA2',
targetId: '0061F00000DXxX2QAL',
recipientIds: ['0051F00000nWKatQAG']
}]}, undefined, console.log)
I think It easy to adapt it to any other use of the actions API, event extending jsforce prototype.

@jesusbarcenilla Any idea if this continues to work now? Because the request gives me a 404 and a message: "Invalid Action Type: customNotificationAction".
@stomita Is there another way to send notifications now?

@nmandyam That code was part of a spike and I think we ended using triggers but It shouldn't be any reason for that code to stop working.
I'd check:

if the custom notification type was created and the id match the customNotifTypeId
if the user has the send notification permission
if the final url the browers use is the right one
You can find a bit more if (not a lot) here https://developer.salesforce.com/docs/atlas.en-us.api_action.meta/api_action/actions_obj_custom_notification.htm

Thanks. Unfortunately, I could not figure out what was wrong - I did check the items you listed before I posted. OTOH, I did make this code work:

I was able to get the above detailed requestPost method to work by setting the API version for the jsforce connection:
const sfConnection = new jsforce.Connection({ version: '56.0' })

Without this, the connection was being created with an API version of 42.0, which is lower than v46.0 where customNotificationAction became available.
I used tooling to find the CustomNotificationType Id by DeveloperName:
const notificationType = await sfConnection.tooling
  .sobject('CustomNotificationType')
  .find({ DeveloperName: 'DocExchange' })
  .execute()

Thanks!!


containerd-shim hung

As mentioned in https://github.com/containerd/containerd/issues/3440, we catch the goroutines of containerd.
the goroutine is stuck in
IO Wait
 internal/poll.runtime_pollWait
 /home/travis/.gimme/versions/go1.10.8.linux.amd64/src/runtime/netpoll.go
 internal/poll.(*pollDesc).wait
 /home/travis/.gimme/versions/go1.10.8.linux.amd64/src/internal/poll/fd_poll_runtime.go
 internal/poll.(*pollDesc).waitWrite
 /home/travis/.gimme/versions/go1.10.8.linux.amd64/src/internal/poll/fd_poll_runtime.go
 internal/poll.(*FD).Write
 /home/travis/.gimme/versions/go1.10.8.linux.amd64/src/internal/poll/fd_unix.go
  os.(*File).write
 /home/travis/.gimme/versions/go1.10.8.linux.amd64/src/os/file_unix.go
 os.(*File).Write
 /home/travis/.gimme/versions/go1.10.8.linux.amd64/src/os/file.go
 github.com/containerd/containerd/vendor/github.com/containerd/fifo.(*fifo).Write"
 /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/fifo/fifo.go
 /home/travis/.gimme/versions/go1.10.8.linux.amd64/src/io/io.go
 /home/travis/.gimme/versions/go1.10.8.linux.amd64/src/io/io.go
 github.com/containerd/containerd/runtime/v1/linux/proc.copyPipes.func2.1
 /home/travis/gopath/src/github.com/containerd/containerd/runtime/v1/linux/proc/io.go
 created by
  github.com/containerd/containerd/runtime/v1/linux/proc.copyPipes.func2
 /home/travis/gopath/src/github.com/containerd/containerd/runtime/v1/linux/proc/io.go

This routine always is in a wait group.
wg.Done() is not executed  here
name: stderr,
			dest: func(wc io.WriteCloser, rc io.Closer) {
				wg.Add(1)
				cwg.Add(1)
				go func() {
					cwg.Done()
					p := bufPool.Get().(*[]byte)
					defer bufPool.Put(p)
					if _, err := io.CopyBuffer(wc, rio.Stderr(), *p); err != nil {
						log.G(ctx).Warn("error copying stderr")
					}
					wg.Done()
					wc.Close()
					if rc != nil {
						rc.Close()
					}
				}()

So when we delete the containerd-shim, delete will be stuck in
sync.(*WaitGroup).Wait
/home/travis/.gimme/versions/go1.10.8.linux.amd64/src/sync/waitgroup.go
github.com/containerd/containerd/runtime/v1/linux/proc.(*Init).delete
/home/travis/gopath/src/github.com/containerd/containerd/runtime/v1/linux/proc/init.go
github.com/containerd/containerd/runtime/v1/linux/proc.(*stoppedState).Delete
/home/travis/gopath/src/github.com/containerd/containerd/runtime/v1/linux/proc/init_state.go
github.com/containerd/containerd/runtime/v1/linux/proc.(*Init).Delete
/home/travis/gopath/src/github.com/containerd/containerd/runtime/v1/linux/proc/init.go
github.com/containerd/containerd/runtime/v1/shim.(*Service).Delete
/home/travis/gopath/src/github.com/containerd/containerd/runtime/v1/shim/service.go
github.com/containerd/containerd/runtime/v1/shim/v1.RegisterShimService.func4
/home/travis/gopath/src/github.com/containerd/containerd/runtime/v1/shim/v1/shim.pb.go
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serviceSet).dispatch
/home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/services.go
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serviceSet).call
/home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/services.go
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serverConn).run.func2
/home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go
created by
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serverConn).run
/home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go

because delete goutine hold a lock of init, so new state request will be stuck in
sync.(*Mutex).Lock
/home/travis/.gimme/versions/go1.10.8.linux.amd64/src/sync/mutex.go
github.com/containerd/containerd/runtime/v1/linux/proc.(*Init).Status
/home/travis/gopath/src/github.com/containerd/containerd/runtime/v1/linux/proc/init.go
github.com/containerd/containerd/runtime/v1/shim.(*Service).State
/home/travis/gopath/src/github.com/containerd/containerd/runtime/v1/shim/service.go
github.com/containerd/containerd/runtime/v1/shim/v1.RegisterShimService.func1
/home/travis/gopath/src/github.com/containerd/containerd/runtime/v1/shim/v1/shim.pb.go
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serviceSet).dispatch
/home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/services.go
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serviceSet).call
/home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/services.go
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serverConn).run.func2

link to https://github.com/containerd/containerd/pull/3361

Please try with 1.2.8+ or 1.3 to see if the issue is resolved. We believe this was fixed by the linked to issue. Please re-open if it continues to occur.


tests: delete unneeded file

The kuberuntime exported code is good enough for us now, assuming this passes jenkins :)

Tests are indeed green.


Omit having all fieldnames present in json-file

Hi together,
Is there a way to have more fieldnames in the template than in the json-file with its default values? I could think of it being helpful if the CLI-interface just starts discovering, which fieldnames are present in the template and asks for possibly all of them when setting up a project, regardless if the fieldname is given in the json-file or not. I read through the docu, but haven't found anything on it. Please forgive me, if I missed out on something. I would appreciate a pointer in t he right direction.
Thanks, M.

Hi @magonser, Cookiecutter doesn't have any discovery feature like the one you want, and can't have it in the future. The reason for that it won't have this feature is that adding it would make the dynamic nature of the cookiecutter.json file impossible, which would break backwards capability.


Display issue with Debate comments

There is an issue with the Debate comments. This is important, because we're launching a new discussion this week and it needs to look good!!
See below:

this have also been fixed this morning, check and re-open if you still see some issues.
The one small bug I still see is for smaller screen there is a Y-axis scroll to fix (leaving this issue open for now)

last little layout bug closed by commit  6e4036d in theme (will be pulled in prod shortly)


Integrity can only be used if the physical files exist

Hey,
I've turned on integrity and experienced errors so far with:

using webpack-dev-server that only stores the files in-memory since they will get rebuilt upon change
sourcemap files that aren't built or not in the place the bundle tracker thinks they are

Tracking this bug leads to https://github.com/django-webpack/webpack-bundle-tracker/blob/4596d47632253ce557ea416d92c17265a3c286f8/lib/index.js#L31 where the readFileSync gets a path that doesn't exist and raises an exception that isn't handled.
This results in webpack silently hanging upon building the bundle with integrities.
Disclaimer: I'm not a webpack dev, just a guy who tries to use this plugin with django, emitting the integrity values in templates. If not otherwise solvable, unavailable integrity calculations should at least raise an error so guys like me would know what happened.

I have something in the works for this. PR coming up.


Server Placement. Group Files

This is a dependent PR for adding and enabling Server Placement Group as Menu in the Core

Manageiq schema changes.
https://github.com/ManageIQ/manageiq-schema/pull/644
Server Placement Group in ManageIQ UI Changes.
https://github.com/ManageIQ/manageiq-ui-classic/pull/8214
PowerVS changes.
https://github.com/ManageIQ/manageiq-providers-ibm_cloud/pull/363
ManageIQ Core changes
https://github.com/ManageIQ/manageiq/pull/21807

@Kuldip-Nanda are you also going to add a server_group model?

This is ready to be merged.

Cross-repo tests on schema and core are green, provider still failing but that can be tackled separately


Can a startup file open multiple ports?

Hello there,
Dear Mr. Walkor, your workerman project is exciting, can php actually do these things? ! ! very impressive.
However, regarding the workman listening port, I have a suggestion:

I developed a "memory MVC framework" based on workerman, which avoids the consumption of traditional php web service process destruction. https://github.com/lynncho/shy

The framework provides command modes, such as "php console list", which is the execution entry for all commands. The framework can start the workerman through "php console http start -d", but since the workerman can only open one port for a startup file, the console command mode of my framework cannot open multiple ports.

I hope that the framework can perform multiple socket services at the same time, so it is recommended that Mr. Walkor change the generation rules of the process id file and allow one startup file to open multiple services.
Written in Shanghai
Lynn, sincerely, salute

Thank you for your post and good work for project https://github.com/lynncho/shy .
Workerman can listening multi port  in one start file like this.
<?php
use Workerman\Worker;
$ws_worker = new Worker('websocket://<IP_ADDRESS>:1234');
$ws_worker->onMessage = function(){};
$http_worker = new Worker('http://<IP_ADDRESS>:5678');
$http_worker->onMessage = function(){};
// ...
Worker::runAll();

I hope this is what you want.

Thanks for your reply, I want to open the port independently, independent socket service.

Worker::$pidFile marked which service to deal with.
You can change Worker::$pidFile by command $argv like this.
<?php
use Workerman\Worker;
$module = $argv[3];
Worker::$pidFile = "/tmp/$module.pid";
call_user_func($module);
function chat() {
    $worker = new Worker('websocket://<IP_ADDRESS>:1234');
    $worker->onMessage = function(){};
}
function api() {
    $worker = new Worker('http://<IP_ADDRESS>:5678');
    $worker->onMessage = function(){};
}
Worker::runAll();

Then start with command like php start.php start -d chat php start.php start -d api.
Stop with command php start.php stop -d chat php start.php stop -d api.

I believe this has been answered, so I'm closing this for now.


linuxPackages.keymash: init at 0.1.1

Description of changes
Added the keymash kernel module. It provides a device driver that supplies endless chars from the QWERTY home row.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 248249 run on x86_64-linux 1

6 packages marked as broken and skipped:

linuxKernel.packages.linux_4_14.keymash
linuxKernel.packages.linux_4_14_hardened.keymash
linuxKernel.packages.linux_4_19.keymash
linuxKernel.packages.linux_4_19_hardened.keymash
linuxKernel.packages.linux_5_4.keymash
linuxKernel.packages.linux_5_4_hardened.keymash

15 packages built:

linuxKernel.packages.linux_5_10.keymash
linuxKernel.packages.linux_5_10_hardened.keymash
linuxKernel.packages.linux_5_15.keymash
linuxKernel.packages.linux_5_15_hardened.keymash
linuxKernel.packages.linux_6_1.keymash
linuxKernel.packages.linux_hardened.keymash (linuxKernel.packages.linux_6_1_hardened.keymash)
linuxKernel.packages.linux_6_4.keymash
linuxKernel.packages.linux_6_4_hardened.keymash
linuxKernel.packages.linux_latest_libre.keymash
linuxKernel.packages.linux_libre.keymash
linuxKernel.packages.linux_lqx.keymash
linuxKernel.packages.linux_testing_bcachefs.keymash
linuxKernel.packages.linux_xanmod.keymash
linuxKernel.packages.linux_xanmod_latest.keymash (linuxKernel.packages.linux_xanmod_stable.keymash)
linuxKernel.packages.linux_zen.keymash

I'm all for funny projects and trying out new stuff like writing kernel modules but this code has no place in a software distribution. For instance, I wrote fpletz/kernelroll as a joke a while back but would never ever advocate to add it to nixpkgs or any other software distribution.

I don't mind having projects like that and for example we also have bottom-rs in our repo but to be honest this might be a better fit for a user repository.

Makes sense. I just made this PR as a joke to see what inputs on less serious projects in nixpkgs would be.


Update to Node 20

Summary of PR changes
Update to Node 20.
PR Requirements

[x] For major, minor, or breaking changes, at least one of the commit messages contains the appropriate +semver: keywords.

See the Incrementing the Version section of the repository's README.md for more details.

[x] The action code does not contain sensitive information.

NOTE: If the repo's workflow could not automatically update the README.md, it should be updated manually with the next version.  For javascript actions, if the repo's workflow could not automatically recompile the action it should also be updated manually as part of the PR.

Can you run a build locally using Node 20 and push it up into this PR @DougDyreng?  I have node 21 on my machine and the build in this repo doesn't like that.


near-operation-file with latest selection set changes breaks fragments imports

Related: https://github.com/dotansimha/graphql-code-generator/issues/2548#issuecomment-534067192

@Swatinem  I think I managed to fix the fragments issues in: https://github.com/dotansimha/graphql-code-generator/pull/2620
Can you please try the latest alpha version? 1.7.1-alpha-59af2256.91

@dotansimha thanks for notifying me.
So the fragment imports are correct now, but there is yet another regression:
It now tries to use the XXXMutation and XXXMutationVariables from the Types (baseTypesPath) namespace, which is wrong.
The declarations are local to each operation file, so no need to use the base types namespace.

It's related to changes by @veeramarni in order to support import-types preset.
I think I'll separate the configuration names in order to fix that.

@Swatinem I managed to fix that in this PR: https://github.com/dotansimha/graphql-code-generator/pull/2667
Can you please try the latest alpha? 1.7.1-alpha-23589acb.118

Fixed in 1.8.0 🚀


RKE1 clusters display incorrect RKE templates in cluster management page

Setup
Rancher version: 2.6-head a6d2050
Browser type & version: Chrome
Describe the bug
RKE1 clusters created with RKE1 templates display incorrect cluster template used for a cluster in cluster management >> config page.
When we create different RKE1 clusters with different RKE1 templates, if we check the config of an RKE1 cluster eg., c1 and switch back to a different cluster with a different RKE1 template eg., c2, RKE1 template of c1 is listed for c2.
To Reproduce

From cluster management page >> RKE1 templates >> Create three templates t1, t2, t3
Create three node driver clusters c1, c2, c3 each with the templates t1, t2 , t3
Once the clusters are created click on the cluster c1 >> config >> Verify the template displayed in the config of the cluster
Switch to either cluster c2  or c3 >> verify the template displayed on the cluster

Result
c1 template is displayed for c2 and c3
Expected Result
Should show the template used to create cluster c2 and c3
Screenshots
During cluster creation, temp2 template was provided :

Post cluster creation, template displayed is incorrect

QA has indicated that this can be looked at post-2.6.5

Could be an iframe cacheing issue that might be fixed

A change was made recently with iFrame catching so this is possibly fixed so we should check.


docs: fix merge conflict in docs (replaces #1248)

Fixes merge conflict text in docs.
replaces PR #1248
Signed-off-by: Jan Broer<EMAIL_ADDRESS>

LGTM


Trying to make the Previews as Test Example

#164  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @Rahkeen and the rest of your teammates on  Graphite


Object: "Cannot access contents of the page. Extension manifest must request permission to access the respective host."

In Chrome I can't load the settings page, Microphone is allowed.
Console shows Object: "Cannot access contents of the page. Extension manifest must request permission to access the respective host."
Brave works fine for me.
Is it an error of mine and if so, how to fix it?

Could manage it: Opened the pinned tab to the left and could access the settings there. Frm now on it works.


haskellPackages.libmpd: Remove unnecessary patch

The patch was merged in https://github.com/vimus/libmpd-haskell/pull/138 and this version is on hackage now
This fixes the xmonad and xmonad-extras build failures in https://github.com/NixOS/nixpkgs/pull/321948
$ nix-build ./ -A haskellPackages.libmpd
/nix/store/blwsajaa1xbv1qdnpv19ixbiqkvss08r-libmpd-<IP_ADDRESS>

$ nix-build ./ -A haskellPackages.xmonad-extras
/nix/store/z9hq49xrsl75znh55mcnxq7h482h8w3v-xmonad-extras-0.17.1

$ nix-build ./ -A haskellPackages.xmonad
/nix/store/4bdr209if7175dc7cm0kprz32jc4harf-xmonad-0.17.2

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
nix-shell -p nixpkgs-review --run "nixpkgs-review pr https://github.com/NixOS/nixpkgs/pull/324308" reports:
Link to currently reviewing PR: https://github.com/NixOS/nixpkgs/pull/324308

1 package built:
xmobar

(I don't know why it does not try to at least build xmonad-extras)

[x] Tested basic functionality of all binary files (usually in ./result/bin/)

24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Great, thx!


Fixing validate-lint script

grep typo that made him way to hungry.. :sweat_smile:
This should fix it 🐹.
🐸
Signed-off-by: Vincent Demeester<EMAIL_ADDRESS>

/ping @ibuildthecloud @dnephin

LGTM


v0.2: Timer and IPI extensions are missing "Function Listing" tables

The Timer and IPI extensions are missing the "Function Listing" tables present in the other extensions.
Although the Timer and IPI extensions have only a single function each, it would be
forward-thinking to define the appropriate function IDs to enable extension in the future.
Also, without function IDs, the switch from the v0.1 legacy encoding to the standard encoding provides no benefit.

I agree. Function IDs was assumed to be zero as the it is the only function. But it should be documented in the specification. I will send a patch. Thanks.

The requested change is already merged. Closing this issue.


[GitHub pages] Release notes link broken

Please provide a succinct description of the issue.
Repro steps
Provide the steps required to reproduce the problem:
Go to http://fsharp.github.io/FSharp.Compiler.Service/index.html
In the left column the link "Release Notes" to https://github.com/fsharp/FSharp.Compiler.Service/blob/master/RELEASE_NOTES.md leads to HTTP Status Code 404
If possible attach a zip file with the repro case. This often makes it easier for others to reproduce.
The zip file should ideally represent the situation just before the call/step that is problematic.
Expected behavior
Either remove the link to release notes or include the relase notes at the target
Actual behavior
GitHub error page for HTTP Status Code 404
Known workarounds
Go here to the release tab https://github.com/fsharp/FSharp.Compiler.Service/releases
Related information
page visited today May 30th, 2019

Thanks for this, looks like we're missing the /fcs/ portion of the path there. I'll try to get that patched up soon.

Having trouble building the docs locally recently. I'll try to tackle this  failure along with build failures after the next integration point (this week expected).


Wrong font rendering on custom border characters

Hi :)
Problem
I have created a custom font of the rounded border characters to make them "filled". I really like this look, as it makes popups more "modern" in my eyes.
This works great in kitty, but doesn't work as expected in ghostty.
Kitty:

Ghostty:

General info
I am using macOS 15.1.1 (24B91) and this is the output of ghostty --version:
Ghostty 1.0.0

Version
  - version: 1.0.0
  - channel: stable
Build Config
  - Zig version: 0.13.0
  - build mode : builtin.OptimizeMode.ReleaseFast
  - app runtime: apprt.Runtime.none
  - font engine: font.main.Backend.coretext
  - renderer   : renderer.Metal
  - libxev     : main.Backend.kqueue

Root cause
Debugging this, I think I found out why this happens, but I am not sure about how to potentially resolve this. Here: https://github.com/ghostty-org/ghostty/blob/a8e5eef11cc67f87f445626f9ca2993373774bf8/src/font/sprite/Box.zig#L251 there seems to be special handling on certain unicodes and how to draw them, and since I have created a custom font and just used some random unicode points (U+e4c3-U+e4d4) they are not handled specially, and therefore not rendered as expected...
I noticed this when trying to copy the rounded border characters to the unicode value I am using, and noticed that they are rendered differently. I also tried to copy my unicode and use some of the characters that are properly handled, and they seem to be rendered as expected then.
However, I don't want to override some very normal border unicode characters.
This is how my border characters look:

and you can find the font here: https://github.com/seblj/dotfiles/blob/master/fonts/corners.ttf

https://ghostty.org/docs/config/reference#font-codepoint-map

No, that is not the issue. The issue is that the border characters are not rendered specially like the other "normal" border characters.
The font is correctly used

Is the issue that they're the wrong size? Perhaps you would want to adjust the cell size down (there's config options for this) so they fit exactly from your font?

Yeah maybe. But the same issue is if you for example copy the border characters from U+256d-U2570 and put them in for example U+e400-U+e403, then they will get rendered incorrectly because the switch never happens

Ghostty only does custom rendering for known codepoints which need to fill the cell in some way. In the future there may be a way to configure certain codepoints to be automatically stretched to the cell width/height, but that is not currently possible.


Rails/OutputSafety reports false positives

Rails/OutputSafety reports no errors on calls to .html_safe that may include user input. This is because of https://github.com/rubocop-hq/rubocop-rails/commit/c8bbfb675c81379105d3b49607dd3af4891e86ff. It seems that the conditional there is excluding non-literal strings, but I'm unfamiliar with Rubocop's internals so I'm not sure how to interpret it.

Expected behavior
Given the following file:
# frozen_string_literal: true
class OutputSafety
  def unsafe(input)
    input.html_safe
  end
end

I expect to receive:
app/models/output_safety.rb:3:11: C: Rails/OutputSafety: Tagging a string as html safe may be a security risk.
    input.html_safe
          ^^^^^^^^^

Actual behavior
Rubocop does not issue any warnings.
Steps to reproduce the problem
See example above.
RuboCop version
Include the output of rubocop -V or bundle exec rubocop -V if using Bundler. Here's an example:
$ bundle exec rubocop -V
0.74.0 (using Parser <IP_ADDRESS>, running on ruby 2.7.0 x86_64-darwin18)

This works correctly for me.
$ cat t.rb
def unsafe(input)
  input.html_safe
end

$ rubocop -r rubocop-rails --only Rails/OutputSafety t.rb
Inspecting 1 file
C

Offenses:

t.rb:2:9: C: Rails/OutputSafety: Tagging a string as html safe may be a security risk.
  input.html_safe
        ^^^^^^^^^

1 file inspected, 1 offense detected

Are you sure you're requiring rubocop-rails?

I switched to use the same command you did and got the same result.
$ cat t.rb
# frozen_string_literal: true
class OutputSafety
  def unsafe(input)
    input.html_safe
  end
end

$ rubocop -r rubocop-rails --only Rails/OutputSafety t.rb
Inspecting 1 file
.

1 file inspected, no offenses detected

$ rubocop -V
0.74.0 (using Parser <IP_ADDRESS>, running on ruby 2.6.5 x86_64-darwin19)

❯ bundle show rubocop-rails
/path/redacted/vendor/gems/2.6.5/ruby/2.6.0/gems/rubocop-rails-2.3.2

So, if I delete .rubocop.yml, it works as expected 🤔. So it's odd since I don't see it being disabled, but I'm pretty sure it's not a problem with rubocop-rails, so I'm closing. Thanks for taking a look.


WIP: Initial generic corrections support

[x] support in the Python API
[ ] support for loading correction data from files
[ ] support for uploading correction data
[ ] support for corrections settings in the GUI
[ ] hot/dead pixel corrections

[ ] XML metadata from SEQ datasets
[ ] From Gatan service configuration file, thx @ju-bar :-)

[ ] per-UDF preferences

[ ] Implement corrections in MaskContainer and ApplyMasksUDF, perhaps as a follow-up issue

[ ] Performance tuning as a follow-up issue

Would love to get some feedback, mostly on the Python API, @uellue
/CC @jhgee
Contributor Checklist:

[x] I have added or updated my entry in the creators.json file
[ ] I have added a changelog entry for my contribution
[ ] I have added/updated documentation for all user-facing changes
[ ] I have added/updated test cases
[ ] I have included the rebuilt production build of the client (only if
changes were made to the GUI)

We can include a dead/hot pixel map as well

Yeah, I was actually already starting to write a comment about that :grin: We have to think about how the user can best specify those. The SEQ datasets actually includes a <BadPixels/> element in an XML sidecar file, so we should probably support something like that. In that case, whole rows are actually marked as bad, so I don't know how well this will work with the current correction code. The XML format also supports single pixels, though. In addition to DataSet-provided dead pixel maps, we should allow user-specified fancy-indexing formats, i.e. boolean arrays or coordinate lists.
In what formats should the GUI accept bad pixel maps? Also as binary maps or coordinate lists, in what formats? .mat, .npz, csv...?

How would that work in this context?

I'm not sure about that yet, I'll add it to the TODO above, though.

We can include a dead/hot pixel map as well

Yeah, I was actually already starting to write a comment about that 😁 We have to think about how the user can best specify those. The SEQ datasets actually includes a <BadPixels/> element in an XML sidecar file, so we should probably support something like that. In that case, whole rows are actually marked as bad, so I don't know how well this will work with the current correction code. The XML format also supports single pixels, though. In addition to DataSet-provided dead pixel maps, we should allow user-specified fancy-indexing formats, i.e. boolean arrays or coordinate lists.

The correction back-end expects a list of coordinates, so that would match well here. The correction back-end can detect and ignore neighboring dead pixels, it just needs at least one good pixel next to each bad pixel.

In what formats should the GUI accept bad pixel maps? Also as binary maps or coordinate lists, in what formats? .mat, .npz, csv...?

See https://groups.google.com/d/msg/libertem/0u_xCjPVf5Y/v7Aj0tJABwAJ :-)

Another question: Corrections are most efficient when applied to a stack of frames. If I understand the code correctly, the DataSet CAN return a stack of frames, which is then applied one by one within the UDFRunner. Do all the DataSet implementations actually do return several frames here? The balance between using the L3 cache for a single frame (frame size in the region of 1/3 MB -- we have two correction maps of the same size as the frame!) vs. performance gains from correcting several frames at once using the correction data in the L1 cache are not so trivial. If one frame is larger than 1/3 MB, a stack of frames is likely more efficient here.
Long story short, tiled processing should be of great advantage when using corrections.
Referring to #335, the correction data would be a perfect use case for shared memory, by the way. That would be a great relief for the L3 cache on machines with many cores.
Another thought about the bad pixel maps: One can construct a boolean sparse matrix from the list, use slicing on that and then convert back to a list for tiled processing. Only disadvantage is that pixels on a tile boundary would not use the entire environment for patching, but only the one within the tile. I am not sure how much of a problem it is, but in some edge cases it might change results depending on the tiling.

Referring to the e-mail from @rafaldb, the code already allows to use a different CorrectionSet class.
The code to patch masks should actually be part of the CorrectionSet, similar to apply(), since it is closely linked to the mathematical model used for correction. The current "constant offset and linear gain, average patching" implementation can patch masks. However, with a non-linear gain correction this wouldn't be possible anymore. That means the UDF should be able to probe the CorrectionSet for correction capabilities and boundary conditions within its proposed does_corrections() method. ApplyMasksUDF, for example, should fall back to receiving corrected data if the correction is not linear.

Another question: Corrections are most efficient when applied to a stack of frames. If I understand the code correctly, the DataSet CAN return a stack of frames, which is then applied one by one within the UDFRunner.

Correct.

Do all the DataSet implementations actually do return several frames here?

Not yet - the SER reader being the only exception. I'm right now returning frame-by-frame to avoid copying, but I think that's easily fixable, even if it may require a patch to ncempy.

The balance between using the L3 cache for a single frame (frame size in the region of 1/3 MB -- we have two correction maps of the same size as the frame!) vs. performance gains from correcting several frames at once using the correction data in the L1 cache are not so trivial. If one frame is larger than 1/3 MB, a stack of frames is likely more efficient here.
Long story short, tiled processing should be of great advantage when using corrections.

Agreed.

Referring to #335, the correction data would be a perfect use case for shared memory, by the way. That would be a great relief for the L3 cache on machines with many cores.

Also agreed.

Another thought about the bad pixel maps: One can construct a boolean sparse matrix from the list, use slicing on that and then convert back to a list for tiled processing. Only disadvantage is that pixels on a tile boundary would not use the entire environment for patching, but only a part within the same tile. I am not sure how much of a problem it is, but in some edge cases it might change results depending on the tiling.

Do we need to add the bad pixel list to the negotiation input, so we can generate a tiling scheme that doesn't have bad pixels at boundaries?

The correction back-end expects a list of coordinates, so that would match well here

The point here being that both are supported, for ease of use (I know it is easy to transform back and forth, but the user shouldn't have to know...)

That means it can patch an entire row without problems

Perfect!

Referring to an e-mail from @rafaldb

Didn't get that one.

The code to patch masks should actually be part of the CorrectionSet, similar to apply(), since it is closely linked to the mathematical model used for correction. The current "constant offset and linear gain, average patching" implementation can patch masks. However, with a non-linear gain correction (edit: or patching) this wouldn't be possible anymore. That means the UDF should be able to probe within its proposed does_corrections() method the CorrectionSet for correction capabilities and boundary conditions. ApplyMasksUDF, for example, should fall back to receiving corrected data if the correction is not linear.

So the interface should allow a UDF to modify the CorrectionSet, for example by deactivating a certain correction?

# default case
def adjust_corrections(self, corrections):
    return corrections

# masks
def adjust_corrections(self, corrections):
    if corrections.something is not "good":
        return corrections.without_something()\
                          .without_something_else()
    return corrections

# unconditionally deactivate:
def adjust_corrections(self, corrections):
    return CorrectionSet()  # or possibly corrections.empty() so the user doesn't have to import stuff

This is then similar to the adjust_tileshape interface of the DataSet.

So the interface should allow a UDF to modify the CorrectionSet, for example by deactivating a certain correction?

Oh, that's a neat idea, to grab parts of the correction! We just have to make sure that the UDF really understands what the correction does, for example by probing if it is an instance of a supposed LinearCorrectionSet, and only takes over if it can make sure the result is equivalent.
With "parts of a correction", do you mean detector correction vs. correction of other physical coordinates in the future, or parts of a detector correction?I'm not sure if deactivating parts of a detector correction would bring much benefit compared to either doing all the detector corrections or none.

Do we need to add the bad pixel list to the negotiation input, so we can generate a tiling scheme that doesn't have bad pixels at boundaries?

Now THAT would be neat! :-) :-) :-)

The point here being that both are supported, for ease of use (I know it is easy to transform back and forth, but the user shouldn't have to know...)

I guess we can just support both since they are easily distinguished by shape. Probably we just convert to the sparse "list of pixels" format immediately in the constructor?

So the interface should allow a UDF to modify the CorrectionSet, for example by deactivating a certain correction?

Ah, just for clarity: The current implementation assumes that the gain correction is a constant multiplier, which is a good first approximation in many cases. However, real-world detectors have non-linear gain curves, as @ju-bar and @rafaldb wrote. In the future the user may want to use a different mathematical model for correction, i.e. a completely different implementation of a CorrectionSet set that takes nonlinearity into account, for example gamma curves as they are used in photography.

Not yet - the SER reader being the only exception. I'm right now returning frame-by-frame to avoid copying, but I think that's easily fixable, even if it may require a patch to ncempy.

An issue for the "good first issue" category?

Do we need to add the bad pixel list to the negotiation input, so we can generate a tiling scheme that doesn't have bad pixels at boundaries?

Now THAT would be neat! :-) :-) :-)

It just won't work (efficiently?) if a whole column of pixels is missing. Another possibility is reading overlapping tiles, which are then cropped by the correction mechanism. I think we can leave that for later, when this becomes a problem in practice.

I guess we can just support both since they are easily distinguished by shape. Probably we just convert to the sparse "list of pixels" format immediately in the constructor?

Yeah... I had in mind that you could just use fancy indexing into a coordinate array with either format without checking the shape, but that doesn't seem to be as easy as I remember.

Ah, just for clarity: The current implementation assumes that the gain correction is a constant multiplier, which is a good first approximation in many cases. However, real-world detectors have non-linear gain curves, as @ju-bar and @rafaldb wrote. In the future the user may want to use a different mathematical model for correction, i.e. a completely different implementation of a CorrectionSet set that takes nonlinearity into account, for example gamma as it is used in photography.

We have to take care that the API stays valid in the future. That means that the return value from adjust_correction should stay valid even in the face of any changes in the future.

An issue for the "good first issue" category?

With copy, yes, without, it is an issue in the openNCEM project /cc @ercius

Do we need to add the bad pixel list to the negotiation input, so we can generate a tiling scheme that doesn't have bad pixels at boundaries?

Now THAT would be neat! :-) :-) :-)

It just won't work (efficiently?) if a whole column of pixels is missing. Another possibility is reading overlapping tiles, which are then cropped by the correction mechanism. I think we can leave that for later, when this becomes a problem in practice.

👍

I guess we can just support both since they are easily distinguished by shape. Probably we just convert to the sparse "list of pixels" format immediately in the constructor?

Yeah... I had in mind that you could just use fancy indexing into a coordinate array with either format without checking the shape, but that doesn't seem to be as easy as I remember.

As far as I still understand my own Numba code, it is built around a list of pixels and wouldn't be so easy to change.

Ah, just for clarity: The current implementation assumes that the gain correction is a constant multiplier, which is a good first approximation in many cases. However, real-world detectors have non-linear gain curves, as @ju-bar and @rafaldb wrote. In the future the user may want to use a different mathematical model for correction, i.e. a completely different implementation of a CorrectionSet set that takes nonlinearity into account, for example gamma as it is used in photography.

We have to take care that the API stays valid in the future. That means that the return value from adjust_correction should stay valid even in the face of any changes in the future.

Hm, that would make an "all or nothing" approach easier. If the UDF positively identifies the correction and knows what to do, it takes over. If anything seems unsure, it falls back to consuming corrected data. IMHO it could even make sense to flag a "constant shift, linear gain, linear patching" correction as something special since it will allow folding the correction into any UDF that is a linear transformation: f(u + cv) = f(u) + cf(v) https://en.wikipedia.org/wiki/Linear_map
As an example, correcting the result of SumUDF would be trivial: Correct and patch it like a single detector frame, just with the dark frame multiplied by the number of frames.

I just tested a few .seq files and the dark corrections work! :-)
As a side note (most probably better located somewhere else) : It would be super helpful if additional frames would be ignored. For example I have scans with 4100 frames for a 64x64 scan. would be nice if the additional 4 frames would be ignored.

It would be super helpful if additional frames would be ignored. For example I have scans with 4100 frames for a 64x64 scan. would be nice if the additional 4 frames would be ignored.

I'm working on that, thanks for the feedback!

Hm, that would make an "all or nothing" approach easier. If the UDF positively identifies the correction and knows what to do, it takes over. If anything seems unsure, it falls back to consuming corrected data.

Sounds good.

IMHO it could even make sense to flag a "constant shift, linear gain, linear patching" correction as something special [...]
As an example, correcting the result of SumUDF would be trivial: Correct and patch it like a single detector frame, just with the dark frame multiplied by the number of frames.

Here are some strawmen examples for discussion:
The very explicit way:
def postprocess(self):
    my_own_apply_corrections_fn(self.results.something_sig_shaped, self.meta.corrections)

def get_correction_preferences(self):
        return {"custom_corrections": self.meta.set_of_corrections == {UDF.CORR_DARK_FRAME, UDF.CORR_GAIN_MAP, UDF.CORR_PATCH_PIXELS}}

Something a bit more "automatic", for supporting all kinds of linear corrections:
def postprocess(self):
    self.meta.corrections.apply_linear_corrections(self.results.something_sig_shaped, number_of_frames)

def get_correction_preferences(self):
    return {"custom_corrections": self.meta.set_of_corrections == UDF.CORR_SET_LINEAR}

Support for patching masks:
def get_task_data(self):
    masks = ...  # possibly need to fold this into the mask container somehow, or the mask factory fn
    self.meta.corrections.patch_masks(masks)
    return {"masks": masks}

def get_correction_preferences(self):
    # we know what can be patched into masks, so we can also provide that as a set of identifiers:
    return {"custom_corrections": self.meta.set_of_corrections == UDF.CORR_SET_MASK_PATCH}

As this PR is already quite large, I'm trying to get it into a state that is ready for check-in, and to tackle remaining issues in separate independent PRs.
@uellue it would be helpful to get feedback on the excluded_pixels and detector.correct-related changes - do you think it is reasonable to use a sparse.COO array as the common interface?

Trying to revive travis...

Closing and re-opening to trigger a travis build

Using a sparse.COO for the excluded pixels sounds good! That makes it easy to change things later, since it builds on the NumPy ndarray interface.
If I understood the code correctly, one would use it like this?
ds = ctx.load('frms6', path=..., )
udf = ...
res = ctx.run_udf(udf=udf, dataset=ds, corrections=ds.get_correction_data())

If the dataset doesn't provide corrections or if you'd like to use different ones, one would explicitly create a CorrectionSet with the desired parameters?
If a dataset has corrections, they will not be applied automatically, right?
Generally, this looks like a good point to start! We can then add more features step by step. :-)

Looking at the code, it will likely generate a significant merge conflict with #760. Let me know if/how I can help if it gets too messy!

Using a sparse.COO for the excluded pixels sounds good! That makes it easy to change things later, since it builds on the NumPy ndarray interface.

:+1:

If I understood the code correctly, one would use it like this?
ds = ctx.load('frms6', path=..., )
udf = ...
res = ctx.run_udf(udf=udf, dataset=ds, corrections=ds.get_correction_data())

If the dataset doesn't provide corrections or if you'd like to use different ones, one would explicitly create a CorrectionSet with the desired parameters?

Yes.

If a dataset has corrections, they will not be applied automatically, right?

They will be automatically applied - currently this is done in api.py and web/jobs.py - I think it may need to be moved to the UDFRunner instead, if possible, so we don't need to set the default behavior in two different places.

Generally, this looks like a good point to start! We can then add more features step by step. :-)

That's the plan! :grin:

Looking at the code, it will likely generate a significant merge conflict with #760. Let me know if/how I can help if it gets too messy!

Yes, indeed, I should manage, but otherwise we can pair on this via skype.

They will be automatically applied - currently this is done in api.py and web/jobs.py - I think it may need to be moved to the UDFRunner instead, if possible, so we don't need to set the default behavior in two different places.

Ah found it! Yes, I looked in UDFRunner... 😄

Yes, indeed, I should manage, but otherwise we can pair on this via skype.

We can also merge this first and then I rebase the CuPy stuff. :-)

Yes, indeed, I should manage, but otherwise we can pair on this via skype.

We can also merge this first and then I rebase the CuPy stuff. :-)

That could work, too. I'll first finish the review of #760, then I'll add finishing touches here and we can merge this.


Updating Pod ressources seams to have problems

when changing the cpu limit of cassandra pods, I ran into this error
E0611 14:43:22.295011       1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/src/github.com/Orange-OpenSource/cassandra-k8s-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/go/src/github.com/Orange-OpenSource/cassandra-k8s-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/Orange-OpenSource/cassandra-k8s-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/panic.go:522
/usr/local/go/src/runtime/panic.go:82
/usr/local/go/src/runtime/signal_unix.go:390
/go/src/github.com/Orange-OpenSource/cassandra-k8s-operator/pkg/controller/cassandracluster/reconcile.go:425
/go/src/github.com/Orange-OpenSource/cassandra-k8s-operator/pkg/controller/cassandracluster/cassandracluster_controller.go:167
/go/src/github.com/Orange-OpenSource/cassandra-k8s-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
/go/src/github.com/Orange-OpenSource/cassandra-k8s-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158
/go/src/github.com/Orange-OpenSource/cassandra-k8s-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/github.com/Orange-OpenSource/cassandra-k8s-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/github.com/Orange-OpenSource/cassandra-k8s-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:1337
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x127db4b]

@allamand I just tried and did not have any issue https://pastebin.com/nLMsDYi1
Any particular change ? Wonder if that was not an issue with a new code change you've tested ? When I look at the current code I think the issue was at https://github.com/Orange-OpenSource/cassandra-k8s-operator/blob/2c08f6f609d4ee0a90a7165cb4d52e100e67212e/pkg/controller/cassandracluster/reconcile.go#L426
But why ? an issue in the index access for some reason ?

didn't reproduce neither on my side.. I'll close for now


use sdelete64.exe for 64-Bit OS

There's a 64-Bit version of sdelete which can be used in 64-Bit OS.

I thought it was not necessary because a 64bit OS can run 32 bits apps. But I think is a good improvement and I merged and released a new version with your change.
Thanks a lot @abyssox!


dependency: bump spring-javaformat-checkstyle.version to 0.0.25

this version bump will allow checkstyle team to reuse this project in regression testing on each commit.
it will make sure there is no false positives be introduced in new version of checkstyle to this project.

@romani Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.

@romani Thank you for signing the Contributor License Agreement!

@elefeint , please take a look.
thanks a lot for your help at https://github.com/checkstyle/checkstyle/pull/8106


[Epic] Snapshots as a First Class Citizen

An epic to improve the usability and accuracy of snapshots.
### P0 - must
- [ ] https://github.com/dbt-labs/dbt-core/issues/4000
- [ ] https://github.com/dbt-labs/dbt-core/issues/10326
- [ ] https://github.com/dbt-labs/dbt-core/issues/10325
- [ ] https://github.com/dbt-labs/dbt-core/issues/10301
- [ ] https://github.com/dbt-labs/dbt-core/issues/10234
- [ ] https://github.com/dbt-labs/dbt-core/issues/10185
- [ ] https://github.com/dbt-labs/dbt-core/issues/10187
- [ ] https://github.com/dbt-labs/dbt-core/issues/10235
- [ ] https://github.com/dbt-labs/dbt-core/issues/7869
- [ ] https://github.com/dbt-labs/dbt-core/issues/10246

### P1 - should
- [ ] https://github.com/dbt-labs/dbt-core/issues/10236
- [ ] https://github.com/dbt-labs/dbt-core/issues/8207

### p2 - could
- [ ] https://github.com/dbt-labs/dbt-core/issues/9992
- [ ] https://github.com/dbt-labs/dbt-core/issues/7138

### revisit, refine, prioritize
- [ ] https://github.com/dbt-labs/dbt-core/issues/10300
- [ ] https://github.com/dbt-labs/dbt-core/issues/10438
- [ ] https://github.com/dbt-labs/dbt-core/issues/7867
- [ ] https://github.com/dbt-labs/dbt-core/issues/2848
- [ ] https://github.com/dbt-labs/dbt-core/issues/4661
- [ ] https://github.com/dbt-labs/dbt-core/issues/3878
- [ ] https://github.com/dbt-labs/dbt-core/issues/10320
- [ ] https://github.com/dbt-labs/dbt-core/issues/10372
- [ ] https://github.com/dbt-labs/dbt-adapters/issues/82

Discussions
Problems & (Potential) Solutions for Snapshots
Toggling values when snapshotting a table with a key violation

another interesting customization could be the ability to configure the value for dbt_valid_to then opening a new record.
dbt default is null, but in many cases I found the need to have a far away timestamp (e.g. '2999-12-31').

@danielefrigo Yes! This is something we'd like to do - issue here -> https://github.com/dbt-labs/dbt-core/issues/10187

Snapshots currently don't support contracts. Would be great have snapshots support contracts/constraints just as models do.

Thanks @jeremyyeo - would you be down to open up a new issue for that feature request?


Add release instructions

[ ] Binary release documentation
[ ] Public test cluster release instructions

#438 involves Terraform init scripts and some new env vars. How to deploy without necessarily cutting a new release?

You can deploy new terraform without having to bump the release version