Receiving segmentation error

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
Running pytest-memray with coverage in urllib3, occasionally seeing segmentation errors (ref):
Fatal Python error: Segmentation fault

Thread 0x00007fe6c2ffe700 (most recent call first):
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/selectors.py", line 468 in select
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/asyncio/base_events.py", line 1823 in _run_once
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/asyncio/base_events.py", line 570 in run_forever
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/asyncio/base_events.py", line 603 in run_until_complete
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/asyncio/runners.py", line 44 in run
  File "/home/runner/work/urllib3/urllib3/dummyserver/server.py", line 241 in _run_and_close_tornado
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/concurrent/futures/thread.py", line 57 in run
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/concurrent/futures/thread.py", line 80 in _worker
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/threading.py", line 870 in run
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/threading.py", line 890 in _bootstrap

Current thread 0x00007fe6c95e7700 (most recent call first):
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/ssl.py", line 1101 in read
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/ssl.py", line 1226 in recv
  File "/home/runner/work/urllib3/urllib3/dummyserver/testcase.py", line 31 in consume_socket
  File "/home/runner/work/urllib3/urllib3/test/test_ssltransport.py", line 121 in socket_handler
  File "/home/runner/work/urllib3/urllib3/dummyserver/server.py", line 145 in _start_server
  File "/home/runner/work/urllib3/urllib3/dummyserver/server.py", line 149 in run
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00007fe6cfe10740 (most recent call first):
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/pytest_memray/plugin.py", line 125 in wrapper
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/_pytest/python.py", line 195 in pytest_pyfunc_call
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/_pytest/python.py", line 1789 in runtest
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/_pytest/runner.py", line 167 in pytest_runtest_call
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/_pytest/runner.py", line 260 in <lambda>
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/_pytest/runner.py", line 339 in from_call
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/flaky/flaky_pytest_plugin.py", line 138 in call_and_report
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/_pytest/runner.py", line 131 in runtestprotocol
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/flaky/flaky_pytest_plugin.py", line 94 in pytest_runtest_protocol
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/_pytest/main.py", line 324 in _main
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/_pytest/main.py", line 270 in wrap_session
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/_pytest/config/__init__.py", line 167 in main
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/_pytest/config/__init__.py", line 190 in console_main
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/pytest/__main__.py", line 5 in <module>
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/coverage/execfile.py", line 199 in run
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/coverage/cmdline.py", line 830 in do_run
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/coverage/cmdline.py", line 659 in command_line
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/coverage/cmdline.py", line 943 in main
  File "/home/runner/work/urllib3/urllib3/.nox/test-3-8/lib/python3.8/site-packages/coverage/__main__.py", line 8 in <module>
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/runpy.py", line 87 in _run_code
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/runpy.py", line 194 in _run_module_as_main

Not sure why this is happening, toughest part is it's a very rare occurrence as most jobs are passing just fine.
Expected Behavior
No crash
Steps To Reproduce

Install memray, pytest-memray, coverage
Run coverage python -m pytest ... on the urllib3 test suite
Sometimes crash with segfault? :(

Memray Version
1.5.0
Python Version
3.8
Operative System
Linux
Anything else?
Let us know if there's something we can do to make capturing a better reproducer for this issue. Looks very similar to this issue: https://github.com/bloomberg/memray/issues/249

Sometimes crash with segfault? :(

😢

Let us know if there's something we can do to make capturing a better reproducer for this issue.

Well, I'm trying to reproduce it in an Ubuntu 20.04 Docker container, but I haven't had any luck so far running the test suite in a loop. You say that you're occasionally seeing segfaults - do you have a feel for how often they occur?
Based on the Python stack that faulthandler captured, the segfault happened in the file reader. That doesn't surprise me; the reader is generally trusting of the file written by the writer, and invalid files can crash the reader as things stand today.
This probably means that the writer has done something wrong and somehow written an invalid capture file, and the reader is choking on it. The fact that you've got several threads running at the point where the file is being read makes me suspicious - there's some tricky work involved in capturing the stacks of threads that already existed at the point where Memray tracking started, so it's possible that our bug is somewhere around there, but that's just a hunch so far.
Unfortunately, until we see a core file or native stack trace, or even a copy of the memray results file, we're really shooting in the dark.
I've found https://github.com/lhotari/action-upterm to be extremely useful for tracking down issues that are occurring in a GitHub Actions context - it spawns an upterm session that you can connect to, allowing you ssh access to the container before or after the tests run. If this is failing 1 out of 5 times, maybe that's a way for us to grab a core file or the results file when it fails, or to run the test suite under gdb to get a backtrace and allow live inspection. But if it's failing 1 out of 1000 times, that's probably not super helpful, since it does require some manual interaction.

I left my computer tunning python3 -bb -m coverage run --parallel-mode -m pytest --memray --hide-memray-summary -v -ra --color=yes --tb=native --no-success-flaky-report --durations=10 --strict-config --strict-markers test/  -vvvv all night in a loop in a docker container with ubuntu:20.04 running Python 3.8 and unfortunately I haven't been able to reproduce it :(
As @godlygeek mentions, we are going to need either a core file, a native stack trace or the memray dump file that's making it crash so we can make progress.

I also executed the whole urllib3 test suite under valgrind and helgrind (which took a lot of time) independently and neither reported any errors :(
The command for valgrind (the ../valgrind.supp  is in the memray  root):
PYTHONMALLOC=malloc valgrind \
        --suppressions=../valgrind.supp \
        --leak-check=full \
        --show-leak-kinds=definite \
        --error-exitcode=1 \
        --gen-suppressions=all python3 -bb -m coverage run --parallel-mode -m pytest --memray --hide-memray-summary -v -ra --color=yes --tb=native --no-success-flaky-report --durations=10 --strict-config --strict-markers test/  -vvvv

The command for helgrind (the ../valgrind.supp  is in the memray  root):
PYTHONMALLOC=malloc valgrind  \
        --suppressions=../valgrind.supp \
        --tool=helgrind \
        --error-exitcode=1 \
        --gen-suppressions=all python3 -bb -m coverage run --parallel-mode -m pytest --memray --hide-memray-summary -v -ra --color=yes --tb=native --no-success-flaky-report --durations=10 --strict-config --strict-markers test/  -vvvv

Thank you both for trying! We'll try to provide that data to you, but it could take days/weeks. Would you prefer closing the issues in the meantime?

Thank you both for trying! We'll try to provide that data to you, but it could take days/weeks

Thanks a lot! We will wait for the data! Not sure what will be easier to obtain, but just in case you can prioritise, this is the order of usefulness:

The results file (this is created by pytest-memray automatically in a temporary directory).
A core file for the crash
A native stack trace (with C functions and locations).

Would you prefer closing the issues in the meantime?

We will discuss this today. Maybe we could add some kind of code that can print some information before crashing, but I am not sure if that's possible without slowing down things.

@pquentin @sethmlarson  One cheap way to get a stack trace on segfault:
With the following code C:
#define __USE_GNU
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <execinfo.h>
#include <unistd.h>
#include <string.h>

void segfault_handler(int signal, siginfo_t* siginfo, void* context) {
  void *array[1000];
  size_t size;

  size = backtrace(array, 1000);
  fprintf(stderr, "Error: signal %d:\n", signal);
  backtrace_symbols_fd(array, size, STDERR_FILENO);
  exit(1);
}

__attribute__((constructor))
void install_sigsegv() {
  struct sigaction sig;
  sig.sa_sigaction = segfault_handler;
  sig.sa_flags = SA_SIGINFO;

  if(sigaction(SIGSEGV, &sig, NULL) < 0) {
    puts("[error] signal registering failed");
    exit(-1);
  }
}

Compile as gcc the_file.c -o segfaulter.so --shared -fpic and you can run any program with LD_PRELOAD=PATH_TO_SEGFAULTER.SO and will print the native trace on segfault
For example usage:
root@98e633069c53:/# LD_PRELOAD=./segfaulter.so python3 -c 'import signal;import os; os.kill(os.getpid(), signal.SIGSEGV)'
Error: signal 11:
./segfaulter.so(segfault_handler+0x3c)[0xffffa8b70b4c]
linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xffffa8bc6790]
/lib/aarch64-linux-gnu/libc.so.6(kill+0xc)[0xffffa88ea8cc]
python3(+0x23e460)[0xaaaac573e460]
python3(+0x10452c)[0xaaaac560452c]
python3(_PyEval_EvalFrameDefault+0x4c10)[0xaaaac55f0bf0]
python3(+0x1e9a80)[0xaaaac56e9a80]
python3(PyEval_EvalCode+0x74)[0xaaaac56e9904]
python3(+0x21f1ec)[0xaaaac571f1ec]
python3(+0x2168d8)[0xaaaac57168d8]
python3(PyRun_StringFlags+0x84)[0xaaaac570e058]
python3(PyRun_SimpleStringFlags+0x40)[0xaaaac570ded4]
python3(Py_RunMain+0x31c)[0xaaaac570cacc]
python3(Py_BytesMain+0x24)[0xaaaac56dab08]
/lib/aarch64-linux-gnu/libc.so.6(+0x273fc)[0xffffa88d73fc]
/lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98)[0xffffa88d74cc]
python3(_start+0x30)[0xaaaac56da9f0]

@pquentin @sethmlarson Good news, by the usage of arcane rituals and a LOT of time (i spent my workday on this), I was able to reproduce it once and I have a stack trace:
(gdb) info threads
  Id   Target Id                         Frame
  1    Thread 0x7f5270804700 (LWP 50462) __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:50
* 2    Thread 0x7f52774cf740 (LWP 50298) 0x00007f527575da54 in std::istream::sentry::sentry(std::istream&, bool) () from /lib/x86_64-linux-gnu/libstdc++.so.6

This is the stack
Thread 2 (Thread 0x7f52774cf740 (LWP 50298)):
#0  0x00007f527575da54 in std::istream::sentry::sentry(std::istream&, bool) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00007f527575ea5f in std::istream::read(char*, long) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007f5275c81e55 in memray::io::FileSource::read(char*, long) () from /usr/local/lib/python3.8/dist-packages/memray/_memray.cpython-38-x86_64-linux-gnu.so
#3  0x00007f5275c9addb in memray::api::RecordReader::nextRecord() () from /usr/local/lib/python3.8/dist-packages/memray/_memray.cpython-38-x86_64-linux-gnu.so
#4  0x00007f5275cbb6ec in __pyx_pf_6memray_7_memray_10FileReader___cinit__(__pyx_obj_6memray_7_memray_FileReader*, _object*, bool) [clone .lto_priv.457] () from /usr/local/lib/python3.8/dist-packages/memray/_memray.cpython-38-x86_64-linux-gnu.so
#5  0x00007f5275caa9b9 in __pyx_tp_new_6memray_7_memray_FileReader(_typeobject*, _object*, _object*) () from /usr/local/lib/python3.8/dist-packages/memray/_memray.cpython-38-x86_64-linux-gnu.so
#6  0x00000000005f65f3 in ?? ()
#7  0x00000000000007a8 in ?? ()
#8  0x00000000005a6c10 in ?? ()
#9  0x00007f5275fce880 in ?? ()
#10 0x0000000001f98840 in ?? ()
#11 0x00007f5275fce89e in ?? ()
#12 0x0000000001f98a00 in ?? ()
#13 0x00000000015fbba0 in ?? ()
#14 0x0000000001f989f0 in ?? ()
#15 0x00007f5275ef8fc0 in ?? ()
#16 0x0000000000572dea in ?? ()
#17 0x00007f5275fce870 in ?? ()
#18 0x0000000001f989a8 in ?? ()
#19 0x00007f5275f65ba0 in ?? ()
#20 0x00007f5275f85610 in ?? ()
#21 0x00007f5275f9da80 in ?? ()
#22 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f5270804700 (LWP 50462)):
#0  __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  <signal handler called>
#2  __GI___pthread_mutex_lock (mutex=0x10) at ../nptl/pthread_mutex_lock.c:67
#3  0x00007f5275c8887f in memray::tracking_api::Tracker::trackAllocationImpl(void*, unsigned long, memray::hooks::Allocator) () from /usr/local/lib/python3.8/dist-packages/memray/_memray.cpython-38-x86_64-linux-gnu.so
#4  0x00007f5275ca0723 in memray::intercept::malloc(unsigned long) () from /usr/local/lib/python3.8/dist-packages/memray/_memray.cpython-38-x86_64-linux-gnu.so
#5  0x00007f527704bc0e in CRYPTO_zalloc () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#6  0x00007f5276f6db41 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#7  0x00007f5276f6b3f3 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#8  0x00007f5276f6bf82 in ASN1_item_ex_d2i () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#9  0x00007f5276f6c00f in ASN1_item_d2i () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#10 0x00007f52763cefbf in d2i_SSL_SESSION () from /lib/x86_64-linux-gnu/libssl.so.1.1
#11 0x00007f52763fc1e2 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
#12 0x00007f52763ee58f in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
#13 0x00007f52763d9d98 in SSL_do_handshake () from /lib/x86_64-linux-gnu/libssl.so.1.1
#14 0x00007f527644a04c in ?? () from /usr/lib/python3.8/lib-dynload/_ssl.cpython-38-x86_64-linux-gnu.so
#15 0x00000000005c43c6 in ?? ()
#16 0x00007f5271b67d60 in ?? ()
#17 0x0000000000575298 in ?? ()
#18 0x00007f5271b67ee8 in ?? ()
#19 0x000000000068dc54 in ?? ()
#20 0x00007f5276393430 in ?? ()
#21 0x0000000000000000 in ?? ()

This is the disasembly around the crash:
   0x00007f5275c88821 <+529>:   callq  0x7f5275c6f250 <_ZdlPv@plt>
   0x00007f5275c88826 <+534>:   jmpq   0x7f5275c889c4 <_ZN6memray12tracking_api7Tracker19trackAllocationImplEPvmNS_5hooks9AllocatorE+948>
   0x00007f5275c8882b <+539>:   nopl   0x0(%rax,%rax,1)
   0x00007f5275c88830 <+544>:   mov    (%rsp),%rax
   0x00007f5275c88834 <+548>:   mov    0x78(%rbp),%rbx
   0x00007f5275c88838 <+552>:   mov    %rax,0xb0(%rsp)
   0x00007f5275c88840 <+560>:   mov    0x8(%rsp),%rax
   0x00007f5275c88845 <+565>:   mov    %rax,0xb8(%rsp)
   0x00007f5275c8884d <+573>:   movzbl 0x18(%rsp),%eax
   0x00007f5275c88852 <+578>:   mov    %al,0xc0(%rsp)
   0x00007f5275c88859 <+585>:   callq  0x7f5275c6eff0 <_ZTHN6memray12tracking_api5t_tidE@plt>
   0x00007f5275c8885e <+590>:   mov    0x2695e3(%rip),%rax        # 0x7f5275ef1e48
   0x00007f5275c88865 <+597>:   cmpq   $0x0,0x269723(%rip)        # 0x7f5275ef1f90
   0x00007f5275c8886d <+605>:   lea    0x10(%rbx),%rbp
   0x00007f5275c88871 <+609>:   mov    %fs:(%rax),%r12
   0x00007f5275c88875 <+613>:   je     0x7f5275c88887 <_ZN6memray12tracking_api7Tracker19trackAllocationImplEPvmNS_5hooks9AllocatorE+631>
   0x00007f5275c88877 <+615>:   mov    %rbp,%rdi
   0x00007f5275c8887a <+618>:   callq  0x7f5275c70d20 <pthread_mutex_lock@plt>
=> 0x00007f5275c8887f <+623>:   test   %eax,%eax
   0x00007f5275c88881 <+625>:   jne    0x7f5275c88dcd <_ZN6memray12tracking_api7Tracker19trackAllocationImplEPvmNS_5hooks9AllocatorE+1981>
   0x00007f5275c88887 <+631>:   mov    0x8(%rbx),%rdi
   0x00007f5275c8888b <+635>:   cmp    0xb0(%rbx),%r12
   0x00007f5275c88892 <+642>:   je     0x7f5275c88d60 <_ZN6memray12tracking_api7Tracker19trackAllocationImplEPvmNS_5hooks9AllocatorE+1872>

Ok, this was very annoying to fix, but we think we got it in #289. We will do a release soon 👍

Thanks a lot for the report btw, and sorry for the trouble!

@pablogsal It's no trouble at all, thank you so much for this package and the quick turnaround. We know that sort of thing doesn't come free! 🙇

We released a new version with the fix yesterday.
Let us know if you experience any other problems ;)


Why not PJON?

Hi there, I really liked your project and how you engineered it. Maybe my opinion could be a little biased because I am the main developer, but, I think you should consider PJON as an alternative to i2c. Good luck with SensorTape.
https://github.com/gioblu/PJON

Reading the attached UIST document I see in the sections "limitations" and "future work" some issues PJON could accomplish efficiently.

Hi, thanks for sharing the protocol. It looks really interesting, and very nice implementation. I will definitely check it out in more details.
I looked at 1-wire and few other protocols, and there wasn't any good alternatives to I2C at the time.
I am currently trying to use serial over RS-485, it can be pretty fast with 0.5Mb/sec and 256 slaves with 8MHz clock.
Some quick questions about PJON:
What limits the speed to 5.3 kbit/sec? Could you use serial module to speed it up, instead of doing in software?
Is 254 devices a physical limit? Realistically they might add capacitance and impedance to the bus, thus limiting number of devices to a smaller number

Ciao @adementyev thank you again, and I am really pleased buy your comments on PJON :+1:
The 5.3kB/s is a bottleneck imposed by micros() resolution (4 microseconds, crap) infact for me is astonishing to see it working so precisely at these speeds.
I sincerely never considered the hardware serial usage and I would not take it for bias :+1:
Could you explain me in more detail what you imagine?
Because of the not enormous communication speed and the initial padding bits or sync procedure the capacitance effect is mild, I tried up to 30 devices (i don't have more) and it worked flawlessly and other users had similar results. Consider that PJON works flawlessly using a human body as a conductor passing from the left hand to the right one without effect in reliability!!
I also will add in 1.2 tri-state logic to completely exclude from the circuit all devices are not actively or passively using it, to further reduce the average capacitance on the bus.
Thank you again and compliments for you build.

@adementyev Serial over RS485 is really a reliable and professional choice, for sure. And is what in general I suggest for many applications, but, I think RS485 has a little old concept and its bias for reliability in some way limits its use in a real multimaster plug&use environment (a master collission could physically blow up a driver). PJON in some way is an attempt to fill this gap.
What I mean is that could be interesting to cut a piece of your tape and let it work indipendently as a single entity or temporarily join it to another one to higher the new entity's capability. On the other side could be nice to phisically build a network of buses (tape sections) in every possible configuration using packet routing with plug and use capability.


Sub-Task 13.1: View dedicated section for metrics

Task: Under the podcast and given you are the creator of said podcast, there will be a "Metrics" tab or button.

Deleting issues, linked to US#13


Plugin for general indexing of formatted output with Elasticsearch

Please describe what you would like to see in this project
Placeholder description; please expand the details.
Add a plugin that is capable of accepting output generally from other plugins, formatting the data into an opinionated document, and optionally indexing that data into an Elasticsearch instance or passing it on to another plugin.
Please describe your use case
This is the base functionality to replicate existing use cases for data retention.
Additional context

Initial Elasticsearch plugin is available.


Cargo psibase assumes local chain port

cargo psibase deploy currently assumes the local chain is running at http://psibase.<IP_ADDRESS>.sslip.io:8080/. There is a -a flag for cargo psibase, but cargo psibase -a http://psibase.<IP_ADDRESS>.sslip.io:8089/ deploy ... still fails with: error sending request for url (http://psibase.<IP_ADDRESS>.sslip.io:8080/common/tapos/head).
Instead, I think the chain location should be configured in the project's Cargo.toml file. Perhaps you can configure named deploy targets, like:
[psibase.hosts]
dev = "http://psibase.<IP_ADDRESS>.sslip.io:8079"
prod = "https://psibase.io"

And then you can use cargo psibase --host prod deploy
@swatanabe thoughts?

Sounds like the way to go! @James-Mart


release raiwidgets and responsibleai v0.19.0

Description
Release raiwidgets and responsibleai v0.19.0
Release Notes
v0.19.0

educational materials

Add pre-built cohort into adult census notebook by @gaugup in https://github.com/microsoft/responsible-ai-toolbox/pull/1243
Add pre-defined cohorts in responsibleaidashboard-diabetes-decision-making.ipynb by @gaugup in https://github.com/microsoft/responsible-ai-toolbox/pull/1252

new features

Responsible AI Dashboard

Add a summary for disaggregated analysis showing ratio and difference by @romanlutz in https://github.com/microsoft/responsible-ai-toolbox/pull/1367
Use global cohort for disaggregated analysis by @romanlutz in https://github.com/microsoft/responsible-ai-toolbox/pull/1370
Add accuracy as multiclass metric by @romanlutz in https://github.com/microsoft/responsible-ai-toolbox/pull/1371
Add a top-level pivot to model overview by @romanlutz in https://github.com/microsoft/responsible-ai-toolbox/pull/1375
Add a toggle to model overview to turn heatmap colors on and off by @romanlutz in https://github.com/microsoft/responsible-ai-toolbox/pull/1376
Add feature and metric configuration flyouts to model overview by @romanlutz in https://github.com/microsoft/responsible-ai-toolbox/pull/1381
Add probability distribution line chart to model overview by @romanlutz in https://github.com/microsoft/responsible-ai-toolbox/pull/1382
Provide feature flight support to dashboard, test environment, and Python widget by @romanlutz in https://github.com/microsoft/responsible-ai-toolbox/pull/1385

Error Analysis

add pyspark support to tree surrogate model in error analysis by @imatiach-msft in https://github.com/microsoft/responsible-ai-toolbox/pull/1251

bug fixes and tests

Responsible AI Dashboard

Fix bug on scatter points in dataset explorer; remove outlier for aggregate feature importance box chart by @tongyu-microsoft in https://github.com/microsoft/responsible-ai-toolbox/pull/1364
Change procedural loops to functions map() and filter() by @gaugup in https://github.com/microsoft/responsible-ai-toolbox/pull/1255
Create rai-e2e package for tests shared with AML by @vinuthakaranth in https://github.com/microsoft/responsible-ai-toolbox/pull/1378
Create e2e package for tests shared with AML by @vinuthakaranth in https://github.com/microsoft/responsible-ai-toolbox/pull/1383
fix race condition in tree view update code which can cause root stats to be ignored on selected cohort in RAI dashboard static view by @imatiach-msft in https://github.com/microsoft/responsible-ai-toolbox/pull/1386
String updates for model overview by @romanlutz in https://github.com/microsoft/responsible-ai-toolbox/pull/1387
Some fixes for data explorer, heatmap, counterfactual components by @tongyu-microsoft in https://github.com/microsoft/responsible-ai-toolbox/pull/1389
disable causal treatment whatif in static view by @imatiach-msft in https://github.com/microsoft/responsible-ai-toolbox/pull/1391
Fix feature selection state carry-over bug in model overview by @romanlutz in https://github.com/microsoft/responsible-ai-toolbox/pull/1390

RAIInsights

Rename _validate_model_analysis_input_parameters() to _validate_rai_insights_input_parameters() by @gaugup in https://github.com/microsoft/responsible-ai-toolbox/pull/1373

Counterfactual

Add missing desired range to counterfactual chart by @gaugup in https://github.com/microsoft/responsible-ai-toolbox/pull/1366
Cache and serialize the counterfactual explainer by @gaugup in https://github.com/microsoft/responsible-ai-toolbox/pull/1374
fix big data set for counter factual chart by @xuke444 in https://github.com/microsoft/responsible-ai-toolbox/pull/1393

Error Analysis

Interpret

Causal

Handle the loading of causal models more gracefully by @gaugup in https://github.com/microsoft/responsible-ai-toolbox/pull/1377
Alignment and style fixes for causal by @tongyu-microsoft in https://github.com/microsoft/responsible-ai-toolbox/pull/1392

other

Add more context to download print statement in fetch_dataset by @gaugup in https://github.com/microsoft/responsible-ai-toolbox/pull/1368
Update PR template to be more intuitive by @romanlutz in https://github.com/microsoft/responsible-ai-toolbox/pull/1116
Update CODEOWNERS file by @romanlutz in https://github.com/microsoft/responsible-ai-toolbox/pull/1369
update code owner by @xuke444 in https://github.com/microsoft/responsible-ai-toolbox/pull/1372

Checklist

[ ] I have added screenshots above for all UI changes.
[ ] Documentation was updated if it was needed.
[ ] New tests were added or changes were manually verified.

Codecov Report

Merging #1413 (a9d98cf) into main (c919d75) will increase coverage by 3.05%.
The diff coverage is n/a.

:exclamation: Current head a9d98cf differs from pull request most recent head f1e35a1. Consider uploading reports for the commit f1e35a1 to get more accurate results

@@            Coverage Diff             @@
##             main    #1413      +/-   ##
==========================================
+ Coverage   85.94%   88.99%   +3.05%
==========================================
  Files         100       34      -66
  Lines        4795     1518    -3277
==========================================
- Hits         4121     1351    -2770
+ Misses        674      167     -507

Flag
Coverage Δ

unittests
88.99% <ø> (+3.05%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...bleai/responsibleai/_managers/explainer_manager.py

...ai/responsibleai/rai_insights/rai_base_insights.py

raiwidgets/raiwidgets/__version__.py

responsibleai/responsibleai/_internal/constants.py

...sponsibleai/responsibleai/rai_insights/__init__.py

...ibleai/responsibleai/managers/explainer_manager.py

...i/responsibleai/managers/error_analysis_manager.py

...onsibleai/_tools/shared/attribute_serialization.py

.../responsibleai/_managers/counterfactual_manager.py

...ibleai/_tools/shared/state_directory_management.py

... and 56 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c919d75...f1e35a1. Read the comment docs.


install_kernel uses python2

Hello,
I spent 1 hour to find that problem :) Using Fedora 27, "python" is python 2.7. But I used "pip3" to install jupyterlab
I found that you are using a script to install lgo kernel where the first bang line is using "python", I had to change that to "python3". Maybe we must find a way to detect the right version ?
BTW: I'm creating a plot lib that already works well with lgo, I will ask you some questions for that
Other BTW: I guess I will write an article on medium.com to explain my plot lib that can only work with you almightly kernel. Really, you beat gopherlab :)

Thank you for the feedback. Actually, I did not think deeply about the environment where Python2 and Python3 coexist because I still stick to Python2 at work.
I updated README.md and added comments about how to install the kernel configuration with Python2 and Python3. Do you think this comment would have saved your 1 hour if it had been in README.md? Let me know if you have a better idea to avoid this Python version issue.

BTW: I guess I will write an article on medium.com to explain my plot lib that can only work with you almightly kernel.  I'm looking forward to your post on medium.com :)

I'm closing this bug now. Let me know if you still have a problem.


Start with development help: trying to just make fails with "make[1]: dirname: Operation not permitted"

Bug Description
I am trying to start development on a feature and run tests, but so far this is not off to a good start:
steps attempted:

followed guide here
added tag and user
tried cloning repo fresh in case of permissions problem
running make invariably runs into this issue:

# make
make
make[1]: dirname: Operation not permitted
make[1]: /bin/bash: Operation not permitted
make[1]: /bin/bash: Operation not permitted
make[1]: /bin/bash: Operation not permitted
make[1]: /bin/bash: Operation not permitted
make[1]: /bin/bash: Operation not permitted
make[1]: /bin/bash: Operation not permitted
make[1]: /bin/bash: Operation not permitted
make[1]: /bin/bash: Operation not permitted
make[1]: /bin/bash: Operation not permitted
make[1]: /bin/bash: Operation not permitted
make[1]: /bin/bash: Operation not permitted
make[1]: /bin/bash: Operation not permitted
Makefile.core.mk:299: warning: overriding recipe for target '/work/out/linux_amd64/'
Makefile.core.mk:299: warning: ignoring old recipe for target '/work/out/linux_amd64/'
make[1]: /bin/bash: Operation not permitted

Version
#  make -v
GNU Make 4.2.1
Built for x86_64-pc-linux-gnu

# docker version
Client: Docker Engine - Community
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:54:27 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:33 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Please point me towards what i am obviously doing wrong here :sweat:
Additional Information
No response

Upgrade docker will fix it

although the upgrade worked fine, i am now running into disk space issues when running make, seems like an IO_WAIT heavy copy in the docker run command, making me think it's some infinite loop of some kind, more details to come later, having to resort to just run go test for unit tests for now..

Any further feedback?

Otherwise let me know if there's anything else you want me to provide


Sudo improvements

This completes the sudo support first introduced in #760, by doing sudo in some more places where it is needed.
(An earlier draft of this PR was in #864, which explains some of the details.)
I tested this branch with a sudo-based installation, EL and CL, and it's working for me.
The main changes are in the first commit. The three commits after that are just some housekeeping, and non-critical.
Renaming cmd() to docompose() made a lot of changes. So it might be easier to read this PR commit-by-commit.

OK I'll need you to add something back in. cmd is user-facing, so even if it just calls docompose with "$@", it needs to exist.

Hmm. Why the change to return in upgrade_compose? The intent was that the script exits, so that if there are errors during the process, the user can handle those manually.

Thanks for this project by the way.
I believe it will keep Ethereum healthy this month!

On vacation btw and focusing on merge prep and getting taxes in. I'll get back to this.


Some question about explicit and implicit time ODE solvers

Hello, I'm trying to implement a time-dependent solver, and curious about how to use the Mult() and ImplicitSolve() functions of TimeDependentOperator class. For example, the easiest linear heat conduction problem $u_t-\nabla\cdot(\kappa\nabla u) = 0$.
In the annotations of ex9 and ex16, MFEM would write the ODE as du/dt = M^{-1}(Ku + b) to apply different ODESolvers, where M is mass matrix and K is stiffness matrix.
The Forward Euler and Backward Euler time discretizations are
$u^{n+1} = (1 - hM^{-1}K) u^n$
and
$u^{n+1} = (M+hK)^{-1}M u^n$,
respectively, where $h$ is the time step.
It seems that Mult() implements the du/dt = M^{-1}(Ku + b) ODE. However, ImplicitSolver() from ex9 or ex16 looks like neither du/dt = M^{-1}(Ku + b) nor $u^{n+1} = (M+hK)^{-1}M u^n$. Besides, shouldn't different ODESolvers have different linear system thus the T_solver in ex16 will be different? How to write a ImplicitSolver() ?

I also wonder why there's no FormLinearSystem in ex9 but is in ex16. Is it because the DG FE space?

Now I'm clear about the ImplicitSolver(), it's about resolving
$\frac{du}{dt} = -M^{-1}K(u+h)\frac{du}{dt}$
$\frac{du}{dt} = -(M+hK)^{-1}Ku$
and in BackwardEuler update $u_{n+1}$ as $u_{n+1} = u_{n} + h\frac{du}{dt} = u - h(M+hK)^{-1}Ku$.
This is different to the BackwardEuler I mentioned in the question, where I don't solve $\frac{du}{dt}$ first but solve $u_{n+1}$ directly as $u_{n+1} = (M+hK)^{-1}Mu_{n}$.
Then I have a question, which scheme should be applied in terms of quicker computation? Though I think the update $\frac{du}{dt}$ one should be used for convinience, since MFEM's ODESolver classes are designed based on it.


question(Solid_State_Telescope_(SST)_data.ipynb)

use your code ,I have not run .
first ,I use spider editor to run ,and the result is none.
also ,I use jupyter notebook ,but it exists error in 'import pyspedas  from pytplot import tplot, options'
I want to plot a spectrum, but I have not related document about this. can you help me ?

Sorry for the delay - just now seeing this!
I ran the SST notebook, and it seems to run. If you're seeing an error importing pyspedas, it's probably due to an installation issue. Can you share the error you're seeing?


Lack of EXIF tag

I'm downloading photo (jpeg format) with EXIF tags on my server using the method bot.dowload_file(...). The saved files does not contains EXIF tags.

I think Telegram strips that information from the file. Reading online, if you wanted this functionality it looks like the user would have to send the file as a document.


Check if colorspace is invertible

Hi,
I'm currently writing an API on top of OCIO and I was wondering if there's a programmatic way to detect if a colorspace is not invertible. This could be helpful to automatically detect when we should avoid applying a colorspace transformation (since we don't use looks, some of our colorspaces also apply cdl/luts to give the final graded look, so being able to filter out non invertible colorspaces is a quite useful feature).
My current understanding is that if in the FileTransform of a Colorspace we have a 1D LUT, then  the transformation is always invertible (it's a 1 to 1 mapping) but if we have a 3D LUT (a potential many to 1 mapping), then the transformation is not invertible unless we have another 3D LUT that goes in the other direction. And that's one first thing to keep in mind when checking for invertibility.
So I thought about checking the from_reference and to_reference directions of each colorspaces, if we have both on a colorspace definition, then it means we have all the informations on how to go to it and come back, which in my mind means it's invertible.
But if we take for example this colorspace:
- !<ColorSpace>
    name: Input - ARRI - V3 LogC (EI320) - Wide Gamut
    family: Input/ARRI
    equalitygroup: ""
    bitdepth: 32f
    description: |
      V3 LogC (EI320) - Wide Gamut

      ACES Transform ID : IDT.ARRI.Alexa-v3-logC-EI320.a1.v1
    isdata: false
    allocation: uniform
    allocationvars: [0, 1]
    to_reference: !<GroupTransform>
      children:
        - !<FileTransform> {src: V3_LogC_320_to_linear.spi1d, interpolation: linear}
        - !<MatrixTransform> {matrix: [0.680206, 0.236137, 0.083658, 0, 0.085415, 1.01747, -0.102886, 0, 0.002057, -0.062563, 1.06051, 0, 0, 0, 0, 1]}

it has got only a 1D Lut and a MatrixTransform. So I guess that if the matrix is invertible than the colorspace itself is actually invertible, even if it doesn't have a from_reference slot.
Am I wrong?
Does OCIO offer any kind of way to check for invertibility?
Thanks,
Valerio

Hi Valerio,
In OCIO v1, all of the transforms/operators that may be used to build a to_reference or from_reference are invertible except for Lut3D.  So if only the to_reference is available and the from_reference is needed (or vice versa), OCIO is typically able to build that on-demand.  In your example, even though only the to_reference is provided, OCIO will calculate the from_reference when needed.
In OCIO v2, we added the ability to also invert Lut3D and so all transforms are now invertible.  (This is a critical capability, in my opinion.)
Singular matrices are I believe the only remaining edge case that could cause inversion to fail.  But we are planning on fixing that for v2 so that a pseudo-inverse will be returned.
There is no API to check for invertibility, but getProcessor will throw if the inversion fails.  There are also other things that could cause it to throw (e.g. a FileTransform may load a file with a syntax error).  So maybe you would want to focus not just on invertibility, but rather on whether getProcessor succeeds?
Given that in OCIO v2 inversion will never fail, I will argue that no invertibility API call will be needed.
thanks,
Doug

Hi @doug-walker ,
thanks a lot for your reply - it's a real gem and it helped me a lot!
I imagined OCIO would do something like that (building on demand) but it's great to actually know it for sure.
Can I ask you how a Lut3D could be inverted, if for example 2 points on the source are being remapped onto 1 single point in the destination space? So given that 1 point on the destination, which one between the 2 would you pick when going backward?
Thanks a lot,
Valerio

In the cases I needed to know this, I just asked OCIO to create the processor (config->getProcessor) and it throws and exception if it can't (e.g., because there is a 3DLUT). If you write your own logic, there's always the chance it could differ from reality, so I just go to the single source of truth.

Hi Valerio,
Regarding the Lut3D inversion, if the forward direction collapses several inputs into one output, then it is obviously not possible to recover the original inputs.  In that case, the inverse maps to one of the set of inputs that gets collapsed together (usually the one on the edge of the bijective part of the domain).  Another way to think of it is that the result of a [Forward --> Inverse --> Forward] is equivalent to just [Forward].
The same is true for the Lut1D inversion.
Doug

Given that in OCIO v2 inversion will never fail, I will argue that no invertibility API call will be needed.

As per our discussions in the OCIO meeting, I believe it is worth making the distinction whether or not a particular transform is invertible by way of reciprocity, despite always succeeding.
Useful from a DCC vantage for UI here.
Does this seem reasonable @doug-walker ?

Yes Troy, as you wrote, that topic has come up.  We'll need to discuss more how to implement it.  For example, it seems to imply that the contents of the src and dst menus for a color space conversion should no longer be the same.  This is potentially something that could be added to the new app helpers that Patrick just did.

For example, it seems to imply that the contents of the src and dst menus for a color space conversion should no longer be the same

Can you elaborate on this a bit if you have time?
Thanks Doug!

I was referring to end-user-facing tools such as OCIOColorSpace Transform that have menus for a source and destination color space.  What I meant is that if we provide a way for config authors to indicate that a certain direction should not be allowed (from_ref/to_ref), then that means that certain color spaces will need to be removed from either the src or dst menus.  Today, apps just iterate over getColorSpace(i) to build the list, there is no concept of whether this is a src or dst list currently.


Consider making MapWithMut a public API

I am using the MapWithMut trait for take() in Fold. I realize it is designed for VisitMut but I have a use case where it makes sense to use take() to prevent an additional clone(). The comments in  the file indicate it is not for public use yet there is documentation that it can be used when implementing VisitMut as an optimization in preference to Fold.
I have some fallible functions that I use to build Stmt (and other node types). Because Fold, VisitMut etc are all infallible I want to build these nodes outside of the Fold implementation and then pass them in. The general pattern looks like this:
fn build_stmt() -> Result<Stmt> {
  // Generate the statement
}

fn fallible_builder(self) -> Result<Self> {
  let stmt = build_stmt()?;
  let mut stmt_fold = StmtFold {stmt};
  self.script.fold_with(&mut stmt_fold);
  Ok(self)
}

struct StmtFold {
  stmt: Stmt,
};
impl Fold for StmtFold {
  fn fold_script(&mut self, mut n:Script) -> Script {
    let stmt = self.stmt.take();
    // Do something with `stmt`
    n
  }
}

Would you consider making MapWithMut and therefore take() a public API?

Yeah, I felt that I should expose it while working on a transform for next.js.
But I should polish it before exposing.


Add convenience methods to get typed property values

MicrodataProperty.getValue returns the property value as a String. It is often required to then parse this value to an int or another basic type. We should provide convenience methods such as MicrodataProperty.getIntValue to ease this scenario.

Added the following MicrodataProperty convenience methods for common primitives:

getBooleanValue
getIntValue
getLongValue
getFloatValue
getDoubleValue

Didn't bother with byte, char nor short until there's a good enough use-case.


Support for DJI guidance module for collision avoidance/ visual odometry

I have already forked the official DJI guidance module:
https://github.com/jgoppert/Guidance-SDK-ROS
It is really just a c++ demo program wrapped into ROS.
The DJI guidance is a 5 direction stereo vision system with ultrasonic sensors using an FPGA for processing. It then sends obstacle distance and visual odometry over usb to ROS. It can also be configued to send depth images.
I plan to rewrite the official DJI ros node so that it publishes topics cleanly and the startup is configurable in the launch file.
Once I get this done I just need to know how to modify mavros to send the data cleanly to the px4 firmware so that we can use the data.
This could be handled with the following mavlink messages:
Messages
VISION_POSITION_ESTIMATE ( 102 )
VISION_SPEED_ESTIMATE ( 104 )
DISTANCE_SENSOR ( #132 )

But this means we will need to make sure mavlink_receiver can publish all of the distance sensor data from the guidance (5 directions).

I think consideration needs to be taken to make sure the interface for this device is the exact same as the interface for the ETHZ solution.

If the controller for collision avoidance is going to rely on the DISTANCE_SENSOR data, the behavior is going to be very basic, if we want to implement a VFH or potential field method, we will want a more exact direction of the closest point. The DJI guidance module is not setup to enable this, but the ETH device could.

@jgoppert count on me to aid you on this. mavlink_receiver is already ready to receive data from DISTANCE_SENSOR, we just need to define the proper ID numbers for each of the 5 sensors. Also, mavros is also prepared to load data into the Pixhawk through the distance_sensor plugin (I wrote it and tested it myself).
The only doubt I have now is how the the FCU will be handling each of the distance_sensor publishing. I assume it is using a multisubsriber/multipublisher framework.

Once I get this done I just need to know how to modify mavros to send the data cleanly to the px4 firmware so that we can use the data.

This is already done on the distance_sensor plugin. I can guide on how to use it if you want.

One thing: we have a plugin for publishing distance using mavros, but right now VISION_SPEED_ESTIMATE is not yet supported on the firmware. We can readd it to LPE, as it was one of the things I've done and made a PR to when you had that initial branch of LPE.

@TSC21 You got the DJI guidance to work on PixHawk? With collision avoidance?

Will interface general VIO framework.

Also, the DJI Guidance works badly without a DJI flight controller connected to it.
If you want to use it with the Pixhawk, you need the send the IMU information of the Pixhawk to the Guidance system with no information about it.


setting primary IP failes when IP assigments are removed and re-assigned

Environment:
nautobot 2.2.5
pynautobot: 2.2.0
Details:
When I remove all existing IP assignmets of an interface, then re-assign all IP addresses to the interface and set the primary IP of the device, the primary IP is NOT set. Using the rest API works well. This could be similar to issue 197!
The follwing code does not work, it deletes the primary IP of the device
# get device, interface and IP
device = nautobot.dcim.devices.get(name='lab.local')
interface = nautobot.dcim.interfaces.get(device_id=device.id, name='Loopback0')
ip_address = nautobot.ipam.ip_addresses.get(address='<IP_ADDRESS>')

# get list of assigments
id_list = nautobot.ipam.ip_address_to_interface.filter(interface=interface.id, ip_address=ip_address.id)

# remove assignments
for assignment in id_list:
    assignment.delete()

# re-assign
assigned = nautobot.ipam.ip_address_to_interface.create({'interface': interface.id, 'ip_address': ip_address.id} )

# save
save = device.save()
# set primary
success = device.primary_ip4 = ip_address
#save
device.save()

When I add the following line to the code
device = nautobot.dcim.devices.get(name='lab.local')

right before setting the primary IP, it works.
I was asked to open this issue (https://networktocode.slack.com/archives/C01NWPK6WHL/p1718816339526429)

Using the rest API works well

Can you provide the REST API calls that work for you?

I apologize for the delay in getting back to this.
I was unable to reproduce this error via pynautobot using the demo site (which is currently on 2.2.7) with the following code:
from pynautobot import api
url = "https://demo.nautobot.com/"
token = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
nautobot = api(url=url, token=token)

# get device, interface and IP
device = nautobot.dcim.devices.get(name='atl01-sdwan-02')
interface = nautobot.dcim.interfaces.get(device_id=device.id, name='Loopback0')
ip_address = nautobot.ipam.ip_addresses.get(address='<IP_ADDRESS>')

## REST OF YOUR ORIGINAL SNIPPET
....

When I remove all existing IP assignmets of an interface

Can you confirm your original code snippet is correct? Your snippet is only removing a single assignment since you are filtering the assignments by the interface and the IP ID's:
id_list = nautobot.ipam.ip_address_to_interface.filter(interface=interface.id, ip_address=ip_address.id)

Is this maybe a bug in your code or did you provide an incorrect snippet?


gh action CI

this is just a skeleton CI that build and publishes Hord to docker cloud.
we can treat the PR as a WIP until the CI tasks is complete with testing and any additional CI tasks that may be needed before a release.

@deantchi If this is still a WIP and not ready for review, can you change this to a draft?

recent run here

Looking great, thank you so much @deantchi!
Could we please rely on the existing dockerfile instead?
Thanks!

done!

Thanks @deantchi!
Looking more closely at the build process, would it be possible to run in parallel:

cargo test
docker build
and only publish the image to docker hub if the test passed?
This simple trick would halve the CI time!

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


ci: auto-close linked issues

Playing with a workflow to close issues on merge into next.
Closes #431

Interestingly this does work for the manually linked issue. But not for the text based one unfortunately, though this would be possible to parse manually so overall this could work. Not really super important though, so I'll park this for now :)


Error on creating a stream while closing

Prevent opening streams or requests while the parent stream is closing, to prevent deadlock.
%gJ9LjBEIPdEBx/CDimsi3rtlJFOy7og/wqWQahzkpls=.sha256

awesome, merging.

merged into 2.0.2


delete archive after extract

What does this PR do?
fix unreleased change, delete embedded python archive after extract
Motivation
accidentally removed in https://github.com/DataDog/datadog-agent/pull/29803
https://datadoghq.atlassian.net/browse/WINA-1176
Describe how to test/QA your changes
install Agent, ensure embedded3.COMPRESSED does not exist
Possible Drawbacks / Trade-offs
Additional Notes

/merge


proposed solution to fix the pagination issue

Hello @Edorka and thanks for your PR ! Seems a perfect fit to fix your issue #48
I'll add some tests in the future for pagination to ensure that we cover most cases


cassandra.contactPoints may not be empty

Hi
Is this support for apache-cassandra-2.0.2 ? As per the document cluster  internet address should be updated in contactPoints in YML file . When we update our internet address  it throwing below error  will you please let me know what is missing
contactPoints: ma-aaa.aaa.bbbb.com
config/config.yml has an error:

cassandra.contactPoints may not be empty

Regards
Pavan Shetty

Hi
Have you looked at any of the tests for an example of how to specify the contactPoints? It looks like you've just provided a String value when this should be a list. Check out https://github.com/composable-systems/dropwizard-cassandra/blob/master/src/test/resources/minimal.yml for an example.
The version of Cassandra shouldn't matter - but the version of the Cassandra driver does matter. So long as you use a compatible driver version (or just use the one provided in the transitive dependencies) you'll be fine.
Hope that helps!
Stuart


Address Binding Issue on Tencent Cloud Server: Binding to <IP_ADDRESS> vs [::] for Service Registration

Tencent Cloud servers cannot directly bind to a public IP address and port; they can only bind to <IP_ADDRESS>:port. However, in this case, the service is registered with the address [::]:port. How can this issue be resolved?

@lizhemingi

@lizhemingi I've reviewed the recently updated code. Thank you.


[wip] introduce go mod

This pr introduces the use of go modules.
There are still some things that need working on, for example how GO111MODULE=on is used everywhere to force the use of modules, since the code is most commonly run within $GOPATH where modules are currently opt-in.
staticcheck and errcheck also need to be revisited.

+245 −208,636 this is a drastic diet :slightly_smiling_face: I haven't had any significant experience with Go modules yet but I believe that GO111MODULE=on shouldn't be necessary if we move out of GOPATH. For CircleCI, it means removing the working_directory parameter (see https://circleci.com/blog/go-v1.11-modules-and-circleci/). But maybe you want it to work seamlessly for local dev?

staticcheck and errcheck also need to be revisited.

Is that because go get modifies mod.sum?
Obviously changes to Makefile.common need to go to prometheus/prometheus first. We had similar discussions in https://github.com/prometheus/prombench/pull/161 as prombench already uses Go modules.

Is that because go get modifies mod.sum?

It was failing because we ignore some imports in /vendor, but they aren't in /vendor anymore.

Obviously changes to Makefile.common need to go to prometheus/prometheus first. We had similar discussions in prometheus/prombench#161 as prombench already uses Go modules.

That's cool, is this something you're interested in working on? You've been helping to keep the common parts in sync between the two projects :)

Re prombench, @sipian said that he was going to be a bit offline for a couple of days. But sure, I'll do some homework on Go modules and try to understand what we need to get in Makefile.common.

@stuartnelson3, quick heads-up. I'm looking into go mod'ing the Prometheus repository and tackle the necessary updates to Makefile.common. Also we chatted briefly about it with @brancz and he was more inclined to keep the vendor/ directory with guarantees reproducible builds and protects us from version control systems being down.

@simonpasquier if you get a chance to push this forward now that the prometheus gomod pr is merged ... :)

Yep it is on my list :smiley:

Superseded by #1630


feat: add german translation

I added a translation to German as well as tests, as described in the contribution guidelines.

Codecov Report

Merging #50 (610cafa) into main (b9e15ef) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##              main       #50   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            1         1
  Lines          250       250
  Branches        44        44
=========================================
  Hits           250       250

Flag
Coverage Δ

core
100.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

@irrelevation Thank you!
I will merge and release this.

@all-contributors please add @irrelevation for translation and test


Implements ADR009: Trailing slash in HTTP resources

Context
https://github.com/openregister/openregister-java/blob/master/doc/arch/adr-009-trailing-slash.md
Changes proposed in this pull request
Redirect trailing slash paths to non-trailing slash
Guidance to review
Note this does not implement rel=canonical as suggested in the ADR, as while they were in the same ADR I would rather implement rel=canonical in a separate PR.

Note this does not implement rel=canonical as suggested in the ADR, as while they were in the same ADR I would rather implement rel=canonical in a separate PR.

👍


Update zio, zio-test, zio-test-sbt to 2.0.21

About this PR
📦 Updates

dev.zio:zio
dev.zio:zio-test
dev.zio:zio-test-sbt

from 2.0.15 to 2.0.21
📜 GitHub Release Notes - Version Diff
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "dev.zio" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "dev.zio" }
}]

labels: library-update, early-semver-patch, semver-spec-patch, commit-count:1

Superseded by #743.


Cannot find java.exe within Windows Subsystem for Linux

I'm trying to set up a Scala development environment within Windows Subystem for Linux (WSL) using both JRE and JDK installed on Windows i.e., not within WSL. I exported JAVA_HOME and JRE_HOME to the appropriate folders and alias java=java.exe. This works quite well: java -version runs and sbt does its job too. I can populate an ensime configuration with sbt ensimeConfig.
The issue is that ensime-vim cannot find Java when starting vim. I suspect this is due to the fact that ensime-vim thinks it's on a Linux system and looks for java within :java-home (as set in .ensime config). However, since it's an installation on Windows, it's actually java.exe. (This is probably related to #383.)
Is there some way to configure ensime-vim to look for java.exe instead of java?
Steps to reproduce

Set-up WSL to use JDK and JRE on Windows
Alias java to java.exe
Follow installation instructions of ensime-vim

Expected behaviour
I open a *.scala file in vim on WSL and "everything just works".
Actual behaviour
In vim ensime-vim comlains:
[ensime] Java not found or not executable, verify :java-home in your .ensime config

Hi Niklaas, could you try checking what Python's notion of os.name is when running under WSL? In Vim you can try this:
:py import os; print(os.name)

Also check the value of os.pathsep too, we might want to update that part of #383's changes to try to rely more on Python's platform-agnosticism help too.

Thanks for your prompt response.
:py import os; print(os.name) gives posix.
:py import os; print(os.pathsep) gives :.
For comparison: On gvim in Windows I get nt and ; respectively.

At the moment I don't have a Linux system at hand to check whether posix and : can be used to distinguish WSL from a normal Linux. Anyway, in my ZSH config I use uname -a | grep Microsoft to find out whether the shell is run within WSL or not:
> uname -a
Linux len-x1 4.4.0-43-Microsoft #1-Microsoft Wed Dec 31 14:42:53 PST 2014 x86_64 x86_64 x86_64 GNU/Linux

At the moment I don't have a Linux system at hand to check whether posix and : can be used to distinguish WSL from a normal Linux.

No, these values are the same as when running in Linux or OS X.
So, we could try checking for an executable named java.exe as a fallback even when os.name is posix, to handle this case of running in WSL when the Java installation comes from Windows. But the next question is how does java.exe -cp behave under these conditions? As a Windows executable is it necessary that the classpath entries given to it are separated by ; even though it's invoked from WSL?
I don't have an environment suitable for testing this, so your quickest route to a solution will likely come if you can try to experiment yourself. The code touched by #383 should be the relevant areas you need to look at: finding the java executable and passing it a classpath (that code could be updated to use os.pathsep.join(classpath) instead of a manual ternary, aside from the unknown point in question).
I'd also like to understand the use cases—I'm not a Windows person at all, so I wonder about why you have (and want) this setup of JRE/JDK installed in Windows, and presumably Vim and maybe Python, etc. in WSL. Might life be easier in other ways too if you installed Java in WSL?


Unable to Select article text

I'm unable to copy/paste/highlight text from the article view, neither through the keyboard, mouse, or Edit menu
NNN: Version 6.1.6 (6127)
MacOS: Sonoma 14.4.1 (23E224)

Can you provide exact steps to reproduce?

After running into this issue for a few days, it seems to have disappeared. I'll try to reproduce again

Could not reproduce


Update to a curl-rust that doesn't use OpenSSL on OSX

This should help continue to mitigate #1420 further

r? @huonw
(rust_highfive has picked a reviewer for you, use r? to override)

@bors: r+ beec3706bd3a269be00d50ef5c8547536ef129c5

:hourglass: Testing commit beec370 with merge b01770c...

:sunny: Test successful - cargo-linux-32, cargo-linux-64, cargo-mac-32, cargo-mac-64, cargo-win-gnu-32, cargo-win-gnu-64, cargo-win-msvc-32, cargo-win-msvc-64


kdePackages.qtpbfimageplugin: 3.1 -> 3.2

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/project/qtpbfimageplugin/versions.
meta.description for kdePackages.qtpbfimageplugin is: Qt image plugin for displaying Mapbox vector tiles
meta.homepage for kdePackages.qtpbfimageplugin is: https://github.com/tumic0/QtPBFImagePlugin
Updates performed

Version update

To inspect upstream changes

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 3.2 in filename of file in /nix/store/4l4nxdf72qg1zj4ghk27ksskfnr76wkb-qtpbfimageplugin-3.2

Rebuild report (if merged into master) (click to expand)

5 total rebuild path(s)

5 package rebuild(s)

First fifty rebuilds by attrpath

kdePackages.qtpbfimageplugin
libsForQt5.qtpbfimageplugin
plasma5Packages.qtpbfimageplugin
qt6Packages.qtpbfimageplugin

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/4l4nxdf72qg1zj4ghk27ksskfnr76wkb-qtpbfimageplugin-3.2 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A kdePackages.qtpbfimageplugin https://github.com/r-ryantm/nixpkgs/archive/db3e7ba816f1fc13762a893b4c89cb8190ee1085.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/db3e7ba816f1fc13762a893b4c89cb8190ee1085#kdePackages.qtpbfimageplugin

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/4l4nxdf72qg1zj4ghk27ksskfnr76wkb-qtpbfimageplugin-3.2
ls -la /nix/store/4l4nxdf72qg1zj4ghk27ksskfnr76wkb-qtpbfimageplugin-3.2/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review

x86_64-linux

:white_check_mark: 2 packages built:

kdePackages.qtpbfimageplugin
libsForQt5.qtpbfimageplugin

Maintainer pings
cc @sikmir for testing.

[!TIP]
As a maintainer, if your package is located under pkgs/by-name/*, you can comment @NixOS/nixpkgs-merge-bot merge to automatically merge this update using the nixpkgs-merge-bot.

Add a :+1: reaction to pull requests you find important.

Darwin build is broken on master, not a regression.


Dachsbun, Tinkatink, & Tinkaton have wrong sprites

Dachsbun, Tinkatink, Tinkatuff, & Tinkaton's sprites are for other Pokemon.
The correct sprites can be found in the links below:
https://bulbapedia.bulbagarden.net/wiki/Dachsbun_(Pokémon)
https://bulbapedia.bulbagarden.net/wiki/Tinkatink_(Pokémon)
https://bulbapedia.bulbagarden.net/wiki/Tinkatuff_(Pokémon)
https://bulbapedia.bulbagarden.net/wiki/Tinkaton_(Pokémon)
Edit:
I've done API calls to the pokemon above, and I see the id's are incorrect, which is probably why they are calling to incorrect sprite id's
The correct id's for the pokemon should be the following:
Dachsbun: #927
Tinkatink: #957
Tinkatuff: #958
Tinkaton: #959
Edit2:
Looks like a lot of other Pokemon form Generation 9 have incorrect ids in their API responses. See the following issue

I can confirm that there is incorrect data affecting generation 9 as a whole. Some point within the last 2 weeks, something has changed and needs to be reverted to the last working version ASAP. The last check I did where everything worked as intended was March 23.


Get phedata mk

Updated a lot and solved few merge conflicts that occurred. @juhis

@juhis can you finalize the review so that it is not left hanging here for long and harder to come back to the code later.

@juhis boing back


Docker buildx with k8s without specifying endpoint argument

What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #
Does this PR introduce a user-facing change?

Additional Notes for your reviewer:
Review Checklist:

[ ] Follows the developer guidelines
[ ] Relevant tests are added or updated
[ ] Relevant docs in this repo added or updated
[ ] Relevant carvel.dev docs added or updated in a separate PR and there's
a link to that PR
[ ] Code is at least as readable and maintainable as it was before this
change

Additional documentation e.g., Proposal, usage docs, etc.:

Also updated the github actions to there latest versions to get rid from the warning we are getting in our actions:
kapp-controller release
Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20<EMAIL_ADDRESS>carvel-dev/setup-action@v1, actions/setup-go@v4. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.

Good to merge once the test case passes.


Filter UI not reset properly in some circumstances

[ ] Resetting values to original levels doesn't clear the filter pillbox.

This works: uncheck 'unknown values' checkbox => pillbox added; recheck unknown values => pillbox removed
Doesn't work properly:
a) Slide value away from min or max;
b) slider value back to maximum/minimum
Actual: pillbox not removed, filter values are simply set to their min/max.
Expected behavior: resetting the slider to a level matching its original level will set that filter as 'cleared'.

[x] Pressing 'x' in the pillbox toggles the 'unknown values' checkbox to the wrong state in some circumstances. This occurs if both the checkbox and the slider have been used. This is caused by the first problem in combination with some incorrect logic about checkbox state.

Doesn't work properly:  For one of the slider data sets (e.g. Total units),

slide filter to some value
uncheck the checkbox
re-check the checkbox
click the 'x' in the pillbox.
Actual: Checkbox is switched to unchecked when the filter is cleared. This does not accurately reflect the state of the filters, which in reality do include nulls. Clicking the checkbox again once does nothing, and clicking a final time restores the sync of state to ui.
Expected: Checkbox remains unchanged when clearing filter because it is already checked.

#451 resolves the second issue. First issue still outstanding.

First issue is addressed in PR #458


Improving mockStore in WritingTests doc

Currently mockStore in the doc Writing Tests would produce done() called multiple times error when assertion error occurs. It causes mocha to produce uninformative reports, instead of useful assertion errors:
1) async actions shall not pass:
     Error: done() called multiple times
      at Suite.<anonymous> (test.js:57:3)
      at Object.<anonymous> (test.js:56:1)
      at loader (node_modules/babel-core/node_modules/babel-register/lib/node.js:127:5)
      at Object.require.extensions.(anonymous function) [as .js] (node_modules/babel-core/node_modules/babel-register/lib/node.js:137:7)
      at Array.forEach (native)
      at node.js:961:3

  2) async actions shall not pass:
     Error: done() called multiple times
      at Suite.<anonymous> (test.js:57:3)
      at Object.<anonymous> (test.js:56:1)
      at loader (node_modules/babel-core/node_modules/babel-register/lib/node.js:127:5)
      at Object.require.extensions.(anonymous function) [as .js] (node_modules/babel-core/node_modules/babel-register/lib/node.js:137:7)
      at Array.forEach (native)
      at node.js:961:3

The detailed analysis and the minimal project to reproduce this bug is listed in this gist: https://gist.github.com/MrOrz/50bda5e23db2c39713e1
This pull request fixes the bug by turning dispatch() in mockStore into no-op after done() is called.

Just found that this patch will cause tests to erroneously pass.
If there are actually more actions dispatched than expectedActions, the test will be marked as "passed", which is not desired.
Haven't come up a fix yet.

Rather than maintain this code, I changed the doc to point to redux-mock-store source. Feel free to improve that package!
Closed via 02ed21cd278951722e9367d840db55c03f016adb.


String translation security request

Would be nice to have esc_attr__ instead of just __ in the string translations, recommended for security reasons.

I don't get your pull request... where do you see a security issue exactly?
I can't see any __() functions changed to esc_attr__

translatable strings like in sample-config.php and any other position if eventually present, when you output translated strings is recommended to have a esc_attr__ as it will prevent breaches and exploit performed through the .mp .po files.
Example:
'title'       => ( 'Controls with Choices', 'kirki' ),
Fixed:
'title'       => esc_attr( 'Controls with Choices', 'kirki' ),
Source:
https://codex.wordpress.org/Function_Reference/esc_attr_2
This can useful when adding new fields to the customizer as it is a requirement for premium themes (and as well a good practice) so when doing copy-paste from the examples we don't need to add esc_attr to all the __ functions manually :)
Thanks!

@qantumthemes I've never heard of an exploit via .mo files... I think WP filters those properly before outputting them.
I think it's an overkill since WP properly sanitizes __() and we're talking about hard-coded values.
If __() were vulnerable, all themes and plugins out there would be in danger, which is clearly not the case.
However, I do see your point so I changed them so that theme devs don't have to do anything else. :+1:

@aristath actually WordPress Internationalizing guidelines recommends escaping string specifically so translators can not embed malicious code.
https://developer.wordpress.org/plugins/internationalization/how-to-internationalize-your-plugin/#escaping-strings
These functions should be use when applicable:
https://developer.wordpress.org/plugins/internationalization/how-to-internationalize-your-plugin/#escape-functions

Hah! Awesome!
I've never seen that before... thanks for the education! :+1:

I didn't knew it too ;) so it has been a hard job when my last themeforest product had to be all sanitized with esc_attr, a real pain :dancer:
Also is good to use esc_js and esc_url in certain cases.
Basically any dynamic content can be source of exploits, and po mo files are particularly vulnerable as are created by users and they can upload files with the wrong permissions very easily.
I've been hacked a couple of times and now I pay a lot of attention in these details :)
Anyway I'm playing these days with your plugin and I must say that is really cool!
I was using my own customizer class before, but the the elegant styling options that you added is really a better result!


Unsigned integer overflow

Compiling with clang++ and the `-fsanitize=unsigned-integer-overflow` flag
(http://clang.llvm.org/docs/UsersManual.html) exposes a few unsigned overflow
issues:

../src/miniz.c:1929:7: runtime error: unsigned integer overflow: 0 - 1 cannot
be represented in type 'mz_uint' (aka 'unsigned int')
../src/miniz.c:964:35: runtime error: unsigned integer overflow: 0 - 1 cannot
be represented in type 'size_t' (aka 'unsigned long')

Originally described at https://github.com/developmentseed/node-blend/issues/27

Original issue reported on code.google.com by<EMAIL_ADDRESS>on 8 Jan 2013 at 2:04

Does not occur anymore.


CHANGELOG: Remove I2C testing from TODO

@curat97 has reported that the I2C interface works with libcsp 2.0 in #547. Therefore, it is being removed from the TODO list.

as noted at https://github.com/libcsp/libcsp/issues/547#issuecomment-2066069870, it's not fully working yet. Closing this for now.


IElasticClient.GetAlias().IsValid throws NullReferenceException if Elasticsearch is not reachable

NEST/Elasticsearch.Net version: 6.3.1
Elasticsearch version: 6.2.4
Description of the problem including expected versus actual behavior:
IGetAliasResponse.IsValid throws NullReferenceException if Elasticsearch is not reachable. It should just return false.
Steps to reproduce:

Make sure Elasticsearch is not running.
Invoke IElasticClient.GetAlias(s => s.Name("any_name_will_do")).IsValid.

IsValid property will throw a null ref exception. Definition of this property is as under in this file:
public override bool IsValid => Indices.Count > 0;
Indices is null when Elasticsearch is not running and that results in this exception. Ideally, IsValid should never fail and its definition should be updated to as under:
public override bool IsValid => Indices != null && Indices.Count > 0;

Thanks for opening @bittusarkar. Your proposal looks reasonable to me. Would you be interested in opening a PR for this?

Sure, I can send a PR for this.

@russcam This does not reproduce anymore in 6.7.0. So I guess something got fixed.

Thanks for reporting back to us @bittusarkar, will close this.


Update network-protection-macos.md

Made changes to the commands as the names have changed
Reference - https://learn.microsoft.com/en-us/microsoft-365/security/defender-endpoint/mac-updates?view=o365-worldwide#set-the-channel-name
Fixes #https://github.com/MicrosoftDocs/microsoft-365-docs/issues/10315

@JohanFreelancer9 Please copy review

Learn Build status updates of commit 7d3c13e:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error: RuningBuildFailed] Some unexpected errors happened when running build, please open a ticket in https://aka.ms/SiteHelp and include the error report for our team to troubleshoot

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Learn Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings or suggestions unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

Learn Build status updates of commit 924e05d:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

microsoft-365/security/defender-endpoint/network-protection-macos.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

Learn Build status updates of commit f7064a9:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

microsoft-365/security/defender-endpoint/network-protection-macos.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel


#[serde_as(deserialize_as = …)] generates #[schemars(deserialize_with(…))], which is invalid

Due to https://github.com/jonasbb/serde_with/blob/master/serde_with_macros/src/lib.rs#L814, a #[serde_as(deserialize_as = …)] will generate #[schemars(deserialize_with(…))] if feature schemars_0_8 is enabled, but https://github.com/GREsau/schemars/blob/master/schemars_derive/src/attr/schemars_to_serde.rs#L8-L31 does not contain deserialize_with, only serialize_with. Generation of #[schemars(deserialize_with(…))] thus results in an compiler error. I suggest to use #[schemars(schema_with = "…")].
The error I get looks like this:
error: unknown schemars attribute `deserialize_with`
  --> cherub/src/foo/bar.rs:23:1
   |
23 | #[serde_as]

Thanks for finding this problem. Indeed, deserialize_with for schemars is not supported. It seems that serialize_with is only "supported" in that it doesn't cause compilation failures, but it won't change the generated schema. The problem is fixed in v3.8.1.


Use cases and shortcomings for the Responsive component

Why
Digital Platform developers traditionally wrote CSS media queries for all their responsive views. It wasn't possible to write unit tests for responsive design, plus writing CSS proved to be time-consuming in general. In response, the <Responsive /> component was made to tackle these issues.
What
The <Responsive /> component was included in TDS as a replacement for CSS media queries. It wraps the react-media community component. The key benefits are:

developers can write unit tests for responsive components
responsive design is visible within JSX, which maintains context and reduces cognitive strain
by wrapping other components, you can have responsive JS such as variables that show a different value on desktop and mobile viewports
performance improvements by letting JS decide which component view to render. This way you can have two or more DOM elements conditionally render rather than having them be present and depend on CSS

There are some known shortcomings with the <Responsive /> component:

the starter kit is missing user agent sniffing functionality, which causes issues with server-side rendering
there is a flash of content when using multiple <Responsive /> components with server-side rendering #461

Discuss
Has the <Responsive /> component fallen short for you? Do you have some particular pain points that we can reconcile?

On server side, instead of rendering both desktop and mobile version. I believe render nothing is better. This is the default behaviour of react-media. Somehow it is altered in the Responsive component.
I don't believe relying on the user-agent is a great idea for the following 2 reasons:

TDS should be a stand-alone lib instead of relying on the hosting app to provide additional informations
I think user-agent can only gives info about browser type and version, but not the width of the browser. (iPad mobile Safari portraits vs landscape )

Re: render nothing on server and default behaviour of react-media. -- Responsive component does not alter the behaviour of react-media.
Looking at react-media source, it uses a state property of matches to determine whether or not to render when there is no window. It is set to true by default so that it always matches. See here: https://github.com/ReactTraining/react-media/blob/master/modules/Media.js#L21-L27.
The Responsive component allows the use of the defaultMatches prop of react-media so that consumers can control this behaviour.
We could set defaultMatches to false in the Responsive component... but not sure if that is the way to go?

Re: user agent
Totally agree that TDS should be standalone! That is why we have not built any user agent stuff into the component and want to rely on the consuming app to build additional functionality on top of the low-level, raw TDS components. We want to keep global app level concerns out of TDS.
And yes, user agent only allows you to make an educated guess about the client's device characteristics, which might be "good enough"?

We could set defaultMatches to false in the Responsive component... but not sure if that is the way to go?

I'm afraid that setting defaultMatches to false by default is not the way to go due to the SEO implications (and defeats the purpose of having an isomorphic starter kit).

@lzcabrera you beat me to it. Yes!
I might suggest a mobile first approach to rendering: ie. when in doubt optimize for mobile.
But all important content should be server rendered. That is essential

See #461.
For reference, we are likely implementing user-agent detection so that the server can intelligently decide what to render depending on the client. This will have the added benefit of potentially small payloads on initial render vs plain media queries.

^^ I feel like user agent detection was a rabbit hole that mobile first design was intended to resolve.
Is this component necessary? I feel like we're potentially adding to the problem rather than removing it.
What will bots see?

Also without doing any detailed analysis a reduction in html on the server side render is unlikely to result in a sizable performance gain relative to the size of your js bundle.
Any idea if this component is in wide use? I might suggest deprecating it quickly and revisiting its value. Just not sure if this is currently solving a problem.

@githubjosh Thanks for your comments and both issues! Good points.

Is this component necessary? I feel like we're potentially adding to the problem rather than removing it.

I think it may be worth revisiting the value of this component given the noted issues. Again, we did not have this issue prior. What problem are we trying to solve here?

I think its best to give the full picture here:
What is the Responsive component?
A component interface to media queries through JavaScript.
What's wrong with media queries in CSS?
CSS media queries are essentially conditional logic embedded in CSS, which (among other things) contributes to making CSS codebases difficult to maintain and reason about over time.
CSS media queries suffer from:

the corresponding markup often must contain multiple elements for all queries
having CSS logic that is separate from markup and JS logic is hard to conceptualize
you cannot easily test CSS media queries
it is difficult to optimize your CSS bundles for various clients
CSS media queries are limited in power to only styling

What does the Responsive component provide?

you can render a component or not based on the result of the query
you can render one component or another based on the query
you can write unit tests for media query behaviour
the component is more likely to be able to easily expand to other platforms (think react-native-web)
because it's "just a component", consumers can easily compose it to build higher order components without reimplementing things (think responsive grid)
with a little more smarts, we can optimize critical path styles and markup for faster load times

Looking at the bigger picture, because TD has embraced "component architecture", providing a JS-first, component-oriented interface to the design system fits nicely. By standardizing the interface across TDS to just components, we can lessen the cognitive load of using the system. If you need something from TDS, it's a component.
What's the catch?
Because Responsive uses window.matchMedia, when rendering on the server the component does not know which styles of components to render. We are seeing a brief flash of content in the browser when the server rendered markup/styles differs from the client side version.
We believe this is due to a delay in JS execution on the client. The pre-rendered app is mounted, styles are applied and the first paint happens, then the componentDidMount is called, triggering the media query, which causes the visible shift in style.
The hypothesis
We can use a contained bit of user agent detection (which seems to be the way others are going too in this situation) to control how the Responsive component behaves on the server, so that is can make a good enough guess at the client and know how to render itself before returning to the client. This should fix the flash of content, while also giving a slight optimization to the initial payload.
Conclusion
I believe that all of this creates a pretty compelling story for the Responsive component. This component in particular opens up a lot of future possibilities and enables a more powerful and expressive way of writing responsive Web applications.

@githubjosh

I feel like user agent detection was a rabbit hole that mobile first design was intended to resolve.

Agreed. We don't intend for this to in any way change how design is done, or replace mobile-first design.
If we do go this route, we could keep the user agent detection contained in scope, and not available for nefarious use.

@githubjosh

What will bots see?

Good question! We would have to keep that in mind. I would propose no change in current behavior, bots are served the default rendering of the app. No special behaviour should be applied to bots is the best thing to do I believe. Serve them just as you would a human in a web browser.

Wanted to update here...
Been having lots of great conversations with others on this. We are currently re-evaluating the implementation of the Responsive component, and re-considering the advice of using user agent detection to help alleviate the server side rendering issues.
Thanks for your patience and feedback. Stay tuned for more details.

Refactor TDS components to use the Sass mixins as needed (e.g. Button, Box, DisplayHeading, Heading, ExpandCollapse) (non-breaking patches)

This item is causing a bug in Annual Reports animation (it is triggering the animation twice), and our release date is coming soon (less than two weeks). See the current behaviour here (same user/pass as wcstage):
https://wcpreview.telus.com/en/bc/about/investors/annualreports/2017
Is it possible to release this particular fix as a patch sooner than later? Not having this will impact our designs and we won't have time to pivot to a new one in time for the release.

Patched in this release: https://github.com/telusdigital/tds/releases/tag/%40tds%2Fcore-responsive%401.1.0


Show scientific name in species drop-down list

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 98.31%. Comparing base (92a6bd9) to head (105cc6b).

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##             main      #84   +/-   ##
=======================================
  Coverage   98.31%   98.31%
=======================================
  Files          59       59
  Lines        2907     2907
  Branches      403      403
=======================================
  Hits         2858     2858
  Misses         30       30
  Partials       19       19

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


DO NOT SUBMIT: Allow mixer config default namespace to be configurable

WIP. DO NOT SUBMIT.
This PR is a proposal for allowing the default config namespace used by Mixer to be configurable. It reuses the updateVersion.sh script that is also controlling ISTIO_NAMESPACE.
Release note:

<PLEASE PUT RELEASE-NOTE HERE, PUT NONE if NO RELEASE-NOTE>

@douglas-reid: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-suite-rbac-auth.sh
85169716a30df6678b1e6e03960f3c759b33f7c9
link
/test e2e-suite-rbac-auth

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@douglas-reid: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-suite-rbac-auth.sh
85169716a30df6678b1e6e03960f3c759b33f7c9
link
/test e2e-suite-rbac-auth

prow/e2e-suite-rbac-no_auth.sh
85169716a30df6678b1e6e03960f3c759b33f7c9
link
/test e2e-suite-rbac-no_auth

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


@return object {...} phpdoc misparsed

Documented type (mixed | null) -> return does not match actual (mixed) -> object for client_hint
...
/** ...
* @param mixed|null $request
* @return object {
* ...
* }
*/
function client_hint($request) {

A bit of stupidity here - basically:
if(md = foo.match(/.../)) {
    if(md[1].match(/.../)) {
        foo = md[1]
    }
}

Second one was supposed to be capturing. Fixed.


note about retracted versions added

Thanks for contributing to the Harness Developer Hub! Our code owners will review your submission.
Description

Please describe your changes: \ adding info boxes about retracted versions due to a critical bug.
Jira/GitHub Issue numbers (if any): \ FFM-11481
Preview links/images (Internal contributors only): __________________

PR lifecycle
We aim to merge PRs within one week or less, but delays happen sometimes.
If your PR is open longer than two weeks without any human activity, please tag a code owner in a comment.
PRs must meet these requirements to be merged:

[X] Successful preview build.
[X] Code owner review.
[X] No merge conflicts.
[X] Release notes/new features docs: Feature/version released to at least one prod environment.

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://66472f729e6c50a6b6b16aa3--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://66473f3d56a651b03b08f92e--harness-developer.netlify.app


NoDrop

Enabled the commented out code for NoDrop Confirmation Dialog added setting LootNoDrop to Ini settings.
Re-ordered the looting checks. to check lore first, if lore and  we don't already own one check no drop check if we want to loot no drop. if just no drop check if we want to loot no drop.

newest push should address the above.


Removing unused maven dependencies

There are some declared but not used dependencies. This pull requests suggests to clean them up.

@orange-buffalo ; only comment is that sfl4j-simple is used by the tests, so this needs to be kept.

@garethahealy my bad, fixed that.


Wisteria leaves can generate at distance 7

Some wisteria leaves generate naturally at distance 7, decaying shortly afterwards

Who added the big wisterias in again they're still broken >:(

I feel like I am soon to be stabbed

you are


Can't Shadow Existing Session

I'm running on Windows 10.  Below is the screenshot of the settings:

As shown, shadowing is enabled.  Yet when I RDP into the computer with the same credentials as the one physically logged on, it doesn't shadow the exiting session.  Instead, it creates a new session with those credentials.  So, if the user is John, in Task Manager, under Users, I see you rows with "John",  but I can't interact with the user, as it's a different session.
What am I doing wrong?
Thanks,

What am I doing wrong?

Creating the issue that already exists: #247.


Test Differ#pick_encoding; move to EncodedString

Followup to https://github.com/rspec/rspec-support/pull/151
especially see https://github.com/rspec/rspec-support/pull/134#issuecomment-68984440

I thinking of simplifying this PR to

Testing Differ#pick_encoding
Moving pick_encoding into EncodedString

Then, in another PR, per any discussion in this one, change the tests in the differ_spec.rb to use an encoding-agnostic be_identical_string matcher, such as in this changset, or just use object equality, for now.

Since I'm going to revise this PR a bit, I thought I might as well push the ArgumentError on EncodingString#split fix for a preview of what it is, though it would go into a subsequent PR.

I want to keep this moving, so I've reduced this PR to just the Differ/EncodedString#pick_encoding

Thanks @bf4

(Note @myronmarston I haven't added a change log because this is pretty much just an internal refactoring)


Update packages & rust

Just maintenance to keep this Docker image updated.

Thanks for this. CI should build the new image and upload it to Docker Hub automatically.

@Michael-F-Bryan Can you please also tag (or release) it?

As well as using latest, the image should be automatically tagged on Docker Hub with the merge commit's hash.
I don't really think it makes sense to create a GitHub release or tag the commit in GitHub because there is no one version number you can use.


Unable to run the application. Getting error No such file exists robots933456.txt'

I am running the angular application and deploying it on Azure.
But it's not running up as it is looking for the robots933456.txt route.
It wasn’t the case before today.
Please guide me on how to proceed further.

Thanks @g138  for reaching out to us , but at this time it does not look like you are referring to a document but to a specific scenario . If that's the case we request you to please post your question on Microsoft Q&A and other forums , we have our engineers and others who are monitoring them and will be happy to help .
If you have a document which you are following and if you think it needs  an update , please let us know about that here .

Since we have not heard back , going ahead and closing the issue.


History page takes time to load

Description
When I navigate to other tabs and come back to the History page the round rainbow ball is observed. It takes longer to load this page than the others.
Steps to reproduce
Navigate to the other tabs and come back to the History tab.
Additional context
No response
Hedera network
other
Version
0.10.0-alpha.1
Operating system
No response

Accepting new tasks from the home page also shows the rainbow round ball which did not happen before this feature was introduced


Bump golang.org/x/sys from 0.0.0-20210514084401-e8d321eab015 to 0.1.0

supersedes #72 and #73
Testing done

Submitter checklist

[ ] Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
[ ] Ensure that the pull request title represents the desired changelog entry
[ ] Please describe what you did
[ ] Link to relevant issues in GitHub or Jira
[ ] Link to relevant pull requests, esp. upstream and downstream changes
[ ] Ensure you have provided tests - that demonstrates feature works or fixes the issue

Will close this PR if #79 goes through

Closing in favor of #79


Improve test coverage

Pull Request Test Coverage Report for Build 390

0 of 0 (NaN%) changed or added relevant lines in 0 files are covered.
6 unchanged lines in 2 files lost coverage.
Overall coverage increased (+0.3%) to 77.574%

Files with Coverage Reduction
New Missed Lines
%

src/index.js
2
81.94%

src/helpers.js
4
75.43%

Totals

Change from base Build 387:
0.3%

Covered Lines:
253

Relevant Lines:
304

💛  - Coveralls