M1 LLVM Runtimedyld Invalid page reloc value assertion error

We are seeing a LLVM Assertion error occurring randomly in our build farm.
The error message is:
Assertion failed: (isInt<33>(Addend) && "Invalid page reloc value."), function encodeAddend, file /path/to/conda-bld/llvmdev_1643905487494/work/lib/ExecutionEngine/RuntimeDyld/Targets/RuntimeDyldMachOAArch64.h, line 210

Earliest report is from gitter on July 15, 2022
The error can be triggered with the below script on bdb2384. The error usually occurs within 10 iteration.
!python setup.py build_ext --inplace
c = 0
_exit_code = 0
tests = """
numba.tests.test_stencils.TestManyStencils.test_basic40
numba.tests.test_stencils.TestManyStencils.test_basic70
numba.tests.test_array_constants.TestConstantArray.test_too_big_to_freeze
numba.tests.test_array_manipulation.TestArrayManipulation.test_fill_diagonal_basic
""".split()
cmdarg = ' '.join(tests)
while _exit_code == 0 and c < 150:
    print(f"c={c}".center(80, '='))
    !NUMBA_OPT=0 python -m unittest -vb $cmdarg $cmdarg
    c += 1
    print(f"exit={_exit_code}")
    assert _exit_code == 0

The error occurs in both LLVM 11 and LLVM 14.
The current hypothesis is that the LLVM Runtimedyld is mishandling far jumps. To relate this to the reproducer above, the situation can be created by:

first JITing some stencil kernels, which tend to be large and esp. larger when OPT=0
allocating large amount of memory as in test_too_big_to_freeze (the compilation and execution bits in the tests can be commented out and it will still trigger the error)
JITing more array operations as in test_fill_diagonal_basic. The assertion error occurs here. The guess is that JITed code emitted for the stencil tests are reused here. The large allocation in between help make sure there is a gap/fragmentation in the memory space such that the fill_diagonal functions are JITed in somewhere far away.

Julia devs is pointing to a broken large code model in LLVM Runtimedyld for MachO aarch64. See https://github.com/JuliaLang/julia/issues/42295#issuecomment-1008427270, https://github.com/JuliaLang/julia/pull/43664.

Dear users, i'm trying to run a script for a python3 program and i get this error 9/10. My script works, but randomly. When the code is too long, sometimes the error appears, some times it doesn't. So i have to run the script multiple times in the hope that it arrives at an end. I paste you the error that i get:
Assertion failed: (isInt<33>(Addend) && "Invalid page reloc value."), function encodeAddend, file /Users/ci/miniconda3-arm64/conda-bld/llvmdev_1643905487494/work/lib/ExecutionEngine/RuntimeDyld/Targets/RuntimeDyldMachOAArch64.h, line 210.
zsh: abort      python3 test2.py
I have no idea how to solve this error, i don't even have idea if i can really solve it or if it depends from llvm. Do you know something about that?
Thanks in advance, i hope in an answer...

Dear users, i'm trying to run a script for a python3 program and i get this error 9/10. My script works, but randomly. When the code is too long, sometimes the error appears, some times it doesn't. So i have to run the script multiple times in the hope that it arrives at an end. I paste you the error that i get:
Assertion failed: (isInt<33>(Addend) && "Invalid page reloc value."), function encodeAddend, file /Users/ci/miniconda3-arm64/conda-bld/llvmdev_1643905487494/work/lib/ExecutionEngine/RuntimeDyld/Targets/RuntimeDyldMachOAArch64.h, line 210. zsh: abort python3 test2.py
i'm running in a macbookPro with M1Pro. I have no idea how to solve this error, i don't even have idea if i can really solve it or if it depends from llvm. Do you know something about that? Thanks in advance, i hope in an answer...

@Francyrad thank you for asking about this. You do indeed encounter the same error as reported in this issue. You may consider Numba (more precisely, LLVM) to be broken on M1. There is currently no known fix or workaround and we are not sure if this has been reported upstream to LLVM or if there is a fix in progress. IIRC @sklam also checked LLVM 14 and it appears as though this has not be fixed. My only remaining guess here would be to try to run your script in a docker container on the M1 using a linux-aarch64 docker image. Performance should not be too bad as the hardware will not be simulated in this case. Note however, that I am guessing at this and it may very well also not work.
TL:DR Running Numba on an M1 may cause the segfaults you see above and the only known workaround is to use different hardware.

@esc thank you for your answer, i hope someone will be able to fix that. Please, let me know when it will be fixed commenting this issue
Thank you again

@esc thank you for your answer, i hope someone will be able to fix that. Please, let me know when it will be fixed commenting this issue
Thank you again

Yes, we hope so too, if you subscribe to this issue, you will receive updates regarding this quest.

Hi, is there any update on this? I'm under Python3.9 and LLVM 11.1.0 and M1 mac, and am having the same issue right now when running multi-processing of a forecast model (AutoCES) under statsforecast package.
I've tried to bootstrap dev versions of both numba (0.57.0.dev0+1257.gce69f3010) and  llvmlite 0.40.0.dev0+70.ge6901e0) from github repos, and still failed and keep facing this issue.
It seems like the temporary fix by https://github.com/numba/numba/pull/8583 is not working for me.
I have other models tested without issues, but they're all with the numba in the backend to speed up the computing. The only difference that I can think of is this specific model using complex values rather than some real-number values.
Can anyone help here? Thanks

I have still the issue. Sometimes I waste more of my time try to running my scripts instead of working

It seems like the temporary fix by #8583 is not working for me.

#8583 only disables the tests so that we can complete the test-suite and ship the package, so it won't actually help with the issue.

Can anyone help here? Thanks @esc @sklam

No, unfortunately not, there is no known workaround, it's broken in LLVM 11 and 14 (supported by next Numba/llvmlite release). I am not aware of anyone working on a fix at present, so your best bet for now will be to use non-M1/Apple silicon, i.e. change hardware. So sorry I don't have better news for you.
@sklam for reference, was this ever reported to the LLVM issue tracker and if so, can you post the issue ID please? Thank you.

Just wanted to mention that I'm having the same issue on Mac M1, llvm-openmp 16.0.2 and llvmlite 0.40.0! I run into this issue when solving systems of PDEs using py-pde. I've subscribed to this issue and fingers crossed that it will get fixed in the near-future.

@iamlll another bug that I don't is that the parallelisation with OpenMP don't work with the following chips:
M1Pro, M1Max and M1Ultra
It works just with M1
Is there some llvm where is it possible to do some report?

Just wanted to mention that I'm having the same issue on Mac M1, llvm-openmp 16.0.2 and llvmlite 0.40.0! I run into this issue when solving systems of PDEs using py-pde. I've subscribed to this issue and fingers crossed that it will get fixed in the near-future.

@iamlll The reason you are seeing this with llvmlite 0.40.0 is because it is based on LLVM 14 and that is indeed buggy.

buggy.

So how can we solve the problem
With OpenMP?

This a problem of the LLVM JIT that we are using (MCJIT) and we need to migrate to OrcJIT (https://github.com/numba/llvmlite/pull/919) so we can use JitLink and hopefully that will fix it.

buggy.

So how can we solve the problem With OpenMP?

This issue is about M1 LLVM Runtimedyld Invalid page reloc value assertion error -- you are inquiring about a different issue here. In order to keep the signal-to-noise low, please open a new issue with the OpenMP issues you are seeing, thank you!

The issue is not limited to Apple M1 or MacOS. We're seeing it on Neoverse-N1 running Ubuntu 20.04 ever since we've uprgraded to Numba 0.57. This is a server machine - and not just one. Unfortunately, we cannot downgrade because we need CUDA 12 support.
Error message:
python: /root/miniconda3/envs/buildenv/conda-bld/llvmdev_1680642098205/work/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp:507: void llvm::RuntimeDyldELF::resolveAArch64Relocation(const llvm::SectionEntry&, uint64_t, uint64_t, uint32_t, int64_t): Assertion `isInt<33>(Result) && "overflow check failed for relocation"' failed.

System info:
uname -a: Linux <hostname_redacted> 5.15.0-46-generic #49~20.04.1-Ubuntu SMP Mon Aug 8 18:51:21 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

cat /etc/os-release | grep PRETTY
  PRETTY_NAME="Ubuntu 20.04.4 LTS"

lscpu
  Architecture:                    aarch64
  CPU op-mode(s):                  32-bit, 64-bit
  Byte Order:                      Little Endian
  CPU(s):                          80
  On-line CPU(s) list:             0-79
  Thread(s) per core:              1
  Core(s) per socket:              80
  Socket(s):                       1
  NUMA node(s):                    1
  Vendor ID:                       ARM
  Model:                           1
  Model name:                      Neoverse-N1
  Stepping:                        r3p1
  Frequency boost:                 disabled
  CPU max MHz:                     3000.0000
  CPU min MHz:                     1000.0000
  BogoMIPS:                        50.00
  L1d cache:                       5 MiB
  L1i cache:                       5 MiB
  L2 cache:                        80 MiB
  NUMA node0 CPU(s):               0-79
  Vulnerability Itlb multihit:     Not affected
  Vulnerability L1tf:              Not affected
  Vulnerability Mds:               Not affected
  Vulnerability Meltdown:          Not affected
  Vulnerability Mmio stale data:   Not affected
  Vulnerability Retbleed:          Not affected
  Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
  Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
  Vulnerability Spectre v2:        Mitigation; CSV2, BHB
  Vulnerability Srbds:             Not affected
  Vulnerability Tsx async abort:   Not affected
  Flags:                           fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs

free -m
                total        used        free      shared  buff/cache   available
  Mem:         514318        5282       73564           6      435471      504537
  Swap:          2047         213        1834

I can confirm being able to reproduce a similar issue on a non-M1 AArch64 - in general we can overflow relocations - the assertion is a little different because Linux on AArch64 us using RuntimeDyldELF and not RuntimDyldMachO, but I think the principle (and the root cause) is the same, but I need to investigate further to be sure. At present I'm reproducing with DALI like:
/opt/dali/dali/test/python# DALI_EXTRA_PATH=/opt/dali_extra python -m nose2 --verbose --plugin=nose2_test_timer.plugin --with-timer --timer-color --timer-top-n 20 -A '!slow' -s operator_1 test_numba_func
test_numba_func.test_multiple_ins ... ok
test_numba_func.test_split_images_col ... ok
test_numba_func.test_numba_func:1
[(10, 10, 10)], <class 'numpy.uint8'>, <function set_all_values_to_255_batch at ... ok
test_numba_func.test_numba_func:2
[(10, 10, 10)], <class 'numpy.uint8'>, <function set_all_values_to_255_sample a ... python: /root/miniconda3/envs/buildenv/conda-bld/llvmdev_1680642098205/work/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp:507: void llvm::RuntimeDyldELF::resolveAArch64Relocation(const llvm::SectionEntry&, uint64_t, uint64_t, uint32_t, int64_t): Assertion `isInt<33>(Result) && "overflow check failed for relocation"' failed.
Aborted (core dumped)

and I need to figure out how to make a Numba-only reproducer.
I'm working on a system very similar to the one reported by @mzient in https://github.com/numba/numba/issues/8567#issuecomment-1556803212 - just some small minor OS / kernel version differences.

I couldn't trigger this issue with @sklam's script from https://github.com/numba/numba/issues/8567#issue-1432286236, even after hundreds of runs on a Linux AArch64 system. However, the following (still using DALI, but without needing a test harness) does reproduce the issue pretty reliably:

test_standalone.py
import numpy as np
from nvidia.dali import pipeline_def
import nvidia.dali as dali
import nvidia.dali.fn as fn
import nvidia.dali.types as dali_types
from nvidia.dali.plugin.numba.fn.experimental import numba_function

def set_all_values_to_255_batch(out0, in0):
    out0[0][:] = 255

def set_all_values_to_255_sample(out0, in0):
    out0[:] = 255

def set_all_values_to_float_batch(out0, in0):
    out0[0][:] = 0.5

def set_all_values_to_float_sample(out0, in0):
    out0[:] = 0.5

def setup_change_out_shape(out_shape, in_shape):
    out0_shape = out_shape[0]
    in0_shape = in_shape[0]
    perm = [1, 2, 0]
    for sample_idx in range(len(out0_shape)):
        for d in range(len(perm)):
            out0_shape[sample_idx][d] = in0_shape[sample_idx][perm[d]]

def change_out_shape_batch(out0, in0):
    for sample_id in range(len(out0)):
        out0[sample_id][:] = 42

def change_out_shape_sample(out0, in0):
    out0[:] = 42

def get_data(shapes, dtype):
    return [np.empty(shape, dtype=dtype) for shape in shapes]

def get_data_zeros(shapes, dtype):
    return [np.zeros(shape, dtype=dtype) for shape in shapes]

@pipeline_def
def numba_func_pipe(shapes, dtype, run_fn=None, out_types=None, in_types=None,
                    outs_ndim=None, ins_ndim=None, setup_fn=None, batch_processing=None):
    data = fn.external_source(lambda: get_data(shapes, dtype), batch=True, device="cpu")
    return numba_function(
        data, run_fn=run_fn, out_types=out_types, in_types=in_types,
        outs_ndim=outs_ndim, ins_ndim=ins_ndim, setup_fn=setup_fn,
        batch_processing=batch_processing)

def _testimpl_numba_func(shapes, dtype, run_fn, out_types, in_types,
                         outs_ndim, ins_ndim, setup_fn, batch_processing, expected_out):
    batch_size = len(shapes)
    pipe = numba_func_pipe(
        batch_size=batch_size, num_threads=1, device_id=0,
        shapes=shapes, dtype=dtype,
        run_fn=run_fn, setup_fn=setup_fn, out_types=out_types,
        in_types=in_types, outs_ndim=outs_ndim, ins_ndim=ins_ndim,
        batch_processing=batch_processing)
    pipe.build()
    for _ in range(3):
        outs = pipe.run()
        for i in range(batch_size):
            out_arr = np.array(outs[0][i])
            assert np.array_equal(out_arr, expected_out[i])

def test_numba_func():
    # shape, dtype, run_fn, out_types,
    # in_types, out_ndim, in_ndim, setup_fn, batch_processing,
    # expected_out
    args = [
        ([(10, 10, 10)], np.uint8, set_all_values_to_255_batch, [dali_types.UINT8],
         [dali_types.UINT8], [3], [3], None, True,
         [np.full((10, 10, 10), 255, dtype=np.uint8)]),
        ([(10, 10, 10)], np.uint8, set_all_values_to_255_sample, [dali_types.UINT8],
         [dali_types.UINT8], [3], [3], None, None,
         [np.full((10, 10, 10), 255, dtype=np.uint8)]),
        ([(10, 10, 10)], np.float32, set_all_values_to_float_batch, [dali_types.FLOAT],
         [dali_types.FLOAT], [3], [3], None, True,
         [np.full((10, 10, 10), 0.5, dtype=np.float32)]),
        ([(10, 10, 10)], np.float32, set_all_values_to_float_sample, [dali_types.FLOAT],
         [dali_types.FLOAT], [3], [3], None, None,
         [np.full((10, 10, 10), 0.5, dtype=np.float32)]),
        ([(10, 20, 30), (20, 10, 30)], np.int64, change_out_shape_batch, [dali_types.INT64],
         [dali_types.INT64], [3], [3], setup_change_out_shape, True,
         [np.full((20, 30, 10), 42, dtype=np.int32),
          np.full((10, 30, 20), 42, dtype=np.int32)]),
        ([(10, 20, 30), (20, 10, 30)], np.int64, change_out_shape_sample, [dali_types.INT64],
         [dali_types.INT64], [3], [3], setup_change_out_shape, None,
         [np.full((20, 30, 10), 42, dtype=np.int32),
          np.full((10, 30, 20), 42, dtype=np.int32)]),
    ]

    for shape, dtype, run_fn, out_types, in_types, outs_ndim, ins_ndim, \
            setup_fn, batch_processing, expected_out in args:
        _testimpl_numba_func(
            shape, dtype, run_fn, out_types, in_types, outs_ndim, ins_ndim, \
            setup_fn, batch_processing, expected_out
        )

test_numba_func()

which gives this on almost every run:
$ python test_standalone.py
python: /root/miniconda3/envs/buildenv/conda-bld/llvmdev_1680642098205/work/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp:507: void llvm::RuntimeDyldELF::resolveAArch64Relocation(const llvm::SectionEntry&, uint64_t, uint64_t, uint32_t, int64_t): Assertion `isInt<33>(Result) && "overflow check failed for relocation"' failed.
Aborted (core dumped)

I've broken the Linux-specific variant of this issue into #9001 to avoid spamming everyone here as I post updates whilst I debug. Please watch / subscribe to that if you want to track as I'm working on the issue on Linux AArch64.

I'm finding this surprisingly hard to reproduce on macOS. My environment is:

numba -s output
$ numba -s
System info:
--------------------------------------------------------------------------------
__Time Stamp__
Report started (local time)                   : 2023-06-07 22:38:43.250968
UTC start time                                : 2023-06-07 21:38:43.250976
Running time (s)                              : 0.822634

__Hardware Information__
Machine                                       : arm64
CPU Name                                      : cyclone
CPU Count                                     : 12
Number of accessible CPUs                     : ?
List of accessible CPUs cores                 : ?
CFS Restrictions (CPUs worth of runtime)      : None

CPU Features                                  :

Memory Total (MB)                             : 32768
Free Memory (MB)                              : 1162

__OS Information__
Platform Name                                 : macOS-13.3.1-arm64-arm-64bit
Platform Release                              : 22.4.0
OS Name                                       : Darwin
OS Version                                    : Darwin Kernel Version 22.4.0: Mon Mar  6 20:59:58 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6020
OS Specific Version                           : 13.3.1   arm64
Libc Version                                  : ?

__Python Information__
Python Compiler                               : Clang 14.0.6
Python Implementation                         : CPython
Python Version                                : 3.10.8
Python Locale                                 : en_GB.UTF-8

__Numba Toolchain Versions__
Numba Version                                 : 0.58.0dev0+190.gc9cc06ba1.dirty
llvmlite Version                              : 0.41.0dev0-38-g70f057b

__LLVM Information__
LLVM Version                                  : 14.0.6

__CUDA Information__
CUDA Device Initialized                       : False
CUDA Driver Version                           : ?
CUDA Runtime Version                          : ?
CUDA NVIDIA Bindings Available                : ?
CUDA NVIDIA Bindings In Use                   : ?
CUDA Minor Version Compatibility Available    : ?
CUDA Minor Version Compatibility Needed       : ?
CUDA Minor Version Compatibility In Use       : ?
CUDA Detect Output:
None
CUDA Libraries Test Output:
None

__NumPy Information__
NumPy Version                                 : 1.23.5
NumPy Supported SIMD features                 : ('NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD', 'FPHP', 'ASIMDHP', 'ASIMDDP')
NumPy Supported SIMD dispatch                 : ('ASIMDHP', 'ASIMDDP', 'ASIMDFHM')
NumPy Supported SIMD baseline                 : ('NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD')
NumPy AVX512_SKX support detected             : False

__SVML Information__
SVML State, config.USING_SVML                 : False
SVML Library Loaded                           : False
llvmlite Using SVML Patched LLVM              : True
SVML Operational                              : False

__Threading Layer Information__
TBB Threading Layer Available                 : False
+--> Disabled due to Unknown import problem.
OpenMP Threading Layer Available              : False
+--> Disabled due to Unknown import problem.
Workqueue Threading Layer Available           : True
+-->Workqueue imported successfully.

This issue recently presented itself to me. Any suggestions where i might dig into when trying to contribute a fix? I see we are somewhere between llvm and llvm lite?

This issue recently presented itself to me. Any suggestions where i might dig into when trying to contribute a fix? I see we are somewhere between llvm and llvm lite?

https://github.com/numba/numba/issues/9001#issuecomment-1581424023 describes the issue - the GOT is allocated more than 4GB away from a text section it refers to. If you'd like to start digging in, I'd suggest looking into the RuntimeDyld allocator to devise a strategy that ensures this can't happen. I understand JITLink has a slab allocator already which can help elide this issue, but I didn't yet get chance to look into it further,

@carstenr I had a little more thought about this recently... One of the problems that makes it hard to think about a fix is that reproducing the issue is a giant pain at present - if you're able to do anything to take the existing reproducers and simplify them at all, that would help make it easier for someone (or yourself) to understand the issue and work on a fix.

@carstenr I had a little more thought about this recently... One of the problems that makes it hard to think about a fix is that reproducing the issue is a giant pain at present - if you're able to do anything to take the existing reproducers and simplify them at all, that would help make it easier for someone (or yourself) to understand the issue and work on a fix.

I can give you a script that is able to reproduce it quite often if that can help

I can give you a script that is able to reproduce it quite often if that can help

Yes please!

please write me in<EMAIL_ADDRESS>The script and the file that you will read is quite big

Alright, that means we got two large cases then to reporoduce. We will focus on reducing is as much as possible.

Might take a while to get there as our developers naturally have a strong python background. We will start with a minimal nixtla setup, which is where this popped up for us. And from there on we will work our way down.

FYI by googling I noticed that when porting Julia to ARM they also hit the same bug. Look at https://github.com/JuliaLang/julia/issues/36617  and search in the page for "Assertion failed: (isInt<33>(Addend) && "Invalid page reloc value."),".
Apparently, if this can help at all, the PR that fixed the issue was https://github.com/JuliaLang/julia/pull/43664 ...

FYI by googling I noticed that when porting Julia to ARM they also hit the same bug. Look at JuliaLang/julia#36617 and search in the page for "Assertion failed: (isInt<33>(Addend) && "Invalid page reloc value."),".
Apparently, if this can help at all, the PR that fixed the issue was JuliaLang/julia#43664 ...

The problem is still present

I can't set the ulimit to such large numbers on Mac. it errors with
ulimit: value exceeds hard limit

The largest ulimit I can set is ulimit -s 65520
but it is not crashing for now...

What number did it get to before you stopped it?

I can let it run the whole night if you tell me it can be useful for you.
Il 19 ott 2023, 6:47 PM +0200, Graham Markall @.***>, ha scritto:

What number did it get to before you stopped it?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: @.***>

That would be great if you could give it a go!

@PhilipVinc Is it still running? :-)

@PhilipVinc Thanks - indeed, that was a Numba limitation. I think in #9001 and https://github.com/gmarkall/numba-issue-9001 we're getting close to a really good reproducer now, so there's probably no need for additional testing here - thanks for everything you've looked into so far :-)

LLVM discourse discussion started to discuss a potential fix: https://discourse.llvm.org/t/llvm-rtdyld-aarch64-abi-relocation-restrictions/74616

@Francyrad @PhilipVinc @carstenr It's early work at the moment, but if you're able to build llvmlite from source with the PR https://github.com/numba/llvmlite/pull/1009, and let me know whether you still observe the issue with it (or observe any other issues) that would be good feedback - hopefully this resolves the issue, but there's a lot of testing / review to be done to have confidence in the strategy.

I have experienced this issue repeatedly over the past month, getting errors similar to the following for my ~150 line code for solving a specific PDE:
Assertion failed: (isInt<33>(Addend) && "Invalid page reloc value."), function encodeAddend, file /Users/ci/miniconda3-arm64/conda-bld/llvmdev_1643905487494/work/lib/ExecutionEngine/RuntimeDyld/Targets/RuntimeDyldMachOAArch64.h, line 210.

@gmarkall, I'm not quite sure how to build from source but am happy to try and test it out.

@jacobjivanov Thanks for sharing this info - fortunately you don't need to build from source to test the fix now, as it's part of the llvmlite 0.42 / Numba 0.59 release candidates. You can follow the instructions here to install the Numba and llvmlite release candidates: https://numba.discourse.group/t/ann-numba-0-59-0rc1-and-llvmlite-0-42-0rc1/2329
If you try this, I'd really appreciate if you can let me know whether it appears to have solved the issue for you.

@gmarkall, I can't confirm whether it'll ever fail, but it no longer fails for the particular script that would fail roughly 50% of the time previously. Ran it ~20 times with different initial conditions.

@gmarkall Your work is greatly appreciated! Switching to the release candidate also solved the issue for one of our packages which would occasionally fail.

With llvmlite now at 0.42.0 and the new memory manager merged, can we close this?

I've not heard of any reports of this issue manifesting in llvmlite 0.42, so I think so.

I've not heard of any reports of this issue manifesting in llvmlite 0.42, so I think so.

Alright, let's put a proverbial checkmark behind this issue. We always have the option to re-open in case.
@gmarkall thank you again for the fix for this, it is much appreciated!


Add ASIR workshop series at FedCSIS

https://dblp.org/db/conf/fedcsis/fedcsis2011.html > 1st International Workshop on Advances in Semantic Information Retrieval
https://dblp.org/db/conf/fedcsis/fedcsis2012.html > 2nd International Workshop on Advances in Semantic Information Retrieval
https://dblp.org/db/conf/fedcsis/fedcsis2013.html > 3rd International Workshop on Advances in Semantic Information Retrieval
https://dblp.org/db/conf/fedcsis/fedcsis2014.html > 4th International Workshop on Advances in Semantic Information Retrieval
https://dblp.org/db/conf/fedcsis/fedcsis2015.html > (no workshop sub-headings)
https://dblp.org/db/conf/fedcsis/fedcsis2016.html > 6th International Workshop on Advances in Semantic Information Retrieval
https://dblp.org/db/conf/fedcsis/fedcsis2017.html > 7th International Workshop on Advances in Semantic Information Retrieval
https://dblp.org/db/conf/fedcsis/fedcsis2018.html > 8th International Workshop on Advances in Semantic Information Retrieval
https://dblp.org/db/conf/fedcsis/fedcsis2019.html > (workshop has no entry; perhaps no papers submitted: https://www.fedcsis.org/2019/asir/)
https://dblp.org/db/conf/fedcsis/fedcsis2020.html > (workshop has no entry; perhaps no papers submitted: https://www.fedcsis.org/2020/asir)

this is a good candidate for manual addition

The conference covers many fields, but only the "International Workshop on Advances in Semantic Information Retrieval" is relevant.


mach_msg_trap skips mach_msg caller

https://share.firefox.dev/3scCzqj
I think there's a mach_msg frame missing.

Not sure what changed, but this profile now has the mach_msg frame: https://share.firefox.dev/3B0s6T0


created time is always the same as lastModified

created should be the file creation time and will not be changed, right?
But I found that it is always consistent with lastModified
My silverbullet runs in Docker on Synology NAS
I don't know if I'm the only one having this problem.

What filesystem are you using? If the FS somehow doesn't report file created time, SilverBullet will fall back to last modified if I remember correctly.

I saw this information in the Docker terminal. The file system should be btrfs. The birth time does not seem to be provided. Is there any way I can enable my Docker to provide the birth time?

I have no idea, all I could do is the same as you, which is google.

Ok, I did some testing and Google searches, it seems that the Synology NAS just can't provide the Birth attribute, 🤣 I will add the creation time frontmatter to the new file

Having the same issue running on docker in Unraid, will investigate as well.

Oh thank you very much, I will try again  @terr-steak

I wanted to add, I experienced similar issues. I had my hard drive set to NTFS, after reformatting it to ext4, it worked as expect.


fastcgi_cache_use_stale

I configured a website using "ee site create domain.com --wpfc".
I've looked around at all the configuration files related to this website, yet I can't find where you specify the "fastcgi_cache_use_stale" line so nginx serves a cached version if PHP goes down.
Can you point me where it is exactly?
As mentionned on this tutorial: https://rtcamp.com/wordpress-nginx/tutorials/single-site/fastcgi-cache-with-purging/
Thanks.

Found it.


Add i128 and u128 types

typedef unsigned long u64;
typedef __uint128_t u128;

u64 f(u64 x, u64 y) {
	u128 mul = (u128)x * (u128)y;
	mul += 1UL << 63;
	return mul >> 64;
}

This produces efficient code that cannot be written in Rust without inline assembly.

It would be great to have 128-bit integer types -- for one, this would make bitflags more useful.

An aside: https://github.com/rust-lang/rust/pull/24612 (new float to decimal conversion code) has a custom bignum module, and it has a note that its generic (uN, u2N) based bignum code would easily gain N=64 support if we had u128 support.

+1, The Duration type could be simplified with this: instead of using a u64 for seconds with a seperate u32 for nanoseconds, it could just be a single u128/i128 count of nanoseconds.

I really want this.

We had quadruple precision floating point types at one point (@thestinger added them) but they were later removed from the language, citing lack of support across compilers.

System libm must support variety of non-trivial mathematical operations (sine, cosine, etc) on f128 on all platforms we support in order for the type to be useful.
This does not apply to i/u128 and, as long as LLVM and Rust support i/u128, nothing else needs to support them. This also means that i/u128 are relatively easier to add compared to f128.

I've actually come to the opinion that since llvm supports arbitrarily-sized integers (e.g., i7 for a 7-bit integer), it'd be great if Rust actually just exposed this directly. I doubt it'll ever happen, but I'd love to see it.

That would actually provide a fairly easy and efficient way of parsing and writing bit-packed binary structures (like TCP packets) you could write a packed struct full of your oddly sized integers and transmute a buffer ptr to your struct ptr.

@rrichardson, it's a perfect way of writing tightly-packed structures, yes. It also allows the programmer to more finely tune the acceptable range of a particular data point if they so wish. For example, if you want to be able to store exactly one hexdigit, an i4 is what you want. This allows you to, for example, implement bigints pretty easily by using an array (or vector since llvm has both) of i4s.
I've been hoping for this feature, but again, I seriously doubt I'll ever see this in Rust.

@rrichardson Currently packed structs will only pack to the granularity of 1 byte AFAICT, so adding non-multiple-of-eight sized integers wouldn't help much without more extensive changes.

@Diggsey, just because they only pack to the nearest octet doesn't mean that arbitrarily-sized fixed-width integers wouldn't make things simpler. Think of them as being a more well-supported version of C's bitfields.

@HalosGhost Packing means how fields are layed out relative to each other. Even when you enable packing, fields are still aligned to 1 bytes, which makes arbitrarily sized integer useless.
C++ bitfields allow you to have bit-level packing because it effectively merges adjacent bitfields together into a single field as a pre-process step before the struct layout code sees them. This is also why bitfields are so poorly supported, because only the struct layout step is standardised on each platform, and that step can't deal with less than 1 byte alignments.
It's not impossible to get bit-level packing, but you just have to go quite a bit further than just adding the types to the language: you have to either do what C++ does by having a strategy for merging sub-byte fields into whole-byte fields, or extend the struct layout code to support sub-byte fields directly, making sure that it never affects the layout for structs without those fields.

I must be misunderstanding you.
struct ex {
    uint8_t a: 4;
    uint8_t b: 4;
};

If you were to take sizeof(struct ex); you would discover that it reports 1.

In your example, 'a' and 'b' are merged into a single uint8_t fields as the first step, and then that field is sent to the layout code, and so of course you get a 1 byte struct. Rust doesn't have that merge step at the moment, and the merge step is part of the reason why bitfields are non-portable and poorly supported, because it's entirely implementation defined.

I suppose. Sounds like that should be addressed as well. Even without bit-level packing in structs, the additional boundaries that the arbitrary-width offers is enough for me to want the feature.

Or maybe the answer is not arbitrary width ints but, bit level pattern
matching and destructuring. Erlang might be a good model for this.
On Thu, Apr 30, 2015, 9:12 PM Sam Stuewe<EMAIL_ADDRESS>wrote:

I suppose. Sounds like that should be addressed as well. Even without
bit-level packing in structs, the additional boundaries that the
arbitrary-width offers is enough for me to want the feature.
—
Reply to this email directly or view it on GitHub
https://github.com/rust-lang/rfcs/issues/521#issuecomment-98014771.

@rrichardson: that could be done with a macro or syntax extension. I've been wanting that for a while now :)

Add +1 request for u128 ints :)

Another +1 for {u,i}128.  I could really use it for an implementation of murmur3.  Is there a downside to adding it?

+1
This would often be helpful for implementing various cryptographic algorithms without needing to either jump through hoops to stay within the bounds of u64/i64 or switching to std::num::BigInts.

This cannot be implemented efficiently without compiler support for various reasons. It is clear that certain people have no interest in this feature and that this issue will be ignored by them unless someone turns it into an RFC.

Does LLVM have native support for 128-bit integers even on 32-bit architectures like x86 or ARM?

LLVM exposes arbitrarily-sized fixed-width integers.

@cesarb That's irrelevant since it can always be emulated before the code is translated to LLVM IR.

@HalosGhost I have found that this claim is only true for machine friendly types. In our experience anything larger than 128-bits on x86_64 is extremely poorly supported. Additionally, when dealing with types smaller than 128-bits one is only safe to use (again, x86_64) 8-, 16-, 32-, 64- and 128-bit integers. Other bit-widths can work, but one has to be careful to not form vectors out of them.
We have also have had issues with storing non-standard bit-width integers in memory, but haven't tried to pinpoint them exactly. Seems like this is very poorly tested area of LLVM. Tread carefully.

Does LLVM have native support for 128-bit integers even on 32-bit architectures like x86 or ARM?

Nope. __uint128_t is not defined when compiling for armv7, which I recently learned when compiling code for the iPhone 4S (which is still supported by Apple).

@mjbshaw A fallback is possible (like rem u64).

@mjbshaw LLVM supports it, clang does not.

I need u128 to easily implement support for the mulx BMI2 instruction on architectures that support it (clang does it this way as well), but wouldn't mind falling back to something slower (e.g. an user-defined type) on architectures that do not support it.

i128/u128 is not needed and IMO not very useful for that kind of thing. I much prefer the intrinsics that Microsoft made: https://msdn.microsoft.com/en-us/library/3dayytw9(v=vs.100).aspx.

@briansmith the API only uses 64bit types, but the implementation (which is not shown there) probably won't.

@gnzlbg It's an intrinsic, so it isn't implemented as a function like that. The compiler treats intrinsics special and compiles them down to a certain set of instructions. The MUL instruction on x86 multiplies RAX and some other i64 value and returns the result in RAX and RDX. So it really is returning the result as two separate i64 values, it's not implemented in terms of i128. Same with DIV, it takes the divisor in RAX and RDX, an i64 divisor, and returns the quotient in RAX and the remainder in RDX. At no point is it working with i128 directly, it is simply doing 64-bit math with a second register holding the high part.
I really wish Rust exposed these intrinsics somehow, they're incredibly useful. Instead all we have are intrinsics for doing multiply with overflow which is far less useful since you don't get the high part, just a bool indicating whether you needed a high part.

@retep998 I failed to mention that I am implementing the intrinsic, in Rust. LLVM doesn't offer it because it can recognize it.

For example, this is what GCC generates for both versions on a platform that does support 128bit integers (but not MULX).
mult64to128(unsigned long, unsigned long, unsigned long*, unsigned long*):
        movl    %edi, %r8d
        movl    %esi, %eax
        shrq    $32, %rdi
        movq    %r8, %r9
        shrq    $32, %rsi
        imulq   %rax, %r9
        imulq   %rdi, %rax
        imulq   %rsi, %r8
        movq    %r9, %r10
        movl    %r9d, %r9d
        shrq    $32, %r10
        addq    %r10, %rax
        imulq   %rsi, %rdi
        movl    %eax, %r10d
        shrq    $32, %rax
        addq    %r10, %r8
        movq    %r8, %rsi
        salq    $32, %r8
        shrq    $32, %rsi
        addq    %r9, %r8
        addq    %rdi, %rax
        addq    %rsi, %rax
        movq    %rax, (%rdx)
        movq    %r8, (%rcx)
        ret
umulx(unsigned long, unsigned long, unsigned long*):
        movq    %rdi, %rax
        movq    %rdx, %rcx
        mulq    %rsi
        movq    %rdx, (%rcx)
        ret

mult64to128 is horrifying and I seriously hope nobody uses that. umulx shows that gcc is capable of optimizing down that pattern to a simple 64-bit multiply. However that doesn't change the fact that we could expose such an intrinsic to do umulx without needing 128-bit integers. Every x86_64 processor supports doing that sort of multiplication, so all it would take is a small amount of effort from someone to add a Rust intrinsic for that.
Although... is there an LLVM intrinsic for umulx? MSVC definitely has an intrinsic, so if we look at the Clang/C2 intrinsics for _umul128... oh, it is an inline function that uses 128 bit integers. Does LLVM not have an intrinsic for this? o_0
static __inline__ unsigned __int64 __DEFAULT_FN_ATTRS
_umul128(unsigned __int64 _Multiplier, unsigned __int64 _Multiplicand,
         unsigned __int64 *_HighProduct) {
  unsigned __int128 _FullProduct =
      (unsigned __int128)_Multiplier * (unsigned __int128)_Multiplicand;
  *_HighProduct = _FullProduct >> 64;
  return _FullProduct;
}

The TL;DR is that, in a nutshell, to be able to use mulx and mulq LLVM expects to see some code that uses 128bit integers (there is no llvm.bmi2.mulx intrinsic in LLVM). Even if we expose a umulx function in the standard library, that function is going to have to be written somewhere. If Rust had 128bit integers, it could be written in plain rust (using target_feature, switching between 128bit and 64bit depending on the architecture) and LLVM would do the right thing. Without 128bit integers, we would still need to be generating 128bit LLVM IR code from within rustc itself for LLVM to recognize it.
@retep998

Although... is there an LLVM intrinsic for umulx?

No, there is no need for it since LLVM recognizes the pattern automatically.
And maybe it is worth mentioning that actually, LLVM removes intrinsics over time as it learns to recognize the patterns behind them. For example, all the TBM instruction set used to be llvm.tbm.___ but when LLVM backend learned to recognize the IR for those they were removed.
And so does GCC by the way. I wouldn't be surprised if the MSVC intrinsic was written in plain C++ as well.

LLVM’s “intrinsic” (note the quotes) for widening multiply is
%x = zext i64 %a to i128
%y = zext i64 %b to i128
%r = mul nuw i128 %x, %y
ret i128 %r

and ends up becoming a
movq	%rsi, %rax
mulq	%rdi
retq

It is somewhat important that it is easy for LLVM to prove both arguments have been zero extended in order to generate such code, because otherwise it very quickly may become a
imulq	%rdx, %rsi
movq	%rdx, %rax
mulq	%rdi
addq	%rsi, %rdx
imulq	%rdi, %rcx
addq	%rcx, %rdx
retq

instead, which you, obviously, do not want.
Implementing a rust intrinsic which emits the first sequence of LLVM instructions exactly guarantees the emission of the 3 instruction case, whereas using arbitrary 128-bit sized integers may make LLVM fall back to the other case even obvious cases for no apparent reason.

@nagisa Thanks! I will give that approach a try, it should work on all architectures. The only "non-issue" is that we cannot offer widening multiply as an intrinsic unless we actually have 128-bit integers, but that is irrelevant for mulx_u64 (I'll just write an intrinsic for that instead).

whereas using arbitrary 128-bit sized integers may make LLVM fall back to the other case even in obvious cases for no apparent reason

That's a huge may though, since it would be a regression, clang relies on this, ... Following the same reasoning we should be generating assembly directly, since even if we generate the correct IR the backend could still "fall back to the other case for no apparent reason". So, while you are technically correct (which is the best kind of correct), I don't think we should give this argument that much weight.

Closing, we now have accepted an RFC for this feature: https://github.com/rust-lang/rust/issues/35118


Add missing documentation (examples) for justifyContent and update item box styling

[X] I have followed (at least) the PR section of the contributing guide.

Closes: #29050

@omarmosid Thanks for pointing that out. I think all of the boxes in grey can be converted to blue (similar to grid demos) in this PR! please go ahead with blue boxes.
BTW, Nice work!

Thankyou for the feedback @siriwatknp.
I have now updated the PR and added a few missing examples for css properties that can be used for flexDirection, alignItems, alignContent, alignSelf, display, flexDirection, flexGrow, flexShrink, flexWrap, and Order
I have also taken the liberty to add links to the relevent MDN documentation under each property title. Hope that's going to be okay. It's worth noting here that these links open in the same page since they are your normal markdown links and we may want to have them open in a new tab. So I could change:
For more information please see [flex-direction](https://developer.mozilla.org/en-US/docs/Web/CSS/flex-direction) on MDN.

with
For more information please see
flex-direction
on MDN.

@omarmosid If you don't mind, I would like to reduce the font-size from 19px to 16px.


Add images and charts

Hi.
Congratulations 👏 for this python package
I think extracting charts, images and tables can be also important to convert pdf files to word file.
If you agree with me I can do this to make it more efficient ✌

Hi, that's great
I agree  @mertz1999


Review defaults and ranges

“Does this number matter?... I will just leave it what it is”

Just to document it:

We will keep the impact hour slope and max ih rate ranges
impact hour slope step = 20
max impact hour rate step = 2


fix install bug

fixes #1191
loaders block the entire page until complete, unless they throw locally. loaders were manually calling a connection failure test in order to mimic the layoutloader connection test, but this was incomplete, resulting inaccurate state detection in some situations.
removing those tests resulted in loader use of a client blocking the page, and blocking any thrown error from generating an error page, until the transport timeout. this obscured any early-thrown prax connection failure, and presented the last loader error, which was also misdetected.
so this pr reimplements a more complete temporary solution to fail loaders. ultimately, we should prefer a transport provider and connect-query.
also cleans up the client package a bit, makes the error page ui consistent
some work was also done on the protobuf package, as #1209, it now re-exports service definitions as well as the registry.

relevant #1202

I feel a bit uneasy about the abortLoader() pattern. It feels a bit too hacky of a workaround.

It's hacky, and it's a workaround, but it is equivalent to the current state of things - calling throwIfPraxNotConnected - but it is standardized and complete.

Isn't the main thing we want is simply to redirect if the user is not connected to Prax? Loaders can return redirect responses. Aka, change all of these:
- await throwIfPraxNotConnectedTimeout();
+ await redirectIfPraxNotConnectedTimeout(); // 302 response

which sends it to our connect page. Then the order would not be important and the first to resolve is the first to issue the redirect.

ah, okay. does the redirect certainly not block until all loaders complete? this would also require new routes in minifront as a sort of "onboarding flow". which is essentially a new feature so i'm inclined to call this out of scope, this pr is a minimal solution

Other consumers will need to have a way to cleanly await and definitively know whether the extension is installed or not.

this is surfaced by the page global, but the loaders as written expect to create and use a client without respecting that status.


How to opt out from any collected data? This surviellance is not acceptable.

Read the docs - https://docs.crewai.com/telemetry/Telemetry/#data-collected-includes

I have to agree with @barshag, there needs to be a way to totally disable it.
Perhaps instead of hard wiring AgentOps in a telemetry provider an interface could be swapped in.  A default  DevNullTelemetryProvider should be the default,  By all means provide an optional AgentOpsTelemetryProvider but don't force it.
CrewAI now depends on AgentOps and you can't opt out of AgentOps TOS.   This will cause complications within corporate environments for sure.  https://github.com/AgentOps-AI/agentops/blob/main/TOS.md
Can this be rolled back until a better solution is provided?

Crew Telemetry is disabled by default - you have to opt-in

You do have to include AgentOps now as the crew.py, agents.py, tool_usage.py, now all "import agentops" otherwise you'll get a dependency error.

We are rolling back agentsops integration for now, it is on a RC so it's not out on a proper version only for people manually installing that specific version.
The crew telemetry is actually enabled but anonymous by default, maybe - you have to opt-in to make it non-anonymous. We are also putting out our own TOS.

I think  you can get Crewai, python ans OS versions stats from pypi directly

The crew telemetry is actually enabled but anonymous by default, maybe - you have to opt-in to make it non-anonymous. We are also putting out our own TOS.

**** Why?****

Even Microsoft  allow you to opt-out from annonymous by default (on windows)!
This was meant to be community effort project and now it's all about survelliance to gain benefits from usages.
This is wrong practical of any Open-source project ethic. You can see it in NextJS or any other big project.

If one is willing to give you data fine, but why make it mandatory for all of us that our privacy is so important for us?

But it's still relevnt


Staff: Create new tables to support Staff UI.

Before creating a pull request make sure that:

[ ] commit messages are meaningful and follow good commit message guidelines
[ ] README and other documentation has been updated / added (if needed)
[ ] tests have been updated / new tests has been added (if needed)

Please remove this line and everything above and fill the following sections:
JIRA link (if applicable) ###https://tools.hmcts.net/jira/browse/RDCC-5420
Change description ### Staff: Create new tables to support Staff UI.
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[ ] No

need to merge to dev branch closing this


Create a New Document

When creating a new document, we probably want following options:

Trixels Width
Trixels Height
Background Color
AutoExpand Option
Expand Manually via a window

Instead of an auto-expand we might just want to check if we can keep inifit mode and look for tools like mypaint for inspiration.


Feature Request: Suggest existing tags when adding new URLs

Type

[ ] General question or discussion
[ ] Propose a brand new feature
[x] Request modification of existing behavior or design

What is the problem that your feature request solves
When adding new URLs, the input field for tags is a plain text field. I sometimes end up writing a tag name slightly different to one that I have used previously, thereby creating two tags for the same meaning.
Describe the ideal specific solution you'd want, and whether it fits into any broader scope of changes
Ideally, the text field would be replaced with a combo box that suggests/autocompletes existing tags, while still allows creating new ones.
Examples: https://github.com/yairEO/tagify or https://mui.com/material-ui/react-select/#chip
What hacks or alternative solutions have you tried to solve the problem?
Keep a second tab open with the list of tags. For every tags I'm not sure about, CTRL+F the list.
How badly do you want this new feature?

[ ] It's an urgent deal-breaker, I can't live without it
[ ] It's important to add it in the near-mid term future
[x] It would be nice to have eventually

[ ] I'm willing to contribute dev time / money to fix this issue
[x] I like ArchiveBox so far / would recommend it to a friend
[ ] I've had a lot of difficulty getting ArchiveBox set up

I think we can do this with the same built-in autocomplete django field that it uses in the Admin UI


Add links to other CHANGELOGs

Added links to CHANGELOGS in other repos where applicable.
[skip ci]

@miq-bot add_label documentation


file: cleanup tmpDir

I got tired of my /tmp filling up.  So lets not fill it up.

oooer, thanks. lgtm


The passbook button works on Firefox and Edge (Windows 10) but is disabled

After removing disabled=true from the submit button using dev tools, the pkpass file was still created and I was able to load it onto my phone successfully.
I would recommend not arbitrarily restricting browsers since these passes work on several apps for android as well, and not everyone uses or wants to use chrome. I was able to find this workaround and still get my pass on my android phone (no chrome installed), but if possible I would suggest the button not be disabled unless there is a functional reason not to enable it.

We've loosened up the rules around this to give people more chances to save this - I believe this resolves the issue noted above.


Implement Cloud Map client-side traffic throttling

Instead of hitting server side throttles and subsequent failures, implement client-side traffic shaping to slow down request submission. Example from app-mesh-controller.

Related issue: https://github.com/aws/aws-cloud-map-mcs-controller-for-k8s/issues/128


Table throws an error when searching in the search bar.

How do you replicate this?

  <TableHeader dataField='id' searchable={true} isKey={true}>Product ID</TableHeader>
  <TableHeader dataField='name' searchable={true}>Product Name</TableHeader>
  <TableHeader dataField='price' searchable={true}>Product Price</TableHeader>
</Table>```

Then just type in the search bar and it should throw the error.

I wonder if this is only in styleguidist.  Table search works in GC for me

I think this has been fixed sometime between when this issue was filed and now.  I can't reproduce.

Yea your right @stuller

Yeah sorry, I had it figured it out, and had it fixed sometime later forgetting to close this.  It was an issue with a mistyped code for that component that we imported over from react-bootstrap-table


Unable to build postgress plugin

Hello Aries Team!
Trying to compile the Postgres plugin by using this documentation  have such error
error[E0512]: cannot transmute between types of different sizes, or dependently-sized types
   --> /Users/yuriyyakovenko/.cargo/registry/src/github.com-1ecc6299db9ec823/socket2-0.3.11/src/sockaddr.rs:156:9
    |
156 |         mem::transmute::<SocketAddrV4, sockaddr_in>(v4);
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: source type: `SocketAddrV4` (48 bits)
    = note: target type: `sockaddr_in` (128 bits)

For more information about this error, try `rustc --explain E0512`.

MacOS
m1 processor
Sonoma 14
cargo 1.68.2 (6feb7c9cf 2023-03-26)
Could you please help, Postgres is a main our storage tool
Regards

@djleonskennedy I think you have to build the plugin using the cargo version  1.63.0 as there was an error while building with new versions and also same version is used in the GitHub actions as well here


Clarification - lifetime of characteristic callbacks

I'm trying to understand the difference in the way callbacks are set and cleaned up by the library.
m_pbleServer = NimBLEDevice::createServer();
m_pbleServer->setCallbacks(new BleServerCallbacks(), true);
m_pbleService = pServer->createService(g_svcid);
NimBLECharacteristic *pSessionCharacteristic = g_sessionService->createCharacteristic(
    g_sescharid,
    //          NIMBLE_PROPERTY::READ |
    NIMBLE_PROPERTY::WRITE |
    NIMBLE_PROPERTY::NOTIFY);
m_pbleCharacteristicCallbacks=new BleCharacteristicCallbacks(&m_cmd);
pSessionCharacteristic->setCallbacks(m_pbleCharacteristicCallbacks);

See the two setCallback() calls?
One appears to allow me to create a new instance of a callback object, which the library will delete on destruction.
The other one appears to require me to handle cleanup myself. There is no parameter to delete the callback object.
I don't understand the discrepancy, and it wouldn't concern me except that now I'm not sure if I'm handling this correctly.
The code above runs - not in a constructor, but in a method that gets called exactly once and almost immediately after construction.
In the destructor of this class, I am calling delete on m_pbleCharacteristicCallbacks if it's non-null.
Would you say that seems like the right thing to do? I'm not exactly clear on where I should be deleting the characteristic callbacks. When is it safe to do so, exactly?

Would you say that seems like the right thing to do? I'm not exactly clear on where I should be deleting the characteristic callbacks. When is it safe to do so, exactly?

You have this correct. You can delete the callback function any time after the characteristic instance has been deleted or after calling it's setCallbacks method with a nullptr.
I understand the confusion and the addition of a cleanup parameter should be added to the characteristic callback setting. This was not considered when added due to the nature of server characteristics not generally changing once setup.

Closing as stale, please reopen if needed.


[typespec-python][bug] Strange method signature

After generating with this tsp spec, begin_create_dev_box gets generated like this:
https://github.com/Azure/azure-sdk-for-python/blob/ad7bc0dcca0c98f987f828b03a527ce49edbdb9f/sdk/devcenter/azure-developer-devcenter/azure/developer/devcenter/operations/_operations.py#L2757C4-L2759C26
The typing for body is showing up as Union[JSON, JSON, IO], I would expect Union[JSON, IO]

cc @iscai-msft, @msyyc


Total number of annotations cannot be higher than the file duration in seconds

Hi,
It seems that the maximum number of annotations that can be written in a EDF+ file with this library is equal to the duration in seconds of the signals. Why?
I have done several tests and for files with 10s, 100s, 1000s or 10000s long records it's not possible to write more than 10, 100, 1000 or 10000 annotations respectively. There is no error raised by the f.writeAnnotation method, I just found it out after opening the newly created file where only the first N=records_duration annotations where present. I don't know if this is a limit of the EDF format but if so, it should be documented and the f.writeAnnotation method should return an error code if a limit is reached.
Best,
Jose

I've encountered this too. It's not strictly 1 per second. The limit is 1 annotation per "record", and the default record size is 1 second. You can change the record size, and thus increase the number of annotations per second, but I've found this to be very buggy, and increasing the total number of records really slows down file loading.
Alternatively, if you're using EDF+ or BDF+, you can set_number_of_annotation_signals to 64. This will allow for up to 64 annotations per record.
Note that a data record may not exceed 61440 bytes, including the annotation signal(s).
64 annot /s * 114 bytes / annot = 7296 bytes, leaving 54144 bytes of data per record.

I agree that it would be nice if there was a warning to tell you that your annotations will be truncated. Raising an error is a bad idea because the problem won't be known until file close. e.g., If your first 1 second of data has 100 annotations, but your next 99 seconds of data have 0 annotations, then you're still OK (I think?). But if you only have 98 more seconds, then your last annotation will be lost.
Maybe an error can be raised after file close.


Zero downtime upgrade (services with longer start)

When using service that needs some time to start up it seems to be impossible to upgrade (or even just scale up) with zero downtime. Reasons:

Rancher seems to add new container to HAProxy immediately when its ENTRYPOINT/CMD is executed (even when exposed ports are not opened yet).
Even though health checks are set up some requests fail since new servers in HAProxy are by default considered "up" until the health checks fail and mark server "down".

The above reasons combined cause that there's actually no option to have a zero-downtime upgrade/scale-up since requests get routed to a service that is not ready for some time (until health checks fail). Is there any way to get around this?
(I guess in many cases this problem is just not really visible since many services start in shorter time than Rancher needs to update HAProxy, but it is a general problem that can occur on any upgrade/scale-up from time to time.)

With the release of Rancher 2.0, development on v1.6 is only limited to critical bug fixes and security patches.


[Add] Glow button

Details
I want to add a neon glow button. This button will glow on clicking.
Screenshots

Type of Contribution

[ ] Update to an existing Animated Button
[X] Adding a new Button
[ ] Resolving a bug
[ ] Proposal to the Repository
[ ] Changes related to documentation or README.md
[ ] Other Changes

Checklist

[X] I have checked the existing issues
[X] I have read the Contributing Guidelines
[X] I am willing to work on this issue
[X] I am a GSSoC'23 contributor

Please check #1346

Sorry for the inconvinience, but we are closing all the issues! Please open a new issue if you want to contribute anything. Also now you can create a new button from the website itself! You will get points for those too! I will add the instructions on how to create a new button soon and create an empty pr for the same to get points!


beginning overhaul of random processes

Overhauling random processes.
All random processes (Markov chains, Poisson processes, Gaussian processes) have a corresponding:

result object that stores one realization
ProbabilitySpace
RandomProcess

Arrival times, interarrival times, and states are now functions defined on the probability space.

I've played around with some of the new random process features, but I haven't really looked at most of the code.  I really like the separate ProbabilitySpace and RandomProcess definitions.  And lots of the features seem to be working properly, e.g. returning the correct distributions.  Here are some initial thoughts.
Suggested Fixes

Need to be able to plot sample paths of discrete time MC (I think you said you were working on this)
For Brownian motion, Var(X[t]) = scale * t, but I think it is more standard to define scale so that Var(X[t]) = scale**2 * t
Can we start the arrival time sequence at 0?  I think it's standard to define the 0th arrival_time as 0, so that W[1] is the time of the actual first arrival.
Likewise, if we want the time of the first arrival to be equal to the first interarrival time, W[1] = T[1], then we need the interarrival_time sequence to start with 0 also.

P = MarkovChainProbabilitySpace(...)
X = RandomProcess(P)
X.draw()

returns ContinuousTimeFunction rather than a state sequence (and X.sim(1) returns blank Result).

You had one example that was X = RandomProcess(GaussianProcess) which didn't always work (e.g. (X[1] & X[2]).sim(100) returned an error).  We should maybe include warning messages that it's either X = RandomProcess(GaussianProcessProbabilitySpace) or X=GaussianProcess but not a mix of the two.

Comments/questions

We can call RV like a function to find its value for a particular probability space outcome.  Can we do something similar for random process objects?  For example, save one draw from a PoissonProcessProbabilitySpace as omega and then access the sample path, arrival times, and interarrival times for this particular outcome.
When defining a RandomProcess on a probability space, should there be available an explicit "identity" function, maybe sample_path, so that X = RandomProcess(P) is equivalent to X = RandomProcess(P, sample_path)?
Is it possible to define independent probability spaces for any of the new objects?  For example, I can add independent Poisson Processes by defining two independent PoissonProcesses and using AssumeIndependent.  But would it be possible to define a probability space supporting two independent Poisson processes, P = PoissonProcessProbabilitySpace(...)*PoissonProcessProbabilitySpace(...), with a result object equal to a product of two result objects, and then define processes on this space like N = RandomProcess(P[0])?  I assume AssumeIndependent is doing something like this under the hood, but when I try to define independent PPPS's directly I get an error.
arrival_times works nicely when first defining a probability space.  But how to get arrival times for a general discrete state continuous time random process?  Is something like the following possible?

N = PoissonProcess(...)
W = N.apply(arrival_times) # or W = arrival_times(N)

For example, if I add two independent Poisson processes using AssumeIndependent as mentioned above I might want to find the arrival times of the resulting process.

In general, how do we apply functionals to a random process?  Like defining stopping times.  I think I need to understand better what the various results objects are and what they can do.
It would be helpful to define a degenerate RandomProcess $X(t) = t$, called Time?, which can be used with any probability space.  For example, X = N - Time.  (I think you did something similar for constants as RV.)
Is there a states function for general continuous time discrete state processes, to return the sequence of states, S = RV(P, states).

I'll continue reviewing and post some notebooks later.  Thanks!

@kevindavisross
I haven't had a chance to address the RandomProcess issues you raised above, but that's next on my list.
What I've done is overhaul the underlying data structures for vectors, infinite vectors, discrete-time functions, etc. I think we should be back to the old functionality based on my tests, but I'm sure I missed a few things. Could you run through your notebooks and tutorial and let me know if there are any bugs I should fix?

Fixed all of the bugs that you reported, except for the one about "Simulating paths of a process along with things like arrival times" (this is on my TODO list).

The last few commits essentially remove RandomProcess, as we discussed offline. RandomProcess still exists in barebones form, mostly for defining processes like random walks and AR processes. For this use case, it is more common to specify the probability space and the index set, but not the function. So I changed the API back to RandomProcess(probSpace, index_set, function).
Right now, random walks and AR processes are very slow. It takes a long time to evaluate X[n] for n > 20. This is a known issue, and I'm working on fixing this. Let me know if you spot any other bugs.

I'll email you some notebooks with examples.  I ran some of the tests of binary operations you suggested.  Most seem to work the way they should.  A few questions

How do we want Vector (or InfiniteVector) + ContinuousTimeFunction to behave (both for RV and result)?  At first I thought this wouldn't make sense.  But do we want (2, 3) + X(t) to return (2 + X(t), 3 + X(t))?  Same for DiscreteTimeFunction.
What kind of Python object is a Symbulate result vector?  Vector + [blank] works when Vector is a result vector, e.g. x + RV(Normal(0, 1)) returns RV-Vector for x = Poisson(2)**2.draw().  But how to replace x with constants, e.g. (2, 3) + RV(Normal(0, 1))?  I tried numpy array but that's not it.
I'm assuming we only want to allow operations on Vectors with same length, so Vector + Infinite-Vector should not work, and it does not (i.e. no recycling).

Other comments/questions

Can you give me a few examples of DiscreteTimeFunction?  I'm not sure how this is intended to work.
We now have ContinuousTimeFunction result.  Are GaussianProcessResult and PoissonProcessResult special cases of CTF result?  Will each special process have its own result type, or should there just be general CTF result?
What are the intended "rules" for unpacking?  Generally, X = RV(P * Q) gets unpacked like X[0] = RV(P), X[1] = RV(Q).  But not if P and Q represent Scalar and Vector, in which case it gets unpacked like one big vector.  For example,

X = RV(Poisson(1) * (Normal(0, 1) ** 2))
X[1]  # is RV(Normal(0, 1))
X[2]  # is RV(Normal(0, 1))

but
X = RV(PoissonProcessProbabilitySpace(1) * (Normal(0, 1) ** 2))
X[1]  # is RV(Normal(0, 1) ** 2)

How do we want Vector (or InfiniteVector) + ContinuousTimeFunction to behave (both for RV and result)? At first I thought this wouldn't make sense. But do we want (2, 3) + X(t) to return (2 + X(t), 3 + X(t))? Same for DiscreteTimeFunction.

I agree that that is one possible interpretation of (2, 3) + X(t), but I'm not sure it makes sense. What type of object would (2 + X(t), 3 + X(t)) be? It doesn't seem to behave like a random vector or a random process. For example, how would you plot it?

What kind of Python object is a Symbulate result vector? Vector + [blank] works when Vector is a result vector, e.g. x + RV(Normal(0, 1)) returns RV-Vector for x = Poisson(2)**2.draw(). But how to replace x with constants, e.g. (2, 3) + RV(Normal(0, 1))? I tried numpy array but that's not it.

It's a Vector object. So you could do Vector((2, 3)) + RV(Normal(0, 1)).
I can actually make (2, 3) + RV(Normal(0, 1)) work. But there's no way to make (2, 3) + (0, 1) work, so I'm not sure if enabling that notation would ultimately be more confusing. Let me know what you think.

I'm assuming we only want to allow operations on Vectors with same length, so Vector + Infinite-Vector should not work, and it does not (i.e. no recycling).

Yes, that was the idea.

Can you give me a few examples of DiscreteTimeFunction? I'm not sure how this is intended to work.

You specify the function X[n]. So lambda n: n at a sampling rate of fs=2 would actually be the sampled version of the function f(t) = 2t.

We now have ContinuousTimeFunction result. Are GaussianProcessResult and PoissonProcessResult special cases of CTF result? Will each special process have its own result type, or should there just be general CTF result?

Yes, they are both special cases (i.e., subclasses) of CTF. The reason for giving them their own class is that they store additional metadata. But they all share, for example, the same .plot() function.
By the way, if x is a GaussianProcessResult, you can check that it is also a CTF using isinstance(x, ContinuousTimeFunction).

What are the intended "rules" for unpacking? Generally, X = RV(P * Q) gets unpacked like X[0] = RV(P), X[1] = RV(Q). But not if P and Q represent Scalar and Vector, in which case it gets unpacked like one big vector.

Right now, the rule is that scalars and vectors get concatenated into one big vector. Is that the behavior we want? Maybe we want * to return Tuples, but ** to return Vectors? So RV(Normal() * Normal() ** 2) would return a tuple of two objects, the second of which is a vector of length 2? What do you think?

How do we want Vector (or InfiniteVector) + ContinuousTimeFunction to behave (both for RV and result)? At first I thought this wouldn't make sense. But do we want (2, 3) + X(t) to return (2 + X(t), 3 + X(t))? Same for DiscreteTimeFunction.

I agree that that is one possible interpretation of (2, 3) + X(t), but I'm not sure it makes sense. What type of object would (2 + X(t), 3 + X(t)) be? It doesn't seem to behave like a random vector or a random process. For example, how would you plot it?
Yes, it probably doesn't make sense.

What kind of Python object is a Symbulate result vector? Vector + [blank] works when Vector is a result vector, e.g. x + RV(Normal(0, 1)) returns RV-Vector for x = Poisson(2)**2.draw(). But how to replace x with constants, e.g. (2, 3) + RV(Normal(0, 1))? I tried numpy array but that's not it.

It's a Vector object. So you could do Vector((2, 3)) + RV(Normal(0, 1)).
I can actually make (2, 3) + RV(Normal(0, 1)) work. But there's no way to make (2, 3) + (0, 1) work, so I'm not sure if enabling that notation would ultimately be more confusing. Let me know what you think.
Thanks.  No need to change notation; Vector is fine.

I'm assuming we only want to allow operations on Vectors with same length, so Vector + Infinite-Vector should not work, and it does not (i.e. no recycling).

Yes, that was the idea.

Can you give me a few examples of DiscreteTimeFunction? I'm not sure how this is intended to work.

You specify the function X[n]. So lambda n: n at a sampling rate of fs=2 would actually be the sampled version of the function f(t) = 2t.
I haven't tested this much out yet, but I will.

We now have ContinuousTimeFunction result. Are GaussianProcessResult and PoissonProcessResult special cases of CTF result? Will each special process have its own result type, or should there just be general CTF result?

Yes, they are both special cases (i.e., subclasses) of CTF. The reason for giving them their own class is that they store additional metadata. But they all share, for example, the same .plot() function.
By the way, if x is a GaussianProcessResult, you can check that it is also a CTF using isinstance(x, ContinuousTimeFunction).
I know you said implementing ContinuousTimeFunction was a pain, but I really like it.  Thanks for making it work!

What are the intended "rules" for unpacking? Generally, X = RV(P * Q) gets unpacked like X[0] = RV(P), X[1] = RV(Q). But not if P and Q represent Scalar and Vector, in which case it gets unpacked like one big vector.

Right now, the rule is that scalars and vectors get concatenated into one big vector. Is that the behavior we want? Maybe we want * to return Tuples, but ** to return Vectors? So RV(Normal() * Normal() ** 2) would return a tuple of two objects, the second of which is a vector of length 2? What do you think?
Yes, I think that * should return Tuples.  I think that would enhance consistency with the other objects.
A few more comments/issues; see the attached notebook for more details
Random process overhaul - bugs.pdf

BrownianMotion: change to (drift=0, scale=1) (You already made the change for ProbSpace)
For histograms, I'm getting " 'normed' kwarg is deprecated" warning, so it would be great if we could merge in Kien's changes which fixed this.
There are some examples of vector-valued processes where unpacking and time-indexing can get confusing.  Say I want X to be a 2-d Brownian motion, and I want to generate 100 pairs at time 1.  But X[1].sim(100) generates 100 sample paths of the first coordinate

Is there a way to distinguish between time index and coordinate index?  Maybe not possible or desirable.  Is there a way to indicate all components of a vector, like X[:], so X[:][1.2] returns vector at time 1.2? Or do we just need to unpack carefully?

Should we allow transformations of processes with common functions using g(X) syntax?
Negative indexing for vectors, e.g. X[-2], - starting with the end.  Do we want to allow?
Two examples giving errors that I can't figure out why

P = BrownianMotionProbabilitySpace(drift=2)
Q = BrownianMotionProbabilitySpace(drift=-2)
X = RV(P * Q)
S = RV(P * Q, sum)
(X & S).sim(10)
# "same prob space error"

P = Uniform(-pi, pi)
t = ContinuousTimeFunction()
Z = RV(P)
X = cos(2 * pi * t + Z)
X.sim(10).plot()
# TypeError: must be real number, not ContinuousTimeFunction

X = RV((BrownianMotionProbabilitySpace(0, 1) ** 3) * (PoissonProcessProbabilitySpace(3) ** 3)) is also unpacking like a vector of length 6 instead of two tuples of length 3 each
I think we would want to add a Vector to a vector-valued process.  But it's not working properly

B = RV(BrownianMotionProbabilitySpace() ** 3)
y = Vector((10, 100, 1000))
X = y + B

Note: I don't think vector-valued process functionality is essential to this version of Symbulate.  But so much does work already, that I'm trying to see how far we can push it.

I haven't been able to figure out how to define a compound PoissonProcess, i.e. a PoissonProcess where the jump sizes are realizations of an InfiniteVector likeRV(Normal() ** inf).  Any ideas?
I didn't mention previously, but you convinced me on 0-indexing for arrival and interarrival times for jump processes.

Here's an example that involves several issues that I've mentioned above, like unpacking and 0-indexing.
I don't know how to define a compound Poisson process, so I tried to define a compound Poisson RV
P = (Normal(0, 1) ** inf) * Poisson(0.1)
X, N = RV(P)
Y = X.apply(cumsum)
Y[N].sim(1000)

This almost works.  The problem is that I want Y=0 when N=0, but here Y = X[0] if N=0 because of the 0-indexing.  I could solve the problem by appending 0 as the first entry of X.  I tried to do this with
P = BoxModel([0], size=1) * (Normal(0, 1) ** inf)
X = RV(P)
N = RV(Poisson(0.1))
X, N = AssumeIndependent(X, N)

but X unpacks into two tuples --- like we've said we want it to do --- instead of a single vector (so I can't use cumsum).
So

Is there a way to append vectors into a single vector?
I think we should have a general protocol for unpacking, but should we allow this to be overridden?  Like with vectorize or tupleize functions?
I have been trying to use AssumeIndependent as little as possible, instead trying to reflect the independence in the probability space.  I generally have been able to do this, though in some situations the AssumeIndependent syntax is easier.  But even if my BoxModel fix above worked, I wouldn't know how to do this example without AssumeIndependent or being able to override the unpacking (I would want BoxModel to go with Normal, but Poisson to be on its own).

Yes, I think that * should return Tuples. I think that would enhance consistency with the other objects.

Done. But just FYI, there are downsides to this approach. Now, (Normal() ** 3).draw() is a Vector, while (Normal() * Normal() * Normal()).draw() is a Tuple. This means that some operations, like .sum() will not be available for the latter. This is unavoidable because * is a binary operation. If we want both Normal() * Normal() and Normal() * Normal() * Normal() to return Vectors, then we'd have to allow multiple Vectors to collapse into one big Vector, as the code was before.
Let me know if you have any thoughts about which approach is better.

BrownianMotion: change to (drift=0, scale=1) (You already made the change for ProbSpace)

Done.

For histograms, I'm getting " 'normed' kwarg is deprecated" warning, so it would be great if we could merge in Kien's changes which fixed this.

I'll do this after merging this pull request.

Is there a way to distinguish between time index and coordinate index? Maybe not possible or desirable. Is there a way to indicate all components of a vector, like X[:], so X[:][1.2] returns vector at time 1.2? Or do we just need to unpack carefully?

This is probably not worth doing until we implement vector-valued random processes.

Should we allow transformations of processes with common functions using g(X) syntax?

This works already, except there was a bug, which I fixed. (See the discussion of the cosine example below.)

Negative indexing for vectors, e.g. X[-2], - starting with the end. Do we want to allow?

This should work already. Are you saying we should disallow it?

Two examples giving errors that I can't figure out why
P = BrownianMotionProbabilitySpace(drift=2)
Q = BrownianMotionProbabilitySpace(drift=-2)
X = RV(P * Q)
S = RV(P * Q, sum)
(X & S).sim(10)
# "same prob space error"

That's because you defined P * Q twice. That's two instances of the same probability space, not the same probability space. Just store P * Q in a variable, and make it common to both X and S.

P = Uniform(-pi, pi)
t = ContinuousTimeFunction()
Z = RV(P)
X = cos(2 * pi * t + Z)
X.sim(10).plot()
TypeError: must be real number, not ContinuousTimeFunction

This was fixed by this commit.
In case you're curious, the bug was this: Functions like cos() simply apply math.cos() to the RV or TimeFunction. The problem here was that cos(2 * pi * t + Z) is two-levels deep: it is an RV that returns a TimeFunction. We can't just apply math.cos() to the RV because you can't call math.cos() on a TimeFunction. I've fixed this problem by making cos() (and other functions) recursive. Now, it will keep applying cos() until it hits a scalar, at which point it calls math.cos().

I'm looking at the new unpacking now.  A few quick comments

isinstance(x, Tuple) returns: 'Tuple' not defined, but isinstance(x, tuple) works.  What's the difference between tuple and Tuple?
I see what you say about the downside to * unpacking as tuple, but the following still works

P = Poisson(1) * Poisson(2) * Poisson(3)
X = RV(P, sum)

and this also works
(Poisson(1) * Poisson(2) * Poisson(3)).sim(10).apply(sum)

Is that what you expected?  Seems great to me.

In the compound Poisson example, I tried to append 0 with 0 & X, but that doesn't work.  But maybe it could?  Joining two RV X & Y produces a tuple; maybe we could allow constants also?

isinstance(x, Tuple) returns: 'Tuple' not defined, but isinstance(x, tuple) works. What's the difference between tuple and Tuple?

Tuple is a subclass of tuple that is defined in Symbulate. The reason for having two classes is to distinguish between probability spaces that return a tuple and collections of outcomes. For example, if you define P to be the DeckOfCards probability space, then P.draw() returns a tuple outcome, like (3, 'hearts'). If you do P ** 2, you want the outcome to be like Tuple([(3, 'hearts'), (2, 'diamonds')]). If we didn't distinguish between tuple and Tuple, then the two tuples would collapse into a single tuple, i.e., (3, 'hearts', 2, 'diamonds'). It's probably too confusing to call one tuple and the other Tuple. If you have another idea for a name for a list of outcomes, let me know.
The reason isinstance(x, Tuple) doesn't work is because Tuple is not exported by the package. It's easy to export it, but does a user really have to define their own Tuple?

I see what you say about the downside to * unpacking as tuple, but the following still works...

Yes, sum will still work, but many other functions will not. For example, compare the following:
P1 = Poisson(1) * Poisson(1) * Poisson(1)
X = RV(P1, cumsum)
X.draw()

P2 = Poisson(1) ** 3
Y = RV(P2, cumsum)
Y.draw()

In the compound Poisson example, I tried to append 0 with 0 & X, but that doesn't work. But maybe it could? Joining two RV X & Y produces a tuple; maybe we could allow constants also?

I agree that we should support the notation 0 & X. But that won't solve your problem where X returns an InfiniteVector, right? For that, I'm thinking about creating a concat(...) function that takes in any number of scalars and vectors (and possibly an InfiniteVector at the end) and concatenates them into a single Vector or InfiniteVector. What do you think of this interface?

I have done a fairly comprehensive test of the new unpacking protocol.  I think it works well; there is a clear, consistent logic to it.
I see about tuple vs Tuple.  I don't know of a better name yet.  I don't think we have to export; I had just been wondering why I could call say isinstance(x, InfiniteVector) but not isinstance(x, Tuple).
Do we need to worry about differences between when something returns type int versus numpy.int32? Potential addition versioning issues?
I like your concat idea.
I agree that vector-valued processes should be deferred, but I'm impressed but how much capability is already there.

I have done a fairly comprehensive test of the new unpacking protocol. I think it works well; there is a clear, consistent logic to it.

Great!

I see about tuple vs Tuple. I don't know of a better name yet. I don't think we have to export; I had just been wondering why I could call say isinstance(x, InfiniteVector) but not isinstance(x, Tuple).

Okay. Let me know if you have a better name. I really want to distinguish it from tuple.

Do we need to worry about differences between when something returns type int versus numpy.int32? Potential addition versioning issues?

Thanks for pointing this out. We do indeed need to worry about that. In this commit, I have systematically replaced all references to isinstance(x, int) to isinstance(x, numbers.Integral), which includes Numpy types.

I like your concat idea.

Done. See attached notebook. Let me know if you notice any bugs or have any suggestions.
Concat and Joint Distributions.pdf

Joining two RV X & Y produces a tuple; maybe we could allow constants also?

Also done. See the attached notebook.

I retested the binary operations after your recent changes, and added a few more tests.  Almost everything that should work does, but I still need to do more tests with DiscreteTimeFunction.
The only exception: The following does NOT work because B.draw() returns tuple and Y.draw() returns Vector.

Y, B = RV(Normal() ** 3 * BrownianMotionProbabilitySpace() ** 3)
X = Y + B
type(X.draw())

But replace ** 3 with ** inf in the above and it DOES work and returns InfiniteVector.  Should something like BrownianMotionProbabilitySpace() ** 3 return Vector (of continuous time functions) rather than tuple?  That would be consistent with the general protocol that ** returns Vector.  This might fall under "vector-valued processes should be deferred", but there does seem to be an inconsistency in the current version.

To replace "Tuple", how about "ProductSpaceOutcome" or "ProductSpaceResult"?  (I realize Normal() ** inf is also a product space but returns Vector, so ProductSpace is not the greatest name.  But I'm having trouble thinking of something better.)
I imagine it's not possible to apply concat in RV space, or you would have implemented it already.  But I think the compound Poisson example would be nicer as

P = Poisson(1) * (Normal() ** inf)
N, Z = RV(P)
X = concat(0, Z).apply(cumsum)
X[N].sim(1000).plot()

This is getting into specific examples now, but another way around the above is if cumsum had an "initialize" option, like y = x.cumsum(initialize=3) would have y[0]=3, y[1]=3+x[0], etc?  Then compound Poisson would be

P = Poisson(1) * (Normal() ** inf)
N, Z = RV(P)
X = Z.apply(cumsum(initialize=0))
X[N].sim(1000).plot()

In general I think the consistency of the new unpacking protocol outweighs any convenience the old protocol might offer for specific examples.  The examples that I have thought of so far when I want to "override" the unpacking with concat all involve situations like the compound Poisson example where I want to basically supplant the 0-indexing and start with x[0]=0
I imagine there is a good reason, but why can't you pass a tuple into concat, like concat((2, (2, 3), 3)) instead of concat(2, (2, 3), 3)?

I can't figure out how to define a process through time rescaling, like $X(t) = W(exp(t))$.  Any ideas?
Would it be possible to add Multinomial distributions?  I might use in 305.
Re: the int issue, is there a similar issue with float?
I don't think this is necessary now, but just a thought: ** takes priority in order of operations for unpacking, so X = RV(BoxModel([0, 1, 2], size=2, replace=False) ** inf) unpacks with X[0] a Vector of size 2.  But one might want to unpack it like X, Y where X is an InfiniteVector with the results of all the first draws.  Since we have concat, would it also make sense to have some kind of "transpose" or "reshape" functions?
(BivariateNormal() ** 3).draw() returns AttributeError: 'BivariateNormal' object has no attribute 'sim_fn'  Same for ** inf.  Seems like this should work like BoxModel example above.

Should something like BrownianMotionProbabilitySpace() ** 3 return Vector (of continuous time functions) rather than tuple? That would be consistent with the general protocol that ** returns Vector. This might fall under "vector-valued processes should be deferred", but there does seem to be an inconsistency in the current version.

Okay, I've made it so that ** returns a Vector, always. This means that I have to allow Vector to store arbitrary objects (whereas I originally conceived of Vector as only storing numbers). But maybe this is for the better. You can now also do things like
a = Vector([Vector([0, 1]), Vector([2, 3])])
a + a

Your Brownian motion example now works; I'm just worried that I might have broken something else in the meantime. Could you please check to make sure that this change doesn't break something else?

To replace "Tuple", how about "ProductSpaceOutcome" or "ProductSpaceResult"? (I realize Normal() ** inf is also a product space but returns Vector, so ProductSpace is not the greatest name. But I'm having trouble thinking of something better.)

This name is on the right track, but a little too verbose for my tastes.

I imagine it's not possible to apply concat in RV space, or you would have implemented it already.

It actually shouldn't be too bad; I just didn't think of it as a use case. Let me think about the best way to implement this.

This is getting into specific examples now, but another way around the above is if cumsum had an "initialize" option, like y = x.cumsum(initialize=3) would have y[0]=3, y[1]=3+x[0], etc?

I'm not sure I like this. The usual behavior of cumsum is for it to start with the first element of the sequence being summed. What would the value of "initialize" be in this case?

I imagine there is a good reason, but why can't you pass a tuple into concat, like concat((2, (2, 3), 3)) instead of concat(2, (2, 3), 3)?

Yes, I should fix this. I should also allow people to pass multiple arguments into Vector.

I can't figure out how to define a process through time rescaling, like $X(t) = W(exp(t))$. Any ideas?

I can't think of any ways to do this right now either.

Would it be possible to add Multinomial distributions? I might use in 305.

I think this is a good idea, but I don't have the bandwidth to do this. Do you think you can take this on? (It's straightforward, but requires some time.)

Re: the int issue, is there a similar issue with float?

I tried to also fix all the float issues when I fixed the int issues. Let me know if you run into any problems.

I don't think this is necessary now, but just a thought: ** takes priority in order of operations for unpacking, so X = RV(BoxModel([0, 1, 2], size=2, replace=False) ** inf) unpacks with X[0] a Vector of size 2. But one might want to unpack it like X, Y where X is an InfiniteVector with the results of all the first draws. Since we have concat, would it also make sense to have some kind of "transpose" or "reshape" functions?

I would prefer to defer this until later.

(BivariateNormal() ** 3).draw() returns AttributeError: 'BivariateNormal' object has no attribute 'sim_fn' Same for ** inf. Seems like this should work like BoxModel example above.

Thanks for catching this. It is fixed now.

Your Brownian motion example now works; I'm just worried that I might have broken something else in the meantime. Could you please check to make sure that this change doesn't break something else?

I reran most of my recent notebooks.  There are some bugs.  Most notably, I'm getting many "This operation is only possible with results of consistent dimension" exceptions when trying to plot sample paths or take means.  A notebook with a few examples is attached.
Random process bugs.pdf
Also, adding Vector to Vector-valued RV-CTF does not work properly (see example in attached), though this might fall under "wait till later"

To replace "Tuple", how about "ProductSpaceOutcome" or "ProductSpaceResult"? (I realize Normal() ** inf is also a product space but returns Vector, so ProductSpace is not the greatest name. But I'm having trouble thinking of something better.)

This name is on the right track, but a little too verbose for my tastes.

"ProductOutcome" any better?

I imagine it's not possible to apply concat in RV space, or you would have implemented it already.

It actually shouldn't be too bad; I just didn't think of it as a use case. Let me think about the best way to implement this.

concat for RV would be nice, but I don't think it's essential now.

I'm not sure I like this. The usual behavior of cumsum is for it to start with the first element of the sequence being summed. What would the value of "initialize" be in this case?

I imagined usual cumsum would be cumsum(initialize=None), but I didn't really like this idea either.  I would prefer the concat for RV method.

Would it be possible to add Multinomial distributions? I might use in 305.

I think this is a good idea, but I don't have the bandwidth to do this. Do you think you can take this on? (It's straightforward, but requires some time.)

Sure, I'll try to do this over the weekend.

I reran most of my recent notebooks. There are some bugs. Most notably, I'm getting many "This operation is only possible with results of consistent dimension" exceptions when trying to plot sample paths or take means. A notebook with a few examples is attached.
Random process bugs.pdf

Thanks for catching this big bug. It was caused by an older change; it had nothing to do with replacing tuples by Vectors. It should be fixed now.

Also, adding Vector to Vector-valued RV-CTF does not work properly (see example in attached), though this might fall under "wait till later"

This example didn't work because the * operator returns a Tuple, not a Vector. The weird behavior is because addition of Tuples is concatenation. I should change that behavior.
Working on some of the other issues now.

I haven't run much yet, but * and & aren't working; problems with join?

Tuple.pdf

Thanks Kevin -- I think I just fixed that problem.
On Tue, Oct 2, 2018 at 12:45 PM kevindavisross<EMAIL_ADDRESS>wrote:

I haven't run much yet, but * and & aren't working; problems with join?
[image: image]
https://user-images.githubusercontent.com/27861382/46372489-2376e080-c640-11e8-8834-595947680585.png
Tuple.pdf https://github.com/dlsun/symbulate/files/2439435/Tuple.pdf
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dlsun/symbulate/pull/67#issuecomment-426405266, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABHvzuHbJS8nE3b6f1gMt4gu46OBg9lkks5ug8JGgaJpZM4VGLYy
.

I'm trying to remember where we left off.  Please correct me if I'm wrong, but I think we had decided that

* unpacks as Tuple
** unpacks as Vector
* took priority in order of operations

This is what was happening before at least.  For example,
X = RV(Poisson(1) ** 2 * Normal() ** 2)
X[0] # Vector of length 2 ~ Poisson(1)**2
X[1] # Vector of length 2 ~ Normal()**2
X[0][0] # int ~ Poisson(1)
X[1][0] # float ~ Normal()

But now  RV(Poisson(1) ** 2 * Normal() ** 2) is behaving like one big vector of length 4.  Is this a change you intended to make?  If so, it will take me longer to check, because that's different than how I had been testing before.


More powerful Copy and Depends

We want something like this:
!Copy
source: /work
rules:
- "*.py"
- "public/js/**/*.js"
- "/config"
- "!*~"
- "!*.old"
path: /some/dest

I.e. a list of rules similar to .gitignore rules, but reverse
/cc @anti-social

Will this feature imprement recursive !Depends on entire folder?

We decided to support only absolute paths for including rules.


Can we add a switch to avoid printing the warning?

WARNING: sonic onlu supportsGo116~1.20 88 CPU amd64, but your environment is not suitable

When a third framework uses Sonic and prints this warning, some users get confused.

Thanks for your reply, my environment is Golang 1.21 and macOS, but users should have different systems and Golang versions, I want to find a way to avoid printing the warning, not only for me. Thanks ~

The warning is expected in the unsupported environments for Sonic, and should not be avoided.

Okay, thanks.


Icon improvement suggestions

Testing https://github.com/Microsoft/vscode/issues/14318
Suggestions:

Increase size of icon as it's smaller than most extensions
Match the keyboard's colors with the One Dark theme (blueish gray with a select few colors for the keys)

Yo @bgashler1 can I get your help here?
Could you produce a slightly larger keyboard icon?
@Tyriar is right. The keyboard icon is noticeably smaller.

Full size.zip

The above zip has these images full size with no padding. Let me know if you need anything else.

Done (a while back, just not closed)


Bump senzing from 3.2.0 to 3.3.0

See https://github.com/Senzing/senzingapi-runtime/issues/31

Pull requests:

https://github.com/Senzing/docker-aptdownloader/pull/37


Updated Skype for Business 2016 client list

Updated the Skype for Business 2016 client list with the KBs released between June - September 2019.

@nenonix @JohanFreelancer9 Copy editing is needed for this PR. Thanks.

@PhillipGarding PR has been copyedited and is ready for final review, could you please check and merge? Thanks!

@PhillipGarding PR has been copyedited and is ready for final review, could you please check and merge? Thanks!


Enable noImplicitAny in the tsconfig

You will need to turn this off initially, and gradually add types as we go.  After all the types are defined, lets set noImplicitAny to true.

@kinwa91 This hasn't happened yet. Are you working on it?


feat: lower created_at limit

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

2 of 3 (66.67%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.06%) to 96.266%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/config/environment.ts
0
1
0.0%

Totals

Change from base Build<PHONE_NUMBER>:
-0.06%

Covered Lines:
900

Relevant Lines:
948

💛  - Coveralls


Flashing this to Syma X15W

Hi, would it be possible to completely ditch the onboard RX(BK2425), stock remote controller, and use NRF24l01+ as RX and NRF24l01+ with arduino as remote controller ?
How would I approach this ?
SOC: STM F032K6
MPU: 6050A

Hi any update on this?


If find tries to search inside a directory that doesn't have permission to access the file search function does nothing

The problem is that check_output raises an exception if the command returns a non success (!= 0) value subprocess.check_output will throw an CalledProcessError that is not catched never.
This could be solved by either catching the exception and reading the output.
try:
    find_out = subprocess.check_output(["find", "/path/I/cant/read", "-name", query])
except subprocess.CalledProcessError as e:
    find_out = e.output

or calling the command with a shell and adding an ; exit 0; at the end
    find_out = subprocess.check_output("find /path/I/cant/read -name "+query+";exit 0", shell=True)

Also getting the default directory (or directories because query could be a list) on the config file would be really useful

feel free to submit these things as pull requests (try + except seems cleaner. I can modify this in a couple hours as I am on mobile atm), but I should note that this is a script intended to be used with lighthouse, not a part of lighthouse itself.  the config file should not include anything related to this script.