CUDA 11.2: Support the built-in Stream Ordered Memory Allocator

While this is working, I mark it as Work in Progress as there are some issues to be discussed with our NVIDIA friends 🙂
May be blocked by #4443 (?)
This PR exposes CUDA's new Stream Ordered Memory Allocator added since 11.2 to CuPy. A new memory type, MemoryAsync, is added, which is backed by cudaMallocAsync() and cudaFreeAsync().
To use this feature, one simply sets the allocator to malloc_async, similar to what's done for managed memory:
import cupy as cp

cp.cuda.set_allocator(cp.cuda.malloc_async)
# from now on the memory is allocated on the current stream from Stream Ordered Memory Allocator

On older CUDA (<11.2) or unsupported devices, using this new allocator will raise an error at runtime.
(I didn't add the support with a customized mempool cudaMemPool*()/cudaMallocFromPoolAsync() -- which could be the next PR -- as it's unclear to me the benefit of using non-default mempools. Also, note that there is no API to expose any current information of the mempool, so it wouldn't be compatible with CuPy's MemoryPool API, such as used_bytes() etc.)
Currently observed issues
I think nothing is wrong with my implementation, most likely these are from CUDA 😁

UPDATE: This is irrelevant of this PR, see https://github.com/cupy/cupy/pull/4537#issuecomment-757429743 and #4538.
It is unclear from the CUDA documentation if a stream is allowed to be destroyed before all memory allocated on it is freed. It could be that the driver performs a ref count internally (so we don't have to), but we need to make sure. If it's not the case, then in MemoryAsync we will also need to hold the reference to the stream (object), not just its pointer.
nvprof python my_script.py will fail if malloc_async is used in the workload:

$ nvprof --device-buffer-size 2048 --profiling-semaphore-pool-size 128000 pytest tests/cupy_tests/fft_tests/test_fft.py -k TestFFt
========================================================================= test session starts =========================================================================
platform linux -- Python 3.7.9, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /home/leofang/dev/cupy_cuda112, configfile: setup.cfg
collecting ... ==31333== NVPROF is profiling process 31333, command: /home/leofang/miniconda3/envs/cupy_cuda112_dev/bin/python /home/leofang/miniconda3/envs/cupy_cuda112_dev/bin/pytest tests/cupy_tests/fft_tests/test_fft.py -k TestFFt
collected 717 items / 410 deselected / 307 selected

tests/cupy_tests/fft_tests/test_fft.py ..................................................==31333== Error: Internal profiling error 3938:999.
.....................................^C======== Warning: 569 records have invalid timestamps due to insufficient device buffer space. You can configure the buffer space using the option --device-buffer-size.
======== Warning: 293 records have invalid timestamps due to insufficient semaphore pool size. You can configure the pool size using the option --profiling-semaphore-pool-size.
======== Profiling result:
...

We need to confirm if this is nvprof's problem/limitation (very likely it is), as it could be annoying to our users.
TODO

[x] Add tests
[x] Add tutorial to docs/source/reference/memory.rst?
[x] Fix/update docstrings
[x] Mark it experimental (as I did)?

cc: @jakirkham @pentschev @maxpkatz Could you help address the three observed issues? 🙂

invalid filename
!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "lgenfe: EDG 6.0", isOptimized: true, runtimeVersion: 0, emissionKind: NoDebug, enums: !2)
!1 = !DIFile(filename: "", directory: "/home/leofang/dev/cupy_cuda112")
invalid filename
!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "lgenfe: EDG 6.0", isOptimized: true, runtimeVersion: 0, emissionKind: NoDebug, enums: !2)
!1 = !DIFile(filename: "", directory: "/home/leofang/dev/cupy_cuda112")

This seems to be a CUDA 11.2 issue, irrelevant of the async allocator. My best guess is from NVRTC. If I call the low-level API directly (as I did in my C reproducer), everything was fine:
ptr = cp.cuda.runtime.mallocAsync(100, 0)
cp.cuda.runtime.freeAsync(ptr, 0)

invalid filename
!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "lgenfe: EDG 6.0", isOptimized: true, runtimeVersion: 0, emissionKind: NoDebug, enums: !2)
!1 = !DIFile(filename: "", directory: "/home/leofang/dev/cupy_cuda112")
invalid filename
!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "lgenfe: EDG 6.0", isOptimized: true, runtimeVersion: 0, emissionKind: NoDebug, enums: !2)
!1 = !DIFile(filename: "", directory: "/home/leofang/dev/cupy_cuda112")

This seems to be a CUDA 11.2 issue, irrelevant of the async allocator. My best guess is from NVRTC. If I call the low-level API directly (as I did in my C reproducer), everything was fine:
ptr = cp.cuda.runtime.mallocAsync(100, 0)
cp.cuda.runtime.freeAsync(ptr, 0)

invalid filename
!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "lgenfe: EDG 6.0", isOptimized: true, runtimeVersion: 0, emissionKind: NoDebug, enums: !2)
!1 = !DIFile(filename: "", directory: "/home/leofang/dev/cupy_cuda112")
invalid filename
!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "lgenfe: EDG 6.0", isOptimized: true, runtimeVersion: 0, emissionKind: NoDebug, enums: !2)
!1 = !DIFile(filename: "", directory: "/home/leofang/dev/cupy_cuda112")

This seems to be a CUDA 11.2 issue, irrelevant of the async allocator. My best guess is from NVRTC.

Bingo. This fixes it:
diff --git a/cupy/cuda/compiler.py b/cupy/cuda/compiler.py
index eb74a92c7..5d9216ff8 100644
--- a/cupy/cuda/compiler.py
+++ b/cupy/cuda/compiler.py
@@ -364,7 +364,7 @@ def _preprocess(source, options, arch, backend):
     if backend == 'nvrtc':
         options += ('-arch=compute_{}'.format(arch),)

-        prog = _NVRTCProgram(source, '')
+        prog = _NVRTCProgram(source, '_cupy_nvrtc_preprocess')
         try:
             result, _ = prog.compile(options)
         except CompileException as e:

invalid filename
!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "lgenfe: EDG 6.0", isOptimized: true, runtimeVersion: 0, emissionKind: NoDebug, enums: !2)
!1 = !DIFile(filename: "", directory: "/home/leofang/dev/cupy_cuda112")
invalid filename
!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "lgenfe: EDG 6.0", isOptimized: true, runtimeVersion: 0, emissionKind: NoDebug, enums: !2)
!1 = !DIFile(filename: "", directory: "/home/leofang/dev/cupy_cuda112")

This seems to be a CUDA 11.2 issue, irrelevant of the async allocator. My best guess is from NVRTC.

Bingo. This fixes it:
diff --git a/cupy/cuda/compiler.py b/cupy/cuda/compiler.py
index eb74a92c7..5d9216ff8 100644
--- a/cupy/cuda/compiler.py
+++ b/cupy/cuda/compiler.py
@@ -364,7 +364,7 @@ def _preprocess(source, options, arch, backend):
     if backend == 'nvrtc':
         options += ('-arch=compute_{}'.format(arch),)

-        prog = _NVRTCProgram(source, '')
+        prog = _NVRTCProgram(source, '_cupy_nvrtc_preprocess')
         try:
             result, _ = prog.compile(options)
         except CompileException as e:

@leofang thanks for working on this, I've raised the question about destroying the stream before all memory allocated on it is freed, I'll let you know as soon as I get an answer about that.
As for nvprof, could you also try Nsight Systems before we raise questions on nvprof? The former has been deprecated, so I'm not entirely sure whether we can expect new features such as the Stream Ordered Memory Allocator should work correctly on it, therefore it's probably best if we know whether this is a problem in both or nvprof only. Sorry for not checking it myself, but I don't have access to a machine with 11.2 at the moment.

@leofang thanks for working on this, I've raised the question about destroying the stream before all memory allocated on it is freed, I'll let you know as soon as I get an answer about that.
As for nvprof, could you also try Nsight Systems before we raise questions on nvprof? The former has been deprecated, so I'm not entirely sure whether we can expect new features such as the Stream Ordered Memory Allocator should work correctly on it, therefore it's probably best if we know whether this is a problem in both or nvprof only. Sorry for not checking it myself, but I don't have access to a machine with 11.2 at the moment.

Regarding the stream destruction, answer from @jrhemstad:

Yes. When you destroy a stream it inserts a callback into that stream that does the actual cleanup. Any pending work before the stream is destroyed is still valid.

Regarding the stream destruction, answer from @jrhemstad:

Yes. When you destroy a stream it inserts a callback into that stream that does the actual cleanup. Any pending work before the stream is destroyed is still valid.

@leofang the following is allowed:
cudaMallocAsync(&ptr, size, stream);
cudaStreamDestroy(stream);
cudaFreeAsync(ptr); // or cudaFree(ptr);

I believe you're fine on implementing it as you see fits best to CuPy.

@leofang the following is allowed:
cudaMallocAsync(&ptr, size, stream);
cudaStreamDestroy(stream);
cudaFreeAsync(ptr); // or cudaFree(ptr);

I believe you're fine on implementing it as you see fits best to CuPy.

cc @jrhemstad @harrism @nsakharnykh (who may have thoughts on this 🙂)

cc @jrhemstad @harrism @nsakharnykh (who may have thoughts on this 🙂)

@leofang the following is allowed:
cudaMallocAsync(&ptr, size, stream);
cudaStreamDestroy(stream);
cudaFreeAsync(ptr); // or cudaFree(ptr);

@pentschev @jrhemstad I don't think this works. In Python I got cudaErrorInvalidResourceHandle while in C I got cudaErrorContextIsDestroyed. I am still trying to figure out why the error code is different, but this doesn't look good...
>>> import cupy as cp
>>> s = cp.cuda.Stream()
>>> s.ptr
94049875546000
>>> ptr = cp.cuda.runtime.mallocAsync(100, s.ptr)
>>> del s
>>> cp.cuda.runtime.freeAsync(ptr,<PHONE_NUMBER>6000)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "cupy_backends/cuda/api/runtime.pyx", line 638, in cupy_backends.cuda.api.runtime.freeAsync
    cpdef freeAsync(intptr_t ptr, intptr_t stream):
  File "cupy_backends/cuda/api/runtime.pyx", line 643, in cupy_backends.cuda.api.runtime.freeAsync
    check_status(status)
  File "cupy_backends/cuda/api/runtime.pyx", line 253, in cupy_backends.cuda.api.runtime.check_status
    raise CUDARuntimeError(status)
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorInvalidResourceHandle: invalid resource handle

C:
// nvcc -std=c++11 -arch=sm_75 test_mallocAsync.cu -o test_mallocAsync
#include <cstdio>
#include <cstdlib>

int main() {
    double* a;
    cudaStream_t s;

    int error = cudaStreamCreate(&s);
    if (error != 0) {
        printf("failed!\n");
        exit(1);
    }
    printf("stream ok!\n");
    cudaDeviceSynchronize();

    error = cudaMallocAsync(&a, 100, s);
    if (error != 0) {
        printf("failed!\n");
        exit(1);
    }
    printf("malloc ok!\n");
    cudaDeviceSynchronize();

    error = cudaStreamDestroy(s);
    if (error != 0) {
        printf("failed!\n");
        exit(1);
    }
    printf("stream destory ok!\n");
    cudaDeviceSynchronize();

    error = cudaFreeAsync(a, s);
    if (error != 0) {
        printf("failed: %i ", error);
        printf("%s\n", cudaGetErrorString((cudaError_t)error));
        exit(1);
    }
    printf("free ok!\n");
    cudaDeviceSynchronize();

    return 0;
}

/* output: */
// stream ok!
// malloc ok!
// stream destory ok!
// failed: 709 context is destroyed

@leofang the following is allowed:
cudaMallocAsync(&ptr, size, stream);
cudaStreamDestroy(stream);
cudaFreeAsync(ptr); // or cudaFree(ptr);

@pentschev @jrhemstad I don't think this works. In Python I got cudaErrorInvalidResourceHandle while in C I got cudaErrorContextIsDestroyed. I am still trying to figure out why the error code is different, but this doesn't look good...
>>> import cupy as cp
>>> s = cp.cuda.Stream()
>>> s.ptr
94049875546000
>>> ptr = cp.cuda.runtime.mallocAsync(100, s.ptr)
>>> del s
>>> cp.cuda.runtime.freeAsync(ptr,<PHONE_NUMBER>6000)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "cupy_backends/cuda/api/runtime.pyx", line 638, in cupy_backends.cuda.api.runtime.freeAsync
    cpdef freeAsync(intptr_t ptr, intptr_t stream):
  File "cupy_backends/cuda/api/runtime.pyx", line 643, in cupy_backends.cuda.api.runtime.freeAsync
    check_status(status)
  File "cupy_backends/cuda/api/runtime.pyx", line 253, in cupy_backends.cuda.api.runtime.check_status
    raise CUDARuntimeError(status)
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorInvalidResourceHandle: invalid resource handle

C:
// nvcc -std=c++11 -arch=sm_75 test_mallocAsync.cu -o test_mallocAsync
#include <cstdio>
#include <cstdlib>

int main() {
    double* a;
    cudaStream_t s;

    int error = cudaStreamCreate(&s);
    if (error != 0) {
        printf("failed!\n");
        exit(1);
    }
    printf("stream ok!\n");
    cudaDeviceSynchronize();

    error = cudaMallocAsync(&a, 100, s);
    if (error != 0) {
        printf("failed!\n");
        exit(1);
    }
    printf("malloc ok!\n");
    cudaDeviceSynchronize();

    error = cudaStreamDestroy(s);
    if (error != 0) {
        printf("failed!\n");
        exit(1);
    }
    printf("stream destory ok!\n");
    cudaDeviceSynchronize();

    error = cudaFreeAsync(a, s);
    if (error != 0) {
        printf("failed: %i ", error);
        printf("%s\n", cudaGetErrorString((cudaError_t)error));
        exit(1);
    }
    printf("free ok!\n");
    cudaDeviceSynchronize();

    return 0;
}

/* output: */
// stream ok!
// malloc ok!
// stream destory ok!
// failed: 709 context is destroyed

(If the stream is hold valid before calling cudaFreeAsync, everything works fine as expected.)

(If the stream is hold valid before calling cudaFreeAsync, everything works fine as expected.)

This isn't valid.
cudaStreamCreate(&s);
cudaMallocAsync(p, s);
cudaStreamDestroy(s)
cudaFreeAsync(a, s);

You can't use a stream after it's been destroyed. There's nothing unique about cudaMallocAsync in this regard. The above would be the same as:
cudaStreamCreate(&s);
kernel<<<..., s>>>(...);
cudaStreamDestroy(s)
kernel<<<..., s>>>(...); // you can't do this

That's a use-after-free error. It's like trying to use a pointer after it's been freed.

This isn't valid.
cudaStreamCreate(&s);
cudaMallocAsync(p, s);
cudaStreamDestroy(s)
cudaFreeAsync(a, s);

You can't use a stream after it's been destroyed. There's nothing unique about cudaMallocAsync in this regard. The above would be the same as:
cudaStreamCreate(&s);
kernel<<<..., s>>>(...);
cudaStreamDestroy(s)
kernel<<<..., s>>>(...); // you can't do this

That's a use-after-free error. It's like trying to use a pointer after it's been freed.

cc: @maxpkatz

cc: @maxpkatz

Thanks for quick reply, @jrhemstad! So, the only legit way to free a stream-ordered memory after the stream is destroyed (see Issue No.2 in the PR description) is to call cudaFree? If so, does it mean we should better always just call cudaFree instead of cudaFreeAsync?

Thanks for quick reply, @jrhemstad! So, the only legit way to free a stream-ordered memory after the stream is destroyed (see Issue No.2 in the PR description) is to call cudaFree? If so, does it mean we should better always just call cudaFree instead of cudaFreeAsync?

@leofang sorry for my original answer, it incomplete at best. In addition to cudaFree, you can also call cudaFreeAsync on a different stream, but never on the stream that has been destroyed already.

@leofang sorry for my original answer, it incomplete at best. In addition to cudaFree, you can also call cudaFreeAsync on a different stream, but never on the stream that has been destroyed already.

So, the only legit way to free a stream-ordered memory after the stream is destroyed (see Issue No.2 in the PR description) is to call cudaFree?

No. You can free on another stream so long as it was somehow ordered with the original stream the pointer was allocated on. There's a number of ways you can do this. Here's one example using events:
cudaMallocAsync(p, s0);
kernel<<<..., s0>>>(p, ...);
cudaEventRecord(e, s0);
cudaStreamWaitEvent(s1, e);
cudaStreamDestroy(s0);
cudaFreeAsync(p, s1);

If so, does it mean we should better always just call cudaFree instead of cudaFreeAsync?

No, because you lose a lot of the benefit of using cudaMalloc/FreeAsync in the first place, i.e., avoiding the device sync implicit in cudaFree.

So, the only legit way to free a stream-ordered memory after the stream is destroyed (see Issue No.2 in the PR description) is to call cudaFree?

No. You can free on another stream so long as it was somehow ordered with the original stream the pointer was allocated on. There's a number of ways you can do this. Here's one example using events:
cudaMallocAsync(p, s0);
kernel<<<..., s0>>>(p, ...);
cudaEventRecord(e, s0);
cudaStreamWaitEvent(s1, e);
cudaStreamDestroy(s0);
cudaFreeAsync(p, s1);

If so, does it mean we should better always just call cudaFree instead of cudaFreeAsync?

No, because you lose a lot of the benefit of using cudaMalloc/FreeAsync in the first place, i.e., avoiding the device sync implicit in cudaFree.

Ah...OK thanks for speedy replies @pentschev @jrhemstad!!! So to follow up further: If at some point all other non-default streams are destroyed, I can always call cudaFreeAsync on the legacy or per-thread default stream, or stream 0, right?

Ah...OK thanks for speedy replies @pentschev @jrhemstad!!! So to follow up further: If at some point all other non-default streams are destroyed, I can always call cudaFreeAsync on the legacy or per-thread default stream, or stream 0, right?

I can always call cudaFreeAsync on the legacy or per-thread default stream, or stream 0, right?

You technically can with the legacy default stream as it implicitly synchronous with other streams (except non-blocking streams!), but I don't suggest going this route. You cannot with the per-thread default stream as it is not implicitly synchronous with other streams.

I can always call cudaFreeAsync on the legacy or per-thread default stream, or stream 0, right?

You technically can with the legacy default stream as it implicitly synchronous with other streams (except non-blocking streams!), but I don't suggest going this route. You cannot with the per-thread default stream as it is not implicitly synchronous with other streams.

OK @jrhemstad I think this is the final question (I hope!)

You can free on another stream so long as it was somehow ordered with the original stream the pointer was allocated on.

You cannot with the per-thread default stream as it is not implicitly synchronous with other streams.

So does it mean the driver does not guarantee the correct stream order and it's users' responsibility to ensure it (via cudaStreamSynchronize or other means)?
For example, for the case I just described (no other streams are alive), if before I destroy the stream on which the memory was allocated I add an event to wait on the PTDS, I can then free it on the PTDS right? Something like
cudaMallocAsync(&x, 1024, s);
// do something with x on s...
cudaEventCreate(&e);
cudaEventRecord(&e, s);
cudaStreamDestroy(s);
cudaStreamWaitEvent(1, e, 0);  // wait on PTDS (1)
cudaFreeAsync(x, 1);  // free on PTDS

OK @jrhemstad I think this is the final question (I hope!)

You can free on another stream so long as it was somehow ordered with the original stream the pointer was allocated on.

You cannot with the per-thread default stream as it is not implicitly synchronous with other streams.

So does it mean the driver does not guarantee the correct stream order and it's users' responsibility to ensure it (via cudaStreamSynchronize or other means)?
For example, for the case I just described (no other streams are alive), if before I destroy the stream on which the memory was allocated I add an event to wait on the PTDS, I can then free it on the PTDS right? Something like
cudaMallocAsync(&x, 1024, s);
// do something with x on s...
cudaEventCreate(&e);
cudaEventRecord(&e, s);
cudaStreamDestroy(s);
cudaStreamWaitEvent(1, e, 0);  // wait on PTDS (1)
cudaFreeAsync(x, 1);  // free on PTDS

So does it mean the driver does not guarantee the correct stream order and it's users' responsibility to ensure it (via cudaStreamSynchronize or other means)?

Yes. The per-thread default stream is effectively the same as any other stream.

For example, for the case I just described (no other streams are alive), if before I destroy the stream on which the memory was allocated I add an event to wait on the PTDS, I can then free it on the PTDS right?

Yep, that works. That said, event creation and even recording events is relatively expensive, so it's not something you want to do all the time if you can avoid it.
@harrism wrote a nice benchmark of event overheads recently: https://github.com/harrism/cuda_event_benchmark

So does it mean the driver does not guarantee the correct stream order and it's users' responsibility to ensure it (via cudaStreamSynchronize or other means)?

Yes. The per-thread default stream is effectively the same as any other stream.

For example, for the case I just described (no other streams are alive), if before I destroy the stream on which the memory was allocated I add an event to wait on the PTDS, I can then free it on the PTDS right?

Yep, that works. That said, event creation and even recording events is relatively expensive, so it's not something you want to do all the time if you can avoid it.
@harrism wrote a nice benchmark of event overheads recently: https://github.com/harrism/cuda_event_benchmark

@emcastillo Thanks! Note that I moved memory.rst from docs/source/reference/ to docs/source/tutorial/, as I think it looks much more like a tutorial than an API reference. I also expanded it to address how to use managed memory/async memory.
One question: Would this be a fair statement that I can add to the tutorial for clarification?

Similar to CuPy's memory pool, Stream Ordered Memory Allocator also allocates and deallocates memory asynchronously from a memory pool in a stream-ordered fashion. The key difference is that it is a built-in feature provided in CUDA by NVIDIA.

@emcastillo Thanks! Note that I moved memory.rst from docs/source/reference/ to docs/source/tutorial/, as I think it looks much more like a tutorial than an API reference. I also expanded it to address how to use managed memory/async memory.
One question: Would this be a fair statement that I can add to the tutorial for clarification?

Similar to CuPy's memory pool, Stream Ordered Memory Allocator also allocates and deallocates memory asynchronously from a memory pool in a stream-ordered fashion. The key difference is that it is a built-in feature provided in CUDA by NVIDIA.

Do we have performance measurements? I wonder if this has a real impact in regular cupy codes or we need to do non-standard things for this to show a noticeable benefit.

Do we have performance measurements? I wonder if this has a real impact in regular cupy codes or we need to do non-standard things for this to show a noticeable benefit.

Do we have performance measurements? I wonder if this has a real impact in regular cupy codes or we need to do non-standard things for this to show a noticeable benefit.

Hi @emcastillo I actually tried switching from CuPy's default pool to this new pool locally (done by the end of cupy/__init__.py), and ran a few test files in tests/. The timing not surprisingly remains the same, because we didn't put the tests on different streams, but at least this shows no performance regression if a user just adopts this new pool as default. I currently don't have workloads that are complicated enough to utilize many streams yet simple enough to permit a drop-in replacement, perhaps I could try digging out some sample codes to test further.
Performance aside, I see two main benefits of making this new pool available:

In the long-term, the maintenance burden of a performant memory pool can be shifted to NVIDIA (quoting an offline discussion with @maxpkatz 😅)
More importantly, it has the potential to eliminate a lot of issues when interoperating with other libraries like PyTorch, Numba, the RAPIDS stack, etc. If everyone agrees to use the device's default memory pool (or any CUDA memory pool with cudaMemPool_t properly exposed to Python), many interoperating challenges to avoid OOM (such as pytorch/pytorch#43144 or the need of cupy.cuda.PythonFunctionAllocator, for example) will be resolved with zero effort (from the community)!

Do we have performance measurements? I wonder if this has a real impact in regular cupy codes or we need to do non-standard things for this to show a noticeable benefit.

Hi @emcastillo I actually tried switching from CuPy's default pool to this new pool locally (done by the end of cupy/__init__.py), and ran a few test files in tests/. The timing not surprisingly remains the same, because we didn't put the tests on different streams, but at least this shows no performance regression if a user just adopts this new pool as default. I currently don't have workloads that are complicated enough to utilize many streams yet simple enough to permit a drop-in replacement, perhaps I could try digging out some sample codes to test further.
Performance aside, I see two main benefits of making this new pool available:

In the long-term, the maintenance burden of a performant memory pool can be shifted to NVIDIA (quoting an offline discussion with @maxpkatz 😅)
More importantly, it has the potential to eliminate a lot of issues when interoperating with other libraries like PyTorch, Numba, the RAPIDS stack, etc. If everyone agrees to use the device's default memory pool (or any CUDA memory pool with cudaMemPool_t properly exposed to Python), many interoperating challenges to avoid OOM (such as pytorch/pytorch#43144 or the need of cupy.cuda.PythonFunctionAllocator, for example) will be resolved with zero effort (from the community)!

If everyone agrees to use the device's default memory pool (or any CUDA memory pool with cudaMemPool_t properly exposed to Python), many interoperating challenges to avoid OOM (such as pytorch/pytorch#43144 or the need of cupy.cuda.PythonFunctionAllocator, for example) will be resolved with zero effort (from the community)!

I want to be very clear that even with cudaMallocAsync there is still tremendous value in allowing an external allocator interface, e.g., layering other functionality on top of allocation such as logging, leak checking, etc. Furthermore, application specific allocators will still usually be able to beat cudaMallocAsync's performance.
We shouldn't be thinking that cudaMallocAsync obviates the need for libraries to provide hooks for users to customize allocation.

If everyone agrees to use the device's default memory pool (or any CUDA memory pool with cudaMemPool_t properly exposed to Python), many interoperating challenges to avoid OOM (such as pytorch/pytorch#43144 or the need of cupy.cuda.PythonFunctionAllocator, for example) will be resolved with zero effort (from the community)!

I want to be very clear that even with cudaMallocAsync there is still tremendous value in allowing an external allocator interface, e.g., layering other functionality on top of allocation such as logging, leak checking, etc. Furthermore, application specific allocators will still usually be able to beat cudaMallocAsync's performance.
We shouldn't be thinking that cudaMallocAsync obviates the need for libraries to provide hooks for users to customize allocation.

Thanks, @jrhemstad.

even with cudaMallocAsync there is still tremendous value in allowing an external allocator interface,

I would have thought that if everyone is on the same pool, the only job an external allocator interface needs to do is to pass around the pool pointer cudaMemPool_t* in use, no?

Furthermore, application specific allocators will still usually be able to beat cudaMallocAsync's performance.

What specific examples you have in mind for this?

Thanks, @jrhemstad.

even with cudaMallocAsync there is still tremendous value in allowing an external allocator interface,

I would have thought that if everyone is on the same pool, the only job an external allocator interface needs to do is to pass around the pool pointer cudaMemPool_t* in use, no?

Furthermore, application specific allocators will still usually be able to beat cudaMallocAsync's performance.

What specific examples you have in mind for this?

I would have thought that if everyone is on the same pool, the only job an external allocator interface needs to do is to pass around the pool pointer cudaMemPool_t* in use, no?

No, I might want to layer additional functionality on top of the CUDA pool, like logging, leak check, etc. Or I might want to use an all together different allocator that is tuned to a specific use case.
Point is, just because CUDA is providing a convenient solution that is good for 80% of cases doesn't mean we should block the flexibility to accomplish the remaining 20% of cases.

I would have thought that if everyone is on the same pool, the only job an external allocator interface needs to do is to pass around the pool pointer cudaMemPool_t* in use, no?

No, I might want to layer additional functionality on top of the CUDA pool, like logging, leak check, etc. Or I might want to use an all together different allocator that is tuned to a specific use case.
Point is, just because CUDA is providing a convenient solution that is good for 80% of cases doesn't mean we should block the flexibility to accomplish the remaining 20% of cases.

I see. Right, I was mainly thinking on the 80% side, but flexibility is definitely also something we would love to have.

I see. Right, I was mainly thinking on the 80% side, but flexibility is definitely also something we would love to have.

@leofang I have not updated the post yet to compare with cudaMallocAsync, but I recommend checking out this blog post we wrote about RAPIDS Memory Manager: https://developer.nvidia.com/blog/fast-flexible-allocation-for-cuda-with-rapids-memory-manager/  But in particular, the RMM pool allocator implementation is currently about 5x faster than cudaMallocAsync+cudaFreeAsync in our benchmarks. I'm sure this will get better, but as Jake pointed out, one can always customize allocation for application-specific allocation patterns to achieve better performance -- a general purpose allocator cannot do that.
The blog post emphasizes the need for what we call External Allocator Interfaces (EAIs) in libraries for the benefit of the ecosystem, in particular to enable the flexible layering of memory-oriented functionality beyond just allocation, as Jake points out.

@leofang I have not updated the post yet to compare with cudaMallocAsync, but I recommend checking out this blog post we wrote about RAPIDS Memory Manager: https://developer.nvidia.com/blog/fast-flexible-allocation-for-cuda-with-rapids-memory-manager/  But in particular, the RMM pool allocator implementation is currently about 5x faster than cudaMallocAsync+cudaFreeAsync in our benchmarks. I'm sure this will get better, but as Jake pointed out, one can always customize allocation for application-specific allocation patterns to achieve better performance -- a general purpose allocator cannot do that.
The blog post emphasizes the need for what we call External Allocator Interfaces (EAIs) in libraries for the benefit of the ecosystem, in particular to enable the flexible layering of memory-oriented functionality beyond just allocation, as Jake points out.

@harrism wrote a nice benchmark of event overheads recently: https://github.com/harrism/cuda_event_benchmark

I recommend checking out this blog post we wrote about RAPIDS Memory Manager: https://developer.nvidia.com/blog/fast-flexible-allocation-for-cuda-with-rapids-memory-manager/

Thanks for the pointers @jrhemstad @harrism! It is very nice to see these performance benchmarks. I think we should add RMM to CuPy's interoperable list 🙂
@emcastillo I split the PR into two, so this PR only exposes the necessary APIs (async malloc & free). In the next PR (#4592) the handling of cudaMemPool_t will be added. This PR should be ready for review now.

Jenkins, test this please

Jenkins CI test (for commit 325571ac954fca5f049b7b11146e616c1f7d5678, target branch master) failed with status FAILURE.

CI errored out due to No space left on device. cc: @kmaehashi

Jenkins, test this please

Jenkins CI test (for commit 325571ac954fca5f049b7b11146e616c1f7d5678, target branch master) failed with status FAILURE.

Jenkins, test this please

Jenkins CI test (for commit f21fe825c1ad7714b7d9198eed4d19e564556037, target branch master) failed with status FAILURE.

Jenkins, test this please

Jenkins CI test (for commit f2f8f4f200b325cc33fe6da8ffe0d316c3658ba2, target branch master) failed with status FAILURE.

Jenkins never started....?
Jenkins, test this please

pfnCI, test this please

Jenkins CI test (for commit f2f8f4f200b325cc33fe6da8ffe0d316c3658ba2, target branch master) failed with status FAILURE.

CI failures are known and unrelated.

Jenkins, test this please

Jenkins CI test (for commit f2f8f4f200b325cc33fe6da8ffe0d316c3658ba2, target branch master) succeeded!

Thanks @emcastillo and all!


How to get the scope which is outside of the form directive

Hi,
There is one scenario where am accessing a ngModel value inside a form directive but the parent scope is not getting inside this form directive. How can i access the parent scope inside this form. The code look like below,

   <form sf-schema="schema" sf-form="form"> {{$scope.customVal}}</form>

In the above code sf-schema is an directive used for dynamic form building, $scope,customVal which is defined inside parent scope. While am trying to print the value of parent scope inside form directive its not getting accessed. So what is the reason behind this issue. Could anyone help me to sort out this issue.
Regards,
Akshath

Please read the contributing guidelines; these types of questions are ill-suited for GitHub. Thanks!


Updated java stack

Coverage remained the same at 92.523% when pulling e573a8a0964630d4fe83e44699e93e88eec04a3e on updated-java-stack-1 into bea053e4229d3e404aa981f5cd6807509f252775 on master.

Coverage remained the same at 92.523% when pulling e573a8a0964630d4fe83e44699e93e88eec04a3e on updated-java-stack-1 into bea053e4229d3e404aa981f5cd6807509f252775 on master.


EO is missing from GitHub's programming languages list

GitHub maintains a list of programming languages developed on GitHub: https://github.com/showcases/programming-languages.
We need to be there.

@yegor256 please, pay attention to this issue

@alexpanov good point, thanks! I've sent an email to GitHub support.

I proposed adding the EO programming language to the list. Now it is in the end of the list.
So, maybe close?
@yegor256

I proposed adding the EO programming language to the list. Now it is in the end of the list.
So, maybe close?
@yegor256

@yegor256 maybe let's close the issue as it is resolved?

@yegor256 maybe let's close the issue as it is resolved?

@nlchar thanks!

@nlchar thanks!

There is an unrecoverable failure on my side. Please, submit it here:
PID: 4@4a07447a-25ae-4dc9-b0d1-e99cf6d3ad97, thread: pool-1155-thread-1
com.zerocracy.radars.github.ReSafe[81] java.io.IOException: com.zerocracy.radars.github.GhProjectNotFoundException: Github project for 'cqfn/eo' repo was not found
com.zerocracy.radars.github.GhProject[69] com.zerocracy.radars.github.GhProjectNotFoundException: Github project for 'cqfn/eo' repo was not found

0.55: Issue: cqfn/eo#120, Action: created

There is an unrecoverable failure on my side. Please, submit it here:
PID: 4@4a07447a-25ae-4dc9-b0d1-e99cf6d3ad97, thread: pool-1155-thread-1
com.zerocracy.radars.github.ReSafe[81] java.io.IOException: com.zerocracy.radars.github.GhProjectNotFoundException: Github project for 'cqfn/eo' repo was not found
com.zerocracy.radars.github.GhProject[69] com.zerocracy.radars.github.GhProjectNotFoundException: Github project for 'cqfn/eo' repo was not found

0.55: Issue: cqfn/eo#120, Action: created

There is an unrecoverable failure on my side. Please, submit it here:
PID: 4@4a07447a-25ae-4dc9-b0d1-e99cf6d3ad97, thread: pool-1158-thread-1
com.zerocracy.radars.github.ReSafe[81] java.io.IOException: com.zerocracy.radars.github.GhProjectNotFoundException: Github project for 'cqfn/eo' repo was not found
com.zerocracy.radars.github.GhProject[69] com.zerocracy.radars.github.GhProjectNotFoundException: Github project for 'cqfn/eo' repo was not found

0.55: Issue: cqfn/eo#120, Action: created

There is an unrecoverable failure on my side. Please, submit it here:
PID: 4@4a07447a-25ae-4dc9-b0d1-e99cf6d3ad97, thread: pool-1158-thread-1
com.zerocracy.radars.github.ReSafe[81] java.io.IOException: com.zerocracy.radars.github.GhProjectNotFoundException: Github project for 'cqfn/eo' repo was not found
com.zerocracy.radars.github.GhProject[69] com.zerocracy.radars.github.GhProjectNotFoundException: Github project for 'cqfn/eo' repo was not found

0.55: Issue: cqfn/eo#120, Action: created

You @nlchar/z need to have one of these roles in order to do "Request order start": ARC, PO; I'm sorry to say this, but at the moment you've got no roles in this project


[BugFix] backport_pr_34317_to_branch-3.1

… primary backend" (#34317)
Why I'm doing:
forget backport pr : https://github.com/StarRocks/starrocks/pull/34317 to branch-3.1
What I'm doing:
manually backport pr : https://github.com/StarRocks/starrocks/pull/34317 to branch-3.1
Fixes #issue
What type of PR is this:

[x] BugFix
[ ] Feature
[ ] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Does this PR entail a change in behavior?

[ ] Yes, this PR will result in a change in behavior.
[x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

[ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
[ ] Parameter changes: default values, similar parameters but with different default values
[ ] Policy changes: use new policy to replace old one, functionality automatically enabled
[ ] Feature removed
[ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto-backported to the target branch

[ ] 3.2
[ ] 3.1
[ ] 3.0
[ ] 2.5

ref https://github.com/StarRocks/starrocks/pull/34317

ref https://github.com/StarRocks/starrocks/pull/34317


Validation driven Callbacks

Adding any validation dataset performance related info into callbacks can be usefull. See PPE

Completed with #197


Confluence Pages with page property reports and labels

This change will:

Init Base Pages (Customers, Detection Rules, etc.) including page property macros

Currently only for Customers as an example
Page Property Report is limited to page labels

Label each Customer Page

Benefit: A nice 'overview' page for each bucket.

Hello @andurin! Thank you very much for your contribution (:
Could you please resolve the conflicts in the PR?
You've created PR to the master branch, but we restrict access to master and we all do PRs only to develop (so I've changed it to develop).

Just created a new PR based on develop branch - this one is no longer needed.


add support for repo regex filter

HMCTS Nightly Build needs support for Repositories regex filter this adds that

Closing old PR


Required fields referenced by optional relations are wrongly removed from create

Summary
There seems to be an issue with the generated SDK when a field is used in a relation that is not required.
Example
Schema
model project {
    // Fields
    name       String   @unique
    main_workspace_name String?

    // Relations
    main_workspace workspace? @relation("primary_workspace", fields: [name, main_workspace_name], references: [project_name, name])
    workspaces workspace[]

    @@unique([name, main_workspace_name])
}

model workspace {
    // Fields
    name       String   @default("default")
    project_name String

    // Relations
    project      project @relation(fields: [project_name], references: [name], onDelete: Cascade)
    is_primary_workspace project? @relation("primary_workspace")

    @@unique([project_name, name])

}

Outputs
This will generate the incorrect .project().create(...) function signature.
pub fn create(self, _params: Vec<SetParam>) -> Create<'a>

If main_workspace_name and main_workspace are made required the generated signature is as follows.
pub fn create(self, main_workspace: super::workspace::UniqueWhereParam, _params: Vec<SetParam>) -> Create<'a>

Result
In both cases, there is no way to set the required field of name without using the vec![Set] arguments but this throws validation errors.
Workaround
The current workaround for anyone running into this issue is to
create
let project = prisma::project::create_unchecked(...);
let created = client.project().create_many(vec![project]).exec().await?;
let project = client.project().find_unque(....).exec().await?;

note: this will only work for people who have a way to create a unique query from the data they used in the
Discord chat record

This should be fixed on rev = "f8ec79ad532b0bbca0d90dcaa6ecc895c82b87c3"

Out of curiosity I plugged this schema into prisma client JS and I got this

Turns out that the JS client would actually give you the same problem if you wanted to use the checked version. I'm not sure if that's a bug or a feature, but I'd like to keep consistency.
I think I'll add a create_unchecked method that's similar to the create_many you're doing.


enso-8194 Copy and pasting a single node

Pull Request Description
Implements  #8194
Peek 2023-11-13 14-15.webm
Checklist
Please ensure that the following checklist has been satisfied before submitting the PR:

[x] The documentation has been updated, if necessary.
[x] Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
[x] All code follows the
Scala,
Java,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
All code has been tested:

[x] Unit tests have been written where possible.
[x] If GUI codebase was changed, the GUI was tested when built using ./run ide build.

Please note that this implementation stores nodes in a custom JSON-based format with text/plain MIME-type, which means any external program will receive a bunch of unnecessary metadata when pasting.
The implementation in the old GUI was using two payloads:

text/plain contains only the code. It is used when pasting in other programs.
web application/enso containing our custom format with metadata.

This was a feature specifically requested on the demo.
Also, we’re not handling node positions here – the node is always placed under the cursor, even if another node is already there. We don‘t have the necessary API for this issue yet, but it must be remembered.

@MichaelMauderer just a heads up, this is ready to merge (all required checks are passing too) but there are new merge conflicts


Adds a CPU check test for Windows

Motivation
Only the Linux CPU check was tested.

Superseded by #8097


Why _ngcontent-c* are always counting up?

Every tag in the SSR page are contains  _ngcontent-c* attributes in the HTML.  I know what is this for, but I'm really curious to know why this attributes are always counting up when I hit Refresh of the page.

This is just part of the compiler and how it handles everything. These are from ViewEncapsulation Emulation, we just want to make sure each request has different ones so that the browser doesn't some cache/display anything improperly! Hope that helps 👍
Nothing to worry about it's all normal :)


tab completion relies too much on type annotation, doesn't show fields of actual object

Describe the bug
I have a variable, and used tab completion, and it did not show the fields of the variable. It only shows the fields of the return type of the function the variable came from. I have an actual object, I want to see it's fields and functions, not just what it can know from type annotations.
Environment
{
"marimo": "0.7.17",
"OS": "Windows",
"OS Version": "10",
"Processor": "Intel64 Family 6 Model 85 Stepping 4, GenuineIntel",
"Python Version": "3.10.11",
"Binaries": {
"Browser": "126.0.6478.127",
"Node": "--"
},
"Requirements": {
"click": "8.1.7",
"importlib-resources": "missing",
"jedi": "0.19.0",
"markdown": "3.6",
"pymdown-extensions": "10.8.1",
"pygments": "2.16.1",
"tomlkit": "0.12.5",
"uvicorn": "0.30.1",
"starlette": "0.37.2",
"websockets": "12.0",
"typing-extensions": "4.7.1",
"ruff": "0.5.5"
}
}
Code to reproduce
First if I just create the object without using type annotation, I can see both fields with tab completion.

Then if I return the same exact object via a type annotated function, I can only see the fields known via the type annotation. It feels like I'm being punished for trying to adopt type annotation by having less access to runtime information.

I believe the behavior makes sense. If return_b claims to return an instance of A, code shouldn't assume it has access to fields of B; b2.b won't pass type-checking of the type checker thinks b2 is of type A. So the type annotated code is stricter, because presumably you're adopting type annotations because you want to enforce better coding practices.

I understand that argument, it does make sense. But marmio multiple uses, not only the best coding practices uses. It's also basically a debugger in a lot of my usage and there I want the runtime information wtihout the need to edit the code I'm debugging. Would you consider a toggle in the settings to change this behavior?
Or at least make it respect a function like
def reveal(x):
    return x

to strip type annotation for debugging.
See reveal not working here

Okay, I understand your perspective, thank you for explaining.
I would indeed consider that. The only thing is that we currently use Jedi for completion and mean to migrate off it, since it's not actively developed. So I'm hesitant to add new features, since it could make migration more difficult.

Closing this in favor of #1963


Autobuyer only searches once

I have tried uninstalling it and reinstalling older versions
the autobuyer only runs once when i click search, and then does not search again or update at all unless i manually hit stop and search again
any ideas on how to fix this?
thank you in advance

Wait time should be a range value like 3-8 , if you give single value like 3 or 8 , the autobuyer will stop


shiboken 6.3.2 seem to break CI.

🧰 Task
CI seem to be breaking on 3.9+pyqt6 since https://pypi.org/project/shiboken6/#history 6.3.2 was released about 2h ago.
I'm unsure the best way to set a constraints without making shiboken a requirement.

Closed by #5073


Action ignores configuration cache if Gradle User home exists

Action will ignore saving/restoring configuration cache from the projects .gradle directory if it detects that Gradle User home exists on the host.
Logs from the gradle-build-action setup step:
Run gradle/gradle-build-action@v2
  with:
    cache-read-only: false
    gradle-version: wrapper
    cache-disabled: false
    cache-write-only: false
    gradle-home-cache-includes: caches
  notifications

    generate-job-summary: true
    gradle-home-cache-strict-match: false
    workflow-job-context: null
Gradle User Home already exists: will not restore from cache.

Following task execution:
Run ./gradlew ktLintCheck
Starting a Gradle Daemon, 1 busy and 1 stopped Daemons could not be reused, use --status for details
Configuration cache is an incubating feature.
Configuration on demand is an incubating feature.
Calculating task graph as no configuration cache is available for tasks: ktLintCheck

The desired outcome would be to restore configuration cache independently of restoring GRADLE_HOME state

I was thinking about the second one and I don't think there is a reasonable way to do this without knowing exactly which files configuration cache would require to be present in Gradle Home.
Do you have any additional resources on this specific topic, I would happily do a small bit of research because I believe there has to be some kind of semantic connection between files in config and build caches?

Since the configuration-cache can contain sensitive information, the gradle-build-action no longer attempts to save and restore this information in the GitHub Actions cache: https://github.com/gradle/gradle-build-action/security/advisories/GHSA-h3qr-39j9-4r5v.
As such, this issue is no longer relevant, since the behaviour is the same for self-hosted and github-hosted runners.


NTR: Ammonium thiosulfate

Could you please add this term.
It is used as a fertilizer in agriculture.
Many thanks!

An entry has now been created for ammonium thiosulfate (CHEBI:147402). The entry will become searchable on the ChEBI website by tomorrow.


App crash while month view change to week or day

Hello,
I am trying to changing month view from month to day or week but in meantime I got error and app is going to crash. I passed all props like startAccessor='start' endAccessor='end' or startAccessor='startDate' endAccessor='endDate' but I don't achieve my goal . When I try to pass props startAccessor='startDate' endAccessor='endDate' app is not going to crash but events are not showing to it related dates. please help me thanks

It would be helpful to get a reproduceable example on codesandbox or to see your code.

We had crash issues switching between month/day views running v0.20.2.  We were able to fix by updating to v0.20.4.

I had the same problem. In order to fix it, you have to make sure that both start and end (or startDate and `endDate``) are date objects, not strings., something like this:
this.events = (result.data || []).map(item => {
  item['start']  = item['start'] && moment(item['start_at']).toDate()
  item['end'] = item['end'] && moment(item['finish_at']).toDate()
  return item
})


Update instead of insert when user close keyboard

I created a chat using XMPP,  and Realm for storing messages.
When the user press the "send message" button, I write an object on realm and  I receive an insert notification, so I can update the tableview UI.
But I have this problem: if the user close the keyboard by RETURN key, and then tap on "send message" button, Realm notify me an update instead of an insert and this cause me an NSInternalInconsistencyException.
Recap:

User press "send" with keyboard visible -> All works right!
User press "send" after closing keyboard -> NSInternalInconsistencyException caused by update instead insert notification

@carmelo-ruota Can you show me your code? I do not know what is different in each cases.

@carmelo-ruota Please fill out the template we provide when you open an issue, so we can better help.

Hi @carmelo-ruota!
I'm sorry, but we need a lot more information from you before we can help.
If you're still having trouble, can you please re-open this issue, go back and properly fill out the template we provided with all of the information requested in it?
Additionally, if you can show us your code responsible for receiving the callback events from the keyboard and performing the Realm write transaction, that would help a lot.
Thanks! Have a happy holiday!


Live

Test PR

Docs Build status updates of commit f7fcc3a:
:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Create a more standard training loop interface for pretraining

Currently, clmbr_train_model conceals more familiar training loops structure from users. In most demos and APIs, the boilerplate looks like what's outlined here https://github.com/PyTorchLightning/pytorch-lightning with this structure
dataset = MNIST(os.getcwd(), download=True, transform=transforms.ToTensor())
train, val = random_split(dataset, [55000, 5000])

autoencoder = LitAutoEncoder()
trainer = pl.Trainer()
trainer.fit(autoencoder, DataLoader(train), DataLoader(val))

basically the form

dataloader
data splits
model architecture
training

Specific details around the loss are configured in the model architecture and the trainer class handles stuff like progress bars, choice of optimizer, etc.
What is the lift required to provide a demo and refactor to support this type of workflow?

The refactor PR puts pre-training into this kind of API:
model = CLMBRFeaturizer(config, info)
dataset = PatientTimelineDataset(extract_path, ontology_path, info_path)
model.fit(dataset)

The original clmbr_train_model still works, just uses this API. I'll leave this issue open until piton_private is updated to reflect these changes.


AudioKit issues with implementing the app "preferences" (Settings) for the app.

macOS Version(s) Used to Build
macOS 13 Ventura
Xcode Version(s)
Xcode 13.4.1
Description
Hi,
I have made an app and finally I decided to have a "preferences" / "Settings" window for my app. After wasting hours, thinking maybe my code is broken, I figured out that AudioKit is causing that. I have created a new Xcode document without AudioKit and implemented the "Settings" there and it worked perfectly. to make sure that the problem is caused by Audiokit, I then added the AudioKit into my project, and there it was. the error shoed up again. AudioKit is using "Settings" in one of their modules which is reserved by Xcode. and that creates the conflict.. I have tested this with AudioKit version 5+ on macOS 13+. I have included a screenshot from the issue
Crash Logs, Screenshots or Other Attachments (if applicable)

If you replace Settings { with SwiftUI.Settings { does it build successfully?

Yes that did it. Thanks a million.


Unity2017.3 , aws lambda sdk  can not use long string.

when I use unity 2017.3 , I can not use long string.
it occur  exception.
Amazon.Lambda.AmazonLambdaException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)
send any long message, about 2000 charactor  ,  to lambda using unity aws sdk.
ResultText.text = "Invoking '" + FunctionNameText.text + " function in Lambda... \n";
string text = @"{""key1"":""aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"" }";
        Debug.Log("text.legnth "+ text.Length);

        Client.InvokeAsync(new Amazon.Lambda.Model.InvokeRequest()
        {
            FunctionName = "HelloLamda",
            Payload = text
        },
        (responseObject) =>
        {
            ResultText.text += "\n";
            if (responseObject.Exception == null)
            {
                ResultText.text += Encoding.ASCII.GetString(responseObject.Response.Payload.ToArray()) + "\n";
            }
            else
            {
                ResultText.text += responseObject.Exception + "\n";
            }
        }
        );

Context
Your Environment
I just used 'AWSSDK.Lambda.<IP_ADDRESS>.unitypackage'
and Unity 2017.3
.NET Core Info

.NET Core version used for development:
.NET Core version installed in the environment where application runs:
Output of dotnet --info:
Contents of project.json/project.csproj:
.net 3.5  and .net 4.6

After upgrading to Unity2017.3 and AWSSDK.Lambda.<IP_ADDRESS>.unitypackage, we also encountered a similar issue.
I can confirm that the issue is only occurring for a request with a 'heavy' payload.
Requests with smaller payloads continue to work properly as they used to...
Please let me know if I can provide any extra info that can help identify the cause of this issue...

Could this be a similar problem as issue #823 ?

@kvasukib
I understood how to avoid this error.
You can avoid errors by setting UnitWebRequest.chunkedTransfer to false.
I'd like to ask you to fix it with AWSSDK.
I responded by the following method.
UnityInitializer.AttachToGameObject(gameObject); gameObject.AddComponent<CustomUnityMainThreadDispatcher>();
CustomUnityMainThreadDispatcher.cs
https://gist.github.com/housei/b38cc93d6e8fe7caa2764d5b0dc34393

For me, it gives error quite often but not always. So what I do, is send the request repeatedly. It gives the error like 7-8 times and then succeeds. But screen waits for that long time. So it is really a show stopper for user experience. My launch is waiting for the resolution of this bug. I won't even do the alpha launch of my game without it being fixed.
So please guys.. Make it a priority...


feat: collect pg db.collection_name attribute

The db.collection.name attribute is conditionally required for Database spans. This PR uses regex to detect the collection name (called table name in PG) and report it as the db.collection.name attribute for PG instrumentation.
closes https://github.com/open-telemetry/opentelemetry-ruby-contrib/issues/1059

I think these tests need updating. I provided a suggestion for what I'm seeing and recommend the fixtures be updated to supply an expected table_name for the assert_equal to use or for the fixture queries to all query "table_name".

Thank you for the feedback and edits, Rob! Helped catch an additional update with the regex. 7bb9db0ac92325469a56682e1ce84ef6dc11db8d

💎 ❤ Awesome. ❤ 💎


ethereum-spec-lint to prevent absolute imports within forks

What was wrong?
There were no existing checks in ethereum-spec-lint for controlling the hygiene of import statements
Related to Issue #418
How was it fixed?
Added check_import to enforce rules outlined in #418
Additionally extended the existing linting such that, the linting is performed recursively. E.g:- individual modules in vm.instructions were not being checked earlier.
Cute Animal Picture

Codecov Report

Merging #446 (2e3c18c) into master (f037baf) will increase coverage by 3.83%.
The diff coverage is 94.76%.

@@            Coverage Diff             @@
##           master     #446      +/-   ##
==========================================
+ Coverage   75.53%   79.36%   +3.83%
==========================================
  Files         152      187      +35
  Lines        7942     9864    +1922
==========================================
+ Hits         5999     7829    +1830
- Misses       1943     2035      +92

Flag
Coverage Δ

unittests
79.36% <94.76%> (+3.83%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

src/ethereum/dao_fork/spec.py
0.00% <0.00%> (ø)

src/ethereum/dao_fork/vm/gas.py
0.00% <0.00%> (ø)

...rc/ethereum/dao_fork/vm/instructions/arithmetic.py
0.00% <0.00%> (ø)

src/ethereum/dao_fork/vm/instructions/system.py
0.00% <0.00%> (ø)

src/ethereum/dao_fork/vm/interpreter.py
0.00% <0.00%> (ø)

src/ethereum/frontier/vm/instructions/system.py
97.29% <33.33%> (-1.33%)
:arrow_down:

src/ethereum/homestead/vm/instructions/system.py
96.21% <33.33%> (-1.04%)
:arrow_down:

...hereum/tangerine_whistle/vm/instructions/system.py
97.97% <33.33%> (-1.00%)
:arrow_down:

src/ethereum/spurious_dragon/utils/json.py
52.94% <52.94%> (ø)

src/ethereum/spurious_dragon/spec.py
80.61% <80.61%> (ø)

... and 49 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1ed7b5d...2e3c18c. Read the comment docs.

@SamWilsn Have included the relevant updates. See the comments on the individual items above.


New agar.io version (550) disconnects agar.io client

Looks like key length has changed. It was working perfectly yesterday, they made a new version.
Here's the example.js stack trace.
$ node ./node_modules/agario-client/example.js
worker: client created
Requesting server in region EU-London
HTTP request answer: <IP_ADDRESS>:443
36XyMmbZ

Connecting to ws://<IP_ADDRESS>:443
worker: connecting...
worker: connected to server
worker: disconnected
worker: spawning
c:\projects\nodejs\agario-dinner-bot\node_modules\agario-client\node_modules\ws\lib\WebSocket.js:214
    else throw new Error('not opened');
               ^
Error: not opened
    at WebSocket.send (c:\projects\nodejs\agario-dinner-bot\node_modules\agario-client\node_modules\ws\lib\WebSocket.js:214:16)
    at Object.Client.send (c:\projects\nodejs\agario-dinner-bot\node_modules\agario-client\agario-client.js:130:17)
    at Object.Client.spawn (c:\projects\nodejs\agario-dinner-bot\node_modules\agario-client\agario-client.js:415:14)
    at Object.<anonymous> (c:\projects\nodejs\agario-dinner-bot\node_modules\agario-client\example.js:55:12)
    at Object.emit (events.js:104:17)
    at null._onTimeout (c:\projects\nodejs\agario-dinner-bot\node_modules\agario-client\agario-client.js:88:20)
    at Timer.listOnTimeout (timers.js:110:15)

Never mind, looks like the new version of agario-client fixed it. Keep up the good work #pulviscriptor!

It is fixed for now. Author of agar.io replaced buf.setUint32(1, 673720361, true); with buf.setUint32(1,<PHONE_NUMBER>, true); and this is (I think) to kill all unofficial clients.
Let's hope he will not do this every day.


Unable to use material-ui/test-utils

material-ui 1.0.0-beta.2
The script fails with the "ReferenceError: context is not defined" exception as the 'material-ui/test-utils' package is imported.
Looks like the issue has been already fixed by @oliviertassinari in 4e2cf38ae3181cf38a5796179bfb2887c402b4ac commit. But this fix was not included in the 1.0.0-beta.2, so the 'test-utils' module available in the current version is completely unusable.
Aren't you planning to release a new beta in the nearest future?

Yes, we will cut a new release shortly with bug fixes. Still have a couple of them to fix though.

same issue


Comstar

Some of the things we modified your bundle to do are:
(a) Added the ability to pass cURL options in the CurlCall constructor
(b) Option to get the headers returned from a cURL call.
(b)(1) Return headers can be stored raw or as an associative array.
(b)(2) New getters getResponseHeaderData and getResponseHeaderObject.
(b)(3) Fixed an issue where a prior cURL POST defaulted to the same method in HttpGetHtml (forced GET).
(c) HttpGetXML - store XML return data in the requestObject as a raw XML or associative array.
(d) Fixed an issue with generateRequestData method was not adding the leading '?'.
(e) Pulled the $curl->exec() function out of each class into CurlCall curlExec() method for parsing headers.

Thank you for your commit.. reviewed all changes, they all seem very good! I will merge them and release v1.2.0


Fix build error AttributeError: 'dict' object has no attribute ...

Running pip wheel ./  in the source checkout with pip>19.0 fails with a AttributeError: 'dict' object has no attribute __NUMPY_SETUP__
Depending on how the setup.py script is executed (as __main__ or not) __builtins__ in the global namespace can be a dictionary or a module (https://docs.python.org/3/library/builtins.html says it can be either and that this is an cpython implementation detail).
Fixed by an explicit import.

Hey thanks! Right now master is failing unit tests due to #203 and as mentioned in #191 so I'll wait for those to pass before merging PRs.

@ales-erjavec Thanks, merging!


[FLINK-3713] [clients, runtime] Use user code class loader when disposing savepoints

Disposing savepoints via the JobManager fails for state handles or descriptors, which contain user classes (for example custom folding state or RocksDB handles).
With this change, the user has to provide the job ID of a running job when disposing a savepoint in order to use the user code class loader of that job or provide the job JARs.
This version breaks the API as the CLI now requires either a JobID or a JAR. I think this is reasonable, because the current approach only works for a subset of the available state variants.
We can port this back for 1.0.4 and make the JobID or JAR arguments optional. What do you think?
I've tested this with a job running on RocksDB state both while the job was running and after it terminated. This was not working with the current 1.0.3 version.
Ideally, we will get rid of the whole disposal business when we make savepoints properly self-contained. I'm going to open a JIRA issue with a proposal to do so soon.

It seems very clumsy to dispose savepoints like that. If that is needed for a fix currently, then I guess we have to live with that.
Would make sense to make the jar and jobid optional, so that non-custom cases can dispose the savepoints in the simple way.
Then, we could ad the RocksDB state backend to the jar in the flink distribution lib folder, so that it is always available and would not need jar or jobid for disposal. That would give the least user friction...

Full ack, but the problem is not just RocksDB, but also savepoints, which reference a user class in the state descriptor, including FS snapshots. So if you configure a file backend and use folding or reducing state, you run into the issue as well.
What about the following: make it optional and fail with a proper error message in case of a missing user code class. SubtaskState (previously StateForTask), which fails to dispose the state snapshots currently only logs the Exception.

Good work @uce :-)
I agree that it currently it is a bit clumsy to discard savepoints of jobs which are no longer running. From the user perspective it should be as easy as possible.
I also think that it would be a good idea to add the RocksDB jar to the flink-dist.jar since all serious user are using it. Furthermore, I think it would be a good idea to not only log possible exceptions in SubtaskState but to communicate it back to the user. Thus, letting them bubble up and handling them on the JobManager level would be the right way to go.
I had some minor comments concerning test coverage and the way the job jars are constructed.

Thanks for review. I will propagate the errors, make the job ID/JAR arguments optional, and try to simplify parts of the CLI as you suggested.
Regarding including the RocksDB jar in dist, I think that should be handled as a separate issue.

I've addressed the comments:

Moved the JAR uploading to a helper:

// Retrieve blob server address and upload JARs to server
uploadJarFiles(ActorGateway, FiniteDuration, List);
// Upload JARs to server
uploadJarFiles(InetSocketAddress, List);
```
I did not remove the JobGraph upload JARs method, but it now calls these helpers. I decided against removing it, because it's called in multiple places with the same pattern (upload, get BLOB keys, set BLOB keys).

I extract the libraries as you suggested (no need for the job graph now)
Removed the job ID variant as it was overloading disposal too much and just stuck to the JAR variant, which is now optional (no script API break)
State disposal errors are now propagated and the a failed disposal with ClassNotFoundException gives a hint to provide the JAR file
I've added an automated test with custom KV state in ClassLoaderITCase

If there are no objections, I would like to merge this.


Error: Attempt to invoke virtual method 'android.content.ClipData$Item android.content.ClipData.getItemAt(int)' on a null object reference

Environment
System:
    OS: macOS 14.5
    CPU: (12) arm64 Apple M2 Pro
    Memory: 82.53 MB / 32.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 18.18.0 - ~/.nvm/versions/node/v18.18.0/bin/node
    Yarn: Not Found
    npm: 9.8.1 - ~/.nvm/versions/node/v18.18.0/bin/npm
    Watchman: 2<IP_ADDRESS> - /opt/homebrew/bin/watchman
  Managers:
    CocoaPods: 1.15.2 - /Users/tom/.rvm/gems/ruby-3.2.2/bin/pod
  SDKs:
    iOS SDK:
      Platforms: DriverKit 23.5, iOS 17.5, macOS 14.5, tvOS 17.5, visionOS 1.2, watchOS 10.5
    Android SDK: Not Found
  IDEs:
    Android Studio: 2023.1 AI-231.9392.1.2311.11076708
    Xcode: 15.4/15F31d - /usr/bin/xcodebuild
  Languages:
    Java: 11.0.23 - /opt/homebrew/opt/openjdk@11/bin/javac
  npmPackages:
    @react-native-community/cli: Not Found
    react: 18.2.0 => 18.2.0
    react-native: 0.71.19 => 0.71.19
    react-native-macos: Not Found
  npmGlobalPackages:
    *react-native*: Not Found

Platforms
Android
Versions

Android: 10 - 14
iOS: N/A
react-native-netinfo: N/A
react-native: 0.71.19
react: 18.2.0

Description
We use Sentry to record issues. All of the errors we see have been caused by the function getItemAt(0) in our case. The exact error message is in the title.
After investigating the issue it seems strange that when the function getItemAt(0) is invoked, an exception of null object reference can be thrown as there is a NULL and a getItemCount check (see below on line 6) in file ClipboardModule.java to prevent that exact scenario.
@ReactMethod
public void getString(Promise promise) {
  try {
    ClipboardManager clipboard = getClipboardService();
    ClipData clipData = clipboard.getPrimaryClip();
    if (clipData != null && clipData.getItemCount() >= 1) {
      ClipData.Item firstItem = clipboard.getPrimaryClip().getItemAt(0);
      promise.resolve("" + firstItem.getText());
    } else {
      promise.resolve("");
    }
  } catch (Exception e) {
    promise.reject(e);
  }
}

I wonder whether that issue could be prevented by using clipData instead of clipboard.getPrimaryClip() like that
if (clipData != null && clipData.getItemCount() >= 1) {
      ClipData.Item firstItem = clipData.getItemAt(0);

since clipData has been checked beforehand. But I could be wrong.
Reproducible Demo
N/A

I did see same error for sentry here.
Stack Trace
Error
Attempt to invoke virtual method 'android.content.ClipData$Item android.content.ClipData.getItemAt(int)' on a null object reference


Bridge Icon and text for start-evaluation is not formatted properly

Your issue may already be reported! Please search on the issue tracker before creating a new one.
On the left hand side of bridge, it shows text sh.keptn.event.start-evaluation and default icon and not text and right icon

Environment

Keptn 0.6.1 - Bridge. 20200402.1046

Affected Component

[ ] Docs
[ ] CLI
[ ] Kubernetes Integration
[ ] Openshift Integration
[ ] Helm
[ ] Istio
[x ] Bridge
[ ] Gatekeeper
[ ] Datastore
[ ] REST API
[ ] eventbroker / distributor
[ ] jmeter

Describe the bug

To Reproduce

Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
Shows 'start evaluation' as text and  and evaluation icon
Current behavior
Shows internal text sh.keptn.event.start-evaluation and default icon
Screenshots

Additional context

Hi Rob, one day earlier I have created an issue for exactly the same. Please see #1667
I closed this issue since it is a duplicate.


Added tropical storm Nangka for 20201013 for HK Exchange

Updated exchange_calendar_xhkg.py
Added tropical storm Nangka on 20201013 to list of adhoc closures for HK Exchange

Hi @leech74 - thanks for contributing. Can you link to a source in your PR comment? Preferably from the exchange website.

Hi @gerrymanoim, here is the link from the exchange website: https://www.hkex.com.hk/News/Market-Communications/2020/2010132news?sc_lang=en


Add slimzsh

Many people use it on daily basis :) It's really good if someone is looking for something small and simple.

Could you please:

[ ] Resolve conflicts
[ ] Drop the A in the description.

Thanks! @elharion

Hey @k4m4. Done :) If there's anything more i could do please let me know. Cheers!


Is the release pipeline broken?

These two files:
http://purl.obolibrary.org/obo/go/snapshot/extensions/go-plus.json
http://purl.obolibrary.org/obo/go/snapshot/extensions/go-plus.owl
are both showing a version IRI of:
http://purl.obolibrary.org/obo/go/releases/2018-02-20/extensions/go-plus.owl
go-plus.owl has been updated several times since 20 Feb, so has something gone wrong with the release pipeline?

Is there an alternative location where I can find an up-to-date version of go-plus.json?

There've been no Jenkins reports for a few days but they've come through this morning.

The version of go-plus.json that we grabbed at about 3am this morning still had the 2018-02-20 timestamp.
I'll try grabbing it again now to see whether it's been updated.

Nope, http://purl.obolibrary.org/obo/go/snapshot/extensions/go-plus.json is still stuck at 2018-02-20

@cmungall Any word on what's happening here?

@tonysawfordebi very sorry about the lack of response
I can try and track down what was going on, but either way it works now:
  "version" : "http://purl.obolibrary.org/obo/go/releases/2018-08-28/extensions/go-plus.owl"


HLint ignores deriving inside family instances

Working example:
{-# LANGUAGE GeneralizedNewtypeDeriving, TypeFamilies #-}
module Main where

import Control.Applicative

class Class x where
    data MyIO x :: * -> *

instance Class Int where
    newtype MyIO Int a = MyIO { unIO :: IO a }
        deriving (Functor, Applicative, Monad)

hello :: MyIO Int ()
hello = do
    MyIO $ putStrLn "Hello"
    MyIO $ putStrLn "World"

main :: IO ()
main = unIO hello

hlint-1.9.27 outputs:
Warning: Unused LANGUAGE pragma
Found:
  {-# LANGUAGE GeneralizedNewtypeDeriving, TypeFamilies #-}
Why not:
  {-# LANGUAGE TypeFamilies #-}

1 hint

Thanks for the test case, all fixed now. I'll make a new release tomorrow.

Released in hlint 1.9.28.


Newest code not published to npm

The updated code, #9 has not been published to npm.

Just published it, sorry about that.


host is required but specified

ftpgrab freebsd 64bit binary from the 5.1.0 release. Two issues.
First I get this even though host is defined in sftp.
Fri, 15 Feb 2019 00:23:38 UTC FTL Improper configuration error="host is required"
ftpgrab --version reprots "dev" instead of the actual version number.

@gregf Can you post your config ull config file please ?

server:
  type: sftp
  sftp:
    host: example.com
    port: 22
    username: gregf
    password:
    key: /home/gregf/.ssh/id_ed25519
    sources: /home/gregf/downloads/tv
    timeout: 30
    max_packet_size: 32768
db:
  enable: true
  path: /home/gregf/.ftpgrab/tv.db

download:
  output: /mnt/media/tv/unsorted
  uid: 1001
  gid: 1001
  chmod_file: 0644
  chmod_dir: 0755
  include:
  exclude:
  since: 0001-01-01T00:00:00Z
  retry: 3
  hide_skipped: false
  create_basedir: false

mail:
  enable: false
  host: localhost
  port: 25
  ssl: false
  insecure_skip_verify: false
  username:
  password:
  from:
  to:

Sources is a slice :
server:
  type: sftp
  sftp:
    host: example.com
    port: 22
    username: gregf
    password:
    key: /home/gregf/.ssh/id_ed25519
    sources:
      - /home/gregf/downloads/tv
    timeout: 30
    max_packet_size: 32768

db:
  enable: true
  path: /home/gregf/.ftpgrab/tv.db

download:
  output: /mnt/media/tv/unsorted
  uid: 1001
  gid: 1001
  chmod_file: 0644
  chmod_dir: 0755
  include:
  exclude:
  since: 0001-01-01T00:00:00Z
  retry: 3
  hide_skipped: false
  create_basedir: false

mail:
  enable: false
  host: localhost
  port: 25
  ssl: false
  insecure_skip_verify: false
  username:
  password:
  from:
  to:

ftpgrab --version reprots "dev" instead of the actual version number.

@gregf That's because you probably not using the official release. Download the latest version and also check the path of ftpgrab.

@crazy-max I said I downloaded the official binary release.

What is the output of which ftpgrab ?

I'm a moron totally forgot I had a old copy installed from source when I was working on the FreeBSD port. It was using that over the copy I through in ~/bin. Feel free to close.


ssh file disappears after first boot

Hi,
I tried to enable ssh by default from a custom plugin, so I added this in my postinst script :
touch /boot/ssh

But after write image on SDCard, the file does not exist in the boot partition.
If I create it manually, it will be erased at first Pi start. I need to recreate it a second time to make SSH work.
What I'm doing wrong ? Is there another mean to enable SSH ?
Thanks,
sgallou

the pi need to boot twice for ssh to work, and it should do it on it's own, first boot creates the keys, keys aren't created cause every device needs unique keys so they can not be part of the image, once they are created the pi should reboot, and ssh should work.  I have no idea what /boot/ssh does?

Adding a /boot/ssh empty file enable SSH on headless RaspberryPi (see "3. Enable SSH on a headless Raspberry Pi" on this page).
Should SSH work without adding this file ?
I let enough time (I hope) for RaspberryPi to make all its initialization, more than 10 minutes. Must I reboot manually to pass the second boot ?


Add timeout for all ITs

Context
 java - spring - tests running IT for over 2 hours

/azp run java - spring - tests

/azp run java - spring - tests

This PR triggered test pipeline  has passed.
I manully triggered the test pipeline  with current main branch to check the differences.

Closing this, this is used to test the errors in pipeline.


so when I am trying to give my repo link in project section the contents are not displayed in webpage

so you can clearly see that I am trying to upload and those contents are not shown on the webpage is there any idea how to work around this problem.

@gokuljs Thanks for testing out mytemplate .
It looks like you have entered the GitHub URL for the repository. Try replacing it with author_slug/repo_slug
Here is an example
# Link 1
gokuljs/machine-monitoring-tool-using-python-
# Link 2
gokuljs/Node.js-Mongodb-and-Express

The output will as follows


Label and resources

Does the Angular WinJS support the resource strings framework that is part of WinJS?
I have .resjson files for multiple languages in a project and migrating from this:
data-win-control="WinJS.UI.NavBarCommand" data-win-res="{winControl: { label: '{{item.label}}'}}"
Into something like this:
<win-split-view-command label="'menuHome'"
The example above simply renders menuHome, and not the resource text value from the resjson file.
Is there a concept of "res" (resources) in Angular WinJS?

Hi @sondreb If you look at SplitViewCommand definition in angular-winjs.js, attributes that are supported are defined in api-variable; icon,label,tooltip and on-invoked.
exists("SplitViewCommand") && module.directive("winSplitViewCommand", function () {
        var api = {
            icon: BINDING_property,
            label: BINDING_property,
            tooltip: BINDING_property,
            onInvoked: BINDING_event,
        };
        return {
            restrict: "E",
            replace: true,
            scope: getScopeForAPI(api),
            template: "

",
            transclude: true,
            link: function ($scope, elements) {
                initializeControl($scope, elements[0], WinJS.UI.SplitViewCommand, api);
            }
        };
    });

Rightway/workaround is to utilize AngularJS builtin localization. Basically you set variable into scope and let AngularJS localization to do the rest.
Some helpful links about AngularJS localization :
https://docs.angularjs.org/guide/i18n
https://github.com/doshprompt/angular-localization
-XDVarpunen

Thanks!
Ended up doing the following, where the first jQuery load downloads the angular-local_{{language}}.js, then the second getJSON downloads the resources.resjson.
$.getScript(filename, function () {
    $.getJSON(filenameJson, function (data) {

        // Set the global value for our localized strings.
        window.strings = data;

        angular.element(document).ready(function () {
            angular.bootstrap(document, ["app"]);
        });

    });
});

This is a good way to load dynamic locale Angular files, it's important that those modules are fully loaded before the bootstrap of the app, or dates/numbers won't translate properly.
Then I wrote a provider and a filter that simply returns window.strings :+1:


Fix unused shared_mem argument, ensure cudaMemset has completed before task terminates

Some minor fixes to CUDA module.

@srirajpaul could you take a look at these quick fixes?


Can't launch scenes in editor. Remote Debugger: Unable to connect. Status: 3

Just did a rebuild for Windows 3.1 alpha ef7b384861d59e566badf1adf28c8b56255c912e
Getting this error message anytime I try to use F5/F6 to any scene on any project.  Will do a rebuild in the morning.  Not clear what might be the source though.

Also got a one-off error, unsure if it's connected but:
   At: drivers\unix\net_socket_posix.cpp:191```


Deeplink url error will not handled

when url that open another app like open app store url (https://apps.apple.com/in/app/facebook/id284882215). webview not able to handle error

Summary

Changelog

[CATEGORY] [TYPE] - Message
Test Plan

Thank you for sending a PR! WebView was extracted from React Native and can now be found here: https://github.com/react-native-community/react-native-webview
Please use that component and send your Pull Request to the repo there :)

@cpojer many developer are still using react-native v0.59.10 so can you merge in react-native v0.59.10 so this is also  fix in this version.


captioned somethign in 04-jgathome.md

I failed to spell something correct

I like this change
+1


'Looking for sources' empty state never times out

Describe the bug
'Looking for sources' empty state never times out
To Reproduce
Steps to reproduce the behavior:

Add random OCP provider
refresh page
'Looking for sources...' never disappears

Expected behavior
If we don't find sources after X seconds we should be timing out with an error message.
Screenshots

Desktop (please complete the following information):

OS: Fedora
Browser Chrome
Version 66

Additional context
ocp provider I added:

Are there any errors in the browser console? Or, any network requests fail?
Not sure what the timeout should be? However, we can start by showing the unexpected error page after 60 seconds?

The logic used to determine if a provider exists needed to change slightly.


Parse Error

https://puu.sh/FgzV6/2f65bd0221.bms

Thanks for opening the issue, maybe I will have a look at the weekend.

I think the parse error is because this BMS file really contains an invalid tag.
At line 382:
...
#BGA02 bga.wmv
...

According to this memo, #BGAXX should have 7 arguments, all of them should be integer, which defines the coordinate and range (cropped area) should specified BMP object display on.
Therefore this is not the fault of this parser.
Currently I'll close this issue, feel free to re-open if there has more problems.

Yeah I see , I found these bms file other BMS player/parser can't parse , too
So I will ignore them.
Thx your reply :D

I found that there is a spicial bms file which work on LR2.
but Bemusilization always keep infinite keep parsing.

https://puu.sh/FieDp/fa588d83ae.bms
http://www.dream-pro.info/~lavalse/LR2IR/search.cgi?mode=ranking&bmsid=117199

Wow it seems to be a very large file...
Marked and give it a test later when I have time.

This file does contains something which crashes the parser (well, just simply spamming illegal characters)
And I see this comment on the file:

#CAUTION スペックに自信がない、およびLR2以外では絶対にプレーしないでください。破裂、火災、故障の原因となります。

The creator of this BMS chart deliberately making it impossible to be parsed other than LR2.
It is quite challenging if I make the Bemusilization be able to parse this, it might make the parser not following the "de-facto standard" behavior or even worse...

Actually I'm sure that's real reason about endless looping.
But I hope Bemusilization can able to parse , or throw exceptions.

But I have tested with the latest branch, although it still uses some time, but it throws exceptions.


wifi-2984: tplink device mount manufacturing data and certificates

The TP-Link device mounts the production_info partition, and apnos will
access the manufacturing data and copy the certificate to its use when
it is first booted.
Signed-off-by: Arthur Su<EMAIL_ADDRESS>

Merged to pending. Thanks.


Requesting “term of use” or “privacy policy” documents.

Hi @Laeyoung
Bad news is, the Wordpress plugin team has shut my plugin down, they send me a email I quote:

A legal issue with the service you're using was brought to our attention.

https://wordpress.org/plugins/corona-virus-data/

Your plugin uses https://github.com/Laeyoung/COVID-19-API which has no published information regarding the license and terms of use for the service. It has them for the source code, which is great, but as a service they're obligated to protect you (and your users) with at the very least a terms of use or privacy policy.

We cannot reopen your plugin with this issue.

So is there any possible you can include a “term of use” or “privacy policy” in your README?
I promise the plugin will never go commercial and will always be free, I can find the ainiza.ai terms and policy goes here:
https://ainize.ai/terms
And the terms of use of “
Novel Coronavirus (COVID-19) Cases, provided by JHU CSSE” here:
https://github.com/CSSEGISandData/COVID-19/blob/master/README.md
I think you could claim your API respect the MIT license and give users freedom to use in none-commercial use including showing data on their website etc.
Thank you and best regards.
Duke Yin

Hi @yduke
I added license description and term of service at README.md.
I didn't know there was term of service of CSSEGISandData/COVID-19 until you let me know. I think the ToS seem a little bit strict as I assume ☹.

Thank you @Laeyoung
I've submit your readme url to the wordpress plugin review team, hopefully the plugin will be reopen to public soon. :)

@yduke I hope so. If you need any helps, let me know : )

[x] [[[[[@]()]()]()]()]()


feat: add ZeebeError for all Zeebe errors

Pull Request Test Coverage Report for Build<PHONE_NUMBER>5
Details

17 of 18 (94.44%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.07%) to 97.051%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pyzeebe/errors/zeebe_errors.py
13
14
92.86%

Totals

Change from base Build<PHONE_NUMBER>2:
-0.07%

Covered Lines:
1086

Relevant Lines:
1119

💛  - Coveralls


Restrict compilationQueue to library-visibility

Its type uses a generic type which is limited to library-visibility.
Eliminating compilationQueue from ResidentCompiler's public interface
makes it possible to mock in tests.

Partially addresses #17051 and #17804.


DASH doesn't work with dash.js, the problem may be the mpd file

Hello everyone,
I found that the mpeg-dash is not working. I used ffmpeg to stream a video to my own server and use nginx-rtmp-module to serve the mpeg-dash stream. However, I failed to used dash.js to display my streaming. I also test the hls and rtmp, and both of them work perfectly. I think the problem may be the mpd file, so I use this website to validate the mpd file. The validation result is below.
<svrl:failed-assert test="if (@type = ‘dynamic’ and not(@publishTime)) then false() else true()" location="/*[local-name()='MPD' and namespace-uri()='urn:mpeg:dash:schema:mpd:2011']"> <svrl:text>If MPD is of type "dynamic" publishTime shall be defined.</svrl:text> </svrl:failed-assert> <svrl:failed-assert test="if (@type=’dynamic’ and not(@id)) then false() else true()" location="/*[local-name()='MPD' and namespace-uri()='urn:mpeg:dash:schema:mpd:2011']"> <svrl:text>If the MPD type is dynamic, the id shall be present </svrl:text> </svrl:failed-assert> <svrl:failed-assert test="if (exists(parent::dash:AdaptationSet/@par) and exists(@sar) and exists(@width) and exists(@height)) then if(abs((number(tokenize(parent::dash:AdaptationSet/@par,’:')[1]) div number(tokenize(parent::dash:AdaptationSet/@par,’:')[2])) – ((number(@width) * number(tokenize(@sar,’:')[1])) div (number(@height) * number(tokenize(@sar,’:')[2])))) < 0.001) then true() else false() else if(not(parent::dash:AdaptationSet/@par) and @sar) then false() else if(exists(parent::dash:AdaptationSet/@par) and not(@sar)) then false() else true()" location="/*[local-name()='MPD' and namespace-uri()='urn:mpeg:dash:schema:mpd:2011']/*[local-name()='Period' and namespace-uri()='urn:mpeg:dash:schema:mpd:2011']/*[local-name()='AdaptationSet' and namespace-uri()='urn:mpeg:dash:schema:mpd:2011'][1]/*[local-name()='Representation' and namespace-uri()='urn:mpeg:dash:schema:mpd:2011']"> <svrl:text>If par is present in AdaptationSet, widh*sarx and height*sary shall have the same ratio.</svrl:text> </svrl:failed-assert>
I also test another mpd validator and the last error does not show up. Maybe it is the validators' problem. No matter what, there are two major errors in the mpd file.

If MPD is of type "dynamic" publishTime shall be defined.
If the MPD type is dynamic, the id shall be present.

I found some sample mpd files from dash.js github. In these correct mpd files, the MPD's type is set to "dynamic" and they do contain publishTime and id. However, the mpd file generated by nginx-rtmp-module is below and we can see there is no id and no publishTime.
<MPD     type="dynamic"     xmlns="urn:mpeg:dash:schema:mpd:2011"     availabilityStartTime="2016-05-06T00:43:33+08:00"     availabilityEndTime="2016-05-06T00:43:48+08:00"     minimumUpdatePeriod="PT5S"     minBufferTime="PT5S"     timeShiftBufferDepth="PT0H0M0.00S"     suggestedPresentationDelay="PT10S"     profiles="urn:hbbtv:dash:profile:isoff-live:2012,urn:mpeg:dash:profile:isoff-live:2011"     xmlns:xsi="http://www.w3.org/2011/XMLSchema-instance"     xsi:schemaLocation="urn:mpeg:DASH:schema:MPD:2011 DASH-MPD.xsd">
I add id and publishTime manually to the mpd file. Finally it pass the mpd validation but I haven't test it on dash.js player. I'll try to modify the ngx_rtmp_dash_module.c, adding publishTime and id, and rebuild the project. Hope it will fix the problem.

I have the same problem. What player do you recommend for yours mpd files?

I found that this module works with modified version of dash.js and with bitdash.
https://en.wikipedia.org/wiki/Dynamic_Adaptive_Streaming_over_HTTP#Servers
I tested it with bitdash... All works!

Very thank you for your information. > <
I'll close the issue. I can finally enjoy the DASH.
Thanks again!!!!

I wrote PHP which is adding those two attributes to mpd. Validators said
it's correct but players still don't work. It must be something else.

Maybe I close the issue too soon...
After thinking a while, I think this is still a problem. Since @sonntex gave a great solution, I found that the modified version of dash.js was updated two years ago. The latest version of dash.js can not play the streaming. And, @misiek08 mentions that the problem is not only in mpd...

Hi Guys, Any update on a fix for this as I've just run into exactly the same issue.
I run my generated .mpd through a couple of validators and got varyting results..
http://www.dashif.org/conformance.html - validator works if you add a publishTime attribute the the MPD when using dynamic.
http://www-itec.uni-klu.ac.at/dash/?page_id=605 - is giving me the issue mentioned in the original post.


[QUESTION] Kubelet logging "transport is closing"

Question
I am trying to understand why these messages pop up every 1-2 minutes in my Kubelet log:
Oct 06 10:48:24 vmx2 kubelet[3248860]: I1006 10:48:24.507519 3248860 clientconn.go:106] scheme "" not registered, fallback to default scheme
Oct 06 10:48:24 vmx2 kubelet[3248860]: I1006 10:48:24.507681 3248860 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/lib/kubelet/plugins/driver.longhorn.io/csi.sock  <nil> 0 <nil>}] <nil> <nil>}
Oct 06 10:48:24 vmx2 kubelet[3248860]: I1006 10:48:24.507728 3248860 clientconn.go:948] ClientConn switching balancer to "pick_first"
Oct 06 10:48:24 vmx2 kubelet[3248860]: I1006 10:48:24.507877 3248860 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
Oct 06 10:48:24 vmx2 kubelet[3248860]: I1006 10:48:24.507989 3248860 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0022f60b0, {CONNECTING <nil>}
Oct 06 10:48:24 vmx2 kubelet[3248860]: I1006 10:48:24.509026 3248860 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0022f60b0, {READY <nil>}
Oct 06 10:48:24 vmx2 kubelet[3248860]: I1006 10:48:24.510935 3248860 clientconn.go:106] parsed scheme: ""
Oct 06 10:48:24 vmx2 kubelet[3248860]: I1006 10:48:24.510985 3248860 clientconn.go:106] scheme "" not registered, fallback to default scheme
Oct 06 10:48:24 vmx2 kubelet[3248860]: I1006 10:48:24.511042 3248860 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Oct 06 10:48:24 vmx2 kubelet[3248860]: I1006 10:48:24.511053 3248860 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/lib/kubelet/plugins/driver.longhorn.io/csi.sock  <nil> 0 <nil>}] <nil> <nil>}
Oct 06 10:48:24 vmx2 kubelet[3248860]: I1006 10:48:24.511187 3248860 clientconn.go:948] ClientConn switching balancer to "pick_first"
Oct 06 10:48:24 vmx2 kubelet[3248860]: I1006 10:48:24.511357 3248860 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
Oct 06 10:48:24 vmx2 kubelet[3248860]: I1006 10:48:24.511397 3248860 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0022f6280, {CONNECTING <nil>}
Oct 06 10:48:24 vmx2 kubelet[3248860]: I1006 10:48:24.512026 3248860 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0022f6280, {READY <nil>}
Oct 06 10:48:24 vmx2 kubelet[3248860]: I1006 10:48:24.517694 3248860 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"

I am not 100% sure if they are all related to the Longhorn, but always appear at the same time as the line with ccResolverWrapper: sending .... driver.longhorn.io/csi.sock. Is this something to be concerned about at all? Longhorn seems to be running correctly and I was unable to find further error logs in instance-managers or longhorn-managers.
Any help where to look next will be much appreciated.
Environment:

Longhorn version: v.1.2.1
Kubernetes version: v.1.21.5
Node config

OS type and version: Ubuntu 20.04
CPU per node: 64
Memory per node: 256G
Disk type: hdd
Network bandwidth and latency between the nodes: 10Gbit

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Baremetal KubeSpray

Additional context
Add any other context about the problem here.

I just directly googled the error message controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing". It seems that this is related to ETCD & API server, or it may be just noises.


Remove unused mix dependencies.

Proposed Changes
I noticed mix.lock still had a few dependencies that were no longer being used. This shouldn't cause an impact.

Thanks for the clean-up PR, Ryan - really appreciate it! :clap: :purple_heart:


Put scipy dependency explicitly in requirements.txt

scikit-learn requires it but wonderfully does not specify it in setup.py :) Specify in ours as workaround, and someone cluebat them for https://github.com/scikit-learn/scikit-learn/blob/master/setup.py#L178

scipy==0.15.1 probably should be added to requirements.txt