Performance boost: CUB-backed _SimpleReductionKernel

Blocked by #2584. Blocked by #3253. Blocked by #3262.
## Important note
DO NOT REVIEW the code yet!
To core devs: I need a quick turnaround feedback first before investing time in polishing this PR. Please read the PR description below. The current implementation is admittedly not perfect, and there're a number of things to be addressed/fixed (see TODO) in order to make code review more comfortable.
Motivation
With the cupy.cuda.cub module we were able to speed up a few commonly encountered reduction operations (sum, min, max, etc), but it would be nice to speed up all of the reduction operations currently existing in (or to be added to) CuPy (see the list here).
However, it is very tedious to add the desired support; we've seen how error-prone it was, and how many additional patches are needed to make it work. Even if we can do these for all reduction functions one by one, the maintenance is going to be difficult. Therefore, it is preferable, if possible, to build on top of the existing infrastructure for code generation in runtime.
What is being done
This PR achieves the goal by providing an alternative reduction kernel implementation based on cub::BlockReduce. All reduction functions returned by create_reduction_func, which is in turn based on _SimpleReductionKernel, will be benefited if the reduced axes are contiguous (can be in either C or F order). The net effect is as if cub::DeviceReduce or cub::DeviceSegmentedReduce were used, depending on whether it's a full or partial reduction.
What is not included

Generalization to arbitrary ReductionKernel: I think it'll have to wait until this PR is merged, as I don't understand a few things there. I will likely need to discuss with the core devs offline.
Support NVRTC: currently this PR requires using nvcc (see below) due to NVlabs/cub#131, but I think as suggested there this issue can be circumvented by using jitify.
Reduction over non-contiguous arrays: I don't think it's possible with CUB...

Implementation detail & limitations
Will add this later. See https://github.com/cupy/cupy/pull/3244#issuecomment-608097646.
Performance benchmarks
Using this script in which all reduction functions in CuPy (searched via grep create_reduction_func) are tested and benchmarked, I see that in most cases, especially a full reduction, a significant boost is achieved.
For CUDA 9.2 + P100, this is what I got.
For CUDA 10.0 + GTX 2080 Ti, this is what I got.
(UPDATE): https://github.com/cupy/cupy/pull/3244#issuecomment-639332979
TODO

[x] Merge with the current master: The conflict is due to #2920, which I need more time to understand, so I won't proceed further unless I get a green light from the core devs.
[x] Merge cupy/core/_cub_simple_reduction.py back to cupy/core/_reduction.pyx. (I added the kernel in a .py file for faster dev & test cycles.)
[ ] Discuss how to enable/disable this feature to make it easier to both users and our CIs. Toggling on/off this feature is currently done by the new flag cupy.core.cub_block_reduction_enabled, but frankly I am not a fan of it.
[ ] Analyze why in certain cases CUB does not win
[x] Provide some architecture tuning parameters such as ITEMS_PER_THREAD, BLOCK_SIZE, etc. This is how it's done in CUB, but perhaps we can defer this to the next PR.
[x] Clean up all the mess and polish the code.
[x] (Optional) Force-push to clean up the commit history (need to delete 3e8eb62)

@emcastillo Sorry for my slow addition...Here it is.
Implementation detail & limitations
I hijack _AbstractReductionKernel._call() and split the code path into two. If the conditions for using CUB is met, I then pass most of the generated strings (pre_map_expr, post_map_expr, reduce_type, etc) to the string template of the CUB-based kernel for runtime code generation.
(Some strings like params are totally different from the current implementation, see below.)
To compile the code, note that NVRTC does not support CUB natively (NVlabs/cub#131), but fortunately we can call the NVCC backend. Will address this issue in a future PR using jitify.
Because we only invoke the CUB kernel for reductions over contiguous axes, it allows a number of simplifications. First, the stride sizes are no longer needed, and instead, for partial reduction like (a, b, c) -> (a,), we just need to know the “segment” size = b * c (following the terminology in CUB). I decide to pass in this info as a kernel argument, so that the generated kernel can be reused for different segment sizes. This also means we don’t need CArray and CIndexer at all, so I just pass the pointers to the input and output arrays to the kernel. In this case, each block handles one segment, so the grid size is simply a.
Second, for full reduction like (a, b, c) -> (), the performance of the above “one-pass” strategy is very bad, as only one block is used! So, in this case we need a “two-pass” reduction -- in the first pass, we assume a fair share among certain number of blocks, say N, so that each block handles a segment of size (a * b * c) / N. Then, in the second pass we invoke 1 block of N threads to handle the single remaining segment and get the final result.
The most challenging part is to allocate a work space to hold the intermediate result between the two passes. Long story in short: we don’t know in advance what’s sizeof(reduce_type). Currently I try to be on the safe side and allocate 32 * N bytes for this purpose. This seems to work for all reductions in CuPy. Anyway, a number of C macros are defined for this purpose such that the string template can handle all cases (one-pass, the first pass, and the second pass).
In any case, items per thread and block size are two compile-time constants in order to specialize a CUB kernel. In a future PR they should be tuned for different architectures, following more or less what CUB is doing for device policies. Also, I realized the strategies taken here are more or less following the internal implementations of cub::DeviceReduce and cub::DeviceSegmentedReduce. This is why I said

The net effect is as if cub::DeviceReduce or cub::DeviceSegmentedReduce were used, depending on whether it’s a full or partial reduction.

One thing I’m considering: there’re some hacks done to make mean() work, which requires to know the input and output array sizes. Currently I hard-coded them in the generated kernel, but perhaps passing them to the kernel as arguments can further increase the kernel reusability.

Related to #2796

Documenting progress for myself: Fixed a few memory corruptions, and I hope this is the only one left. Will resume when I have time...
import cupy as cp
import numpy as np

def test_sum_all2(dtype):
    a_np = cp.testing.shaped_arange((20, 30, 40), np, dtype=dtype)
    out_np= a_np.sum()

    a_cp = cp.testing.shaped_arange((20, 30, 40), cp, dtype=dtype)
    out_cp= a_cp.sum()
    print(dtype, out_np, out_cp, cp.allclose(out_np, out_cp))

test_sum_all2(np.float64)
test_sum_all2(np.float32)
test_sum_all2(np.float16)
test_sum_all2(np.int8)    # <--- buggy, why?

The reason I believe it's also a memory corruption is that if I just run the last line (test_sum_all2(np.int8)) without others, the result is correct.

The reason I believe it's also a memory corruption is that if I just run the last line (test_sum_all2(np.int8)) without others, the result is correct.

Fixed in 6885655!

All tests in tests/cupy_tests passed! I will proceed to clean up/refactor/support Optuna.

@emcastillo I am going to make this PR unblocked by #2584 so that it can be queued for review sooner.

@emcastillo This is ready for review! a34c060 make it decoupled from #2584, so nothing blocks this PR now. Two env vars are used:

CUPY_CUB_PATH is respected at runtime to discover the CUB location.
CUPY_CUB_BLOCK_REDUCTION_DISABLED (analogous to CUB_DISABLED) to toggle this feature on and off (default is off). This can be used in your CI.

Below I give a performance benchmark script and result. The results with Optuna turned on are labeled with opt. General remarks:

Without Optuna: CUB block reduction (this PR) generally performs much better than the CuPy reduction, occasionally even better than CUB device reduction (from cupy.cuda.cub) if it's available (most of time CUB block is slightly slower than CUB device).
With Optuna: CUB block reduction can (but not always) be further optimized, so at least it's safe to claim the optimized context implementation is working, and the optimized CUB block reduction always wins the optimized CuPy reduction.

script: (click me)
import sys
import warnings
warnings.filterwarnings("ignore", category=FutureWarning)

import numpy as np
import cupy as cp
from cupyx.time import repeat
from cupyx import optimizing

shape = (256, 512, 512)
dtype = cp.float32

a = cp.random.random(shape).astype(dtype)
a_np = cp.asnumpy(a)
CUB_supported = ['sum', 'prod', 'min', 'max', 'argmin', 'argmax']
REST = ['amin', 'amax', 'nanmin', 'nanmax', 'nanargmin', 'nanargmax',
        'mean', 'nanmean',
        'var', 'nanvar', 'nansum', 'nanprod',
        'all', 'any', 'count_nonzero']

for reduce_func in CUB_supported + REST:
    for axis in [(2,), (1, 2), (0, 1, 2)]: # [(0,1,2)]:
        print("testing", reduce_func, "with axis = ", axis, '...')
        func = getattr(cp, reduce_func)

        # get numpy answer for comparison
        if reduce_func not in ('argmin', 'argmax', 'nanargmin', 'nanargmax'):
            ans = getattr(np, reduce_func)(a_np, axis)
        elif len(axis) == 1:
            ans = getattr(np, reduce_func)(a_np, axis[0])
        else:
            ans = None

        cp.cuda.cub_enabled = False
        cp.core.cub_block_reduction_enabled = False
        data = repeat(func, (a, axis), n_repeat=100)
        results = [data._to_str_per_item('GPU', data.gpu_times), data._to_str_per_item('CPU', data.cpu_times)]
        print('{:<15s} (old kernel):{}'.format(reduce_func, ' '.join(results)))
        b = func(a, axis)

        cp.cuda.cub_enabled = False
        cp.core.cub_block_reduction_enabled = False
        with optimizing.optimize(key=None):
            data = repeat(func, (a, axis), n_repeat=100)
            results = [data._to_str_per_item('GPU', data.gpu_times), data._to_str_per_item('CPU', data.cpu_times)]
        print('{:<11s} (old opt kernel):{}'.format(reduce_func, ' '.join(results)))
        f = func(a, axis)

        if reduce_func in CUB_supported:
            cp.cuda.cub_enabled = True
            cp.core.cub_block_reduction_enabled = False
            data = repeat(func, (a, axis), n_repeat=100)
            results = [data._to_str_per_item('GPU', data.gpu_times), data._to_str_per_item('CPU', data.cpu_times)]
            print('{:<15s} (CUB device):{}'.format(reduce_func, ' '.join(results)))
            c = func(a, axis)
        else:
            print('{:<15s} (CUB device):{}'.format(reduce_func, '    (CUB device-wide reduction not available)'))
            c = None

        cp.cuda.cub_enabled = False
        cp.core.cub_block_reduction_enabled = True
        data = repeat(func, (a, axis), n_repeat=100)
        results = [data._to_str_per_item('GPU', data.gpu_times), data._to_str_per_item('CPU', data.cpu_times)]
        print('{:<15s} (CUB blocks):{}'.format(reduce_func, ' '.join(results)))
        d = func(a, axis)

        cp.cuda.cub_enabled = False
        cp.core.cub_block_reduction_enabled = True
        with optimizing.optimize(key=None):
            data = repeat(func, (a, axis), n_repeat=100)
            e = func(a, axis)
        results = [data._to_str_per_item('GPU', data.gpu_times), data._to_str_per_item('CPU', data.cpu_times)]
        print('{:<11s} (CUB opt blocks):{}'.format(reduce_func, ' '.join(results)))

        try:
            cp.cuda.cub_enabled = False
            cp.core.cub_block_reduction_enabled = False
            if ans is not None:
                assert cp.allclose(ans, b, rtol=1E-4)
                if c is not None:
                    assert cp.allclose(ans, c, rtol=1E-4)
                assert cp.allclose(ans, d, rtol=1E-4)
                assert cp.allclose(ans, e, rtol=1E-4)
                assert cp.allclose(ans, f, rtol=1E-4)
        except AssertionError:
            print("Result not match! (function: {}, axis: {})".format(reduce_func, axis), file=sys.stderr)
            print(ans, b, d, e)
            raise
        finally:
            print()

result: (click me)
bench_test_cub_reduction_kernel5-3.txt

I think the existing test suite should provide enough coverage, but if you wish I can try adding a few simple tests targeting this feature.

I would specialize the _AbstractKernel into _CubSimpleAbstractReductionKernel so that we have a separation of concerns

I would specialize the _AbstractKernel into _CubSimpleAbstractReductionKernel so that we have a separation of concerns

That would make the code cleaner indeed, but I don't see how it could be done given the goal is to hijack create_reduction_func(). We need a lot of preparation (and code petition) before we can decide if CUB is usable or not. Any thoughts?

Yeah, I see that, I wonder if it would be ok moving some of it around using a Factory pattern.
The Factory just does all that common preparation and then spits out a CUB or an old style kernel

The Factory just does all that common preparation and then spits out a CUB or an old style kernel

Isn't _AbstractReductionKernel already the best factory for us? 😇
Actually, I just recalled that in the very beginning I wanted to support ReductionKernel too, so I thought the base class is the best place to modify. But then I had trouble in understanding ReductionKernel (for example, why it could take multiple inputs and outputs? Which input is reduced to which output? How does it work with @cupy.fuse? etc), so I put that ambitious plan on hold and focused on _SimpleReductionKernel. So I'd think any decision made here should have this possible extension in mind.

How does it work with @cupy.fuse

This might be actually an issue here

Right, I worried about that too, but luckily so far no problem (I ran the entire test suite locally several times). I guess it's because @cupy.fuse uses ReductionKernel instead of _SimpleReductionKernel, so it didn't even bother to try CUB (I set try_use_cub=False in the former).

Hi @asi1024, with #3410 and the following patch I still see the traceback associated with cupy.cuda.driver.CUDADriverError being printed out, but the optimization continues without disruption. Is this expected?
diff --git a/cupy/core/_reduction.pyx b/cupy/core/_reduction.pyx
index e932f2e74..c20612398 100644
--- a/cupy/core/_reduction.pyx
+++ b/cupy/core/_reduction.pyx
@@ -933,6 +933,8 @@ cdef class _AbstractReductionKernel:
             self, optimize_config, in_args, out_args, in_shape, out_shape,
             type_map, map_expr, reduce_expr, post_map_expr, reduce_type,
             stream, full_reduction, out_block_num, contiguous_size, params):
+        from cupy.cuda.driver import CUDADriverError
+
         out_size = internal.prod(out_shape)
         in_args = [_optimizer_copy_arg(a) for a in in_args]
         out_args = [_optimizer_copy_arg(a) for a in out_args]
@@ -951,7 +953,7 @@ cdef class _AbstractReductionKernel:
                 post_map_expr, reduce_type, stream, params, cub_params)

         def suggest_func(trial):
-            block_size_log = trial.suggest_int('block_size_log', 5, 9)
+            block_size_log = trial.suggest_int('block_size_log', 5, 10)
             block_size = 2 ** block_size_log
             items_per_thread = trial.suggest_int(
                 'items_per_thread', 2, 32, step=2)
@@ -962,10 +964,10 @@ cdef class _AbstractReductionKernel:
         optimize_impl = optimize_config.optimize_impl
         best = optimize_impl(
             optimize_config, target_func, suggest_func,
-            default_best={
-                'block_size_log': default_block_size_log,
-                'items_per_thread': default_items_per_thread,
-            })
+            default_best={'block_size_log': default_block_size_log,
+                          'items_per_thread': default_items_per_thread,},
+            # CUDA_ERROR_LAUNCH_OUT_OF_RESOURCES might happen
+            ignore_error=(CUDADriverError,))

         return best.params['items_per_thread'], best.user_attrs['block_size']

Hmm, it's unexpected behavior. Could you show me a reproducer?

I simply set up this PR with the above patch, set the two env vars in https://github.com/cupy/cupy/pull/3244#issuecomment-639332979, and then run the script in the comment. Not sure if there's a better, shorter, and deterministic reproducer I can provide. Perhaps I can rerun and post the full output log?

@leofang I agree with the support of CUB-backed _SimpleReduction to unify some internal implementations (e.g., allocations of output memory, decisions of dtypes).
However, I don't want to heavily mix the implementation for cub-support in _reduciton.pyx. We will also support cuTENSOR or other external libraries in reduction functions in the future, so I want to keep extensibility for multi-backends.
Could you split the implementations into another file _cub_reduction.pyx?

Hi @asi1024

Could you split the implementations into another file _cub_reduction.pyx?

I thought about this too, after @emcastillo suggested the possibility of making a factory method. However, I really don't see how I can do this. I still think the only way to support CUB (or any other backends) is to intercept _AbstractReductionKernel._call(). Any suggestions? Perhaps I just call a function defined in _cub_reduction.pyx from _AbstractReductionKernel._call()?

I think it's OK just to call a function defined in _cud_reduction.pyx in this PR.

@asi1024 I am working on the change you suggested. Although I have not finished this migration yet, I already foresee a problem.
I was able to move all _*cub*() helper functions to _cub_reduction.pyx, but not the logic in _AbstractReductionKernel._call(). The reason is the number of lines in _call() would not decrease: I need to pass in a whole lot of arguments to _cub_reduction.pyx, get them processed, and return a bunch of parameters back to _call() for kernel launch. I think this change only makes it harder to maintain. May I suggest keeping _call() as is?

Is it not possible to add a _AbstractCUBReductionKernel that inherits _AbstractCUBReductionKernel and overrides whats needed?

Did you mean to inherit _AbstractReductionKernel and create a _SimpleCUBReductionKernel? Yes, it's always possible. The problem is we must hijack and split the code path somewhere, so the problem is three-fold:

If we do any alternative approach (other than the current implementation), where to split?
At the split point, the code would be a bit messy (due to the need to test some conditions to decide which path). Can you accept that?
Would the proposal be flexible enough that we can easily extend it later (see https://github.com/cupy/cupy/pull/3244#issuecomment-640392142).

My concern for No. 1 is that the split point may have to happen as early as create_reduction_func is called if we do this inheritance, but we won't be prepared sufficiently to make any judgement (for using CUB or not) at that point. I don't see where in the existing code path the _SimpleCUBReductionKernel can enter .
@asi1024 @emcastillo I think in the above discussions we are kinda mix these issues together. I will wait for a consensus to be reached and proceed to change.

I think that all the points that you mention make complete sense.
I will merge the PR in its current state and then try to re-factor it myself while working with @asi1024 in an extensible design.
Sorry for kept asking changes.

Jenkins, test this please

Jenkins CI test (for commit a4acb8121d9103baa8045d36c565c0409c571ad5, target branch master) succeeded!

That was quick! Thanks a lot @emcastillo! Yeah no worries, those are perfectly legit requests. I just don't have good ideas (for now). We can work together to improve the designs.
I will also in the meantime start looking into hijacking create_ufunc (don't be mad at me 🙇♂) and see if CUB helps (I hope it does!)

@leofang What is the difference between CUB_PATH and CUPY_CUB_PATH?

@asi1024 Thanks for asking, I should have remember to leave a note 😅
Currently:

CUB_PATH is used at build time to build cupy.cuda.cub
CUPY_CUB_PATH is used at runtime to support this PR

After #2584 is merged:

CUB_PATH  is used at build time to build cupy.cuda.cub, but will be deprecated
CUPY_CUB_PATH is used at build time to build cupy.cuda.cub and at runtime to support this PR

For the purpose of supporting Conda-Forge, it's best to have all package-specific env vars to be prefixed by the package name (i.e., CUPY_) whenever possible, so that we can make better distinction and reduce interference among packages.

@leofang Thanks!

Do you have some recent benchmarks with these changes Leo?

Hi John, see https://github.com/cupy/cupy/pull/3244#issuecomment-639332979 (sorry GitHub folded the long thread...) We might get some slight improvement in master, because after posting that result I also did a few cosmetic changes (loop unroll, inline, etc). If you need I can rerun the script.

Can we remove these lines for each routine now?
https://github.com/cupy/cupy/blob/57235e7b203dd3c666762fdcbc4812606a18b896/cupy/core/_routines_math.pyx#L94-L99

Not really, no. Yeah it would be great if these branches (which means all reduction routines from cupy.cuda.cub!) could be eliminated, but AFAIK they need to stay for now for a few reasons:

It seems when they are usable, they are usually the fastest, although there are exceptions;
There is a possibility that nvcc is not in PATH (say using conda as in #3403, or on some HPC computing nodes to which the full compiler toolchain is not visible) so this PR would not be usable:
https://github.com/cupy/cupy/blob/9c60736ee212b956b2eb0e5337972276ad75a631/cupy/core/_reduction.pyx#L350-L352
I would suggest to first merge #2584 and then evaluate.

Exactly, the cub reductions in some routines are special cases in cub that are (or should be) more optimized than the general reduction algorithm.

Ah, forgot to add -- perhaps it is obvious -- that the current implementation ensures the calling precedence is 1. routines from cupy.cuda.cub, 2. this PR, 3. CuPy's original reduction implementation.


Memory leak using ORC.

in this nim code
import owlkettle

viewable App:
  counter: int

method view(app: AppState): Widget =
  result = gui:
    Window:
      title = "Counter"
      defaultSize = (200, 60)

      Box(orient = OrientX, margin = 12, spacing = 20):
        Button {.expand: false.}:
          text = "+"
          style = [ButtonSuggested]
          proc clicked() =
            app.counter += 1
        Label(text = $app.counter)
        Button {.expand: false.}:
          text = "-"
          style = [ButtonSuggested]
          proc clicked()=
              app.counter -= 1

brew(gui(App()))

when compiling with:
nim c -d:release -d:lto -d:useMalloc main.nim

valgrind reports leaks
==49509== Memcheck, a memory error detector
==49509== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==49509== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==49509== Command: ./main
==49509==
==49509== Warning: noted but unhandled ioctl 0x644f with no size/direction hints.
==49509==    This could cause spurious value errors to appear.
==49509==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==49509== Warning: noted but unhandled ioctl 0x6444 with no size/direction hints.
==49509==    This could cause spurious value errors to appear.
==49509==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==49509== Invalid read of size 1
==49509==    at 0x4CF106C: ??? (in /usr/lib/x86_64-linux-gnu/libgtk-4.so.1.600.9)
==49509==    by 0x4CF820B: ??? (in /usr/lib/x86_64-linux-gnu/libgtk-4.so.1.600.9)
==49509==    by 0x4CF824C: ??? (in /usr/lib/x86_64-linux-gnu/libgtk-4.so.1.600.9)
==49509==    by 0x4CE6E74: ??? (in /usr/lib/x86_64-linux-gnu/libgtk-4.so.1.600.9)
==49509==    by 0x4CCD799: gsk_renderer_render (in /usr/lib/x86_64-linux-gnu/libgtk-4.so.1.600.9)
==49509==    by 0x4B714C1: ??? (in /usr/lib/x86_64-linux-gnu/libgtk-4.so.1.600.9)
==49509==    by 0x4B71E28: ??? (in /usr/lib/x86_64-linux-gnu/libgtk-4.so.1.600.9)
==49509==    by 0x4C4BB77: ??? (in /usr/lib/x86_64-linux-gnu/libgtk-4.so.1.600.9)
==49509==    by 0x52716FF: g_signal_emit_valist (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.7200.4)
==49509==    by 0x5271862: g_signal_emit (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.7200.4)
==49509==    by 0x4C77202: ??? (in /usr/lib/x86_64-linux-gnu/libgtk-4.so.1.600.9)
==49509==    by 0x52716FF: g_signal_emit_valist (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.7200.4)
==49509==  Address 0x6596690 is 32 bytes before a block of size 4,096 in arena "client"
==49509==
==49509==
==49509== HEAP SUMMARY:
==49509==     in use at exit: 4,703,508 bytes in 20,876 blocks
==49509==   total heap usage: 296,643 allocs, 275,767 frees, 69,116,495 bytes allocated
==49509==
==49509== LEAK SUMMARY:
==49509==    definitely lost: 3,240 bytes in 33 blocks
==49509==    indirectly lost: 19,232 bytes in 405 blocks
==49509==      possibly lost: 12,644 bytes in 85 blocks
==49509==    still reachable: 4,589,016 bytes in 19,650 blocks
==49509==         suppressed: 0 bytes in 0 blocks
==49509== Rerun with --leak-check=full to see details of leaked memory
==49509==
==49509== For lists of detected and suppressed errors, rerun with: -s
==49509== ERROR SUMMARY: 16 errors from 1 contexts (suppressed: 0 from 0)

I am however aware that there are currently some cases where we don't properly deal with memory deallocation in owlkettle. E.g. Stylesheet does currently not have a destructor & at least according to the compiler warnings the =destroy hooks got changed in Nim 2.0, so those would need to be updated.

I am however aware that there are currently some cases where we don't properly deal with memory deallocation in owlkettle. E.g. Stylesheet does currently not have a destructor & at least according to the compiler warnings the =destroy hooks got changed in Nim 2.0, so those would need to be updated.

Bro fix it

Sorry but this comment just seems kind of rude. I work on owlkettle in my limited free time and therefore need to choose which areas to focus on. Since the change to the destroy hooks only came out recently, I did neither have the time nor the interest to work on this issue. However you are free to work on updating the =destroy hooks yourself. It should be quite easy to do since the Nim compiler literally points them out to you.

@can-lehmann mind keeping the issue open?
Despite Faisal's behaviour it makes sense to have this be an open issue for somebody to work on.
I have not done any work so far with destroy hooks in general so when I can find some time this might be something I'd dig into.
Or would you rather have a dedicated separate issue for that?

I decided to create dedicated issues for the individual problems in order to be able to easily track progress on them:

https://github.com/can-lehmann/owlkettle/issues/76
https://github.com/can-lehmann/owlkettle/issues/75


Enabling SSL in LibCurl on Raspberry / Ubuntu22.04

I have a single question. Is there a way to enable ssl in libcurl in any way?
Any help is really welcome!
marcell@raspberry:~$ curl -v ssl://pecl.php.net:443
* Protocol "ssl" not supported or disabled in libcurl
* Closing connection -1
curl: (1) Protocol "ssl" not supported or disabled in libcurl

OpenSSL Version: OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)
CURL Version: curl 7.81.0 (aarch64-unknown-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.13
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd

This is a question for the curl folks rather than OpenSSL.

Should the command line not be curl -v https://pecl.php.net:443 ?
From curl's perspective, the "protocol" would be https (as opposed to http).


HTML5 Instead Of Adobe Flash

Hi,
I love your awesome desktop player! Thank you very much <3
Is it possible to use HTML5 instead of Adobe Flash?

Since it uses mp3 and html5 requires to have license to play that audio format this is not possible at the moment. Google Chrome has it license, but let's hope Google would switch to an open source audio format in a near future. I'm also reading python's unofficial music API for Play Google, but I don't promise anything.


Peak Element in an Array

Problem Statement:
Given an array of n elements, find the position of the element which is greater than or equal to both of its neighboring elements.
Notes:

First and last elements have only one neighbor.
It is guaranteed that at-least one peak exists.
Do not use linear scan.
Your program should run in logarithmic complexity.

I  would be happy to contribute to this issue. @amritansh22
Thank you! :)

Yes sure @ArvindR07 assigned it to you


Cartographer & Velodyne Lidar

Hi! i'm new working with slam and acquisition im in an internship working with a VLP16, also new in ROS, so i'm sorry is this is a dumb question, i want to know if theres a way to do the mapping by acquiring the data from the velodyne packets (velodyne_pointcloud), and if there is a way, how can i do it?
I can see that you work with diferent types of topics, nodes and msgs, can i modify something to get this working with my Velodyne?
Thank You

The backpack_3d configuration is set up for consuming Velodyne data. You'll need to set up the driver and plumbing yourself though. For help with that, I suggest reaching out on http://answers.ros.org/.


Test: StatusBarItem properties id and name

Refs #74972

[x] anyOS @roblourens
[x] anyOS @weinand
[ ] anyOS @chrisdias (Optional)

Complexity: 3
Create Issue

You can now pass a id to the createStatusBarItem method for the item and set a name property dynamically.
Test:

write a simple extension that adds a StatusBarItem to the window
verify you can use this new API without enabling proposed API
the name you pick appears as name in the context menu on the status bar for your entry (see example below)
if you access id property of StatusBarItem from extension code, the value will be the extension identifier unless you specify an identifier when creating the item
if you access name property of StatusBarItem from extension code, the value will be undefined unless you specify it and once specified will match what you were assigning it to

@bpasero if the id of a StatusBarItem is not explicitely specified, accessing the property gives a undefined_publisher.statusbar, not undefined.

Let me know if this is an error and if you want me to create a bug...

Thanks I updated the test plan, for id this is expected.

Looks good, but why allow changing the name? Seems like this is something that would be static

It is a display property and all the other display properties can be changed as well. We had a API discussion about it.


Oops, needed to update the package.json

Closed because we've already gone several versions further.


L.StripePattern doesnt work with Canvas selected as renderer.

When the renderer is forced to use Casvas instead of SVG, the L.StripePattern doesnt work any more.

any one got solution for the same?

Same here :/


[TRACKER] Command line-related issues

This issue tracks bugs related to command line support in Godot.
Bugs
Command-line argument handling

[ ] https://github.com/godotengine/godot/issues/17778

Command-line debugger

[ ] https://github.com/godotengine/godot/issues/19775
[ ] https://github.com/godotengine/godot/issues/32693
[ ] https://github.com/godotengine/godot/issues/32713
[ ] https://github.com/godotengine/godot/issues/50170
[ ] https://github.com/godotengine/godot/issues/51387

Pull requests

[ ] https://github.com/godotengine/godot/pull/36252
[ ] https://github.com/godotengine/godot/pull/40627
[ ] https://github.com/godotengine/godot/pull/44594
[ ] https://github.com/godotengine/godot/pull/51028

I think it's worth mentioning #49362, although I am not sure what's going to happen to it. In general, an issue I found was that the CLI tool was not just executing a tool, but actually initializing the editor.


Site Slowness- frequently observed

After converting the sites to Modern, site slowness is observed frequently

Hey @InfosysAdmin , I'm closing this issue as it's not something related to page transformation. Hope the above answers provided you the feedback you needed.


:sparkles: adding postgres docstore

Description
Introducing a postgres implementation of the KVStore and the Docstore. This should simply the infra stack for many people as they can use postgres for all most everything.
The put_all implementation has been developed for postgres in particular to use raw statements for bulk upserts rather than the sqlalchemy ORM.
Fixes # (issue)
Type of Change
Please delete options that are not relevant.

[X] New feature (non-breaking change which adds functionality)

How Has This Been Tested?
I have developed some test cases that spin up a postgres docker container and test the functionality of the kvstore and the docstore.

[X] Added new unit/integration tests

Suggested Checklist:

[X] I have performed a self-review of my own code
[X] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added Google Colab support for the newly added notebooks.
[X] My changes generate no new warnings
[X] I have added tests that prove my fix is effective or that my feature works
[X] New and existing unit tests pass locally with my changes
[X] I ran make format; make lint to appease the lint gods

@jordanparker6 hmm, we may need to skip the postgres unit tests, or mock out the constructor to avoid the dependencies?

@jordanparker6 hmm, we may need to skip the postgres unit tests, or mock out the constructor to avoid the dependencies?

I have tried pushed a new commit so that the tests wont run if the postgres drivers are not there.
It may be worth having a new group in the poetry for integration testing (e.g. --group integration) so that more involved testing can be done as I usually catch most my bugs at the integration testing point.

@jordanparker6 fair point. In the coming weeks, each integration is going to be moved to its own pypi package (a huge effort driven by @nerdai ❤), so we can enable proper testing once that is done.

Please include the new PostgresDocumentStore class into storage/docstore/init.py


bpf: FIB fixes

Please ensure your pull request adheres to the following guidelines:

[ ] For first time contributors, read Submitting a pull request
[ ] All code is covered by unit and/or runtime tests where feasible.
[ ] All commits contain a well written commit description including a title,
description and a Fixes: #XXX line if the commit addresses a particular
GitHub issue.
[ ] If your commit description contains a Fixes: <commit-id> tag, then
please add the commit author[s] as reviewer[s] to this issue.
[ ] All commits are signed off. See the section Developer’s Certificate of Origin
[ ] Provide a title or release-note blurb suitable for the release notes.
[ ] Are you a user of Cilium? Please add yourself to the Users doc
[ ] Thanks for contributing!

Fixes: #issue-number
Fix drops with DROP_NO_FIB in the Load-Balancer reply path when using XDP or pre-5.10 kernels.

/test

/test

/test


DomainStrategy Setting Has no Effect in Freedom Outbound while Using jsonv5 Format

What version of V2Ray are you using?

Tested both 5.4.1 and 5.3.0
What's your scenario of using V2Ray?

On the server side, process v2ray uses custom dns setting from config.json to resolve domain name instead of operate system setting in /etc/resolv.conf
What problems have you encountered?

version 5.3.0
On version 5.3.0, when I tried to add "domainStrategy":"UseIP" in freedom settings, like
{
  "protocol": "freedom",
  "settings": {
    "domainStrategy": "UseIP"
  }
}

it failed to start and here's the output
V2Ray 5.3.0 (V2Fly, a community-driven edition of V2Ray.) Custom (go1.20 linux/amd64)
A unified platform for anti-censorship.
Failed to start: main/commands: failed to load config: [config.json] > infra/conf/v5cfg: unable to build config > infra/conf/v5cfg: unable to load outbound protocol config > common/registry: unable to parse json content > unknown field "domainStrategy" in v2ray.core.proxy.freedom.SimplifiedConfig

Then I tried to insepct the source code, and figured that DomainStrategy is not in proxy/freedom/config.proto, line 29, message SimplifiedConfig{} while it is shown in line 23,message Config{}.
And also DomainStrategy is not in proxy/freedom/config.pb.go, line 191, type SimplifiedConfig struct{} while it is shown in line 123, type Config struct{}.
I wonder if some changes are needed in those files.
version 5.4.1
After that, I tried to switch to version 5.4.1 because I found pull request #2334 , in which added DomainStrategy for all outbound.
Based on the useage in pull request #2334, I changed the outbound setting like
{
  "protocol": "freedom",
  "domainStrategy": "UseIP",
  "settings": {}
}

And the v2ray started successfully this time. When I created  connection through v2ray by curl, the resloving packages captured by tcpdump were sending to <IP_ADDRESS> which is the setting in /etc/resolv.conf, while they were supposed to send to  <IP_ADDRESS> which is the setting in config.json. Here's the log
V2Ray 5.4.1 (V2Fly, a community-driven edition of V2Ray.) Custom (go1.20.2 linux/amd64)
A unified platform for anti-censorship.
2023/03/31 12:23:28 [Info] app/dns: DNS: created UDP client initialized for <IP_ADDRESS>:53
2023/03/31 12:23:28 [Info] transport/internet/tcp: listening TCP on <IP_ADDRESS>:8080
2023/03/31 12:23:28 [Warning] V2Ray 5.4.1 started
2023/03/31 12:23:38 [Info] [2258904320] proxy/http: request to Method [CONNECT] Host [github.com:443] with URL [//github.com:443]
2023/03/31 12:23:38 [Warning] [2258904320] app/dispatcher: default route for tcp:github.com:443
2023/03/31 12:23:38 [Info] [2258904320] proxy/freedom: opening connection to tcp:github.com:443
2023/03/31 12:23:38 [Info] [2258904320] transport/internet/tcp: dialing TCP to tcp:github.com:443

Since I am not good at golang, I can't figure out why domainStrategy next to protocol has no effect.
Then I changed the outbound setting like
{
  "protocol": "freedom",
  "settings": {
    "domainStrategy": "UseIP"
  }
}

And it failed to start just like version 5.3.0
V2Ray 5.4.1 (V2Fly, a community-driven edition of V2Ray.) Custom (go1.20.2 linux/amd64)
A unified platform for anti-censorship.
Failed to start: main/commands: failed to load config: [config.json] > infra/conf/v5cfg: unable to build config > infra/conf/v5cfg: unable to load outbound protocol config > common/registry: unable to parse json content > unknown field "domainStrategy" in v2ray.core.proxy.freedom.SimplifiedConfig

The problem seems to be the same as version 5.3.0 this time.
Currently, I changed the function useIP() in source file proxy/freedom/config.go to force the UseIP strategy and rebuild the v2ray.
func (c *Config) useIP() bool {
	return true
}

It might cause some bug, but it's fine so far.
What's your expectation?
With format jsonv5, the process will use internal dns settings to resolve domain name when outbound setting is either
{
  "protocol": "freedom",
  "domainStrategy": "UseIP",
  "settings": {}
}

or
{
  "protocol": "freedom",
  "settings": {
    "domainStrategy": "UseIP"
  }
}

I don't use v4 json format because 'bindToDevice' is not supported in it.
Please attach your configuration here

Server configuration:
Process failed to start with this configuration:
{
  "dns": {
    "nameServer": [{
      "address": {
        "address": "<IP_ADDRESS>",
        "port": 53
      }
    }]
  },
  "inbounds": [
    {
      "port": 8080,
      "protocol": "http",
      "listen": "<IP_ADDRESS>"
    }
  ],
    "outbounds": [
    {
      "protocol": "freedom",
      //"domainStrategy": "UseIP",
      "settings": {
        "domainStrategy": "UseIP"
      }
    },
    {
      "protocol": "blackhole",
      "tag": "block"
    }
  ],
  "router": {
    "domainStrategy": "AsIs"
  },
  "log": {
    "error": {
      "type": "Console",
      "level": "Info"
    }
  }
}

"domainStrategy": "UseIP" has no effect with this configuration
{
  "dns": {
    "nameServer": [{
      "address": {
        "address": "<IP_ADDRESS>",
        "port": 53
      }
    }]
  },
  "inbounds": [
    {
      "port": 8080,
      "protocol": "http",
      "listen": "<IP_ADDRESS>"
    }
  ],
    "outbounds": [
    {
      "protocol": "freedom",
      "domainStrategy": "UseIP",
      "settings": {
        //"domainStrategy": "UseIP"
      }
    },
    {
      "protocol": "blackhole",
      "tag": "block"
    }
  ],
  "router": {
    "domainStrategy": "AsIs"
  },
  "log": {
    "error": {
      "type": "Console",
      "level": "Info"
    }
  }
}

Client configuration:
Using command curl
https_proxy=http://<IP_ADDRESS>:1080 curl https://github.com

While I am using v5.4.1 and had tried every methods to resolve IP for every requests before sending them to outbound (that's the purpose what "domainStrategy": "UseIP" phrase designed for), none of them works. I also tested same configuration under xray environment with success.  That's should be some bugs cause the issue and needed to be fixed, to bring this key feature works again.

While I am using v5.4.1 and had tried every methods to resolve IP for every requests before sending them to outbound (that's the purpose what "domainStrategy": "UseIP" phrase designed for), none of them works. I also tested same configuration under xray environment with success. That's should be some bugs cause the issue and needed to be fixed, to bring this key feature works again.

After switch back to v5.2.1, the UseIP stragegy worked normally, which resolve IP for routed request first, then initiate new request using resolved IP to the outbound chain.

@YVjXuB

Based on the useage in pull request https://github.com/v2fly/v2ray-core/pull/2334, I changed the outbound setting like
{
 "protocol": "freedom",
 "domainStrategy": "UseIP",
 "settings": {}
}

And the v2ray started successfully this time. When I created connection through v2ray by curl, the resloving packages captured by tcpdump were sending to <IP_ADDRESS> which is the setting in /etc/resolv.conf, while they were supposed to send to <IP_ADDRESS> which is the setting in config.json.

This is because in #2334 I only added config support for jsonv4, and forgot to add it for jsonv5 (because I am still sticking to jsonv4). Jsonv5 support is missing in
https://github.com/v2fly/v2ray-core/blob/acbb5e6e0820fc113c476bd25386a098abcac07c/infra/conf/v5cfg/outbound.go#L14-L23

Then I tried to insepct the source code, and figured that DomainStrategy is not in proxy/freedom/config.proto, line 29, message SimplifiedConfig{} while it is shown in line 23,message Config{}.
And also DomainStrategy is not in proxy/freedom/config.pb.go, line 191, type SimplifiedConfig struct{} while it is shown in line 123, type Config struct{}.
I wonder if some changes are needed in those files.
Yes, it is missing and I also wonder the reason. Since we have domainStrategy for all outbounds now, should we only use the global one or add to freedom outbound settings again for compatibility? @AkinoKaede

@bayard

While I am using v5.4.1 and had tried every methods to resolve IP for every requests before sending them to outbound (that's the purpose what "domainStrategy": "UseIP" phrase designed for), none of them works. I also tested same configuration under xray environment with success. That's should be some bugs cause the issue and needed to be fixed, to bring this key feature works again.

Are you using jsonv4 or jsonv5? For jsonv4 I've just tested that both way mentioned in this issue will work. Could you share you config here?

@bayard

While I am using v5.4.1 and had tried every methods to resolve IP for every requests before sending them to outbound (that's the purpose what "domainStrategy": "UseIP" phrase designed for), none of them works. I also tested same configuration under xray environment with success. That's should be some bugs cause the issue and needed to be fixed, to bring this key feature works again.

Are you using jsonv4 or jsonv5? For jsonv4 I've just tested that both way mentioned in this issue will work. Could you share you config here?

Thanks for your reply.
Because I had switch to xray and the config file might had been modified to compatible to xray's syntax, the following config may not be exact the same with mentioned one in my previous reply:
`{
// Cloudflare的proxy模式中，40000端口的socks无法通过udp dns查询，导致dns解析失败。劫持基于IP地址的DoH dns请求，通过代理直接解析
"log": {
"loglevel": "debug"
},
"dns": {
"servers": [
"https://<IP_ADDRESS>/dns-query",
"https://<IP_ADDRESS>/dns-query"
    ]
},
"outbounds": [
    {
        "tag": "dns-out",
        "protocol": "dns"
    },
    {
        "tag": "proxy-dispatcher",
        "mux": {
            "enabled": true,
            "concurrency": -1,
            "xudpConcurrency": 256,
            "xudpProxyUDP443": "allow"
        },
        "protocol": "freedom",
        "settings": {
            "domainStrategy": "UseIP"
        },
        "proxySettings": {
            "tag": "proxy"
        }
    },
    {
        "tag": "overseasdns-dispatcher",
        "protocol": "freedom",
        "proxySettings": {
            "tag": "proxy",
            "transportLayer": true
        },
        "mux": {
            "enabled": true,
            "concurrency": -1,
            "xudpConcurrency": 256,
            "xudpProxyUDP443": "allow"
        }
    },
    {
        "protocol": "socks",
        "settings": {
            "servers": [
                {
                    "address": "<IP_ADDRESS>",
                    "port": 40000
                }
            ]
        },
        "tag": "proxy"
    },
    {
        "tag": "direct",
        "protocol": "freedom",
        "settings": {}
    }
],
"inbounds": [
    {
        "tag": "socks-in",
        "listen": "<IP_ADDRESS>",
        "port": 2162,
        "protocol": "socks",
        "settings": {
            "udp": true
        },
        "sniffing": {
            "enabled": true,
            "destOverride": [
                "http",
                "tls"
            ]
        }
    },
    {
        "tag": "http-in",
        "listen": "<IP_ADDRESS>",
        "port": 2161,
        "protocol": "http",
        "sniffing": {
            "enabled": true,
            "destOverride": [
                "http",
                "tls"
            ]
        },
        "settings": {
            "auth": "noauth",
            "udp": false
        }
    }
],
"routing": {
    "domainStrategy": "IPIfNonMatch",
    "rules": [
        {
            "type": "field",
            "inboundTag": [
                "socks-in",
                "http-in"
            ],
            "port": 53,
            "outboundTag": "dns-out"
        },
        {
            "type": "field",
            "ip": [
                "<IP_ADDRESS>",
                "<IP_ADDRESS>"
            ],
            "outboundTag": "overseasdns-dispatcher"
        },
        {
            "type": "field",
            "outboundTag": "Reject",
            "domain": [
                "geosite:category-ads-all"
            ]
        },
        {
            "type": "field",
            "outboundTag": "direct",
            "domain": [
                "geosite:private",
                "geosite:cn"
            ]
        },
        {
            "type": "field",
            "outboundTag": "direct",
            "ip": [
                "geoip:cn",
                "geoip:private"
            ]
        },
        {
            "type": "field",
            "inboundTag": [
                "socks-in",
                "http-in"
            ],
            "outboundTag": "proxy-dispatcher"
        }
    ]
}

}`

@bayard I've discovered the root cause of your issue. It is indeed a regression bug introduced after #2334.
There are two reasons for your failure:

After #2334, using domainStrategy with proxySettings, the original outbound's domainStrategy will not be honored. This is because domain resolving is postponed to the very end before dialing, and proxySettings happened before it, and the domainStrategy information is lost.

Even if we pass the domainStrategy information from freedom outbound to socks5 outbound, or just specify domainStrategy for socks5 outbound, the domain will still not be resolved, as socks5 outbound will only try to dial its proxy address (<IP_ADDRESS> in this case), so domainStrategy for such outbound (socks5, vmess, etc) are for specifying the strategy for resolving their remote server addresses.

I'll figure out a way to resolve this issue.

Btw, are your using freedom and proxySettings to socks5 is for resolving the proxied domain in client side instead of socks5 server side?

This also raises a question not discussed in #2334: By using domainStrategy for outbounds like socks5 and vmess, what domain should be resolved? :

The proxy server's address (will be used for dialing)
The connection destination (written in request header and sent to proxy server)

Should it be 1 (current behavior), or 2 (so it behaves like freedom outbound), or both?
@AkinoKaede @xiaokangwang

@bayard I've discovered the root cause of your issue. It is indeed a regression bug introduced after #2334.
There are two reasons for your failure:

After [app/proxyman] DomainStrategy support for all outbounds #2334, using domainStrategy with proxySettings, the original outbound's domainStrategy will not be honored. This is because domain resolving is postponed to the very end before dialing, and proxySettings happened before it, and the domainStrategy information is lost.
Even if we pass the domainStrategy information from freedom outbound to socks5 outbound, or just specify domainStrategy for socks5 outbound, the domain will still not be resolved, as socks5 outbound will only try to dial its proxy address (<IP_ADDRESS> in this case), so domainStrategy for such outbound (socks5, vmess, etc) are for specifying the strategy for resolving their remote server addresses.

I'll figure out a way to resolve this issue.

From the traced log output, the v2ray's behavior was like what you explained:
What I expected was the DNS requests(hijacked by dns-out)  should be routed to DNS servers specified in DNS directive, then dial outbound proxy using resolved IP instead of domain name because the socks proxy unable to handle DNS resolution correctly (passing requests with domain name to cloudflare WARP's socks proxy will failed, the reason might related to socks version compatibility or internal DNS address blocked by GFW).
But the log revealed that hijacked DNS routed and IP resolved correctly, and parallelly a request using original domain name also dialed to outbound  socks proxy and failed because the proxy unable to handle requests using domain name. It seemed that  UseIP domainStrategy with proxySettings had no effect in this circumstance.

Btw, are your using freedom and proxySettings to socks5 is for resolving the proxied domain in client side instead of socks5 server side?

YES. The outbound socks5 was cloudflare WARP, it seems that the proxy only support socks4 or internal DNS resolution blocked/poisoned by GFW, so I tried to resolve IP in client side.

YES. The outbound socks5 was cloudflare WARP, it seems that the proxy only support socks4 or internal DNS resolution blocked/poisoned by GFW, so I tried to resolve IP in client side.

So this proves that for domainStrategy for socks and vmess outbound, we need option 3: both proxy server's address and target destination will be resolved.


Conditionals should be supported in state contracts.

Different substories inside the switch conditional should be allowed to assign state variable with the same name.

Superseded by #627


use button groups for translations

@saeedmb8 review me please!

:+1: :cake: :cake: :cake: :cake: :tada:

Waiting for this to land on master

@miltontony can we close this PR?


Transfer learning and fine-tuning tts

Hi everybody !
I have trained on my computer a Vietnamese female voice model in 500k steps. and I found the voice quite clear. I want to train another male Vietnamese voice.
I learned there is a training method based on a previously trained model, which will shorten the training time.
Can someone help me with that method.
Thank you !

Hello. Not sure if a reply this late will help, but that is simply known as transfer learning. You take your first model's checkpoint, pass it as the pre_trained model, and warm start from that point. This will ensure that your model now has the new speaker's voice as well as benefitting from the previous training.

Hello. Not sure if a reply this late will help, but that is simply known as transfer learning. You take your first model's checkpoint, pass it as the pre_trained model, and warm start from that point. This will ensure that your model now has the new speaker's voice as well as benefitting from the previous training.

Hello, thank you for your response.
I understand that to fine-tune the model in such a way, I just need to replace the audio files and metadata of the new model in the location of the old model, and continue training, right? I would appreciate your feedback!

Yes. I have done it with tacotron 2 model, and it for sure works. Basically you would do the training just as in the first time: you get your audio dataset ready, and give the model your new audios and the audios' transcriptions.

Just the only difference is you choose the pre trained model to be your previously trained model's checkpoint. 2) You make use of warm starting. It should be like a parameter in hparams that you set to TRUE.
One more thing: do not change the original batch size. At least in tacotron 2 whenever I changed batch size it printed out errors.


New version: goplus.gop 1.2.3

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?
[x] This PR only modifies one (1) manifest
[x] Have you validated your manifest locally with winget validate --manifest <path>?

[x] Does your manifest conform to the 1.6 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

Automatic Validation ended with:

Executable C:\Users\validator\AppData\Local\Microsoft\WinGet\Links\gop.exe returned exit code: 2
Executable C:\Users\validator\AppData\Local\Microsoft\WinGet\Links\gopfmt.exe returned exit code: 2
Executable C:\Users\Validator\AppData\Local\Microsoft\WinGet\Links\gop.exe returned exit code: 2
Executable C:\Users\Validator\AppData\Local\Microsoft\WinGet\Links\gopfmt.exe returned exit code: 2

(Automated response - build 869.)

@wingetbot waivers Add Validation-Executable-Error


Spacy EntityRuler

Hi, I try to use Spacy V3, but I got a warning that "The component 'entity_ruler' does not have any patterns defined."
So, should I specify a specific pattern for entity_ruler?
Thanks

It has to work with Spacy v2.3.7, and please make sure you install the en-core-web-sm (or -lg) v2.3.1.


Allow area to display multiple carets and selections

First attempt at resolving the first aspect of #222.
There are a few questions that still need to be answered:

~How should the caret's bounds on screen be calculated? The old approach would force the ParagraphBox and ParagraphText to layout their respective contents before querying the bounds. Does this still need to be done or will the bounds be inaccurate? Also, how should one determine whether the caret is outside of the viewport (i.e. it's parent was removed) and distinguish that event from one in which the caret is moved to another paragraph within the viewport?~

The old approach still works. I couldn't get it to work previously because there was a very subtle bug that had to be fixed first.

How should the area's pseudo class has caret be handled now? Do we rename that pseudo class to has-main-caret and add another pseudo class has-a-caret for each ParagraphBox, so that the first one applies when it has the main caret and the second one applies when it has any non-main caret?
Should the main caret be prevented from being removed? Others might wish to use their own custom caret as the main caret. If allowed, how would the default behavior work since the caret is used considerably for that? If prohibited, how should it be prevented from being removed?

This requires special handling as one cannot just replace the caret with their own version. They need to replace it with a CaretSelectionBind, so that the default behavior still works since it updates the selection, which propagates changes to the caret.

~How should a caret be distinguishable from another caret? I'm currently using a List (as it makes it easier to implement ObservableList via ModifiableObservableListBase, which allows the list change listeners), but this allows the same caret to be added twice. Thus, I've added additional checking to insure that can't occur. However, the better structure to use would be a Set, but I don't think they have an ModifiableObservableSetBase class, making such an implementation its own larger issue. Since there isn't one distinguishable thing about a given caret, how should they be equaled/hashed to be different? One idea is to name them.~

Naming them seems simple enough.

~Currently, the bounds related tests fail in this PR. I'm not sure whether the tests should be changed or the bounds intialization/updating should be changed.~
All tests currently pass on Linux. Not sure about Windows/Mac
Also, I still need to write tests for this.

Looks like some more tests need to be run via the JFX App Thread, not the test thread

The multiple selections doesn't quite work yet since it doesn't properly remove the selections from the area. Also, some tests fail because they aren't being run on the JFX App thread. I'll fix that on another day.

I've updated the tests so that they pass. However, 2 tests do not pass. I ran these tests on code from various commits in the project that do not include any commits from this PR and they fail there, too. I'm left wondering whether a newer Java version broke something.
When I watch the two failing tests on my screen, the TextFlow/mouse doesn't not seem to work correctly

I'm left wondering whether a newer Java version broke something.

Travis CI is using Java 8u151 still in this build and in a prior build that passed. So this really is an issue introduced somewhere in this PR.

Things left to do:

[ ] Write much clearer javadoc for various things
[ ] Write tests for the multiple caret/selection code
[ ] Add demo for multi caret/selection

During this process, I learned that Selection#selectionBoundsProperty throws an exception because it tries to calculate itself before the area is finished updating. I've corrected it in this PR as well.

This is ready to be merged. I would appreciate any feedback on this.


Feature/plus minus board

Changelogs:

like functionality implemented
edit functionality (currently disabled because it was using a text area which didn't look good)
drag and drop enabled
cards, box shadow, show delete and like on hover
other UI changes

Issues:

linear lists (two list items don't arrange themselves in one row)

In general I would prefer smaller PR, as otherwise review becomes very difficult :)


Ensure we set the cache policy on all of our I/O operations, we set i…

…t to be FetchMissing so we use the cached value for anything we download -- so snapshots should not change. The download pool by default is 6, this is low i think for most scenarios where you are fetching a lot of small files and are incurring overheads of making lots of SSL connections
Context/goal: After adding some of the courser stuff i've been trying to recover the latency introduced by getting the extra sha details. I did notice we did not usually seem to be populating the coursier cache unlike the cache we had been building with aether.
using the built jar and export COURSIER_CACHE=$HOME/.coursier-cache has that working, need to do some confirmations if we need to build the fat jar for it to work right. But thats not in this PR anyway.
export COURSIER_CACHE=$HOME/.coursier-cache has that working
Latency 10+min and 700MB downloads -> 2min with very little downloaded at all in test repo

Great, thanks, @ianoc-stripe !


fix: get more details in the errors from grpc

Codecov Report

Merging #58 (810b225) into develop (26690e8) will increase coverage by 0.09%.
The diff coverage is 0.00%.

@@             Coverage Diff             @@
##           develop      #58      +/-   ##
===========================================
+ Coverage    77.11%   77.21%   +0.09%
===========================================
  Files           37       37
  Lines         1604     1602       -2
===========================================
  Hits          1237     1237
+ Misses         314      312       -2
  Partials        53       53

Flag
Coverage Δ

unittests
77.21% <0.00%> (+0.09%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

resources/manager.go
34.54% <0.00%> (+1.21%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f6a4de7...810b225. Read the comment docs.


bug: harbor project create command fails due to unhandled error.

harbor project create commands due to unhandled error in the getRegistryList.

https://github.com/goharbor/harbor-cli/blob/381ec2dd7305dc515f290e9236d0f387e6a4470b/pkg/views/project/create/view.go#L39
./harbor project create
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x9b1cbd]

goroutine 1 [running]:
github.com/goharbor/harbor-cli/pkg/views/project/create.CreateProjectView(0xc000392d00)
        /home/mehul/lfx/harbor-cli/pkg/views/project/create/view.go:45 +0x9d
github.com/goharbor/harbor-cli/cmd/harbor/root/project.createProjectView(0x0?)
        /home/mehul/lfx/harbor-cli/cmd/harbor/root/project/create.go:67 +0x58
github.com/goharbor/harbor-cli/cmd/harbor/root/project.CreateProjectCommand.func1(0xc0001c8d00?, {0x10c6680?, 0x4?, 0xaf28a6?})
        /home/mehul/lfx/harbor-cli/cmd/harbor/root/project/create.go:37 +0xec
github.com/spf13/cobra.(*Command).execute(0xc0002f6f00, {0x10c6680, 0x0, 0x0})
        /home/mehul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987 +0xaa3
github.com/spf13/cobra.(*Command).ExecuteC(0xc0002f6300)
        /home/mehul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(0xc0000061a0?)
        /home/mehul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039 +0x13
main.main()
        /home/mehul/lfx/harbor-cli/cmd/harbor/main.go:10 +0x18

Still the project creation fails with the below error.

./harbor project create

ERRO[0013] failed to create project: [POST /projects][400] createProjectBadRequest  &{Errors:[0xc0000fa0e0]}

It seems you must set the registry first before you can create a project via the CLI. The API endpoint POST /project requires a registry ID to be present and should be greater than zero. Therefore, you cannot create a project without specifying the registry ID.
So, We need to let the user know that they need to create the registry first to proceed with project creation.
But on the WebUi, this is not the case the user can create the project even without having a registry.


🐛 Bug Report: Backstage express/morgan request logs do not identify the security principal making the request.

📜 Description
Backstage does not populate the remote-user token within the morgan / express logger formats. When viewing the logs, one does not know which user or service is making the request and performing the action. This makes security forensics impossible.
I see two possible outcomes for this concern:

make the core backstage correctly populate the remote_user within express / morgan logs.
provide documentation on how users can perform this action. For example, if you feel that logging the name of the user is a PII concern, then at least tell others how to do it.

I'm able to work around this, and will post code in the ticket on how I'm doing so. I just feel like others in the community shouldn't have to figure this out, and it should work correctly out of the gate.
👍 Expected behavior
Log the security principal associated with the express request in the remote_user field. Like this:
[0] [1] 2024-05-03T14:05:13.992Z rootHttpRouter info ::1 - service:legacy-backend-to-backend [03/May/2024:14:05:13 +0000] "GET /api/catalog/entities?limit=500&offset=0 HTTP/1.1" 200 - "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type=incomingRequest
[0] [1] 2024-05-03T14:12:43.276Z rootHttpRouter info ::1 - user:default/von.jones [03/May/2024:14:12:43 +0000] "POST /api/catalog/entities/by-refs/ HTTP/1.1" 200 - "http://localhost:3000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36" type=incomingRequest
[0] [1] 2024-05-03T14:21:52.282Z rootHttpRouter info ::1 - service:backstage-smoke-tests [03/May/2024:14:21:52 +0000] "GET /api/status/readiness HTTP/1.1" 200 15 "-" "curl/8.4.0" type=incomingRequest

👎 Actual Behavior with Screenshots
The remote_user field is not present in the logs. It shows as a dash, like this:
[0] [1] 2024-05-03T14:05:13.992Z rootHttpRouter info ::1 - - [03/May/2024:14:05:13 +0000] "GET /api/catalog/entities?limit=500&offset=0 HTTP/1.1" 200 - "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type=incomingRequest
[0] [1] 2024-05-03T14:12:43.276Z rootHttpRouter info ::1 - - [03/May/2024:14:12:43 +0000] "POST /api/catalog/entities/by-refs/ HTTP/1.1" 200 - "http://localhost:3000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36" type=incomingRequest
[0] [1] 2024-05-03T14:21:52.282Z rootHttpRouter info ::1 - - [03/May/2024:14:21:52 +0000] "GET /api/status/readiness HTTP/1.1" 200 15 "-" "curl/8.4.0" type=incomingRequest

👟 Reproduction steps
View the logs, filtered to requests.
📃 Provide the context for the Bug.
I am unable to track which actions are performed by which users on which teams. If a user reports a bug, I cannot view the logs associated with their requests. I am unable to discern which actions are user actions, which are backend service-to-service actions, and which are external API callers we created tokens for.
🖥 Your Environment
OS:   Darwin 23.4.0 - darwin/x64
node: v20.10.0
yarn: 1.22.19
cli:  0.26.2 (installed)
backstage:  1.25.2
Dependencies:
@backstage/app-defaults                                          1.5.3
@backstage/backend-app-api                                       0.6.2, 0.7.1
@backstage/backend-common                                        0.21.6, 0.21.7
@backstage/backend-defaults                                      0.2.17
@backstage/backend-dev-utils                                     0.1.4
@backstage/backend-openapi-utils                                 0.1.10, 0.1.9
@backstage/backend-plugin-api                                    0.6.16, 0.6.17
@backstage/backend-tasks                                         0.5.21, 0.5.22
@backstage/catalog-client                                        1.6.3, 1.6.4
@backstage/catalog-model                                         1.4.5
@backstage/cli-common                                            0.1.13
@backstage/cli-node                                              0.2.4, 0.2.5
@backstage/cli                                                   0.26.2
@backstage/config-loader                                         1.7.0, 1.8.0
@backstage/config                                                1.2.0
@backstage/core-app-api                                          1.12.3
@backstage/core-compat-api                                       0.2.3, 0.2.4
@backstage/core-components                                       0.14.5
@backstage/core-plugin-api                                       1.9.1, 1.9.2
@backstage/dev-utils                                             1.0.30
@backstage/errors                                                1.2.4
@backstage/eslint-plugin                                         0.1.6
@backstage/frontend-plugin-api                                   0.6.3, 0.6.4
@backstage/integration-aws-node                                  0.1.12
@backstage/integration-react                                     1.1.25, 1.1.26
@backstage/integration                                           1.10.0, 1.9.1
@backstage/plugin-api-docs-module-protoc-gen-doc                 0.1.6
@backstage/plugin-api-docs                                       0.11.3
@backstage/plugin-app-backend                                    0.3.64
@backstage/plugin-app-node                                       0.1.16
@backstage/plugin-auth-backend-module-atlassian-provider         0.1.8
@backstage/plugin-auth-backend-module-aws-alb-provider           0.1.8
@backstage/plugin-auth-backend-module-gcp-iap-provider           0.2.11
@backstage/plugin-auth-backend-module-github-provider            0.1.13
@backstage/plugin-auth-backend-module-gitlab-provider            0.1.13
@backstage/plugin-auth-backend-module-google-provider            0.1.13
@backstage/plugin-auth-backend-module-microsoft-provider         0.1.11
@backstage/plugin-auth-backend-module-oauth2-provider            0.1.13
@backstage/plugin-auth-backend-module-oauth2-proxy-provider      0.1.9
@backstage/plugin-auth-backend-module-oidc-provider              0.1.7
@backstage/plugin-auth-backend-module-okta-provider              0.0.9
@backstage/plugin-auth-backend                                   0.22.3
@backstage/plugin-auth-node                                      0.4.11, 0.4.12
@backstage/plugin-auth-react                                     0.0.3
@backstage/plugin-catalog-backend-module-github                  0.5.7
@backstage/plugin-catalog-backend-module-scaffolder-entity-model 0.1.14
@backstage/plugin-catalog-backend                                1.21.0
@backstage/plugin-catalog-common                                 1.0.22
@backstage/plugin-catalog-graph                                  0.4.3
@backstage/plugin-catalog-import                                 0.10.9
@backstage/plugin-catalog-node                                   1.11.0, 1.11.1
@backstage/plugin-catalog-react                                  1.11.2, 1.11.3
@backstage/plugin-catalog                                        1.18.2
@backstage/plugin-events-backend-module-github                   0.2.3
@backstage/plugin-events-backend                                 0.3.4
@backstage/plugin-events-node                                    0.3.3, 0.3.2
@backstage/plugin-home-react                                     0.1.11
@backstage/plugin-home                                           0.7.2
@backstage/plugin-org                                            0.6.23
@backstage/plugin-permission-common                              0.7.13
@backstage/plugin-permission-node                                0.7.27, 0.7.28
@backstage/plugin-permission-react                               0.4.21, 0.4.22
@backstage/plugin-proxy-backend                                  0.4.14
@backstage/plugin-scaffolder-backend-module-azure                0.1.8
@backstage/plugin-scaffolder-backend-module-bitbucket-cloud      0.1.6
@backstage/plugin-scaffolder-backend-module-bitbucket-server     0.1.6
@backstage/plugin-scaffolder-backend-module-bitbucket            0.2.6
@backstage/plugin-scaffolder-backend-module-gerrit               0.1.8
@backstage/plugin-scaffolder-backend-module-gitea                0.1.6
@backstage/plugin-scaffolder-backend-module-github               0.2.6, 0.2.7
@backstage/plugin-scaffolder-backend-module-gitlab               0.3.2
@backstage/plugin-scaffolder-backend                             1.22.3
@backstage/plugin-scaffolder-common                              1.5.1
@backstage/plugin-scaffolder-node                                0.4.2, 0.4.3
@backstage/plugin-scaffolder-react                               1.8.3
@backstage/plugin-scaffolder                                     1.19.2
@backstage/plugin-search-backend-module-catalog                  0.1.21
@backstage/plugin-search-backend-module-pg                       0.5.25
@backstage/plugin-search-backend-module-techdocs                 0.1.21
@backstage/plugin-search-backend-node                            1.2.20
@backstage/plugin-search-backend                                 1.5.6
@backstage/plugin-search-common                                  1.2.11
@backstage/plugin-search-react                                   1.7.10, 1.7.9
@backstage/plugin-search                                         1.4.9
@backstage/plugin-techdocs-backend                               1.10.3
@backstage/plugin-techdocs-module-addons-contrib                 1.1.8
@backstage/plugin-techdocs-node                                  1.12.2
@backstage/plugin-techdocs-react                                 1.2.2
@backstage/plugin-techdocs                                       1.10.3
@backstage/plugin-user-settings                                  0.8.4
@backstage/release-manifests                                     0.0.11
@backstage/test-utils                                            1.5.3
@backstage/theme                                                 0.5.2, 0.5.3
@backstage/types                                                 1.1.1
@backstage/version-bridge                                        1.0.7, 1.0.8
✨  Done in 1.27s.
👀 Have you spent some time to check if this bug has been raised before?

[X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

[X] I have read the Code of Conduct

Are you willing to submit PR?
No, but I'm happy to collaborate on a PR with someone else

Here's how I'm solving this in the new backend systems:
// src/packages/backend/src/services/LoggingAugmenterService.ts
import {
  coreServices,
  createServiceFactory,
} from '@backstage/backend-plugin-api';
import { Request, RequestHandler } from 'express';
import morgan from 'morgan';
import { getRootLogger } from '@backstage/backend-common';
import { createServiceRef } from '@backstage/backend-plugin-api';
import { ServerResponse } from 'http';

interface LoggingAugmenterService {}

const loggingAugmenterServiceRef = createServiceRef<LoggingAugmenterService>({
  id: 'wpe-loging-augmenter',
});

// Test this using source .env && curl -H "Authorization: Bearer $<ENVVAR like GROUP_GENERATION_SECRET>" -H  -X GET  http://localhost:7007/api/status/readiness

// Ref: https://backstage.io/docs/backend-system/architecture/services/
// Ref: https://github.com/backstage/backstage/blob/master/docs/plugins/backend-plugin.md#making-use-of-the-users-identity
// Ref: https://github.com/backstage/backstage/blob/9910c6babc104d8b765c9646ee89604b09dc15de/packages/backend-app-api/src/services/implementations/httpAuth/httpAuthServiceFactory.ts#L126
// Ref: https://github.com/backstage/backstage/blob/9910c6babc104d8b765c9646ee89604b09dc15de/packages/backend-app-api/src/services/implementations/httpRouter/httpRouterServiceFactory.ts#L79
// Ref: https://github.com/backstage/backstage/blob/9910c6babc104d8b765c9646ee89604b09dc15de/packages/backend-app-api/src/services/implementations/httpRouter/createCredentialsBarrier.ts#L62
export const loggingAugmenterService = createServiceFactory(() => ({
  service: loggingAugmenterServiceRef,
  initialization: 'always',
  deps: {
    httpRouter: coreServices.httpRouter,
    httpAuth: coreServices.httpAuth,
  },
  factory({ httpRouter, httpAuth }) {
    const credentialAugmenterMiddleware: RequestHandler = async (
      req: Request,
      _,
      next,
    ) => {
      const credentials = await httpAuth.credentials(req);
      // we are adding the credentials to the request object for logging purposes, Used by the morgan.token code below.
      // @ts-ignore we are adding a field not in the original interface
      req.credentials = credentials;
      next();
      return;
    };
    httpRouter.use(credentialAugmenterMiddleware);
    return {};
  },
}));

// log the remote-user field in common logging formats of express request logs
// Ref: https://expressjs.com/en/resources/middleware/morgan.html
morgan.token('remote-user', function getRemoteUserToken(req: Request, res: ServerResponse) {
  try {
    const credentials: any =
      // since morgan.token does now allow us to make the async call, we added this to the request object above in the service
      // @ts-ignore we are adding a field not in the original interface
      req?.credentials;

    if (credentials && credentials?.principal?.type !== 'none') {
      if (credentials.principal?.type === 'user') {
        return credentials.principal.userEntityRef;
      }
      if (credentials.principal?.type === 'service') {
        return `${credentials.principal.type}:${credentials.principal.subject}`;
      }
    }

    if (
      (req.baseUrl === '/api/cookie' && req.path === '/') ||
      req.path?.startsWith('/api/auth/okta/') ||
      req.baseUrl === '/api/auth/okta' ||
      req.path?.startsWith('/api/auth/google/') ||
      req.baseUrl === '/api/auth/google' ||
      req.path?.startsWith('/api/auth/github/') ||
      req.baseUrl === '/api/auth/github'
    ) {
      // calls to the identity system in the process of authenticating the user
      return 'anonymous-authenticating';
    }
    if ([200, 304].includes(res.statusCode)) {
      return 'unauthenticated:public-content';
    }

    // this should be an error case
    return 'unauthenticated:error';
  } catch (error) {
    getRootLogger().error(
      `Error identifying the remote-user for logging: ${error}`,
    );
  }
  return 'remote-user-error';
});

// src/packages/backend/src/index.ts
...
import { loggingAugmenterService } from './services/LoggingAugmenterService';

...
const backend = createBackend();
...
/*
 * Backstage extension services, from the src/packages/backend/src/services directory
 */
backend.add(loggingAugmenterService);

backend.start();

Here's how I was doing it with the old backend system, extending the prior service-to-service auth pattern:
// packages/backend/src/authMiddleware.ts
...
export const createAuthMiddleware = async (
  config: Config,
  appEnv: PluginEnvironment,
) => {
...
      try {
        req.user = await appEnv.identity.getIdentity({ request: req });
      } catch {
...
}

// log the remote-user field in common logging formats of express request logs
morgan.token('remote-user', function getRemoteUserToken(req: Request) {
  try {
    const identity: BackstageIdentityResponse | undefined =
      req?.user as BackstageIdentityResponse;

    if (identity?.identity?.type === 'user') {
      // an authenticated user
      return identity?.identity?.userEntityRef;
    }
    if (identity?.identity?.type === SERVER_TOKEN_SUB) {
      // an authenticated server-to-server request, Ref:
      // https://github.com/backstage/backstage/blob/master/contrib/docs/tutorials/authenticate-api-requests.md
      // It could also be an external API caller that is authorized via
      // https://backstage.io/docs/auth/service-to-service-auth/#usage-in-external-callers . Ideally,
      // the user agent identifies the caller, which is logged separately.
      return 'backstage-or-external-api-caller';
    }
    if (
      (req.baseUrl === '/api/cookie' && req.path === '/') ||
      req.path?.startsWith('/api/auth/okta/') ||
      req.baseUrl === '/api/auth/okta' ||
      req.path?.startsWith('/api/auth/google/') ||
      req.baseUrl === '/api/auth/google' ||
      req.path?.startsWith('/api/auth/github/') ||
      req.baseUrl === '/api/auth/github'
    ) {
      // calls to the identity system in the process of authenticating the user
      return 'anonymous-authenticating';
    }
    // Ref: https://github.com/backstage/backstage/blob/master/packages/app/public/
    const publicDirectoryContents = [
      "/android-chrome-192x192.png",
      "/android-chrome-512x512.png",
      "/apple-touch-icon.png",
      "/favicon-16x16.png",
      "/favicon-32x32.png",
      "/favicon.ico",
      "/favicon.svg",
      "/index.html",
      "/manifest.json",
      "/robots.txt",
      "/safari-pinned-tab.svg"
    ];
    if (
      (req.baseUrl === '/api/auth' && req.path === '/.well-known/jwks.json') ||
      req.baseUrl.startsWith('/static') || req.path.startsWith('/static') ||
      (req.baseUrl === '' && publicDirectoryContents.includes(req.path)) ||
      (req.path.includes('/assets/stylesheets/'))
    ) {
      // public APIs which are intentionally unauthenticated
      return 'anonymous-public-content';
    }

    // this should be an error case
    return 'anonymous-other';
  } catch (error) {
    getRootLogger().error(`Error identifying the remote-user for logging: ${error}`);
  }
  return 'remote-user-error'
});

Love the idea! 👍
I think we may need to explore other options to using morgan, the lack of async isn't a non-starter but it's certainly not great. Bigger issue is that it's a singleton and we're mutating that state withing our service implementation, make it hard to customer cleanly

This might also be possibly related to #21605, and a larger conversation about audit trails where logging may be one of a few venues where this information is tracked.


fix: improve the variables step

so that the generated variables appear before the existing file if it exists to avoid overwriting things
also added KANIKO_FLAGS and any other entries from the docker registry configmap

/approve

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: jstrachan
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [jstrachan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


applySourceMap uses wrong relative path

I'm using gulp-sourcemap to generate source maps with gulp-typescript. My folder structure is as follows:
js/
ts/
    foo/
        fizz.ts
    bar/
        buzz.ts

Gulp is set up to take files from ts, run them through gulp-typescript, and output them to js with separate source maps files. The sources property of the map files is incorrect; for example, a file that is output to js/foo/fizz.js.map has a source property of ../ts/foo/fizz.ts whereas the correct value is ../../ts/foo/fizz.ts.
The problem is constant regardless of nested folder depth, i.e. ts/my/really/deep/folder/hello.ts will be correctly placed in js/my/really/deep/folder/hello.ts and js/my/really/deep/folder/hello.ts.map but the source property of the map still only refers to ../ts/my/really/deep/folder/hello.ts (up one level, instead of several).
From debugging the code, it looks like the problem is related to line 89 of output.ts. Changing output.base to output.dirname fixes the problem for me.
Expected behavior:
source property is set to ../../ts/foo/fizz.ts
Actual behavior:
source property is set to ../ts/foo/fizz.ts.
Your gulpfile:
Include your gulpfile, or only the related task (with ts.createProject).
const gulp = require('gulp');
const ts = require('gulp-typescript');
const sourcemaps = require('gulp-sourcemaps');

const tsProject = ts.createProject('tsconfig.json');

gulp.task('default', () => {
  tsProject.src()
    .pipe(sourcemaps.init())
    .pipe(tsProject())
    .pipe(sourcemaps.write('./'))
    .pipe(gulp.dest('./js'));
});

tsconfig.json
Include your tsconfig, if related to this issue.
{
    "compilerOptions": {
        "module": "commonjs",
        "target": "es5",
        "noImplicitAny": false,
        "sourceMap": true,
        "outDir": "./js",
        "rootDir": "./ts"
    },
    "include": [
        "./ts/**/*.ts"
    ]
}

Code
Include your TypeScript code, if necessary.

Demo of the problem

If I'm not mistaken, gulp-sourcemaps' docs say that the sources property should be relative to the base folder of the emitted files. In your setup, the base folder would be /js/ (and the relative part of the file is foo/fizz.js. Resolving ../ts/foo/fizz.ts relative to the base path does yield the correct source location, but I understand that this behaviour is not really desired for you. I think that the sourceRoot option might help. I might be mistaken, I'll take a better look at this later.

Ooof. Just realized my mistake. I wasn't setting destPath in sourcemaps.write(). Changed my gulpfile to sourcemaps.write('.', { includeContent: false, destPath: './js' }) and it's working flawlessly now. Thanks for your help!


Particles Stretch on Div Stretch during Webpage Load

Hello coders,
I've set up a two-layered div on my website, with particles.js running in the background and the company logo in the foreground. Unfortunately on some occasions the particles.js loads first before the logo. When the logo loads after the particles, the particles become stretched. The issue is fixed immediately just by resizing the browser, or by refreshing the screen (in most occasions).
Is there a sneaky way I can load particles.js only after the image loads? I want to avoid using min-height css tricks because the logo scales depending on window size.
Website: http://chaostheorem.com/
Thanks,
Tim

Update: I've put the particles.js and app.js files right at the bottom of the index.html page so now they load last. This appears to be helping.

Just as a comment, I was having same issue and even if im not quite sure, why it works, but it actually does, i added the following css code to the theme and it works like a charm:
.particles-js-canvas-el {
height: 50vmax !important;
width: 100vmax !important;
}
I really have no idea how did i came up with that, as i dont even know what that css code exactly does (never used the vmax before) but it does.

Where to add index.html, app.js and particle.json files.
Not a dev, please help.


problem with ?

I have problem when use <table> with other attribute like border, align, ... to generate my reports.
how to solve?

This package wraps https://parall.ax/products/jspdf.
Any rendering issues should be reported to them.

thanks,


Support for SF Symbols 5

Are there any plans to support SF Symbols 5? This new version was officially released a few weeks ago.
https://developer.apple.com/sf-symbols/
Thank you very much for this great library! I use it in all of my apps. ❤

Any updates?

SFSafeSymbols 5.2.0 is now released


 Unfortunately I don't have time to investigate and fix it, because SeleniumLibrary development is taking all my available time for open source. Most likely bug is in here: https://github.com/andriyko/sublime-robot-framework-assistant/blob/master/Robot.tmLanguage
I am actively looking for a new lead developer for the project, would you happen to know anyone who would be interested?

#223 That I just submitted might have inadvertently fixed this? In my sample file I used a few test cases that started with "I" and they worked

@jimpriest , does the fix #223 resolve your issue?

Yes!  I had actually hacked up a similar fix of my own but @jsdavis  beat me to it.


Add Acronym practice exercise

@kephas The entry in the config.json file seems to be missing. See https://github.com/exercism/lfe/actions/runs/7927859604/job/21644993462?pr=194

Sorry, I didn't think to use configlet. There is no warning anymore.


Support arrow cursors in "llm chat" mode

https://twitter.com/josh_m/status/1727472264688677314

llm doesn't like arrow keys in a (I think) stock macOS terminal, it just inserts control chars.

This seems to fix it:
diff --git a/llm/cli.py b/llm/cli.py
index af37feb..3fa2ecc 100644
--- a/llm/cli.py
+++ b/llm/cli.py
@@ -28,6 +28,7 @@ from .plugins import pm
 import base64
 import pathlib
 import pydantic
+import readline
 from runpy import run_module
 import shutil
 import sqlite_utils
@@ -340,6 +341,7 @@ def chat(
     """
     Hold an ongoing chat with a model.
     """
+    readline.parse_and_bind("")
     log_path = logs_db_path()
     (log_path.parent).mkdir(parents=True, exist_ok=True)
     db = sqlite_utils.Database(log_path)

This was an incomplete fix, but I addressed it properly here:

#376


Update mockito-scala, ... to 1.16.1

Updates

org.mockito:mockito-scala
org.mockito:mockito-scala-scalatest

from 1.16.0 to 1.16.1.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.mockito" } ]

labels: test-library-update, semver-patch

Codecov Report

Merging #836 (4819a99) into master (726457b) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #836   +/-   ##
=======================================
  Coverage   82.80%   82.80%
=======================================
  Files          73       73
  Lines        5404     5404
  Branches      146      146
=======================================
  Hits         4475     4475
  Misses        929      929

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 726457b...4819a99. Read the comment docs.


[BUG FIX] Fixes fileouput directory issue

probably a good idea to wait for #62 to be merged to master, because there's some functionality that will come in handy for validating output specs.


"It's not how much money you make, but how much money you keep, how hard it works for you, and how many generations you keep it for."  join investment today if you are interested ask me how

"It's not how much money you make, but how much money you keep, how hard it works for you, and how many generations you keep it for."  join investment today if you are interested ask me how