parallel mode ignores divide by zero errors, sometimes fills with nonsense

Reporting a bug

[x] I have tried using the latest released version of Numba (most recent is
visible in the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
[x] I have included a self contained code sample to reproduce the problem.
i.e. it's possible to run as 'python bug.py'.

Case 1 nested
The size the array seems to matter for whether the output is non-deterministic
import numpy as np
from numba import njit, prange

def foo(x, y):
    return x / y

cfoo = njit(foo)

def bar(x, y):
    out = np.zeros(len(x))
    for i in prange(len(x)):
        out[i] = cfoo(x[i], y)
    return out

Errors with no parallelization, the expected result
njit(bar)(np.ones(100), 0)

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-42-aebfaf53ad4a> in <module>
----> 1 njit(bar)(np.ones(100), 0)

ZeroDivisionError: division by zero

With parallelization, no error, and nonsense results
njit(parallel=True)(bar)(np.ones(100), 0)

array([ 0.00000000e+000, -1.36311572e+057, -1.36311572e+057,
       -1.36311572e+057, -1.36311572e+057, -1.36311572e+057,
       -1.36311572e+057,  0.00000000e+000, -1.36311572e+057,
       -1.36311572e+057, -1.36311572e+057, -1.36311572e+057,
       -1.36311572e+057, -1.36311572e+057,  0.00000000e+000,
       -1.36311572e+057, -1.36311572e+057, -1.36311572e+057,
       -1.36311572e+057, -1.36311572e+057, -1.36311572e+057,
        0.00000000e+000, -1.36311572e+057, -1.36311572e+057,
       -1.36311572e+057, -1.36311572e+057, -1.36311572e+057,
       -1.36311572e+057,  0.00000000e+000, -1.36311572e+057,
       -1.36311572e+057, -1.36311572e+057,  2.41298017e-314,
       -1.49166815e-154,  0.00000000e+000,  0.00000000e+000,
        2.41463797e-314,  0.00000000e+000,  0.00000000e+000,
        6.94200297e-310,  0.00000000e+000, -1.49166815e-154,
        1.38338381e-322,  2.33419537e-313,  1.28303992e-138,
        1.28528994e-138,  0.00000000e+000,  1.28765272e-138,
        1.28527008e-138,  1.28642776e-138,  1.28643801e-138,
        1.28663790e-138,  0.00000000e+000,  1.28747333e-138,
        1.28748358e-138,  4.94065646e-324,              nan,
        6.94199943e-310,  0.00000000e+000,  8.48798317e-314,
        1.27319747e-313,  1.83684208e-313,  1.69759663e-313,
        5.56268465e-308,  0.00000000e+000,  2.41108763e-314,
        2.41098414e-314,  7.53351297e-320, -1.49166815e-154,
       -1.29074172e-231,  0.00000000e+000,  1.28617150e-138,
        4.33875185e-311, -1.49166815e-154,  2.41463797e-314,
        2.41465960e-314,  0.00000000e+000,  2.41465859e-314,
        2.12199579e-314,  4.94065646e-324,  0.00000000e+000,
        0.00000000e+000,  0.00000000e+000,  8.51419406e-308,
        2.41465605e-314,  2.41465778e-314,  2.41465865e-314,
        2.41465960e-314,  0.00000000e+000,  2.41465866e-314,
        2.41465867e-314,  2.41465882e-314,  2.41465883e-314,
        2.41465947e-314,  0.00000000e+000,  9.73469813e-309,
       -1.49166815e-154,  1.73060171e-077,  2.41463799e-314,
        2.41465867e-314])

Case 2 non-nested
def baz(x, y):
    out = np.zeros(len(x))

    for i in numba.prange(len(x)):
        out[i] = x[i] / y

    return out

This at least follows numpy's semantics, though numpy would warn.
njit(parallel=True)(baz)(np.ones(100), 0)

array([inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf,
       inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf,
       inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf,
       inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf,
       inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf,
       inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf,
       inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf,
       inf, inf, inf, inf, inf, inf, inf, inf, inf])

Expected results
I would have expected all these cases to error, following from the documentation for jit.
Version info
print(numba.__version__)
print(numba.threading_layer())
# 0.53.1
# workqueue

Thanks for the report.

For case 1, I can't seem to reproduce.
import numpy as np
from numba import njit, prange
from numba import version_info

print(version_info.short)

def foo(x, y):
    return x / y

cfoo = njit(foo)

def bar(x, y):
    out = np.zeros(len(x))
    for i in prange(len(x)):
        out[i] = cfoo(x[i], y)
    return out

njit(parallel=True)(bar)(np.ones(100), 0)

with Numba 0.52, 0.53 and mainline are all reporting ZeroDivisionError for me.

For case 2: I can reproduce, it returns an array of Inf.

I suspect this all comes down to the error_model and where it's been defined. In case 1 that foo is defined with njit means it will have the default cpython error model, I think this will lead to code generation with the div-by-zero check in it. Calling this cpython error model function from a parallel=True function, which has the error_model set as 'numpy', i.e. don't do the div-by-zero check, will not eliminate the check as the flag won't propagate. In case 2, as there's just one function and it's parallel=True the NumPy error model will be in use and the result is expected.
I think this needs adding to the considerations around compilation flags (there's large scale refactoring ongoing in relation to this for 0.54/0.55).

For the part of the first case that you can't reproduce, I would suspect that it's related to the parallel backend ('workqueue' on my machine) as I had run into very similar behaviour around these kinds of return values with the actual function in question. Also, whoops, adding version info above.

For the part of the first case that you can't reproduce, I would suspect that it's related to the parallel backend ('workqueue' on my machine) as I had run into very similar behaviour around these kinds of return values with the actual function in question. Also, whoops, adding version info above.

hmmm, this is interesting. First, which OS are you seeing this on, and second, what is it you are observing? There's three backends, would be interesting to know what goes on in each. Thanks :)

Ah, on linux, this:
$ NUMBA_THREADING_LAYER=tbb python issue6976.py
(0, 54)
ZDE hit
tbb

$ NUMBA_THREADING_LAYER=omp python issue6976.py
(0, 54)
OMP: Info #274: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
ZDE hit
omp

$ NUMBA_THREADING_LAYER=workqueue python issue6976.py
(0, 54)
workqueue

from this:
import numpy as np
from numba import njit, prange, threading_layer
from numba import version_info

print(version_info.short)

def foo(x, y):
    return x / y

cfoo = njit(foo)

def bar(x, y):
    out = np.zeros(len(x))
    for i in prange(len(x)):
        out[i] = cfoo(x[i], y)
    return out

try:
    njit(parallel=True)(bar)(np.ones(100), 0)
except ZeroDivisionError:
    print("ZDE hit")
    pass

print(threading_layer())

suggests workqueue has a problem. Which now mentioned, seems familiar.

First, which OS are you seeing this on

This is macOS

second, what is it you are observing?

It's a bit late for me, so I can write more on this tomorrow. It's the same function that led to my previous report here, and possibly a few other bug reports.
This particular case actually came in as a bug report (https://github.com/theislab/scanpy/issues/1698#issuecomment-824634639), but I've detailed some of my debugging here: https://github.com/theislab/scanpy/issues/1698#issuecomment-826712541

There's three backends, would be interesting to know what goes on in each.

This has been a bit of perennial problem for me. I haven't been able to get either of the other two backends installed and working (though the last time I tried was a couple months ago).

First, which OS are you seeing this on

This is macOS

second, what is it you are observing?

It's a bit late for me, so I can write more on this tomorrow. It's the same function that led to my previous report here, and possibly a few other bug reports.
This particular case actually came in as a bug report (theislab/scanpy#1698 (comment)), but I've detailed some of my debugging here: theislab/scanpy#1698 (comment)

Think this comes down to how exceptions are handled in the threading backends. In OpenMP and TBB the main thread joins the pool doing the parallel execution, so if it hits e.g. a div-by-zero then the exception is triggered. Whereas in workqueue, the main thread schedules work for the thread pool but doesn't take part in the work, so exceptions are essentially ignored.

There's three backends, would be interesting to know what goes on in each.

This has been a bit of perennial problem for me. I haven't been able to get either of the other two backends installed and working (though the last time I tried was a couple months ago).

Sorry to hear this, that one of TBB or OpenMP is not available is unfortunate. Will try and debug it a bit.

This has been a bit of perennial problem for me. I haven't been able to get either of the other two backends installed and working (though the last time I tried was a couple months ago).

Sorry to hear this, that one of TBB or OpenMP is not available is unfortunate. Will try and debug it a bit.

@sklam and I just looked, seems like lack of supporting OpenMP headers at wheel build time could  be the cause of the missing OpenMP backend.

Sorry to hear this, that one of TBB or OpenMP is not available is unfortunate.

I could probably give some info on this, but I'm not sure if either package has a bug tracker. I'm able to reproduce this in both a conda env and outside of one. It seems to work if I install with conda, but never with pip.

It's a bit late for me, so I can write more on this tomorrow.

I'm looking at this again, but I'm not sure what's useful for me to add. I would say that this particular function has been a fount of numba parallelization errors for me (e.g. #4484, related to https://github.com/theislab/scanpy/pull/915/commits/9a783b65fbb954a0135cd75ceb6d72530af32540) so it could be useful test case? But please let me know anything in particular you'd like to know about my use case here.

Actually, I do have one thing to add. I noticed that using the version of my function specialized for sparse arrays was returning consistent results for equivalent data. It looks like it's still not returning the correct results, but instead is filling chunks of the result array with zeros after encountering a divide by zero. Here's an example:

 setup
import numpy as np
import scipy.sparse
import numba

def _gearys_c_mtx_csr(
    g_data, g_indices, g_indptr, x_data, x_indices, x_indptr, x_shape
):
    M, N = x_shape
    W = g_data.sum()
    out = np.zeros(M, dtype=np.float_)
    x_data_list = np.split(x_data, x_indptr[1:-1])
    x_indices_list = np.split(x_indices, x_indptr[1:-1])
    for k in numba.prange(M):
        out[k] = _gearys_c_inner_sparse_x_sparsevec(
            g_data,
            g_indices,
            g_indptr,
            x_data_list[k],
            x_indices_list[k],
            N,
            W,
        )
    return out

def _gearys_c_inner_sparse_x_sparsevec(
    g_data, g_indices, g_indptr, x_data, x_indices, N, W
):
    x = np.zeros(N, dtype=np.float_)
    x[x_indices] = x_data
    x_bar = np.sum(x_data) / N
    total = 0.0
    N = len(x)
    for i in numba.prange(N):
        s = slice(g_indptr[i], g_indptr[i + 1])
        i_indices = g_indices[s]
        i_data = g_data[s]
        total += np.sum(i_data * ((x[i] - x[i_indices]) ** 2))
    numer = (N - 1) * total
    # Expanded from 2 * W * ((x_k - x_k_bar) ** 2).sum(), but uses sparsity
    # to skip some calculations
    # fmt: off
    denom = (
        2 * W
        * (
            np.sum(x_data ** 2)
            - np.sum(x_data * x_bar * 2)
            + (x_bar ** 2) * N
        )
    )
    # fmt: on
    C = numer / denom
    return C

def test_decorator(inner_decorator, outer_decorator, g, X):
    @inner_decorator
    def _gearys_c_inner_sparse_x_sparsevec(
        g_data, g_indices, g_indptr, x_data, x_indices, N, W
    ):
        x = np.zeros(N, dtype=np.float_)
        x[x_indices] = x_data
        x_bar = np.sum(x_data) / N
        total = 0.0
        N = len(x)
        for i in numba.prange(N):
            s = slice(g_indptr[i], g_indptr[i + 1])
            i_indices = g_indices[s]
            i_data = g_data[s]
            total += np.sum(i_data * ((x[i] - x[i_indices]) ** 2))
        numer = (N - 1) * total
        # Expanded from 2 * W * ((x_k - x_k_bar) ** 2).sum(), but uses sparsity
        # to skip some calculations
        # fmt: off
        denom = (
            2 * W
            * (
                np.sum(x_data ** 2)
                - np.sum(x_data * x_bar * 2)
                + (x_bar ** 2) * N
            )
        )
        # fmt: on
        C = numer / denom
        return C

    @outer_decorator
    def _gearys_c_mtx_csr(
        g_data, g_indices, g_indptr, x_data, x_indices, x_indptr, x_shape
    ):
        M, N = x_shape
        W = g_data.sum()
        out = np.zeros(M, dtype=np.float_)
        x_data_list = np.split(x_data, x_indptr[1:-1])
        x_indices_list = np.split(x_indices, x_indptr[1:-1])
        for k in numba.prange(M):
            out[k] = _gearys_c_inner_sparse_x_sparsevec(
                g_data,
                g_indices,
                g_indptr,
                x_data_list[k],
                x_indices_list[k],
                N,
                W,
            )
        return out

    g = scipy.sparse.csr_matrix(g)
    X = scipy.sparse.csr_matrix(X)

    return _gearys_c_mtx_csr(
        g.data, g.indices, g.indptr, X.data, X.indices, X.indptr, X.shape
    )

identity = lambda x: x

def gen_X(m, n, idx):
    X = scipy.sparse.random(m, n, density=0.1, format="csr")
    X[idx, :] = 0
    return X

X = gen_X(100, 100, [2, 12, 90])
g = scipy.sparse.random(100, 100, format="csr", density=.1)

Input arrays are g and X, both sparse csr arrays, with shape constraints g.shape = (M, M), x.shape = (N, M), and the output of the function is a numpy array of shape N. In this case X has constant values on rows 2, 12, and 90 which will cause division by zero in the inner function.
Cases
Python
test_decorator(identity, identity, g, X)

array([1.01459874, 0.9593495 ,        nan, 0.92378192, 0.98912519,
       1.11105868, 0.9237929 , 1.00199953, 1.0473778 , 0.89726875,
       0.91095727, 0.98098525,        nan, 1.06641939, 0.8550019 ,
       1.03667488, 1.04289153, 0.96342758, 1.16369452, 0.8795137 ,
       0.95353743, 0.94629377, 1.05775005, 0.82455052, 1.16925157,
       1.06700197, 0.89021435, 1.07046724, 0.98635352, 1.03487087,
       0.87623032, 1.13291671, 1.1144185 , 0.98632352, 0.91674741,
       0.98358401, 0.87575519, 0.78970287, 0.82850676, 1.0575536 ,
       1.14980637, 1.25518776, 0.97476218, 1.17197319, 0.99370747,
       0.94696166, 1.10395619, 1.09552527, 1.04145689, 1.00786312,
       1.12144619, 0.87249599, 1.02204677, 0.80832584, 1.07331229,
       1.03909514, 0.97139034, 0.86515497, 0.97439893, 1.12686683,
       1.12114041, 0.86674782, 0.78915942, 0.94386858, 1.0714797 ,
       0.77890142, 1.22414133, 0.90829404, 1.0304547 , 1.03939956,
       0.79671678, 0.94347584, 1.09156576, 0.93155959, 0.99819332,
       0.86329434, 0.94059095, 1.09062337, 0.90695409, 0.87844441,
       0.9802381 , 0.9453733 , 0.9199573 , 1.01535701, 1.05485224,
       0.99843007, 0.99859551, 0.98578795, 0.94867376, 1.02783281,
              nan, 0.94921226, 0.92394616, 0.99367206, 0.92030381,
       0.90544909, 0.88176731, 0.87415911, 1.0410088 , 1.0085156 ])

single thread | multi thread
test_decorator(numba.njit(), numba.njit(parallel=True), g, X)

array([1.01459874, 0.9593495 , 0.        , 0.        , 0.        ,
       0.        , 0.        , 1.00199953, 1.0473778 , 0.89726875,
       0.91095727, 0.98098525, 0.        , 0.        , 0.8550019 ,
       1.03667488, 1.04289153, 0.96342758, 1.16369452, 0.8795137 ,
       0.95353743, 0.94629377, 1.05775005, 0.82455052, 1.16925157,
       1.06700197, 0.89021435, 1.07046724, 0.98635352, 1.03487087,
       0.87623032, 1.13291671, 1.1144185 , 0.98632352, 0.91674741,
       0.98358401, 0.87575519, 0.78970287, 0.82850676, 1.0575536 ,
       1.14980637, 1.25518776, 0.97476218, 1.17197319, 0.99370747,
       0.94696166, 1.10395619, 1.09552527, 1.04145689, 1.00786312,
       1.12144619, 0.87249599, 1.02204677, 0.80832584, 1.07331229,
       1.03909514, 0.97139034, 0.86515497, 0.97439893, 1.12686683,
       1.12114041, 0.86674782, 0.78915942, 0.94386858, 1.0714797 ,
       0.77890142, 1.22414133, 0.90829404, 1.0304547 , 1.03939956,
       0.79671678, 0.94347584, 1.09156576, 0.93155959, 0.99819332,
       0.86329434, 0.94059095, 1.09062337, 0.90695409, 0.87844441,
       0.9802381 , 0.9453733 , 0.9199573 , 1.01535701, 1.05485224,
       0.99843007, 0.99859551, 0.98578795, 0.94867376, 1.02783281,
       0.        , 0.        , 0.        , 0.        , 0.92030381,
       0.90544909, 0.88176731, 0.87415911, 1.0410088 , 1.0085156 ])

A series of values following the division by zero are not written to. It looks like this is the rest of the chunk being operated on.
single thread | single thread
test_decorator(numba.njit(), numba.njit(), g, X)

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-4-d77f03d5429f> in <module>
----> 1 test_decorator(numba.njit(), numba.njit(), g, X)

<ipython-input-1-329522ca7ee2> in test_decorator(inner_decorator, outer_decorator, g, X)
    110     X = scipy.sparse.csr_matrix(X)
    111
--> 112     return _gearys_c_mtx_csr(
    113         g.data, g.indices, g.indptr, X.data, X.indices, X.indptr, X.shape
    114     )

ZeroDivisionError: division by zero

Multi thread | multi thread
test_decorator(numba.njit(parallel=True), numba.njit(), g, X)

This kills the process with:
>>> test_decorator(numba.njit(parallel=True), numba.njit(parallel=True), g, X)
Terminating: Nested parallel kernel launch detected, the workqueue threading layer does not supported nested parallelism. Try the TBB threading layer.
Terminating: Nested parallel kernel launch detected, the workqueue threading layer does not supported nested parallelism. Try the TBB threading layer.
Terminating: Nested parallel kernel launch detected, the workqueue threading layer does not supported nested parallelism. Try the TBB threading layer.
[1]    49964 abort      python3

Interestingly, if I use the tbb or omp threading backend (in a conda environment) all of the numba compiled functions raise division by zero errors. None have the numpy warning behavior.

I suspect this all comes down to the error_model and where it's been defined. In case 1 that foo is defined with njit means it will have the default cpython error model, I think this will lead to code generation with the div-by-zero check in it. Calling this cpython error model function from a parallel=True function, which has the error_model set as 'numpy'

Looking into the tbb and omp backends a bit more, they seem to both sorta support setting the error_model. They will allow the computation to run, and give nan results (same as with the plain python version), but won't warn. This occurs with both the sparse and dense versions of the function.

Sorry to hear this, that one of TBB or OpenMP is not available is unfortunate.

I could probably give some info on this, but I'm not sure if either package has a bug tracker. I'm able to reproduce this in both a conda env and outside of one. It seems to work if I install with conda, but never with pip.

The wheels for OSX are not built with TBB or OpenMP support at present, we're looking at enabling this. The main issue is that even if the support is added, there's a reasonable chance that the libraries won't be found due to various inconsistencies in paths etc potentially present in pip installed environments. There's also this: https://discuss.python.org/t/how-to-disable-vendoring-of-libraries-in-wheel/5010. With conda packages, full support for the all the threading backends is built in and the threading libraries are also available as conda packages and everything "just works" as you've discovered.

It's a bit late for me, so I can write more on this tomorrow.

I'm looking at this again, but I'm not sure what's useful for me to add. I would say that this particular function has been a fount of numba parallelization errors for me (e.g. #4484, related to theislab/scanpy@9a783b6) so it could be useful test case? But please let me know anything in particular you'd like to know about my use case here.

The threading backends do not all work in the same way, some support more complex types of concurrent behaviours (e.g. fork(2) and use from multiple threads) and it's important to select a threading layer that matches the behaviours present in your application. There's more information on this in this documentation. In terms of the design of these threading backend, effort has been put in to try and detect situations a given threading backend cannot handle and report this to the user.

I think the key thing here is to differentiate between code generation and code execution. The error_model impacts code generation and what is actually compiled to executable code, the threading backend impacts the execution of this executable code.
During execution, in the case of an exception being raised, the mechanics of this will happen irrespective of the threading backend. What matters is whether the "main" thread (the one that instigated the execution of the compiled function) "notices" the exception.
In the case of OpenMP and TBB, (usually) the main thread joins the execution pool that does the work in the threading backend. If an exception occurs in the parallel region and it happens to be on the main thread, then the main thread will pick it up and raise on return to the interpreter. In the workqueue threading layer, the main thread does not join the execution pool, it just hands schedules the work for the threads in the pool, as a result, if there's an exception raised it's impossible (at present) for the main thread to know about it.
This whole area needs work!

@ivirshup another thought about the mysterious zeros appearing under parallel=True. I think this is due to a thread in the workqueue threadpool hitting the div-by-zero and an exception being raised, so it stops, but due to the issue with exceptions not transmitting back to Python the only observable effect is that there's some zeros present. This is highlighted by doing:
NUMBA_NUM_THREADS=1 NUMBA_THREADING_LAYER=workqueue python <the reproducer>

with just one thread running, as soon as it hits a div-by-zero it stops processing and the rest of the output is just the zeros from the original allocation (if you use np.ones to allocate instead, the matrix just has ones past the point of the first exception).


JavaScript-lesson

[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b8906861-1357-0292-ce64-9a8db681a98c
Version Independent ID: 096eea5a-63a3-f6bb-5c7b-5a0ecbf69d37
Content: Share quickstart - Visual Studio Live Share
Content Source: docs/quickstart/share.md
Product: visual-studio-family
Technology: live-share
GitHub Login: @Chuxel
Microsoft Alias: clantz

Hey @keiron123jp closing this bug due lack of information. Please reopen it with more details if this issue persists. Thanks!


How to install egui on OpenSuse Tumbleweed Linux

Hello,
To install egui on OpenSuse Tumbleweed Linux do:
zypper install clang clang-devel clang-tools libxkbcommon-devel pkgconf-pkg-config libopenssl-devel libxcb-devel gtk3-devel libatk-1_0-0 fontconfig-devel

Then when you start in order to not give you warnings on wininit pointer you have to ser the XCURSOR_THEME temporarilly in a environment variable, so concatenate the following  XCURSOR_THEME=Adwaita before cargo run or the execution of your egui program. This will start the demo program without any error.
XCURSOR_THEME=Adwaita cargo run --release -p egui_demo_app

Note: If this could possiblly be placed on the starting page after the ubuntu and the fedora install instructions it would be really nice :-)
Thank you,
Best regards,
João Carvalho

Make a PR to add it and I'll merge it! :)


NPE in ODocument creation

OrientDB Version: 2.2.16
Java Version: 8
It's allowed to create ODocument with className null, but if you specify null oClass - it will through NPE.
new ODocument((String)null);  // It's OK
new ODocument((OClass)null); It's failing, **but in OrientDB 2.1.19 it was OK**.

fixed closing


feat: subscription policies

As part of https://github.com/chainbound/msg-rs/issues/13, we want to add the ability for subscribers to specify the "durability policy" that they want to use.
Here's how that would look like:

subscriber (ReqSocket or SubSocket) connects to a target, specifying the policy: if they would like to receive messages from the connection point onward, or if they want to also receive cached messages that were already broadcasted earlier than the connection time.
On the publisher's side there will always be a limit of how many messages to replay for the specific subscriber. Nonetheless, this will be useful for when clients are starting up to make sure they don't lose initial messages.

I can take this.

@quangkeu95 this issue is blocked by #13, which we still need to figure out how to implement. Will ping you once we have a better idea of how to move forward!


Error in Documentation

Current Behavior
Docu page https://dashboard.flowfuse.com/layouts/:
We currently offer three different layout options:
Grid
Fixed
Notebook
Tabs
Expected Behavior
"three" should be "four"
Steps To Reproduce
No response
Environment

Dashboard version:
Node-RED version:
Node.js version:
npm version:
Platform/OS:
Browser:

Have you provided an initial effort estimate for this issue?
I am not a FlowFuse team member

Thanks for reporting this bug, @dastriebel
While reviewing the issue you pointed out, I also noticed that the reference to the Tabs layout was missing from the "Getting Started" page under the "Layout Options" section. I've added both now.
https://github.com/FlowFuse/node-red-dashboard/pull/1267

Thanks for that, so a smal ishue ist grwowing ;-)


cilium on microk8s: core-dns, hostpath-provisioner and hubble fails.

Enabling cilium on microk8s results into cluster not working.

coredns stops working
hospath pod stops working
looking into pod describe state shows warning (not sure if related): failed to find plugin "cilium-cni" in path [/var/snap/microk8s/2551/opt/cni/bin]

ENV:
Distributor ID:	Ubuntu
Description:	Ubuntu 21.10
Release:	21.10
Codename:	impish
microk8s 1.22/stable (Single node physical server)

inspection-report-20211018_140644.tar.gz
Plugins
microk8s is running
high-availability: no
  datastore master nodes: <IP_ADDRESS>:19001
  datastore standby nodes: none
addons:
  enabled:
    cilium               # SDN, fast with full network policy
    dns                  # CoreDNS
    ha-cluster           # Configure high availability on the current node
    helm3                # Helm 3 - Kubernetes package manager
    metallb              # Loadbalancer for your Kubernetes cluster
    storage              # Storage class; allocates storage from host directory

Enabling cilium plugin causes

dns pod fails (No policy applied)
hostPath provision fails (storage)

.:53
[INFO] plugin/reload: Running configuration MD5 = be0f52d3c13480652e0c73672f2fa263
CoreDNS-1.8.0
linux/amd64, go1.15.3, 054c9ae
[INFO] <IP_ADDRESS>:38711 - 2364 "HINFO IN<PHONE_NUMBER>512818014.7305508818995797580. udp 57 false 512" NOERROR - 0 6.001203496s
[ERROR] plugin/errors: 2<PHONE_NUMBER>512818014.7305508818995797580. HINFO: read udp <IP_ADDRESS>:50304-><IP_ADDRESS>:53: i/o timeout
[INFO] <IP_ADDRESS>:40286 - 35177 "HINFO IN<PHONE_NUMBER>512818014.7305508818995797580. udp 57 false 512" NOERROR - 0 6.000765939s
[ERROR] plugin/errors: 2<PHONE_NUMBER>512818014.7305508818995797580. HINFO: read udp <IP_ADDRESS>:33405-><IP_ADDRESS>:53: i/o timeout
[INFO] <IP_ADDRESS>:58037 - 43993 "HINFO IN<PHONE_NUMBER>512818014.7305508818995797580. udp 57 false 512" NOERROR - 0 4.000622412s
[ERROR] plugin/errors: 2<PHONE_NUMBER>512818014.7305508818995797580. HINFO: read udp <IP_ADDRESS>:60612-><IP_ADDRESS>:53: i/o timeout
I1018 12:06:34.490262       1 trace.go:205] Trace[774965466]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.19.2/tools/cache/reflector.go:156 (18-Oct-2021 12:06:04.489) (total time: 30000ms):
Trace[774965466]: [30.000588379s] [30.000588379s] END
E1018 12:06:34.490290       1 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.2/tools/cache/reflector.go:156: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Get "https://<IP_ADDRESS>:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp <IP_ADDRESS>:443: i/o timeout
I1018 12:06:34.490319       1 trace.go:205] Trace[436340495]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.19.2/tools/cache/reflector.go:156 (18-Oct-2021 12:06:04.489) (total time: 30000ms):

Looking at pod describe:
Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Warning  FailedScheduling        66m                    default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
  Warning  FailedScheduling        65m                    default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
  Normal   Scheduled               58m                    default-scheduler  Successfully assigned kube-system/coredns-7f7f9dc75b-fq7wx to antaresinc-cluster
  Warning  FailedCreatePodSandBox  58m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cf739b3ee917271f23fd6187b9e9f5f6417ecca0a942aba36acbf2bc25ee0b80": failed to find plugin "cilium-cni" in path [/var/snap/microk8s/2551/opt/cni/bin]

Cilium Hubble
Warning  FailedCreatePodSandBox  34m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d54e5a6079466384ab662149f9dc6f3c6865a82669b06a8b953d0bcc84817f8f": failed to find plugin "cilium-cni" in path [/var/snap/microk8s/2551/opt/cni/bin]

Can you start fresh using microk8s.enable cilium and record exactly what errors occur? It is difficult to pinpoint a specific problem if you have tried three different approaches, because it's not clear which approach is causing the problems.
For reference, helm and cilium-cli direct install into microk8s is not the expected path. The official Cilium docs say that to install into microk8s, you should use microk8s enable cilium: https://docs.cilium.io/en/stable/gettingstarted/microk8s/#gs-microk8s

OK great, this is a different error (Readiness probe failed: Get "http://<IP_ADDRESS>:8181/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)). Now are there any errors or warnings in the cilium-agent pod logs?

Ah right, yes that's the one that I was looking for. Are there errros or warnings there?

nop. only info messages.
evel=info msg="Rewrote endpoint BPF program" containerID= datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=668 identity=4 ipv4= ipv6= k8sPodName=/ subsys=endpoint
level=info msg="Removed endpoint" containerID= datapathPolicyRevision=1 desiredPolicyRevision=1 endpointID=668 identity=4 ipv4= ipv6= k8sPodName=/ subsys=endpoint
level=info msg="New endpoint" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=1979 ipv4= ipv6= k8sPodName=/ subsys=endpoint
level=info msg="Resolving identity labels (blocking)" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=1979 identityLabels="reserved:health" ipv4= ipv6= k8sPodName=/ subsys=endpoint
level=info msg="Identity of endpoint changed" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=1979 identity=4 identityLabels="reserved:health" ipv4= ipv6= k8sPodName=/ oldIdentity="no identity" subsys=endpoint

ip route show
<IP_ADDRESS>/24 via <IP_ADDRESS> dev cilium_host src <IP_ADDRESS>
<IP_ADDRESS> dev cilium_host scope link

The above output is not complete. What IP range is used for connecting to the network? Is it a local network address range like 172.x.y.z or 10.x.y.z or 192.168.x.y?

default via <IP_ADDRESS> dev eno2 proto dhcp src <IP_ADDRESS> metric 100
<IP_ADDRESS>/24 via <IP_ADDRESS> dev cilium_host src <IP_ADDRESS>
<IP_ADDRESS> dev cilium_host scope link
<IP_ADDRESS>/24 dev eno2 proto kernel scope link src <IP_ADDRESS>
<IP_ADDRESS> dev eno2 proto dhcp scope link src <IP_ADDRESS> metric 100

I think the next steps would probably need to involve trying something like using an older Ubuntu version to see whether this is a regression in the combination of Ubuntu 21.10 and Cilium 1.10.x. I can tell you that I was able to successfully install microk8s+cilium on Ubuntu 20.04 LTS here with a Linux 5.10.x kernel. If you're able to try that out, it could help to confirm whether there is a problem with the integration to Ubuntu 21.10 or whether there is another problem in your environment.
In the mean time, I reached out to the cilium.io Slack community #development channel to see whether any others have experience running Cilium on this new version of Ubuntu yet, so hopefully we can find out whether that's the problem and whether there are any mitigations or fixes there.

Have you already tried installing the extra headers for 21.10?
sudo apt install linux-modules-extra-raspi

No, I haven't.

Have you already tried installing the extra headers for 21.10? sudo apt install linux-modules-extra-raspi

I just ran into this problem myself. It looks like the issue has reproduced and worked around here: https://github.com/cilium/cilium/issues/18131#issuecomment-988160016.
I'm not sure if that is the optimal solution (since it seems like it may be disabling some sort of reverse path filtering), but it did the trick for me.


How to stop swift mailer exception if host is not valid?

{
"message": "Connection could not be established with host emailpwd.com1 :stream_socket_client(): php_network_getaddresses: getaddrinfo failed: No such host is known. ",
"exception": "Swift_TransportException",
"file": "\vendor\swiftmailer\swiftmailer\lib\classes\Swift\Transport\StreamBuffer.php",
"line": 261,
"trace": [
{
"function": "{closure}",
"class": "Swift_Transport_StreamBuffer",
"type": "->"
},

This is not an issue with the Laravel framework.


Support setting the default language for code blocks in Hovers

It would be really helpful if we could provide the language that should be used for code blocks in Hovers. Currently we can pass a string of markdown (where code will be uncoloured) or a chunk of text where the whole thing will be considered code in a language.
In Dart, comments are already markdown, so we have code like this:
/// This is a great method.
///
/// Here is some example code:
///
///     final name = "Danny";
///     print(name);
danny() {

}

The string we get is exactly like that, minus the /// . In order to get Dart highlighting code the code, we would have to split the document up, detect the code blocks and then created MarkedStrings with the language set to Dart as each one. This is quite complicated because there are other things that are indented the same way with blank lines before/after (like nested lists). It would be great if we could just set the code language and pass markdown directly.

You can use the normal <lang_name>code

@jrieken I understand that, but it means figuring out where the code blocks are and rewriting them. Code is already doing Markdown parsing, so it seems a bit silly for us to have to do it too. All language extensions comments are probably going to have code blocks in the same language, so this seemed like it could save duplicated effort of us all writing our own markdown re-writer if comments are already markdown and contain code blocks?

In Dart, comments are already markdown, so we have code like this:

Maybe it's a formatting issue with GH but I really don't see any markdown in that sample, esp no code-block

The lines indented by 4 spaces are code, but maybe it wasn't a very clear example.
Here's a better (real) example; the Future<T> class. This is the real dartdoc from the class. We get this string (stripped of the comment markers) and do some basic tweaking of it.  Code's rendering in hovers does correctly interpret these things indented with 4-spaces as code, but it doesn't know that they're dart so they don't get syntax highlighting.
Since this comment isn't written by us (extension author) but the end user, we can't ask them to add backticks (and it's also not the convention, as in Dart we know it's Dart). Surrounding the code blocks with backticks ourselves (in the hover provider) is possible but does mean we'll have to parse the Markdown on some level (because nested lists look a little like code blocks; they have blank lines before/after and are indented with 4 spaces).
I'm not against rewriting the markdown ourselves, but I think a more elegant solution would be if we could just tell Code that all code blocks in this markdown should be considered Dart. If you don't think that's a good feature; that's fine; I just wanted to suggest it before I embarked on Markdown-parsing :-)
Hope this is clearer!
/**
 * An object representing a delayed computation.
 *
 * A [Future] is used to represent a potential value, or error,
 * that will be available at some time in the future.
 * Receivers of a [Future] can register callbacks
 * that handle the value or error once it is available.
 * For example:
 *
 *     Future<int> future = getFuture();
 *     future.then((value) => handleValue(value))
 *           .catchError((error) => handleError(error));
 *
 * A [Future] can complete in two ways:
 * with a value ("the future succeeds")
 * or with an error ("the future fails").
 * Users can install callbacks for each case.
 * The result of registering a pair of callbacks is a new Future (the
 * "successor") which in turn is completed with the result of invoking the
 * corresponding callback.
 * The successor is completed with an error if the invoked callback throws.
 * For example:
 *
 *     Future<int> successor = future.then((int value) {
 *         // Invoked when the future is completed with a value.
 *         return 42;  // The successor is completed with the value 42.
 *       },
 *       onError: (e) {
 *         // Invoked when the future is completed with an error.
 *         if (canHandle(e)) {
 *           return 499;  // The successor is completed with the value 499.
 *         } else {
 *           throw e;  // The successor is completed with the error e.
 *         }
 *       });
 *
 * If a future does not have a successor when it completes with an error,
 * it forwards the error message to the global error-handler.
 * This behavior makes sure that no error is silently dropped.
 * However, it also means that error handlers should be installed early,
 * so that they are present as soon as a future is completed with an error.
 * The following example demonstrates this potential bug:
 *
 *     var future = getFuture();
 *     new Timer(new Duration(milliseconds: 5), () {
 *       // The error-handler is not attached until 5 ms after the future has
 *       // been received. If the future fails before that, the error is
 *       // forwarded to the global error-handler, even though there is code
 *       // (just below) to eventually handle the error.
 *       future.then((value) { useValue(value); },
 *                   onError: (e) { handleError(e); });
 *     });
 *
 * When registering callbacks, it's often more readable to register the two
 * callbacks separately, by first using [then] with one argument
 * (the value handler) and using a second [catchError] for handling errors.
 * Each of these will forward the result that they don't handle
 * to their successors, and together they handle both value and error result.
 * It also has the additional benefit of the [catchError] handling errors in the
 * [then] value callback too.
 * Using sequential handlers instead of parallel ones often leads to code that
 * is easier to reason about.
 * It also makes asynchronous code very similar to synchronous code:
 *
 *     // Synchronous code.
 *     try {
 *       int value = foo();
 *       return bar(value);
 *     } catch (e) {
 *       return 499;
 *     }
 *
 * Equivalent asynchronous code, based on futures:
 *
 *     Future<int> future = new Future(foo);  // Result of foo() as a future.
 *     future.then((int value) => bar(value))
 *           .catchError((e) => 499);
 *
 * Similar to the synchronous code, the error handler (registered with
 * [catchError]) is handling any errors thrown by either `foo` or `bar`.
 * If the error-handler had been registered as the `onError` parameter of
 * the `then` call, it would not catch errors from the `bar` call.
 *
 * Futures can have more than one callback-pair registered. Each successor is
 * treated independently and is handled as if it was the only successor.
 *
 * A future may also fail to ever complete. In that case, no callbacks are
 * called.
 */
abstract class Future<T> {
  // <snip>
}

Thanks for clarifying - I actually didn't know that you can format code blocks using space indentation... Now it makes sense to me.
Thinking of a way to achieve this... We have this and it should be save to access the modeId of the current model in case the value is missing. I'll investigate

That means we won't allow to set the default (unless a really good idea for that comes up) but simply take the context as default

Ah sorry, I should've provided a better example to start and maybe it would've been more obvious what I meant!
I don't fully understand the last comment; do you mean you'll infer it from the language of the HoverProvider rather than letting us set it? (if so, that seems perfect to me!)

Yeah - it would be the default when no language is set. Slightly on thin ice with that cos it changes existing behaviour but I'll give it a shot beginning of September

Awesome; thank you! :+1:

I'm guessing this isn't the expected behavior, as dart code doesn't syntax highlight either hover or markdown comments in code...Or is it because you haven't updated Dart-Code to utilize this functionality?
@DanTup

How could I verify this works?

@octref It did work when I tested it in Insiders 18 days ago (see https://github.com/Dart-Code/Dart-Code/issues/160#issuecomment-246192748) but I haven't tested it recently. I'll make a note to give it another look tomorrow when I'm at a PC.

@octref Indeed, it seems broken now; the colouring is all gone (was like that before I updated today - was probably a week old? and with todays version).

@DanTup It's moved to October.
@chrmarti As this is not working and moved to Oct I'm removing verification-needed to get it out of the query.

@alexandrudima Is this due to the mode/tokenization changes you made?

Yeah - looks like a regression from https://github.com/Microsoft/vscode/commit/04f43c375c4f6b40107de54aa96729d17a5d6079 where the fallback to the editor mode id got lost

@jrieken reopening to assess if this is a possible candidate for the stable branch

Removing the candidate for September stable since it is not a regression.

Fixed in the latest.

Hover works, code in comment doesn't.

I'm looking at this file: https://github.com/angular/angular.dart/blob/master/lib/formatter/filter.dart
Also wondering, is this something we should do? In case of angular.dart there are also a lot of HTML code in comment. Marking them as dart doesn't make sense, but providing a custom way for author to say which lang their code is in doesn't sounds good either.

@octref How should we know that half the comment is in dart and half the comment is in html ?

I thought the comment section are supposed to behave like hover, which marks everything as dart, in this dart file.
I think we should only allow github-style 3-backstick code blocks in comment.

This issue was about falling back to the language currently being shown in the editor if the markdown code block doesn't specify one.

Using @DanTup's dart extension this works for me. Looking at dart-lang/sdk's future.dart with an untrained eye. Marking verified.


Assign None as default for Optional Type

Hi,
I think it would be awesome if the default value for Optional types would be set to None.
Here is a snippet I hoped would work:
import attr

@attr.s(auto_attribs=True)
class Book:
    name: str
    author: Optional[str]

data = {'name': 'Fluent Python'}
Book(**data)

TypeError: __init__() missing 1 required positional argument: 'author'

You can very easily add = None to the class definition.
Also default args don't play nice if they are in the middle.  e.g.
@attr.s(auto_attribs=True)
class Book:
    name: str
    author: Optional[str]
    isbn: str

This is what I am currently doing - assigning None to every field in 20-field schema and it is looking ugly.
I believe this is extra information which is already indicated by the Optional type.

I'm not sure I agree that the optional type indicates that the default should be None; it only indicates that the value may be None.
That said, it does seem a reasonable default-default.  But I'm a little wary of doing that implicitly, and @euresti's point about defaults in the middle would have to be addressed.


Bulk load from JDBC source

TODO
Migrated from marklogic/data-movement#9.

@sammefford , should this be marked as ship ?

@jmakeig No, I need to first add docs to BulkLoadFromJdbc*.java

@vivekmuniyandi looks like I'm only getting in BulkLoadFromJdbcRaw this time.  Could you take a look and make sure the docs seem helpful?

Looks good to me.


tf.learn update

Update contrib/{learn,losses,layers,metrics,framework,slim} to reflect changes since r0.9 branch split.

Ignoring CLA because all commits are cherry-picks.

Giving up. This is not worth the risk of breaking all sorts of other things.


[Direct3D12] Update ResourceDescription.cs

Methods Texture2D and Texture3D doesn't work because use enum ResourceDimension.Texture1D instead of ResourceDimension.Texture2D and ResourceDimension.Texture3D

Thanks it is merged by commit 2a174b019cc1707d9f7512f3ab89306da077a9b2


Scanner struggles with reflections

I have the ability for users to upload a photo image of the VIN barcode. I notice that if there's any sort of reflection at all, the scanner cannot find the barcode. Does anyone have ideas to improve performance in that area?
Here is what I'm using currently:
quagga.decodeSingle({
    decoder: {
        readers: ['code_39_vin_reader']
    },
    locate: true, // locate barcode in the image, regardless of barcode orientation
    src: URL.createObjectURL(e.target.files[0]) // pass local URL of uploaded image
})

You can see the issue in action by trying the attached images at this link: https://serratus.github.io/quaggaJS/examples/file_input.html
Doesn't work:

Works:

I've been playing with locate: false and it does seem to be quite a bit faster at scanning at much less battery consumption.  I haven't noticed a significant increase in accuracy of it, or ability to read things that it couldn't read with locate: true, though, after a few hours of using it to scan books and things like that.  I have noticed as I spent quite a bit of time scanning in weird lighting today, that significant differences in brightness across a barcode will fail to register, much like this.  So it's not just the VIN reader, seems to probably be a problem with determining overall what is in the barcode.


Diarization is not working fine for all audios

Diarization is not working fine for me. For some audios, all segments are identifies by Speaker 0.  Are there some solutions to improve diarization quality ?

I had the same problem

I just ran into this. Do you get the same results if you were to try the transcription on just a few minutes of your audio at a time?
I have an audio file that's 41 minutes long and has about 16 speakers, total, and every single speaker comes out as "Speaker 0" in the transcript. However, if I extract a 5-minute chunk of the audio file and transcribe just that chunk, I get pretty good results, with to similar-sounding speakers sometimes getting categorized together (understandable, though).
I have also noticed that I get different speaker identification in parallel vs. non-parallel diarize.py with the five-minute clip. For my clip, the parallel version looked more correct (the actual transcript was identical between the two; it was just the speaker identification that was more wrong in the non-parallel version).

@GuyPaddock , can you upload the audio file to reproduce?

Sadly, it's a student project with some IP I can't share publicly, but if there's some private way to share it I could.

Also, thank you for the fast reply! I am happy to help any way that I can.

you can upload it to a drive link and share it with me
<EMAIL_ADDRESS>

Shared! Let me know if you don't receive an email from Drive.

Got it, I'm currently working om pushing some updates to this repo, I'll debug it after

Thanks!

BTW I am not using stems because I got better transcription with this particular audio file without it. So, my command line looks like this:
python diarize_parallel.py --audio ./input.wav --language en --whisper-model large-v3 --device 'cuda' --no-stem

@MahmoudAshraf97 Today I tried actually listening to the audio at the timestamps indicated in the SRT for the file that I shared with you and I'm finding that the times don't line up with the line that's been spoken. In many cases, the timestamps in the SRT are 5-15 seconds before the text that was transcribed, and might not even contain the entire line of dialogue. I wonder if that's impacting speaker attribution?

facing the same problem, is there any update? My situation is that when I use a single person's 6 seconds audio, it could be identified into 2 speakers, however, when I use a 30+ minutes and more than maybe 8 people audio, it could only identify 1 speakers which cause all speakers are 'Speaker 0'.


restore original behavior of WithUser

cc @strib

Oh, this is what was breaking the app yesterday!  @mmaxim thanks for finding it.  It resulted in a UI exception that was not easy to repro. We have a ticket in this week's sprint to look into it.
CC @nathunsmitty  / @mlsteele / @chrisnojima  who were affected.

Also cc @heronhaye it wasn't related to GAPU as you said.


chore: update fork url

I have limited experience in this area, but I have taken the following steps:

Examined how it is done in the Foundry repository.
Collected hardcoded URLs in the Solidity file.
Added names based on the URLs and referenced them in Solidity.
Ran cargo test.

The hardcoded URLs in the file crates/edr_solidity_tests/tests/testdata/default/cheats/RpcUrls.t.sol crates/tools/js/benchmark/src/solidity-tests.ts  have not been modified.
#643

Thanks for the changes @iosh! Please also fix the Clippy/formatting issues mentioned here: https://github.com/NomicFoundation/edr/pull/707#pullrequestreview-2394794060

Sorry, I forgot to run cargo clippy. I have now re-run cargo clippy, cargo fmt, and cargo test.

Hi @agostbiro can help me find the next issue I can try?


xtensa: include core-macros.h on esp32 only

core-macros.h includes other files not part of the xtensa HAL, make this
esp32 specific now until the headers are recovered.
Fixes #31301

PTAL this #31313

PTAL this #31313


feat: implement transaction builder and script handlers mechanism

@quake Please help check the pattern is acceptable. See the file example.go.


Add link to the new playground in the dashboard

We should link to the new playground in the sidebar of the dashboard

Text (Playground with experimental flair) and link to the new playground
And for the old playground add deprecated next to it

@xmonader  the two links should have the same label?  Playground

with experimental flair or anything that indicates it's a new thing (maybe alpha flair or something)

Status: PR #425 ready for review

Status: merged, waiting for release

verified


Update ip_pools.md

The increased limit on IP pools was release to customers on 2016/09/14

duplicate of #2178


Vpdatum on HPC cluster

Hi, can the vpdatum package be used on a linux-based HPC cluster, or an ubuntu computer? I have only been able to use the package on Windows succesfully.

VyperDatum at currently relies on NGS' VDatum package which is windows only.  The updated version we are working on will remove that restriction and should be cross platform.

Thank you!

Hi Barry, do you know more or less when the updated version will be released?
Thank you

There are a few phases for this.  Get a working alpha and test in the Mississippi River in the next two months.  Have a working national framework beta for a local install in roughly four months.  Get our datasets into the Proj CDN for a more seamless final solution, I'm not sure how long that part would take.


what does chmod 2>&1 mean?

it's in the deploy:writable task from common recipe.Line: 243

'2' is 'stderr', and '1' is 'stdout'.
So basically, all error messages (stderr) will be displayed as standard output.

the output of command "chmod 2>&1" is like this
chmod: missing operand
Try `chmod --help' for more information.
so what exactly is line:243 checking for
if (strpos(run("chmod 2>&1; true"), '+a') !== false)

chmod 2>&1 seems to return usage information on OS X, but the GNU coreutils chmod will respond with "missing operand". The latter also does not seem to provide a +a option at all. Here's the output of chmod --help for reference:
Usage: chmod [OPTION]... MODE[,MODE]... FILE...
  or:  chmod [OPTION]... OCTAL-MODE FILE...
  or:  chmod [OPTION]... --reference=RFILE FILE...
Change the mode of each FILE to MODE.
With --reference, change the mode of each FILE to that of RFILE.

  -c, --changes          like verbose but report only when a change is made
  -f, --silent, --quiet  suppress most error messages
  -v, --verbose          output a diagnostic for every file processed
      --no-preserve-root  do not treat '/' specially (the default)
      --preserve-root    fail to operate recursively on '/'
      --reference=RFILE  use RFILE's mode instead of MODE values
  -R, --recursive        change files and directories recursively
      --help     display this help and exit
      --version  output version information and exit

Each MODE is of the form '[ugoa]*([-+=]([rwxXst]*|[ugo]))+|[-+=][0-7]+'.

GNU coreutils online help: <http://www.gnu.org/software/coreutils/>
Full documentation at: <http://www.gnu.org/software/coreutils/chmod>
or available locally via: info '(coreutils) chmod invocation'

It’s for detecting a option.


feat: include statically analyzable dynamic imports in the eszip

I'm not sure why this was disallowed. This allows people to run deno compile with statically analyzable dynamic imports.

@lucacasonato see here https://github.com/denoland/eszip/pull/115/files#diff-ff42aba765ab96f68cb209933964efa01404f135126af4e1cfe45d5e7c1fd0e9R936


SQL Client: Unable to load DLL 'sni.dll'

Downloaded latest CLI build (beta-2175), created new project with dotnet new, added aspnetcidev myget feed to nuget.config, added AspNetCore, MVCCore, and EFCore dependencies, created context and model for existing database, SQL Server query fails with:

dbug: Microsoft.EntityFrameworkCore.Query.Internal.SqlServerQueryCompilationContextFactory[5]
TRACKED: True
(QueryContext queryContext) => IEnumerable _ShapedQuery(
queryContext: queryContext,
shaperCommandContext: SelectExpression:
SELECT [p].[Id], [p].[Name]
FROM [Products] AS [p]
,
shaper: UnbufferedEntityShaper
)

dbug: Microsoft.EntityFrameworkCore.Storage.Internal.SqlServerConnection[3]
Opening connection to database 'ProductsDb' on server 'localhost\SQLEXPRESS'.
fail: Microsoft.EntityFrameworkCore.Query.Internal.SqlServerQueryCompilationContextFactory[1]
An exception occurred in the database while iterating the results of a query.
System.DllNotFoundException: Unable to load DLL 'sni.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)
at System.Data.SqlClient.SNINativeMethodWrapper.UnmanagedIsTokenRestricted(IntPtr token, Boolean& isRestricted)
Easiest way to see this in action is to download the cli-samples (https://github.com/aspnet/cli-samples), using HelloMvcApi add "Microsoft.EntityFrameworkCore.SqlServer": "1.0.0-rc2-*" to project.json dependencies, query anything with a context.
Apologies if this is a dupe of known issue in #3925, thought it was worth mentioning because it's occurring again, but, this time on any EF+SqlServer server connections- no tools/ef-commands here.
Also, I understand the churn in these repos. Let me know if there's no need to report such mundane/obvious issues right after changes in the core/cli nightlies.

After looking further I see this is not an issue with EF. Targeting System.Data.SqlClient directly and opening a connection throws the same exception ala dotnet/corefx#5252 and the issue probably belongs there

Keeping this open to track the issue from the EF side, but opened https://github.com/dotnet/corefx/issues/7518 for SQL Client to fix the issue.

This has been fixed in CLI. I verified sqlclient works with cli-samples + sql server on win7 and win10.
dotnet --info
.NET Command Line Tools (1.0.0-rc2-002351)

Product Information:
 Version:     1.0.0-rc2-002351
 Commit Sha:  0128033afd

Runtime Environment:
 OS Name:     Windows
 OS Version:  6.1.7601
 OS Platform: Windows
 RID:         win7-x64


AVS response time

Hi, I came across your great work in hopes of finding someone who might be able to improve the response time from AVS with an RPI. Check out: https://forums.developer.amazon.com/questions/30086/streaming-audio-to-avs-to-improve-unacceptable-res.html, amazon replied with help on how to do this, but it will probably take me weeks to figure it out. I was wondering if you might consider modifying your alexa_client.py project code in the ask function to enable chunked streaming vs file input. It would help with the 3-5 second response time. Mine is actually more like 5-10 seconds :(. I would love to see how it is done and will share if I figure it out, but my python skills are at a beginner level.  Thanks for your project!

Sorry for taking so long to get to this issue. I tried opening the link you pasted but it didn't work. Just went to the frontpage of the developer forums. Maybe this issue is already resolved by itself? I have not had any problem on my RPi. Thanks.


Fe homework 13

Metric
Value

alert_status
:white_check_mark: OK

bugs
0

code_smells
8

reliability_rating
A (0 Bugs)Note: A being best, E being worst

security_rating
A (0 Vulnerabilities)Note: A being best, E being worst

vulnerabilities
0


Cannot install and run theme from fresh clone

After cloning this repo, adding .env and running yarn:
❯ yarn dev
yarn run v1.22.21
$ yarn setup && next dev
$ yarn sync && yarn graphql:generate && yarn theme:generate
$ node -r dotenv/config ./build-utils/sync-storefront.mjs
✔ Successfully set editor values for [abc].
Enabled editor: true
Source theme id: horizon

exec error: Error: Command failed: swell storefronts push --id $SWELL_STOREFRONT_ID -t settings -s defaults -f ./config/defaults.json --secret-key $SWELL_SECRET_KEY --store $SWELL_STORE_ID

✔ Successfully pushed theme.horizon.memberships model to [abc] store.

✔ Successfully pushed theme.horizon.multiple_features model to [abc] store.

✔ Successfully pushed theme.horizon.multiple_panels model to [abc] store.

✔ Successfully pushed theme.horizon.press_mentions_carousel model to [abc] store.

✔ Successfully pushed theme.horizon.products_preview model to [abc]store.

✔ Successfully pushed theme.horizon.full_width_media model to [abc]store.

✔ Successfully pushed theme.horizon.reviews model to [abc] store.

✔ Successfully pushed theme.horizon.heading_with_text model to [abc] store.

✔ Successfully pushed theme.horizon.categories_preview model to [abc] store.

✔ Successfully pushed theme.horizon.divider model to [abc] store.

✔ Successfully pushed theme.horizon.quiz_question model to [abc] store.

{
  errors: {
    fields: {
      message: "Content field name '6563e6bfc6f1320011833ef2' should not start with a number",
      code: 'INVALID'
    }
  }
}

{
  errors: {
    fields: {
      message: "Content field name '6563e6bf9328cf00126e2330' should not start with a number",
      code: 'INVALID'
    }
  }
}

✔ Successfully pushed theme.horizon.quiz_transition model to [abc] store.

✔ Successfully pushed theme.horizon.category model to [abc] store.

✔ Successfully pushed theme.horizon.product model to [abc] store.

exec error: Error: Command failed: swell storefronts push --id $SWELL_STOREFRONT_ID -t editor -s defaults -f ./config/editor.json --secret-key $SWELL_SECRET_KEY --store $SWELL_STORE_ID

$ graphql-codegen --config codegen.yml -r dotenv/config
✔ Parse Configuration
✔ Generate outputs
$ node -r dotenv/config ./build-utils/generate-theme.mjs
file:///code/[abc]/build-utils/styles.mjs:15
  Object.entries(object).forEach(([key, value]) => {
         ^

TypeError: Cannot convert undefined or null to object
    at Function.entries (<anonymous>)
    at getVarsFromObject (file:///code/[abc]/build-utils/styles.mjs:15:10)
    at generateCssVars (file:///code/[abc]/build-utils/styles.mjs:33:21)
    at file:///code/[abc]/build-utils/generate-theme.mjs:13:19
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Node.js v18.18.2
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Swell is aware of this. Work around available here: https://github.com/orgs/swellstores/discussions/451

I am running into a similar issue, the posted workaround above did not work. Fresh install of Horizon, just trying to use the theme for the first time.
yarn run v1.22.21
warning ../../../package.json: No license field
$ yarn setup && next dev
warning ../../../package.json: No license field
$ yarn sync && yarn graphql:generate && yarn theme:generate
warning ../../../package.json: No license field
$ node -r dotenv/config ./build-utils/sync-storefront.mjs
✔ Successfully set editor values for xxx storefront XXXX.
Enabled editor: true
Source theme id: horizon

✔ Successfully updated settings configuration for xxx storefront XXX.

✔ Successfully pushed theme.horizon.divider model to xxx store.

✔ Successfully pushed theme.horizon.heading_with_text model to xxx store.

✔ Successfully pushed theme.horizon.full_width_media model to xxx store.

✔ Successfully pushed theme.horizon.categories_preview model to xxx store.

✔ Successfully pushed theme.horizon.memberships model to xxx store.

✔ Successfully pushed theme.horizon.multiple_panels model to xxx store.

✔ Successfully pushed theme.horizon.press_mentions_carousel model to xxx store.

✔ Successfully pushed theme.horizon.quiz_transition model to xxx store.

✔ Successfully pushed theme.horizon.reviews model to xxx store.

✔ Successfully pushed theme.horizon.quiz_question model to xxx store.

✔ Successfully pushed theme.horizon.products_preview model to xxx store.

✔ Successfully pushed theme.horizon.multiple_features model to xxx store.

✔ Successfully pushed theme.horizon.product model to xxx store.

✔ Successfully pushed theme.horizon.quiz model to xxx store.

✔ Successfully updated editor configuration for xxx storefront XXX.

✔ Successfully pushed theme.horizon.category model to xxx store.

✔ Successfully pushed theme.horizon.page model to xxx store.

warning ../../../package.json: No license field
$ graphql-codegen --config codegen.yml -r dotenv/config
✔ Parse Configuration
⚠ Generate outputs
  ❯ Generate to ./graphql.schema.json
    ✔ Load GraphQL schemas
    ✔ Load GraphQL documents
    ✖ GraphQL Document Validation failed with 9 errors;
      Error 0: Cannot query field "content" on type "SwellCategory".
- Swell Storefront/horizon/lib/graphql/que…
      Error 1: Cannot query field "productHighlights" on type "SwellProductContent".
    Swell Storefront/horizon/lib/graphql/que…
      Error 2: Cannot query field "expandableDetails" on type "SwellProductContent".
   Swell Storefront/horizon/lib/graphql/que…
      Error 3: Cannot query field "layoutOptions" on type "SwellProductContent".
   Swell Storefront/horizon/lib/graphql/que…
      Error 4: Cannot query field "showStockLevels" on type "SwellProductContent".
     Swell Storefront/horizon/lib/graphql/que…
      Error 5: Cannot query field "lowStockIndicator" on type "SwellProductContent".
      at / Swell Storefront/horizon/lib/graphql/que…
      Error 6: Cannot query field "calloutTitle" on type "SwellProductContent".
      at / Swell Storefront/horizon/lib/graphql/que…
      Error 7: Cannot query field "calloutDescription" on type "SwellProductContent".
      at / - Swell Storefront/horizon/lib/graphql/que…
      Error 8: Cannot query field "enableProductCounter" on type "SwellProductContent".
      at / - Swell Storefront/horizon/lib/graphql/que…
  ❯ Generate to lib/graphql/generated/sdk.ts
    ✔ Load GraphQL schemas
    ✔ Load GraphQL documents
    ✖ GraphQL Document Validation failed with 9 errors;
      Error 0: Cannot query field "content" on type "SwellCategory".
      Error 1: Cannot query field "productHighlights" on type "SwellProductContent".
      Error 2: Cannot query field "expandableDetails" on type "SwellProductContent".
      Error 3: Cannot query field "layoutOptions" on type "SwellProductContent".
      Error 4: Cannot query field "showStockLevels" on type "SwellProductContent".
      Error 5: Cannot query field "lowStockIndicator" on type "SwellProductContent".
      Error 6: Cannot query field "calloutTitle" on type "SwellProductContent".
      Error 7: Cannot query field "calloutDescription" on type "SwellProductContent".
      Error 8: Cannot query field "enableProductCounter" on type "SwellProductContent".
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.```


Extend search result fields

Is your feature request related to a problem?
Verdaccio search endpoints (/-/all or /v1/search) return standard fields like name, version, author, license... I'd like to extend custom fields into the search result.
 export function prepareSearchPackage(data: Package, time: unknown): any {
   ...
       name: version.name,
+      displayName: (version as any).displayName,
       description: version.description,

It's a very customized request, I can leave this in a branch, but I wonder if there's a way to move this into a plugin.
Describe the solution you'd like
Ideally, I can get a hook point like the middleware plugin to have a callback into the prepareSearchPackage function.
export function prepareSearchPackage(data: Package, time: unknown): any {
...
    const version: Version = data.versions[latest];
    const versions: any = { [latest]: 'latest' };
    const pkg: any = {
     ...
    };
    if (some_hook_point) {
      some_hook_point(pkg, version, versions);
    }

Describe alternatives you've considered
Maintainers can also reject this request if it's not considered useful for others.
Additional context
n/a

Could you elaborate a bit more? which custom fields? where those fields would be? I have an idea what you mean but maybe you can give me more.

Custom package solution that built on the top of NPM may add custom fields to the search endpoints to present custom information. For Unity package manager, it adds custom fields like displayName, unityVersion to every package meta file (package.json). This is the one I'm trying to solve.
I'm not sure this is generic enough for other usages. So I'm okay with rejection.

I'd vote to keep it, sounds interesting, unique consumer for npm search is, either npm or pnpm nowadays. Let's wait new search endpoint and see whether this fits there somehow.


⚠ Samsung has degraded performance

In 306107b, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 3302 ms

Resolved: Samsung performance has improved in 8a02fb3.


Slack link should work

See title. It should be a link inviting the user to the slack channel.

This works now.


Library Publication Requirements

the following is a document related to this: https://docs.google.com/document/d/1TA8iGRChZHwLuPxHVH-iP-anR5zPbtG2_6QQnkBYbv0/edit

see #7, #2, #6 for related issues


fix: eliminate SAML type import by flattening MetadataType to string union

Problem
Previously, users would need to import types to specify the saml MetadataType, like this:

import { UserPoolIdentityProviderSamlMetadataType } from 'aws-cdk-lib/aws-cognito';

metadata: {
metadataContent: samlMetadataContent,
metadataType: UserPoolIdentityProviderSamlMetadataType.FILE,
},

Issue number, if available:
Changes
Now, customers can simply specify "FILE" or "URL" without importing any types.

metadata: {
metadataContent: samlMetadataContent,
metadataType: 'FILE',
},

DX:

Corresponding docs PR, if applicable:
Validation

Checklist

[x] If this PR includes a functional change to the runtime behavior of the code, I have added or updated automated test coverage for this change.
[x] If this PR requires a change to the Project Architecture README, I have included that update in this PR.
[ ] If this PR requires a docs update, I have linked to that docs PR above.
[x] If this PR modifies E2E tests, makes changes to resource provisioning, or makes SDK calls, I have run the PR checks with the run-e2e label set.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Please fix the api check


WindowsIdentity.AuthenticationType throws under IIS but not IIS Express

When performing windows authentication I am accessing User.Identity.AuthenticationType to know if "Negotiate" or "NTLM" was used for authentication. On IIS Express this works, but when published to IIS I get this exception:
2017-01-06 12:20:55.109 -05:00 [Error] An unhandled exception has occurred while executing the request
System.UnauthorizedAccessException: Attempted to perform an unauthorized operation.
   at System.Security.Principal.WindowsIdentity.get_AuthenticationType()
   at IdentityServer4.Quickstart.UI.AccountController.<ExternalLogin>d__9.MoveNext() in C:\ballen\github\identity\IdSvr4\IdentityServer4\src\Host\Quickstart\Account\AccountController.cs:line 179

And I see the code is throwing here: https://referencesource.microsoft.com/#mscorlib/system/security/principal/windowsidentity.cs,315
So I am guessing that IIS Express, running as me as admin, is allowed to make the Win32 call, whereas IIS' app pool identity is not.
I think this permissions idea is sort of a red herring, though, as I'd expect the m_AuthType to be initialized when the windows identity is created. This SO article seems to corroborate it: https://stackoverflow.com/questions/11587305/unauthorizedaccessexception-after-using-logonuser/11588736#11588736
So I'm not sure where the WindowsIdentity is created, but it seems to make sense to pass in the auth type to the ctor, rather then defer to a Win32 call.
Originally opened here: https://github.com/IdentityServer/IdentityServer4/issues/650#issuecomment-270806080

See https://github.com/aspnet/IISIntegration/issues/231

Ok, closing as a dup. Thanks.

I'm on aspnetcore 2.1 and just inserted @this.User.Identity.AuthenticationType to a razor page. Shows 'NTLM' or 'Kerberos" on my local IISExpress machine (different browsers) and kaboom explodes on real IIS due to this UnauthorizedAccessException.
2018 is done soon, these tickets are more than 2 years old and all closed/solved in a way I don't know if just some workarounds have been applied here and there or this problem got reintroduced.

ASP.NET Core worked around the issue like this: https://github.com/aspnet/IISIntegration/pull/282/files
If you want a fix for WindowsIdentity you'll need to ask https://github.com/dotnet/corefx.


Update ROADMAP.md

Suggestion to add afterRead so you can create functions to alter objects before serving them -- more specifically, stripping passwords from user objects.

@cburatto you already can do this with toJSON: http://sailsjs.org/#!/documentation/reference/waterline/records/toJSON.html

Awesome, thanks for point that out! should I remove this request?

@cburatto Well done adding your feature request as a PR to the roadmap, definitely the right way to do it!  In this case since we have a similar feature implemented already, we'll just close this one.


Error: Timed out while waiting for handshake

Connecting with ssh manually works for me,
when i try to deploy using ssh_deploy i get
Error: Timed out while waiting for handshake
My config:
secret: grunt.file.readJSON('secret.json'),

environments: {
    theme: {
        options: {
            host: '<%= secret.theme.host %>',
            username: '<%= secret.theme.username %>',
            password: '<%= secret.theme.password %>',
            port: '<%= secret.theme.port %>',
            deploy_path: '/public_html/newfolder',
	readyTimeout: 99999,
            local_path: '../wp-content/themes/mytheme/',
            current_symlink: 'current',
            debug: console.log
        }
    },

The command:
grunt ssh_deploy:theme
The Output:
Running "ssh_deploy:theme" (ssh_deploy) task
Connecting :: myserver.com
DEBUG: Parser: STATE_INIT

hot to fix this issue?

+1 I have the same issue

+1 I have the same issue as well. It might be good to have an option of specifying your own timeout.


Update x/sys to support go 1.17

What does this PR do?
Updates x/sys to support go 1.17 and unblock brew. More details here: https://github.com/Homebrew/homebrew-core/pull/83413
Motivation
brew is blocked
Additional Notes
Please cut a release after merging this

@roopakv
Hello.
Looking at https://github.com/Homebrew/homebrew-core/pull/83413 , I'm a bit confused by your diagnostic, especially since this issue is lacking details about what is failing. Notably:

it seems like when you were discussing this issue, and if I'm looking at the history on https://github.com/Homebrew/homebrew-core/commits/c1700179ceb4878d580dac84fa996fd41e898914/Formula/traefik.rb , when you were running these tests, you were not on traefik 2.5, but on traefik 2.4 (which does not support Go 1.17). Am I wrong, can you clarify that?

As of now, if I'm running brew test (on Catalina) with:

mbp:traefik mpl$ brew info traefik
traefik: stable 2.5.0 (bottled), HEAD
Modern reverse proxy
https://traefik.io/
/usr/local/Cellar/traefik/2.5.0 (8 files, 99.4MB) *
  Poured from bottle on 2021-08-19 at 15:00:43
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/traefik.rb

I get
mbp:traefik mpl$ brew test --retry --verbose traefik
/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.3_2/bin/bundle install
Fetching gem metadata from https://rubygems.org/.........
...
Installing rubocop-performance 1.11.5
Bundle complete! 33 Gemfile dependencies, 72 gems now installed.
Bundled gems are installed into `../../../../../../usr/local/Homebrew/Library/Homebrew/vendor/bundle`
Removing rubocop-performance (1.11.4)
==> Testing traefik
/usr/bin/sandbox-exec -f /private/tmp/homebrew20210819-12914-1pyb6c8.sb ruby -W1 -- /usr/local/Homebrew/Library/Homebrew/test.rb /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/traefik.rb --verbose --retry
INFO[0000] Configuration loaded from file: /private/tmp/traefik-test-20210819-12915-g8tcsm/traefik.toml
==> curl -sIm3 -XGET http://<IP_ADDRESS>:52417/
==> curl -sIm3 -XGET http://<IP_ADDRESS>:52416/dashboard/
==> /usr/local/Cellar/traefik/2.5.0/bin/traefik version 2>&1

which seems to be working.

If the problems are only limited to ARM (not sure, as the linked PR is confusing on that aspect), which would explain why the test is working on my non-ARM laptop, then can you tell us what exactly is failing on ARM? it is true that we do not run any tests for ARM builds, but at least we provide binaries for all architectures, so we do know that at least traefik 2.5 is building on ARM.

@mpl
Take a look at this: https://github.com/golang/go/issues/45702
Basically with go 1.17 traefik will not build unless x/sys is upgraded
I can write up something a bit more detailed in a bit :)

@roopakv currently traefik (v2.5.0) is built with go1.17

@ldez interesting. Let me see why the arm bottle failed again. I was sure that all failures I collected were the x/sys issue.

@ldez interesting. Let me see why the arm bottle failed again. I was sure that all failures I collected were the x/sys issue.

Yeah, in my point 3. above, it was implicit but I meant we are building traefik, for ARM, with go 1.17.

If someone knows which commit in x/sys fixes the issue, then people could check if the version they use is has that commit; do we know which specific commit fixes the issue?

@mpl from the logs on the golang brew upgrade:

this is definitely the x/sys issue. Upgrading x/sys as mentioned here: https://github.com/golang/go/issues/46763
and https://github.com/golang/go/issues/45702 will unblock building traefick with go 1.17 and unblock homebrew as well.

Can you provide the hash or the tag of traefik that you are trying to build?

531a8ff24843bcd903c195dcf418dbb0eb5c8e0d with go 1.17

As suggested by @thaJeztah, could you point the exact commit in x/sys that fixes the problem?

@ldez this is the commit that fixes the problem in x/sys https://github.com/golang/sys/commit/a76c4d0a0096537dc565908b53073460d96c8539


Account for adoptium producing incorrect metadata for staticlibs

Checklist

[ ] You added tests to cover the change
[ ] mvn clean install build and test completes
[ ] You changed or added to the documentation

Codecov Report

Merging #195 (6b223f0) into main (4ff0fb5) will increase coverage by 0.09%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main     #195      +/-   ##
============================================
+ Coverage     64.57%   64.67%   +0.09%
- Complexity      202      204       +2
============================================
  Files           118      118
  Lines          3162     3168       +6
  Branches        466      467       +1
============================================
+ Hits           2042     2049       +7
+ Misses          888      887       -1
  Partials        232      232

Impacted Files
Coverage Δ

...otlin/net/adoptopenjdk/api/v3/AdoptReposBuilder.kt
69.86% <100.00%> (ø)

...topenjdk/api/v3/mapping/adopt/AdoptBinaryMapper.kt
92.41% <100.00%> (+0.32%)
:arrow_up:

...penjdk/api/v3/dataSources/mongo/InternalDbStore.kt
100.00% <0.00%> (+5.00%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4ff0fb5...6b223f0. Read the comment docs.

@jerboaa, it seems that the metadata file is only incorrect for staticlibs, the rest are fine

it seems that the metadata file is only incorrect for staticlibs, the rest are fine

At the risk of asking a silly question, have we now fixed the metadata generation too?

@sxa https://github.com/adoptium/ci-jenkins-pipelines/commit/df113be6717b7c56acfd187b27ce828d45bf9fe9#diff-b4bb07e2f448f9ad2986d69423ad0b704dcdc4d217d947f465e6ace3fde7a4b3 should have done it

@sxa adoptium/ci-jenkins-pipelines@df113be#diff-b4bb07e2f448f9ad2986d69423ad0b704dcdc4d217d947f465e6ace3fde7a4b3 should have done it

Yes, and it did:
https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-linux-aarch64-hotspot/967/artifact/workspace/target/OpenJDK11U-static-libs-glibc_aarch64_linux_hotspot_2021-10-04-09-34.tar.gz.json
So it's weird to have this special case.

So it's weird to have this special case.

Unless the API is tripping up on the ones published before that fix went in.

So it's weird to have this special case.

Unless the API is tripping up on the ones published before that fix went in.

Yes, that's the issue. Some nightlies had the wrong metadata. This patch is a band-aid to fix it.


Unable to  discover  ServiceMonitor  created in different namespace

What did you do?
I am using all yamls from  manifests  located in Kube-prometheus. I have services running in different namespace(sandbox). I want to create new service monitors for all those services in the same namespace(sandbox) or monitoring namespace. I notice that prometheus-prometheus.yaml has
serviceMonitorNamespaceSelector: {}
 serviceMonitorSelector: {}

so they will choose all servicemonitors from all  namepaces.  But the  mongoservicemonitor deployed in either  monitoring or  sandbox are never discovered by the prometheus service running in monitoring.
e.g.  mongoservicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: mongodb
  labels:
    name: mongodb
spec:
  namespaceSelector:
    any: true
  selector:
    matchLabels:
       name: mongodb
  endpoints:
  - targetPort: 27017

e.g. mongodb.yaml (service)
apiVersion: v1
kind: Service
metadata:
  name: mongodb
  labels:
    name: mongodb
spec:
  type: ClusterIP
  selector:
    app: mongodb
  ports:
  - port: 27017
    targetPort: 27017

What did you expect to see?
I don't see the  mongodb service monitor in the prometheus service discovery.  I am using latest release of manifests from github and Kubernetes version v1.10.1.
@brancz

This could be due to multiple reasons. Could you share the generated configuration that is shown in the /config page of the Prometheus UI?

Full /config page shared on google drive

https://drive.google.com/file/d/1mHzDdUb5FZXjk6q6Mc_U--8FkRZP9n2W/view?usp=sharing
mongodbmonitor config is shown here

job_name: sandbox/mongodbmonitor/0
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
kubernetes_sd_configs:

role: endpoints
relabel_configs:
source_labels: [__meta_kubernetes_service_label_name]
separator: ;
regex: mongodb
replacement: $1
action: keep
source_labels: [__meta_kubernetes_pod_container_port_number]
separator: ;
regex: "27017"
replacement: $1
action: keep
source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
separator: ;
regex: (.*)
target_label: endpoint
replacement: "27017"
action: replace

Could you try:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: mongodb
  labels:
    name: mongodb
spec:
  namespaceSelector:
    any: true
  selector:
    matchLabels:
       name: mongodb
  endpoints:
  - port: metrics

and
apiVersion: v1
kind: Service
metadata:
  name: mongodb
  labels:
    name: mongodb
spec:
  type: ClusterIP
  selector:
    app: mongodb
  ports:
  - name: metrics
    port: 27017
    targetPort: <port-name-in-pod>

I still cant find the service monitors with service discovery
The config logs for mongodb monitor is here:

job_name: sandbox/mongodb/0
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
kubernetes_sd_configs:

role: endpoints
relabel_configs:
source_labels: [__meta_kubernetes_service_label_name]
separator: ;
regex: mongodb
replacement: $1
action: keep
source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
separator: ;
regex: (.*)
target_label: endpoint
replacement: metrics
action: replace

Can you share the output of the "Service Discovery" page of the Prometheus UI that is respective to this config section?

I dont see any  logs for mongodb  in the my service discovery page.

Could you check the Prometheus logs? This sounds like an RBAC issue.

Prometheus

Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:monitoring:prometheus-k8s" cannot list pods in the namespace "sandbox""
You are right. It's RBAC issue. Do I need to alter example-app-deployment.yaml to make it work?

Your Prometheus server needs additional roles to list/watch all Pods. That's because you're asking it to with the any namespace selector.

Your Prometheus server needs additional roles to list/watch all Pods. That's because you're asking it to with the any namespace selector.

Will I have to create *role.yaml and *roleBinding.yaml as its shown in contrib/kube-prometheus/examples/example-app/prometheus-frontend-role.yaml and corresponding binding.yaml.

Added Role and Rolebindig for new namspace as explained in https://github.com/coreos/prometheus-operator/blob/master/contrib/kube-prometheus/docs/monitoring-other-namespaces.md
Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:monitoring:prometheus-k8s" cannot list endpoints at the cluster scope"

Added Role and Rolebindig for new namspace as explained in https://github.com/coreos/prometheus-operator/blob/master/contrib/kube-prometheus/docs/monitoring-other-namespaces.md
1.Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:monitoring:prometheus-k8s" cannot list endpoints at the cluster scope"
2.Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:monitoring:prometheus-k8s" cannot list endpoints in the namespace "sandbox""

@brancz

I added a new Namespace in prometheus(Role and Rolebinding)
Installing @26.0 version of prometheus  with jb install
jb update and build.sh of example.jsonnet to create new manifests
Running those manifest caused

"system:serviceaccount:monitoring:prometheus-k8s" cannot list endpoints in the namespace "monitoring""

I get this error while running those manifests

error: error validating "manifests/prometheus-namespaces.yaml": error validating data: invalid object to validate; if you choose to ignore these errors, turn validation off with --validate=false

The prometheus-namespaces.yaml file has

default
kube-system
sandbox

The kubelet service is not discovered( 0 /10 active targets)

The /config logs shared here
https://drive.google.com/file/d/1buJ6dncBgWMMUR2ByXKtf880o7in956j/view?usp=sharing

The Prometheus service pods logs are fine.

Hello, I have similar problem, cannot discover ServiceMonitors and PrometheusRules from namespaces different than openshift-monitoring, so I decided to add description of my problem here, let me know if I should start separate issue.
What did you do?
Installed Cluster Monitoring Operator, installed second instance of Prometheus in the same namespace and tried to configure it to discover ServiceMonitors and PrometheusRules from entire cluster.
What did you expect to see?
ServiceMonitors defined in all namespaces available in service discovery and Alerts defined in PrometheuRules in all namespaces available in Prometheus alerts.
What did you see instead? Under which circumstances?
Only CRDs defined in namespace openshift-monitoring are discovered in Prometheus.
Service Discovery: only ServiceMonitors defined in openshift-monitoring namespace available
Rules:
Prometheus configuration:
rule_files:
- /etc/prometheus/rules/prometheus-ngp-rulefiles-0/*.yaml

only PrometheusRules defined in  openshift-monitoring namespace mounted in  /etc/prometheus/rules/prometheus-ngp-rulefiles-0/* and available in Prometheus
Environment
openshift v3.9.0+ba7faec-1
kubernetes v1.9.1+a0ce1bc657

Cluster Monitoring Operator tested versions (same results):
quay.io/coreos/cluster-monitoring-operator:v0.0.6 and registry.access.redhat.com/openshift3/ose-cluster-monitoring-operator:v3.11.16-3

Prometheus Operator tested versions (same results):
quay.io/coreos/prometheus-operator:v0.23.2 and registry.access.redhat.com/openshift3/ose-prometheus-operator:v3.11.16-3

Prometheus related config:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: ngp
  namespace: openshift-monitoring
spec:
  (...)
  serviceMonitorNamespaceSelector:
    any: true
  serviceMonitorSelector:
    any: true
  ruleSelector:
    any: true
  ruleNamespaceSelector:
    any: true
  version: v2.3.2

RBAC configuration

apiVersion: v1
kind: ClusterRole
metadata:
 annotations:
 name: prometheus-k8s-apps
rules:
- apiGroups:
 - ""
 attributeRestrictions: null
 resources:
 - endpoints
 - nodes
 - pods
 - prometheusrules
 - services
 verbs:
 - get
 - list
 - watch
- apiGroups:
 - ""
 attributeRestrictions: null
 resources:
 - configmaps
 verbs:
 - get
- apiGroups:
 - ""
 attributeRestrictions: null
 resources:
 - tokenreviews
 verbs:
 - '*'
- apiGroups:
 - monitoring.coreos.com
 attributeRestrictions: null
 resources:
 - alertmanagers
 - alertmanagers/finalizers
 - prometheuses
 - prometheuses/finalizers
 - prometheusrules
 - servicemonitors
 verbs:
 - '*'

clusterrolebinding - binded to:
- system:serviceaccount:openshift-monitoring:prometheus-k8s
- system:serviceaccount:openshift-monitoring:prometheus-ngp
- system:serviceaccount:openshift-monitoring:prometheus-operator

**kind: ClusterRole**
metadata:
 creationTimestamp: 2018-10-08T13:31:25Z
 name: prometheus-operator
 resourceVersion: "5735837"
 selfLink: /oapi/v1/clusterroles/prometheus-operator
 uid: 741e8ebd-cafe-11e8-8ce3-02159dc2c52c
rules:
- apiGroups:
 - apiextensions.k8s.io
 attributeRestrictions: null
 resources:
 - customresourcedefinitions
 verbs:
 - '*'
- apiGroups:
 - monitoring.coreos.com
 attributeRestrictions: null
 resources:
 - alertmanagers
 - alertmanagers/finalizers
 - prometheuses
 - prometheuses/finalizers
 - prometheusrules
 - servicemonitors
 verbs:
 - '*'
- apiGroups:
 - apps
 attributeRestrictions: null
 resources:
 - statefulsets
 verbs:
 - '*'
- apiGroups:
 - ""
 attributeRestrictions: null
 resources:
 - configmaps
 - secrets
 verbs:
 - '*'
- apiGroups:
 - ""
 attributeRestrictions: null
 resources:
 - pods
 verbs:
 - delete
 - list
- apiGroups:
 - ""
 attributeRestrictions: null
 resources:
 - endpoints
 - services
 verbs:
 - create
 - get
 - update
- apiGroups:
 - ""
 attributeRestrictions: null
 resources:
 - nodes
 verbs:
 - list
 - watch
- apiGroups:
 - ""
 attributeRestrictions: null
 resources:
 - namespaces
 verbs:
 - list
 - watch

clusterrolebinding - binded to:
- system:serviceaccount:openshift-monitoring:prometheus-operator

Prometheus Operator Logs:
(both Prometheus synced, there is some issue with watching resources, not sure if relevant)

  | level=info ts=2018-10-26T06:47:06.483377492Z caller=operator.go:893 component=prometheusoperator msg="sync prometheus" key=openshift-monitoring/k8s
  | level=info ts=2018-10-26T06:47:07.015122256Z caller=operator.go:893 component=prometheusoperator msg="sync prometheus" key=openshift-monitoring/ngp
  | level=info ts=2018-10-26T06:47:07.178506954Z caller=operator.go:893 component=prometheusoperator msg="sync prometheus" key=openshift-monitoring/k8s
  | level=info ts=2018-10-26T06:47:54.709893379Z caller=operator.go:402 component=alertmanageroperator msg="sync alertmanager" key=openshift-monitoring/main
  | (...)
  | level=info ts=2018-10-26T07:04:27.749493736Z caller=operator.go:402 component=alertmanageroperator msg="sync alertmanager" key=openshift-monitoring/main
  | W1026 07:05:24.352643       1 reflector.go:341] github.com/coreos/prometheus-operator/pkg/prometheus/operator.go:341: watch of *v1.Secret ended with: The resourceVersion for the provided watch is too old.
  | W1026 07:05:49.541097       1 reflector.go:341] github.com/coreos/prometheus-operator/pkg/alertmanager/operator.go:200: watch of *v1beta2.StatefulSet ended with: The resourceVersion for the provided watch is too old.

Prometheus Logs:
(some problems with scraping etcd related to reusing certificats from default  prometheus-k8s, not sure if relevant)

  | level=info ts=2018-10-25T23:58:09.941614592Z caller=kubernetes.go:188 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
  | level=info ts=2018-10-25T23:58:10.18926207Z caller=main.go:629 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
  | level=error ts=2018-10-25T23:58:14.795752516Z caller=scrape.go:148 component="scrape manager" scrape_pool=openshift-monitoring/etcd/0 msg="Error creating HTTP client" err="unable to use specified client cert (/etc/prometheus/secrets/kube-etcd-client-certs/etcd-client.crt) & key (/etc/prometheus/secrets/kube-etcd-client-certs/etcd-client.key): tls: failed to find any PEM data in certificate input"
  | level=info ts=2018-10-26T01:00:04.680857788Z caller=compact.go:398 component=tsdb msg="write block" mint=1540504800000 maxt=1540512000000 ulid=01CTPY561Z9A3ZP7CSYRNPTD06
  | level=info ts=2018-10-26T01:00:05.084914907Z caller=head.go:348 component=tsdb msg="head GC completed" duration=108.960634ms
  | level=info ts=2018-10-26T01:00:05.085332503Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=4.453µs

@domino7 as an OpenShift customer, please open an issue through that system. tl;dr the cluster-monitoring stack is not mean to be used for anything but what it's already configured for. The Prometheus Operator is specifically setup to only watch the openshift-monitoring namespace, therefore what you are looking for will never work on 3.11 (which is also the only supported version of the cluster-monitoring stack, you're going to have a lot of trouble with 3.9).

@brancz
Is the cross namespace monitoring  supported in 26.0. if yes then, Could you please attach a working example of cross namespace monitoring with the example's yamls from prometheus-operator/example/user-guides.
I haden't done anything different from what was suggested in the documentation but a working example would help many devops team who don't wish to run multiple Prometheus servers in different namespaces.
Thanks

@brancz

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: prometheus-k8s
namespace: foo
rules:

apiGroups:

""
resources:
nodes
services
endpoints
pods
verbs:
get
list
watch
kind: RoleList

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: prometheus-k8s
namespace: foo
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: prometheus-k8s
subjects:

kind: ServiceAccount
name: prometheus-k8s
namespace: monitoring
kind: RoleBindingList

Above are the Role and Rolebinding created for Foo namespace. I still  get this error in<EMAIL_ADDRESS>"github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:287: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list services in the namespace \"monitoring\""

I'm sorry, but the way your comment is formatted is not comprehensible. Could you can format code with 3 backward ticks
`

Thank you :slightly_smiling_face:

@brancz
It's working fine. I had to include monitoring namespace along with foo.

prometheus+:: {
namespaces: ["default", "kube-system","monitoring", "foo"],
},

@dhirajnitk , can you please provide the RBAC, RBAC rolebinding, clusterrole, clusterrolebinding, other config yaml file as you have got the solution to cross namespace problem.
complete yaml file would be more helpful for "foo" namespace with "default", "kube-system","monitoring", "foo"


Python versions

Hey @pchlap, just wanted to see how we wanted to handle pydicer moving forward with python versions. I think as of now it works for 3.8-3.9 but not anything more recent. I am creating a devcontainer for version 3.9 and make development at least more streamlined that way, but we might want to look at 3.10 and 3..11, and see what needs to be be updated under the hood with pyradiomics, thanks!

Hi @dalmouiee, thats for being proactive on this. I agree, it would be great to make sure we can support more recent Python versions.
I do think pyradiomics will be the key (or only) sticking point. I noticed this PR was merged a couple of months back: https://github.com/AIM-Harvard/pyradiomics/pull/836
If I remember correctly then this was the problem we were facing in our poetry builds, but I would need to revisit to confirm. I also think we need to wait for pyradiomics to actually make a 3.1.1 release which they haven't done yet.
I can try to revisit this over the next weeks when I get some time. Or if you have capacity it would be great if we can confirm what the exact issue is/was. Basically I think we just need to add 3.10 and 3.11 back into our Github PR pipelines and see what issues arise. If we are sure the 3.1.1 pyradiomics version is what we need, then we can post on their GitHub to see if there is anything we can do to help get this moving along.


no more folder in Finder(mac) when folder's name is menu and numeric prefix

really strange issue, first time i saw that, don't know if it's related with GRAV
when i copy folder from finder in MAC, and add numeric folder, can't access  the folder from the finder, ok from GRAV
After some investigation it happen only if the folder named is menu (see screenshot)
and file type is MenuExtra plugin in Finder

This is an OS-specific feature, there's nothing Grav can do. Same with adding a 01.pages with the Pages app installed. Finder will consider that an application-specific file. Not sure if there are OSX utilities or hacks to prevent this.

@flaviocopes thanks

@dimitrilongo @flaviocopes: Thank you both!
It looks strange though that Gantry developers didn't find a solution to this problem...As far as I can see there is only one work around, that is uninstalling Pages...I searched Mac support to find a system to prevent this behaviour, but there is nothing :(