krakend-plugin-builder image tag 2.1.3 not published

Greetings,
I'm using the devopsfaith/krakend-plugin-builder image to compile my plugins for compatibility with the containerized version of KrakenD. I see that the 2.1.3 and 2.1.3-alpine tags aren't published. The 2.1-alpine tag uses Alpine 3.16.2, while the devopsfaith/krakend:2.1.3 image uses Alpine 3.16.3. This causes my plugins not to load due to a platform mismatch. Can you publish the missing tags?
I can work in version 2.1.2 for now, but would eventually like to adopt 2.1.3 so we can use the hide_version_header config flag. Thanks a lot!

Hi @klaude
We are deprecating de devopsfaith namespace, and the "plugin-builder" new name is krakend/builder.
I've pushed a copy of the containers to devopsfaith/krakend-plugin-builder.
Future versions will only be pushed to krakend/builder.
Thanks for reporting!


proposed change to output file naming

Currently, pypiper output files start with the pipeline name and then the file type, like PIPE_log.md and PIPE_profile.md, etc. (see docs)
I did it this way to make it so you could find stuff by multiple pipelines easier. But the thing is, I've realized that 95% of the time we're running only a single pipeline, ad it's annoying to have 4 files all starting with PIPE_* for tab completion.
PIPE_log.md
PIPE_profile.md
PIPE_status.flag
PIPE_commands.md
instead, these could be:
log_PIPE.md
profile_PIPE.md
status_PIPE.flag
etc. The advantage being tab completion is now much easier for folders running a single pipeline, which is most of the time.
But this is kind of a big change. Any opposed? Ideas?

I am not favorable to that change. The advantage you mention is tiny compared to how disruptive that change could be.

Ok, the advantage is only relevant if you are going through output files a lot, which you may not do, but I do.
The disruption would just be if scripts rely on that naming scheme, but I can't think of many... maybe just flagCheck.sh. So it doesn't seem very disruptive to me... nothing reads log files, profile files, or command files. so for those the file names can be changed without effect.

Well, I do look at log files. And indeed it might not be as disruptive. I am not favorable for that it potentially affects lots of undocumented scripts and projects without bringing any significant improvements to what Pypiper really is about.
See what other users have to say on that. I would comply to the general consensus.

Well, if anyone has examples of scripts that rely on this naming scheme, let us know.
The only one I can think of is the checks for the flags, which are mostly programmatic; like flagCheck and looper also checks for flags. These would be very easy to change.
I can't think of anything currently checking the log, profile, or commands files, so I think it wouldn't be too bad but maybe I'm not thinking of something.
the tab-completion I think woudl really help for general navigation; it saves a complete step because you can just say "l " for the log file instead of R to get RRBS, and then l to get log.md, see? It starts to add up when you're checking lots of log files in a project running only 1 pipeline, which is 95% of the time.

I have no preference for either, you can always use some sort of regex to get whatever files, no?
The find command is super powerful, I use it all the time e.g.
find project/results_pipeline -iname "*log*.md*" | grep something | xargs less

of course, but I think you're missing the point... I'm just talking about looking at a single sample, and saving the split second of an extra tab-completion step, because everything starts with the same string. typing out a 'find' command is already way overkill for the use case I'm trying to describe.

One issue: re-runs of a currently completed pipeline would re-run everything one time because they wouldn't find the new complete flag name.

I think I will go ahead with it but leave the flag files unchanged, so it shouldn't disrupt anything. If anyone feels strongly that this is a bad idea, speak now! @cdietzgit you were opposed to this, are you still?


I use sd 1.5 and lcm is OK,but use sdxl turbo print error

turbo checkpoint is not supported yet


Udate CHANGES

/cc @talex5 and @avsm

Maybe note that the images now live in https://hub.docker.com/u/datakit/ ?

Update with feedback.


Batch compatible install

The install command should be batch-compatible. Currently, it asks ! Rename the binary?, which prevents automation, and batch usage.
One way to fix this may be a --rename (or --as) option for install command:
$ stew install knative-sandbox/kn-plugin-event --as kn-event
knative-sandbox/kn-plugin-event
⬇  Downloading asset: 100% |██████████████████████████████████████████████████████████| (57/57 MB, 34.474 MB/s)
✅ Downloaded kn-event-linux-amd64 to /home/ksuszyns/.local/share/stew/pkg
📄 Updated /home/ksuszyns/.local/share/stew/Stewfile.lock.json
✨ Successfully installed the kn-event binary in /home/ksuszyns/bin

Alternative, could be a format of the installation. For example:
$ stew install knative-sandbox/kn-plugin-event!kn-event

This is addressed by the new ability to have reproducible batch installs from the Stewfile.lock.json (https://github.com/marwanhawari/stew/pull/15, https://github.com/marwanhawari/stew/pull/35, https://github.com/marwanhawari/stew/pull/42, https://github.com/marwanhawari/stew/pull/43). This was a part of the v0.4.0 release.


support configuring the filter bits per key

adds a new db opt named filter_bits_per_key which lets the user specify the number of filter bits to use per key for each bloom filter.
also changes the way that SsTableFormat is built to use a builder rather than a new() method with a bunch of parameters. This will let us add new format parameters without having to change all the places where we create a format

Is this related to #163? I haven't gone in-depth yet.

Is this related to https://github.com/slatedb/slatedb/issues/163? I haven't gone in-depth yet.

yeah, it covers one of the bullets from that issue.


[DOCFIX] Update the Chinese version of Contributing-to-Alluxio.md

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/10820/Build result: ABORTED[...truncated 2874 lines...]	at org.jvnet.hudson.maven3.launcher.Maven31Launcher.main(Maven31Launcher.java:132)	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)	at java.lang.reflect.Method.invoke(Method.java:606)	at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)	at jenkins.maven3.agent.Maven31Main.launch(Maven31Main.java:181)	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)	at java.lang.reflect.Method.invoke(Method.java:606)	at hudson.maven.Maven3Builder.call(Maven3Builder.java:136)	at hudson.maven.Maven3Builder.call(Maven3Builder.java:71)	at hudson.remoting.UserRequest.perform(UserRequest.java:120)	at hudson.remoting.UserRequest.perform(UserRequest.java:48)	at hudson.remoting.Request$2.run(Request.java:326)	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)	at java.util.concurrent.FutureTask.run(FutureTask.java:262)	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)	at java.lang.Thread.run(Thread.java:745)[INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ alluxio-tests ---
Test FAILed.

Jenkins, test this please

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/10829/
Test PASSed.

Merged build finished. Test PASSed.

Thanks for the contributation. I have some minor comments. Please handle that, thanks! @saltylin

Thanks very much for your careful review!! I have addressed your comments! @RongGu

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/10832/
Test PASSed.

LGTM.

LGTM

alluxio-bot, check this please


[WIP] Bounds checking

This is still a work in progress, but feedback is welcome. I'm still new to LLVM so let me know if I should be generating the code in a better way.
I have changed the API of get_item_pointer[2] to include the context argument. This is required to raise an exception.
Still TODO:

[ ] Add tests
[ ] Make the error message match object mode (should I try to match the NumPy error message?)
[ ] Add the flag to the public API (right now it is enabled by default for
testing purposes.

I'll need help on how to do the last item.
There is also a boundcheck flag in some places in the code, which doesn't seem to do anything. I have named my flag boundscheck with an s, as that seemed better, but I don't particularly care if you decide another name is better.

There is also a boundcheck flag in some places in the code, which doesn't seem to do anything. I have named my flag boundscheck with an s, as that seemed better, but I don't particularly care if you decide another name is better.

Agree, boundscheck makes sense, I also think renaming the current flag to that and reusing it if convenient would be good, else just remove the current flag as IIRC it's dead.

I guess this isn't accounting for the numpy shape:
>>> import numba
>>> import numpy
>>> @numba.jit
... def test(x):
...     return x[:, 0]
>>> test(numpy.empty((0, 1)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: index is out of bounds

This is the source of the test failures.
Is the numpy shape saved somewhere? If not, I guess we can just disable bounds checks if one of the shapes is 0. We can't get segfaults or nonsense memory values in that case. It mainly would help for type checking.

I guess this isn't accounting for the numpy shape:
>>> import numba
>>> import numpy
>>> @numba.jit
... def test(x):
...     return x[:, 0]
>>> test(numpy.empty((0, 1)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: index is out of bounds

This is the source of the test failures.
Is the numpy shape saved somewhere? If not, I guess we can just disable bounds checks if one of the shapes is 0. We can't get segfaults or nonsense memory values in that case. It mainly would help for type checking.

The shape is an argument to the get_item_pointer2 function, I guess you can generate a branch based on that? I'm not sure that disabling bounds checking on arrays where one of the shape elements is 0 would be desirable, you can still access outside the allocated region and indeed the page, after all it's just a GEP that's being emitted. e.g. with no bounds check:
import numba
import numpy
@numba.jit
def test(x):
    big_index =<PHONE_NUMBER>0000000
    return x[big_index, big_index]

print(test(numpy.empty((0, 1))))

will likely segfault.

This little snippet might help you with debugging, it lets you print an iterable from the compiled code:
        def print_iterable(builder, name, fmt, iterable):
            printf(builder, "{}\n".format(name))
            for x in iterable:
                printf(builder, fmt, x)
            printf(builder, "\n")

        print_iterable(builder, "shape", "%d, ", shape)
        print_iterable(builder, "indices", "%d, ", indices)

if you paste that into the if boundscheck: branch body and run the failing example above you should see:
shape
0, 1,
indices
0, 0,
Traceback (most recent call last):
  File "test_4432.py", line 21, in <module>
    print(test(numpy.empty((0, 1))))
IndexError: index is out of bounds branch1

To accommodate the 0d case I suspect a:
builder.if_else(zero_d_predicate) as (then, otherwise):
  with then:
    <stuff>
  with otherwise:
    <stuff>

might be useful?
Hope this helps?

Oh the problem isn't that it's removing the numpy shape as I was assuming. It's that bounds checking for an empty dimension shouldn't do the index >= dimension check.
Also regarding the error thing, I hadn't realized that it reuses the pickled exception if it is used multiple times. I had seen it twice in there but only because I had different error messages for the upper and lower out of bounds at the time. So I was worried that you would get 2n pickled exceptions saved to the compiled function for n array accesses, but that isn't the case.
Thanks for the debug suggestion. I didn't notice the printf function.

Can we distinguish these two cases inside numba?
>>> a = np.empty((0, 1))
>>> a[:, 0]
array([], dtype=float64)
>>> a[0, 0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: index 0 is out of bounds for axis 0 with size 0

The above printf code shows the shape as (0, 1) and indices as (0, 0) for both.

This code does not segfault for me (in numba master)
from numba import njit
@njit
def test(x):
    return x[:,<PHONE_NUMBER>0000000000]
import numpy as np
a = np.empty((0, 1))
test(a)

I think what we have to do is to implement separate bounds checking in basic_indexing() in arrayobj.py (and possibly other places if they deal with slices as well). That function already does its own wraparound logic (it's the reason there is a flag for it in get_item_pointer, so it can avoid doing it twice), so it's reasonable for it to do bounds checking as well. As far as I can tell the input to get_item_pointer for a[:, 0] and a[0, 0] is identical. Only basic_indexing() knows when the index is a slice, and hence can be used on an empty dimension.
I don't actually understand how numpy handles shape-0 arrays. Does it just create a placeholder pointer?

I've implemented that in the latest commit.

>>> a = np.empty((0, 1))
>>> a[:, 0]
array([], dtype=float64)
>>> a[0, 0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: index 0 is out of bounds for axis 0 with size 0

By the time we get to GEP I think it's probably too late, the contextual slice info is lost. I think this is because a slice looks like accessing index 0, i.e. "where does the data start for a slice == what is index 0". This https://github.com/numba/numba/pull/4432/files#diff-36c9ab8fa31f756e0da52905db0a95deR717-R722 is just working out where in data block the relevant data is.

I figured out that issue (see my most recent comments/commits).
Right now I'm trying to figure out the test failures, which are presently failing because bounds checking is still enabled by default. It looks like there are some issues with both cuda and parallel.
Here's an example:
from numba import njit
import numpy as np

@njit(parallel=True)
def test(X):
    X[:0] += 1
    return X

test(np.ones(10))

Enabling the debug output which I just added, you get
debug: IndexError: index 1 is out of bounds for axis 0 with size 0
debug: IndexError: index 0 is out of bounds for axis 0 with size 0
debug: IndexError: index 7 is out of bounds for axis 0 with size 0
debug: IndexError: index 5 is out of bounds for axis 0 with size 0
debug: IndexError: index 9 is out of bounds for axis 0 with size 0
debug: IndexError: index 4 is out of bounds for axis 0 with size 0
debug: IndexError: index 3 is out of bounds for axis 0 with size 0
debug: IndexError: index 6 is out of bounds for axis 0 with size 0
debug: IndexError: index 8 is out of bounds for axis 0 with size 0
debug: IndexError: index 2 is out of bounds for axis 0 with size 0
IndexError: index is out of bounds

I guess parallel takes slices and converts them into explicit indices. But bounds checking should not be enabled for slices, so I need to figure out where that is happening and how to disable it for that code path, similar to what I did for the single core slice code path.
I haven't looked at the cuda issues yet, but I think they are similar.
As an aside, it could be a good idea to have a CI build that runs the test suite with bounds checking enabled. I already found one function in numba that failed with it enabled (see https://github.com/numba/numba/pull/4432/commits/0b95b7ca8e287dd772b96fce9bff80829ff59efc).

I figured out that issue (see my most recent comments/commits).
Right now I'm trying to figure out the test failures, which are presently failing because bounds checking is still enabled by default. It looks like there are some issues with both cuda and parallel.
Here's an example:
from numba import njit
import numpy as np

@njit(parallel=True)
def test(X):
    X[:0] += 1
    return X

test(np.ones(10))

Enabling the debug output which I just added, you get
debug: IndexError: index 1 is out of bounds for axis 0 with size 0
debug: IndexError: index 0 is out of bounds for axis 0 with size 0
debug: IndexError: index 7 is out of bounds for axis 0 with size 0
debug: IndexError: index 5 is out of bounds for axis 0 with size 0
debug: IndexError: index 9 is out of bounds for axis 0 with size 0
debug: IndexError: index 4 is out of bounds for axis 0 with size 0
debug: IndexError: index 3 is out of bounds for axis 0 with size 0
debug: IndexError: index 6 is out of bounds for axis 0 with size 0
debug: IndexError: index 8 is out of bounds for axis 0 with size 0
debug: IndexError: index 2 is out of bounds for axis 0 with size 0
IndexError: index is out of bounds

I guess parallel takes slices and converts them into explicit indices. But bounds checking should not be enabled for slices, so I need to figure out where that is happening and how to disable it for that code path, similar to what I did for the single core slice code path.

Were I to guess, this is where I'd start looking. https://github.com/numba/numba/blob/f629736997eaf83040caecd1fea7322577b7fd89/numba/parfor.py#L1921-L1934

I haven't looked at the cuda issues yet, but I think they are similar.
As an aside, it could be a good idea to have a CI build that runs the test suite with bounds checking enabled. I already found one function in numba that failed with it enabled (see 0b95b7c).

If we have the compute capacity then I think this makes sense.

Note to self: need to find a way to get an ir.Loc to the site the exception is being created else the exception will correctly report out of bounds, but the site it will be raised from will be the jitted function, not the corresponding source line.

@DrTodd13 This is the issue with parfors and bounds check, if you have any ideas they'd be appreciated, thanks https://github.com/numba/numba/pull/4432#issuecomment-522099410 !

The bounds check should only happen when the getitem is physically called, unless I am misunderstanding how things work. Is the parfor generated from the slice doing unnecessary redundant getitems? I wasn't able yet to debug why it happens (any suggestions for how to debug this sort of thing, beyond "learn how to read LLVM"?)
And if it is and they're not easy to remove, is there some way to annotate the getitem so that the get_item_pointer can know it shouldn't be bounds checked? The parfor logic happens in a completely different stage from the get_item_pointer.

This also should be extended to catch broadcast errors like the following:
>>> def test(a, b):
...     a += b
>>> test(np.empty((3, 4)), np.empty((3, 5)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in test
ValueError: operands could not be broadcast together with shapes (3,4) (3,5) (3,4)
>>> @njit
... def test(a, b):
...     a += b
>>> test(np.empty((3, 4)), np.empty((3, 5))) # In this branch
>>>

Perhaps that should be considered a bug in the existing numba. If you instead do return a + b it gives an error ValueError: unable to broadcast argument 1 to output array.

I guess the exception issue is even worse than I thought, because if you have nested jitted functions, the exception is only reported for the top-level one.
from numba import njit
import numpy as np

@njit
def func1():
    x = np.empty((0,))
    func2(x)

@njit
def func2(x):
    x[0] = 1

func1()

$python test.py
Traceback (most recent call last):
  File "test.py", line 13, in <module>
    func1()
IndexError: index is out of bounds

For now, we should probably document with this option that the best way to debug these errors is to turn jitting off.

Perhaps that should be considered a bug in the existing numba. If you instead do return a + b it gives an error ValueError: unable to broadcast argument 1 to output array.

Please could you raise an issue, thanks.

I guess the exception issue is even worse than I thought, because if you have nested jitted functions, the exception is only reported for the top-level one.
from numba import njit
import numpy as np

@njit
def func1():
    x = np.empty((0,))
    func2(x)

@njit
def func2(x):
    x[0] = 1

func1()

$python test.py
Traceback (most recent call last):
  File "test.py", line 13, in <module>
    func1()
IndexError: index is out of bounds

For now, we should probably document with this option that the best way to debug these errors is to turn jitting off.

What you are seeing is a result of synthesising stack frames based on static compilation information, the issue is that the location information hasn't been propagated to the site where the exception is being generated (which is my note to self here: https://github.com/numba/numba/pull/4432#issuecomment-526361878). This works fine:
from numba import njit
import numpy as np

@njit
def func1():
    x = np.empty((0,))
    func2(x)

@njit
def func2(x):
    raise ValueError("message")

func1()

and gives:
$ python test_4432_2.py
Traceback (most recent call last):
  File "test_4432_2.py", line 13, in <module>
    func1()
  File "test_4432_2.py", line 11, in func2
    raise ValueError("message")
ValueError: message

because the ir.Loc information for this code path is given to the equivalent function call to that in this PR here:
https://github.com/numba/numba/pull/4432/files#diff-36c9ab8fa31f756e0da52905db0a95deR683
were the ir.Loc for the source line somehow passed through to lowering so it were available where the bounds checking code is generating the exceptions in GEP then the frames/traceback would look reasonable.

OK, regarding parfor, I'm pretty sure there's a bug
>>> from numba import njit
>>> import numpy as np
>>> @njit(parallel=True)
... def test(X):
...     X[:0] = 2
...     return X
>>> test(np.ones(10))
array([2., 2., 2., 2., 2., 2., 2., 2., 2., 2.])
>>> def test2(X):
...     X[:0] = 2
...     return X
>>> test2(np.ones(10))
array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

I opened https://github.com/numba/numba/issues/4630 for it.

Regarding the test_array_ctypes_ref_error_in_parallel test failure (#2887), it seems to be unrelated to this change. I think there's a race condition for the memory being freed and the extra time spent by the bounds check causes it to be hit.
Consider this behavior from the test slightly modified (to call the callback twice). This is in master:
# test.py
from numba import njit
import numpy as np

# Issue #2887
from ctypes import CFUNCTYPE, c_void_p, c_int32, c_double, c_bool

@CFUNCTYPE(c_bool, c_void_p, c_int32, c_void_p)
def callback(inptr, size, outptr):
    # A ctypes callback that manipulate the incoming pointers.
    try:
        inbuf = (c_double * size).from_address(inptr)
        outbuf = (c_double * 1).from_address(outptr)
        a = np.ndarray(size, buffer=inbuf, dtype=np.float64)
        b = np.ndarray(1, buffer=outbuf, dtype=np.float64)
        b[0] = (a + a.size)[0]
        return True
    except:
        import traceback
        traceback.print_exception()
        return False

# parallel=True is required to reproduce the error.
@njit()
def foo(size):
    arr = np.ones(size)
    out = np.empty(1)
    # Exercise array.ctypes
    inct = arr.ctypes
    outct = out.ctypes
    # The reference to `arr` is dead by now
    # print(inct.data)
    status = callback(inct.data, size, outct.data)
    print(status, out)
    status = callback(inct.data, size, outct.data)
    print(status, out)

size = 3
foo(size)

$ python test.py
True [4.]
True [-9.86830992e+148]

If you remove the @njit decorator, you get
True [4.]
True [4.]

I've also removed the parallel decorator, which isn't needed to reproduce it. Probably the parallel=True also caused the same race condition to be hit with its overhead, as the function doesn't actually use any parallelism.
So I think issue https://github.com/numba/numba/issues/2887 wasn't actually fixed.

I've added a boundscheck flag to @jit (replacing the old unused boundcheck flag), defaulting to False. I also added a NUMBA_BOUNDSCHECK environment variable which globally overrides the flag. I've also added documentation for both.
What remains to do is to add tests for this, and also add (and fix) a test suite build with NUMBA_BOUNDSCHECK=1. I'm hoping someone else will look at the parfor issue (https://github.com/numba/numba/issues/4630) and the https://github.com/numba/numba/pull/2893 issue. I still need to look into the CUDA failures and see if bounds checking even makes sense on CUDA or if the flag should be disallowed there (unfortunately the Azure build logs from before are gone now, so I have to wait for a rebuild; would it be possible to change the Azure build logs so they stick around forever?).
By the way, why is jit documented separately in its docstring and in the RST docs? Wouldn't it be better to pull in the docstring using autodoc?

Aside from a lack of tests, which I will work on soon, this is ready for review.
One question about tests: is there a good way to test the environment variable behavior? I want to test that it overrides the flag correctly, but it would need to be done in a subprocess.

Also where's the best place to add a CI build with bounds checking enabled (assuming you still agree it's a good idea)?

Aside from a lack of tests, which I will work on soon, this is ready for review.
One question about tests: is there a good way to test the environment variable behavior? I want to test that it overrides the flag correctly, but it would need to be done in a subprocess.

@asmeurer probably want something like this: https://github.com/numba/numba/blob/5bd018fd55ab2ff0916ab2c6360334fe94e471a4/numba/tests/test_warnings.py#L173-L206

Also where's the best place to add a CI build with bounds checking enabled (assuming you still agree it's a good idea)?

Probably the Azure tests: https://github.com/numba/numba/blob/5bd018fd55ab2ff0916ab2c6360334fe94e471a4/azure-pipelines.yml
with a conditional export here:
https://github.com/numba/numba/blob/5bd018fd55ab2ff0916ab2c6360334fe94e471a4/buildscripts/incremental/test.sh#L17-L22
I think it'd be wise to see how much of an impact it has on runtime before committing to having it on in public CI. Have you run the test suite locally with it on/off to get some indication? Thanks.

I'll run the tests locally now, but based on old CI builds on this branch (enabled and disabled), it doesn't seem like there is much of a difference.

Replying to https://github.com/numba/numba/pull/4432#issuecomment-536103107 regarding issue #2887.
TL;DR
Pass array.ctypes into a C-callable instead of passing array.ctypes.data.  The latter is not holding a reference to array.
Details
The problem seen here is different. The use of arr.ctypes.data yields an integer type with no reference to the array arr.  It is possible that arr is deallocated at that time.
The numpy docs mention the following:

Note that unlike data_as, a reference will not be kept to the array: code like ctypes.c_void_p((a + b).ctypes.data) will result in a pointer to a deallocated array, and should be spelt (a + b).ctypes.data_as(ctypes.c_void_p)

(from https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.ctypes.html)
The trouble is complicated by:

CPython would keep arr around if it is a local variable.  But that is an implementation detail. Numba is more eager to cleanup dead variable (determined by static analysis). So numba can remove unused variable very early.
In pure python, such code will likely give the illusion of working. 1) arr.ctypes creates a cyclic reference that defers deallocation further (untils the cyclic detector kicks in).  2) Numpy deallocation routine does not erase the underlying data or unmap it.  Reading and writing to the deallocated buffer will likely not cause any error. On the other hand, Numba overwrites the first cacheline of the to-be deallocated buffer to make use-after-free error more obvious (at https://github.com/numba/numba/blob/5bd018fd55ab2ff0916ab2c6360334fe94e471a4/numba/runtime/nrt.c#L205).  The -9.86830992e+148 value is the 0xDEDEDEDEDEDEDEDE sequence interpreted as a float64.

Since numba doesn't implement the arr.ctypes.data_as, the recommendation is to pass arr.ctypes, for which ctypes will gladly take for an argument that is expecting a pointer type like c_void_p.

@sklam what should be done about the test case for that is in the code for that? It fails when bounds checking is enabled?

changing https://github.com/numba/numba/blob/8638f8cc6939781d98bec40aa77c6092529b75d5/numba/tests/test_array_attr.py#L272 to status = callback(inct, size, outct) seems to work.

OK, that seems fine. Does that still test the original issue?

The tests seem to take about the same amount of time with NUMBA_BOUNDSCHECK=1. Maybe it would be useful to benchmark the bounds checking in general. Is there a good example you know of that I can benchmark it on?

Maybe something like a mergesort on array: https://github.com/numba/numba/blob/master/examples/mergesort.py

Looks like about 10% to 30% slower.
No bounds checking
In [1]: import examples.mergesort

In [2]: import numpy as np

In [3]: arr = np.random.random(10**5)

In [4]: # warmup

In [5]: examples.mergesort.mergesort(arr.copy());

In [6]: examples.mergesort.mergesort_inplace(arr.copy());

In [7]: %timeit examples.mergesort.mergesort(arr.copy())
14.5 ms ± 45.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [8]: %timeit examples.mergesort.mergesort_inplace(arr.copy())
1.83 s ± 22.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [9]: # Confirm no bounds checking

In [10]: from numba import njit

In [11]: @njit
    ...: def test(x):
    ...:     return x[1]
    ...:

In [12]: test(np.array([1]))
Out[12]:<PHONE_NUMBER>17040

Bounds checking

In [1]: import examples.mergesort

In [2]: import numpy as np

In [3]: arr = np.random.random(10**5)

In [4]: # warmup

In [5]: examples.mergesort.mergesort(arr.copy());

In [6]: examples.mergesort.mergesort_inplace(arr.copy());

In [7]: %timeit examples.mergesort.mergesort(arr.copy())
16.5 ms ± 45.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [8]: %timeit examples.mergesort.mergesort_inplace(arr.copy())
2.42 s ± 10.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [9]: # Confirm bounds checking is enabled

In [10]: from numba import njit

In [11]: @njit
    ...: def test(x):
    ...:     return x[1]
    ...:

In [12]: test(np.array([1]))
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-12-63f95a072376> in <module>
----> 1 test(np.array([1]))

IndexError: index is out of bounds

In [13]: 16.5/14.5
Out[13]: 1.1379310344827587

In [14]: 2.42/1.83
Out[14]: 1.3224043715846994

I do wonder, though, going back to https://github.com/numba/numba/pull/4432#discussion_r313080879, to what degree is LLVM able to optimize out the bounds checks. Most of the indices in the mergesort algorithm are from loops that are based on the array.size. So it's not clear to me if every index actually gets a bounds check or if LLVM is able to optimize some of them away, and I don't know how to check that. Is there a good example where the array indices cannot possibly be known at compile time, like if they are inputs to the function?

I just noticed this doesn't handle fancy indexing.

I'm working on this again.
I looked into the cuda failures. It looks like if you enable boundschecking in a cuda function, it stops the execution but doesn't raise an exception. Should we just disable boundschecking for CUDA?
Note that some of the cuda tests use out of bounds accesses and hence fail with NUMBA_BOUNDSCHECK=1. For example, here https://github.com/numba/numba/blob/e4674eea90b451ae2141a20546727bef4ac8b352/numba/cuda/tests/cudapy/test_sync.py#L33-L38
The cuda tests that fail are
test_nvvm_memset_fixup
test_const_record
test_const_record_align
test_dyn_shared_memory
test_selp

I haven't looked into why for each one.
You can check by adding return to this line https://github.com/asmeurer/numba/blob/boundscheck/numba/cgutils.py#L674 (otherwise you get a bunch of errors about printf, I guess the tests run with FULL_TRACEBACKS enabled?), then run
NUMBA_BOUNDSCHECK=1 ./runtests.py -m -v numba.cuda.tests

I also get some cuda test failures in master
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aaronmeurer/Documents/numba/numba/cuda/cudadrv/autotune.py", line 245, in warp_occupancy
    limits = PHYSICAL_LIMITS[cc]
KeyError: (7, 5)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aaronmeurer/Documents/numba/numba/cuda/tests/cudapy/test_deprecation.py", line 29, in test_autotune
    foo.autotune
  File "/home/aaronmeurer/Documents/numba/numba/cuda/compiler.py", line 727, in autotune
    at = AutoTuner(info=cufunc.attrs, cc=cufunc.device.compute_capability)
  File "/home/aaronmeurer/Documents/numba/numba/cuda/cudadrv/autotune.py", line 49, in __init__
    self._table = warp_occupancy(info=info, cc=cc)
  File "/home/aaronmeurer/Documents/numba/numba/cuda/cudadrv/autotune.py", line 248, in warp_occupancy
    % ".".join(str(c) for c in cc))
ValueError: 7.5 is not a supported compute capability

======================================================================
ERROR: test_independent_scheduling (numba.cuda.tests.cudapy.test_warp_ops.TestCudaWarpOperations)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aaronmeurer/Documents/numba/numba/cuda/tests/cudapy/test_warp_ops.py", line 241, in test_independent_scheduling
    self.assertTrue(np.all(ary_out == exp))
NameError: name 'ary_out' is not defined

======================================================================
FAIL: test_match_any_sync (numba.cuda.tests.cudapy.test_warp_ops.TestCudaWarpOperations)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aaronmeurer/Documents/numba/numba/cuda/tests/cudapy/test_warp_ops.py", line 219, in test_match_any_sync
    self.assertTrue(np.all(ary_out == exp))
AssertionError: False is not true

----------------------------------------------------------------------
Ran 515 tests in 37.872s

And the tests segfault if I don't run them with -m -v
Current thread 0x00007faee3e62740 (most recent call first):
  File "/home/aaronmeurer/Documents/numba/numba/cuda/cudadrv/driver.py", line 293 in safe_cuda_api_call
  File "/home/aaronmeurer/Documents/numba/numba/cuda/cudadrv/driver.py", line 1622 in launch_kernel
  File "/home/aaronmeurer/Documents/numba/numba/cuda/cudadrv/driver.py", line 1578 in __call__
  File "/home/aaronmeurer/Documents/numba/numba/cuda/compiler.py", line 611 in _kernel_call
  File "/home/aaronmeurer/Documents/numba/numba/cuda/compiler.py", line 537 in __call__
  File "/home/aaronmeurer/Documents/numba/numba/cuda/compiler.py", line 801 in __call__
  File "/home/aaronmeurer/Documents/numba/numba/cuda/kernels/reduction.py", line 252 in __call__
  File "/home/aaronmeurer/Documents/numba/numba/cuda/tests/cudapy/test_reduction.py", line 71 in test_result_on_device
  File "/home/aaronmeurer/anaconda3/lib/python3.7/unittest/case.py", line 615 in run
  File "/home/aaronmeurer/anaconda3/lib/python3.7/unittest/case.py", line 663 in __call__
  File "/home/aaronmeurer/anaconda3/lib/python3.7/unittest/suite.py", line 122 in run
  File "/home/aaronmeurer/anaconda3/lib/python3.7/unittest/suite.py", line 84 in __call__
  File "/home/aaronmeurer/anaconda3/lib/python3.7/unittest/runner.py", line 176 in run
  File "/home/aaronmeurer/Documents/numba/numba/testing/main.py", line 123 in run
  File "/home/aaronmeurer/anaconda3/lib/python3.7/unittest/main.py", line 271 in runTests
  File "/home/aaronmeurer/Documents/numba/numba/testing/main.py", line 354 in run_tests_real
  File "/home/aaronmeurer/Documents/numba/numba/testing/main.py", line 369 in runTests
  File "/home/aaronmeurer/anaconda3/lib/python3.7/unittest/main.py", line 101 in __init__
  File "/home/aaronmeurer/Documents/numba/numba/testing/main.py", line 163 in __init__
  File "/home/aaronmeurer/Documents/numba/numba/testing/__init__.py", line 75 in run_tests
  File "/home/aaronmeurer/Documents/numba/numba/_runtests.py", line 28 in _main
  File "/home/aaronmeurer/Documents/numba/numba/runtests.py", line 9 in <module>
  File "/home/aaronmeurer/anaconda3/lib/python3.7/runpy.py", line 85 in _run_code
  File "/home/aaronmeurer/anaconda3/lib/python3.7/runpy.py", line 208 in run_module
  File "./runtests.py", line 11 in <module>
Segmentation fault (core dumped)

I've no idea if the master failures are some issue with my environment or bugs in numba. Here's my machine info:
$ nvidia-smi
Fri Nov 15 17:10:24 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 435.21       Driver Version: 435.21       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 2080    Off  | 00000000:01:00.0  On |                  N/A |
|  7%   54C    P8    12W / 215W |    879MiB /  7959MiB |      2%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1242      G   /usr/lib/xorg/Xorg                           349MiB |
|    0      6487      G   /usr/bin/kwin_x11                            158MiB |
|    0      6495      G   /usr/bin/krunner                               6MiB |
|    0      6498      G   /usr/bin/plasmashell                          82MiB |
|    0      7646      G   /proc/self/exe                                46MiB |
|    0      7651      G   ...quest-channel-token=7692621858562035617   167MiB |
|    0      7716      G   /usr/bin/akonadi_archivemail_agent             6MiB |
|    0      7785      G   /usr/bin/akonadi_mailfilter_agent              6MiB |
|    0      7795      G   /usr/bin/akonadi_sendlater_agent               6MiB |
|    0      7796      G   /usr/bin/akonadi_unifiedmailbox_agent          6MiB |
|    0      8214      G   ...quest-channel-token=6964761366230685101    40MiB |
+-----------------------------------------------------------------------------+
$ uname -a
Linux unbuntu 5.3.0-23-generic #25-Ubuntu SMP Tue Nov 12 09:22:33 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ python -c 'import sys; print(sys.version)'
3.7.3 | packaged by conda-forge | (default, Jul  1 2019, 21:52:21)
[GCC 7.3.0]

On another note, is there a reason the old Azure logs are deleted? It would be helpful if they weren't.

To enable exception checking on kernel exit, we need cuda.jit(debug=True).  The debug flag does the followings:

enable boundchecking code generation.
enable exception propagation code in cuda target.
enable runtime code for exception state setup and exception state checking.
other debug mode de-optimization and debuginfo emission at nvvm unrelated to boundschecking.

Because the CUDA kernel launch is asynchronous to the host, the post-call exception state checking must force a synchronization on the kernel call to raise the exception "in-time".
I think we should keep the current behavior of only enabling boundschecking if debug=True for CUDA target.
As for the failing tests on master, our buildfarm does not have GPU of CC>=7.0. The failure in autotune can be ignored because it is exercise deprecated features. The other two failures in test_warp_ops will need to be investigated. I don't remember if we have ever test them on Volta.

enable boundchecking code generation.

Do you mean there is already some boundschecking that happens for CUDA?

Yes, CUDA target has bounds-checking enabled

There were some boundchecking in certain methods, but not in __getitem__ for ndarray.

Hmm... actually

enable boundchecking code generation.

is not quite true. I remembered it wrong. So, it is only the enabling of exception handling for CUDA that enabled the raised IndexError to propagate to the host. i.e. this is a test for tuple OOB in CUDA: https://github.com/numba/numba/blob/e4674eea90b451ae2141a20546727bef4ac8b352/numba/cuda/tests/cudapy/test_exception.py#L9-L28

I have gone through the failed test with NUMBA_BOUNDSCHECK=1:
Failed due to strict assumption in emitted code:
- test_nvvm_memset_fixup
- test_const_record
- test_const_record_align
- test_selp
These tests just cannot run with boundscheck enabled because it is checking what the compiler must be emitting.
test_dyn_shared_memory failed because dynamic shared memory doesn't propagate
the .shape attribute.  Dynamic shared memory is implemented in such a way that it is not compatible with boundschecking.
These failures are safe to ignore when NUMBA_BOUNDSCHECK=1.

OK, I'll add code to completely disable boundschecking in CUDA. (by the way, my GPU is Turing, not Volta, if I'm not mistaken).

I pushed a fix for parfor converting slices to get/setitem (https://github.com/numba/numba/pull/4432#issuecomment-522099410). It works for that test, but then other parfor tests fail, and I don't understand why.
The fix itself is a bit hacky, so let me know if it is OK. If you can suggest a better way to do it, let me know.
See the most recent commit message.

The latest commit also causes random segfaults when I run code with something that uses threads like an interactive terminal or pudb. I don't understand how it could be causing any of these problems. Is it because some exception get swallowed somewhere?
Another option for fixing this would be to change the parfor code to generate a loop based on min(slice endpoint, array size). That way it never produces an out of bounds access. I might need help on how to do that, though, since that code is written in numba IR (_setitem_to_parfor in numba/parfor.py).
That's probably a more ideal fix since not only is it less hacky, but it should make things more efficient in some cases, since currently parallel=True iterates over the whole shape of an array even if the slice is much smaller:
import time

import numpy as np
from numba import njit

X = np.ones(1000000000)

def test(X):
    X[:0] += 1
    return X

t = time.time()
test(X)
print("No numba:", time.time() - t, "seconds")

t = time.time()
njit()(test)(X)
print("Numba no parallel:", time.time() - t, "seconds")

t = time.time()
njit(parallel=True)(test)(X)
print("Numba parallel:", time.time() - t, "seconds")

No numba: 2.1219253540039062e-05 seconds
Numba no parallel: 0.07966017723083496 seconds
Numba parallel: 0.7951962947845459 seconds

The slice thing is the only issue left here that I'm aware of. I still need to run the full test suite with bounds checking enable to make sure there aren't other failures I missed.
As an aside, one test in the parfor tests confirmed that bounds checking does indeed break vectorization.

I just noticed that the test_parfor_slice19 bug is fixed by https://github.com/numba/numba/pull/4648. I fixed all other known issues on this PR. Please put it in the review queue. I guess we should wait until #4648 is merged before merging this.
When testing this, the tests should be run with NUMBA_BOUNDSCHECK=1 to ensure there aren't any latent out of bounds issues in the library functions.

I've addressed the review comments here.

This issue shouldn't block this pull request, but regarding getting the location of the exception, it seems that the most straightforward way is to store it on the context. Here is an example implementation:
diff --git a/numba/cgutils.py b/numba/cgutils.py
index cdce25131..c89a70565 100644
--- a/numba/cgutils.py
+++ b/numba/cgutils.py
@@ -691,12 +691,12 @@ def do_boundscheck(context, builder, ind, dimlen, axis=None):
     with if_unlikely(builder, out_of_bounds_upper):
         if config.FULL_TRACEBACKS:
             _dbg()
-        context.call_conv.return_user_exc(builder, IndexError, (msg,))
+        context.call_conv.return_user_exc(builder, IndexError, (msg,), loc=context.loc)
     out_of_bounds_lower = builder.icmp_signed('<', ind, ind.type(0))
     with if_unlikely(builder, out_of_bounds_lower):
         if config.FULL_TRACEBACKS:
             _dbg()
-        context.call_conv.return_user_exc(builder, IndexError, (msg,))
+        context.call_conv.return_user_exc(builder, IndexError, (msg,), loc=context.loc)

 def get_item_pointer2(context, builder, data, shape, strides, layout, inds,
diff --git a/numba/lowering.py b/numba/lowering.py
index 3fb48c03f..824f61208 100644
--- a/numba/lowering.py
+++ b/numba/lowering.py
@@ -254,6 +254,7 @@ class BaseLower(object):
         self.pre_block(block)
         for inst in block.body:
             self.loc = inst.loc
+            self.context.loc = inst.loc
             defaulterrcls = partial(LoweringError, loc=self.loc)
             with new_error_context('lowering "{inst}" at {loc}', inst=inst,
                                    loc=self.loc, errcls_=defaulterrcls):
diff --git a/numba/targets/base.py b/numba/targets/base.py
index a7b52d69e..c89b1e56c 100644
--- a/numba/targets/base.py
+++ b/numba/targets/base.py
@@ -254,6 +254,8 @@ class BaseContext(object):

         self._boundscheck = False

+        self.loc = None
+
         self.data_model_manager = datamodel.default_manager

         # Initialize

from numba import njit
import numpy as np
@njit(boundscheck=True)
def test(x):
    return x[1]
test(np.array([1]))

$ python test-loc.py
Traceback (most recent call last):
  File "test-loc.py", line 6, in <module>
    test(np.array([1]))
  File "test-loc.py", line 5, in test
    return x[1]
IndexError: index is out of bounds

This issue shouldn't block this pull request, but regarding getting the location of the exception, it seems that the most straightforward way is to store it on the context. Here is an example implementation:

I ran the tests with this and there were no failures. Should I push it up here?

RE: https://github.com/numba/numba/pull/4432#issuecomment-561916179
Let's keep it a different PR.

Will do. I also have some thoughts on getting the error message to include a more complete error message (which I haven't implemented yet). I'll start a new PR once this one is merged.

running internal CI: numba_77

this has passed internal CI


[log] we need to log crashing panics

What does this PR do?
Adds panicwrap such that we can capture and log crashing panics, as opposed to just having those go to stderr which posed all sorts of challenges to put something in place that was sane across all different environments and distros.
When the agent panics it will log to panic.log in the expected locations. I considered just logging to the same agent.log file, and ultimately decided not to in case it could mess with the seelog logging for the main process somehow (it's probably not a "thing", just did out of conservatism).
Motivation
We were kind of blind to agent crashes, this will give us some insight and debugging ability in the event of an unexpected crash, OOM kill, etc...

Let's not do this in the agent itself.


when converting v1 to v2, transform build tags (#4729)

Pull request opened by the merge tool on behalf of #4729

Merged into branches iso5 in 3eb77f3e9f03989599075eaac49adb9a454d9bc1


Many crates in workspace no longer build on their own

PR https://github.com/tikv/tikv/pull/9358 changed cargo to use resolver = 2, which is described here.
This changes, among other things, feature resolution to be based on the crates selected to build/check, instead of the workspace crate. After this change, building many crates results in the appropriate protobuf-codec/prost-codec features not being set, as the individual crates were relying on the tikv crate to properly configure all crates.
Using the newer resolver is probably the right thing to do long term though, since that is how future-cargo is expected to work.
My workflow typically involves running cargo check on individual crates, so I expect all crates to be buildable by default. In some cases it is probably reasonable for crates to not have a buildable default feature set, but in tikv's case, I think it's best for the contributor experience that every crate can build on its own. Otherwise, one needs to look at each crate's manifest and figure out what features to turn on.
Unfortunately, realizing this the obvious way requires adding a lot more boilerplate to all the manifests in the workspace, setting default = ["protobuf-codec"], and correctly setting default-features = false everywhere necessary.
I've started working on a patch that does this, as well as introducing a script that ensures that all crates always build under a variety of configurations. It does though feel like something is missing in the rust tooling that managing complex trees of features is so incredibly arduous.

The test-engines-rocksdb/test-engines-panic features have a similar problem, though it's not as bad since these aren't as widespread throughout the tree.

The change to resolver 2 also caused a number of crates to stop building on their own under cargo test. I'm trying to fix all these cases so that every crate builds under all feature configurations.

The change to resolver 2 also caused a number of crates to stop building on their own under cargo test. I'm trying to fix all these cases so that every crate builds under all feature configurations.


Set heat_metadata_server_url

This change adds the capability to detect and set the cfn route as the heat_metadata_server_url in the heat.conf file

We need a Watches() here for the route. So we need to reconcile once the openstack-operator has created the CFN route, which will update the heat.conf file with the Route URL.

Can you explain why we need to do the route lookup? Don't we have already that info passed in via the spec.heatCfnAPI.override.service.public.endpointURL:
  spec:
  ...
    heatCfnAPI:
        override:
          service:
            ...
            public:
              endpointURL: https://heat-cfnapi-public-openstack.apps-crc.testing

which is also used to register the endpoint:
sh-5.1$ openstack endpoint list | grep heat
| 02e7a0925e1b426a963ef6f3ba1a7925 | regionOne | heat         | orchestration  | True    | public    | https://heat-api-public-openstack.apps-crc.testing/v1/%(tenant_id)s   |
| 7ea6768b31a144dcad25dbfe85017695 | regionOne | heat-cfn     | cloudformation | True    | public    | https://heat-cfnapi-public-openstack.apps-crc.testing/v1              |
| ba7dd48987274ac299afd1ec2cf53dad | regionOne | heat-cfn     | cloudformation | True    | internal  | https://heat-cfnapi-internal.openstack.svc:8000/v1                    |
| cb1d444f717644028767fd45e900f9b9 | regionOne | heat         | orchestration  | True    | internal  | https://heat-api-internal.openstack.svc:8004/v1/%(tenant_id)s         |

And if none is provided it should be just be the k8s service, same how we handle the endpoint registration.  Like the public apiEndpoint without the dataPath https://github.com/openstack-k8s-operators/heat-operator/blob/main/controllers/heatcfnapi_controller.go#L486C3-L486C37

Can you explain why we need to do the route lookup? Don't we have already that info passed in via the spec.heatCfnAPI.override.service.public.endpointURL:
  spec:
  ...
    heatCfnAPI:
        override:
          service:
            ...
            public:
              endpointURL: https://heat-cfnapi-public-openstack.apps-crc.testing

which is also used to register the endpoint:
sh-5.1$ openstack endpoint list | grep heat
| 02e7a0925e1b426a963ef6f3ba1a7925 | regionOne | heat         | orchestration  | True    | public    | https://heat-api-public-openstack.apps-crc.testing/v1/%(tenant_id)s   |
| 7ea6768b31a144dcad25dbfe85017695 | regionOne | heat-cfn     | cloudformation | True    | public    | https://heat-cfnapi-public-openstack.apps-crc.testing/v1              |
| ba7dd48987274ac299afd1ec2cf53dad | regionOne | heat-cfn     | cloudformation | True    | internal  | https://heat-cfnapi-internal.openstack.svc:8000/v1                    |
| cb1d444f717644028767fd45e900f9b9 | regionOne | heat         | orchestration  | True    | internal  | https://heat-api-internal.openstack.svc:8004/v1/%(tenant_id)s         |

And if none is provided it should be just be the k8s service, same how we handle the endpoint registration. Like the public apiEndpoint without the dataPath https://github.com/openstack-k8s-operators/heat-operator/blob/main/controllers/heatcfnapi_controller.go#L486C3-L486C37

Yeah, I too realised this after I put in all this effort...
I have another iteration of this in testing atm using the endpointURL value. But it seems this isn't going to fix the problem I was trying to fix in tempest jobs anyway. I had initially speculated that the Tempest jobs were failing because it couldn't reach the metadata endpoint (this heat_metadata_server_url). Which we set in tripleo here:
https://github.com/openstack-archive/tripleo-heat-templates/blob/stable/wallaby/deployment/heat/heat-engine-container-puppet.yaml#L174-L178
So I can use this CFN endpointURL to set that, but the actual problem I need to solve is the signal URL. It resolves it from Keystone as the internal URL and this parameter I'm trying to set here won't change that.
Anyway, I'll close this and come back to my other patch once I figure out the immediate problem.


Refactoring go code

Mainly decluttering the main function by moving each even handler registration into a separate function.
There should currently only be one functional change. Namely that RoundEndOfficial events do not overwrite EndTick if it was properly set by RoundEnd
Also planning to add some tests where easily possible.

This is now also ready for a review.
There is still some stuff do to, but that is outside the scope of this PR.
For example there are still some extremely complex functions and there is also a lot of duplication with BombEvent handling.


Compilation Error OpenCV Make

Hi,
I get the following error when when I try to build. I am using Ubuntu 14.04.
Scanning dependencies of target HeadPoseEstimationFramework [  2%] Building CXX object HeadPoseEstimationFramework/CMakeFiles/HeadPoseEstimationFramework.dir/Grabber/PCDGrabber.cpp.o [  5%] Building CXX object HeadPoseEstimationFramework/CMakeFiles/HeadPoseEstimationFramework.dir/Grabber/OpenNIGrabber.cpp.o [  8%] Building CXX object HeadPoseEstimationFramework/CMakeFiles/HeadPoseEstimationFramework.dir/Grabber/ProviderGrabber.cpp.o [ 11%] Building CXX object HeadPoseEstimationFramework/CMakeFiles/HeadPoseEstimationFramework.dir/Grabber/GrabberBase.cpp.o [ 14%] Building CXX object HeadPoseEstimationFramework/CMakeFiles/HeadPoseEstimationFramework.dir/Converter/KinectDataConverter.cpp.o [ 17%] Building CXX object HeadPoseEstimationFramework/CMakeFiles/HeadPoseEstimationFramework.dir/Features/Calculator/CylinderOptimizedFeatureCalculator.cpp.o [ 20%] Building CXX object HeadPoseEstimationFramework/CMakeFiles/HeadPoseEstimationFramework.dir/Features/Sampler/CylinderSampler.cpp.o /home/psllsi/FaceCept3D/FaceCept3D/HeadPoseEstimationFramework/Features/Sampler/CylinderSampler.cpp:3:30: fatal error: opencv2\opencv.hpp: No such file or directory  #include <opencv2\opencv.hpp>                               ^ compilation terminated.
Could you please let me know what's the issue?
When I try to open pcl_openni_viewer, it throws a 'No Device Connected' error. Does this have anything to do with OpenNI1 vs OpenNI2 compatibility issue?
Thanks.

Hi,
Your compiler is not able to find <opencv2\opencv.hpp>.
Did you install opencv?

@sergeytulyakov  <opencv2\opencv.hpp>  should be <opencv2/opencv.hpp>  in linux :)


Issue #80: Use receiveAsync() to aggregate messages into shared queue for partitioned consumer

Motivation
Context can be found in #80. This is a slightly different approach from #83, using receiveAsync() and pausing and resuming reads depending on the shared queue utilization.

CLA is valid!


Pin deploy pages ubuntu version to fix failing CD

Now the deploy pages part of the CD (which was not run during my test run in my fork) fails because the previously used ubuntu-latest image changed to 24.04 which does not support Python 3.7. This pins the version to the previous 22.04 to fix this issue

@poggenhans could you quickly approve this PR if you have the time? The CD pipeline keeps failing because of external dependency changes 🙈


Project/Stack name case sensitivity issue

Basically, if you have a project name (referred to in the v0.31.0 UI as Stacks) that is for example "Test", and then you provide the rancher-compose CLI with -p Test, it will convert that to lowercase (test) and then be unable to find the stack/project due to test !== Test, so it will try and create test but get back a NotUnique because test and Test are the same when case insensitive.
:sob:

This is currently logged in Rancher at https://github.com/rancher/rancher/issues/1684 and the fixes need to happen in Rancher and Rancher-compose are:  https://github.com/rancher/rancher/issues/1693 and  https://github.com/rancher/rancher/issues/1694


[HUDI-886] Replace jsc.hadoopConfiguration by hadoop configuration in hudi-client testcase

What is the purpose of the pull request

Sub task of HUDI-538 Restructuring hudi client module for multi engine support

Brief change log

Replace jsc.hadoopConfiguration by hadoop configuration in hudi-client testcase

Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end.
Added HoodieClientWriteTest to verify the change.
Manually verified the change by running a job locally.

Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

Looks good.


Move meta.lua to it's own top-level location

Since meta.lua is currently within the Utils folder, other Utils can't use a lot of the helper functions I've made. This is because meta.lua is loaded before everything else so. that it can double check that importing is happening is happening for all non-exluded files. Side effect of loading meta first is that I also load all of the Utils folder (is this true?? may need to check) so if a utils file calls, for example my ltp function (alias for listTableProps), it will throw a global 'ltp' is not callable (a nil value) error

This issue actually has more to do with import ordering after further inspection. The task here is to create a declarative way of specifying import orders. For instance, I will want my print functions and associated aliases to import before everything else so that those aliases will be available for use by all files imported after.

I could see a simple labeling system being sufficient for my purposes:

Root
Debug
Development
Production

Not married to this list, but something to this effect would be helpful. This system should effectively decouple the file structure from the import order. Then, if I am working in a file that is typically a root or debug kind of file, I could temporarily set it's type to 'Development' to allow access to other util-like functions that it normally would not have access to.

There's also the approach of importing files without executing them. This would have more to do with how I write my scripts generally. A heavier focus on OOP would probably give this to me for free. Will keep thinking about this


*: make CWrappedNative interface consistent with CCollateralCap

I think we also need totalCollateralTokens interface on wrappedNative too, right?


Pending contents for 20210913

https://internals.tidb.io/t/topic/382

https://internals.tidb.io/t/topic/374

https://github.com/pingcap/tidb/pull/27325

https://internals.tidb.io/t/topic/385


Contents of directory pointed to by --db-path is lost if the database cannot be read

Describe the bug
When a VERSION file exists in the --db-path directory, but for any reason, the database cannot be read (could be a spurious IO error), the current behavior of meilisearch is to silently remove the entire contents of that directory and replace it with a fresh database.
To Reproduce
Steps to reproduce the behavior:

Create a fresh db: meilisearch -- --db-path tmp-db.ms
Add something to the db:curl \
    -X POST 'http://localhost:7700/indexes' \
    -H 'Content-Type: application/json' \
    --data-binary '{
        "uid": "movies",
        "primaryKey": "id"
    }'

Check it is there:curl -X GET 'http://localhost:7700/indexes'

returns:{"results":  [{"uid":"movies","createdAt":"2022-11-16T13:44:58.985399Z","updatedAt":"2022-11-16T13:44:58.988185Z","primaryKey":"id"}],"offset":0,"limit":20,"total":1}

Stop meilisearch, then simulate IO errors: chmod -r tmp-db.ms/tasks/data.mdb
Restart meilisearch on the DB, in this instance, it exits with an error: Error: Permission denied (os error 13)
Restart meilisearch again, this time it restarts although we didn't chmod +r
curl -X GET 'http://localhost:7700/indexes' returns {"results":[],"offset":0,"limit":20,"total":0}

Expected behavior

The returned error indicates that the database cannot be read
The database is not removed

Meilisearch version: v0.30.0-rc1
Additional context
Encountered in a less contrived case of having mistakenly opened a database created with a different (future) version of lmdb, resulting in the loss of that database.

Fixed by https://github.com/meilisearch/meilisearch/pull/3077


11193 DEV update baseMigrations/migrate.js to populate supportEmailAddress

https://github.com/Shippable/pm/issues/11193

This is done.


feature: allow push to current branch

Hello everybody,
We are using trunk based development and want to allow pushing the changes made by the template to the main branch.
I want to push to the current branch from the azure:repo:push action.
Currently, this fails with the branch already exists because it tries to create a new one.
I have created a fix for it but am not allowed to create branches in this project.
Please find the code snippet below in case you prefer to add it yourself. Otherwise, I am happy to submit a PR if you give me permission.
helpers.ts
...
 const currentBranch = await git.currentBranch({dir})

  if (currentBranch !== branch) {
    await git.branch({
      dir,
      ref: branch,
    })

    await git.checkout({
      dir,
      ref: branch,
    });
  }
...

Let me know if you need more info :)
Thanks!

@JetteBke  you can fork the project, create the changes in your tree and then PR back against our tree.
Have a try and let me know if you need help :)

hi @hevans-dglcom , I am gonna try that, thank you! :)

hi again @hevans-dglcom , I added a PR: #13
I didn't find a good way to test this locally, but it should be fine.
Let me know if it works.

Thank you so much @hevans-dglcom , it worked! 🥳


Introducing the MSOutput module for Reqmgr2ms.

Fixes #9606
Status
In development && not-tested
Description
Introduces the new MSOutput module for Reqmgr2ms. For more details see the issue to be fixed withe the current PR
Is it backward compatible (if not, which system it affects?)
YES
Related PRs
We need the configuration parameters to be set in the deployment scripts:
https://github.com/dmwm/deployment/pull/890
External dependencies / deployment changes
No

Of course unit tests will fail here. It's importing the DDM module, which has not been merged yet. I'm looking into it

test this please


[求助/Help] how to delete a server IP once a VM is deleted

Hello!
I'm currently evaluating Cloudpods all in one, and was able to provision a QEMU VM with no issues. However, once I deleted the VM, it seems like the IP on the VNIC doesn't get cleaned up.
This leads me to have a network on my VPC that I cannot delete. I tried looking for an option in climc but the closest command only lists the network ips.

@eddiewang the server might be in pending delete state, please check with command line:
climc server-list --pending-delete

You may clean up the server permanently by:
climc server-delete -f <sid>

thank you @swordqiu - that worked great. do you know why the server would be in pending delete state?

@eddiewang this is a feature by design, to give user a chance to recover deleted VM. If a VM was pending deleted for  curtain period (three days by default), the VM will be finally cleaned up.


Failing Test - "should remove a recipe from the meal plan"

The following test is failing in testMealPlan.js
See details below:

Removing from the meal plan was being considered as trying to add an undefined recipe to the database.
The check for valid recipeIDs was altered to accommodate removal functionality

Resolved in pull request #35


The editor crashes when exiting playmode while it is creating the LLM service

Describe the bug
This happens if asynchronous startup is true. When you exit playmode after the "Server command" log but before the "LLM service created" log, then the editor just closes.
Steps to reproduce
No response
LLMUnity version
2.0.0
Operating System
Windows

Same error. But happens regardless of async startup bool value. v1.2.9 example projects worked fine, but started a new project with 2.0.2 and they error out now.
LLMUnity Version
2.0.2
Operating System
Mac

I can't replicate it so far.
@stanchiang do you get the crashes in general or only when you exit playmode before the llm is created?


TLCC revise FPC transaction validation

Description
TLCC FPC transaction validation logic needs to be aligned with recent changes in ecc-vscc as introduced with PR #371.
Another issue is that the current tlcc code requires a ecc_ prefix which was removed with PR #320. Quick and dirty fix applied but needs to resolved. More detailed discussion on PR #323

Additional info in:

comment https://github.com/hyperledger-labs/fabric-private-chaincode/pull/388/files#r447985598 (including suggestion to look for the validation plugin)


CorpNetSaw and CorpNetPublic are not valid service tags in NSG

I am not able to add the default rules required for the Managed Instances.
https://docs.microsoft.com/en-us/azure/azure-sql/managed-instance/connectivity-architecture-overview#mandatory-inbound-security-rules-with-service-aided-subnet-configuration

@muditmittal1985, as per the Azure doc, you cannot manually select the "CorpNetSaw" service tag from the portal. You must use Azure PowerShell or Azure CLI to manually configure a rule that uses the CorpNetSaw service tag.
Get-AzureRmNetworkSecurityGroup -Name "nsg-name" -ResourceGroupName "resource-group-name" | Add-AzureRmNetworkSecurityRuleConfig -Name "new-rule-name" -Access "Allow" -Protocol "TCP" -Direction "Inbound" -Priority "priority-number" -SourceAddressPrefix "CorpNetSaw" -SourcePortRange "" -DestinationPortRange "3389" -DestinationAddressPrefix "" | Set-AzureRmNetworkSecurityGroup

SS:

Ref Doc: https://docs.microsoft.com/en-us/azure/active-directory-domain-services/network-considerations

@muditmittal1985, we are closing this issue for now. If there are further questions regarding this matter, please reply and we will gladly continue the discussion.


New version: LifeInsuranceDataModel v0.6.2

Registering package: LifeInsuranceDataModel
Repository: https://github.com/Actuarial-Sciences-for-Africa-ASA/LifeInsuranceDataModel.jl
Created by: @michaelfliegner
Version: v0.6.2
Commit: e73d11012bf48c79ffd46b2b07a1a9cb7a524569
Reviewed by: @michaelfliegner
Reference: https://github.com/Actuarial-Sciences-for-Africa-ASA/LifeInsuranceDataModel.jl/commit/e73d11012bf48c79ffd46b2b07a1a9cb7a524569#commitcomment-98771408
Description: bitemporal data management for prototypical life insurance data model

@JuliaRegistrator register


vuex and vue-worker

Is it possible to access the vuex store from within a function passed onto a worker ?

I would like to know simply how to call a worker from a VueX action. Can somebody provide some guidance?

Is it possible to access the vuex store from within a function passed onto a worker ?

This is likely not possible (or not feasible), because Web Workers don't share any data with the main thread. By design, communication between workers is done by passing messages, and such data is copied but never "shared".
While technically it could be possible to set up a bunch of messages so that a worker could ask for data from the main thread, this is likely going to increase the overhead and the complexity of the code, and making it much slower to run and harder to maintain.

Not true. Currently indexedb is blocking  - and if you  want to sync the store with your indexedb instance - using workers is the only way to make it truly non blocking. And yes, it is possible.  The app will be faster.  However I do not know if the module allows it, I have used worker-loader module before.


feat: wait for initial params after connection init

After connection init, the mixer sends all current state params to the client. Commands that depend on state, e.g. "toggle mute", cannot be reliably executed before all state is present.
We now wait until the state hasn't changed for a while which indicates that all ~5000 params have been received. In case the state changes for a longer time, we also have a timeout that ends the connection process.

@Kaputtnique This should solve the problem that toggleMute() does not work immediately after connection to the mixer.

alright, cool, thanks for the fast and very good support 👍


Baseurl support

May I use a global config like axios set the baseurl? This feature is useful.

superagent-use


DO NOT MERGE -- warn about explicit meta-metatypes

A naive grep of the source compatibility suite suggests that users aren't using meta-metatypes. Let's test that theory.
For random people looking at this pull request, I'm of the opinion that removing meta-metatypes from the language allows for better internal optimizations.

@swift-ci please test source compatibility

It looks like the compatibility testing doesn't output to the console (and I really doubt that every test compiles warning free). Retrying as an error now to force failures rather than warnings.
@swift-ci please clean test source compatibility

@swift-ci please test source compatibility

@swift-ci please test source compatibility

Please update the base branch to main by Oct 5th otherwise the pull request will be closed automatically.

How to change the base branch: (Link)
More detail about the branch update: (Link)


Add virtualenv usage when running from source

Hey, it slipped my mind to do the virtualenv . and source bin/activate when installing & running this app from source. I presume that it should be executed in this order for it to work properly?

You shouldn't need virtualenv with this project unless you're re-building packages.zip. If you run the python code via run_map.py or through Electron, the dependency tree should be taken care of.

Oh, okay - I'll take your word for it. Thanks!


Snap not working in 6.5.0

Describe the bug
Snapping does not work correctly in 6.5.0 - Works 100% in 6.4.3
To Reproduce
Drag line (starting some distance from line), then drag to point and snap.  You will see that snap occurs on point at the same distance that dragging started from line.  Maybe to do with delta_ in ol/interaction/Modify:
Modify.prototype.handleDragEvent = function (evt) {
        this.ignoreNextSingleClick_ = false;
        this.willModifyFeatures_(evt, this.dragSegments_);
        var vertex = [
            evt.coordinate[0] + this.delta_[0],
            evt.coordinate[1] + this.delta_[1],
        ];

See Code and video below
Video:
https://user-images.githubusercontent.com/1152458/106648236-aa358280-6598-11eb-80fb-b18868f2e536.mp4
Code:
import "ol/ol.css";
import Map from "ol/Map";
import View from "ol/View";
import GeoJSON from "ol/format/GeoJSON";
import { Modify, Snap } from "ol/interaction";
import { Vector as VectorSource } from "ol/source";
import { Vector as VectorLayer } from "ol/layer";
import { transform } from "ol/proj";

var vector1 = new VectorLayer({
  source: new VectorSource({
    format: new GeoJSON(),
    features: new GeoJSON().readFeatures({
      type: "FeatureCollection",
      features: [
        {
          type: "Feature",
          geometry: {
            type: "LineString",
            coordinates: [
              [2059761.8846943479, -4001987.9903377485],
              [2059183.3271955752, -4002069.1811360097]
            ]
          },
          properties: {}
        }
      ]
    })
  })
});

var vector2 = new VectorLayer({
  source: new VectorSource({
    format: new GeoJSON(),
    features: new GeoJSON().readFeatures({
      type: "FeatureCollection",
      features: [
        {
          type: "Feature",
          geometry: {
            type: "Point",
            coordinates: [2059524.189759539, -4002117.2223334215]
          },
          properties: {}
        }
      ]
    })
  })
});

var map = new Map({
  layers: [vector1, vector2],
  target: "map",
  view: new View({
    center: transform(
      [18.50310560948664, -33.799754138117294],
      "EPSG:4326",
      "EPSG:3857"
    ),
    zoom: 17
  })
});

var ExampleModify = {
  init: function () {
    this.modify = new Modify({
      source: vector1.getSource()
    });
    map.addInteraction(this.modify);
  },
  setActive: function (active) {
    this.modify.setActive(active);
  }
};
ExampleModify.init();

ExampleModify.setActive(true);

// The snap interaction must be added after the Modify and Draw interactions
// in order for its map browser event handlers to be fired first. Its handlers
// are responsible of doing the snapping.
var snap = new Snap({
  source: vector2.getSource()
});
map.addInteraction(snap);

Test:
https://codesandbox.io/s/snap-forked-g2dge
Expected behavior
Should snap correctly

Thanks for the report.
This is likely caused by #11769 and the change to the dragged features position under the mouse pointer.
The placement under the mouse pointer used to be the center of the point, with 6.5.0 it was changed to the position it was clicked on.
The version on codesandbox for this change is 6.4.4-dev.1607296357109.

Ok, so how would I go about for the above code to work in 6.5.0 ?

This is an unintentional regression.
I gave it a first shot with https://github.com/MoonE/openlayers/commit/63e0a7dec113c66d76abfe312843e5afdf54f33e, but the snap distance is not working correctly this way because the origin of the snap tolerance measurement is the mouse pointer, and not the vertex center, like it should be.

Did you take the code example above and tested it with 6.4.3 and then again with 6.5.0?  You will see that in 6.4.3 it works as expected, but not in 6.5.0.  You are correct that the mouse pointer should be used to measure the snap tolerance, and it works that way when starting the drag. The problem is when you snap after a drag to an existing point, then it does not snap to the target point, but rather with the same offset when the drag started (on the source point).

I did test it with both versions and with my commit above it should snap to the correct position.
But if the mouse and vertex center are different the snap tolerance is larger to the side where the mouse pointer is.
The problem with implementing this correctly now is that there is kind of a circular dependency.
Snap depends on Modify to use the correct snap tolerance center point from the dragged vertex.
Modify depends on Snap to get the correct coordinate.
Not sure how to best resolve this.

Not sure if it is a solution in your case @1Map, but the problem goes away when you also make the source geometry also a snap target, which is e.g. the case in https://openlayers.org/en/latest/examples/snap.html.
@MoonE: the problem you're describing regarding vertex center vs mouse position must have existed prior to the introduction of the delta calculation in ol/interaction/Snap as well.
Going forward, I think the best option is to make the delta calculation optional. ol/interaction/Modify could get a new option snapToPointer. When set to true, no delta between vertex/point and pointer will be calculated and applied.

Hi @ahocevar
The problem does go away when the source geometry is also a snap target, but this will not solve my problem as in my case I want to prohibit users to snap to certain features.
The real problem here is as follow:
You use delta (cursor position to feature position) when start dragging a snappped feature (which is 100% correct and also works 100%).  The problem now is, because that code lives in Modify it does the same logic when a dragged feature is snapped to a target feature.  Here the feature do get snapped, but with the new calculated delta (cursor position to target snap feature), which is wrong as we want the feature to snap to exactly the centroid of the target feature.  So, the code should only use delta if you start dragging, but not when you drop the feature.

@1Map Yes, I do understand the problem. Please test with ba4077237b, and configure the Modify interaction with snapToPointer: true.

Hi @ahocevar
Thanks, this works perfect.  Will this change be in the next release, and when can the public expect this?

I'm going to finalize a pull request with the above change. Once that is reviewed and merged, you can use it with npm install ol@dev, and it will also be in the next release.

#11978 is the pull request that introduces the snapToPointer option mentioned in https://github.com/openlayers/openlayers/issues/11976#issuecomment-772295018.


[Bug]: Fixing issues in disabled Choreo samples

Connector Name
connectors/samples (Connector Samples/Templates)
What happened?
There are 22 samples that are hidden on Choreo due to various issues on them.
We need to identify all issues, group the issues accordingly, and fix them.

Disabled samples
github_issue_summary_to_email'
gcalendar_to_twilio'
github_new_commit_to_google_sheet_row'
sfdc_outbound_msg_to_gsheet'
twilio_new_sms_to_gmail_inbox'
twilio_sms_to_gsheet'
gdrive_new_file_to_gsheet_rows'
gmail_new_attachment_to_drive'
gdrive_new_file_to_twilio_sms'
json_to_xml_conversion'
xml_navigation'
circuit_breaker'
github_issue_assigned_to_sms'
github_issue_assigned_to_twilio_sms'
github_new_release_to_twilio_sms'
github_release_to_slack'
gsheet_new_row_to_github_new_issue'
gsheet_new_row_to_sfdc_new_lead'
gsheet_new_row_to_sfdc_new_record'
gsheet_row_update_to_gmail'
gsheet_update_row_to_sfdc_update_record'
slack_new_message_to_twilio_sms'
Relevant log output
No response

Related issues
https://github.com/wso2-enterprise/choreo/issues/11797
https://github.com/wso2-enterprise/choreo/issues/11798
https://github.com/wso2-enterprise/choreo/issues/11800
https://github.com/wso2-enterprise/choreo/issues/11801
https://github.com/wso2-enterprise/choreo/issues/11802
https://github.com/wso2-enterprise/choreo/issues/11803
https://github.com/wso2-enterprise/choreo/issues/11805
https://github.com/wso2-enterprise/choreo/issues/11803
https://github.com/wso2-enterprise/choreo/issues/11806
https://github.com/wso2-enterprise/choreo/issues/11850

All samples are now enabled except circuit_breaker [1]
[1] https://github.com/wso2-enterprise/choreo/issues/11850
Issue can be closed after enabling [1]

Closing since its fixed via https://github.com/wso2-enterprise/choreo/issues/11850


Controller.start uses threading

https://github.com/aio-libs/aiosmtpd/blob/3ced778177418d8ae47bc3c300879bf59e731f1a/aiosmtpd/controller.py#L54
It seems to be that Controller.start using Threading is not required and will lead to problems.
Like the asyncio http server starting an SMTP server should not involve threading, but just loop.run_forever().

@samuelcolvin so if I don't misunderstood, you want to split out the thread-starting ability from Controller?

Why is the default behavior to run in a thread?
Mirroring Samuel's request, having the ability to run it in the foreground without any threading would be useful.

@nhoad One thing to realize is that Controller is just a "convenience class" to instantiate the SMTP server.
The code for Controller is very well designed; if you need a non-threading, in-the-foreground solution, you can simply take the code within the Controller._run() method (that is, select the text, copy them, and paste in your code) and run it.
All the intelligence of the SMTP server happens in the aiosmtpd.smtp.SMTP class; it's self-contained.

I have a branch waiting for #247 to be merged first.

Boy oh boy... apparently implementing unthreaded controllers is much more complex than I thought.
Testing them properly is a whole 'nother level.

If anyone wants to take a peek and/or kick the tires, be my guest: https://github.com/pepoluan/aiosmtpd/tree/unthreaded-controller

The branch is about 95% complete, documentation also updated.
Feel free to check it out @samuelcolvin @nhoad


Mock Service Worker Conversion - Initial Setup

Initial setup for replacing custom mocking and fetching layer mocking with
Mock Service Worker allowing for mocking in the network layer of requests and responses,
which will allow for full testing of code with minimal mocking of it.
Tests converted to Mock Service Worker w/ included request input checks in handlers inline in certain tests:

r2.test.ts
whoami.test.ts

Special thanks to @caass for the fetch fall-through logic

emphasizing the Delta on internal 😆

I feel a bit uncomfortable about how we mock out a load of "happy path" requests by default. Although this might make writing tests simpler, it means that it is not possible to tell in any given tests, which endpoints we expect to hit and which not. Have you considered using helpers in specific tests to add in handlers to msw on a needs-must basis?

The pattern is that the test without inlined Mocks would be using the default handlers, the inline mocks which look like this msw.use(rest.<method>(URL, (req, res) => response.once()) would indicate a deviation from the default behavior, explicitly indicating what paths are being tested.

Have you considered using helpers in specific tests to add in handlers to msw on a needs-must basis?

This would basically be using the inline handler pattern (even if abstracted with a helper) for every test or test suite not just edge cases or deviations from "happy path". I feel like this misses out on a MSW standard/practice of top level handlers with the initialized MSW server.

This looks fine to me -- great work! A few thoughts for future work:

It feels like there should be one level of abstraction here. I'm not sure what, exactly, yet, but I think it will become obvious as we write more of these tests.
It's worth creating a tracking issue for the tests that still have to be converted, and smaller issues tracked in there.
Some small helpers (e.g. ctx.ok for JSON ok responses and ctx.error for JSON error responses) wouldn't go amiss, I don't think

I agree with these 3 points and will create an issue regarding them, then break them into separate issues


Twitch Video onSeek not fired and bug for seekTo when pause

With any Twitch videos with controls, if I seek from the Twitch video controls, the onSeek method doesn't be called but the onPlay and onPause methods work.
There is a bug on Twitch videos when call seekTo. It works when playing is true but when the video is on pause. SeekTo method not work. Normally it's possible because on Twitch Video controls, we are able to seek the video when we are on pause or playing.
Thanks you for your awesome work

I just saw on the CookPete demo, the seekTo method doesn't work properly on Twitch video when playing is true (and the video time seek is not loading yet) and not work at all when playing is false.

There is no onSeek event for the Twitch player – it's nowhere in the Twitch embed docs.
As for seekTo, ReactPlayer is calling the seek method exactly as the docs describe. It's a problem with the Twitch embedded player if it doesn't work while paused. The docs even say

Seeks to the specified timestamp (in seconds) in the video and resumes playing if paused.

which is obviously not what it actually does. There's nothing that this library can really do about this.

@xVanilla did you find a workaround? We are having the same issue :(

This should now be fixed - seek can be called at any time (but won't work for live streams), and a seek event was added.

That's awesome! Thank you!


Xiaomi MiIO Fan: Xiaomi Air Purifier 2 integration

Description:
This is a custom component for home assistant to integrate the Xiaomi Air Purifier 2.
https://github.com/syssi/xiaomi_airpurifier
Features

On, Off
Operation modes (auto, silent, favorite, idle)
Buzzer (on, off)
LED (on, off), LED brightness (bright, dim, off)
Favorite Level
States

power
aqi
humidity
temperature
mode
led
led_brightness
buzzer
child_lock
brightness
favorite_level
filter1_life
f1_hour_used
use_time
motor1_speed

Pull request in home-assistant.github.io with documentation (if applicable): https://github.com/home-assistant/home-assistant.github.io/pull/3605
Example entry for configuration.yaml (if applicable):
# confugration.yaml

fan:
  - platform: xiaomi_airpurifier
    name: Xiaomi Air Purifier 2
    host: <IP_ADDRESS>
    token: b7c4a758c251955d2c24b1d9e41ce47d

Ready to merge as soon as @balloob is satisfied with the service names.


Support for mix/max reduction in OpenMP reduction

cc @diorcety this should fix the issue we had during the live session.


Padded nav y extent.

Relates to #571.

Possibly go for padding = [0, 0.04], to be consistent with the primary chart, and you don't need to declare it as a variable? (0.08 / 2 = 0.04, and we don't want to pad the bottom value?)


[4.x] Bump okio to 3.3.0

fixes https://github.com/square/okhttp/issues/7654

That didn't work because okio is kotlin 1.8. Not sure I want to bring that in for OkHttp 4.x at this time.
So took a different approach.

Found it. https://github.com/square/okhttp/issues/7654


Update cuemod init workflow

Currently cuemod init command looks for the value provided in the --tool flag and supports k8s, argocd, tektoncd, crossplane, and clusterapi tools.
It is proposed to enhance the above workflow to also accept OCI artifact URLs to create Cue workspace.
The artifact stored in the OCI registry should contain the following minimum requirements for creating a Cue workspace:
.
├── cue.mod
└── policy.cue

The cue.mod subdirectory Must contain all the upstream APIs in Cue format for the technology we intend to work with.

closed in #60


Goto Definition is not working

Description
What steps will reproduce the problem?

After right clicking on the function, its not showing "goto defition option".

Hi,
I am using 4.1.4 version.
Thanks,
Somanna
On Tue, 4 Aug 2020, 11:06 pm Stephannie Jimenez Gacha, <
<EMAIL_ADDRESS>wrote:

Hi @SomannaKuri https://github.com/SomannaKuri,
With which function are you trying to use the goto definition option?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/spyder-ide/spyder/issues/13461#issuecomment-668730374,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJTGCSTC3YSBA6FWPWNOM73R7BBIFANCNFSM4PTAKPDQ
.

Can you please let me know which function gives you the error? In this way we can try to reproduce the issue

When I right click on the function, which is called from other package or
within same file, and select "go to definition", it's not working.
Thanks & Regards,
Somanna
On Wed, 5 Aug 2020, 1:23 am Stephannie Jimenez Gacha, <
<EMAIL_ADDRESS>wrote:

Can you please let me know which function gives you the error? In this way
we can try to reproduce the issue
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/spyder-ide/spyder/issues/13461#issuecomment-668793283,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJTGCSTGNEQ4FJPSI447V33R7BRKTANCNFSM4PTAKPDQ
.

Can you please upload a gif to see the behavior of the error?

Hi Stephannie,
Thank you so much for your response. After restarting the system its
working fine.
Please find the attached file.
Thank you,
Somanna
On Wed, Aug 5, 2020 at 1:37 AM Stephannie Jimenez Gacha <
<EMAIL_ADDRESS>wrote:

Can you please upload a gif to see the behavior of the error?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/spyder-ide/spyder/issues/13461#issuecomment-668799419,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJTGCSUYEHSUGT3FSJHQSC3R7BS7NANCNFSM4PTAKPDQ
.

So your problem is fixed now?

Yes it is fixed now.
Thank you,
Somanna
On Wed, 5 Aug 2020, 8:43 pm Stephannie Jimenez Gacha, <
<EMAIL_ADDRESS>wrote:

So your problem is fixed now?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/spyder-ide/spyder/issues/13461#issuecomment-669251928,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJTGCSS2QHBAGTGVLHPUZ6TR7FZJ5ANCNFSM4PTAKPDQ
.

I'm closing this issue since it is fixed, if you encounter another error please open a new issue.
Thanks for reporting!


Fix filename for ModelValidatorProviderContext (fixes #4798)

I found another one as well. I'm still exploring the repository.

Thanks for the PR! I'll merge this in shortly.

Thanks again!


Drop individual icon imports from styled-icons package

Problem
So I can't publish styled-icons v9.6.0 because we have hit a file count limit and NPM is rejecting the publish. 🙂 There are now 67,776 files in that package after the addition of the material-* variant icon packs.
Today we are proxying all icons in the styled-icons package to the corresponding @styled-icons/* packages - we proxy both the pack as a whole and each individual icon.  Each proxied icon results in 5 files in the npm package.
Proposal
We should release a new major release (v10) where only the icon packs themselves are proxied, and if consumers wish to import icons individually, they would need to import from @styled-icons/pack.  Specifically:
Today (v9)
// Works:
import {material} from 'styled-icons'
import * as material from 'styled-icons/material'
import {Alarm} from 'styled-icons/material'
import {Alarm} from 'styled-icons/material/Alarm'

// Also works:
import * as material from '@styled-icons/material'
import {Alarm} from '@styled-icons/material'
import {Alarm} from '@styled-icons/material/Alarm'

Proposed (v10)
// Works:
import {material} from 'styled-icons'
import * as material from 'styled-icons/material'
import {Alarm} from 'styled-icons/material'

// Does not work anymore:
import {Alarm} from 'styled-icons/material/Alarm'

// Still works:
import * as material from '@styled-icons/material'
import {Alarm} from '@styled-icons/material'
import {Alarm} from '@styled-icons/material/Alarm'

This would cut down the number of files to a very reasonable amount.
@stramel does this seem okay to you?

I was actually going to take a stab at re-working the structure last night but didn't get time to.
I'm definitely on board for the proposed. Which is basically styled-icons becoming a bunch of export * as material from '@styled-icons/material.
With that being said, I'm not sure I see the benefit of maintaining that package anymore. If you want multiple, you could easily just pull in the packs you want. It will save install time. I imagine the primary use-case is a single package anyways.
We could add a deprecation notice for v10 and make styled-icons just a bunch of re-exports. ex.
export * as foundation from '@styled-icons/foundation'
export * as material from '@styled-icons/material'
// ....

Opened #1174 to implement this - I don't think want to necessarily deprecate styled-icons directly (as in, we don't want a NPM deprecation message) as I know some of the apps I work will will still want to import from styled-icons to pick up everything, but I've adjusted the docs to highlight the @styled-icons/PACK packages instead.

I was thinking by deprecation notice, it would be a console message maybe  a postinstall script that would encourage users to switch to the new syntax so in a future release we can drop support for styled-icons. But will leave it up to you

I was thinking by deprecation notice, it would be a console message maybe a postinstall script

If/when we do deprecate styled-icons, just a note that Yarn silences the output of all postinstall scripts, so only marking packages as deprecated on npmjs.com actually displays something in the consumer's terminal (NPM does display postinstall output IIRC).


[BugFix] Fix BE crash when enable filter_unused_columns and dictionary inconsistent

Usually this doesn't happen. This is because the only known inconsistency that occurs is triggering DELETE. This PR is only programmed as a defense.
What type of PR is this:

[x] BugFix
[ ] Feature
[ ] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Does this PR entail a change in behavior?

[ ] Yes, this PR will result in a change in behavior.
[x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

[ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
[ ] Parameter changes: default values, similar parameters but with different default values
[ ] Policy changes: use new policy to replace old one, functionality automatically enabled
[ ] Feature removed
[ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto-backported to the target branch

[x] 3.1
[x] 3.0
[x] 2.5
[ ] 2.4

[BE Incremental Coverage Report]
:heart_eyes: pass : 10 / 11 (90.91%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
src/storage/rowset/segment_iterator.cpp
10
11
90.91%
[1401]

[FE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[BE Incremental Coverage Report]
:heart_eyes: pass : 10 / 11 (90.91%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
src/storage/rowset/segment_iterator.cpp
10
11
90.91%
[1412]

@Mergifyio backport branch-3.1

@Mergifyio backport branch-2.5


Sensor drivers for Disco L475 IoT board

Provide drivers (and update) for following drivers built in on Disco L475 Board
LPS22HB pressure and temperature
LIS3MDL 3D axis magnetometer
LSM6DSL accelerometer and gyrometer

Thanks Maureen and Bogdan for your reviews.
I've integrated all the comments except DEVICE_AND_API_INIT on which I'm waiting your confirmation (as DEVICE_INIT is widely used in sensor drivers)

Second update with remaining comments

@erwango Don't forget to update the wiki with the new sensors

Done, thanks for the reminder!


Sunset subject:organizationalUnitName (Section <IP_ADDRESS>.2.i, CAB-Forum BR)

This pull request adds a lint for "Sunset subject:organizationalUnitName" - Section <IP_ADDRESS>.2.i of CAB BR (v1.7.9 - https://cabforum.org/wp-content/uploads/CA-Browser-Forum-BR-1.7.9.pdf). In this pull request the time for the CABFBRs_1_8_0_Date in time.go (from 21st of August to 25th) has also been adjusted (see Section 1.2.1, https://cabforum.org/wp-content/uploads/CA-Browser-Forum-BR-1.8.0.pdf) and you may consider merging only this change to the main project.

Is this pull request still considered for inclusion? Is there somethig to do on our side?
Please note this change: https://github.com/zmap/zlint/pull/643#discussion_r734993945
which is interesting for the main project and is independent of this new lint. I could make a new pull request that adresses only this issue.

Comments have been addressed. The duplicate lint to #675 is removed along with the corresponding test data. Only e_subject_contains_organizational_unit_name_and_no_organization_name is still present.