Pure Cling demo and improvements to C++ JIT infrastructure

This is based on the Cling standalone you gave me, @vgvassilev.

@lukasheinrich, I should include you here, as this will be relevant for my talk to ATLAS, just as #1295 is.

@jpivarski, is the C++ code available somewhere?

The code you gave me is not included in this PR. I haven't figured out how to distribute it, yet, or even how much will be in clangdev itself.
The C++ samples I showed you yesterday (in Slack) were generated. I can copy them here.

Although you can't test it without @vgvassilev's InterpreterUtils.cpp (and maybe I can integrate that into Awkward's compiled code? no, not without depending on Cling...), this version is able to do the following:
>>> import awkward as ak
>>> from awkward._v2._connect.cling import CppStatements
>>>
>>> a = ak._v2.Array([[0.0, 1.1, 2.2], [], [3.3, 4.4]])
>>>
>>> f = CppStatements("""
...
... for (int i = 0; i < array.size(); i++) {
...   printf("[\\n");
...   for (int j = 0; j < array[i].size(); j++) {
...     printf("  %g\\n", array[i][j]);
...   }
...   printf("]\\n");
... }
... """, array=a)
>>>
>>> f(array=a)
[
  0
  1.1
  2.2
]
[
]
[
  3.3
  4.4
]

The array=a passed to the CppStatements constructor could instead be a Form, like array=a.layout.form (or more likely, something derived from a Parquet schema).
Behind the scenes, this is the C++ code that was generated and JIT-compiled:
#include<sys/types.h>
extern "C" int printf(const char*, ...);

namespace awkward {
  class ArrayView {
  public:
    ArrayView(ssize_t start, ssize_t stop, ssize_t which, ssize_t* ptrs)
      : start_(start), stop_(stop), which_(which), ptrs_(ptrs) { }

    size_t size() const noexcept {{
      return stop_ - start_;
    }}

    bool empty() const noexcept {{
      return start_ == stop_;
    }}

  protected:
    ssize_t start_;
    ssize_t stop_;
    ssize_t which_;
    ssize_t* ptrs_;
  };
}
namespace awkward {
  class NumpyArray_float64_bdYFlthWcck: public ArrayView {
  public:
    NumpyArray_float64_bdYFlthWcck(ssize_t start, ssize_t stop, ssize_t which, ssize_t* ptrs)
      : ArrayView(start, stop, which, ptrs) { }

    typedef double value_type;

    value_type operator[](size_t at) const noexcept {
      return reinterpret_cast<double*>(ptrs_[which_ + 1])[start_ + at];
    }
  };
}
namespace awkward {
  class ListArray_ms7qTHCb7Ik: public ArrayView {
  public:
    ListArray_ms7qTHCb7Ik(ssize_t start, ssize_t stop, ssize_t which, ssize_t* ptrs)
      : ArrayView(start, stop, which, ptrs) { }

    typedef NumpyArray_float64_bdYFlthWcck value_type;

    value_type operator[](size_t at) const noexcept {
      ssize_t start = reinterpret_cast<int64_t*>(ptrs_[which_ + 1])[start_ + at];
      ssize_t stop = reinterpret_cast<int64_t*>(ptrs_[which_ + 2])[start_ + at];
      return value_type(start, stop, ptrs_[which_ + 3], ptrs_);
    }
  };
}

void awkward_function_0(ssize_t awkward_argument_1_length, ssize_t awkward_argument_1_ptrs) {
auto array = awkward::ListArray_ms7qTHCb7Ik(0, awkward_argument_1_length, 0, reinterpret_cast<ssize_t*>(awkward_argument_1_ptrs));

for (int i = 0; i < array.size(); i++) {
  printf("[\n");
  for (int j = 0; j < array[i].size(); j++) {
    printf("  %g\n", array[i][j]);
  }
  printf("]\n");
}

}

The user code is the last part, in the function with the generated name. Still thinking about interface...
Thanks to @vgvassilev for all the help in getting this to work! (It involved quite a lot of compiler details I didn't know about.)

For future reference (so that the information exists somewhere other than my hard drive), we needed
  std::vector<const char *> ClangArgv = {"-Xclang", "-emit-llvm-only",
                                         "-fPIC", "-fno-rtti", "-fno-exceptions"};

to get this to work because the clangdev package that ships in conda-forge has exceptions turned off.
Also, it seems to me that this is the place to put "-O3", unless it can also be controlled by a #pragma.

For future reference (so that the information exists somewhere other than my hard drive), we needed
  std::vector<const char *> ClangArgv = {"-Xclang", "-emit-llvm-only",
                                         "-fPIC", "-fno-rtti", "-fno-exceptions"};

to get this to work because the clangdev package that ships in conda-forge has exceptions turned off.
Also, it seems to me that this is the place to put "-O3", unless it can also be controlled by a #pragma.

This is the sign of someone who has been burned by this before :smile:

For future reference (so that the information exists somewhere other than my hard drive), we needed
  std::vector<const char *> ClangArgv = {"-Xclang", "-emit-llvm-only",
                                         "-fPIC", "-fno-rtti", "-fno-exceptions"};

to get this to work because the clangdev package that ships in conda-forge has exceptions turned off.
Also, it seems to me that this is the place to put "-O3", unless it can also be controlled by a #pragma.

Maybe we do not need fPIC and sure, we can add -O3, I think -O2 is safer.

Now we can use ArrayBuilder in C++ (pure Cling; adopted from @ianna's implementation in RDataFrame).
For example,
>>> import awkward as ak
>>> from awkward._v2._connect.cling import CppStatements
>>>
>>> a = ak._v2.Array([[0.0, 1.1, 2.2], [], [3.3, 4.4]])
>>> b = ak._v2.ArrayBuilder()
>>>
>>> f = CppStatements("""
... for (int i = 0; i < array.size(); i++) {
...   builder.begin_list();
...   for (int j = 0; j < array[i].size(); j++) {
...     builder.real(array[i][j]);
...     builder.real(array[i][j]);
...   }
...   builder.end_list();
... }
... """, builder=b, array=a)
>>>
>>> f(array=a, builder=b)
>>>
>>> b.snapshot().show()   # all of the numbers have been inserted twice
[0, 0, 1.1, 1.1, 2.2, 2.2],
[],
[3.3, 3.3, 4.4, 4.4]]

With iterators and ArrayBuilder append methods, now the following works:
>>> import awkward as ak
>>> from awkward._v2._connect.cling import CppStatements
>>>
>>> a = ak._v2.Array([[0.0, 1.1, 2.2], [], [3.3, 4.4]])
>>> b = ak._v2.ArrayBuilder()
>>>
>>> f = CppStatements("""
... for (auto inner : array) {
...   builder.begin_list();
...   for (auto x : inner) {
...     builder.append(x);
...     builder.append(x);
...   }
...   builder.end_list();
... }
... """, builder=b, array=a)
>>>
>>> f(array=a, builder=b)
>>>
>>> b.snapshot().show()   # all of the numbers have been inserted twice
[0, 0, 1.1, 1.1, 2.2, 2.2],
[],
[3.3, 3.3, 4.4, 4.4]]

@ianna, we should get together to talk about converging our PRs. I've taken your ArrayBuilder and reimplemented it with all the methods and pure C style function invocation (i.e. without bringing in std::invoke). The iterators I've implemented are rather different from yours—they seem to be a lot more minimal. I don't know if there will be any unexpected effects from calling ROOT's ProcessLine, but if we're going to point at free-standing header files like src/awkward/_v2/_connect/_cling/is_iterable.h, we'll have to use Python resources to find them, or else it likely won't work in the package we distribute (they'll be in all sorts of different locations, and I don't know if ROOT will know what to take as the right local path). I'm not sure why we need an is_iterable compile-time function.
From https://stackoverflow.com/a/36606852/1623645, I saw that the minimum we need for a class to be usable in a for-each loop is begin and end methods that return a type with operator*, operator++, and operator!=, so that's what the Iterator<ARRAY, VALUE> is. (Yeah, I used templates to define it when I could have generated it with Python, but in this case, the template is easier because it's just setting the array type, nothing more. Also, if somebody looks at the generated C++, they won't have to scroll past a bunch of Python-generated iterator classes; the C++ templates are written once, though the effective code-generation is the same.)
These iterators make a stack-bound copy of the ArrayView (32 bytes, better than trying to ensure that an iterator doesn't outlive the array it was made from) and an at integer (8 more bytes, total of 40), which is the only thing that gets incremented in the loop. We can't, in general, represent the iterator with a single pointer and increment it because not all Awkward types have a single buffer of data to point to.
...
Although I did just think of a way to make it a bit more efficient. One moment...

This last commit puts the data that used to be in at_ into start_ and constructs ArrayViews on the fly to produce values. So now Iterators are 32 bytes, rather than 40. The operator!= should be faster, too, because most of the comparisons will be in the start_.
I'm not sure how much of the operator[](size_t at) with at=0 the compiler will optimize, but it wouldn't be too ridiculous to add a code-path for operator_at_zero() (no arguments). This is auto-generated code, after all.
But more importantly, this last commit establishes an API for the Iterator that we can later optimize. It has exactly the same member content as the ArrayViews themselves (but it has to be separate because ArrayViews are immutable and Iterators walk).

Actually, we could also drop the which from all the ArrayViews, RecordViews, and Iterators by moving the ptrs when we descend into nested structure. ArrayViews and Iterators could be 24 bytes and RecordViews could be 16 bytes.
I won't look at that, though, until I have a large-scale performance test set up.

Thinking about optimization derailed me. The main thing is, we have to figure out a merge strategy between this and #1295. If this one merges with main first, #1295 will need to adjust before it can merge.

Thinking about optimization derailed me. The main thing is, we have to figure out a merge strategy between this and #1295. If this one merges with main first, #1295 will need to adjust before it can merge.

@jpivarski - It looks great! I really like it.  I’ll either cherry pick from this PR or rebase mine.

Okay, as long as it's not going to be difficult to merge later. Having this in main would let me try those variations.

I like difficult :-)

Actually, we could also drop the which from all the ArrayViews, RecordViews, and Iterators by moving the ptrs when we descend into nested structure. ArrayViews and Iterators could be 24 bytes and RecordViews could be 16 bytes.

For posterity: I looked into that, and hadn't remembered that the values in the "Lookup" array are absolute, not relative. I'd have to change a lot of things—including Numba, because it uses the same lookup—just to perform the test.
Also, I'm going to guess that it won't make much of a difference if I did. Numba ArrayViews are 40 bytes and C++ ArrayViews are 32 bytes (because Numba needs an extra pointer to allow for the possibility of returning an array slice as a Python object), but they can be iterated over in nearly identical time, probably dominated by CPU cache misses.
Using the zlib9-jagged3.parquet from here (the "triply nested" data in this paper),
array = ak._v2.from_parquet("zlib9-jagged3.parquet")
builder = ak._v2.ArrayBuilder()

@nb.njit
def f_numba(array, builder):
    out = 0.0
    for inner1 in array:
        for inner2 in inner1:
            for inner3 in inner2:
                for inner4 in inner3:
                    out += inner4
    builder.real(out)

starttime = time.time()
f_numba(array, builder)
print("Numba time", time.time() - starttime)

print("result", builder.snapshot()[0])

consistently takes 1.72 seconds (standard deviation of 0.01 seconds in 10 trials, without re-reading the array), and
array = ak._v2.from_parquet("zlib9-jagged3.parquet")
builder = ak._v2.ArrayBuilder()

f_cpp = CppStatements("""
double out = 0.0;
for (auto inner1 : array) {
  for (auto inner2 : inner1) {
    for (auto inner3 : inner2) {
      for (auto inner4 : inner3) {
        out += inner4;
      }
    }
  }
}
builder.real(out);
""", array=array, builder=builder)

starttime = time.time()
f_cpp(array=array, builder=builder)
print("C++ time", time.time() - starttime)

print("result", builder.snapshot()[0])

consistently takes 1.78 seconds (standard deviation of 0.01 seconds in 10 trials). That's with -O3, which is the equivalent of what Numba uses.
For scale, a tenth of the dataset, converted into Python builtin types (lists and floats), takes 2.93 seconds (standard deviation of 0.03 seconds in 10 trials), so both the C++ and Numba are about 17× faster than pure Python here. (Python is particularly fast with its builtin types.)
By contrast, iterating through a (already filled!) std::vector<std::vector<std::vector<std::vector<float>>>> of the same data takes 12.69 seconds (standard deviation of 0.08 seconds in 10 trials), which is 2.3× faster than Python but 7.2× slower than the ArrayView iterators. This is an -O3-compiled, standalone program. Although the cause of the difference between ArrayView iterators and std::vector hasn't been narrowed down conclusively, there are at least two contenders:

std::vector has a per-vector header that would be 24 bytes if this SO answer is correct, and more data to ship from RAM into CPU cache costs time.
The buffer associated with a std::vector is heap-allocated, so the data aren't even sequential in memory. That sounds like the biggest issue.

Both of the above apply to Python lists as well as std::vector, and a Python list (in Python 3) has 56 bytes of overhead. That could account for why Python lists and std::vector are in the same ballpark (differ by a factor of 2.3×), with the remaining slowdown in Python going to per-item type-checking. (Conjecture!)
The bottom line is that ArrayView iterators, whether in Numba or C++, are about equally fast and probably optimal. The fact that Numba's iterators are 8 bytes larger doesn't seem to make any perceptible difference, so I won't even go through the effort of making the C++ iterators another 8 bytes smaller.

By contrast, writing data with ArrayBuilders is expensive.
Writing 1/10th of the data, copying all of its nested structure in C++:
f_cpp = CppStatements("""
for (auto inner1 : array) {
  builder.begin_list();
  for (auto inner2 : inner1) {
    builder.begin_list();
    for (auto inner3 : inner2) {
      builder.begin_list();
      for (auto inner4 : inner3) {
        builder.real(inner4);
      }
      builder.end_list();
    }
    builder.end_list();
  }
  builder.end_list();
}
""", array=array, builder=builder)

is consistently 21.8 seconds. (Remember, that's 1/10th of the data, so if I had enough memory for two whole copies of this array, the second would take 218 seconds to write. It took 1.78 seconds to read: 122× faster.)
Without structure is faster:
q_cpp = CppStatements("""
for (auto inner1 : array) {
  for (auto inner2 : inner1) {
    for (auto inner3 : inner2) {
      for (auto inner4 : inner3) {
        builder.real(inner4);
      }
    }
  }
}
""", array=array, builder=builder)

is consistently 6.9 seconds. (Still, 1/10th of the data.)
Somehow, we managed to make this faster in Numba, although it's also going through external library function pointers, unable to optimize through the ArrayBuilder calls.
@nb.njit
def f_numba(array, builder):
    for inner1 in array:
        builder.begin_list()
        for inner2 in inner1:
            builder.begin_list()
            for inner3 in inner2:
                builder.begin_list()
                for inner4 in inner3:
                    builder.real(inner4)
                builder.end_list()
            builder.end_list()
        builder.end_list()

is consistently 4.8 seconds. (Still 1/10th of the data.)
@nb.njit
def q_numba(array, builder):
    for inner1 in array:
        for inner2 in inner1:
            for inner3 in inner2:
                for inner4 in inner3:
                    builder.real(inner4)

is consistently 2.0 seconds. (Still 1/10th of the data.)
The ArrayBuilder is dynamically typed, evaluating whether it should change its type model with every insertion. I don't expect it to be fast. It would be interesting to see what the LayoutBuilder can do here, since the LayoutBuilder is specialized to a given type, but the big difference between Numba and C++ makes me think that there's a bigger bottleneck before we get to the internals of ArrayBuilder vs LayoutBuilder.
In Numba, we put the function pointer from the external .so into a "dynamic addresses" part of Numba's infrastructure: context.add_dynamic_addr in
https://github.com/scikit-hep/awkward-1.0/blob/413f02a01027f8b2ec8b3168c021dd61db8b1c12/src/awkward/_v2/_connect/numba/builder.py#L14-L23
In C++, we JIT-compile a reinterpret_cast<FunctionPointerType>(RAW_INTEGER_IN_SOURCE_CODE) into the source:
https://github.com/scikit-hep/awkward-1.0/blob/413f02a01027f8b2ec8b3168c021dd61db8b1c12/src/awkward/_v2/_connect/cling.py#L211-L396
The single-curly expressions like {ctypes.cast(ak._libawkward.ArrayBuilder_length, ctypes.c_voidp).value} evaluate (in Python) to literal integers, and those literal integers go into the code to be reinterpret-casted as function pointers, then called.
An ideal thing to do, since we have a JIT-compiler on hand, would be to generate specialized LayoutBuilder code (in C++, not Forth) that the compiler can optimize into the surrounding code that calls it. That would bypass whatever's wrong with the C++ function pointer calling, above, and function pointer calling in general (as well as the fact that Forth isn't as fast as C++). But writing such a thing would take a lot of time, before we ever know if it's going to pay off.

I wonder, if dropping an ArrayBuilder shim class makes any difference:
https://github.com/scikit-hep/awkward-1.0/pull/1295/files#diff-a48bdafeba779feab0a7581cdabf4d98ad62952bb52eec7ae76870ddda2f6641R131-R246
I haven't dropped this code from the PR yet. There is a test:
    rdf = ROOT.RDataFrame(10).Define("x", "gRandom->Rndm()")

    builder = ak._v2.highlevel.ArrayBuilder()
    func = ak._v2._connect.rdataframe.from_rdataframe.connect_ArrayBuilder(
        compiler, builder
    )
    compiler(
        f"""
    uint8_t
    my_x_record(double x) {{
        {func["beginrecord"]}();
        {func["field_fast"]}("one");
        {func["real"]}(x);
        return {func["endrecord"]}();
    }}
    """
    )

    rdf.Foreach["std::function<uint8_t(double)>"](ROOT.my_x_record, ["x"])

    array = builder.snapshot()

You mean just go through functions, without having a class instance to organize it? I would have thought that generates the same bytecode.
Actually, perhaps I ought to play with godbolt and find out what bytecode the function pointer generates.
Meanwhile, I've been thinking about adding an interface in which arrays and std::vector in the user code can be exported as a container for ak._v2.from_buffers. Using that would require more expertise from the user, but it's the fastest way to generate an array. (I need to demonstrate on Friday that it's possible to get data into and out of ATLAS at a reasonable rate, even if only by experts.)

What if we extend an ArrayBuilder API with an append_range<T>?  We'd not need to update its builder, just copy the range of the n values (or append a contiguous buffer) to its GrowableBuffer.
The following loop:
@nb.njit
def q_numba(array, builder):
    for inner1 in array:
        for inner2 in inner1:
            for inner3 in inner2:
                for inner4 in inner3:
                    builder.real(inner4)

would become:
@nb.njit
def q_numba(array, builder):
    for inner1 in array:
        for inner2 in inner1:
            for inner3 in inner2:
                builder.append_range_real(inner3.begin(), inner3.end()):

That would reduce the number of external function pointer calls, but it's only applicable to data of primitive type (like append_range_real). Lists in HEP tend to be small, such as leptons with an average number of 1.0 or even 0.1 per event: calling a range-fill for a lot of empty events can actually be worse than not calling it in a loop. But then again, a user could add an if statement to avoid calling it on a loop, so yeah: having that range-fill function and users who know how to use it would be a net positive.
But rather than just work around the slow external function pointer, I'd like to understand why it's so slow in C++ and not in Numba. Numba provides an existence proof that external functions can be called a lot faster than they are with the reinterpret_cast. Something is a bottleneck in the C++. And for all we know at present, it might only be the incremental parser I'm using here, not the one you get in ROOT.
FYI: I've also put zlib9-jagged3.parquet in a publicly accessible place, for repeatability: https://pivarski-princeton.s3.amazonaws.com/chep-2021-jagged-jagged-jagged/zlib9-jagged3.parquet

Cc: @sudo-panda, @wlav. I think both of them would be interested to follow this work here.

But rather than just work around the slow external function pointer, I'd like to understand why it's so slow in C++
and not in Numba.

Just a guess, but a straight-up function pointer is considerably faster than a relocation through the PLT (I saw -fPIC way up in this thread) in a tight loop, especially if you have lots of relocatable functions and two or more of them used in the inner loop are some distance in the table: the function pointer allows prediction, the jump through the PLT does not.
I also saw ProcessLine mentioned, which I think enables the null checker pass. I actually don't know whether it applies to function pointers (Vassil?) but it's trouble in tight loops for two reasons: 1) it introduces a branch, and 2) it kills pattern-based optimizations (more of a problem for arrays than single pointers, though).

The above doesn't use ProcessLine: this is directly through Clang Incremental, not ROOT.
There are two approaches being developed in parallel: @ianna is going through RDataFrame, which JIT-compiles strings through ROOT and Cling, in PR #1295.
This PR is about Clang Incremental, using a ctypes-enabled library (a prototype—not for production). The performance plots above are all Clang Incremental vs Numba, and both of them access ArrayBuilder through external function pointers. The fact that the performance is so different is a clue that it's being done in a different way—there's some source of overhead in the Clang Incremental set-up that isn't present in the Numba.

Hm, I suppose we can dump the llvm IR for both and compare them to see what gets optimized and how? @sudo-panda, do you want to take a look?

Hm, I suppose we can dump the llvm IR for both and compare them to see what gets optimized and how? @sudo-panda, do you want to take a look?

Yeah!

There must be more to it than just the IR. A single function pointer dispatch (of type double(*)(double) from Numba looks like this after disasm:
    vmovsd   6(%rsp), %xmm0
    movabsq  $_ZN8__main__11go_fast_241B42c8tJTIeFCjyCbUFRqqOAK_2f6h0kCng1maAA_3d_3dEd, %rax
    leaq     8(%rsp), %rdi
    movq     $0, 8(%rsp)
    callq    *%rax
    vmovsd   8(%rsp), %xmm0

So Numba is using absolute addressing for the call through the function pointer. I don't have "clang incremental" handy to try, but clang++ actually generates relative addressing starting from -O1 (i.e. no load and using a single instruction, albeit with %rax pushed on the stack for some reason, which is gone by -O2). Iow., Numba is actually not being optimally efficient here ...
What about the loops themselves? As seen in this disasm, Numba enables -mavx by default. I don't see that argument listed in the set above in this thread for Clang?

The loops themselves are fast: https://github.com/scikit-hep/awkward-1.0/pull/1359#issuecomment-1068534763.

Not convinced that that is comparable ... there the operation "+= on native type double" is fully visible (i.e. the compiler can deduce it has no side-effects). It can thus e.g. re-order the loops if that improves the memory layout (needs -O3, though). With the function pointer, it has to assume a worst case of the call having side-effects. I'd figure that with -mavx in both cases the inner loop could be unrolled and if so, I'd be curious to see whether the compiler reloads the function pointer in between invocations (I presume Numba can declare it const, in which case it would not need to).

Well, color me puzzled. I'll need to get a hand on "clang incremental" then to try it out or someone should post the disasm.
Just as some more data points, I compared cppyy.cppdef() which is (from PyPI, not the ancient stuff that ROOT distributes!) my optimized equivalent of Cling's declare(). (Aside, it JITs about 10x faster than Numba, as it does not run the expensive loop optimizations, of all things!)
Fastest by far is passing a function pointer on the stack (i.e. as an argument; this outperforms Numba, too, but obviously no surprise as this removes the load), slowest is when JITted in but through a relocation. But even then, the spread is only ~16%.

@vgvassilev, I hope it's alright: I've posted the code you sent me on https://gist.github.com/jpivarski/aad015ac893a0d9cca6c6f42a90a9505 with instructions for reproducing the above. I think all the warnings that I put there (and the fact that it's a gist, not a regular repo) should prevent anyone from taking it too seriously as production code.

I am fine posting that code.

How to get the parquet file? Closest I seem to get when following the white rabbit is this script, but it still needs inputs that don't seem to be available?

Ah, found it in the thread above. :)

I can't get the Numba example to work yet:
    out = 0.0
    ^

This error may have been caused by the following argument(s):
- argument 0: Cannot determine Numba type of <class 'awkward._v2.highlevel.Array'>
- argument 1: Cannot determine Numba type of <class 'awkward._v2.highlevel.ArrayBuilder'>

might be obvious, but I'm still new to Numba.
However, when checking with top (yay for simple tools :) ), I find that both slow Cpp examples are massive memory hogs: the 6.9s one runs up to 8GB, then jumps to ~17GB at the end. The other increases slowly up to ~14GB, then jumps to ~19GB at the end.
I still see a rough 3x difference just like you do (more like 2.6x, from 57s and 22s, but I don't think I restricted the data set), and it can't be swapping, so smells to me that there are superfluous copies being made? With the slower one, which uses more memory, making more copies?
Just putting that out there in case it rings a bell? Will check in more detail once I get the Numba code to work ...

Sorry about the issue with the Numba examples. Run this:
ak._v2.numba.register()

It's because everything in the ak._v2 submodule is a second implementation of the library, with the intention of replacing it when Awkward 2.0 is released (somewhat like the ROOT::Experimental namespace). The Numba extensions for Awkward v1 are automatically registered, but v2 is opt-in to make sure that there are no conflicts that break it for existing Numba users. (There shouldn't be—the type names are different—but I'm paranoid about breaking things for production users.)
With that, Numba should recognize these _v2 arrays as things that can be in an @nb.njit function.

I would not expect any changes in memory when going through the external function pointer, other than the fact that the ArrayBuilder contains GrowableBuffers that fill up like push_back on std::vector—i.e. replaces its buffer with another 1.5× larger every time the limit is reached. The memory use should jump exponentially larger, logarithmically less often. That effect would be clearest to see in the example without builder.begin_list()/end_list(). If it only has one builder.real(...) call, then it only has one GrowableBuffer, and you wouldn't see different GrowableBuffers expanding at different times.
If there's a large memory use when snapshot() happens, that can be the final transfer from ArrayBuilder to the new Array.

Thanks! With that the Numba example works! It doesn't spike in memory like the other ones. Yes, growing with 1.5x will cause lots of memory copying all around and is the most likely culprit. Yes, the number of jumps slows down logarithmically with size, but the amount of copying needed on each jump grows exponentially (okay exp(n-1)).
A q&d profiling with perf checks the whole program, not just the timed portion, but total time scaling is pretty close to the load/store count scaling.
A q&d profiling with operf, also whole program, unequivocally puts the blame on _ext.cpython-38-x86_64-linux-gnu.so. Doesn't show symbols, I guess no -g used, but it is indeed the home of GrowableBuffer, per nm -D.
Is there any way you can use the semantic equivalent of reserve() somewhere?
Aside, mixing Numba and AArrayclangInterpreter in the same file results in a segfault in llvm::TargetPassConfig::addPass(llvm::Pass*).

Since the ArrayBuilder is used in both the Numba and C++ demos, they should show the same memory growth and copying, though the C++ demo still takes longer than the Numba one. As for the copying, we're slowly moving to a model that requires less copying: since snapshot() is rare, we no longer require that to be a view, and so the accumulated data no longer needs to be contiguous. It could be a linked list of large buffers (or similar), with the discontiguous pieces concatenated in a single copy in snapshot().
https://github.com/scikit-hep/awkward-1.0/pull/1359#issuecomment-1069583963 is the semantic equivalent of "reserve": the arrays that get filled in this example are preallocated with a given size. In general use-cases, we won't know the size, so we can't always use this technique. That's the one that's 2.04 seconds instead of 218 seconds (with 1.78 seconds to iterate without producing output).
To really zero in on the external function pointer, instead of ArrayBuilder's memory usage, how about
import numpy as np
import awkward as ak
from awkward._v2._connect.cling import CppStatements

builder = ak._v2.ArrayBuilder()

f = CppStatements("""
for (int i = 0;  i <<PHONE_NUMBER>;  i++) {
  int64_t result;
  builder.length(&result);
}
""", builder=builder)

f(builder=builder)

versus
g = CppStatements("""
for (int i = 0;  i <<PHONE_NUMBER>;  i++) {
  dummy[0] = dummy[0] * 2 - 1;   // complicated enough that the optimizer can't remove it?
}
""", dummy=dummy)

g(dummy=dummy)

because builder.length has to go through an external pointer, but does not fill anything. f takes 8.1 seconds and g takes 0.06 seconds for me.
The Numba equivalent of f:
import numba as nb
import awkward as ak
ak._v2.numba.register()

@nb.njit
def f(builder):
    for i in range(1000000000):
        len(builder)

f(builder)

takes 3.0 seconds, which is noticeably less than 8.1 seconds. That's the odd thing: that they're both external function calls, but Numba is faster. I would have thought they'd be the same. It's an indication that something is slowing down Clang Incremental's use of external function pointers.

Aside, mixing Numba and AArrayclangInterpreter in the same file results in a segfault in llvm::TargetPassConfig::addPass(llvm::Pass*).

I noticed that, too, and forgot to report it to @vgvassilev! (I didn't narrow it down, though, just noticed that I needed separate conda environments for Numba and clangdev.) Thanks for doing so.

Not sure what dummy is above, but if an array of double, then that 2nd example's loop gets unrolled (at -O3) using AVX (16x) and FMA (2x), giving a 32x speedup. That's not from 8.1s -> 0.06, though, but then, there's also zero memory access (not even the cache) until final write as both constants 2 and 1 are powers of two ...
As for the first and last, on my system, they are basically the same speed, both ~4.7s.

Aside, I just took the examples as-is, but if I add a warmup call, then Clang easily outperfoms by ~20%.

I forgot to copy in
dummy = np.array([0])

so you were right (integers, though). I fixed the comment above.

What's the "warmup"? Repeated application? Because with f defined using Clang:
>>> s = time.time(); f(builder=builder); time.time() - s
8.109642505645752
>>> s = time.time(); f(builder=builder); time.time() - s
8.104896306991577
>>> s = time.time(); f(builder=builder); time.time() - s
8.123145818710327
>>> s = time.time(); f(builder=builder); time.time() - s
8.121228694915771
>>> s = time.time(); f(builder=builder); time.time() - s
8.182328701019287

and with f defined using Numba:
>>> s = time.time(); f(builder=builder); time.time() - s
3.039454460144043
>>> s = time.time(); f(builder=builder); time.time() - s
3.0041792392730713
>>> s = time.time(); f(builder=builder); time.time() - s
3.0177931785583496
>>> s = time.time(); f(builder=builder); time.time() - s
3.0152335166931152
>>> s = time.time(); f(builder=builder); time.time() - s
3.0572359561920166

This is hitting the same external pointer one billion times per call but not doing anything beyond that.
If you're getting different results on repeated application, then we might have different hardware and that would be a clue to the origin of this effect. On the other hand, if this effect is something that even changes direction with hardware (Clang being eventually faster on your computer by 20%), then it's probably not something to deep-dive into, since we wouldn't be able to control for it in a product. Regardless of Clang or Numba, external pointers should be avoided if possible.

Yes, is what I meant. It's not directly clear to me what Clang is still doing on lookup, although the code as written doesn't look free, parsing is already done at that point. Numba, however, doesn't JIT until first use (where types become known).
As for hardware, I'm using a Linux box with an AMD EPYC 7702P and 1TB of RAM.

Regardless of Clang or Numba, external pointers should be avoided if possible.

Yes, absolutely agree with that generally speaking. For cppyy/numba, that is what I want to achieve, too: place the IR from Cling C++ into the IR from Numba Python, effectively inlining it. Not only gets rid of the indirections, but also adds further optimization opportunities.

One more quick note: yes, with an int array for dummy, that whole loop gets optimized out of existence already at -O1.

Aside, mixing Numba and AArrayclangInterpreter in the same file results in a segfault in llvm::TargetPassConfig::addPass(llvm::Pass*).

I noticed that, too, and forgot to report it to @vgvassilev! (I didn't narrow it down, though, just noticed that I needed separate conda environments for Numba and clangdev.) Thanks for doing so.

Can you set me up with a minimal reproducer?

@vgvassilev This seems to be the most minimal example (note that there's another bug with the arguments of the C++ function). The basic point is to do "something" with Numba which creates its setup (doesn't need to run the full JIT by having the nb_noop() call execute), then do have to perform the function call through Clang incremental (which goes back to my earlier question, regarding warmup, how much is deferred to the actual call).
from numba import jit
from all_call_fn import InteractiveCppEnv

@jit(nopython=True)
def nb_noop():
    pass

CppEnv = InteractiveCppEnv()
CppEnv.cpp_compile("void c_noop() {}")
CppEnv.Cpp.c_noop(0)   # <- arg required? (bug)


Roll Engine from 6fa0fb005910 to 7676dbc9eb06 (17 revisions)

https://github.com/flutter/engine/compare/6fa0fb005910...7676dbc9eb06
2021-04-23<EMAIL_ADDRESS>Roll Skia from 600bc360ff01 to 92934d75284d (3 revisions) (flutter/engine#25733)
2021-04-23<EMAIL_ADDRESS>Roll Dart SDK from 4ead30825322 to b207c9873e6c (1 revision) (flutter/engine#25731)
2021-04-23<EMAIL_ADDRESS>Roll CanvasKit to 0.26.0 (flutter/engine#25732)
2021-04-23<EMAIL_ADDRESS>Roll Skia from 7978abafee41 to 600bc360ff01 (5 revisions) (flutter/engine#25730)
2021-04-23<EMAIL_ADDRESS>Fix accessibility of embedded views in Android 9 and below (flutter/engine#25709)
2021-04-23<EMAIL_ADDRESS>Roll Skia from 7b8f14991dae to 7978abafee41 (1 revision) (flutter/engine#25725)
2021-04-23<EMAIL_ADDRESS>Roll Dart SDK from 805527e063af to 4ead30825322 (1 revision) (flutter/engine#25723)
2021-04-23<EMAIL_ADDRESS>Roll Skia from 94d3dd82f350 to 7b8f14991dae (1 revision) (flutter/engine#25722)
2021-04-23<EMAIL_ADDRESS>Roll Dart SDK from 7f16d6f67d91 to 805527e063af (1 revision) (flutter/engine#25721)
2021-04-23<EMAIL_ADDRESS>Roll Skia from 7b60deedbd98 to 94d3dd82f350 (3 revisions) (flutter/engine#25720)
2021-04-23<EMAIL_ADDRESS>Roll Skia from 395274e664cf to 7b60deedbd98 (1 revision) (flutter/engine#25719)
2021-04-23<EMAIL_ADDRESS>Roll Dart SDK from d05e7fc462ba to 7f16d6f67d91 (14 revisions) (flutter/engine#25718)
2021-04-23<EMAIL_ADDRESS>Roll Skia from 82007f568d90 to 395274e664cf (7 revisions) (flutter/engine#25717)
2021-04-23<EMAIL_ADDRESS>Roll Clang Mac from UStSqd7xn... to bf4GDBSQE... (flutter/engine#25715)
2021-04-23<EMAIL_ADDRESS>Roll Clang Linux from GiGTah6EB... to zRvL_ACCG... (flutter/engine#25714)
2021-04-22<EMAIL_ADDRESS>Roll Skia from 071182ed1da5 to 82007f568d90 (4 revisions) (flutter/engine#25712)
2021-04-22<EMAIL_ADDRESS>Refactor Android PlatformView constructors (flutter/engine#25632)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/master/autoroll/README.md

Commit queue failed; closing this roll.


dockerTools.pullImage: use skopeo to pull the image

Motivation for this change
Fixes #29271
Before this patch, a VM was used to spawn docker that pulled the
VM. Now, the tool Skopeo does this job well so we can simplify our
dockerTools since we doesn't need Docker anymore:)
This also fixe the regression described in
https://github.com/NixOS/nixpkgs/issues/29271 : cntlm proxy doesn't
work in 17.09 while it worked in 17.03.
Note Skopeo doesn't produce the same output than docker pull so, we
have to update sha.
Things done
Successfully run:
nix-build ./ -A dockerTools.examples.nix && docker load -i result

And also the ubuntu:latest image.

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] Linux

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Fits CONTRIBUTING.md.

cc @matejc @grahamc

+1 for the idea in general because it is faster and better maintainable. I think there should be a release note item or hint that we broke sha256 of the previous version.

This looks great to me, but I haven't tested it yet.

I'm reverting the skopeo stuff for now.. I generally think it's a good idea but creates too many incompatibilities right now.. sorry and I really hope you don't feel discouraged by that as you have been doing awesome work on the dockerTools! But having a clean release is a priority there.. I really hope we can improve this to make this compatible as it is definitely superior otherwise..

Did you revert on master or on the release branch?

Ah ok so mesos was the reason here: dabb296c76dea4446665e47990653de909ac054f

@globin Don't worry. I don't have any problems with these reverts! I'm waiting for some feedbacks from the Skopeo community before resubmitting improved versions of these patches.
How could I reproduce your issue with Mesos?
Or could you describe a little bit what was your issue?

Just running nix-build -A nixos/tests/mesos.nix should be enough now that mesos itself is fixed. I hope I'll get a bit more involved with this after the release, but don't really have time right now..


Fixed issue #56: broken link

Amended the Code of Conduct link in the contributing.md with the correct path (https://github.com/msandfor/10-Easy-Steps/blob/master/code_of_conduct.md)

@all-contributors please add @Keatnuxsuo for bug doc maintenance mentoring tutorial userTesting


Add class name prefixes to fix potential style leaks

As it currently stands we are not prefixing our classNames e.g. we have .sidebar
which means consumers CSS is likely to leak into the timeline.
I suggest we prefix all our classNames with rt- to avoid this.

We have .react-timelines wrapper which adds some level of protection but agree there is still likely to be some leakage.
Something like .rt or .rtl would help at the expense of readability / noise in the code-base. Would also not guarantee working okay e.g. users can just style p directly and there isn't much we can do about this.


python38Packages.osqp: 0.6.2.post0 -> 0.6.2.post4

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python38Packages.osqp is: "The Operator Splitting QP Solver"
meta.homepage for python38Packages.osqp is: "https://osqp.org/"
meta.changelog for python38Packages.osqp is: ""
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS

Warning: a test defined in passthru.tests did not pass

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 0.6.2.post4 with grep in /nix/store/1cjv2s0dhr93306wabkxxvfw5pr7ym1f-python3.8-osqp-0.6.2.post4

found 0.6.2.post4 in filename of file in /nix/store/1cjv2s0dhr93306wabkxxvfw5pr7ym1f-python3.8-osqp-0.6.2.post4

Rebuild report (if merged into master) (click to expand)

42 total rebuild path(s)

21 package rebuild(s)

21 x86_64-linux rebuild(s)
21 x86_64-darwin rebuild(s)

First fifty rebuilds by attrpath
docbookrx
python38Packages.cvxpy
python38Packages.osqp
python38Packages.qiskit
python38Packages.qiskit-aer
python38Packages.qiskit-aqua
python38Packages.qiskit-finance
python38Packages.qiskit-ibmq-provider
python38Packages.qiskit-ignis
python38Packages.qiskit-optimization
python38Packages.scikit-survival
python39Packages.cvxpy
python39Packages.osqp
python39Packages.qiskit
python39Packages.qiskit-aer
python39Packages.qiskit-aqua
python39Packages.qiskit-finance
python39Packages.qiskit-ibmq-provider
python39Packages.qiskit-ignis
python39Packages.qiskit-optimization
python39Packages.scikit-survival

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/1cjv2s0dhr93306wabkxxvfw5pr7ym1f-python3.8-osqp-0.6.2.post4 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python38Packages.osqp https://github.com/r-ryantm/nixpkgs/archive/718a42bd39d26d1db0093104aae6fd8aed0a6514.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/1cjv2s0dhr93306wabkxxvfw5pr7ym1f-python3.8-osqp-0.6.2.post4
ls -la /nix/store/1cjv2s0dhr93306wabkxxvfw5pr7ym1f-python3.8-osqp-0.6.2.post4/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

18 packages failed to build:

python38Packages.cvxpy
python38Packages.qiskit
python38Packages.qiskit-aer
python38Packages.qiskit-aqua
python38Packages.qiskit-finance
python38Packages.qiskit-ibmq-provider
python38Packages.qiskit-ignis
python38Packages.qiskit-optimization
python38Packages.scikit-survival
python39Packages.cvxpy
python39Packages.qiskit
python39Packages.qiskit-aer
python39Packages.qiskit-aqua
python39Packages.qiskit-finance
python39Packages.qiskit-ibmq-provider
python39Packages.qiskit-ignis
python39Packages.qiskit-optimization
python39Packages.scikit-survival

2 packages built:

python38Packages.osqp
python39Packages.osqp

Maintainer pings
cc @drewrisinger for testing.

Build failures confirmed. Needs debugging to figure out why downstream builds are failing. Maybe some issue with version numbers?

Closing in favor of #154235 or #154646 (both updates to newer version, 0.6.2post5)


latest uglify-js release appears to minify all benchmarks correctly

uglify-js supports ES6+ syntax now up to ES2020 and appears to minify the two libraries that previously failed:
$ node_modules/uglify-js/bin/uglifyjs -V
uglify-js 3.13.1

$ node_modules/uglify-js/bin/uglifyjs node_modules/d3/dist/d3.js -mc | wc -c
  267653

$ node_modules/uglify-js/bin/uglifyjs node_modules/terser/dist/bundle.min.js -mc | wc -c
  407144

Might you be able to update this benchmark to the latest release of uglify-js?
--- a/package.json
+++ b/package.json
@@ -41 +41 @@
-               "uglify-js": "^3.12.5",
+               "uglify-js": "^3.13.1",

Unfortunately I'm not running pnpm so I can't give you a PR.
Thanks in advance.

Thanks for the tip! Updated

@privatenumber This readme comment is also out of date:

UglifyJS takes first place for minification performance. This is quite impressive as it doesn't support or leverage new and concise ES6+ syntax (hence the failed minifications for "terser v5.5.1").

No failed minification and it now handles ES6+ up to ES2020.
It is still written in ES5 though.

Ah good catch! Forgot to update. Fixed in https://github.com/privatenumber/minification-benchmarks/commit/b74b988441da93b339a6a093d6696089295dcfec
Thanks again for the info!
BTW, I see you're a major contributor of UglifyJS—is there a CHANGELOG or release notes for UglifyJS to keep updated with these new features?

Thanks.
Just the commit log and the README. The project is rather low key.


can't replace string in PUT method

First of all , thanks for share this compenent !
So, let's start. In my prctice project, I request a url like this:  $http.put('/api/user/order/{uuid}/');
but I can't get the value of 'uuid' in the url.
My manager use a clever way to resolve it:
$http.put(apiMap('/api/user/order/{uuid}/').replace('{uuid}', this.uuid), {
status: this.add.status,
note: this.add.note
},
it can works! But I think it's not a best way to resolve it.
I can't find anything about the put methods waning in the document, so I put a issued to ask.
Thanks for answering.

It will also work when passing the uuid with params option.
$http.put('/api/user/order/{uuid}/', {...}, {params: {uuid: 123}});


Rule AA0240 recognizes a GUID as Phone No. or E-Mail

1. Describe the bug
The code analysis rule AA0240 recognizes a GUID that is defined as label as a phone no. or e-mail and therefore is triggered. I often declare well-known GUIDs that are used in external interfaces as labels, to have them as similar as I can get to constants. This should not trigger AA0240.
2. To Reproduce
Declare a label like so:
TheLabelOfLifeTok: Label '{8f4ce2e6-4bfa-4c10-a611-972201006156}', Locked = true;

3. Expected behavior
This declaration should not raise any warnings.
4. Actual behavior
CodeCop rule AA0240 is triggered.
5. Versions:

AL Language: v9.0.615906
Visual Studio Code: 1.67.0

Seems already reported and fixed in https://github.com/microsoft/AL/issues/6337 - must have come back?

Have the same issue: warning AA0240: The Label 'NotificationGuidTxt' must not contain email addresses or phone numbers.
codeunit 50000 "Notification Mgt."
{
    var
        NotificationGuidTxt: Label '42cf998c-b9b6-4799-9752-ad62d4080c4a', Locked = true;
[..]

The fix for this issue has been checked in to the master branch. It will be available in the bcinsider.azurecr.io/bcsandbox-master Docker image starting from platform build number 42332 and VS Code Extension Version 10.0.640052.
If you don’t have access to these images you need to become part of the Ready2Go program: aka.ms/readytogo
For more details on code branches and docker images please read:
https://blogs.msdn.microsoft.com/nav/2018/05/03/al-developer-previews-multiple-releases-and-github/
https://freddysblog.com/2020/06/25/working-with-artifacts/


Implement weekly build in the CD/CI pipeline

https://blog.gskinner.com/archives/2022/01/flutter-easily-add-ci-testing-with-github-actions.html reference

@bozzelliandrea apparently the action doesn't work, I've tried to remove the "hardcoded" flutter version but it still doesn't work. Can you look into it ?

@matteasu yep, i'm on it

@bozzelliandrea https://github.com/RIP-Comm/sossoldi/actions/runs/3245749270/jobs/5323709127

I'm closing this as it seems has been resolved.


Unable to overload arrow function in module augmentation

I'm trying to overload an existing arrow function in a module, exported as const. It works perfectly fine with overrides for function, class and interface, but failing for arrow function declared as const.
Also tried with:
    export interface myVar {
        (param: number): number
    }

It works, but then the arrow function become unusable because it treats it as interface and it's not callable.
TypeScript Version:  3.9.2
Search Terms: const, module augmentation, declaration merging, overload, block-scope variable
Expected behavior:
Allow to override block-scoped variables if it's referring to an arrow function. Just like a function.
Actual behavior:
Cannot redeclare block-scoped variable 'myVar'.(2451)

Related Issues:

https://github.com/microsoft/TypeScript/issues/18163 (maybe related)

Code
declare module 'test' {
    // export function works
    export function myFunc(param: string): string;

    // export const doesn't work
    export const myVar: (param: string) => string;
}

declare module 'test' {
    export function myFunc(param: number): number;
    export const myVar: (param: number) => number;
}

Compiler Options
{
  "compilerOptions": {
    "noImplicitAny": true,
    "strictNullChecks": true,
    "strictFunctionTypes": true,
    "strictPropertyInitialization": true,
    "strictBindCallApply": true,
    "noImplicitThis": true,
    "noImplicitReturns": true,
    "useDefineForClassFields": false,
    "alwaysStrict": true,
    "allowUnreachableCode": false,
    "allowUnusedLabels": false,
    "downlevelIteration": false,
    "noEmitHelpers": false,
    "noLib": false,
    "noStrictGenericChecks": false,
    "noUnusedLocals": false,
    "noUnusedParameters": false,
    "esModuleInterop": true,
    "preserveConstEnums": false,
    "removeComments": false,
    "skipLibCheck": false,
    "checkJs": false,
    "allowJs": false,
    "declaration": true,
    "experimentalDecorators": false,
    "emitDecoratorMetadata": false,
    "target": "ES2017",
    "module": "ESNext"
  }
}

Playground Link: Provided

This is the intended behavior; consts are assumed to be in conflict but multiple function declarations adds new overloads.

@RyanCavanaugh But const that point to an arrow function is basically a function...

The goal of that message (I assume) is to match runtime behavior where the function declaration  acts like var vs the const which never allow overriding:
❯ node
Welcome to Node.js v13.6.0.
Type ".help" for more information.
> function a() {}
undefined
> function a() {}
undefined

> const b = () => {}
undefined
> const b = () => {}
Uncaught SyntaxError: Identifier 'b' has already been declared


[Feature] Add one parameter to decide the way to display the result

Motivation
It used showing the result in the form in default.
However, it will cause the problem for automatic test, since CI can't close the image, so the Ci test will pausing at the status of displaying images and the other cases won't be performed.
Modification
add one parameter to decide the way to display the result

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Please sign the CLA, so that we can review your code.

please fix the lint refer to the code style guide

I have fixed the lint error.

I'm sorry to inform you that the previous master(dev) branch has been abandoned, and therefore, this pull request (PR) based on the master(dev) branch will be closed.
We have integrated the previous mmcls and mmselfsup into a new repo named mmpretrain, and you are welcome to use it.


doc: improve debug mode documentation

Debug mode is very useful to analyze various runtime problems.
However, it's not sufficiently documented.
This PR improves documentation to some extent.

Fixed.

jenkins test this please

LGTM


Make a noisy sampling simulator using _channel_

Using cirq.channel you can get the krauss operator / quantum operator ( https://en.wikipedia.org/wiki/Quantum_operation ) for each operation in a circuit. You can then use this to update a density matrix state (I think). Start with a specified state, apply all the operations, sample from the result. Find optimizations to make it faster in special cases, such as using cirq.unitary first.

Question: does it make sense to have an assumption that the same specific noise operator (e.g. amplitude damping) would apply to each operation in the given circuit?

Then this issue isn't really about noise at all then, but rather implementing simulation of channels, right?

So should this issue be based on assumption that every operation already implements SupportsChannel class?
If yes, then adding this dependency seems like a prerequisite to this one.
(E.g. assume that the chosen simulation device would somehow transform the abstract operations from the circuit "blueprint" into channels). Does it make sense?

Perhaps the way to go is to define a SupportsApplyChannel protocol analogous to SupportsApplyUnitary. Any implementation of SupportsApplyUnitary should naturally transfer over to SupportsApplyChannel.

DensityMatrixSimulator exists. Does that satisfy the conditions for closing this issue @Strilanc ?

Closing this because it roughly exists.


feat: Links the generated dialog to its schema

Description
Currently one can navigate from a form dialog schema to its generated dialog.
This PR adds the capability to do the reverse, from generated dialog navigate to its the form dialog schema. (if it's still there)
Task Item
#minor
Screenshots

Coverage increased (+0.004%) to 55.301% when pulling f036db07ba4047c574add7baaf18d09b2fbd9848 on hatpick:sorgh/fd_link into 032f17a3ea4f07cc94a582dbeef430795e5aed57 on microsoft:main.


Missing field image_id in API Call /api/books

Describe the Bug
The documentation mentions a field image_id but the API does not deliver it in API Call /api/books
Steps to Reproduce

Log in to Bookstack
Create a book
Add a title image to the book
Open /api/books
Field image_id is missing for every book

Expected Behaviour
The field image_id is returned for every book
Screenshots or Additional Context
The field is mentioned in the source code, but isn't returned anyway
https://github.com/BookStackApp/BookStack/blob/7c597a05f69c3c4637d213405e55e922844b46a8/app/Http/Controllers/Api/BookApiController.php#L40
Browser Details
No response
Exact BookStack Version
v22.04.2
PHP Version
No response
Hosting Environment
https://demo.bookstackapp.com

Thanks for reporting @bytepoets-ssi. Looks like this property has been hidden for a couple of years, since about v0.30.0.
I'd suggest we simply remove it from the listing call and the API docs since I can't see it providing much value as-is.

As per my last comment, this property has now been removed for documentation.
Change will be included in the next feature release.


Remove the 'build queue' link from the topbar

It's already linked prominently in the footer, it doesn't need to be in both.

Test failure isn't linked to your PR apparently so I merge. Thanks!

Test failure isn't linked to your PR apparently so I merge. Thanks!

The test failure is https://github.com/rust-lang/rust/issues/84790.


unable to setup solaris_exporter

followed all the instructions but when I tryied to launch solaris_exporter by
python2.7 solaris_exporter.py

Traceback (most recent call last):
File "solaris_exporter.py", line 133, in
from prometheus_client.core import REGISTRY, Counter, Gauge, GaugeMetricFamily, CounterMetricFamily, UntypedMetricFamily
File "/usr/lib/python2.7/site-packages/prometheus_client/init.py", line 3, in
from . import (
File "/usr/lib/python2.7/site-packages/prometheus_client/exposition.py", line 113
def make_wsgi_app(registry: CollectorRegistry = REGISTRY, disable_compression: bool = False) -> Callable:
^
SyntaxError: invalid syntax

root@solaris-test:/opt/solaris_exporter# uname -a
SunOS solaris-test 5.11 <IP_ADDRESS>.0 i86pc i386 i86pc
Can some one help me to get this running.
thanks

Hello,
Use the latest README.
pip2.7 install prometheus_client==0.7.1
# Note than current versions of prometheus_client (0.14.1 +) for some reasons not works with Python 2.7,
# that is why I fixed its version to 0.7.1, it is tested.

Hello, Use the latest README.
pip2.7 install prometheus_client==0.7.1 # Note than current versions of prometheus_client (0.14.1 +) for some reasons not works with Python 2.7, # that is why I fixed its version to 0.7.1, it is tested.

Thank you so much for the prompt respond
let me check

Hello, Use the latest README.
pip2.7 install prometheus_client==0.7.1 # Note than current versions of prometheus_client (0.14.1 +) for some reasons not works with Python 2.7, # that is why I fixed its version to 0.7.1, it is tested.

Thank you so much for the prompt respond let me check

Thank you its working now.
++++++++++++++++++++++++++++++++++
but when I tried to set it up as a service I get the following messages and do not see the output on web like I gout it earlier

> root@solaris-test:/opt/solaris_exporter# ./solaris_exporter_smf.sh install
> ./solaris_exporter_smf.sh Script for adding solaris_exporter.py to Solaris SMF as prometheus/solaris_exporter
>      r    Removing "Prometheus::solaris:cmd:::/usr/sbin/fmadm:uid=0" from  /etc/security/exec_attr
>      r    Removing "Prometheus::solaris:cmd:::/usr/sbin/nvmeadm:uid=0" from  /etc/security/exec_attr
>      r    Removing "Prometheus::solaris:cmd:::/usr/sbin/raidctl:uid=0" from  /etc/security/exec_attr
>      r    Removing "Prometheus::solaris:cmd:::/usr/sbin/zlogin:uid=0" from  /etc/security/exec_attr
>      r    Removing "Prometheus:::User could run some monitoring commands" from /etc/security/prof_attr
>      r    Current profiles of monitor user is:
> SMF prometheus/solaris_exporter removed
> Installing solaris_exporter.py as SMF prometheus/solaris_exporter
>     i
> [ 2022 Sep  4 02:24:04 svc.startd could not set context for method:  ]
> chdir: No such file or directory
> [ 2022 Sep  4 02:24:04 Stopping because all processes in service exited. ]
> [ 2022 Sep  4 02:24:04 Executing start method ("/usr/bin/python2.7 /opt/solaris_exporter/solaris_exporter.py"). ]
> [ 2022 Sep  4 02:24:04 Environment variable "LC_ALL=" already set. ]
> [ 2022 Sep  4 02:24:04 svc.startd could not set context for method:  ]
> chdir: No such file or directory
> online          2:24:04 svc:/prometheus/solaris_exporter:default
> root@solaris-test:/opt/solaris_exporter#
> root@solaris-test:/opt/solaris_exporter#
> root@solaris-test:/opt/solaris_exporter#
> root@solaris-test:/opt/solaris_exporter#
> root@solaris-test:/opt/solaris_exporter#
> root@solaris-test:/opt/solaris_exporter#
> root@solaris-test:/opt/solaris_exporter# svcs solaris_exporter
> STATE          STIME    FMRI
> online          2:24:48 svc:/prometheus/solaris_exporter:default

Kind regards,

after creating the /exporter/home/monitor directory and setting its permission to user monitor
the service configured and it is working.
BTW will this same setup work in solaris <IP_ADDRESS>.0  (0.5.11)


Use uniform syntax for bl command line switch

This is just a tiny adjustment to make the syntax in the code block for
the bl switch match the preceding "-binaryLogger" text as well as
the other switches in the document.

@brendanzagaeski : Thanks for your contribution! The author, @mikejo5000, has been notified to review your proposed change.

#sign-off

@brendanzagaeski Thanks! Looks good.


Add support for HttpApplication events in MEF exports

When using MEF there is no entry-point (aka main) called inside the initialized plugin. Furthermore the MEF plugins have no notion about application and session startup and teardown.
MEF endpoints must be able to expose an interface in order to get notifications about HttpApplicationevents (from global.asax.cs).

this interface should also support a Priority and Version as #13 and #9


Remove Dataset Label

How would I be able to delete the dataset label for legend on the chart? I would like to set my own way of label/legend for the chart...
and is it possible to have custom color for each dot on the linechart?
thanks... :)

@mamarilmanson to change the legend labels, use the following in your config:
options: {
  legend: {
    labels: {
      generateLabels: function(chart) {
         /* return objects containing all of the necessary info */
      }
    }
  }
}

The default implementation is https://github.com/chartjs/Chart.js/blob/master/src/core/core.legend.js#L44-L59

To have a custom color for each dot on a line chart, set the pointBackgroundColor attribute of the dataset to an array of colors.

@etimberg ~ re legend labels... I would mean remove it...
like this...

from the left one to the right? :)

oh, you want the entire thing gone?
in your config:
options: {
  legend: {
    display: false
  }
}

This is the one! Thanks! :)


Nested Adapter - Bad performance when having many items (gets laggy)

Hey,
I'm using the nested adapter approach to have horizontally scrolling collectionViews inside a vertically scrolling collectionView (just like in the nested adapter example).
The approach works fine when you only have a few cells inside the horizontal collectionViews. When I tried having about 500+ items, it starts to stutter every time a new row is being displayed. The scrolling stops for a few milliseconds and then goes on. I created a demo project to showcase this (I exaggerated the amount of items to make it clearer):
https://github.com/d3mueller/NestedAdapterTest

I looked into it and it looks like the problem is here (in the section controller that's holding the nested list adapter):
override func cellForItem(at index: Int) -> UICollectionViewCell {
        if let cell = collectionContext!.dequeueReusableCell(of:  NestedCell.self, for: self, at: index) as?  NestedCell {

            // This call takes a long time (sometimes a few dozen or hundred milliseconds)
            listAdapter.collectionView = cell.collectionView

            return cell
        }
        fatalError()
    }

The listAdapter.collectionView property has a setter method that runs every time the cell is being displayed. And something inside there takes a very long time. And that is causing problems while scrolling.
I guess this setter has to access all items from the data source (maybe to get the contentSize for the collectionView?), which takes some time for big arrays.
Having thousands of items is no problem for a single collectionView, it's still incredibly smooth. But when you have these nested adapters, vertical scrolling is quite laggy for many items per row.
Is there any way to improve the performance of this? Can I maybe cache some things to make setting listAdapter.collectionView faster? If yes, how would I do that?
If that's not possible, I would have to reduce the amount of items per row. I thought about dynamically loading new items as you scroll (while also deleting items you have long scrolled past). (Also, I load the data locally using CoreData, I don't have to download them or anything). That kind of works, but it has a big problem:
Using the ideas from #242, I'm trying to maintain the contentOffset while removing items at the beginning of the collectionView. It works, but after setting the contentOffset, every active scrolling animation is stopped, so there's no way to smoothly scroll through the cells. It stops every time new data is being loaded. I implemented this in the demo project. It's commented out
I've been trying to figure this out for the past week or so, I can't think of any solutions to this :D.
I'm thankful for any thoughts/ideas you might have. Thanks!
General information

IGListKit version: master branch
iOS version(s): 12.1
CocoaPods/Carthage version: 1.5.3
Xcode version: 10.1
Devices/Simulators affected: All (but it's hard to see in the simulator)
Reproducible in the demo project? (Yes/No): Yes
Related issues: #242 maybe

Okay, I think I found a possible solution, that still has one small problem.
I took the approach of dynamically loading the data (as described above). I solved the problem with the scrolling by replacing
listAdapter.collectionView?.setReversedContentOffset(reversedContentOffset, animated: false)

with
listAdapter.collectionView?.reversedContentOffset = reversedContentOffset

Then it continues to scroll after the updates. This seems to be working really well, except for one visual glitch:
Before I can update the reversedContentOffset to adjust for the missing items in the beginning, the collectionView renders the new data at the old contentOffset. So it looks like it's jumping to a new position and then back, creating a flicker on the screen.
I tried capturing this behavior using breakpoints (reload is triggered when cell 41 is being displayed):

Is there any way to update the contentOffset a little bit earlier? At the moment, I'm doing it right after performUpdates() finishes:
...

let reversedContentOffset = listAdapter.collectionView!.reversedContentOffset
listAdapter.performUpdates(animated: false) { (finished) in
    listAdapter.collectionView?.reversedContentOffset = reversedContentOffset
    ...
}

...

And something inside there takes a very long time

Did you run Time Profiler to trace exactly what is slow?

Is there any way to update the contentOffset a little bit earlier?

Unfortunately we run into this problem too where we want to "fix" the scroll offset while updates happen. There are some hacks to make this work. See #242

Did you run Time Profiler to trace exactly what is slow?

I think it's this method inside IGListAdapter.m:
 -[IGListAdapter _updateObjects:dataSource:]

This is being called every time a collectionView is set. Inside, there are quite a few iterations over (probably) all objects in the data source. This looks like code that's needed to set up the collectionView.

Unfortunately we run into this problem too where we want to "fix" the scroll offset while updates happen. There are some hacks to make this work. See #242

I think I have the same problem as yood in this answer:(https://github.com/Instagram/IGListKit/issues/242#issuecomment-392850132)
However, his proposed solution doesn't fix it for me. The contentOffset is still being updated too late. Which is strange because the completion(finished) block is being called before actually committing the CATransaction.

Did you have any luck with this issue? It looks like I having same problems with performance on embedded / reused collectionviews.

Unfortunately, no. I just changed my approach so that I don't have too many items inside my collectionViews at any point in time. I dynamically load them as needed (which still has the above mentioned visual glitch though).

Is there anyone who solved this problem?
I am using nested adapter but got lag issue while scrolling vertically.
@rnystrom , can you help me?


Reg files (.reg) cause occasionally Git conflicts on updating Scoop

Bug Report
Package Name: issue does not apply to specific packages, the repository itself is affected
Current Behaviour
Occasionally Git conflicts happen on .reg files and scoop update throws an error due to that.
I can't understand when exactly this happens. It can be that it happens when new .reg files are added to the Git index.
Last time it happened to me with these two .reg files which were added two days ago: https://github.com/ScoopInstaller/Extras/tree/master/scripts/everything
Expected Behaviour
No Git conflicts should happen.
Additional context/output
Executing a git reset --hard didn't help to fix this issue. I had to do this:
git rm .gitattributes
git add -A
git reset --hard

Source: https://stackoverflow.com/a/19588910/7593710
Possible Solution
Specify how to deal with .reg files in .gitattributes.
I'm not 100% sure but this could be a solution:
*.reg text working-tree-encoding=UTF-16LE-BOM eol=CRLF

System details
Windows version: 10
OS architecture: 64bit
PowerShell version:
Major  Minor  Build  Revision
-----  -----  -----  --------
5      1      18362  1801

Additional software: not relevant
Scoop Configuration
{
    "lastupdate":  "2021-12-16T08:50:52.5175566+01:00",
    "SCOOP_REPO":  "https://github.com/lukesampson/scoop",
    "SCOOP_BRANCH":  "master",
    "aria2-enabled":  true,
    "aria2-warning-enabled":  false
}

Hmm, never faced this problem before. @tukanos could this be related to encoding?

@rashil2000  good question.  The question is, if the filename is correctly encoded.  Only the UTF-16 does not suffice.  You need to know if the file contains BOM, if it LE/BE (little/big endian).

I suggest using UTF8 and avoid the BOM nonsense. Every .reg file I have ever created or exported was UTF8 and I've never had an issue.

Default encoding of .reg files is UCS-2 LE BOM (UTF-16) with \r\n endings
UTF-8 and UTF-8 BOM seem to work as well.
More info here: https://gist.github.com/SalviaSage/8eba542dc27eea3379a1f7dad3f729a0

Default encoding of .reg files is UCS-2 LE BOM (UTF-16) with \r\n endings.

I understand it's the default, but I have never heard that it's required. UTF-8 w/o BOM has worked 100% of the time for me.

@rasa True. So you would suggest this as the solution, right?
*.reg text eol=crlf working-tree-encoding=UTF-8

I suggest using UTF8 and avoid the BOM nonsense. Every .reg file I have ever created or exported was UTF8 and I've never had an issue.

I understand Windows 10 has redesigned support for newer formats and line endings, but is this also the case for 7 and 8?

Default encoding of .reg files is UCS-2 LE BOM (UTF-16) with \r\n endings.

I understand it's the default, but I have never heard that it's required. UTF-8 w/o BOM has worked 100% of the time for me.

That gist is mentioned in my PR. Simply export reg file from regedit or do a reg export and you will see that it is always  UTF16LE BOM.
The UTF16LE BOM is recommended and preferred - read the gist why.  You may experience issues with uploading registry files in different encoding.  If something works for you 100% of time it does not mean that it will work for everyone and everywhere 100% of time.  It is always wise to use the recommended solution.

I would also vote for this:
*.reg text working-tree-encoding=UTF-16LE-BOM eol=CRLF

Single issue with this solution would be older Git client versions, since working-tree-encoding was added approx. in March 2018.

Scoop only works on Win7+ 🙃
But can't vouch for point no. 1.

Instead of migrating to UTF16 why not replace the Windows Registry Editor Version 5.00 prefix with REGEDIT4?
All the .reg files in main and extras are currently UTF8, so this means no format converting. I think BOMs and UTF16 will cause more problems that it will solve, imo.

On #7574 @rashil2000 suggested:
PS C:\Users\xxxx> scoop bucket rm extras
PS C:\Users\xxxx> scoop bucket add extras
Checking repo... ok
The extras bucket was added successfully.

This seemed to work OK with no errors but I didn't see the list of updated apps under the 'extras' bucket list.

You will see the list next time you run scoop update.

related discussion: https://github.com/ScoopInstaller/Scoop/pull/4586

https://github.com/ScoopInstaller/Scoop/pull/4654

If you try to edit a UTF16-LE file on GitHub, you will see the message
We’ve detected the file encoding as UTF-16LE. When you commit changes we will transcode it to UTF-8.

I would prefer we stick with formats that the tools we use support.

I encounter this problem every time and it prevents me from updating!
Even git restore is executed, it still cause conflict and actully the files is not modified at all.
Updating 6ec259ca1..de3c07def
error: Your local changes to the following files would be overwritten by merge:
        scripts/everything/install-context.reg
        scripts/everything/uninstall-context.reg
Please commit your changes or stash them before you merge.
Aborting

@scratch-er Please post the output of the following commands:
git config -l --show-origin | findstr /r /c:"core\.[aes][uoa]"

git ls-files --eol "%userprofile%\scoop\buckets\extras\scripts\everything"

git config -l --show-origin | findstr /r /c:"core.[aes][uoa]" outputs nothing.
And here is the core section of ~/scoop/buckets/extras/.git/config
[core]
        repositoryformatversion = 0
        filemode = false
        bare = false
        logallrefupdates = true
        symlinks = false
        ignorecase = true

git ls-files --eol "%userprofile%\scoop\buckets\extras\scripts\everything" outputs nothing.
#7661 suggests the following workaround:
git rm .gitattributes
git add -A
git reset --hard

It works on my machine. I have noted that in .gitattributes, something about the line-ending is configured. Is it line-endings
that caused this problem?

@scratch-er I'm sorry you're having troubles. I had the same experience. In trying to see what's going on, I ran:
$ file -i *
install-context.reg:        text/x-ms-regedit; charset=utf-8
uninstall-context.reg:      text/x-ms-regedit; charset=us-ascii

and did the same on the original files:
$ git cat-file blob 5db260c1b9607d9643bcc98b76a8196a6beccffc:./uninstall-context.reg > uninstall-context.reg.orig
$ git cat-file blob 5db260c1b9607d9643bcc98b76a8196a6beccffc:./install-context.reg > install-context.reg.orig

$ file -i *.orig
install-context.reg.orig:   text/x-ms-regedit; charset=utf-8
uninstall-context.reg.orig: text/x-ms-regedit; charset=us-ascii

The only difference between the original files, and the current files, is they have a final newline:
$ diff install-context.reg*
23c23
< @="$app_path -path \"%1\""
---
> @="$app_path -path \"%1\""
\ No newline at end of file

$ diff uninstall-context.reg*
8c8
< [-HKEY_CURRENT_USER\Software\Classes\Drive\shell\Everything\command]
---
> [-HKEY_CURRENT_USER\Software\Classes\Drive\shell\Everything\command]
\ No newline at end of file

The only thing I found is that everything/install-context.reg has a UTF-8 dash here. But if that's the problem, why does git fail on uninstall-context.reg which is pure ASCII? And why doesn't it also fail on grepwin/install-context.reg which has the same UTF-8 dash?
I'm stumped.

I can't add anything to the diagnosis, but for me deleting the json file that git was complaining about allowed the update command to complete without error.
As a workround for a user this is easier than using git or scoop commands.

This problem is not just limited to .reg files. Many people have reported that they can't update buckets because of conflicting JSON files. I'm not sure what's the cause though. It affects all buckets. Maybe the .gitattributes file?

I'm not sure what's the cause though. It affects all buckets. Maybe the .gitattributes file?

I think the problem is mostly caused by uploading files via Github's web interface. Github will keep CRLF rather than replace them by LF. This also explains why most of the problems are found in .reg files.

Github will keep CRLF rather than replacing them with LF.

Because we're asking it to, no?
https://github.com/ScoopInstaller/Extras/blob/60f3969aaddc7717f15ec69f620acbb3bff359b8/.gitattributes#L1-L2

The workaround for now is to remove and add the bucket again

Another workaround is to execute this in the bucket directory:
git rm .gitattributes
git add -A
git reset --hard

But we need a solution here, not workarounds 😄
This error happened to me again recently but this time in app manifests, not in Windows Registry files 🤔
For example when this manifest was added: https://github.com/ScoopInstaller/Extras/blob/master/bucket/elisa.json

@rashil2000 @r15ch13 Is there any other concerns about using * text eol=crlf in .gitattributes?
(for example, to avoid some errors in certain files?)
If not so, we can easily change it to * text eol=lf and solve the problem.

I was actually thinking of removing the file altogether. Modern Windows deals alright with LFs, but I'm not sure how the change will affect Win 7/8 users.

Not sure if fully removing the .gitattributes file will solve this 🤔
This would happen in this case:

If the text attribute is unspecified, Git uses the core.autocrlf configuration variable to determine if the file should be converted.

Couldn't it still break depending on the personal core.autocrlf configuration?
I think it would be better to keep the file and to concretize how to handle specific files, e.g. like proposed in my initial comment in Possible Solution block.

We can test it. Can you make a PR?

@rashil2000 Here you go #8285

One more interesting thing I found out.
Most files use the ASCII charset. But there are some with ISO-8859-1, UTF-8 , WINDOWS-1252 but also exotic ones like WINDOWS-1258 or ISO-8859-9. Is there a need to unify this? Maybe this is something which causes trouble?
$ find -type f -not -path "./.git/*" -printf "%f: " -exec uchardet "{}" \; | grep -v ASCII

audioswitcher.json: ISO-8859-1
bes.json: UTF-8
diffpdf.json: WINDOWS-1252
flutter.json: WINDOWS-1252
linqpad.json: WINDOWS-1252
love.json: UTF-8
ontopreplica.json: UTF-8
teamspeak3.json: UTF-8
thunderbird.json: WINDOWS-1258
tortoisesvn.json: UTF-8
totalcommander.json: UTF-8
trojita.json: ISO-8859-9
wireshark.json: UTF-8
install-context.reg: UTF-8
flutter-dev-setup.ps1: UTF-8

We can test the current changes for a while, and if they don't work as intended, we now know where to look next :)

@rashil2000 @rasa @tukanos @issaclin32 @scratch-er
So, some time has passed. The error hasn't happened to me personally anymore. How about you?
Would you like to propagate this fix to other repos like ScoopInstaller/Main, ScoopInstaller/Versions and ScoopInstaller/Java?

Would you like to propagate this fix to other repos?

Absolutely.
I have propagated the fix to all official buckets. Thanks for the follow up.

Thanks, @issaclin32. Hope it will work flawlessly 😁

I have also made the change in Tests and BucketTemplate repos


Downloads mysteriously fail

There are at least four different observed failed states:

File is created, but contains no video data, and is usually small (under 50 KiB)
File is created, but contains only black frames at 540p
File is created, but recording is severed partway through and the result is unplayable
No file is created, log is fill with "HandleCtrl, Ping ..."

The last seems to be caused by the server failing to respond and ffmpeg failing to timeout.

File created with no video data samples
broken-no-video-data.tar.gz

HandleCtrl, Ping is a common refrain in at least 1) and 4). It's rare but not unheard of in working downloads though, so I can't break on one.

no-output-file.tar.gz

Mostly fixed in development branch, though there are still occasional oddities so I'm leaving this open.


before_filter skiped by invitation limit alert

When the invitation limit is reached application_controller.rb before_filters are skipped.
Example:
before_filter :load_activities

def load_activities
  @activities = current_user.activities
end

@activities returns nil when the inivtation limit is reached.
Any ideas on this?

Closing, using prepend_before_filter is the way Devise adds filters in their controllers too. You should override Invitation Controller to fix this.


[4.0.0] Is there any way to have the drawer toggle button visable at any of width

Hi,
Is there any way to have the button in the red circle visible all the time as below picture

Sure, try out any Behaviour without "RESPONSIVE" in it.

@natezhengbne could you be more specific about the desired UI behaviour?

Hi @appreciated , sorry for the late reply.
I want the toggle button is visible when the screen width greater than 800px also.
I think I want the drawer behaviour was running as same as any of screen width.

@natezhengbne That is the case for all LeftLayouts.* that don't contain the keyword Responsive.

@natezhengbne That is the case for all LeftLayouts.* that don't contain the keyword Responsive.


Unicode 8 Nature

Branched from #25.
I've still yet to figure out the ordering too :stuck_out_tongue:
Also, iOS has renamed "Nature" to "Animals and Nature". Should we follow suit?

Also, iOS has renamed "Nature" to "Animals and Nature". Should we follow suit?

:+1: I think so.

Just pushed up my best-effort at reordering the nature category, which is now called animalsandnature. What do you think of underscore-casing categories as well => animals_and_nature?

What do you think of underscore-casing categories as well => animals_and_nature?

:+1: for underscore-casing them. wouldn't make sense if sand matches in animalsandnature.

awesome. Just did this. I left the other categories alone for now (though they should also be snakecased soon) to prevent conflicts with @yoshokatana and other's work.

hmm, not sure what's up with Travis here. Everything passes locally..

@wbinnssmith I added a test on master, which checks if the number of keys matches number of emoji.

Ah yeah. Forgot that Travis runs the merge of the base and target branches.
I'll fix this up.
On Mon, Oct 26, 2015 at 5:51 PM Mu-An Chiou<EMAIL_ADDRESS>wrote:

@wbinnssmith https://github.com/wbinnssmith I added a test on master,
which checks if the number of keys matches number of emoji.
—
Reply to this email directly or view it on GitHub
https://github.com/muan/emojilib/pull/26#issuecomment-151327950.

Yup! I added a comment to that test to update the number when more emoji were added, but perhaps it should be in the readme somewhere :p

just fixed it up!

Hi! Is this merge-able now? Just checking. 🔍
I haven't upgraded to El Capitan. 😬

Yep! As long as there aren't any conflicts (which there are bound to be
with the other PRs, happy to resolve if those are merged first)
On Wed, Nov 11, 2015 at 5:43 PM Mu-An Chiou<EMAIL_ADDRESS>wrote:

Hi! Is this merge-able now? Just checking. 🔍
I haven't upgraded to El Capitan. 😬
—
Reply to this email directly or view it on GitHub
https://github.com/muan/emojilib/pull/26#issuecomment-155970277.

Sorry it took so long. Just merged this :sparkles:!


Perform not well in wsgi script.

OS: RedHat 6.5
python: 3.6
python-docx: 0.8.10
Hello,
I can create a docx file correctly in a flask APP.  The size of empty docx file is  36446 bytes.
But when I deploy this flask APP to httpd using mod_wsgi, there is something wrong with the docx. The size of empty docx file 20588 bytes.   It cannot be opened, saying "The file is corrupt and cannot be opened".
Anyone know why? Thanks!
bad_empy_file_from_httpd.docx
good_empty_file_from_flask_or_cmd.docx

So, this problem solved.
I save two docxs as zip file and find that the bad one's document.xml is empty, I realized there is something wrong with the code about generating XML files.
I finally debug in to "python3.6/site-packages/docx/opc/oxml.py:58  serialize_part_xml " find that in line 57
return etree.tostring(part_elm, encoding='UTF-8', standalone=True)
before this line, part_elm is normal, but after etree.tostring, it returns nothing.
so I change search keywords to mod_wsgi etree tostring and find a useful link to this problem.
It tell me there is something wrong with my version of libxml because of mod_php5 in my conf. So, it's OK after I remove mod_php5 in mod_wsgi.

solved.


Statistical and mathmatical evaluation of data

Mathmatical and statistical evaluation of provided data using the PolyMath framework.

EPIC should be to implement general statistical operations
Implementing Poly Math is a necessary first step/ user story

Thanks, you're right. I fixed that.


serria OS broken url

url adding button is broken under 10.12.6

Please test our new version v2.1 and get back to us if you run into any more issues.


Expose ElementReferences

Expose ElementReferences to underlying HTML Elements inside Components like the  element in MudTextField.
Some context why I came to this idea:
I am trying to call "FocusAsync" on an MudTextField but this is only possible by exposing its underlying reference to the element.
Yay or nay?

This is going to be something many users will want to do right? If so, we should probably expose the method FocusAsync on MudTextField.
As for a general discussion of exposing a ref to the input element, I have no strong opinion on that. Let us weigh the options in light of the nature of the library.
Positive: It might increase the usefulness for use cases that weren't foreseen by us
What would be the downside? Somebody might shoot themselves in the foot? I wouldn't mind.
Any more insights?
Are there performance considerations?

This is going to be something many users will want to do right? If so, we should probably expose the method FocusAsync on MudTextField.

@henon Agreed. I think it's better to expose the things you need in the API for the component, such as FocusAsync method in this case, instead of exposing the elementreference internals which I feel is a bit too complicated if we are aiming for simplicity.

Whats the state of this? I would be satisfied if crucial APIs like FocusAsync() are exposed and delegated to the element reference. But exposing the "core" element reference for "advanced" users would be the best.

@Flaflo: Can you please PR the things you need to the API? My main concern with exposing element references is that everyone will just bypass the component's public API and thus our components will not get better/more useful. Also their code might break more easily if we make internal changes. Not exposing the ref forces us (and you to help) improve the components.
Here are some more thoughts:

On the other hand, it might be necessary to have access to html elements for advanced use-cases that involve JS
Also, on some components there are multiple internal html elements, we can't expose them all without spamming our API. So, I am thinking, can we find a generic way to allow advanced users internal access to html elements via queries, much like in bUnit?

@Flaflo: Can you please PR the things you need to the API? My main concern with exposing element references is that everyone will just bypass the component's public API and thus our components will not get better/more useful. Also their code might break more easily if we make internal changes. Not exposing the ref forces us (and you to help) improve the components.
Here are some more thoughts:
* On the other hand, it might be necessary to have access to html elements for advanced use-cases that involve JS

* Also, on some components there are multiple internal html elements, we can't expose them all without spamming our API. So, I am thinking, can we find a generic way to allow advanced users internal access to html elements via queries, much like in bUnit?

Yea sure, but at the moment its just FocusAsync() so is it a good idea to PR it? I mean if you gonna go and proxy all api calls through a mudcomponent will mean that you need to maintain them frequently otherwise some might break with an update.
If you ask me: this is a hard design decision, or more likely a decision about how lazy you are :D

I really want the lib to be useful to all and not limit expert users for the sake of non-experts. You are right that our current man power (or lazyness if you will :D) doesn't allow us to proxy everything. It also means we don't have the time to expose every internal html element.
So how about instead of making a big decision for the library component let's handle each case separately? Expose functions that are most likely to be needed by many and expose refs to internal html elements otherwise. Please go ahead and PR a solution to your use-case.

Well I just figured out that my FocusAsync would need either .NET 5 or a custom JSInterop and I dont really think you not gonna like the latter :D

ad JS: when there is no other way we do use JS.
ad .net5: We have received a lot of positive feedback on supporting it, so we will make a net5 branch and nuget. Do you want to get started with the net5 branch?

Well there is another way obviously so I guess I'll start the .NET 5 branch.
I guess I branch of from master?

Doesn't matter. You can also branch off develop.

We finally are giving in to exposing the input reference: PR #4025


Remove gochist from sig-docs-en-owners

To make Korean localization easier, I've been listed in sig-docs-en-owner so I can approve changes to the i18n directory and the README-ko.md file by #15692. However, there are side effects that I recommend as a reviewer or assignee in other English documents.
/cc @jimangel @zacharysarah

/lgtm
/approve


Cognitive Search Readme Issue

Section link1:

Reason:
Client is not defined
Suggestion:
Add code
client = SearchIndexClient(service_endpoint, AzureKeyCredential(key))
@rohit-ganguly  @lmazuel @hector-norza  @pallavit  @xiangyan99 for notification.

Thank you for the feedback @jerryshia . We will investigate and get back to you asap.

It is defined in above blocks. https://github.com/azure-sdk/azure-sdk-for-python/tree/main/sdk/search/azure-search-documents#querying

Fixed.


Parameter passing prototype

This is the first PR for a series of PRs (i.e., incremental PRs with new features will be merged into this one). Everything will be merged into sdf10 once the whole series is completed.
Parameter passing reference: http://sdformat.org/tutorials?tut=param_passing_proposal&branch=param_passing

Codecov Report

Merging #413 (30b2317) into sdf10 (efd1a7f) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##            sdf10     #413   +/-   ##
=======================================
  Coverage   87.36%   87.37%
=======================================
  Files          60       60
  Lines        9221     9227    +6
=======================================
+ Hits         8056     8062    +6
  Misses       1165     1165

Impacted Files
Coverage Δ

src/parser.cc
78.74% <100.00%> (+0.14%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update efd1a7f...30b2317. Read the comment docs.

30b2317: Parser checks for //include/experimental:params tag and added test

Just minor comments about either removing commented out blocks in the sdf or adding doc to describe what they are for

I removed the unneeded comments in a9238215ab6ecae86f1d5b93aa72e75fdfec977b and also added a test that compares the result with the expected output for nested models


Need examples

"To be consumable in a C++ project, a package must target the native framework. "
How do I do this in a nuspec file?  There isn't any documentation I can find.    Please provide an example nuspec file for a native C++ dll.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 3ae24ac6-e495-0b9f-9fc9-eef609e2492b
Version Independent ID: 86486a38-2180-49aa-68ac-36f8dd0629e1
Content: Creating Native NuGet Packages
Content Source: docs/create-packages/Native-Packages.md
Product: nuget
GitHub Login: @karann-msft
Microsoft Alias: karann

Totally agree. This is almost worthless documentation, without an example and supporting commentary.

bump ;)
I'm trying to package binaries given by a third part, as they don't when to release it as a nupkg.
Reading the docs was a deception :s

Ok, after some digging here is how I found an example.

I created a hello world C++ project in Visual Studio.
I opened up Nuget package manager (right-click on Project > Manage Nuget Package).
I looked for boost which is a popular C++ library. I downloaded the package.
I navigated to my temp project and under the packages folder, I looked inside boost.
Next I used Nuget Package Explorer and opened the nuget package.

Voila! Now you have your example!

I totally get your point, you are probably using the C++ way more than i would have dreamed about :s
This i what i did, with other package than boost
My experience with the C++ ecosystem is near to 0, and the binaries i have to package are already built by a third part vendor.
That vendor ships binaries for multiple platform/arc and also ship managed wrapper
In this scenario Boost won't help much better.
i tried with multiple package like SqLite
The thing is that since nuget 2.7 / 3.5 etc ... there was breaking changes in the folder structure of how nuget is scaffold so i can't just take a random package.
Once i got there the packing is not that straightforward when packing both managed/native binaries

the nuget "downloaded" does not all respect the spec (warning or fail with latest nuget CLI)
folder are supposed to match RID when doing XPlat but the random samples does not uses the same layout
reference the package in .net project reference only a subset of what it should do
native lib were sometime not loaded during debug
native lib need to be copied on publish folder specific to the associated RID when running dotnet publish --runtime RID
have to actually load x86 or x64 based on Build time parameter

That's a small list of what happen and that i can't guess randomly when picking  "boost" or "sqlite" or ....
We finally made it work after lots of efforts
That why i went here during the process and after because  even though we looked at the documentation we could not find detailed explanations about native/wrapper/rid/scaffolding
There are "notion" of it but not how to properly glue it together and make it work.
The point is not that that developper are looking for something to work.
The point was about giving developper a solid foundation of how it actually work written by the nuget team, They are doing an amazing work
i believe that multiple developper can be stuck on this and are trying to take a look at the doc for that

#1013 has an example using .csproj

@Mikejo5000
yep looked at it as an "inspiration" but it's not always quite explicit about the layout / Xplat or the structure to adopt so that dotnet publish --runtime ... copy properly every binaries.
And then the same question shows up when you add managed wrapper.
We managed to have something that we think is working but it does not match what we found in the documentation.
Also the only doc i found about XPlat was in UWP section IIRC
just trying to use already existing issue to bump up an important point.
As .net core is going XPlat, there are chances that we might see native+wrapper package show up more and more

With a good working sample, as simple as possible, I could update the doc.

yep
that could be used as a base example
An other uncovered part would be to explain how to package already existing native dll a vendor gives you.
i managed to have something packing but it also pack the csproj and the obj folder
i still don't understand why
the main issue is not only missing example but the documentation explaining how should we actually build the .proj itself in such scenario
or for example with with even
<NoBuild>true</NoBuild>
<NoRestore>true</NoRestore>

it still generate obj folder for the dotnet pack ?

@zenakuten @MikeDiack @tebeco @thedevilfish - I would suggest taking a look at VCPkg that is targeted specifically at this scenario - https://github.com/Microsoft/vcpkg#vcpkg

i can't use VCpkg when third part vendor refuse to package binaries
they only ship

Win32/x64 native => raw dll file
linux native specific => raw sofile
respectively x86/x64 managed wrapper => raw dll


cmd/compile: incorrect liveness information

This bug is not present in release-branch.go1.8 nor earlier versions of Go. It is the root cause for some failures reported on #19029 (but not the ones using Go 1.7.5).

Note that in bb 0~r1 has magically become live between the start of the function and the call to newobject. That's wrong. It is not initialized until bb 3.
My guess is this was introduced by Keith's change to the handling of results in liveness bitmaps, but that's just a guess.
Again, not present in Go 1.8rc3, nor in Go 1.7.5. Those correctly delay ~r1 being live until after it is initialized.
/cc @mdempsky @randall77

@rsc, cool. Glad that this helps!  So again, this is only reproducible on current Go master (2017-02-14).

This keeps popping up on the issue tracker under various guises. I think it'd be good to prioritize it.

I'm working on it now.  The fix is pretty easy.  The test is hard.

@randall77 What's your planned fix? Only add Addrtaken PPARAMOUT variables to livedefer?

My planned fix is to zero results earlier. Currently we walk down the results list once, issuing zeroings and move-to-heaps as we go.  I'm changing it to walk down the list twice, all the zeroings first and then all the move-to-heaps after that.

I thought for a while about a more complicated fix, which was to be more precise about when result slots were live.  We'd need to mark every instruction that could panic so we could mark all the result slots as live at each of those points.  It is a more precise fix but it is also more complicated and fragile.


Backports from Bitcoin

We recently investigated the Bitcoin issues which are related to privacy protection, vulnerability patches, or security enhancements. We have also checked the dash source code. Results show that these issues and their PRs are not backported yet. Henceforth, we suggest that dash should backport the PRs listed below for the considerations of software security and integrity.

[x] Bitcoin PR#17906, avoid uninitialized reads.
[ ] Bitcoin PR#16572, fix a Char variable used as Bool.
[ ] Bitcoin PR#15323, fix race condition in mempool_persist.py.
[x] Bitcoin PR#15305, fix crash when disconnecting fail.
[x] Bitcoin PR#15039, avoid leaking nLockTime fingerprint.
[ ] Bitcoin PR#14993, fix data race in InterruptRPC().
[x] Bitcoin PR#13808, shuffle coins before grouping, for privacy protection.

Some of these issues and PRs are not severe security-related, but backports can avoid the chaos ecosystem of Bitcoin-forked projects and the potential vulnerabilities in the future.
Reported by de957ad9679f28a38f02f00cc7928bce8fb424882ff060a3c09c32895b1474cc.

Tracking for all backports is located in issue #4000
Also, numerous of the backports that you've mentioned have already been backported into dash core, including 17906, 15305, 15039 and 13808.
If you'd like to do any other of these backports, feel free.

[ ] The Bitcoin PR#16251 should also be considered to backport for improving signrawtransaction error reporting.