ENH: Rewrite minkowski metric in C++ with pybind11

Reference issue
https://github.com/scipy/scipy/issues/13629#issuecomment-790831819
What does this implement/fix?
This implements a framework for pdist and cdist in C++ using pybind11 and demonstrates it with minkowski and related metrics (chebyshev, cityblock and euclidean).  The code supports weights, strided arrays, out arguments, type promotion and also adds long double support.
While there's a lot of new code here, most of it is metric-independant. To add a new metric, you only need to do 3 things:

Create a class in distance_metrics.h that implements the metric with and without weights.
Register pdist and cdist bindings in distance_pybind.cpp, e.g.

    m.def("pdist_euclidean",
          [](py::object x, py::object w, py::object out) {
              return pdist(out, x, w, EuclideanDistance{});
          },
          "x"_a, "w"_a=py::none(), "out"_a=py::none());

Update the MetricInfo object in distance.py

Benchmark results
All metrics changed here have improved in performance. Both in terms of fixed overhead and in terms of calculation time. Weighted metrics in particular show 1000x speedup on large inputs, going from milliseconds to microseconds.
       before           after         ratio
     [9acb9368]       [8f10dc4f]
     <master>         <distance-rework>
-        47.2±2μs       44.6±0.3μs     0.95  spatial.Xdist.time_cdist(10, 'mahalanobis')
-     1.82±0.03ms      1.73±0.03ms     0.95  spatial.XdistWeighted.time_cdist(10, 'sokalsneath')
-      23.9±0.1μs       22.4±0.3μs     0.93  spatial.Xdist.time_cdist(100, 'euclidean')
-         164±7ms        153±0.6ms     0.93  spatial.XdistWeighted.time_cdist(100, 'canberra')
-      24.2±0.2μs       22.4±0.2μs     0.93  spatial.Xdist.time_cdist(100, 'minkowski')
-         658±4μs          606±1μs     0.92  spatial.Xdist.time_pdist(1000, 'cityblock')
-        1.30±0ms      1.18±0.02ms     0.91  spatial.Xdist.time_cdist(1000, 'cityblock')
-      11.6±0.1μs       10.4±0.1μs     0.89  spatial.Xdist.time_pdist(100, 'chebyshev')
-      19.9±0.1μs      17.5±0.02μs     0.88  spatial.Xdist.time_cdist(100, 'chebyshev')
-      10.4±0.1μs      8.73±0.02μs     0.84  spatial.Xdist.time_pdist(100, 'cityblock')
-      17.8±0.1μs      14.6±0.09μs     0.82  spatial.Xdist.time_cdist(100, 'cityblock')
-      29.6±0.3μs      22.6±0.03μs     0.76  spatial.XdistWeighted.time_cdist(20, 'minkowski-P3')
-     20.7±0.09μs       15.3±0.2μs     0.74  spatial.XdistWeighted.time_pdist(100, 'chebyshev')
-      36.2±0.3μs      26.5±0.09μs     0.73  spatial.XdistWeighted.time_cdist(100, 'chebyshev')
-     9.88±0.02μs       6.88±0.1μs     0.70  spatial.Xdist.time_cdist(10, 'minkowski-P3')
-     6.60±0.09μs      4.42±0.06μs     0.67  spatial.Xdist.time_pdist(10, 'minkowski-P3')
-      19.2±0.4μs       12.5±0.1μs     0.65  spatial.XdistWeighted.time_pdist(20, 'minkowski-P3')
-     3.78±0.04μs      2.02±0.01μs     0.54  spatial.Xdist.time_pdist(10, 'euclidean')
-     3.76±0.02μs      1.95±0.01μs     0.52  spatial.Xdist.time_pdist(10, 'cityblock')
-     3.83±0.06μs      1.98±0.02μs     0.52  spatial.Xdist.time_pdist(10, 'chebyshev')
-     4.04±0.01μs      2.07±0.02μs     0.51  spatial.Xdist.time_pdist(10, 'minkowski')
-     14.5±0.05μs      7.33±0.02μs     0.50  spatial.XdistWeighted.time_cdist(10, 'minkowski-P3')
-     4.64±0.01μs      2.20±0.04μs     0.47  spatial.Xdist.time_cdist(10, 'euclidean')
-     5.01±0.01μs      2.23±0.02μs     0.44  spatial.Xdist.time_cdist(10, 'minkowski')
-     4.75±0.01μs         2.09±0μs     0.44  spatial.Xdist.time_cdist(10, 'chebyshev')
-     4.70±0.04μs      2.05±0.02μs     0.44  spatial.Xdist.time_cdist(10, 'cityblock')
-      11.6±0.1μs      4.83±0.04μs     0.42  spatial.XdistWeighted.time_pdist(10, 'minkowski-P3')
-     10.5±0.04μs      3.50±0.03μs     0.33  spatial.XdistWeighted.time_cdist(20, 'chebyshev')
-     9.26±0.03μs      2.98±0.06μs     0.32  spatial.XdistWeighted.time_pdist(20, 'chebyshev')
-     8.82±0.02μs      2.37±0.03μs     0.27  spatial.XdistWeighted.time_pdist(10, 'chebyshev')
-     9.76±0.01μs      2.55±0.02μs     0.26  spatial.XdistWeighted.time_cdist(10, 'chebyshev')
-     11.3±0.04μs      2.46±0.01μs     0.22  spatial.XdistWeighted.time_pdist(10, 'minkowski')
-     14.4±0.03μs      2.67±0.01μs     0.19  spatial.XdistWeighted.time_cdist(10, 'minkowski')
-     18.6±0.05μs      3.08±0.02μs     0.17  spatial.XdistWeighted.time_pdist(20, 'minkowski')
-      28.4±0.1μs      3.67±0.02μs     0.13  spatial.XdistWeighted.time_cdist(20, 'minkowski')
-       257±0.5μs       15.6±0.2μs     0.06  spatial.XdistWeighted.time_pdist(100, 'minkowski')
-         499±3μs      27.6±0.04μs     0.06  spatial.XdistWeighted.time_cdist(100, 'minkowski')
-       320±0.8μs      2.37±0.04μs     0.01  spatial.XdistWeighted.time_pdist(10, 'cityblock')
-        404±10μs      2.42±0.04μs     0.01  spatial.XdistWeighted.time_pdist(10, 'euclidean')
-         698±4μs      2.53±0.02μs     0.00  spatial.XdistWeighted.time_cdist(10, 'cityblock')
-         855±7μs      2.62±0.01μs     0.00  spatial.XdistWeighted.time_cdist(10, 'euclidean')
-     1.29±0.01ms      2.92±0.01μs     0.00  spatial.XdistWeighted.time_pdist(20, 'cityblock')
-     1.58±0.01ms      3.08±0.01μs     0.00  spatial.XdistWeighted.time_pdist(20, 'euclidean')
-     2.76±0.02ms      3.38±0.03μs     0.00  spatial.XdistWeighted.time_cdist(20, 'cityblock')
-     3.40±0.05ms      3.64±0.01μs     0.00  spatial.XdistWeighted.time_cdist(20, 'euclidean')
-     32.8±0.07ms      13.1±0.02μs     0.00  spatial.XdistWeighted.time_pdist(100, 'cityblock')
-      40.5±0.3ms      15.4±0.07μs     0.00  spatial.XdistWeighted.time_pdist(100, 'euclidean')
-      84.2±0.1ms      27.5±0.07μs     0.00  spatial.XdistWeighted.time_cdist(100, 'euclidean')
-      68.2±0.3ms      22.2±0.04μs     0.00  spatial.XdistWeighted.time_cdist(100, 'cityblock')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

The code uses int64_t. As we also support 32-bit systems, we should not use int64_t in the loops, but rather one that adjusts with the length of a pointer. std::intptr_t would be the most obvious. Also, int64_t is not even a sta dard type like std::int64_t but a vendor extension on some compilers.

Good that @sturlamolden is checking the C++ a bit

Any measurable drawbacks on the compiling of generics? Binary size/compile time pretty constant?

What is the advantage of using C++ and pybind11 here, compared to an improved version in Cython or Cython and C? I do not really see why C++ should be more efficient here.
Maintaining C++ is harder, as fewer are able to do it.
Also, is pybind11 used in SciPy before, or is this a new dependency?

All boiler-plate code aside, the performance critical code is the loops in distance_metrics.h. I do not see anything there that could not equally well be written in Cyhton or C with the same effect.

Yes, pybind11 is already a dependency as you can find using Github's search.
You are welcome to your personal opinion about C++, but the project has already had this discussion. pybind11-wrapped C++ is an acceptable choice for contributions.

You are welcome to your personal opinion about C++, but the project has already had this discussion.

I have contributed quite a bit of C++ to SciPy, so it is not about my personal opinion.
There has always been an order of preference for compiled languages, with Cython > C > C++ > Fortran.  Using C++ should thus at least come with some advantage, that cannot be achieved with Cython or C, as it has the disadvantage of reducing the number of maintainers by a factor of 100. So I am just wondering what that migt be? For example in cKDTree the use of C++ simplified memory management due to RAII. It avoided conditional branches in performance critical loops due to templates. It avoided function pointers in performance critical loops due to templates. It avoided error checking and error handling in performance critical loops due to exception handling. STL gave access to efficient containers and fast selection algorithms. As a result it got quite a bit faster than what could reasonably have been achieved with Cython or C, but at the cost of being considerably harder to maintain. Most of it has been done by @peterbell10, @rainwoodman and me. When I look at this PR however, I don't see any of the reasons for preferring C++ that we had in cKDTree. So I am just wondering what it might be? So it is just a question. I am not saying C++ or pybind11 is inappropriate. I was just wondering what the reasons for using C++ instead of C here might be. As opposed to just improving the C we already have, that is.

@tylerjereddy

Needing to define a custom testing object on top of everything else seems a bit much maybe. Is enforcement of contiguity a new thing with this change? Can the tests pass just fine without modification in a first pass?

I think you have it backwards: enforcing contiguous out is the old behaviour, the new code supports non-contiguous out arrays and doesn't raise an exception.

Any measurable drawbacks on the compiling of generics? Binary size/compile time pretty constant?

It currently takes 3 seconds to compile, and compiles to a ~175kB shared object. While bigger than the current _distance_wrap, it's smaller than e.g. _hausdorff which is just a 40 line Cython function.
I've also written cdist and pdist to not be templated on the metric itself, so there won't be an explosion of code as more metrics are added.
@sturlamolden

The code uses int64_t. As we also support 32-bit systems,

As I see it, the potential for indexing calculation bugs outweighs the potential gain from using 32-bit indexing in loops. e.g. indexing past the end in the dimension with the largest stride could over-flow an int and invoke undefined behavior, even though the data itself fits in a 32-bit address space.
Are there any platforms in wide use where 64-bit integer calculation are prohibitively expensive? AFAIK the biggest reason 32-bit executables are used today is lower memory consumption. Not lack of support/performance with 64-bit arithmetic.

What is the advantage of using C++ and pybind11 here, compared to an improved version in Cython or Cython and C? I do not really see why C++ should be more efficient here.

Cython is superior to pybind11 for interacting with Python objects. However, I find it very limiting in other areas. For example, I don't know of a clean way to write higher-order functions like this in cython without accidentally creating Python objects. I also find their "fused types" much harder to use than C++ templates.
That said, I'm definitely biased because I use C++ much more often.

Also, is pybind11 used in SciPy before, or is this a new dependency?

It's used in scipy.fft.

the new code supports non-contiguous out arrays and doesn't raise an exception.

Has anyone asked for this? What's the use case? Can we just keep the default old behavior/tests and have an optional flag/completely separate testing for the new feature? From my end, it is a little easier to review if the old tests basically remain untouched.
FWIW, I'm mostly in agreement with Sturla re: order of language preference, although there were recent adjustments to this policy (in fact, an attempt was made to prefer C++ over C by default, which luckily we were able to hold off, at least for now, but it does worry me).
In any case, I think my only role here is to review the code as best I can regardless of what language it is in & what bindings are used. The performance bumps are nice at least.

Has anyone asked for this? What's the use case? Can we just keep the default old behavior/tests and have an optional flag/completely separate testing for the new feature? From my end, it is a little easier to review if the old tests basically remain untouched.

Okay, I've removed it for now.

There has always been an order of preference for compiled languages, with Cython > C > C++ > Fortran. Using C++ should thus at least come with some advantage, that cannot be achieved with Cython or C, as it has the disadvantage of reducing the number of maintainers by a factor of 100.

This has been updated a bit. http://scipy.github.io/devdocs/dev/contributor/compiled_code.html#other-languages is the reference you're talking about, it now says: "Our basic advice is: use Pythran or Cython for accelerating smaller pieces of code. In cases where Pythran or Cython are no longer enough, prefer C or C++."
(side note, this is from the unreleased devdocs; making Pythran non-optional is not finalized yet, needs revisiting before the next release)

as it has the disadvantage of reducing the number of maintainers by a factor of 100.

This is not true anymore, C++ has really matured; we seem to now have more people interested in writing new C++ than C; and (assuming you avoid the more magical features) can be more maintainable - especially when you need to support many dtypes, the numpy-specific .c.src /** begin repeat C templating is quite bad in comparison.
Regarding rewriting spatial.distance, we have had two major attempts that both stalled years ago. The whole module is in bad shape. If @peterbell10 is going to rewrite the whole thing in modern C++, with improved performance and fixing a host of bugs in the process, that would be a major win.

the project has already had this discussion. pybind11-wrapped C++ is an acceptable choice for contributions.

Agreed. Let's focus the discussion on the PR contents, not on language choice.

All metrics changed here have improved in performance. Both in terms of fixed overhead and in terms of calculation time. Weighted metrics in particular show 1000x speedup on large inputs, going from milliseconds to microseconds.

This looks great! Could you commit the benchmarks?

There has always been an order of preference for compiled languages, with Cython > C > C++ > Fortran.

For C++ vs. python, recently scipy merged pythran support in #8306, cf. #13308; it might be that pythran sits somewhere between cython an C++ on this scale. Not sure how much of this rewrite could be done natively in pythran, but perhaps @serge-sans-paille can weigh in?

Let me repeat: "Let's focus the discussion on the PR contents, not on language choice."

@rgommers Sorry, I had started this comment and then got pulled away; didn't see your comment when I hit send.

No worries, thanks @h-vetinari

This looks great! Could you commit the benchmarks?

I'm using the asv benchmarks that already exist in master. Xdist:
https://github.com/scipy/scipy/blob/102b7fa5a2f2f97715ef5c40902e1c6f73c19941/benchmarks/benchmarks/spatial.py#L327
And XDistWeighted which I added in #13785:
https://github.com/scipy/scipy/blob/102b7fa5a2f2f97715ef5c40902e1c6f73c19941/benchmarks/benchmarks/spatial.py#L362

I'm showing a few performance drops with local benchmarks (I ran them twice with similar results). Not trying to criticize, just checking if this is related to i.e., the recent revisions/fixes from review? I'm on MacOS/Python 3.7 at the moment, using virtualenvs for asv.
asv continuous -e -b "Xdist.*" master distance-rework
      before           after         ratio
     [2a263d55]       [429dbbc2]
     <master>         <distance-rework>
+      2.83±0.1ms      3.85±0.05ms     1.36  spatial.Xdist.time_cdist(1000, 'chebyshev')
+      3.64±0.2ms       4.73±0.3ms     1.30  spatial.Xdist.time_cdist(1000, 'euclidean')
+     1.84±0.09ms      2.24±0.03ms     1.22  spatial.Xdist.time_pdist(1000, 'minkowski')
+      1.89±0.2ms      2.28±0.02ms     1.21  spatial.Xdist.time_pdist(1000, 'euclidean')
+     1.56±0.08ms      1.83±0.03ms     1.17  spatial.Xdist.time_pdist(1000, 'chebyshev')
+      34.1±0.7μs         39.7±1μs     1.17  spatial.Xdist.time_cdist(100, 'chebyshev')
+      7.85±0.5ms       8.84±0.1ms     1.13  spatial.XdistWeighted.time_pdist(20, 'dice')
-      37.6±0.9μs       32.6±0.6μs     0.87  spatial.Xdist.time_cdist(100, 'cityblock')
-      3.76±0.2ms      3.09±0.08ms     0.82  spatial.XdistWeighted.time_pdist(10, 'cosine')
-        58.5±3μs         47.0±2μs     0.80  spatial.XdistWeighted.time_cdist(100, 'chebyshev')
-      21.3±0.5μs       15.4±0.3μs     0.72  spatial.Xdist.time_cdist(10, 'minkowski-P3')
-        14.1±1μs       9.87±0.6μs     0.70  spatial.Xdist.time_pdist(10, 'minkowski-P3')
-        59.1±4μs         41.2±2μs     0.70  spatial.XdistWeighted.time_cdist(20, 'minkowski-P3')
-      40.6±0.6μs       25.7±0.4μs     0.63  spatial.XdistWeighted.time_pdist(100, 'chebyshev')
-        39.9±3μs       24.9±0.4μs     0.63  spatial.XdistWeighted.time_pdist(20, 'minkowski-P3')
-      8.84±0.8μs      5.52±0.08μs     0.62  spatial.Xdist.time_pdist(10, 'chebyshev')
-      8.47±0.6μs       5.25±0.2μs     0.62  spatial.Xdist.time_pdist(10, 'cityblock')
-      10.2±0.3μs       6.17±0.4μs     0.60  spatial.Xdist.time_cdist(10, 'cityblock')
-      9.43±0.4μs       5.51±0.2μs     0.58  spatial.Xdist.time_pdist(10, 'minkowski')
-      8.86±0.5μs      5.09±0.06μs     0.57  spatial.Xdist.time_pdist(10, 'euclidean')
-      10.2±0.2μs       5.78±0.4μs     0.56  spatial.Xdist.time_cdist(10, 'chebyshev')
-      10.4±0.1μs       5.58±0.2μs     0.54  spatial.Xdist.time_cdist(10, 'euclidean')
-      11.3±0.2μs      5.46±0.07μs     0.48  spatial.Xdist.time_cdist(10, 'minkowski')
-        33.6±1μs       15.6±0.1μs     0.46  spatial.XdistWeighted.time_cdist(10, 'minkowski-P3')
-        26.1±1μs       11.2±0.3μs     0.43  spatial.XdistWeighted.time_pdist(10, 'minkowski-P3')
-        23.9±1μs       8.66±0.7μs     0.36  spatial.XdistWeighted.time_cdist(20, 'chebyshev')
-        22.1±1μs       7.15±0.1μs     0.32  spatial.XdistWeighted.time_cdist(10, 'chebyshev')
-        22.9±2μs       6.89±0.4μs     0.30  spatial.XdistWeighted.time_pdist(20, 'chebyshev')
-      20.5±0.6μs       6.06±0.1μs     0.30  spatial.XdistWeighted.time_pdist(10, 'chebyshev')
-      23.6±0.2μs      5.76±0.08μs     0.24  spatial.XdistWeighted.time_pdist(10, 'minkowski')
-        31.6±2μs       7.01±0.4μs     0.22  spatial.XdistWeighted.time_cdist(10, 'minkowski')
-      40.2±0.5μs       6.92±0.3μs     0.17  spatial.XdistWeighted.time_pdist(20, 'minkowski')
-        59.2±4μs       8.24±0.1μs     0.14  spatial.XdistWeighted.time_cdist(20, 'minkowski')
-         483±9μs       25.1±0.4μs     0.05  spatial.XdistWeighted.time_pdist(100, 'minkowski')
-        910±50μs         42.3±2μs     0.05  spatial.XdistWeighted.time_cdist(100, 'minkowski')
-        606±20μs       5.90±0.1μs     0.01  spatial.XdistWeighted.time_pdist(10, 'cityblock')
-         741±5μs       6.36±0.5μs     0.01  spatial.XdistWeighted.time_pdist(10, 'euclidean')
-     1.49±0.06ms       7.39±0.8μs     0.00  spatial.XdistWeighted.time_cdist(10, 'cityblock')
-      1.85±0.1ms       6.80±0.1μs     0.00  spatial.XdistWeighted.time_cdist(10, 'euclidean')
-     2.71±0.04ms       6.94±0.2μs     0.00  spatial.XdistWeighted.time_pdist(20, 'cityblock')
-     3.39±0.02ms       6.83±0.4μs     0.00  spatial.XdistWeighted.time_pdist(20, 'euclidean')
-      5.77±0.3ms      8.10±0.06μs     0.00  spatial.XdistWeighted.time_cdist(20, 'cityblock')
-      7.53±0.2ms       8.33±0.2μs     0.00  spatial.XdistWeighted.time_cdist(20, 'euclidean')
-        69.1±5ms       24.8±0.1μs     0.00  spatial.XdistWeighted.time_pdist(100, 'cityblock')
-         141±5ms         41.4±2μs     0.00  spatial.XdistWeighted.time_cdist(100, 'cityblock')
-        93.3±2ms       25.7±0.3μs     0.00  spatial.XdistWeighted.time_pdist(100, 'euclidean')
-         175±8ms         42.8±1μs     0.00  spatial.XdistWeighted.time_cdist(100, 'euclidean')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE DECREASED.

@tylerjereddy I still see "performance increased" locally, and no massive drops in performance. So, I'm guessing the difference is either:

The system. e.g. different hardware, background programs interfering with benchmark, etc.
The compiler. e.g. on mac I guess you're using clang.

I'm using gcc/g++ 10.2 and try to shutdown background programs when running benchmarks to reduce noise.

I'll try a third time later--those are super recent versions of the gcc toolchain you are using and yes I'm likely using clang locally though I also sometimes use "native" gcc on mac.
I already ran it twice though so the likelihood that I was selectively doing more work at the exact time as the benchmarks were running for this feature branch probably isn't super high.

The clang++ build output on my mac is super noisy for one of the source files (distance_pybind.cpp) added here--perhaps that offers a clue for the slowdown? I'll paste that part of the build output below the fold. I ran the benchmarks a third time with more apps closed and still see the same result.

building 'scipy.spatial._distance_pybind' extension
compiling C++ sources
C compiler: /usr/bin/clang++ -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/opt/expat/include

compile options: '-DNPY_NO_DEPRECATED_API=NPY_1_9_API_VERSION -DHAVE_CBLAS -I/Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/pybind11/include -I/Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/pybind11/include -I/Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/numpy/core/include -I/Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/numpy/core/include -I/Users/treddy/github_projects/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/python-3.7.6-3gaqcongtvfw2l553jlr5lz3pug2z27w/include/python3.7m -c'
extra options: '-std=c++14 -mmacosx-version-min=10.9 -fvisibility=hidden'
clang++: scipy/spatial/src/distance_pybind.cpp
In file included from scipy/spatial/src/distance_pybind.cpp:1:
In file included from /Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/pybind11/include/pybind11/pybind11.h:44:
In file included from /Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/pybind11/include/pybind11/attr.h:13:
/Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/pybind11/include/pybind11/cast.h:2058:26: warning: loop variable 'a' is always a copy because the range of type 'detail::args_proxy' does not return a reference [-Wrange-loop-analysis]
        for (const auto &a : ap)
                         ^
/Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/pybind11/include/pybind11/cast.h:2058:14: note: use non-reference type 'pybind11::handle'
        for (const auto &a : ap)
             ^~~~~~~~~~~~~~~
/Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/pybind11/include/pybind11/cast.h:2090:26: warning: loop variable 'k' is always a copy because the range of type 'pybind11::dict' does not return a reference [-Wrange-loop-analysis]
        for (const auto &k : reinterpret_borrow<dict>(kp)) {
                         ^
/Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/pybind11/include/pybind11/cast.h:2090:14: note: use non-reference type 'std::__1::pair<pybind11::handle, pybind11::handle>'
        for (const auto &k : reinterpret_borrow<dict>(kp)) {
             ^~~~~~~~~~~~~~~
In file included from scipy/spatial/src/distance_pybind.cpp:1:
/Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/pybind11/include/pybind11/pybind11.h:1415:34: warning: loop variable 'kv' is always a copy because the range of type 'pybind11::dict' does not return a reference [-Wrange-loop-analysis]
                for (const auto &kv : entries) {
                                 ^
/Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/pybind11/include/pybind11/pybind11.h:1415:22: note: use non-reference type 'std::__1::pair<pybind11::handle, pybind11::handle>'
                for (const auto &kv : entries) {
                     ^~~~~~~~~~~~~~~~
/Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/pybind11/include/pybind11/pybind11.h:1427:34: warning: loop variable 'kv' is always a copy because the range of type 'pybind11::dict' does not return a reference [-Wrange-loop-analysis]
                for (const auto &kv : entries) {
                                 ^
/Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/pybind11/include/pybind11/pybind11.h:1427:22: note: use non-reference type 'std::__1::pair<pybind11::handle, pybind11::handle>'
                for (const auto &kv : entries) {
                     ^~~~~~~~~~~~~~~~
/Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/pybind11/include/pybind11/pybind11.h:1442:34: warning: loop variable 'kv' is always a copy because the range of type 'pybind11::dict' does not return a reference [-Wrange-loop-analysis]
                for (const auto &kv : entries) {
                                 ^
/Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/pybind11/include/pybind11/pybind11.h:1442:22: note: use non-reference type 'std::__1::pair<pybind11::handle, pybind11::handle>'
                for (const auto &kv : entries) {
                     ^~~~~~~~~~~~~~~~
/Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/pybind11/include/pybind11/pybind11.h:1456:34: warning: loop variable 'kv' is always a copy because the range of type 'pybind11::dict' does not return a reference [-Wrange-loop-analysis]
                for (const auto &kv : entries)
                                 ^
/Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/pybind11/include/pybind11/pybind11.h:1456:22: note: use non-reference type 'std::__1::pair<pybind11::handle, pybind11::handle>'
                for (const auto &kv : entries)
                     ^~~~~~~~~~~~~~~~
/Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/pybind11/include/pybind11/pybind11.h:1544:26: warning: loop variable 'kv' is always a copy because the range of type 'pybind11::dict' does not return a reference [-Wrange-loop-analysis]
        for (const auto &kv : entries)
                         ^
/Users/treddy/python_venvs/python_376_venv/lib/python3.7/site-packages/pybind11/include/pybind11/pybind11.h:1544:14: note: use non-reference type 'std::__1::pair<pybind11::handle, pybind11::handle>'
        for (const auto &kv : entries)
             ^~~~~~~~~~~~~~~~
scipy/spatial/src/distance_pybind.cpp:111:43: warning: unused variable 'num_cols' [-Wunused-variable]
    const intptr_t num_rows = x.shape[0], num_cols = x.shape[1];
                                          ^
scipy/spatial/src/distance_pybind.cpp:411:12: warning: redundant move in return statement [-Wredundant-move]
    return std::move(type);
           ^
scipy/spatial/src/distance_pybind.cpp:411:12: note: remove std::move call here
    return std::move(type);
           ^~~~~~~~~~    ~
scipy/spatial/src/distance_pybind.cpp:378:14: warning: comparison of integers of different signs: 'const long' and 'std::__1::array::size_type' (aka 'unsigned long') [-Wsign-compare]
    if (ndim != out_shape.size() ||
        ~~~~ ^  ~~~~~~~~~~~~~~~~
scipy/spatial/src/distance_pybind.cpp:486:20: note: in instantiation of function template specialization '(anonymous namespace)::prepare_out_argument<std::__1::array<long, 1> >' requested here
        auto out = prepare_out_argument(out_obj, dtype, out_shape);
                   ^
scipy/spatial/src/distance_pybind.cpp:547:22: note: in instantiation of function template specialization '(anonymous namespace)::pdist<ChebyshevDistance>' requested here
              return pdist(out, x, w, ChebyshevDistance{});
                     ^
scipy/spatial/src/distance_pybind.cpp:273:12: warning: local variable 'out' will be copied despite being returned by name [-Wreturn-std-move]
    return out;
           ^~~
scipy/spatial/src/distance_pybind.cpp:488:13: note: in instantiation of function template specialization '(anonymous namespace)::pdist_unweighted<double>' requested here
            pdist_unweighted<scalar_t>(out, x, f);
            ^
scipy/spatial/src/distance_pybind.cpp:547:22: note: in instantiation of function template specialization '(anonymous namespace)::pdist<ChebyshevDistance>' requested here
              return pdist(out, x, w, ChebyshevDistance{});
                     ^
scipy/spatial/src/distance_pybind.cpp:273:12: note: call 'std::move' explicitly to avoid copying
    return out;
           ^~~
           std::move(out)
scipy/spatial/src/distance_pybind.cpp:273:12: warning: local variable 'out' will be copied despite being returned by name [-Wreturn-std-move]
    return out;
           ^~~
scipy/spatial/src/distance_pybind.cpp:488:13: note: in instantiation of function template specialization '(anonymous namespace)::pdist_unweighted<long double>' requested here
            pdist_unweighted<scalar_t>(out, x, f);
            ^
scipy/spatial/src/distance_pybind.cpp:547:22: note: in instantiation of function template specialization '(anonymous namespace)::pdist<ChebyshevDistance>' requested here
              return pdist(out, x, w, ChebyshevDistance{});
                     ^
scipy/spatial/src/distance_pybind.cpp:273:12: note: call 'std::move' explicitly to avoid copying
    return out;
           ^~~
           std::move(out)
scipy/spatial/src/distance_pybind.cpp:297:12: warning: local variable 'out' will be copied despite being returned by name [-Wreturn-std-move]
    return out;
           ^~~
scipy/spatial/src/distance_pybind.cpp:497:9: note: in instantiation of function template specialization '(anonymous namespace)::pdist_weighted<double>' requested here
        pdist_weighted<scalar_t>(out, x, w, f);
        ^
scipy/spatial/src/distance_pybind.cpp:547:22: note: in instantiation of function template specialization '(anonymous namespace)::pdist<ChebyshevDistance>' requested here
              return pdist(out, x, w, ChebyshevDistance{});
                     ^
scipy/spatial/src/distance_pybind.cpp:297:12: note: call 'std::move' explicitly to avoid copying
    return out;
           ^~~
           std::move(out)
scipy/spatial/src/distance_pybind.cpp:297:12: warning: local variable 'out' will be copied despite being returned by name [-Wreturn-std-move]
    return out;
           ^~~
scipy/spatial/src/distance_pybind.cpp:497:9: note: in instantiation of function template specialization '(anonymous namespace)::pdist_weighted<long double>' requested here
        pdist_weighted<scalar_t>(out, x, w, f);
        ^
scipy/spatial/src/distance_pybind.cpp:547:22: note: in instantiation of function template specialization '(anonymous namespace)::pdist<ChebyshevDistance>' requested here
              return pdist(out, x, w, ChebyshevDistance{});
                     ^
scipy/spatial/src/distance_pybind.cpp:297:12: note: call 'std::move' explicitly to avoid copying
    return out;
           ^~~
           std::move(out)
scipy/spatial/src/distance_pybind.cpp:378:14: warning: comparison of integers of different signs: 'const long' and 'std::__1::array::size_type' (aka 'unsigned long') [-Wsign-compare]
    if (ndim != out_shape.size() ||
        ~~~~ ^  ~~~~~~~~~~~~~~~~
scipy/spatial/src/distance_pybind.cpp:523:20: note: in instantiation of function template specialization '(anonymous namespace)::prepare_out_argument<std::__1::array<long, 2> >' requested here
        auto out = prepare_out_argument(out_obj, dtype, out_shape);
                   ^
scipy/spatial/src/distance_pybind.cpp:575:22: note: in instantiation of function template specialization '(anonymous namespace)::cdist<ChebyshevDistance>' requested here
              return cdist(out, x, y, w, ChebyshevDistance{});
                     ^
scipy/spatial/src/distance_pybind.cpp:319:12: warning: local variable 'out' will be copied despite being returned by name [-Wreturn-std-move]
    return out;
           ^~~
scipy/spatial/src/distance_pybind.cpp:525:13: note: in instantiation of function template specialization '(anonymous namespace)::cdist_unweighted<double>' requested here
            cdist_unweighted<scalar_t>(out, x, y, f);
            ^
scipy/spatial/src/distance_pybind.cpp:575:22: note: in instantiation of function template specialization '(anonymous namespace)::cdist<ChebyshevDistance>' requested here
              return cdist(out, x, y, w, ChebyshevDistance{});
                     ^
scipy/spatial/src/distance_pybind.cpp:319:12: note: call 'std::move' explicitly to avoid copying
    return out;
           ^~~
           std::move(out)
scipy/spatial/src/distance_pybind.cpp:319:12: warning: local variable 'out' will be copied despite being returned by name [-Wreturn-std-move]
    return out;
           ^~~
scipy/spatial/src/distance_pybind.cpp:525:13: note: in instantiation of function template specialization '(anonymous namespace)::cdist_unweighted<long double>' requested here
            cdist_unweighted<scalar_t>(out, x, y, f);
            ^
scipy/spatial/src/distance_pybind.cpp:575:22: note: in instantiation of function template specialization '(anonymous namespace)::cdist<ChebyshevDistance>' requested here
              return cdist(out, x, y, w, ChebyshevDistance{});
                     ^
scipy/spatial/src/distance_pybind.cpp:319:12: note: call 'std::move' explicitly to avoid copying
    return out;
           ^~~
           std::move(out)
scipy/spatial/src/distance_pybind.cpp:349:12: warning: local variable 'out' will be copied despite being returned by name [-Wreturn-std-move]
    return out;
           ^~~
scipy/spatial/src/distance_pybind.cpp:535:9: note: in instantiation of function template specialization '(anonymous namespace)::cdist_weighted<double>' requested here
        cdist_weighted<scalar_t>(out, x, y, w, f);
        ^
scipy/spatial/src/distance_pybind.cpp:575:22: note: in instantiation of function template specialization '(anonymous namespace)::cdist<ChebyshevDistance>' requested here
              return cdist(out, x, y, w, ChebyshevDistance{});
                     ^
scipy/spatial/src/distance_pybind.cpp:349:12: note: call 'std::move' explicitly to avoid copying
    return out;
           ^~~
           std::move(out)
scipy/spatial/src/distance_pybind.cpp:349:12: warning: local variable 'out' will be copied despite being returned by name [-Wreturn-std-move]
    return out;
           ^~~
scipy/spatial/src/distance_pybind.cpp:535:9: note: in instantiation of function template specialization '(anonymous namespace)::cdist_weighted<long double>' requested here
        cdist_weighted<scalar_t>(out, x, y, w, f);
        ^
scipy/spatial/src/distance_pybind.cpp:575:22: note: in instantiation of function template specialization '(anonymous namespace)::cdist<ChebyshevDistance>' requested here
              return cdist(out, x, y, w, ChebyshevDistance{});
                     ^
scipy/spatial/src/distance_pybind.cpp:349:12: note: call 'std::move' explicitly to avoid copying
    return out;
           ^~~
           std::move(out)
19 warnings generated.
/usr/bin/clang++ -bundle -undefined dynamic_lookup -L/Users/treddy/github_projects/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/gettext-0.20.1-qbrme2sb4pm3mglxuacgwdxfba464ych/lib -L/Users/treddy/github_projects/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/readline-8.0-n4beggn2l64dzrf6elkjcixrklp46ghh/lib -L/Users/treddy/github_projects/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/ncurses-6.1-jdnritbfxcmk4kfhfsw7d3xuaeluqqvw/lib -L/Users/treddy/github_projects/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/openssl-1.1.1d-axb6agqybj5ykz6tsroiirkfk2327l2h/lib -L/Users/treddy/github_projects/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/sqlite-3.30.1-gnvq3bbsxwzlkbi2cgsgtfhypc7afpo2/lib -L/Users/treddy/github_projects/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/gdbm-1.18.1-jxyriuyf6a477imqrwkmwbanpy6ngoex/lib -L/Users/treddy/github_projects/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/zlib-1.2.11-cvrek7vtvob6v2dfnvdowu2rkqtqtmmi/lib -L/Users/treddy/github_projects/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/bzip2-1.0.8-boqgsigbjrp3xfvcfphkqhsexqrw2iwd/lib -L/Users/treddy/github_projects/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/xz-5.2.4-nahnpskrw2jjhtsviwietm4qcs5sa7oe/lib -L/Users/treddy/github_projects/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/expat-2.2.9-ctgvqggynhtr2zdidy2f64gk5vgt6jpg/lib -L/Users/treddy/github_projects/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/libffi-3.2.1-2mukefjdfv5eb2oyrsawyoqdkh2zndsd/lib -L/usr/local/opt/expat/lib -I/usr/local/opt/expat/include build/temp.macosx-10.15.3-x86_64-3.7/scipy/spatial/src/distance_pybind.o -Lbuild/temp.macosx-10.15.3-x86_64-3.7 -o build/lib.macosx-10.15.3-x86_64-3.7/scipy/spatial/_distance_pybind.cpython-37m-darwin.so -mmacosx-version-min=10.9

g++ --version output:
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 12.0.0 (clang-1<IP_ADDRESS>)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Do you think this would be ready for the release? This is a major improvement over the actual behaviour that could benefit lots of users.

I looked into the clang performance issues, and it seems clang isn't as good at optimizing the strided address calculations. So, I've included other performance improvements that I was going to leave for another PR. I now see benchmark improvements on both clang and gcc, even without weights.
Clang results
       before           after         ratio
     [8637e42e]       [423fef47]
     <master>         <distance-rework>
-      25.4±0.1μs       24.2±0.2μs     0.95  spatial.Xdist.time_pdist(100, 'yule')
-      1.69±0.1ms         1.60±0ms     0.95  spatial.Xdist.time_cdist(1000, 'correlation')
-      23.8±0.2μs       22.2±0.2μs     0.93  spatial.Xdist.time_cdist(100, 'euclidean')
-      24.1±0.2μs       22.4±0.1μs     0.93  spatial.Xdist.time_cdist(100, 'minkowski')
-     10.6±0.08μs      9.23±0.02μs     0.87  spatial.Xdist.time_pdist(100, 'chebyshev')
-      9.88±0.2μs      8.47±0.01μs     0.86  spatial.Xdist.time_pdist(100, 'cityblock')
-      15.9±0.2μs      13.5±0.08μs     0.85  spatial.Xdist.time_cdist(100, 'cityblock')
-      18.5±0.5μs      15.5±0.09μs     0.83  spatial.Xdist.time_cdist(100, 'chebyshev')
-         494±1μs          388±4μs     0.78  spatial.Xdist.time_cdist(100, 'minkowski-P3')
-       251±0.8μs          197±1μs     0.78  spatial.Xdist.time_pdist(100, 'minkowski-P3')
-     24.7±0.05ms       19.3±0.6ms     0.78  spatial.Xdist.time_pdist(1000, 'minkowski-P3')
-      49.0±0.2ms      37.9±0.05ms     0.77  spatial.Xdist.time_cdist(1000, 'minkowski-P3')
-      10.1±0.1μs      6.61±0.03μs     0.66  spatial.Xdist.time_cdist(10, 'minkowski-P3')
-        6.61±0μs      4.25±0.02μs     0.64  spatial.Xdist.time_pdist(10, 'minkowski-P3')
-     3.77±0.06μs      2.05±0.03μs     0.54  spatial.Xdist.time_pdist(10, 'euclidean')
-     3.72±0.04μs      2.01±0.03μs     0.54  spatial.Xdist.time_pdist(10, 'cityblock')
-     3.87±0.07μs         2.05±0μs     0.53  spatial.Xdist.time_pdist(10, 'chebyshev')
-     4.01±0.04μs      2.09±0.02μs     0.52  spatial.Xdist.time_pdist(10, 'minkowski')
-     4.84±0.06μs      2.27±0.02μs     0.47  spatial.Xdist.time_cdist(10, 'euclidean')
-     5.07±0.08μs      2.27±0.01μs     0.45  spatial.Xdist.time_cdist(10, 'minkowski')
-     4.79±0.02μs      2.12±0.01μs     0.44  spatial.Xdist.time_cdist(10, 'cityblock')
-     4.91±0.05μs      2.13±0.03μs     0.43  spatial.Xdist.time_cdist(10, 'chebyshev')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

GCC results
       before           after         ratio
     [8637e42e]       [423fef47]
     <master>         <distance-rework>
-      23.8±0.5μs       22.2±0.3μs     0.93  spatial.Xdist.time_cdist(100, 'euclidean')
-      23.8±0.2μs       22.2±0.3μs     0.93  spatial.Xdist.time_cdist(100, 'minkowski')
-      9.63±0.1μs      8.52±0.08μs     0.88  spatial.Xdist.time_pdist(100, 'cityblock')
-     18.2±0.07μs      15.6±0.08μs     0.86  spatial.Xdist.time_cdist(100, 'chebyshev')
-      11.0±0.1μs       9.35±0.2μs     0.85  spatial.Xdist.time_pdist(100, 'chebyshev')
-      16.1±0.2μs       13.5±0.2μs     0.84  spatial.Xdist.time_cdist(100, 'cityblock')
-         252±2μs          205±7μs     0.81  spatial.Xdist.time_pdist(100, 'minkowski-P3')
-         495±5μs        389±0.3μs     0.79  spatial.Xdist.time_cdist(100, 'minkowski-P3')
-      24.8±0.2ms       19.4±0.2ms     0.78  spatial.Xdist.time_pdist(1000, 'minkowski-P3')
-      49.6±0.1ms       38.2±0.3ms     0.77  spatial.Xdist.time_cdist(1000, 'minkowski-P3')
-     9.89±0.09μs      6.74±0.04μs     0.68  spatial.Xdist.time_cdist(10, 'minkowski-P3')
-     6.66±0.01μs      4.26±0.04μs     0.64  spatial.Xdist.time_pdist(10, 'minkowski-P3')
-     3.74±0.05μs      2.05±0.02μs     0.55  spatial.Xdist.time_pdist(10, 'euclidean')
-     3.68±0.07μs      1.98±0.01μs     0.54  spatial.Xdist.time_pdist(10, 'cityblock')
-     3.84±0.01μs      2.02±0.01μs     0.52  spatial.Xdist.time_pdist(10, 'chebyshev')
-     3.95±0.05μs      2.05±0.01μs     0.52  spatial.Xdist.time_pdist(10, 'minkowski')
-     4.78±0.01μs      2.22±0.07μs     0.46  spatial.Xdist.time_cdist(10, 'euclidean')
-     4.70±0.04μs      2.07±0.01μs     0.44  spatial.Xdist.time_cdist(10, 'cityblock')
-     5.07±0.02μs      2.22±0.01μs     0.44  spatial.Xdist.time_cdist(10, 'minkowski')
-     4.81±0.04μs      2.09±0.03μs     0.43  spatial.Xdist.time_cdist(10, 'chebyshev')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

Would it be possible to also address the know issues with distance function definitions in https://github.com/scipy/scipy/pull/3163#issuecomment-34224341 in this effort?
There was another change to a distance function recently (https://github.com/scipy/scipy/commit/3b22d1da98dc1b5f64bc944c21f398d4ba782bce), so repairing the definitions does not seem off limits.

Would it be possible to also address the know issues with distance function definitions in #3163 (comment) in this effort?

Yes, I think that would be good. I believe most of those could be considered bugs rather than there being two options and us picking the less commonly used one.


[SMALLFIX]Use static imports for standard test utilities

Use static imports for standard test utilities in /alluxio/core/common/src/test/java/alluxio/master/block/BlockIdTest.java
Change
import org.junit.Assert;
to
import static org.junit.Assert.assertEquals;
AND
Update all
Assert.{{ method }}*
to
{{ method }}*

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/2808/
Test PASSed.


Release date for tensorboard in pypi

Hi,
Thanks for the amazing work,
Do you have an expected release date for this package in pypi ? It's hard to recommend this package to other people for now since it has to be built from source, which is not very easy.

The Pypi repository is no longer this version of tensorboard : https://github.com/dmlc/tensorboard/issues/50

@zihaolucky I don't need another logger, I already use tensorboar dX (https://github.com/lanpa/tensorboard-pytorch). What I want is an alternative to tensorboard, because it has a tensorflow dependancy (and I don't want to make people install tensorflow just for drawing graphs).
Your package is also a logger, thus needs tensorboard and tensorflow.

Indeed, it's always our motivation and objective. We used to use Travis to build and publish the PyPI package, then it's super easy to use, but this solution takes too long to satisfy the Travis restriction in building time.
The maintenance of current solution still remains a problem as we have to include the TensorFlow/TensorBoard as submodule.


Fixes #39 - Ensure we respect the raise_on_failure and only confirm m…

…essages that should be confirmed.

This needs tests.

I'm not seeing where the raise_on_failure = false leads to publish not waiting for ack? It appears both paths still lead to calling ch.confirm_select and  ch.wait_for_confirms calls

@ryanwi Fixed.  I got excited about the connection pools portion :) .  We now do not use confirm_select if raise_on_failure is false.

@ryanwi How is this looking?  I yanked the if/else block and one lined that as suggested.

great, that looks right. But, the builds appeared to have broken?

👍 from me. @dmvt you want to take a look since this affects all publishing.

👍 from me as well... :shipit:


Function autoFit(ts,maxP,maxD,maxQ) question

Hi there,
Now I am using the ARIMA model in the sparkts library. However, when I was reviewing the code on github, I found some errors about function autoFit() in ARIMA.scala
autoFit(ts, maxP, maxD, maxQ)
   * @param ts time series to which to automatically fit an ARIMA model
   * @param maxP limit for the AR order
   * @param maxD limit for differencing order
   * @param maxQ limit for the MA order
   * @return an ARIMAModel

As you can see above, you can see the role of each parameter.
So, when I did autoFit(ts,0,0,0) I thought that results were p = 0, q = 0 because maxP = 0 and maxQ = 0.
But result was maxP = 2, maxQ = 2.
I don't know why these results came out.

In the picture I attached, the maxD, maxP, maxQ values are specified in the code to limit the value. In this code, although the max value is specified, the result is higher than the max value.I can not be sure if I do not understand or if there is an error in this code.
I'll wait for your reply.
Thank you.

From the sparkts Library, autoFit() automatically fit an ARIMA model. By default maxP =5, maxD=2 and maxQ= 5.


PHP 7.4 compatibility issue

I'm currently migrating to PHP7.4  (GA release will be on November, 28th) on my dev environment.
I get an error :
Function ReflectionType::__toString() is deprecated thrown on a  $graph->organization() call
I think I've found a fixe, but not sure about this enough to make a PR :
$this->getOrCreate($type);

=>
$this->getOrCreate($type ? $type->getName() : null);

(in Spatie\SchemaOrg\Graph)

Thanks for pointing out. Right now PHP7.4 isn't supported but after reading the docs the __toString() method is deprecated since PHP5.5 😮
So will try to find a fix. The problem is that getName() is only available on ReflectionNamedType.

In english version of the doc, it says PHP7.1.0, not 5.5... Probably a bug in DE doc ^^
Anyway, thanks for the fix to come !

Similar issue on symfony repo : https://github.com/symfony/symfony/issues/32397
They used the getName() solution, even if they were also dealing with ReflectionType class.. I don't get it either, but I tried the fix I proposed earlier, and it fixes the issue.
Hope it helps.
Note : I tested this only on a PHP7.4 env.
EDIT :  If I dd(get_class($type)); it seems that get getReturnType returns a ReflectionNamedType, and not a ReflectionType as the doc says. (Probably the same for ReflectionParameter::getType() by the way)

Sorry for the delay! Had to fight a cold the last days.
Will start with a fix and rely on the ReflectionNamedType but will add a instanceof check before to prevent method not exists exceptions.
PR will come. :)

https://github.com/spatie/schema-org/releases/tag/2.10.0


Customization of DeclarationOrder

I really like a rule like DeclarationOrder, except when I need to break it… a lot.  This mostly happens with breakpoints.
.cat {
  width: 1000px;
  @include breakpoint($handheld) {
    width: 200px;
  }
}

I suppose we could say I'm looking to relax it when using a @context mixin.

Thanks for the suggestion. However, according to the documentation, I think this should already be the case:

Rule sets should be ordered as follows: @extend declarations, @include declarations without inner @content, properties, @include declarations with inner @content, then nested rule sets.

...

Placing @include declarations without inner @content before properties serves to group them with @extend declarations and provides the opportunity to overwrite them later in the rule set.

@includes with inner @content often involve @media rules that rely on the cascade or nested rule sets, which justifies their inclusion after regular properties.

And, I think we have specs that verify this behavior as well: https://github.com/causes/scss-lint/blob/4b83c6c248e27f9896efc57621b23855c883510d/spec/scss_lint/linter/declaration_order_spec.rb#L168
Is there a specific warning that you are seeing?

Also, what version of scss-lint are you using?

Ah, so my problem is even more of an edge case.  I want to be able add breakpoints (include w/ content) after nested rules.
.cat {
  // Default/Mobile case
  &:before {
    content: 'W';
  }

  // Specific breakpoints
  @include breakpoint($desktop) {
    &:before {
      content: 'Whiskers';
    }
  }
}

This example could easily be refactored, but I have more complicated cases.  Again, love everything else about DeclarationOrder.

I've decided in my specific cases, I'll just add inline disable rules, not too bad of an experience.


Can I add a proxy to devServer?

Every time I need to debug with the backend, I have to package the code and deploy it to the server where the backend is located. This process is quite cumbersome. Therefore, can I set up a proxy for the development server to save a lot of time?

I found that the configuration for devServer is located in tasks.ts, but I don't have control over it.

Hey @fenglier,
you're absolutely right that there should be a way to alter the devServer config. I'm thinking about supporting this in builder.config.mjs, but it will probably be a week or so until I have a decent block of time to look into it. I'll update you here once I get to it.
Cheers!

Hey @fenglier, you're absolutely right that there should be a way to alter the devServer config. I'm thinking about supporting this in builder.config.mjs, but it will probably be a week or so until I have a decent block of time to look into it. I'll update you here once I get to it. Cheers!

oo, thanks


Create documentation pages for main navigation component

[x] I’ve read and understood the Contributing guidelines and have done my best effort to follow them.
[x] I’ve read and agree to the Code of Conduct.
[x] I’ve searched for any related issues and avoided creating a duplicate issue.

Checklist

[x] Overview
[x] Rationale
[ ] Accessiblity
[ ] Code
[x] Discussion
[ ] Live demo
[x] Thumbnail icon
[x] Tags
[x] Related components
[ ] Dependencies
[x] Contributors (including getting permission for new contributors)

Completion of the documentation are currently waiting for finalised components.


Release bigtable 0.30.0

This pull request was generated using releasetool.

@sduskis Do we want to land #5838 before making a release?

@tseaver, we're going to release a Beta release in the near future, and I'm ok deferring #5838 in order to get this release out.

@tseaver, I think I addressed most of your concerns.  I was not able to get PRs for some line items that were committed directly on master without a PR.

@sduskis I've cleaned up a bit, fixing the nesting, putting the bullet points into imperative mode, and moving all the test-only changes to the bottom section.  PTAL.

@sduskis Looks like you just overwrote my changes. :(

@tseaver, I reverted to your version...  Sorry about that.

Release is made: https://github.com/GoogleCloudPlatform/google-cloud-python/releases/tag/bigtable-0.30.0

Tag build succeeded, pushed distributions to PyPI.


Health check?

I periodically have to restart hc and have had trouble determining why from the logs.
Do you have any ideas of a "heath check" request that could be sent over the network?

If an accessory is not reachable, I always use Discovery app to find the related dns-sd entry of type _hap._tcp for the accessory. This tells me if anything is wrong with hc or the network.
PS: I'm currently working dnssd to fix some small issue. Maybe those are related to your problem.

Should I test the unreleased changes on dnssd master or are you still working on them? going to release?

If you want to test the unreleased changes of dnssd you should checkout the branch refactoring of dnssd.
go get github.com/brutella/dnssd
cd $GOPATH/src/github.com/brutella/dnssd
git checkout refactoring

Then in your go.mod use
replace github.com/brutella/dnssd => ./<path-to-dnssd-folder>


FOR REVIEW: Debt ceiling, debt floor, RWA Agreement

Staging branch link: https://github.com/makerdao/governance-manual/pull/182, https://github.com/makerdao/governance-manual/pull/190

@LongForWisdom incorporated your comments, let me know if it's all good and I'll change the title to DONE :)


Detect Ubuntu release after (and including) 16.04 LTS

Hmmm didn't want to add all the changes!  I'll close this and see if I can generate a PR with just my change!

No this is my bad for not having at least merged the changes from the upstream master to my 2.7.0, which I should have fixed shortly ;)

Are you able to merge from this PR, as I'm having issues generating the PR this end (inexperience with git!)

Hmm, I have now merged the various small changes from the upstream master so looks good to merge this... with some tweaks to fit with the coding style etc :)
I concur that there is a steep learning curve for a lot of the branching and merging features on github, having myself inadvertently merged my dev code into a stable branch when I meant to do the opposite..
But it seems like you have the right idea with creating dedicated branches for minor fixes to allow for clean merges, assuming of course I keep my branches at least in sync with the upstream master ;)

The merge seems to have gone to plan, I will try to test on an ubuntu 16.10 install and if successful will manually merge the changes into my 2.6.0 branch.... and from there hopefully into @jprochazka 's 2.6.0 release :)
Thanks again for the contribution.

You're welcome - they're just the changes I made to get things installing
on my own Ubuntu 16.10 system here (albeit from @jprochazka
https://github.com/jprochazka 's master branch).  Next one is to have a
go at the RPi (when I get a chance!)
Thanks to you and @jprochazka https://github.com/jprochazka for taking
the time to streamline the installation of what (up till now) was a mildly
painful process getting everything talking!

--

A J Corbett
G0WFV

Yeah assuming it's -json on both php versions then it should be possible to at least parameterize the phone version and reuse that when specifying the packages to install :)


Refactor validator to func

@ivankivanov As always, code readability. If I see something which can be written simpler I rewrite it, and by doing this I keep the code up-to-date.


Generating the classic RISC-V bin file

The current build setup only generates AVD images, is there a way to generate the traditional RISC-V bin files? I want to test the OS (in shell mode) on off-the-shelf RISC-V emulators.

no.
you should be able to use something like this on the end of your launch_cvd command line to point to your custom qemu though (obviously replacing the path with the equivalent path to your qemu):
      --vm_manager=qemu_cli \
      --qemu_binary_dir=$ANDROID_BUILD_TOP/device/google/cuttlefish_vmm/qemu/x86_64-linux-gnu/bin


Added multitenancy support

The multi-tenancy support added is the support for the app to connect to multiple databases during startup and manage migration for multiple databases(working but a little finicky if complex situation arises).
The feature of having new tenants set up their new schema(or new db) is left to the user to implement as needed(may be implemented in the future if need arises).
This has laid down the bare code to be able to build a normal multi-tenancy system without having to configure anything(hopefully) in regards to migrations or multi db and the user can just get started on the actual implementation of it.


python310Packages.bandit: 1.7.2 -> 1.7.3

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python310Packages.bandit is: "Security oriented static analyser for python code"
meta.homepage for python310Packages.bandit is: "https://bandit.readthedocs.io/en/latest/"
meta.changelog for python310Packages.bandit is: ""
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
Warning: a test defined in passthru.tests did not pass
Warning: no invocation of /nix/store/qmc4f7va22r04fmjdw3lpf676g953la8-python3.10-bandit-1.7.3/bin/bandit had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/qmc4f7va22r04fmjdw3lpf676g953la8-python3.10-bandit-1.7.3/bin/bandit-baseline had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/qmc4f7va22r04fmjdw3lpf676g953la8-python3.10-bandit-1.7.3/bin/.bandit-wrapped had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/qmc4f7va22r04fmjdw3lpf676g953la8-python3.10-bandit-1.7.3/bin/.bandit-baseline-wrapped had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/qmc4f7va22r04fmjdw3lpf676g953la8-python3.10-bandit-1.7.3/bin/bandit-config-generator had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/qmc4f7va22r04fmjdw3lpf676g953la8-python3.10-bandit-1.7.3/bin/.bandit-config-generator-wrapped had a zero exit code or showed the expected version
0 of 6 passed binary check by having a zero exit code.
0 of 6 passed binary check by having the new version present in output.
found 1.7.3 with grep in /nix/store/qmc4f7va22r04fmjdw3lpf676g953la8-python3.10-bandit-1.7.3
found 1.7.3 in filename of file in /nix/store/qmc4f7va22r04fmjdw3lpf676g953la8-python3.10-bandit-1.7.3

Rebuild report (if merged into master) (click to expand)

6 total rebuild path(s)

3 package rebuild(s)

3 x86_64-linux rebuild(s)
3 x86_64-darwin rebuild(s)

First fifty rebuilds by attrpath
prospector
python310Packages.bandit
python39Packages.bandit

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/qmc4f7va22r04fmjdw3lpf676g953la8-python3.10-bandit-1.7.3 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python310Packages.bandit https://github.com/r-ryantm/nixpkgs/archive/28557d89a07f540899cc6609fd1ae6f566027ada.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/qmc4f7va22r04fmjdw3lpf676g953la8-python3.10-bandit-1.7.3
ls -la /nix/store/qmc4f7va22r04fmjdw3lpf676g953la8-python3.10-bandit-1.7.3/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

Maintainer pings
cc @kamadorueda for testing.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/750

This is labelled with 10.rebuild-linux: 0 because this is a duplicate of #162196 (and the diff is exactly the same) which got merged 20 minutes before this PR is created. So I am closing this instead :-)

@bobby285271 Thank you! didn't knew that could happen


CVE-2024-49580

We have an automated vulnerability check in our build pipelines.
They started failing on Monday due to the above CVE in ktor-client
The issue is fixed in Version 3.0.0
See the following advisories

https://github.com/advisories/GHSA-8qv4-773j-c979
https://www.jetbrains.com/privacy-security/issues-fixed/

Do you plan to upgrade to version 3.0.0?

Hi @toplac, thank you for reaching out! If I'm not mistaken, this vulnerability is related to ktor's HttpPlugin, which we don't use in our SDK.
Yes, we'll upgrade ktor to 3.0.0 eventually.

@z4kn4fein Thanks for clarification!

Hi @toplac,
I'm happy to inform you that our latest release 4.0.0 contains the ktor update.


style: Repair code style issues in org.assertj.core.internal.Dates

Repairing Code Style Issues
UnnecessaryLocalVariable
A local variable is declared and in the next line returned. This can be replaced by an instant return.
Changes:

Inlined return statement return checkLowerBoundaryPeriod && checkUpperBoundaryPeriod

Check List:

Fixes Code Style Issue reported by IntelliJ
Unit tests :  NA
Javadoc with a code example (on API only) : YES / NO / NA
PR meets the contributing guidelines

Following the contributing guidelines will make it easier for us to review and accept your PR.

Integrated thanks @MartinWitt!


Can you move mongodb to a peerDependency?

Hey guys, so I have a bunch of packages dependent on mongodb in my project, and it seems right to have it as peerDependency in libraries like mongodb-memory-server. Is there any specific reason why it's not a peerDependency?

To my understanding peerDependencies are made for plugins, things that are not required to run the project and or are exposed to consuming code, and neither applies to mongodb-memory-server, it requires mongodb to function and does not really expose any of that code to the consumer. (If you do want to directly interact with the database (which most likely you do), it does not matter what version mongodb-memory-server uses for setup)
Additionally as a historic reason, npm 6 and below didnt automatically install peerDependencies.
my references for this understanding:

npm: package.json#peerdependencies
this stackoverflow explanation

Also on modern package managers, if the versions are compatible, they will be de-duplicated anyway. (npm, yarn, pnpm)


Error thrown in client.startDiscovery()

When running the following code (copied from the readme with a few lines commented out):
const Hs100Api = require('hs100-api');

const client = new Hs100Api.Client();
//const plug = client.getPlug({host: '<IP_ADDRESS>'});
//plug.getInfo().then(console.log);
//plug.setPowerState(true);

// Look for plugs, log to console, and turn them on
client.startDiscovery().on('plug-new', (plug) => {
  plug.getInfo().then(console.log);
  //plug.setPowerState(true);
});

The following error is thrown as a result of calling startDiscovery:
/usr/src/app/node_modules/hs100-api/lib/plug.js:61
    this.supportsConsumption = (sysInfo.feature.includes('ENE'));
                                               ^

TypeError: Cannot read property 'includes' of undefined
    at Plug.set sysInfo [as sysInfo] (/usr/src/app/node_modules/hs100-api/lib/plug.js:61:48)
    at Client.<anonymous> (/usr/src/app/node_modules/hs100-api/lib/client.js:75:22)
    at emitTwo (events.js:106:13)
    at Socket.emit (events.js:191:7)
    at UDP.onMessage (dgram.js:550:8)

It looks like the sysInfo being passed into the setSysInfo block doesnt have a feature property, but I haven't had a chance to dig deeper.
This is with hs100-api 0.4.0: "hs100-api": "^0.4.0",

I added debugging and saw this in the console output:
DEBUG - {"emeter":{"err_code":-2001,"err_msg":"Module not support"},"system":{"get_sysinfo":{"sw_ver":"1.4.3 Build 170504 Rel.144921","hw_ver":"1.0","model":"LB100(US)","description":"Smart Wi-Fi LED Bulb with Dimmable Light","alias":"Alfa Romeo","mic_type":"IOT.SMARTBULB","dev_state":"normal","mic_mac":"50C7BF310610","deviceId":"{DEVICEID}","oemId":"{OEMID}","hwId":"{HWID}","is_factory":false,"disco_ver":"1.0","ctrl_protocols":{"name":"Linkie","version":"1.0"},"light_state":{"on_off":0,"dft_on_state":{"mode":"normal","hue":0,"saturation":0,"color_temp":2700,"brightness":100}},"is_dimmable":1,"is_color":0,"is_variable_color_temp":0,"preferred_state":[{"index":0,"hue":0,"saturation":0,"color_temp":2700,"brightness":100},{"index":1,"hue":0,"saturation":0,"color_temp":2700,"brightness":75},{"index":2,"hue":0,"saturation":0,"color_temp":2700,"brightness":25},{"index":3,"hue":0,"saturation":0,"color_temp":2700,"brightness":1}],"rssi":-47,"active_mode":"none","heapsize":302452,"err_code":0}}} from <IP_ADDRESS>

It is probably a firmware issue. Sometimes the data that is being received is incomplete. I don’t know if it’s isolated to UDP.
I had to catch and wait for a retry when I wrote code in C#.

The feature issue should be fixed now. I never had any bulbs to test with, but I have a simulator now. It's allowed me to really improve support!


Update snakeyaml to 2.2

About this PR
📦 Updates org.yaml:snakeyaml from 2.0 to 2.2
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

🔍 Files still referring to the old version number
The following files still refer to the old version number (2.0).
You might want to review and update them manually.
LICENSE.txt
Quickstart.sc
build.sbt
client/src/main/scala/skuber/json/PlayJsonSupportForPekkoHttp.scala
client/src/test/resources/examplePodList.json
client/src/test/resources/examplePodWithNodeAffinity.json
client/src/test/scala/skuber/api/ListQueryParamSpec.scala
client/src/test/scala/skuber/json/PodFormatSpec.scala
docs/README.md
docs/index.html

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.yaml", artifactId = "snakeyaml" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "org.yaml", artifactId = "snakeyaml" }
}]

labels: library-update, old-version-remains, commit-count:1

Superseded by #416.


Documentation Issue

Documentation Issue
Compress-Archive
Issue: When using -Path parameter with shares, the share path is appended (without '') to file names.
For example, if archiving 'SomeInfo.txt' on share '\myshare\somefolder', the resulting filename in the zip file is 'mysharesomefolderSomeInfo.txt'
Workaround: To resolve this, I used the -LiteralPathparameter instead.

@jmorey4158 The cmdlet Compress-Archive is part of the module Microsoft.PowerShell.Archive and has his own github repository : github/Microsoft.PowerShell.Archive
You should move your issue in this repo.


Are you a maintainer, or core-contributor, to the project?

[X] Yes

We already have a Pulumi.com account?

[X] Yes, we have created an account

Do you agree to regular check-ins, no more than once every 90 days, with a Developer Advocate from the Pulumi team?

[X] Yes

What's your Open Source project called?
nf-core
Where is the GitHub repo where your organization's infra code is located?
https://github.com/nf-core/
What's Pulumi.com team ID?
nf-core
Do/would you like any assistance with your Pulumi adoption from the DevRel team?

[X] Yes

Can you specifically point me at the repo that has the Pulumi code. We will link to that specific repo.

Sure thing! @aaronkao https://github.com/nf-core/ops

Hey, just bumping this. Any updates?

@edmundmiller hey this in progress right now. if you need help now, hit me up on Community Slack.

@edmundmiller you should be good to go. Let me know if you need anything else

Awesome, thank you so much!


How do i do RangeBar Charts?

they arent available in the SeriesType

Hi,
Have you checked the samples, https://apexcharts.github.io/Blazor-ApexCharts/rangebar-charts

Nope thats what i needed Thank you :D

Where can I view the sample data? So that I know how to feed the chart.

Hi the samples are part of the rep, https://github.com/apexcharts/Blazor-ApexCharts/tree/master/docs/BlazorApexCharts.Docs


Can't run movie quality definitions script - Debian 11

I am trying to run the script for the movie quality definitions but I am getting the below error message, .NET is installed and I am running Debian 11. When I throw the URL into web browser it can be accessed no problems.
[ERR] HTTP error while communicating with Radarr
Flurl.Http.FlurlHttpException: Call failed. The SSL connection could not be established, see inner exception: GET https://<IP_ADDRESS>/radarr/api/v3/qualitydefinition?apikey=REDACTED
 ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure: RemoteCertificateNameMismatch
   at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
   --- End of inner exception stack trace ---
   at Flurl.Http.FlurlRequest.HandleExceptionAsync(FlurlCall call, Exception ex, CancellationToken token)
   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
   at Flurl.Http.ResponseExtensions.ReceiveJson[T](Task`1 response)
   at TrashLib.Radarr.QualityDefinition.Api.QualityDefinitionService.GetQualityDefinition()
   at TrashLib.Radarr.QualityDefinition.RadarrQualityDefinitionUpdater.ProcessQualityDefinition(IEnumerable`1 guideQuality)
   at TrashLib.Radarr.QualityDefinition.RadarrQualityDefinitionUpdater.Process(Boolean isPreview, RadarrConfiguration config)
   at Trash.Command.RadarrCommand.Process()
Exiting due to previous exception```

Looks like the issue is that you're using a self-signed certificate. That is not a scenario I have tested. I would recommend you use http if you're going to use localhost. https will work but it requires an actual certificate, like one from letsencrypt.

any chance to add insecure option to config (and skip the ssl verification on the api calls)?
ive got the same problem, all my setup is via https only with private ca signed certs...

Looks like the issue is that you're using a self-signed certificate. That is not a scenario I have tested. I would recommend you use http if you're going to use localhost. https will work but it requires an actual certificate, like one from letsencrypt.

I have tried using http in the .yaml but still get same error message

I have tried using http in the .yaml but still get same error message

Most likely, Radarr is redirecting http to https. If you are getting the same error message, it's due to SSL.
I'm working on adding settings support. I'll make sure that I add an option to disable SSL as well.

It worked for me after adding my self signed certificate into trust store of my linux. I'm using ubuntu 20.04 and done following steps:

Copy the .crt file to /usr/local/share/ca-certificates/
Run sudo update-ca-certificates

Check out release v1.7.0 which has a new settings file and option inside of it to disable SSL verification. There's a Settings Reference page in the wiki with more details.

Check out release v1.7.0 which has a new settings file and option inside of it to disable SSL verification. There's a Settings Reference page in the wiki with more details.

Still getting same error message.
[INF] Processing Release Profile: Series
[ERR] HTTP error while communicating with Sonarr
Flurl.Http.FlurlHttpException: Call failed. The SSL connection could not be established, see inner exception: GET https://<IP_ADDRESS>/8989/api/v3/system/status?apikey=REDACTED
 ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure: RemoteCertificateNameMismatch```

The ssl verification setting goes into the settings file not the trash.yml

./trash sonarr && ./trash radarr

I now get this error when trying to run it...
htpc@HTPC:~$ ./trash sonarr && ./trash radarr
[WRN] Security Risk: Certificate validation is being DISABLED because setting `enable_ssl_certificate_validation` is set to `false`
[INF] Processing Release Profile: Series
[ERR] HTTP error while communicating with Sonarr
Flurl.Http.FlurlHttpException: Call failed with status code 401 (Unauthorized): GET https://<IP_ADDRESS>/sonarr/api/v3/system/status?apikey=REDACTED
   at Flurl.Http.FlurlRequest.HandleExceptionAsync(FlurlCall call, Exception ex, CancellationToken token)
   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
   at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
   at Flurl.Http.ResponseExtensions.ReceiveJson(Task`1 response)
   at TrashLib.Sonarr.SonarrCompatibility.<>c__DisplayClass0_0.<<-ctor>b__0>d.MoveNext() in D:\a\trash-updater\trash-updater\src\TrashLib\Sonarr\SonarrCompatibility.cs:line 14
--- End of stack trace from previous location ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in /_/Rx.NET/Source/src/System.Reactive/Internal/ExceptionServicesImpl.cs:line 19
   at System.Reactive.ExceptionHelpers.Throw(Exception exception) in /_/Rx.NET/Source/src/System.Reactive/Internal/ExceptionServices.cs:line 16
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in /_/Rx.NET/Source/src/System.Reactive/Subjects/AsyncSubject.cs:line 430
   at TrashLib.Sonarr.ReleaseProfile.ReleaseProfileUpdater.DoVersionEnforcement() in D:\a\trash-updater\trash-updater\src\TrashLib\Sonarr\ReleaseProfile\ReleaseProfileUpdater.cs:line 65
   at TrashLib.Sonarr.ReleaseProfile.ReleaseProfileUpdater.ProcessReleaseProfiles(IDictionary`2 profiles, ReleaseProfileConfig config) in D:\a\trash-updater\trash-updater\src\TrashLib\Sonarr\ReleaseProfile\ReleaseProfileUpdater.cs:line 139
   at TrashLib.Sonarr.ReleaseProfile.ReleaseProfileUpdater.Process(Boolean isPreview, SonarrConfiguration config) in D:\a\trash-updater\trash-updater\src\TrashLib\Sonarr\ReleaseProfile\ReleaseProfileUpdater.cs:line 53
   at Trash.Command.SonarrCommand.Process() in D:\a\trash-updater\trash-updater\src\Trash\Command\SonarrCommand.cs:line 54
Exiting due to previous exception

Your hint is here:

401 (Unauthorized)

This means you are not authorized to connect to Sonarr. Check your API key.

I'm going to lock this issue. @gyCfjSnO please start a new discussion for further support if you still encounter issues.


Unable to assign subscription

Version Info
18.1 master
Describe the bug
After entering the marketplace portal and trying to assign a subscription we receive a message about its positive assignment, but after entering the MailDev we see that the transaction has been rollback
How to Reproduce
Steps to reproduce the behavior:

Go to the page oscm-portal/marketplace/index.jsf
Try assign any subscription
Check e-mail

Observed behavior
The assigned subscription still has the status "pending"
Expected behavior
The subscription will be added to the user
The subscriptions window will display a list of assigned subscriptions
Screenshots

Additional context
Notification of Catalog Manager about the provisioning process for subscription 'Lorem ipsum' failed.

What is service based on? Can you access the application underneath? Is is it up and running? Please check the docker logs, e.g.

Run following command and provoke the error
docker logs -f oscm-app
Check last entries in system.log file
 docker exec oscm-core cat /logs/system.log

For the supplier that was created during Pull_and_start on Jenkins, the assignment request works fine. We get an e-mail about its assignment, but we still can't see the subscription in the "My subcriptions" tab. However, for a new supplier, which we will create from the administrator account with the "Supplier" and "Technology provider" roles, after creating the subscription the situation is as described above. We receive errors in the console:
Subscription_oscm_core_console
Subscription_system_log

Sorry to repeat my question: Which service is used here?
Expected

As long as subscription is in pending state the user assignment is not available.
If the service is public and setup correctly, it is subscribe-able by any customer user.
The subscription state should switch to ready state after some time.

The attached error logs indicate that the service is not setup correctly. The technical service is created with another organization that doesn't own the service.
40125: The current organization doesn't own the technical service of the subscription

Note that the owning organization can be configured in the APP. For OpenStack, AWS, VMware or Azure services this can be done as platform administration in the APP UI. For the Sample Controller and for the Shell Integration this has to be done with respective docker-compose commands via command line. See the Operator's Guide "Updating Controller Configuration Settings" on page 21 how to do this.

@KabacinskiPiotr
Please align the responsible organization of the used APP controller with technology provider organization that owns the respective technical service.
This issue should be solved with the instructions I have given here. If you are still facing these issues though, then let me know and provide the details about the service. Otherwise please close this.


Refactor remaining client objects to avoid fat-arrow methods

Finishes removing fat-arrow methods throughout the codebase; this was started by #380. Using fat-arrow methods assigned as fields implicitly binds their value of this to the value present in the constructor; this can cause unexpected interactions with the JS prototype inheritance model.
(This is a part of the changeset currently known as #376; I'm working on factoring out some of its parts that are separable concerns and make the diffs hard to read if all rolled into one thing.)

(This is a part of the changeset currently known as #376; I'm working on factoring out some of its parts that are separable concerns and make the diffs hard to read if all rolled into one thing.)

(What's the process for merges in this project? Should I mash the green button now, or get more reviews?)


XenServer

add support to XenServer 7.1 (issue #65 )
Possibility to reuse previous extract updates (issue #54 )

Don't have means to test at the moment, but it looks sane to me! Many thanks for your efforts!

If you want, you can test XenServer inside a VM ( since 6.5 version).
The off-line functionality it'snt perfect because, the script need to have Internet access.


Code/ocr countries list

Add efficiency to code base, ocr.py; removed redundant code through use of sets to generate countries list and remove non-matches

Will close this branch, and rework from a new pull/new branch.


Cache not being utilized when called through Docker build

I've successfully cloned the turborepo-remote-cache and I've hooked it up in AWS. Everything is working fine, up until I try to use the cache from docker. Here's my dockerfile:
FROM docker.artifactory.moveaws.com/node:16 as build
COPY . /app
WORKDIR /app
RUN yarn install
RUN yarn build
RUN npx turbo run build

Both RUN yarn build and RUN npx turbo run build are not updating the cache. Originally I thought that my docker instance couldn't communicate with the deployed service, but when I added a CURL inside the container, I could verify that outbound calls are properly triggered.
My question is, has anyone run into this? Maybe someone has an example of a dockerfile already out there? Or what could possibly be the issue that the commands won't trigger the cache during the build phase?

You can follow here https://github.com/fox1t/turborepo-remote-cache#deploy-on-docker
You are not setting the vars.
You can also use the official image we pushed on docker hub.

You can find the Dockerfile here: https://github.com/fox1t/turborepo-remote-cache/blob/main/Dockerfile

@fox1t I believe @mkotsollaris is referring to building a docker image that uses the remote-cache and not deploying the remote cache. I am also running into issues when trying to reach out to my deployed remote-cache. When docker is running a turbo command it is not using hosted remote cache

every S3 bucket is a remote cache, no matter where the docker image runs. I am not sure why you can’t use the provided docker image. What are the limitations?

Ok, I finally got it. You are trying to connect to a remotely deployed remote-cache server from a docker container. Right?

Yes, we are basically just testing this implementation. It works great up until we try use remote-caching during docker build, for some reason it won't work! 🤔

Can you check if the same setup works with Vercel's own remote caching? We should exclude some kind of unexpected setting/bug on the turborepo client-side.

ref: https://github.com/vercel/turborepo/issues/1508

@StevenMatchett, I couldn't make it work with your hack. Can you please write every step you took to make it work?

I've just done some experiments, and it turned out that you don't need VERCEL_ vars (for me, turborepo-remote-cache is not working using VERCEL_ vars).
I added my findings to the README: https://github.com/fox1t/turborepo-remote-cache#enable-remote-caching-in-docker
Also, we needed to update turborepo-remote-cache to version 1.4, introducing the support for the slug query string parameter. Unfortunately, when you use the --team CLI option, it passes it as slug instead of teamId. I don't know why they decided like that or if it is just a bug. This is the ref: https://github.com/vercel/turborepo/blob/d83130a8b30b01ae2ce1627f4188e791598ea8b3/cli/internal/config/config.go#L172

Awesome thank you! I am curious if its a linux thing rather than a docker thing

@fox1t here is a PR for the fix in turborepo
https://github.com/vercel/turborepo/pull/1527

FIY - I still had the problem with the last updates.
With the node:16-slim turbo was not fetching cache from the remote server.
-> I had to switch for a node:16-alpine.
I do not know the reason, however it could be interesting to add it in the documentation :)

@LucM
What Architecture are you running your setup on?

Hello!
Had this similar issue myself and after some Wiresharking the connection between my local machine and turbo-cache I noticed that the TLS Handshake failed between my docker build and remote cache. No error was produced on the build logs about bad certificate. The certificate I use on remote cache is valid AWS Issued certificate.
I used node:18-slim base image and after installing ca-certificates package:
RUN apt-get update && apt-get install -y \
    ca-certificates \
&& rm -rf /var/lib/apt/lists/*

I was able to hit and connect to cache.

@LucM
As you said, I changed to use the node:16-alpine version and it worked! Thanx 🙏


chore: npm audit updates

vulnerability count now = 0

@121watts  Hey can you check if this breaks HMR for you? We were holding off on upgrading to 1.12.0 due to this, and also since the audit warnings are a false positive.

@chnn def breaks hmr.  closing pr.


[Bug] Summarizing on empty data should not return an error

Describe the bug
Whether summarizing or not summarizing, if input is empty, it should return an empty value rather than throwing an error.
According to me, whether we:

receive an input with several elements, but all element do not pass the set Filter
or receive an input with no element
or receive an input with no element, while not using the summarizing function

the same result shall be received: an empty Table.
This is important as we cannot know in advance if our API call will return an array with 0 or several elements. It also impacts the use within alerts.
Steps to reproduce

Source: Inline
Summarize: sum(value)
Summarize by: group
Filter : value > 100
data: []

Result:

error getting data frame from inline data. summarize by field not found. Not applying summarize

I would expect the same result as follows:

Source: inline
Summarize: sum(value)
Summarize by: group
Filter : value > 100
data :

[
  {
    "group": "A",
    "id": 1,
    "value": 42
  },
  {
    "group": "B",
    "id": 2,
    "value": 37
  },
  {
    "group": "A",
    "id": 3,
    "value": 28
  },
  {
    "group": "C",
    "id": 4,
    "value": 53
  },
  {
    "group": "B",
    "id": 5,
    "value": 19
  },
  {
    "group": "C",
    "id": 6,
    "value": 45
  },
  {
    "group": "A",
    "id": 7,
    "value": 32
  },
  {
    "group": "B",
    "id": 8,
    "value": 29
  },
  {
    "group": "C",
    "id": 9,
    "value": 61
  }
]

Result: <0 series returned> (aka empty table)
or for

Source: Inline
Filter : value > 100
data: []

Result: <0 series returned> (aka empty table)
Version Details:

Grafana version : 10.4.1
Plugin version : 0.7.1

I have also had this issue and it trips me up often!


Paging forward resets UIViewController.view layout, paging backward does not

I'm not even sure how to describe what I'm seeing. Basically I have a container view that houses a PagingViewController
The pages all have different heights, which tell the container view to change height on pagingViewController: PagingViewController, didScrollToItem
When I score forward, the appears, slides off to the left of the window, and then flies back in, its a very jarring transformation.
When I scroll backwards though, everything smoothly animates.
I'm at a total loss. Nothing different is happening when I am swiping forward vs swiping backward
Here is a gif to demonstrate it

Hi @bryan1anderson! I'm not really sure what would be causing this. Are you able to reproduce it in a smaller case/using the example targets? I'm also struggling a bit too understand which parts are which here. What is the menu here and what is the content view controllers?

Need more details to help with this. Would be great if you could open new issues where you explain a bit more what the actual issue is and provide a way to reproduce it 🙂


Implement internal SFTP subsystem

Removes the need for Docker based SFTP, simplifies error handling, reduces errors caused by containers not syncing correctly. Also makes it so users can login using their username and password from the Panel.
Current known limitations:

OSX: Transmit client does not support download/upload of large files.
OSX: Cyberduck does not work at all.

assoc: https://github.com/Pterodactyl/Panel/pull/703

@Dane we suggested you to implement separated user access to a specified servers.
(if you remember: 1 server, multiple users with specified perms)


feature: Allow specifying custom scalars when generating queries using Kotlin query projections

Describe the Feature Request
When using the new Kotlin Query projections, it is currently impossible to use custom scalar configuration when constructing the query.
Using custom scalars is available when using the GraphQLQueryRequest API, so this looks like a missing feature in the new API.
I understand that the new Kotlin codegen is still experimental, but having checked the code, I found there is currently no way to possibly achieve this, hence this feature request.
Describe Preferred Solution
It would be ideal if we could provide custom scalar implementations to the new query projections in a similar fashion it is currently done for GraphQLQueryRequest.
Describe Alternatives
I can imagine being able to register custom scalar implementations in a different way, possibly on the client level or such. But the only other alternative seems to be the current scenario where this is not possible at all.

cc @mbossenbroek

That should be supported, unless you're thinking of something different than what we have...
Check out this integration test example:
https://github.com/Netflix/dgs-codegen/tree/master/graphql-dgs-codegen-core/src/integTest/kotlin/com/netflix/graphql/dgs/codegen/cases/dataClassWithMappedTypes
There are a couple of custom scalars in there, mapped with the same configuration as the normal codegen stuff. This is where it gets wired up for these tests:
https://github.com/Netflix/dgs-codegen/blob/master/graphql-dgs-codegen-core/src/integTest/kotlin/com/netflix/graphql/dgs/codegen/Kotlin2CodeGenTest.kt#L52-L58
In gradle, the same config would look something like this:
typeMapping = [
  "Long" : "kotlin.Long",
  "DateTime" : "java.time.OffsetDateTime",
  "PageInfo" : "graphql.relay.PageInfo",
  "EntityConnection" : "graphql.relay.SimpleListConnection<com.netflix.graphql.dgs.codegen.cases.dataClassWithMappedTypes.expected.types.EntityEdge>"
]

This was the type mapping setup that the codegen stuff was using when I made this kotlin version. It looks like that there's a new way of passing this mapping in code that we don't support in the new DSL. Let me know if the configuration mapping won't work for you & I can take another look.

Hi @mbossenbroek,
Thanks for a prompt response. It seems I didn't make myself clear enough, sorry.
The problem I'm experiencing is not related to the code generation itself, that actually works fine. But it is related to the type mapping. Let me explain.
Context
We have the codegen configured with type mapping like so:
    typeMapping =
        mutableMapOf(
            "ISO8601Date" to "java.time.LocalDate",
            "UUID" to "java.util.UUID",
            "ID" to "com.productboard.common.graphql.id.GraphqlId"
        )

where the class we map the ID to looks like this:
data class GraphqlId(val type: String, val id: String)

This correctly generates Kotlin classes where the GraphqlId is used for roperties representing GQL ID fields.
We then use something like this
@DgsScalar(name = "ID")
class GraphQlIdScalar : Coercing<GraphqlId, String> {
    // implementation here
}

to plug the custom type into the framework. So far so good.
The problem
However, when I try actually generating a query using a projection with input arguments containing this custom type, it doesn't validate.
Each GraphqlId value gets incorrectly serialized into the query as
{id: 'someId', type: 'someType'}

This is apparently because the projection doesn't know it should serialize this type using the corresponding Coercing and so it uses the default property-based serialization.
While this seems to be supported in the "old" API via the option to pass explicit scalars into a GraphQLQueryRequest, there's no such possibility when using DgsClient.build*.
Having checked the code, it looks like when generating the query string, the program reaches this point where it tries to serialize the value. However, the inputSerializer is instantiated like this, i.e., with no arguments.
This is different from the aforementioned GraphQLQueryRequest usage where it takes the provided scalars configuration (if any) into account, see here.
Note that in my case, this manifests in tests where I'm using the projection of a query with input arguments to generate a query string which then gets executed.

Hope this make it clearer. I see how my original post could have easily been misunderstood, sorry about that 🙂

Gotcha - that makes more sense. I'll take a deeper look at this, but it should be doable. That said, I might not get to it for a few weeks, and as always - feel free to drop a PR with the change :)

I'll do my best to have a look 🙃 I just think it won't be exactly trivial to implement for someone with zero knowledge of the codegen internals such as myself.
I suspect this will require changes to the code generation itself to make the option available on the DgsClient which is generated, unlike the GraphQLQueryRequest. Correct? 🤔

This is by no means a blocker for us, so I'm more than happy to wait 🙃 Thanks for being so helpful though 👍

Hey @dmoidl , I was just about to do this today & somebody else dropped this PR, which I believe will satisfy your need, right? Lmk if you think this will work or not... https://github.com/Netflix/dgs-codegen/pull/675

This is released now

Thank @mbossenbroek for letting me know 🙂 I think this will do just fine. However, I believe it's not yet released, is it? 🤔 The latest available version seems to be 6.1.10 and I can't see the new InputValueSerializerProvider in there.
No worries though, I'll wait until it makes it to a released version.

Ahh apologies - I changed the original PR a bit in this PR: https://github.com/Netflix/dgs-codegen/pull/677/files#diff-39435f9dc1a43ff8f5570174fdbd1d26970cc2fe5b3b94c22f43e06f403bd8b0
The old way used a thread local to pass in a custom serializer; the new way generates the projections such that they take an optional parameter that's an InputValueSerializerInterface. I was trying to dig up the test I wrote for this as an example, but now I can't find it 🤔 . Maybe it got lost in a merge somewhere?
In any case, lmk if that doesn't make sense & I'll reinstate an example/test for you.


etcd(ticdc): adjust etcd worker rate limit

What problem does this PR solve?

Issue Number: close #7374
What is changed and how it works?

changefeed rate.Limit(tickRate) to rate.Every(timerInterval),  this will fix if the tick interval is greater than  1 second, etcd worker will be always rate limited.
increase burst of rate limit from 1 to 2, since the p2p scheduler will not write too much etcd data

Check List
Tests

Unit test
Integration test

Questions
Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?
Release note
`None`.

Codecov Report

Merging #7375 (5388ad5) into master (11dbe58) will decrease coverage by 0.0468%.
The diff coverage is 45.0142%.

Additional details and impacted files

Flag
Coverage Δ

cdc
67.6125% <42.2310%> (-0.1490%)
:arrow_down:

dm
51.9787% <52.0000%> (+0.0352%)
:arrow_up:

engine
60.8048% <ø> (-0.0814%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.
@@               Coverage Diff                @@
##             master      #7375        +/-   ##
================================================
- Coverage   59.8873%   59.8405%   -0.0469%
================================================
  Files           799        800         +1
  Lines         91637      91804       +167
================================================
+ Hits          54879      54936        +57
- Misses        31982      32097       +115
+ Partials       4776       4771         -5

/merge

/merge

/run-verify


Updated to the latest version (1.0.0-rc2) of Google Cloud Datastore API

Updated to the latest version (1.0.0-rc2) of Google Cloud Datastore API.

The rc2 version has the below breaking changes:

DateTime and DateTimeValue are gone, replaced by Timestamp and TimestampValue
HttpTransportOptions is now in a new package.

Done.


Warning: Received NaN for the offset attribute. If this is expected, cast the value to a string.

I have been getting this warning when using the Linear Progress Bar
This is my code:

lol afik that wasn't intended.
Thanks! I'll try to add a fix ❤

@ItsFlash10 A bit late but fixed in<EMAIL_ADDRESS>Thanks for using Frogress 🙇


ERROR: No Factors were found!

Trying to set up this tool but the cli exists with the following error
Info: Calling https://<org>.okta.com/api/v1/authn
ERROR: No Factors were found!

Do you have any factors configured for your Okta user account?


Read from Github API

This is a feature request/question.
I was wondering if you guys had any development plans to support reading/calls to the github api? I was able to implement a relatively simple extension of the http module's open, and it works well for uncompressed files (also I realize that it would be simple enough for a user just to pass the bearer api token as header, read, json.loads, and decode so might or might not necessarily be a valuable code addition for this use case). However, this does not work well for compressed files, and in the user flow users wouldn't be able to leverage the built in decompression.
A use case I could see this being very helpful is an ETL flow of a version controlled compressed csv in a private repo. It would operate something like below (fake file path):
import pandas, os
from smart_open import open

url = "https://api.github.com/repos/<owner>/<repo>/contents/data_file.bz2"
with open(url, transport_params={"git_token" : os.environ.get("TOKEN","")} ) as file_object:
    df = pandas.read_csv(file_object)

# transforms and writes
...

Also with github, writing would be a bit of an issue with needing to specify a commit message and generally not wanting to push one file at a time. So I thought it made sense to fall into the http module because it is already read only.

No, not really. I don't think the benefits justify the cost. For example, the abstraction that smart_open uses is streaming. In this particular case, this abstraction doesn't fit well with github in the write case, as you have correctly pointed out.
For people that need read stuff from github, HTTPS approach you detailed is probably good enough. Would you be interested in adding that to the howtos.md file in a PR?

Gotcha, yeah that makes sense. Reading in a full response, modifying, and wrapping in a stream isn't exactly the same thing as a proper stream haha.
Sure, yeah I'll add a section in that file about the http approach above and create a PR.

I see that it's merged in so going to close the issue.