./libsymbolizer/build.sh - build error via pip upgrade of Sentry

The following error occurred while trying to upgrade a working installation of Sentry 8.8 to 8.9 via pip:
Failed to build symsynd
Installing collected packages: cffi, symsynd, uwsgi, raven, progressbar2, percy, sentry, setuptools, cryptography, pytz
  Found existing installation: cffi 1.8.2
    Uninstalling cffi-1.8.2:
      Successfully uninstalled cffi-1.8.2
  Running setup.py install for symsynd
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-BeCqMC/symsynd/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-iFqd58-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    ./libsymbolizer/build.sh
    ./libsymbolizer/build.sh: 9: [: Linux: unexpected operator
    ./libsymbolizer/build.sh: 21: [: x0: unexpected operator
    ./libsymbolizer/build.sh: 32: ./libsymbolizer/build.sh: cmake: not found
    make: *** [build] Error 127

    ----------------------------------------
Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-BeCqMC/symsynd/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-iFqd58-record/install-record.txt --single-version-externally-managed --compile" failed with error code 2 in /tmp/pip-build-BeCqMC/symsynd

Let me know what logs, data or other details you require, happy to assist further!

I'm on Ubuntu 14.04 server. Had to install cmake (via apt) but now I get...
  ./libsymbolizer/build.sh
  ./libsymbolizer/build.sh: 9: [: Linux: unexpected operator
  ./libsymbolizer/build.sh: 21: [: x0: unexpected operator
  CMake Error at CMakeLists.txt:3 (cmake_minimum_required):
    CMake 3.4.3 or higher is required.  You are running version <IP_ADDRESS>

  -- Configuring incomplete, errors occurred!
  make: *** [build] Error 1

  ----------------------------------------
  Failed building wheel for symsynd
Failed to build symsynd
Installing collected packages: symsynd, uwsgi, raven, progressbar2, percy, sentry, setuptools, cryptography, pytz
  Running setup.py install for symsynd
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-VAciXx/symsynd/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-ekSIst-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    ./libsymbolizer/build.sh
    ./libsymbolizer/build.sh: 9: [: Linux: unexpected operator
    ./libsymbolizer/build.sh: 21: [: x0: unexpected operator
    CMake Error at CMakeLists.txt:3 (cmake_minimum_required):
      CMake 3.4.3 or higher is required.  You are running version <IP_ADDRESS>

    -- Configuring incomplete, errors occurred!
    make: *** [build] Error 1

    ----------------------------------------
Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-VAciXx/symsynd/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-ekSIst-record/install-record.txt --single-version-externally-managed --compile" failed with error code 2 in /tmp/pip-build-VAciXx/symsynd

Perhaps the module needs requirements updated to include cmake and newer version(s) at very least?

If I don't use the symbols with iOS etc, can the installation of symsynd be skipped to use Sentry?   I don't want to ppa or build my own cmake if I don't need this use case...

Hmm, why is the wheel not being used for installation here? @mitsuhiko
@gregwjacobs Do you have an up to date version of pip? Not sure when linux wheel support was added off the top of my head, but that'd let it install without having all these crazy dependencies. cmake is just the tip of the iceburg in what you'd need for it to fully succeed.

I guess you need pip>=8.1. @gregwjacobs Can you verify with your pip --version?

I have the same error on travis. I use the latest version of pip-8.1.2.
Logs: https://travis-ci.org/bogdal/sentry-youtrack/builds/165314799

@bogdal You're not using pip at all to install. You're doing:  python setup.py install. To use pip, do: pip install . instead. That should then resolve all dependencies through pip.

Ah, right. Thanks @mattrobenolt 👍

Confirmed, thanks gents. Yes I had prior this version of pip. Once upgraded from 7.1.0 to pip version 8.1.2 the install/upgrade to Sentry worked fine.

You might want to have a requirement or warning that a newer pip is required perhaps?


Multiple static placeholders in single page throws error.

Summary

Expected behaviour
Actual behaviour
Environment

Python version:
Django version:
django CMS version:

The error is,
ValueError: Cannot assign "1": "StaticPlaceholder.site" must be a "Site" instance.
I fixed the issue in my project by changing below thing in "cms/templatetags/cms_js_tags.py" in function "render_cms_structure_js" line 49.
Before
kwargs['site'] = renderer.current_site.pk
After
kwargs['site_id'] = renderer.current_site.pk
This kwargs is used in get_or_create(**kwargs) function below which is a standard django function and it requires either the site object or the site_id for getting or creating StaticPlaceholder object.
Hope this helps someone !!!!

This appears to have been resolved, I'm assuming that this was probably caused by the use of a different Django version than what was supported. Closing for now.


Optional filter specification for "ext" doesn't work

Videos that are provided only in "mp4" by YouTube break youtube-dl when using a filter specification such as:
$ youtube-dl -f 'bestvideo[ext=?webm]+bestaudio' -g https://www.youtube.com/watch?v=1UE2x2OcfxI
ERROR: requested format not available

I have [ext=?webm] stored as a default option, so that WebM is preferred, but if it doesn't exist (like in the video given in the example above), then H.264 video is OK. The point is that WebM should be preferred if both are available, but it shouldn't fail if WebM is not available, since the filter operator is =?, not just =.
In case it matters, the same thing happens with audio. Same error message if I try bestvideo+bestaudio[ext=?webm].
A workaround exists, but it can get rather long. For example, this doesn't work:
bestvideo[ext=?webm]+bestaudio[ext=?webm]

But the longer equivalent does work:
bestvideo[ext=webm]+bestaudio[ext=webm]/bestvideo[ext=mp4]+bestaudio[ext=webm]/bestvideo[ext=mp4]+bestaudio[ext=mp4]/bestvideo[ext=webm]+bestaudio[ext=mp4]

(The reason I prefer webm is because YouTube's H.264 encodes are lower quality compared to their VP9 encodes, even though the former has a higher bitrate compared to the latter.)

That's not how =? works. [ext=?webm] matches either webm or unset ext values, but all of the formats of the video have non-webm extensions.

Oh. Is that useful though? Shouldn't it mean "webm, and if that's not available, then whatever you would have picked instead."

@realnc You may want to try -f 'bestvideo[ext=webm]+bestaudio/bestvideo+bestaudio/best'.

@yan1212 As mentioned already, I'm using bestvideo[ext=webm]+bestaudio[ext=webm]/bestvideo[ext=mp4]+bestaudio[ext=webm]/bestvideo[ext=mp4]+bestaudio[ext=mp4]/bestvideo[ext=webm]+bestaudio[ext=mp4], which is kind of a monstrosity. But it that's how it's designed to be...

FYI: Grouping in format filters is requested in #5242 and implemented in #6124. You may want to try it. I guess (bestvideo[ext=webm]/bestvideo[ext=mp4])+(bestaudio[ext=webm]/bestaudio[ext=mp4]) works the same as the previous filter.


Docs: Adding and Removing Projectors and Reactors

A while ago, in #131, I said I'd add the docs for the function: Projectionist::withoutEventHandler($class).
Here it is! I also included some typo fixes and link updates for the v3 docs.

Thank you!


Fix Custom Token editor and Card Token editor bug

The Custom Token editor and Card Token editor have a bug where you are able to edit a token, delete it at the same time, and then save it. This throws a NullPointerException.

Backlogged: Due to a last-minute redesign of the token editor, wasn’t able to add many checks or prevention measures for more specific bugs like this.


1417 policy validations

Summary
Create a validation webhook for policy CRD and their tests
Fixes: #1417
cc: @hectorj2f @mattmoor @vaikas

Codecov Report

Merging #1548 (6b2ef77) into main (4b2c3c0) will decrease coverage by 0.27%.
The diff coverage is 55.76%.

@@            Coverage Diff             @@
##             main    #1548      +/-   ##
==========================================
- Coverage   26.49%   26.21%   -0.28%
==========================================
  Files         126      131       +5
  Lines        7214     7510     +296
==========================================
+ Hits         1911     1969      +58
- Misses       5093     5327     +234
- Partials      210      214       +4

Impacted Files
Coverage Δ

cmd/cosign/webhook/main.go
0.00% <0.00%> (ø)

...kg/apis/cosigned/v1alpha1/zz_generated.deepcopy.go
0.00% <0.00%> (ø)

...cosigned/v1alpha1/clusterimagepolicy_validation.go
82.85% <82.85%> (ø)

...apis/cosigned/v1alpha1/clusterimagepolicy_types.go
0.00% <0.00%> (ø)

...s/cosigned/v1alpha1/clusterimagepolicy_defaults.go
0.00% <0.00%> (ø)

pkg/apis/cosigned/v1alpha1/register.go
0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b2c3c0...6b2ef77. Read the comment docs.

I'm just curious why the test run needed to be approved since it says first time contributor, but we got the other PR merged yesterday, so I don't understand why it blocked you again? 🤔

It didn't need approval to run when I was the author. Looks like Adam has to contribute a PR under his name to get the permission. The merged PR was under my name.


StackVisualizer.cs wishlist

Description

You can define what layer StackVisualizer.cs goes on. (This is so you could have the StackVisualizer and then something over it.
Use alphas. This entails assigning a color: to a layer in the stackLayers.
Show specific stacks for specific items. I don't know how this would play out YAML wise but maybe something like:

- type: StackVisualizer
  composite: true
  stackLayers:
    - welder
      prototypes:
        - id: Welder
        - id: WelderEmergency
    - wrench
       prototypes:
         - id: Wrench

Something like that. This is made more confusing with tags.
4. StackVisualizers for inhands and onmobs.  Stack visualizer is very simple. It assumes that things are generic. I.e. you have 0/10, 1/10, 2/20, etc. items.
I agree that declaring layer is an improvement for StackVisualizer.
But the rest seem like some kind of SlotVisualizer (name pending).

Most of these are implemented so i'm happy.


Perf monitor kafka

Karan, I've closed this as it is old and we need to rethink how we approach detector evaluation. RMSE is problematic in cases where the model simply tracks current observations (e.g. EWMA with alpha=1).


Explicitly expose ports on driver container, fixes #617

What changes were proposed in this pull request?
Expose ports explicitly in the driver container. The driver Service created expects to reach the driver Pod at specific ports which before this change, were not explicitly exposed and would likely cause connection issues (see #617).
How was this patch tested?
Failure in #617 reproduced on Kubernetes 1.6.x and 1.8.x. Built the driver image with this patch and observed fixed #617 on Kubernetes 1.6.x.

We have never run into this issue as reported here. @foxish.

I imagine if a K8s cluster is secured with a network policy or something pods are not exposed by default which would make this change necessary.

@liyinan926 Just to confirm that we've seen a very similar issue with specific network overlays - Romana - but not encountered it with others

@rvesse are you able to test this patch and see if it fixes your issue?

@liyinan926 We have a different patch internally which fixes our issue, not sure if it is the exact same problem though clearly closely relate. The developer who did the debugging and patch is currently out on jury duty until mid next week so I am waiting for her to get back and take a look at this

Can this be posted against apache/spark instead?

+1, please post the fix against the upstream repo. We'll backport it here
at some point.
On Wed, Mar 7, 2018 at 1:13 PM mccheah<EMAIL_ADDRESS>wrote:

Can this be posted against apache/spark instead?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache-spark-on-k8s/spark/pull/618#issuecomment-371286574,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA3U57zyIYJaDzqkHvo9wRh3L5ef1N6kks5tcE1vgaJpZM4SV42C
.

--
Anirudh Ramanathan

I'm the developer that works with rvesse. We have the exact same stacktrace, but I think our underlying problem is different. As far as we could figure out, the problem was coming from that Romana relies on entries in the routing table to route traffic from 10. to 192. and the headless service has no IP and doesn't get written to that table. Very long story short I was able to work around it by changing
val driverHostname = s"${driverService.getMetadata.getName}.$namespace.svc.cluster.local"
in resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/submitsteps/DriverServiceBootstrapStep.scala to
val driverHostname = s"${driverService.getMetadata.getName}"
because the driver service name would also resolve to the IP of the driver pod, which is written to the routing table.
We can try this fix, but I'm doubtful it will be a fix as Romana still won't know how to talk to the node.


#1: Connect interesting C++ functions to Python through pybind11.

For real valued matrices, I added eigenvectors and eigenvalues from both scalar and float type matrices.


fix: convert coroutine to task

Change-Id: Id760f2caac61bc6f773552472b3f965e96939245
Depends-On: #158

This pull request is part of a stack:

chore: rename git_push_stack to mergify_cli (#156)
fix: Make worktree compliant (#157)
feat: get token through gh auth token if not provided by --token or GITHUB_TOKEN env var (#158)
fix: convert coroutine to task (#159)
feat: Add control on draft or open status through a param (#160)
feat: Define trunk through param --trunk/-t or git config mergify-cli.stack-trunk (#161)


DBZ-2654 Edit Customizing Topic Auto-Creation

Jira DBZ-2654
WIP: Pending meeting with @rk3rn3r to discuss some aspects of the material.
I tested the file as-is in a local Antora build, as well as in a local downstream build.

@roldanbob, @rk3rn3r, this is still evolving, right?

@roldanbob, @rk3rn3r, this is still evolving, right?

Yes, I've still got some updates to incorporate from my meeting with @rk3rn3r, and I'm waiting for him to review the current draft.

Updated draft. I think this is improved/ready and addresses input with the exception of the question about terminology (automatic topic creation).  Let me know whether you think that the terminology issue is a blocker.


Feature Request: add ability to run -03 twice

This bug first occurs in version 0.54 (0.49-0.53 are OK). This results in a performance decrease of a factor of 4 in a bit optimized summation and a factor of 2 in a naive implementation.
Example
import numpy as np
import numba as nb
import time

f = np.random.rand(32,33,2000,64)
b = np.random.rand(32,64)

@nb.njit(fastmath=True)
def test_numba_1(f,b):
    fnew = np.zeros((f.shape[1],f.shape[2]))

    for i in range(f.shape[0]):
        for j in range(f.shape[1]):
            for k in range(f.shape[2]):
                for l in range(f.shape[3]):
                    fnew[j,k] += f[i,j,k,l] * b[i,l]
    return fnew

@nb.njit(fastmath=True)
def test_numba_2(f,b):
    fnew = np.zeros((f.shape[1],f.shape[2]))

    for i in range(f.shape[0]):
        for j in range(f.shape[1]):
            for k in range(f.shape[2]):
                acc=fnew[j,k]
                for l in range(f.shape[3]):
                    acc += f[i,j,k,l] * b[i,l]
                fnew[j,k]=acc
    return fnew

%timeit test_numba_1(f,b)
%timeit test_numba_2(f,b)

#'0.53.1' Last version without bug
#108 ms ± 1.35 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
#53.9 ms ± 809 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

#'0.54.0' First version with bug
#216 ms ± 4.68 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
#200 ms ± 2.93 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

#'0.55.0'
#218 ms ± 5.36 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
#201 ms ± 1.73 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Another question would be if, a rewrite pass is possible in Numba to optimize test_numba_1 automatically to test_numba_2. Clang shows exactly the same behavior as Numba before version 0.54, therefore this likely has to be fixed before generating LLVM-IR.
//clang v11
//clang -shared -O3 -march=native -ffast-math -o summation.dll summation.c
__declspec(dllexport) void eq_nb_1(size_t s0,size_t s1,size_t s2,size_t s3,double f[][s1][s2][s3],double b[][s3],double fnew[][s2])
{
  for (size_t i=0;i<s0;i++){
    for (size_t j=0;j<s1;j++){
      for (size_t k=0;k<s2;k++){
         for (size_t l=0;l<s3;l++){
           fnew[j][k]+=f[i][j][k][l]*b[i][l];
         }
      }
    }
  }
}

__declspec(dllexport) void eq_nb_2(size_t s0,size_t s1,size_t s2,size_t s3,double f[][s1][s2][s3],double b[][s3],double fnew[][s2])
{
  for (size_t i=0;i<s0;i++){
    for (size_t j=0;j<s1;j++){
      for (size_t k=0;k<s2;k++){
         double acc=fnew[j][k];
         for (size_t l=0;l<s3;l++){
           acc+=f[i][j][k][l]*b[i][l];
         }
         fnew[j][k]=acc;
      }
    }
  }
}

import numpy as np
import ctypes

lib = ctypes.cdll.LoadLibrary("summation.dll")

def eq_nb_1(f,b):
    fnew = np.zeros((f.shape[1],f.shape[2]))
    s_0=ctypes.c_size_t(f.shape[0])
    s_1=ctypes.c_size_t(f.shape[1])
    s_2=ctypes.c_size_t(f.shape[2])
    s_3=ctypes.c_size_t(f.shape[3])

    f_p   =f.ctypes
    b_p   =b.ctypes
    fnew_p=fnew.ctypes

    lib.eq_nb_1(s_0,s_1,s_2,s_3,f_p,b_p,fnew_p)
    return fnew

def eq_nb_2(f,b):
    fnew = np.zeros((f.shape[1],f.shape[2]))
    s_0=ctypes.c_size_t(f.shape[0])
    s_1=ctypes.c_size_t(f.shape[1])
    s_2=ctypes.c_size_t(f.shape[2])
    s_3=ctypes.c_size_t(f.shape[3])

    f_p   =f.ctypes
    b_p   =b.ctypes
    fnew_p=fnew.ctypes

    lib.eq_nb_2(s_0,s_1,s_2,s_3,f_p,b_p,fnew_p)
    return fnew

%timeit eq_nb_1(f,b)
%timeit eq_nb_2(f,b)
#100 ms ± 311 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
#50.5 ms ± 1.08 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Hi @max9111, thanks for the report. I can confirm there's a performance regression when moving from Numba 0.53 to newer versions. I'll mark this for discussion at the next triage meeting, which will happen next Tuesday.

IIRC 0.54 was the first version using LLVM 11 and there were some things to do with missed vectorization that Numba encountered when the upgrade was undertaken (which were fixed, but I guess this one was missed).
This diff recovers the 0.53 performance, but at the cost of increased compilation effort:
diff --git a/numba/core/codegen.py b/numba/core/codegen.py
--- a/numba/core/codegen.py
+++ b/numba/core/codegen.py
@@ -1189,7 +1189,7 @@ class CPUCodegen(Codegen):
         self._data_layout = str(self._target_data)
         self._mpm_cheap = self._module_pass_manager(loop_vectorize=False,
                                                     slp_vectorize=False,
-                                                    opt=0,
+                                                    opt=2,
                                                     cost="cheap")
         self._mpm_full = self._module_pass_manager()

which suggests it's still possible to do the expected optimisation and it's just a question of working out precisely what combination of passes that is!

What appears to be the same performance bug has just been reported to https://stackoverflow.com/questions/75833328/why-do-these-two-different-ways-to-sum-a-2d-array-have-such-different-performanc . It's the fix planned for the next release?

This issue was raised at the Numba public meeting on 2023-03-28. The outcome is that the 0.58 release series will have support for an environment variable that if set will reinstate the previous behaviour (effectively running -O3 twice).

I have updated the title of this Issue to reflect it's change in state. It is now a feature request scheduled for inclusion in Numba 0.58.
The feature request is: as a Numba user I want to be able to set the environment variable NUMBA_RUN_O3_TWICE which will effectively "run -O3 twice" and run the code path from the patch at: https://github.com/numba/numba/issues/8172#issuecomment-1160474583

I have a general question:
Why is that the new default?
A real-world function without a severe performance drop (factor 1,5-5x) is hard to find. The improvement of compilation time is usually neglectable, it's often a few percent, with caching it doesn’t matter at all.
What is the reason for this? It absolutely makes no sense at all. Compilation times can be high in heavily NumPy - vectorized codes. If Numba is able to compile that at all (quite some features missing, but anyway extremely complicate to optimize this code) the perfomance benefit is quite often neglectable (no real usecase). Usually there is some simple (or maybe not as simple but doable)  rewrite necessary to get a good result in this case.
All this without reimplementation in (C/Fortran)+Wrapper+Dependencies, which is a different story in terms of effort. That's the beauty of Numba.
I really don't get the point to make this bug or “feature” as a default. To my opinion it is one of the most significant bugs in the last years.

The patch from #8172 (comment) slows down the fast variant in the reproducer from #9038. Without the patch, dist_cityblock_2() runs in:
2.34 µs ± 10.9 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

With the patch, it runs in:
4.05 µs ± 12.7 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

so applying the patch above does not look like it would universally improve performance. I can see why a user would want the option to use opt=2 at their discretion though.

Numba 0.57 seems to change the behavoir, on 0.56 I can't reproduce this behaviour:
%timeit dist_cityblock_1(x, y)
#0.56, windows installed via pip (default)
#3.82 µs ± 1.45 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
#0.56, windows installed via pip (opt=2)
#2.02 µs ± 24.3 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

#0.57, windows installed via pip (default)
#3.76 µs ± 0.962 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
#0.57, windows installed via pip (opt=2)
#4.13 µs ± 61.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit dist_cityblock_2(x, y)
#0.56, windows installed via pip (default)
#6.56 µs ± 4.09 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
#0.56, windows installed via pip (opt=2)
#2.02 µs ± 24.3 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

#0.57, windows installed via pip (default)
#2.09 µs ± 19.5 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
#0.57, windows installed via pip (opt=2)
#5.61 µs ± 2.13 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

The issue with the initial question seems to be fixed with 0.57, but opt=2 seems to be slower than the default now:
%timeit test_numba_1(f,b)
#0.56, windows installed via pip (default)
#123 ms ± 290 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
#0.56, windows installed via pip (opt=2)
#78.1 ms ± 182 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

#0.57, windows installed via pip (default)
#77.3 ms ± 499 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
#0.57, windows installed via pip (opt=2)
#77.6 ms ± 491 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit test_numba_2(f,b)
#0.56, windows installed via pip (default)
#120 ms ± 914 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
#0.56, windows installed via pip (opt=2)
#36.2 ms ± 556 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

#0.57, windows installed via pip (default)
#38 ms ± 1.04 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
#0.57, windows installed via pip (opt=2)
#71 ms ± 341 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

On other examples, there seems to be still a bit slower timings with 0.57 (default) to previous versions with (opt=2).

Numba 0.57 seems to change the behavoir

Numba 0.57 went from LLVM 11 to 14, so that might explain a lot of changes - unfortunately it does make it hard to directly compare the two.


ci: add lighthouse CI

This is not the final copy of the action probably, this is just to get started, we can improve it as we use :D
@eddiejaoude you need to do this before this PR can be merged: https://github.com/GoogleChrome/lighthouse-ci/blob/main/docs/getting-started.md#github-app-method-recommended

This repo has the action I mentioned on the livestream. You also need to add a config file though and allow lighthouse ci access to the repo. I’m not sure how the two compare, but mine stops a deploy if scores fall below a certain level. We maybe need to decide if that is something we want or if lighthouse scores are just a nice to have that we can keep an eye on and raise issues ourselves if values fall below a certain level. https://github.com/emmalearnscode/learn-cool-problems/blob/main/lighthouserc.json

Hi @emmalearnscode, could you review this once? 🙂

#1152 (Sorry, I think I deleted the branch)


Bring Medium cards

I guess we need some stuff to reflect new design even on our Medium to reflect upgrades ;)
@XyloDrone ? ❤🔥
https://medium.com/kodadot

https://medium.com/kodadot
https://medium.com/metaprime
check mate

Link to latest medium cards


Cherry-pick Performance page improvements to release

Porting https://github.com/gradle/gradle/pull/21613 to release
Context
This change should exist in the next release.
Contributor Checklist

[X] Review Contribution Guidelines
[X] Make sure that all commits are signed off to indicate that you agree to the terms of Developer Certificate of Origin.
[X] Make sure all contributed code can be distributed under the terms of the Apache License 2.0, e.g. the code was written by yourself or the original code is licensed under a license compatible to Apache License 2.0.
[X] Check "Allow edit from maintainers" option in pull request so that additional changes can be pushed by Gradle team
[X] Provide integration tests (under <subproject>/src/integTest) to verify changes from a user perspective
[X] Provide unit tests (under <subproject>/src/test) to verify logic
[X] Update User Guide, DSL Reference, and Javadoc for public-facing changes
[X] Ensure that tests pass sanity check: ./gradlew sanityCheck
[X] Ensure that tests pass locally: ./gradlew <changed-subproject>:quickTest

Gradle Core Team Checklist

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation
[ ] Recognize contributor in release notes

OK, I've already triggered the following builds for you:

BuildDistributions build

Your PR is queued. See the queue page for details.

OK, I've already triggered a build for you.


Add @Startup for CassandraTrackingQuery

Force to create tables at start up.

Some unit test have no test container for cassandra. Drop it for now.


fix debian package after addition of license header in pom.xml

PR#737 add lines  in pom.xml which break debian build.
This fix debian package build issue.

Verified that the command works as expected.
LGTM

LGTM - I am going to close and open to see if Travis will pass.

@pdion891 If you force push your commits, Travis will run again. It will then probably pass and we can merge it. Thanks!

@pdion891 All green, merging!

Thanks!
On Thu, Aug 27, 2015 at 8:13 AM, asfgit<EMAIL_ADDRESS>wrote:

Merged #746 https://github.com/apache/cloudstack/pull/746.
—
Reply to this email directly or view it on GitHub
https://github.com/apache/cloudstack/pull/746#event-393942468.


Allow disabling thumbnail preview (also by default)

Problem
The preview widget aids the artist to give an idea of the playblast outcome, nevertheless for very large and complex scenes it's unnecessarily slow because it will try to update on each option change that results in different output. Instead it would be easier for those scenes to allow the artist to update the preview only manually.
Solution
Allow disabling the auto-updating feature of the thumbnail (and maybe even hide it to gain screen space?).

Allow disabling the auto-updating feature of the thumbnail (and maybe even hide it to gain screen space?).
This is currently the case, after the user clicks preview the widget pops up but it will not auto-update when settings are changed.
Instead it would be easier for those scenes to allow the artist to update the preview only manually.
Adding a preview state attribute to the preview widget which can be set by a checkbox, if checked it will be able refresh, if not it will return.
Open for discussion : create error message box to warn the user if preview widget has been disable or just print the error. I opted for the first one.
Note:  I was not able to push the branch to this repository.

Toggling display with checkbox works and does not refresh when toggled.
Implemented warning message when preview is disables and preview is clicked.


Footer on root page should be at the bottom of window

If the footer can stay at the bottom, I think this can be tagged as v0.8.0.

Closed in favor of https://github.com/dcrdata/dcrdata/pull/114


Add section for location based updates.

Fixes https://github.com/uptane/uptane-standard/issues/161.

@trishankkarthik @JustinCappos  could one of you review this for the technical content? It's just one paragraph of text.

I like RFC reference.
Let's double-check that I've understood this correctly:

imagerepo re-generates targets and includes geo://foo as location-specific updates comes in
client reports geo://bar in VVM
director checks that geo://bar is within geo://foo and updates targets metadata by adding geo://foo for client's ECU-id bumping metadata version
client follows standard verification procedure and also checks for custom geo:// metadata
once the standard verification passed, the targtes is accepted by client (new version is stored)
if the geo://foo is still matches the vehicle location, the update is installed, otherwise it's ignored

Am I interpreting this right? Could there be confusion/misunderstanding due to vehicle position change between the time VVM is reported and the time when new targets is received?
Overall, I think it's worth merging as-is already and if further clarifications would be necessary than we can make it in follow-up PRs.

Let's double-check that I've understood this correctly:

Sounds about right to me.

Could there be confusion/misunderstanding due to vehicle position change between the time VVM is reported and the time when new targets is received?

That is a good question. I hadn't considered that. I think that might be an implementation-specific question, but I could see adding some suggestions here if you think that's wise.

@patrickvacek Yes, if you could add some suggestions that @zabbal recommends, that would be great.

Thanks for all the input! I've updated the PR to include Lois's suggestion as well as some considerations about what to do if the location on the device changes as Max brought up.

@trishankatdatadog  can you take another look now?

Approved. I have just one suggestion, and then we are good to go.

I have just one suggestion, and then we are good to go.

Done. Thanks!

Ok. Then, can I merge it?

Ok. Then, can I merge it?

Feel free, Lois 🙂


conform debugging

Codecov Report

Merging #48 into master will decrease coverage by -2.89%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master      #48      +/-   ##
==========================================
- Coverage   94.84%   91.95%   -2.89%
==========================================
  Files          34       34
  Lines        1609     1666      +57
==========================================
+ Hits         1526     1532       +6
- Misses         83      134      +51

Impacted Files
Coverage Δ

Sources/Node/Utilities/Errors.swift
20% <0%> (+20%)
:white_check_mark:

Sources/Node/Setters.swift
37.5% <0%> (-62.5%)
:x:

Sources/Node/Core/Node.swift
27.27% <0%> (-2.73%)
:x:

Tests/NodeTests/NodeExtractTests.swift

Sources/Node/Extract/Node+Extract.swift

Sources/Node/Getters.swift
93.54% <0%> (ø)

Tests/NodeTests/NodeGetterTests.swift
94.07% <0%> (ø)

Sources/Node/Core/Node+Polymorphic.swift
82.24% <0%> (+0.33%)
:white_check_mark:

Sources/Node/NodeBacked.swift
97.61% <0%> (+0.39%)
:white_check_mark:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c94e556...6752843. Read the comment docs.


Payment Screen: MailGun

[ ] When click confirm send email to whichever email is inputted.
[ ] Link to mailgun api.

Delayed for later sprint


Upgrade protobuf

Fixes the following build error that I was getting:
cannot find value `VERSION_3_4_0` in crate `protobuf`

Closing because this change was included in a recent cargo upgrade commit.


Support the display of only the top N results

A command-line option like --results 3 or --top 3 could be used to only show the top 3 search results of movies and series.

Hi I'd like to work on this!

Awesome! Go ahead 😃 If you need any help just let me know

There was an oversight on my part on PR #22, I would prefer if the top value would be an Option<usize> instead of Option<u16>. I'm fixing that in PR #23, I hope you don't mind?

Yeah of course. It's used as usize in the end, so it's better to define it as usize from the beginning.


Memory Leaky

When I call the function "cv.imread" very many times, the cost of cpu memory will be greater and greater.
 local im
while 1 do
  im = cv.imread{path, loadType}
collectgarbage()
end
Who can explain it?

What do you mean by "cost of cpu memory"?

That means "the usage of memory".

In the original version of your post, there was no collectgarbage(). With that code Lua will of course load your memory because previously-read images won't be immediately destroyed.
However, with very this code there is no problem, memory usage won't increase. There are no memory leaks or bugs here.

Sorry, I must say that simple code will result in memory leaky very slowly. I will give a complicate code and usage of memory to illustrate this problem soon.

The complicate code is as following:
local cv = require 'cv'
local ffi = require 'ffi'

require 'cv.imgcodecs'
require 'cv.imgproc'
require 'cv.highgui'
collectgarbage()

local src, dst

local loadType = cv.IMREAD_UNCHANGED
local loop = 0
local k = 0
local maskPath, objectPath
local mask, object
local MAX_LOOP = 5
local MAX_ITER = 6

function process(img)
   local img = img:clone()
   while 1 do
      if loop > MAX_LOOP or loop == 0 then
         loop = 1

         maskPath = ffi.string("/home/ytoon/PycharmProjects/face/data/2007_000032_00.png")
         objectPath = ffi.string("/home/ytoon/PycharmProjects/face/data/2007_000032_00.jpg")

         mask = cv.imread{maskPath, loadType}
         object = cv.imread{objectPath, loadType}
         collectgarbage()
         collectgarbage()
      end

      if k > MAX_ITER then
         k = 0
         loop = loop + 1

         mask = cv.imread{maskPath, loadType}
         object = cv.imread{objectPath, loadType}
         collectgarbage()
         collectgarbage()
      end

      if 1 then
         local seed = math.random(0, 1)
         local fx, fy = 1.0, 1.0
         if seed == 0 then
            fx = 1.1
         else
            fy = 1.1
         end

         mask = cv.resize{src=mask, fx=fx, fy=fy, interpolation=cv.INTER_CUBIC}
         object = cv.resize{src=object, fx=fx, fy=fy, interpolation=cv.INTER_CUBIC}
         collectgarbage()
         collectgarbage()
         k = k + 1
      end
   end
end

for i = 1, 100 do
	src = cv.imread{'/home/ytoon/PycharmProjects/face/data/1.jpg', loadType}
	dst = process(src)
end

The usage of memory is as following:
beginning:
usage of memory is 1.6% (8G)
after about 30 minutes:
usage of memory is 9%(8G)

The complicate code is as following:
local cv = require 'cv'
local ffi = require 'ffi'

require 'cv.imgcodecs'
require 'cv.imgproc'
require 'cv.highgui'
collectgarbage()

local src, dst

local loadType = cv.IMREAD_UNCHANGED
local loop = 0
local k = 0
local maskPath, objectPath
local mask, object
local MAX_LOOP = 5
local MAX_ITER = 6

function process(img)
   local img = img:clone()
   while 1 do
      if loop > MAX_LOOP or loop == 0 then
         loop = 1

         maskPath = ffi.string("/home/ytoon/PycharmProjects/face/data/2007_000032_00.png")
         objectPath = ffi.string("/home/ytoon/PycharmProjects/face/data/2007_000032_00.jpg")

         mask = cv.imread{maskPath, loadType}
         object = cv.imread{objectPath, loadType}
         collectgarbage()
         collectgarbage()
      end

      if k > MAX_ITER then
         k = 0
         loop = loop + 1

         mask = cv.imread{maskPath, loadType}
         object = cv.imread{objectPath, loadType}
         collectgarbage()
         collectgarbage()
      end

      if 1 then
         local seed = math.random(0, 1)
         local fx, fy = 1.0, 1.0
         if seed == 0 then
            fx = 1.1
         else
            fy = 1.1
         end

         mask = cv.resize{src=mask, fx=fx, fy=fy, interpolation=cv.INTER_CUBIC}
         object = cv.resize{src=object, fx=fx, fy=fy, interpolation=cv.INTER_CUBIC}
         collectgarbage()
         collectgarbage()
         k = k + 1
      end
   end
end

for i = 1, 100 do
	src = cv.imread{'/home/ytoon/PycharmProjects/face/data/1.jpg', loadType}
	dst = process(src)
end

The usage of memory is as following:
beginning:
usage of memory is 1.6% (8G)
after about 30 minutes:
usage of memory is 9%(8G)

You scale up a picture in an infinite loop, so it gets bigger and bigger. No wonder it eventually needs several hundreds of megabytes. Nothing to do with memory leaks.
Please, when trying to demonstrate a bug, create a MCVE so that others don't waste time puzzling out dozens of irrelevant code lines.

The usage of memory doesn't scale very much because the code will re-load the same image after some iterations.

I let this code run for 30 minutes and saw no increase in memory usage. There is just an infinite loop that resizes the image and reloads it. i is always 1. Same as your example in post head.

Thank you. I will test it again.

Could you tell me the setting environments of your computer? @shrubb

I reinstalled the torch, opencv-3.1 and torch-opencv. Besides, I simplified the test code.
The code is as following:
local cv = require 'cv'
local ffi = require 'ffi'

require 'cv.imgcodecs'
require 'cv.imgproc'
require 'cv.highgui'
collectgarbage()

local loadType = cv.IMREAD_UNCHANGED
local maskPath, objectPath
local mask, object

function process()
   while 1 do
      maskPath = ffi.string("/home/ytoon/PycharmProjects/face/data/2007_000032_00.png")
      objectPath = ffi.string("/home/ytoon/PycharmProjects/face/data/2007_000032_00.jpg")
      mask = cv.imread{maskPath, loadType}
      object = cv.imread{objectPath, loadType}

      collectgarbage()
      collectgarbage()

      mask = cv.resize{src=mask, fx=fx, fy=fy, interpolation=cv.INTER_CUBIC}
      object = cv.resize{src=object, fx=fx, fy=fy, interpolation=cv.INTER_CUBIC}
   end
end

process()

The image wasn't still increase very large because the code re-load the same image each iteration.
I also replaced the functions of torch-opencv with the respondent functions of torch-image.  There was no memory leaky.
Computer Settings
Ubuntu 14.04
OpenCV 3.1
torch(latest version)
torch-opencv(latest version)

Hmm, now it's better. Turns out I should've taken a very very small image to reproduce the bug. I'm investigating the issue...

Cleared with #179


GH1173: Adds integration tests for Cake.Common.Tools.Cake.CakeAliases

releates to #1173

@devlead LGTM!


AST rewrite

3x performance improvement in tests against dockyard.com
Tasks

[x] Replace binaries with AST
[x] Pre-compile template AST and store in ETS
[x] Store routes in ETS - paths and metadata (page_id, layout_id, and modules)
[x] Lookup ETS routes by path (including paths with dynamic segments)
[x] Render templates dynamically from AST stored in ETS
[x] @beacon_path_params
[x] Propagate updates
[x] Double check docs and generator (new apps)

I've got to fix the test suite, and there are some additonal performance optimizations that can be gotten by reducing the number of functions generated.

🎉


backend: fix for loop iteration in ssl_port_status function

Not sure if this is the best/correct way to fix the issue I was experiencing. Essentially the nested for loops had trouble iterating over the manifest variable. The below code changes did fix my build.

What version of rust are you running?

I was just using the build process here: https://github.com/Start9Labs/embassy-os/tree/master/build#readme
I don't have rust installed on my Ubuntu 20.04 Desktop.

I ended up going with a different approach in #1202 due to the need to do an interactive rebase. But thank you for your contribution regardless!


[Review Only] Internal PAL improvements

Prototyping some internal PAL improvements to making supporting new platforms easier.

I can add generic "No-Op" implementations that can be used to build unsupported platforms.
Yes, non-open source platforms will provide their implementations of PlatformInitialize, IHttpProvider, and IWebSocketProvider in separate repos which will get built and linked in to the lhc lib.

In reply to:<PHONE_NUMBER>


New class for structs and consts

(instead of 0.08 everywhere)

Put the graphics scale and the enums in actor into this class.

Also include vector conversion methods (sfml <--> box2d)


Implement rust-notifica instead of rust-notify

Why?
Since we experience problems with rust-notify we should maybe look for an alternative, I present rust-notifica. It is cross-platform and maintained. If we don't have problems then feel free to delete this Issue.
Ressources
https://github.com/frewsxcv/rust-notifica

Great library! We might fork it or use it directly if it supports images


Update bifrost.memory's understanding of memory spaces

This PR updates the "memset" and "memcpy" functions in bifrost.memory to work with ndarrays that live in non-system accessible spaces.

These test failures might be related to #220.

It looks like these failures (and some of those part of the testing for https://github.com/ledatelescope/bifrost/pull/166) are related to missing file(s) needed by the extended testbench tests.  Maybe we should drop those tests?


Check for scalar cubes before attempting to concatenate a CubeList

📰 Custom Issue

Hi guys, a variation on the theme of concatenation/merge on which there are a few issues already open - it'd be nice if iris handled scalar cubes as big bad wolves and looked for them in, say, a CubeList to be concatenated/merged, and raise a nice exception that identifies that (or those) scalar cubes - I recently found out that scalar cubes produce havoc when one tries to concatenate - raised (inadvertently) by @bouweandela in this comment - it all happens in _io._concatenate_overlapping_cubes(cubes) - after extract_time() (one of our utilities) the resulting cube may be scalar - so iris concatenation fails due to scalar cubes not having dim_metadata attributes in iris._concatenate_CubeSignature class (they do, but they are empty lists).
Cheers muchos :beer:

#3234 seems relevant

cheers @pp-mo :beer: Indeed, let's add it to the workshop!

@pp-mo I have created a discussion point on the ESMValTool discussions pages that can hold a list of all these types of issues to be discussed at the call on next Monday, and the wkshp, if you got a bit of time it'd be nice to populate that list with what you reckon could be discussed - or, label these issues with some specific label so I can pull them over there. Cheers muchly as ever :beer:

... label these issues with some specific label so I can pull them over there.

We do have the "Feature: ESMValTool" label.
I just added that one to a few more existing Iris tickets.

We do have the "Feature: ESMValTool" label.

@bjlittle also reminds me -- we also have a Scitools project, though TBH we haven't paid much attention to it lately.

We discussed this issue a bit in the ESMValTool community, and the request is mostly about a nicer error message. @valeriupredoi Could you provide an example of the error message and explain why it is not clear and how you think it could be improved?


Folia support (timed remove AdvancementAPI from Folia)

About AdvancementAPI

I'm gonna wait for HuskHomes first.

@mani1232 Not sure if you're interested -- but If you'd like to rework this based on the work I did for HuskHomes, that'd be fantastic https://github.com/WiIIiam278/HuskHomes2/pull/376


Make the artifacts volume size configurable

Currently the artifacts volume size is hardcoded to 1Gi https://github.com/kubeshop/helm-charts/blob/develop/charts/testkube-api/pvc-template.yml.tmpl#L16
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: {{ .Name }}-pvc
  namespace: {{ .Namespace }}
spec:
  storageClassName: {{ .ArtifactRequest.StorageClassName }}
  accessModes:
  {{- if .ArtifactRequest.SharedBetweenPods }}
    - ReadWriteMany
  {{- else }}
    - ReadWriteOnce
  {{- end }}
  resources:
    requests:
      storage: 1Gi

hey, @lsoica you can adjust thsi template in your Test CRD, in executionRequest we have pvcTemplate field, that works similar to jobTemplate http://docs.testkube.io/articles/creating-tests#changing-the-default-job-template-used-for-test-execution

if it's not clear, please reopen


feat: improve robustness of subject user handling

closes #1516

@bmulholland - this should handle that super edge case you had this week in the event that a github account is deleted


Support configuring and disabling healthcheck extension

The healthcheck probes are configured like this, without customization options.
    livenessProbe:
      httpGet:
        path: /
        port: 13133
    readinessProbe:
      httpGet:
        path: /
        port: 13133

I would like the following options

ability to change path, port
ability to set other options like periodSeconds
ability to disable healthcheck extension all together.

While I agree that generally healthcheck extension + healthcheck probes should always be in use, I do think this should be up to the user. I think defaulting to the current configuration (enabled with sane defaults) is great.
I would be willing to take on this work if the community agrees.

@jsirianni this topic has come up before, here is the related issue for context: https://github.com/open-telemetry/opentelemetry-helm-charts/issues/242

Do you have a use case in mind that requires more configurability to the readiness and liveness sections?

Thanks for linking that. I am not sure I agree with the statement that health checks are mandatory in k8s, but I do agree that they are highly recommended. My argument has already been discussed there by the OP, so I can concede. It does feel "forced" but I understand the reasoning.
I do not have a concrete usecase, but I can imagine wanting to dial in probe settings for a service that might be scaling up and down frequently.  To me, this was less important than having the ability to disable the checks.
If the stanza to absolutely always require the healthcheck extension, feel free to close this issue.

I'd like to see the ability to extend the health check timeout as if you have a big configuration with a lot of processors it can take a while for the collector to report back healthy causing issues causing restarts again.

Being able to configure ports is also acceptable.

Hi, would it be possible to also add option to disable readiness / liveness probe? At times I need to debug the otel collector using the delve debugger and I usually ran into the pod I am attached to getting killed by the liveness probe (when you pause the process, the liveness probe is not responding), forcing me to copy the helm chart and remove the liveness probe.


Update MD file on new branch 2017-12-03_01-39-39

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

split/updateTestRef1.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Apps - Should be installable in the same way as plugins - static-apps like static-plugins and composer repository from story

Please describe the feature you would like to see implemented.
Currently all apps land in the custom/apps folder.
However, due to automated deployments, it should be possible to install them using:

composer store repository
using a static folder that is untouched by the extension manager.

That way e.g. a developer can pre-install necessary apps during deployment but the end-user is still able to manually install additional apps without those two interferring.
This should be easily doable by automatically creating a composer.json for apps by the repository. Except the target folder.
It is also possible to add static apps the same way as plugins that way. So a developer could create own apps and provide them during deployments easily.
Furthermore, not sure if this has already been fixed or not, but in <IP_ADDRESS> it was still the case: If an app folder is deleted, the app is not recognized as removed which can have weird side effects.
Especially,` in combination with the app refresh command not correctly detecting all existing plugins but uninstalling already installed plugins and adding only those that were not installed before - might have been fixed though?

Hello,
Within our company roadmap and work capacity, we try to address each bug or improvement request but admit that not each one will be resolved. To continue our culture of honesty and openness, we are closing this issue to focus on our roadmap on behalf of all Shopware users.
If you feel like this issue is still impacting you, please create a new issue and let us know.
Thank you again for your valuable feedback.


Add more recipes

I have my own recipe database, created from my own prodding at the API. I've also gathered recipes from various sources (result was always verified with an API call).
recipes.zip
Something I've observed while gathering recipes is that, on rare occasions, the recipes can change. I've seen a few recipes that used to work, but now result in "Nothing" (likely intentional censoring), and 1 case of a recipe just getting a completely different result.

Thanks for your contribution, your dataset has been merged and pushed to https://expitau.github.io/InfiniteCraftWiki/. You've been acknowledged as a contributor in the README


compiler: implement short-circuiting with complex conditions

See #699, #700 .
We need to handle bool expressions of any depth in any context (assignment, if, for, switch etc.).

Closed via #700 . (yes, the name of the PR is a bit confusing).


Is your feature request related to a problem? Please describe.
Yes , adding a preloader will increase the user attention to the site
Describe the solution you'd like
I would like to a animation that will render until the page is being loaded
Alternatives
The page would be completely responsive
Additional Context
Kindly assign this task to me . I am familiar with adding animations to the site and quite experienced too.
Are you interested in contributing?

[X] I'd like to work on this issue

Code of Conduct

[X] I have used the search function to check if an issue already exists
[X] I agree to follow this project's Code of Conduct

Which program are you contributing in?
ssoc

#986


Add bjohansebas as repo captain for expressjs.com

I also fixed a small typo in the Contributing Guide and made a minor formatting change.

I will grant you permissions soon @bjohansebas


[YUNIKORN-2582]Update atomic.Int32 for atomic operations

What is this PR for?
Type atomic.Int32 was introduced in go 1.19, we should adopt it for better code clarity.
Below variable type should be changed:

asyncDispatchCount in dispatcher.go
registerCount/UpdateAllocationCount/UpdateApplicationCount/UpdateNodeCount in schedulerapi_mock.go

What type of PR is it?

[ ] - Bug Fix
[x] - Improvement
[ ] - Feature
[ ] - Documentation
[ ] - Hot Fix
[ ] - Refactoring

Todos

[ ] - Task

What is the Jira issue?

https://issues.apache.org/jira/browse/YUNIKORN-2582

How should this be tested?
All unit tests should be passed.
Screenshots (if appropriate)
Questions:

[ ] - The licenses files need update.
[ ] - There is breaking changes for older versions.
[ ] - It needs documentation.

Assuming PR will be updated with the other locations? (schedulerapi_mock.go)

Assuming PR will be updated with the other locations? (schedulerapi_mock.go)

Hahaha. You are right. The second is coming~


Changing screenProps does not trigger a re-render

I've got a single instance of a TabNavigator, which I'm using to pass through various props via screenProps. When the page updates state, I use the same instance of the navigator and send the new props through. The screens though do not update.
Pseudo code:

constructor(props) {
  super(props);
  this.navigator = TabNavigator(props.pages);
}

render() {
  return (
    <this.navigator
       screenProps={{
         data: this.state.data,
       }}
    />
  );
}

When data updates in state, my pages/screens do not get the new data. Is this expected?
The reason for doing it this way, is that currently when state updates and I return a new TabNavigator, the page resets back to the first page. I was hoping this was a way around it.
EDIT: Using beta6

Related https://github.com/react-community/react-navigation/issues/577

Doing some digging around, it seems as though TabView wraps withCachedChildNavigation, which is a PureComponent. If screenProps contains a nested object it won't update.

Closing as it's a duplicate of #577


Update concept-endpoints.md

typo fixed manged to managed

@JaredMSFT : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off

@PeterCLu: I'm sorry - only the author of this article, @rsethur, can sign off on your changes. But we do have an exception process - if you are on the Microsoft content or product team for this product area, you can ask the PR review team to review and merge it by sending mail to the techdocprs alias.


ENS compatibility

For #810
Allows you to write/paste any valid ENS name into the box and it will be translated into the corresponding address

@hboon ok.
Actually, it is better to simply check for "." as any text input that has a dot in it can be implied to be an ENS name, it is also compatible with subdomains. User is protected because an invalid name will still return a null value.

@hboon 2 is not an issue, it is able to detect the ens name if you use the black and white paste button

@James-Sangalli I was referring to this white on black Paste button. Does it trigger ENS resolution?

Another future enhancement when we get the Ethereum name and address to show up together: handle when user types in an ENS name, maybe start resolving say, 0.5 seconds after user stops typing.


[Bug]: The "sorted(gpu_ids)" operation in ray_gpu_executor.py causes an incorrect order of GPU IDs When using the NVIDIA HGX A100 (16-GPU) platform for model inference.

Your current environment
Unable to obtain environmental information at the moment.
🐛 Describe the bug
In the code vllm/executor/ray_gpu_executor.py:line 142, if the number of GPUs on a node exceeds 10 (such as NVIDIA HGX A100 with 16-GPU), the result of sorted(gpu_ids) would be 0,10,11,12,13,14,15,2,3,4,5,6,7,8,9, instead of 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15. This results in an NCCL Error, because the order of GPUs in the Ray Executor (lexicographical order) is inconsistent with the order of GPUs in NCCL (actual numerical order).
The correct way should be
node_gpus[node_id] = sorted(gpu_ids, key=lambda x: int(x))

@JiantaoXu thanks for the information. Can you try if #5604 solves this issue? I don't have 16GPU machine to test it.


Support custom signing implementations in Metadata.sign method

Description of issue or feature request:
Allow TUF integrators to easily use custom metadata signing implementations instead of the currently supported securesystemslib one.
An urgent use case for this is the PEP458/Warehouse integration, where metadata must be signed with keys stored in a Hashicorp Vault and the implementation is already available.
NOTE: This feature request only concerns the tuf.api.Metadata.sign method and not the legacy signing facilities write and writeall in repository tool.
Current behavior:
tuf.api.Metadata.sign(key, ...) calls securesystemslib.keys.create_signature with the passed sslib/json-formatted private key.
Expected behavior:
tuf.api.Metadata.sign takes an abstract signer parameter, which implements signing. A default signer may resolve to the currently used  securesystemslib.keys.create_signature.
Implementation considerations:

Different signers require completely different "keys", e.g. the currently used generic implementation requires an sslib/json-formatted private key, whereas signing implementations that don't have direct access to the key may only need a keyid (see GPG or HSM references below). As a consequence the key parameter in Metadata.sign should probably be encapsulated within the signer.

The abstract signer interface must strictly define the format of the returned signature for metadata interoperability.

The abstract signer interface design should also keep in mind other key-related tasks, such as signature verification, public key export to sslib metadata format, and keyid generation.

securesystemslib.keys.create_signature is already an abstract interface, where the json-formatted private key parameter format is generic enough to hold different key types and to choose one of many supported signing algorithms, identified by the key's scheme field. (see public key metadata format reference below). However, it is not suited to extend to custom implementations. A new abstract signer interface should integrate well with the existing securesystemslib infrastructure and/or aim to replace it. It should not become yet another co-existing interface and/or layer of abstraction (see public API overview reference below).

Related work and references: (for the very ambitious reader)

securesystemslib key overview  (to coordinate this feature request with the existing infrastructure)

general public API overview -- sslib#270
generic signing interface currently used by TUF -- sslib.keys.create_signature
GPG signing interface -- sslib.gpg.functions.create_signature
WIP: HSM signing interface -- sslib.hsm.create_signature
general public key metadata format -- sslib#308
in-memory key representation for generic signing interface -- sslib#310
stand-alone on-disk key format for generic signing interface -- sslib#309

Recently added abstract filesystem interface (for design inspiration) -- https://github.com/theupdateframework/tuf/issues/1009

@trishankatdatadog's Hashicorp Vault interface POC (as reference for PEP458/warehouse implementation and for design inspiration) -- VaultKey

Abstract pyca/cryptograhpy key interface with sign method (for design inspiration) -- RSAPrivateKey

"Historical" external signing API feature request (has some interesting discussion and designs but targets legacy TUF codebase) -- https://github.com/theupdateframework/tuf/issues/864

in-toto metadata model sign methods (alternative but less flexible approach to support different signing implementations)

implementation -- Metadata.sign*
related design discussion (replace serialization/deserialization with sign/verify!) -- sslib#272

A very basic implementation could look something like this:
# in securesystemslib
class Signer:
    @abc.abstractmethod
    def sign(bytes: payload) -> Signature:
        raise NotImplementedError

class SSlibSigner(Signer):
    def __init__(self, sslib_private_key):
        self.key = sslib_private_key

    def sign(self, payload):
        return sslib_keys.create_signature(self.key, payload)

class GPGSigner(Signer):
    def __init__(self, gpg_keyid):
        self.keyid = gpg_keyid

    def sign(self, payload):
        return sslib_gpg.create_signature(payload, self.keyid)

# in TUF
class Metadata:
    def sign(sslib.Signer: signer):
        signer.sign(self.signed.to_canonical_bytes())

# in Warehouse
class VaultSigner(Signer):
    def sign(self, payload):
        # ... you get the idea

metadata = tuf.Metadata(...)
metadata.sign(VaultSigner(...))

A very basic implementation could look something like this:
# in securesystemslib
class Signer:
    @abc.abstractmethod
    def sign(bytes: payload) -> Signature:
        raise NotImplementedError

class SSlibSigner(Signer):
    def __init__(self, sslib_private_key):
        self.key = sslib_private_key

    def sign(self, payload):
        return sslib_keys.create_signature(self.key, payload)

class GPGSigner(Signer):
    def __init__(self, gpg_keyid):
        self.keyid = gpg_keyid

    def sign(self, payload):
        return sslib_gpg.create_signature(payload, self.keyid)

# in TUF
class Metadata:
    def sign(sslib.Signer: signer):
        signer.sign(self.signed.to_canonical_bytes())

# in Warehouse
class VaultSigner(Signer):
    def sign(self, payload):
        # ... you get the idea

metadata = tuf.Metadata(...)
metadata.sign(VaultSigner(...))

class VaultSigner(Signer):

Great idea. I would add only that it might be useful to add in SSLib itself...

class VaultSigner(Signer):

Great idea. I would add only that it might be useful to add in SSLib itself...

I would add only that it might be useful to add in SSLib itself...

Agreed.

I would add only that it might be useful to add in SSLib itself...

Agreed.

Thank you for the detailed issue @lukpueh.
Bit of an aside, but do you imagine the GPG and HSM signing interfaces in securesystemslib becoming implementations of the interface we create here? That seems logical to me, but I'm not familiar with the GPG and HSM signing interfaces.

Thank you for the detailed issue @lukpueh.
Bit of an aside, but do you imagine the GPG and HSM signing interfaces in securesystemslib becoming implementations of the interface we create here? That seems logical to me, but I'm not familiar with the GPG and HSM signing interfaces.

Bit of an aside, but do you imagine the GPG and HSM signing interfaces in securesystemslib becoming implementations of the interface we create here?

I would suggest so, yes. Do you have reservations?

Bit of an aside, but do you imagine the GPG and HSM signing interfaces in securesystemslib becoming implementations of the interface we create here?

I would suggest so, yes. Do you have reservations?

No reservations, just checking whether my expectation was reasonable. Thank you.

No reservations, just checking whether my expectation was reasonable. Thank you.

I'd like to add some ontological thoughts here. In particular, whether the Signer class could instead be a SigningKey class, given that a Signer, as described above, is associated with exactly one key (or key identifier) anyway.
What speaks against combining Signer and Key in one class:
In pure modeling terms a key data container and an entity that implements a signing protocol are quite different things. Also, a private key might be used with different signers (protocols, algorithms, schemes, etc.), which would require duplicating key data on multiple signers.
What speaks for it:
securesystemslib needs private keys only for their signing capabilities and it is unlikely that any given private key will be used with different signers during its in-memory life time.
Outlook
Maybe we should at least tentatively explore the expected use cases for private keys in a new repository library (#1136).
I have already summarized the main tasks for keys (private and public!) in https://github.com/secure-systems-lab/securesystemslib/issues/310, albeit with a focus on public keys, which need to implement serialization to and deserialization from TUF metadata format (https://github.com/secure-systems-lab/securesystemslib/issues/308) and verification. Both may or may not be tied to the protocol that also implements the signing.
@trishankatdatadog, maybe you can share your experience from implementing tuf-on-a-plane?

I'd like to add some ontological thoughts here. In particular, whether the Signer class could instead be a SigningKey class, given that a Signer, as described above, is associated with exactly one key (or key identifier) anyway.
What speaks against combining Signer and Key in one class:
In pure modeling terms a key data container and an entity that implements a signing protocol are quite different things. Also, a private key might be used with different signers (protocols, algorithms, schemes, etc.), which would require duplicating key data on multiple signers.
What speaks for it:
securesystemslib needs private keys only for their signing capabilities and it is unlikely that any given private key will be used with different signers during its in-memory life time.
Outlook
Maybe we should at least tentatively explore the expected use cases for private keys in a new repository library (#1136).
I have already summarized the main tasks for keys (private and public!) in https://github.com/secure-systems-lab/securesystemslib/issues/310, albeit with a focus on public keys, which need to implement serialization to and deserialization from TUF metadata format (https://github.com/secure-systems-lab/securesystemslib/issues/308) and verification. Both may or may not be tied to the protocol that also implements the signing.
@trishankatdatadog, maybe you can share your experience from implementing tuf-on-a-plane?

What speaks against combining Signer and Key in one class:
In pure modeling terms a key data container and an entity that implements a signing protocol are quite different things. Also, a private key might be used with different signers (protocols, algorithms, schemes, etc.), which would require duplicating key data on multiple signers.

I am asking myself does the Key class has meaning outside the signing operations? If not, then probably it makes sense to keep it there?
Also, I am not sure if there is a modeling contradiction. SignerKey will be a container class for key data and at the same time that class would have functionality only related to that data, so they are already in a sense strongly connected.
At the same time, if we have a SignerKey for the signing operations, then how we are going to do the verification operations?
There is no sense to have a separate VerifierKey class.
Maybe we can look the other way around and have a Key class with signature and verification functionality.

What speaks against combining Signer and Key in one class:
In pure modeling terms a key data container and an entity that implements a signing protocol are quite different things. Also, a private key might be used with different signers (protocols, algorithms, schemes, etc.), which would require duplicating key data on multiple signers.

I am asking myself does the Key class has meaning outside the signing operations? If not, then probably it makes sense to keep it there?
Also, I am not sure if there is a modeling contradiction. SignerKey will be a container class for key data and at the same time that class would have functionality only related to that data, so they are already in a sense strongly connected.
At the same time, if we have a SignerKey for the signing operations, then how we are going to do the verification operations?
There is no sense to have a separate VerifierKey class.
Maybe we can look the other way around and have a Key class with signature and verification functionality.

@trishankatdatadog, maybe you can share your experience from implementing tuf-on-a-plane?

It's a good question. I didn't implement any signing interface, only verification interfaces. But, if it helps, what I found is that rather than designing on pen and paper, writing, trying, and feeling the code worked out very well. So try writing what comes naturally, and see if it makes sense? I know I sound like a mystic, but that's what worked for me.

@trishankatdatadog, maybe you can share your experience from implementing tuf-on-a-plane?

It's a good question. I didn't implement any signing interface, only verification interfaces. But, if it helps, what I found is that rather than designing on pen and paper, writing, trying, and feeling the code worked out very well. So try writing what comes naturally, and see if it makes sense? I know I sound like a mystic, but that's what worked for me.

Thanks for your comments, @MVrachev and @trishankatdatadog!

SignerKey will be a container class for key data and at the same time that class would have functionality only related to that data, so they are already in a sense strongly connected.

Yes they are, but there are some variables that change for the signer but not for the key, such as padding, hashing, scheme, etc. To me it feels a bit odd to modify the key object, in order to use a different signing scheme, e.g.:
k = SigningKey(...) # Initialize once e.g. with RSA secret exponent
k.scheme = "rsassa-pss-sha256"
k.sign(data) # Sign using one scheme

k.scheme = "rsa-pkcs1v15-sha512"
k.sign(data) # Sign using another scheme

But maybe switching schemes during life time is just not a use case, and even then, the odd looks alone don't seem to be a strong argument against it.
An alternative would be to accept these variables as a parameters to SigningKey.sign. But given that our Metadata.sign will call the sign method of whichever passed SigningKey subclass, the interface needs to be generic for all key types and as simple as possible.

Maybe we can look the other way around and have a Key class with signature and verification functionality?

Yes, but I think there is an argument for separating public and private key in the model. If they were combined, we'd often have half-empty objects, e.g. on the client where only the public parts are available. And even in the signing context we only really need the private portion.
Furthermore, given that public keys are included in TUF metadata their class representation should look similar to the metadata representation for recognizability and have a focus on type/schema checking and serialization tasks. Whereas we care a lot less about what the private key class looks like as long as it can be used for signing.
Does this make sense? I have a feeling that hands-on @trishankatdatadog might say, I'm overthinking this. :P

Thanks for your comments, @MVrachev and @trishankatdatadog!

SignerKey will be a container class for key data and at the same time that class would have functionality only related to that data, so they are already in a sense strongly connected.

Yes they are, but there are some variables that change for the signer but not for the key, such as padding, hashing, scheme, etc. To me it feels a bit odd to modify the key object, in order to use a different signing scheme, e.g.:
k = SigningKey(...) # Initialize once e.g. with RSA secret exponent
k.scheme = "rsassa-pss-sha256"
k.sign(data) # Sign using one scheme

k.scheme = "rsa-pkcs1v15-sha512"
k.sign(data) # Sign using another scheme

But maybe switching schemes during life time is just not a use case, and even then, the odd looks alone don't seem to be a strong argument against it.
An alternative would be to accept these variables as a parameters to SigningKey.sign. But given that our Metadata.sign will call the sign method of whichever passed SigningKey subclass, the interface needs to be generic for all key types and as simple as possible.

Maybe we can look the other way around and have a Key class with signature and verification functionality?

Yes, but I think there is an argument for separating public and private key in the model. If they were combined, we'd often have half-empty objects, e.g. on the client where only the public parts are available. And even in the signing context we only really need the private portion.
Furthermore, given that public keys are included in TUF metadata their class representation should look similar to the metadata representation for recognizability and have a focus on type/schema checking and serialization tasks. Whereas we care a lot less about what the private key class looks like as long as it can be used for signing.
Does this make sense? I have a feeling that hands-on @trishankatdatadog might say, I'm overthinking this. :P

Does this make sense? I have a feeling that hands-on @trishankatdatadog might say, I'm overthinking this. :P

This requires some meditation TBH. There are generic cryptographic algorithms and then there are specific schemes. What code do we expect delegators and delegatees to write? I would approach it that way.

Does this make sense? I have a feeling that hands-on @trishankatdatadog might say, I'm overthinking this. :P

This requires some meditation TBH. There are generic cryptographic algorithms and then there are specific schemes. What code do we expect delegators and delegatees to write? I would approach it that way.

Does this make sense? I have a feeling that hands-on @trishankatdatadog might say, I'm overthinking this. :P

It makes sense and those are valid points if you ask me.
After all of your thoughts, I feel like the best option is to have a Key class in tuf/metadata/api representing exactly what is written in the spec and a different Signer class which contains a Key class instance, basically the way you initially proposed it.
That way you have a model literally representing a key the way it's defined in the spec, you can easily change the key schema, won't have the problem with half-empty objects, and have a logical encapsulation for each of the classes.
Additional thinks we discussed with @jku:

The interface @lukpueh proposed is fairly clear and easy to implement.
The important question in the interface seems to be what should Signature be or the return type of the sign() operation?

Jussi looked into what is the returned value of all possible implementors of the Signer interface:

Default implementation :  return value of keys.create_signature() is a dict of keyid and bytes
Gpg: Return value is also a dict: it is used to return multiple other pieces of data as well -- it is unclear if these are actually useful to create_signature() callers or only for intermediate processing before that
HSM: Returns a dict like the generic keys.create_signature()
Vault: Vault signature is vault:<key-version-number>:<base64-encoded-signature> -- should be usable as bytes
https://www.vaultproject.io/api/secret/transit#sign-data
https://hvac.readthedocs.io/en/stable/usage/secrets_engines/transit.html#sign-data

So, to summarize it seems achievable to return bytes instead of a custom Signature object which is unclear what should it be.

Does this make sense? I have a feeling that hands-on @trishankatdatadog might say, I'm overthinking this. :P

It makes sense and those are valid points if you ask me.
After all of your thoughts, I feel like the best option is to have a Key class in tuf/metadata/api representing exactly what is written in the spec and a different Signer class which contains a Key class instance, basically the way you initially proposed it.
That way you have a model literally representing a key the way it's defined in the spec, you can easily change the key schema, won't have the problem with half-empty objects, and have a logical encapsulation for each of the classes.
Additional thinks we discussed with @jku:

The interface @lukpueh proposed is fairly clear and easy to implement.
The important question in the interface seems to be what should Signature be or the return type of the sign() operation?

Jussi looked into what is the returned value of all possible implementors of the Signer interface:

Default implementation :  return value of keys.create_signature() is a dict of keyid and bytes
Gpg: Return value is also a dict: it is used to return multiple other pieces of data as well -- it is unclear if these are actually useful to create_signature() callers or only for intermediate processing before that
HSM: Returns a dict like the generic keys.create_signature()
Vault: Vault signature is vault:<key-version-number>:<base64-encoded-signature> -- should be usable as bytes
https://www.vaultproject.io/api/secret/transit#sign-data
https://hvac.readthedocs.io/en/stable/usage/secrets_engines/transit.html#sign-data

So, to summarize it seems achievable to return bytes instead of a custom Signature object which is unclear what should it be.

So, to summarize it seems achievable to return bytes instead of a custom Signature object which is unclear what should it be.

That does make sense. But then every SignerKey must also expose a keyid. Because the caller is likely to store the signature bytes together with a keyid, otherwise the signature can't be mapped to a public key for later verification.
class Metadata:
  ...
  def sign(self, signer):
    # NOTE: Let's assume signatures are still in the old/current dictionary format
    self.signatures.append({
        "keyid": signer.keyid,
        "sig": signer.sign(self.signed.to_canonical_bytes())
      })

I think your proposal makes the SignerKey.sign interface a bit cleaner, but the overall SignerKey class a bit less blackboxy.

Gpg: Return value is also a dict: it is used to return multiple other pieces of data as well -- it is unclear if these are actually useful to create_signature() callers or only for intermediate processing before that

Yeah, unfortunately these are required for verification. Although it might be possible to return them as part of the signature bytes and then parse upon verification.

So, to summarize it seems achievable to return bytes instead of a custom Signature object which is unclear what should it be.

That does make sense. But then every SignerKey must also expose a keyid. Because the caller is likely to store the signature bytes together with a keyid, otherwise the signature can't be mapped to a public key for later verification.
class Metadata:
  ...
  def sign(self, signer):
    # NOTE: Let's assume signatures are still in the old/current dictionary format
    self.signatures.append({
        "keyid": signer.keyid,
        "sig": signer.sign(self.signed.to_canonical_bytes())
      })

I think your proposal makes the SignerKey.sign interface a bit cleaner, but the overall SignerKey class a bit less blackboxy.

Gpg: Return value is also a dict: it is used to return multiple other pieces of data as well -- it is unclear if these are actually useful to create_signature() callers or only for intermediate processing before that

Yeah, unfortunately these are required for verification. Although it might be possible to return them as part of the signature bytes and then parse upon verification.

...custom fields like the one on gpg signatures might also be an argument for SigningKey.sign returning a complex signature object.

...custom fields like the one on gpg signatures might also be an argument for SigningKey.sign returning a complex signature object.

It makes sense and those are valid points if you ask me.
After all of your thoughts, I feel like the best option is to have a Key class in tuf/metadata/api representing exactly what is written in the spec and a different Signer class which contains a Key class instance, basically the way you initially proposed it.
That way you have a model literally representing a key the way it's defined in the spec, you can easily change the key schema, won't have the problem with half-empty objects, and have a logical encapsulation for each of the classes.

It seems I misunderstand @lukpueh. Lukas seems to mean to combine Signer and the private part from the key.
I am only not sure about the SignerKey name. It feels a strange concatenation of the functionality it provides - signing and the data it stores - key data. Probably it makes sense just to call it Signer as it was originally proposed and it will have attributes representing key data.

That does make sense. But then every SignerKey must also expose a keyid. Because the caller is likely to store the signature bytes together with a keyid, otherwise the signature can't be mapped to a public key for later verification.

You have a point here. We can indeed create a Signature class and store there both the bytes signature and keyid. Then we would return an instance of it when calling sign().
We can also use this Signature to add functionality connected to the signature.
Like verification operations?

It makes sense and those are valid points if you ask me.
After all of your thoughts, I feel like the best option is to have a Key class in tuf/metadata/api representing exactly what is written in the spec and a different Signer class which contains a Key class instance, basically the way you initially proposed it.
That way you have a model literally representing a key the way it's defined in the spec, you can easily change the key schema, won't have the problem with half-empty objects, and have a logical encapsulation for each of the classes.

It seems I misunderstand @lukpueh. Lukas seems to mean to combine Signer and the private part from the key.
I am only not sure about the SignerKey name. It feels a strange concatenation of the functionality it provides - signing and the data it stores - key data. Probably it makes sense just to call it Signer as it was originally proposed and it will have attributes representing key data.

That does make sense. But then every SignerKey must also expose a keyid. Because the caller is likely to store the signature bytes together with a keyid, otherwise the signature can't be mapped to a public key for later verification.

You have a point here. We can indeed create a Signature class and store there both the bytes signature and keyid. Then we would return an instance of it when calling sign().
We can also use this Signature to add functionality connected to the signature.
Like verification operations?

Probably it makes sense just to call it Signer as it was originally proposed and it will have attributes representing key data.

I'm fine with Signer. :)

We can indeed create a Signature class and store there both the bytes signature and keyid. Then we would return an instance of it when calling sign().

SGTM! :)

We can also use this Signature to add functionality connected to the signature. Like verification operations?

I'm a bit unsure about that. I think in most cases sign can just return a generic Signature with a keyid and some raw signature bytes, independently of the key type, scheme, etc. used to create the signature. If we implement verify on the Signature class it has to account for all those variables, or, more likely, we'd use different Signature subclasses for different verify implementations.
As a consequence a concrete Signer  subclass implementation must know which concrete Signature subclass to return, in order to provide the correct verify method. I don't know if we want to put that extra responsibility on the Signer. To me, this feels more like a task for the public key.

Probably it makes sense just to call it Signer as it was originally proposed and it will have attributes representing key data.

I'm fine with Signer. :)

We can indeed create a Signature class and store there both the bytes signature and keyid. Then we would return an instance of it when calling sign().

SGTM! :)

We can also use this Signature to add functionality connected to the signature. Like verification operations?

I'm a bit unsure about that. I think in most cases sign can just return a generic Signature with a keyid and some raw signature bytes, independently of the key type, scheme, etc. used to create the signature. If we implement verify on the Signature class it has to account for all those variables, or, more likely, we'd use different Signature subclasses for different verify implementations.
As a consequence a concrete Signer  subclass implementation must know which concrete Signature subclass to return, in order to provide the correct verify method. I don't know if we want to put that extra responsibility on the Signer. To me, this feels more like a task for the public key.

I'm a bit unsure about that. I think in most cases sign can just return a generic Signature with a keyid and some raw signature bytes, independently of the key type, scheme, etc. used to create the signature. If we implement verify on the Signature class it has to account for all those variables, or, more likely, we'd use different Signature subclasses for different verify implementations.
As a consequence a concrete Signer subclass implementation must know which concrete Signature subclass to return, in order to provide the correct verify method. I don't know if we want to put that extra responsibility on the Signer. To me, this feels more like a task for the public key.

Yes, you are right. Those arguments are logical.
Today we had a quick discussion with @jku  about the Signer return type and he shared some concerns if we return our own Signature` type.
Jussi, can you please add your opinion to the thread?

I'm a bit unsure about that. I think in most cases sign can just return a generic Signature with a keyid and some raw signature bytes, independently of the key type, scheme, etc. used to create the signature. If we implement verify on the Signature class it has to account for all those variables, or, more likely, we'd use different Signature subclasses for different verify implementations.
As a consequence a concrete Signer subclass implementation must know which concrete Signature subclass to return, in order to provide the correct verify method. I don't know if we want to put that extra responsibility on the Signer. To me, this feels more like a task for the public key.

Yes, you are right. Those arguments are logical.
Today we had a quick discussion with @jku  about the Signer return type and he shared some concerns if we return our own Signature` type.
Jussi, can you please add your opinion to the thread?

A very basic implementation could look something like this:
# in securesystemslib
class Signer:
    @abc.abstractmethod
    def sign(bytes: payload) -> Signature:
        raise NotImplementedError

class SSlibSigner(Signer):
    def __init__(self, sslib_private_key):
        self.key = sslib_private_key

    def sign(self, payload):
        return sslib_keys.create_signature(self.key, payload)

class GPGSigner(Signer):
    def __init__(self, gpg_keyid):
        self.keyid = gpg_keyid

    def sign(self, payload):
        return sslib_gpg.create_signature(payload, self.keyid)

# in TUF
class Metadata:
    def sign(sslib.Signer: signer):
        signer.sign(self.signed.to_canonical_bytes())

# in Warehouse
class VaultSigner(Signer):
    def sign(self, payload):
        # ... you get the idea

metadata = tuf.Metadata(...)
metadata.sign(VaultSigner(...))

I just found out that we have to pass not only the sslib_private_key_dict in the __init__, but the whole key dictionary containing
the keytype, scheme, keyid and keyval containing both private and public key if we want to use securesystemlib.create_signature(self.key_dict, payload).

As a consequence a concrete Signer subclass implementation must know which concrete Signature subclass to return, in order to provide the correct verify method. I don't know if we want to put that extra responsibility on the Signer. To me, this feels more like a task for the public key.

Yes, you are right. Those arguments are logical.
Today we had a quick discussion with @jku about the Signer return type and he shared some concerns if we return our own Signature type.

I think it was mostly that it seemed like an object isn't needed in this case at all (I'm assuming that usually when you sign something you are doing that to create metadata to write to a file: an array of bytes is fine for that)?
Also I don't know how the opposite case (verifying portions of metadata while de-serializing it) is going to look like and that seemed more relevant for a potential Signature object?  As in how is Metadata API used with e.g. Vault in that case?

@jku, this is roughly how canonicalization, signing/verifying and de/serialization currently play together:
# On the server
1. create canonical bytes for payload
2. create signature over canonical bytes
3. serialize non-canonical payload + signature(s)

# On the client
1. deserialize non-canonical payload + signature(s)
2. create canonical bytes for payload
3. verify signature over canonical bytes

Note that this might change in the future. Currently we are discussing an alternative protocol, which does not require parsing the full payload before verifying the signature (see  Secure Systems Lab signing specification).
Especially, in the light of that new proposal and the likely short-livedness of signature objects I agree with you that a class model might not be needed.
However, I think a class helps to clearly define the required signature format, even if it only consists of two fields (keyid and signature bytes). And it also aligns with our decision in ADR 4 to create classes for all complex metadata attributes, e.g. in order to add self-validation behavior.

I have pushed the initial version of the signing interface and the necessary TUF changes to support it in the prs linked just above this. comment.

While working on SSlibSigner and GPGSigner I realized that if we want to store the necessary information to verify the signatures with securesystemslib.kets.verify_signature()  and securesystemslib.gpg.functions.verify_signature() we would need to add more key metadata in Signature.
For the SSlibSigner, we would need to add the key_dict: https://github.com/secure-systems-lab/securesystemslib/blob/dff4425e5663c58c954447a698efb17c4b23b0f8/securesystemslib/keys.py#L738
and for the GPGSigner, we would need to add the pubkey info https://github.com/secure-systems-lab/securesystemslib/blob/dff4425e5663c58c954447a698efb17c4b23b0f8/securesystemslib/gpg/functions.py#L170
the way we are doing this in the tests: https://github.com/secure-systems-lab/securesystemslib/blob/dff4425e5663c58c954447a698efb17c4b23b0f8/tests/test_gpg.py#L571

Because of the above problem, I am not sure if I can create a relationship between the Signature class and the GPGSignature.
Before, I thought I can just inherit the Signature class used by the SSLibSigner and add specific fields related to GPG, but I am not that sure anymore.
With the observations from the above comment, I made the Signature class like this:

the problem is that for GPGSignature I won't need the self.keydict = keydict field or it will be totally different.
The pubkey info could be one of three other schemas

and each one of them with its own specifics...

@MVrachev, I thought you agreed with my arguments above for not implementing verification as method of the Signature class? (And even if we did, I wouldn't store the public key on the Signature object but rather pass it to the verification method as argument.)

Yes, we agreed that we would do the verification elsewhere.
I just thought we want to store all of the information needed for the signer verification into the Signer class.
We discussed that with Lukas and we would store only the keyid and the signature.
The keyid will give us the necessary additional information for the verification.

I will add a GPGSigner in another pr, but I decided to mark this issue as fixed in my pr #319 because it fixed the main point of this issue - creating a Signer interface and a securesystemslib implementation of it.


Add Timeout Config to ChatOpenAI Object from LLMFactory - Target Version 0.2.0b1

[x] Just add it and all the necessary consequentials

Done with 0.2.0b1


getstorybook --html does not work

getstorybook --html does not work, it reports "unknown option '--html'"
storybook/cli 3.4.6
storybook/html 4.0.0-alpha.4
MacOS 10.13.5
Node 9.7.1

You probably need to use @storybook/cli 4 as well.

Great! I can confirm this works:
@storybook/cli 4.0.0-alpha.8
@storybook/html 4.0.0-alpha.8
I have a few other issues but I will dig into that and open a separate issue when needed, thanx!


[All Hosts] (tools) add link to yo office repo

Doesn't need a review.

Docs Build status updates of commit 3247ad6:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/develop/yeoman-generator-overview.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


mod.request is not a function

Hi guys,
I'm trying to make a call with the javascript SDK from node js 7.10.1 with the following code:
var client = MessagemediaRestApi.ApiClient.instance;
var basic = client.authentications["basic"];

basic.username = user.MessageMediaUser;
basic.password = user.MessageMediaSecret;

var api = new MessagemediaRestApi.MessagingApi();
var post = new MessagemediaRestApi.Messages(mmMessages);

api.sendMessages(post, function (error, data, response) {
     ...
});

At the point sendMessages is called I get an exception mod.request is not a function with the following callstack:
at Request.request (/var/lib/arangodb3-apps/_db/nsw_justice/wyp_apis/APP/node_modules/superagent/lib/node/index.js:748:28)
2017-08-04T04:30:32Z [20466] ERROR     at Request.end (/var/lib/arangodb3-apps/_db/nsw_justice/wyp_apis/APP/node_modules/superagent/lib/node/index.js:868:18)
2017-08-04T04:30:32Z [20466] ERROR     at callApi (/var/lib/arangodb3-apps/_db/nsw_justice/wyp_apis/APP/node_modules/messagemedia-rest-api/src/ApiClient.js:405:13)
2017-08-04T04:30:32Z [20466] ERROR     at sendMessages (/var/lib/arangodb3-apps/_db/nsw_justice/wyp_apis/APP/node_modules/messagemedia-rest-api/src/api/MessagingApi.js:141:29)

Let it be known that I'm not the most worldly node developer, but, upon inspecting the superagent code it does seem a little crackers and I'd expect it not to work... but clearly it's working for you.
If you look at superagent/lib/node/index.js where the exception is occurring the offending line looks like this:
// initiate request
var mod = exports.protocols[url.protocol];

// request
var req = this.req = mod.request(options);

where exports.protocls has previously been defined as:
exports.protocols = {
  'http:': http,
  'https:': https
};

Do you guys have any insight into this?

Ok, apologies for raising this. Turns out the code is being run in ArangoDb, which is a subset of node (I actually think it's an older io.js)... that is the problem.


Windows phone 8.1

When Windows phone 8.1 version?
Thanks

Doesn't it work?

no

That's unexpected as I have a WP 8.1 device with the plugin working.
Can you show me your project? Not just a sample, all of it please.


fix: remove blank mission

:tada: This PR is included in version 3.0.13 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Macro applications

I'm adding an issue to save a feature suggested by @kha in another thread.
For printf and even more complex applications, we could also implement a macro system that allows them to use the full power of the VM and the meta level.
@[macro]
meta def printf (s : string) : tactic expr := ...

#check printf "%d: %s" -- nat -> string -> string

Here is a possible way to implement it.
Whenever the elaborator finds a pre-term headed by a constant tagged with [macro], it fully elaborates the macro arguments, makes sure they are closed terms, and then invokes the VM to get the resulting expression.
Example: givenprintf "%d: %s" t s, we first elaborate printf "%d: %s", and execute. Then we replace this pre-term with the resulting term. Then, we elaborate t and s using the information about their expected type.

Alternatively, this could be implemented via user-defined notations, which I believe we wanted to have anyway:
reserve prefix `printf`:100

@[notation]
meta def printf_macro (_ : parse $ tk "printf") (s : parse parser.string) : tactic pexpr := ...

The only downside I can see is that we lose namespacing. On the other hand, I think I'd prefer a nicer notation like f"$x $y" (that makes use of to_format) anyway.

@kha For printf, the nicer notation is indeed cool.
Both features seem to be useful: user-defined notations and macro applications.
Macro applications are processed by the elaborator. So, they are more flexible since they can access the local context (if we want). At parsing time, the local context is not really available (the types are missing or are pre-terms).

So, they are more flexible since they can access the local context (if we want).

Ah, I overlooked that fact. Providing separate parsing mechanisms for the two seems redundant, so how about exposing a Lean API for creating new macro_definitions? They would come with a callback for elaborating them and could be inserted by user-defined notation parsers.

how about exposing a Lean API for creating new macro_definitions? They would come with a callback for elaborating them and could be inserted by user-defined notation parsers.

This is cool, but it is not clear how much work would it be.
There is also a dependency problem. The macro_definition is defined in the Kernel. So, it is not clear how the elaboration callback would look like.
To expose the current macro_definition, we would also have to expose the abstract_type_context functionality.

These macros should be purely pre-elaboration, so I don't think they need to be expandable(?). The callback would be invoked by the elaborator, just like the current visit_macroX handlers.

Actually, can we just by for that? Or could it be too "lazy" for some use cases?
meta def build_printf (s : string) : tactic expr := ...

reserve prefix `printf`:100
@[notation]
meta def printf_macro (_ : parse $ tk "printf") (s : parse parser.string) : tactic pexpr :=
pure ``(by build_printf %%(reflect s))

Btw. what do you think about splicing locals even into ``() quotations? I didn't implement it originally because it didn't "feel right", but the fewer reflect the merrier, imo.

I'm closing this issue. @kha has already implemented the approach described in the previous comment. Moreover, issue #1674 subsumes the rest.