[NETBEANS-54] Module Review libs.xerces

Updated maven coordinates for Xerces 2.8.0 (Apache 2).
Added xerces notice from https://svn.apache.org/repos/asf/xerces/java/tags/Xerces-J_2_8_0/NOTICE

Please recheck this as this breaks the build. My local build breaks different than the protocol from travis, but reproducible. To check yourself, please run in the repository:
git clean -x -f
ant

The git clean part removes all files, that are not part of the repository. The downloaded file gets a different name and is can't be made available at build and compile time.
Hava a look at project.properties - this should do it:
# This patch file was generated by NetBeans IDE
# It uses platform neutral UTF-8 encoding and \n newlines.
--- a/libs.xerces/nbproject/project.properties
+++ b/libs.xerces/nbproject/project.properties
@@ -16,6 +16,6 @@
 # under the License.

 is.autoload=true
-release.external/xerces-2.8.0.jar=modules/ext/xerces-2.8.0.jar
+release.external/xercesImpl-2.8.0.jar=modules/ext/xerces-2.8.0.jar
 module.jar.verifylinkageignores=org.apache.xerces.util.XMLCatalogResolver
 spec.version.base=1.39.0

Should be ready now. Sorry for the trouble. It would be nice getting a travis warning in case of build failure.

Your wish was already granted - the travis configuration was added to the repository literally hours ago. That's the reason I saw that. :-)

It would be nice getting a pair of beers in case of build success! :-D

For the record: The content of the two binaries is identical, the files from maven central and the original netbeans file were zipped with different timestamps and differing directory order.
Diffoscope output is here: https://try.diffoscope.org/qayzdzqugqge.html
To check yourself: unzip both files and run a recursive diff over it.

I squashed your two commits together and merged this.

Thanks Matthias. If you ever happen to visit Madrid you know you have some free beers here.


Implement search()

Provide implementation for search() definition here in file src/Modules/Searcher.php.
Ensure all tests on this function is passed before submitting changes.

Issue Fixed


tf.signal CPU FFT implementation is slower than NumPy, PyTorch, etc.

What related GitHub issues or StackOverflow threads have you found by searching the web for your problem?
Environment info
Operating System:
Ubuntu 16.04 LTS 64bit
Installed version of CUDA and cuDNN:
(please attach the output of ls -l /path/to/cuda/lib/libcud*):
-rw-r--r-- 1 root root   558720 9月  15 07:02 libcudadevrt.a lrwxrwxrwx 1 root root       16 9月  15 07:05 libcudart.so -> libcudart.so.8.0 lrwxrwxrwx 1 root root       19 9月  15 07:05 libcudart.so.8.0 -> libcudart.so.8.0.44 -rw-r--r-- 1 root root   415432 9月  15 07:02 libcudart.so.8.0.44 -rw-r--r-- 1 root root   775162 9月  15 07:02 libcudart_static.a -rwxr-xr-x 1 root root 79337624 10月 27 23:13 libcudnn.so -rwxr-xr-x 1 root root 79337624 10月 27 23:13 libcudnn.so.5 -rwxr-xr-x 1 root root 79337624 10月 27 23:13 libcudnn.so.5.1.5 -rw-r--r-- 1 root root 69756172 10月 27 23:13 libcudnn_static.a
If installed from binary pip package, provide:

A link to the pip package you installed:

The output from python -c "import tensorflow; print(tensorflow.__version__)".
0.12.head
If installed from source, provide

The commit hash (git rev-parse HEAD)

The output of bazel version

If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)
import numpy as np
import tensorflow as tf
import time

wav = np.random.random_sample((1024,))
spec = np.fft.fft(wav)[:513]

x = tf.placeholder(dtype=tf.complex64, shape=[513])
result = tf.ifft(x)
sess = tf.Session()

start = time.time()
for i in range(10000):
    something = sess.run(result, feed_dict={x:spec})
print 'tensorflow:{}s'.format(time.time()-start)

start = time.time()
for i in range(10000):
   	something = np.fft.ifft(spec)
print 'numpy:{}s'.format(time.time() - start)

tensorflow:25.7219519615s
numpy:0.391902923584s

What other attempted solutions have you tried?
Logs or other output that would be helpful
(If logs are large, please upload as attachment or provide link).

Likely you are measuring the overhead of .run(), the copying of the feed variable spec which doesn't happen in numpy. In general, you will get good performance if each .run() does a significant portion of work. For example, I move your computation into a while loop inside tensorflow graph and get
tensorflow:0.0105199813843s
numpy:0.040412902832s

Here's the code:
import numpy as np
import tensorflow as tf
import time

niterations=1000
N=1024
wav = np.random.random_sample((N,))
spec = np.fft.fft(wav)[:N/2+1]

sess = tf.Session()

# Create and initialize variables
cnt = tf.Variable(tf.constant(niterations))
specVar = tf.Variable(spec, dtype=tf.complex64)
sess.run(tf.variables_initializer([specVar,cnt]))

# While loop that counts down to zero and computes reverse and forward fft's
def condition(x,cnt):
  return cnt <= 0

def body(x,cnt):
  xrev=tf.ifft(x)
  xnew=tf.fft(xrev)
  cntnew=cnt-1
  return xnew, cntnew

start = time.time()
tf.while_loop(condition, body, [specVar,cnt], parallel_iterations=1)

print 'tensorflow:{}s'.format(time.time()-start)

# Equivalent numpy loop
start = time.time()
x = spec
for i in range(niterations):
   xrev = np.fft.ifft(x)
   x= np.fft.fft(xrev)

print 'numpy:{}s'.format(time.time() - start)

@aselle do you run the ops in your version or just construct them?

Good catch @vrv, forgot the run() call. When putting in a run() call (see below code), I get
tensorflow:1.78562903404s
numpy:0.0487790107727s

which I suppose is comparible to your results (yours was numpy 66x faster, and mine was like numpy 33x faster). One explanation is that the GPU FFT implementation is really not tuned to smalls sizes, so that it can't achieve the same performance of the CPU FFT on a relatively small 513 element array. That's only 2 KBytes of data, which is not much for throughput optimized devices. As such I ran another one where I set N=16384 which will basically use 8193 element array. Then I got
tensorflow:5.21
numpy:41.5

import numpy as np
import tensorflow as tf
import time

niterations=1000
N=16384
wav = np.random.random_sample((N,))
spec = np.fft.fft(wav)[:N/2+1]

with tf.Session() as sess:
  # Create and initialize variables
  cnt = tf.Variable(tf.constant(niterations))
  specVar = tf.Variable(spec, dtype=tf.complex64)
  sess.run(tf.variables_initializer([specVar,cnt]))

  # While loop that counts down to zero and computes reverse and forward fft's
  def condition(x,cnt):
    return cnt > 0

  def body(x,cnt):
    xrev=tf.ifft(x)
    xnew=tf.fft(xrev)
    cntnew=cnt-1
    return xnew, cntnew

  start = time.time()

  final, cnt= tf.while_loop(condition, body, [specVar,cnt], parallel_iterations=1)
  final, cnt =  sess.run([final,cnt])

  print 'tensorflow:{}s'.format(time.time()-start)

# Equivalent numpy loop
start = time.time()
x = spec
for i in range(niterations):
   xrev = np.fft.ifft(x)
   x= np.fft.fft(xrev)

print 'numpy:{}s'.format(time.time() - start)

Closing due to lack of recent activity. We will reopen when additional information becomes available. Thanks!

I've done some simple profiling and it seems the CPU version of tf.spectrum.fft is slow in small nfft because of EIGEN fft function. The most time spent on the execution of CPU fft kernel is by EIGEN fft function.

I have done some profiling of my own using the trace profiler and the results suggest that the tensorflow FFT2D is about 50 times slower than numpy's.
Here is what I have done (with IFFT2D but the results are the same for FFT2D):
from keras.layers import Layer, Input, Lambda
from keras.models import Model
import numpy as np
import tensorflow as tf
from tensorflow.python.client import timeline
from tensorflow.signal import ifft2d

def tf_unmasked_adj_op(x):
    return tf.expand_dims(ifft2d(x[..., 0]), axis=-1)

# Model definition (we basically perform and inverse fft and then get the module)
input_size=(320, None, 1)
kspace_input = Input(input_size, dtype='complex64', name='kspace_input')
zero_filled = Lambda(tf_unmasked_adj_op, output_shape=input_size, name='ifft_simple')(kspace_input)
zero_filled_abs = Lambda(tf.math.abs)(zero_filled)
model = Model(inputs=kspace_input, outputs=zero_filled_abs)
run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
run_metadata = tf.RunMetadata()

model.compile(
    optimizer='adam',
    loss='mse',
    options=run_options,
    run_metadata=run_metadata,
)

# fake data
data_x = np.random.rand(35, 320, 320, 1) + 1j * np.random.rand(35, 320, 320, 1)
data_y = np.random.rand(35, 320, 320, 1)

# a single pass
model.fit(
    x=data_x,
    y=data_y,
    batch_size=35,
    epochs=1,
    verbose=2,
    shuffle=False,
)

# profiling output
tl = timeline.Timeline(run_metadata.step_stats)
ctf = tl.generate_chrome_trace_format()
with open('timeline_fft_test.json', 'w') as f:
    f.write(ctf)

On 8 CPUs (intel i7), ubuntu 16.04, tensorflow 1.14.0, keras 2.2.4, the fitting time is 5s. Looking into the timeline (chrome://tracing/), I find that the IFFT2D has a duration of 5000ms (=5s).
As a comparison, the numpy version is only 80ms.
Here is the code I used to perform the numpy profiling (in a jupyter notebook, with the same data_x):
def np_inv_fft(x):
    return np.fft.ifft2(x)

%%timeit
for i in range(len(data_x)):
    np_inv_fft(np.squeeze(data_x[i]))

Maybe I am doing the profiling in a wrong way, but it does feel like the FFT2D is the bottleneck (and a very big one) in my actual models (this is just a minimal reproducible example).
I haven't taken the time to see if it was due to the fourier transform 2D being in a Lambda layer in a keras model, but I guess it shouldn't. I will try to do this in pure tensorflow to see how it is.

This is what I tried in pure tensorflow with the same results:
import numpy as np
import tensorflow as tf
from tensorflow.python.client import timeline
from tensorflow.signal import ifft2d

def tf_unmasked_adj_op(x):
    return tf.expand_dims(ifft2d(x[..., 0]), axis=-1)

# Model definition (we basically perform and inverse fft and then get the module)
kspace_input = tf.placeholder(dtype='complex64', shape=(None, 320, None, 1))
fake_output = tf.placeholder(dtype='float32', shape=(None, 320, None, 1))
zero_filled = tf_unmasked_adj_op(kspace_input)
zero_filled_abs = tf.math.abs(zero_filled)
# this is just to have an adam optimizer similar to the keras training in order to keep things similar (we just compare the IFFT2D compute time in the timeline anyway)
w = tf.Variable((1,), trainable=True, dtype=tf.float32)
zero_filled_abs = tf.math.multiply(zero_filled_abs, w)

loss = tf.losses.mean_squared_error(fake_output, zero_filled_abs)

adam = tf.train.AdamOptimizer(learning_rate=0.3)
a = adam.minimize(loss, var_list=[w])

run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
run_metadata = tf.RunMetadata()

# fake data
data_x = np.random.rand(35, 320, 320, 1) + 1j * np.random.rand(35, 320, 320, 1)
data_y = np.random.rand(35, 320, 320, 1)

# a single pass
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for i in range(1):
        sess.run(a, feed_dict={kspace_input: data_x, fake_output: data_y}, options=run_options, run_metadata=run_metadata)

# profiling output
tl = timeline.Timeline(run_metadata.step_stats)
ctf = tl.generate_chrome_trace_format()
with open('timeline_fft_test.json', 'w') as f:
    f.write(ctf)

So I tried one further thing to see how big the problem was.
I wrapped the numpy IFFT2D in a tensorflow layer using concepts from odl (itself inspired by gists shared in #1095).
This is the code I used to create the tensorflow layer:
import numpy as np
import uuid
import tensorflow as tf
from tensorflow.python.framework import ops

# Define custom py_func which takes also a grad op as argument:
def py_func(func, inp, Tout, name=None, grad=None):
    if grad is None:
        return tf.py_func(func, inp, Tout, stateful=False, name=name)
    else:
        override_name = 'PyFuncStateless'

        # Need to generate a unique name to avoid duplicates:
        rnd_name = override_name + 'Grad' + str(uuid.uuid4())

        tf.RegisterGradient(rnd_name)(grad)
        g = tf.get_default_graph()

        with g.gradient_override_map({override_name: rnd_name}):
            return tf.py_func(func, inp, Tout, stateful=False,
                              name=name)

def fft_layer(mask, op_name='forward'):
    def forward_op(imgs):
        fft_coeffs = np.empty_like(imgs)
        for i, img in enumerate(imgs):
            fft_coeffs[i] = mask * np.fft.ifftshift(np.fft.fft2(np.fft.fftshift(img[..., 0]), norm='ortho'))[..., None]
        return fft_coeffs

    def adj_op(kspaces):
        imgs = np.empty_like(kspaces)
        for i, kspace in enumerate(kspaces):
            masked_fft_coeffs = mask * kspace[..., 0]
            imgs[i] = np.fft.fftshift(np.fft.ifft2(np.fft.ifftshift(masked_fft_coeffs), norm='ortho'))[..., None]
        return imgs

    if op_name == 'forward':
        op = forward_op
        grad_op = adj_op
    else:
        op = adj_op
        grad_op = forward_op

    def tf_grad_op(x, dy, name):
        with tf.name_scope(name):
            out_shape = x.shape()
            with ops.name_scope(name + '_pyfunc', values=[x, dy]) as name_call:
                result = py_func(
                    grad_op,
                    [x, dy],
                    [tf.complex64],
                    name=name_call,
                )

                # We must manually set the output shape since tensorflow cannot
                # figure it out
                result = result[0]
                result.set_shape(out_shape)
                return result

    # Def custom square function using np.square instead of tf.square:
    def tf_op(x, name=None):
        with tf.name_scope(name, op_name, values=[x]) as name:
            x_shape = x.get_shape()
            with ops.name_scope(name + '_pyfunc', values=[x]) as name_call:
                result = py_func(
                    op,
                    [x],
                    [tf.complex64],
                    name=name,
                    grad=tf_grad_op,
                )
                # We must manually set the output shape since tensorflow cannot
                # figure it out
                result = result[0]
                result.set_shape(x_shape)
                return result

    return tf_op

(a bit more complex than a simple inverse fourier transform but I need it for my application)
Even with this slightly more complex function, in numpy, on CPU, the operation now takes 200ms.
I don't know if I am doing things incorrectly, if so please tell me. I don't think I could help solve this problem unfortunately as I don't know how to code in C (I guess if there is a problem it's related to the eigen support of FFT2D).
@aselle @vrv I see that you were involved in this discussion earlier, do you have any idea of what's going on? (sorry to ping you directly but this is really a bottleneck for me).

But even for a batch size of 1 (i.e. the caching isn't involved), TensorFlow's FFT is still slower than numpy's wrapped (in a TensorFlow py_func) IFFT2D (130ms compared to 25ms on CPU).
I also don't understand how the benchmark I did relates to @aselle 's one. Is it because I am considering 2D?

Actually, something must be wrong with my benchmark. A slightly more complex model (where I basically concatenate the real and imaginary part then make them back complex) is ~15 times faster.
I opened an SO question where I asked for some inputs on this but if you have any idea of what's going on, I am impatient to understand.

It looks like you're producing trace timelines (great!) -- can you post them?

Sure. Attached you will find a zip tf_fft_timelines.zip containing 4 files. They correspond to the experiments I showed above. Basically the network is doing an IFFT2D, masking the result, and taking the complex module. The batch size by default is 35. I am fitting the network, but since there is no parameters to fit, it's just doing a forward pass (results are roughly the same for only a batch prediction).

timeline_fft_test_tf_bs1.json : the IFFT2D is done with tf.signal.ifft2d and the batch size is 1. Here the measured time for the op is 130ms.
timeline_fft_test_tf.json : the IFFT2D is done with tf.signal.ifft2d. Here the measured time for the ops is 3900ms.
timeline_fft_test_tf_numpy_bs1.json : the IFFT2D is done with the layer described in this comment (i.e. wrapping np.fft.ifft2d in a tensorFlow layer with some extra tweaks) and the batch size is 1. Here the measured time for the op is 28ms.
timeline_fft_test_tf_numpy.json : the IFFT2D is done with the layer described in this comment (i.e. wrapping np.fft.ifft2d in a tensorFlow layer with some extra tweaks). Here the measured time for the op is 228ms.

This benchmark was done with Ubuntu 16.04, on CPU only.

timeline_fft_test_tf_bs1.json : the IFFT2D is done with tf.signal.ifft2d and the batch size is 1. Here the measured time for the op is 130ms.
timeline_fft_test_tf.json : the IFFT2D is done with tf.signal.ifft2d. Here the measured time for the ops is 3900ms.
timeline_fft_test_tf_numpy_bs1.json : the IFFT2D is done with the layer described in this comment (i.e. wrapping np.fft.ifft2d in a tensorFlow layer with some extra tweaks) and the batch size is 1. Here the measured time for the op is 28ms.
timeline_fft_test_tf_numpy.json : the IFFT2D is done with the layer described in this comment (i.e. wrapping np.fft.ifft2d in a tensorFlow layer with some extra tweaks). Here the measured time for the op is 228ms.

Thanks for that! What CPU are you using? Does it have AVX support and was TensorFlow built with AVX support?

What CPU are you using?

Output of lshw:
*-cpu:0
          product: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
          vendor: Intel Corp.
          physical id: 6
          bus info: cpu@0
          size: 800MHz
          capacity: 3GHz
          width: 64 bits

Does it have AVX support and was TensorFlow built with AVX support?

I did not build TensorFlow from source so it was not built with AVX support. However, even when I use my GPU (Quadro P5000), the IFFT2D is still slower with tf.signal.ifft2d than with np.fft.ifft2d wrapped (don't know if it's supposed to make a big difference), but faster than on CPU.

I just did another simple benchmark, this time comparing TensorFlow (used with keras) to pytorch for the IFFT2D. I find that for prediction, pytorch is 40 times faster than tensorflow when using the IFFT2D.

Thanks @zaccharieramzi!
Could you please run a GPU benchmark as well? Based on this I think PyTorch uses cuFFT and MKL, so on GPU TensorFlow and PyTorch should be using the same underlying FFT implementation. It would be nice to confirm whether or not there is a difference when both systems are using cuFFT because that would point to inefficiencies outside the particular FFT implementation that need to be addressed.

I'm reopening because it is a known issue that TensorFlow's CPU FFT implementation is based on Eigen's TensorFFT, which is not fast compared to FFTW, MKL (what PyTorch uses), or FFTPACK (what NumPy uses).
For projects I work on at Google, we use TPUs and GPUs for FFTs in both training and serving, so I think we are not noticing the pain of the CPU FFTs being slow. Sorry about that :(. Unfortunately we are a little stuck in terms of legal approval for the above mentioned libraries. When we implemented tf.signal Eigen's TensorFFT was the best option in terms of features (non-power-of-2 FFTs via Bluestein's algorithm is a hard requirement) and legal compatibility.

I am training a model on my main GPU right now, but as soon as I can I will run the same benchmarks. But as I recall (and wrote here), even with GPU, tf.signal.ifft2d is quite slow.
I will come back with exact numbers coming from the trace timelines (and a zip with all of them).

Ok, so after running the benchmark on GPU, I ran into the following problem: the FFT ops is not being run on GPU but on CPU even when the GPU is visible to TensorFlow. Other operations are run on GPU however, so I guess the elongated time comes from the fact that there is an overhead due to the communication between CPU and GPU.
Attached you will find the zip containing the same 4 files as before, this time with GPU visible (you can see some of the ops being run on GPU).
In the numpy ones, the IFFT2D is still run on CPU (that's I guess expected), but the TensorFlow ones are in the same situation (even if the other ops are run on GPU).
Do you think it's due to the Lambda layer I am using?

Ok, so after running the benchmark on GPU, I ran into the following problem: the FFT ops is not being run on GPU but on CPU even when the GPU is visible to TensorFlow. Other operations are run on GPU however, so I guess the elongated time comes from the fact that there is an overhead due to the communication between CPU and GPU.
Attached you will find the zip tf_fft_timelines_gpu.zip containing the same 4 files as before, this time with GPU visible (you can see some of the ops being run on GPU).
In the numpy ones, the IFFT2D is still run on CPU (that's I guess expected), but the TensorFlow ones are in the same situation (even if the other ops are run on GPU).
Do you think it's due to the Lambda layer I am using?

I ran the benchmarks on GPU. You can find a zip tf_fft_timelines_gpu.zip attached with similar nomenclature as before. (You will notice that only one run "epoch" appears in each trace, I did not find a way -did not look very hard as well- to have all the "epochs" in once trace)
The first "epoch" is way slower (that's why I did more than 1 "epoch" this time). Was the remark about not caching the plan only applicable to CPU?
This time the pure tf.signal op takes only 1.2 ms on GPU after being cached (500 ms before being cached so still slower than numpy on CPU in that regard, to be considered in my case where all the images don't have the same shape) for a batch size of 35.
I also enhanced my pytorch vs keras benchmark, to feature trace timelines
keras_vs_pytorch_fft_timelines.zip
for both for the forward passes, and taking into account caching. I think the IFFT2D times are equivalent when looking at the timelines (but there is some much stuff in there I am not sure how to read them, if you have a resource it would be welcome).
However, when I look at the overall run time for, for example, prediction, pytorch is 4 times faster (7 times on fitting). I don't think it's only due to the fact that the tensors are already in GPU for pytorch. But maybe it's due to something else that I don't get.

Thanks again @zaccharieramzi. While an end-to-end benchmark is useful, for the purpose of this bug I'm only concerned with the FFT ops themselves, so I think it could be simplified.
Some issues I noticed:

The PyTorch benchmark starts with x already on the GPU, while the TensorFlow one has to transfer the NumPy array from host to device.
The PyTorch benchmark does not measure device to host time for the result tensor (the final r in the predict benchmark is a device = cuda object, while I believe Keras would implicitly fetch the tensor from device to host when running predict_on_batch.
The TensorFlow benchmark is running with tracing enabled -- tracing has some overhead, and so should not be enabled in a benchmark.
Instead of %%time, it could give a lower variance estimate if you use %%timeit. Both of these methods are subject to variance introduced by overheads in the Python interpreter, etc.

Here's a more "raw" benchmark for TensorFlow that can give a better sense of what's going on.
import tensorflow as tf
import numpy as np

tf.reset_default_graph()

x_np = np.random.rand(35, 320, 320) + 1j * np.random.rand(35, 320, 320)
with tf.device('/gpu:0'):
  x_tf = tf.constant(x_np)
  y = tf.signal.ifft2d(x_tf)

sess = tf.Session()

And then:
%%timeit -n1000
sess.run(y.op)  # Note the .op avoids fetching y from device to host.

On a public Colab runtime with a K80 GPU, I'm observing a very high initial run time, followed by a steady result of:
1000 loops, best of 3: 3.47 ms per loop

Compared to PyTorch, which seems to be faster by about 2x:
import torch
dtype = torch.float32
device = torch.device("cuda")
x = torch.randn(35, 320, 320, 2, device=device, dtype=dtype)

And then:
%%timeit -n1000
y_pt = torch.ifft(x, 2)

I'm getting:
1000 loops, best of 3: 1.45 ms per loop

I believe this is apples-to-apples, since we're not measuring device/host transfer time, and the input shapes are the same.
Here is a screenshot of a trace of the above TensorFlow graph executing on the same K80:

You can see there are no memcpy D2H/H2Ds (good). You can see there is over 0.5 ms of time between the IFFT2D op execution (GPU:0 on the bottom) and the CUDA stream beginning work (stream:0). Then the CUDA FFT kernel itself takes about 3ms total.
I'll take a look at how PyTorch invokes cuFFT to see how they differ from TensorFlow. The performance should be identical on GPU.

While an end-to-end benchmark is useful, for the purpose of this bug I'm only concerned with the FFT ops themselves, so I think it could be simplified.

Totally agree. Thanks for the remarks on the different aspects of the benchmark.
However, the trace timelines I got (still relevant I think) are not exactly the same as yours for TensorFlow. For example, I only have 2 items in the CUDA stream.
The overall time taken by the IFFT (1.7ms) is also much closer to that of pytorch (1.4ms). Do you think it might be due to differences in the GPU we are using?
Anyway, I do think your benchmark shows that it might be worth investigating.

I slightly modified mentioned above script for pytorch:
import torch
dtype = torch.float32
device = torch.device("cuda")
fft_size = 1024 * 8
batch_size = 100 * 1024 * 1024 // fft_size # to fit into GPU memory

And then:
%%timeit -n1 -r1
x = torch.randn(batch_size,fft_size,2, device=device, dtype=dtype)
y = torch.randn(batch_size,fft_size,2, device=device, dtype=dtype)
for i in range(200):
    f = torch.fft(x, 1)   # 26 mults per complex sample
    z = torch.mul(f, y)   #  3 mults per complex sample
    a = torch.ifft(z, 1)  # 26 mults per complex sample

Script output after the first run after kernel restart in Jupiter notebook:
2.42 s ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)
Script output after the second and all other consecutive runs keeping all structures in GPU memory:
34.2 ms ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)
Both input arrays (x and y) have 100M complex FP32 samples each. If I calculated correct there are 26*2+3=55 FP32 mults per input sample. Taking into account 200 iterations in a loop I would estimate number of required operations as 100M * 200 * 55 = 1.1 TFLOP.
I have GTX 1070 installed on my PC and by spec it can perform up to 6.5 TFLOPS for FP32 operations. Looks like I did some mistake in my calculations because script supposedly performs 1.1 TFLOP each 34.2 ms which can't be correct.
Would anyone point out on my mistake, please?

Removing the support label, as this describes a real issue with tensorflow that is not yet resolved.

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue?

It has been 31 days with no activity and the awaiting response label was assigned. Is this still an issue?

We are closing this issue for now due to lack of activity. Please comment if this is still an issue for you. Thanks!

It's still an issue -- bad bot, @tensorflowbutler!

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue?

Here is a simple benchmark comparing tensorflow and numpy's real-valued 1D FFT on CPU, showing that TF is slower by a factor of 10:
https://colab.research.google.com/gist/Andreas5739738/fc603468829ee0fc7e40a2e27d8a6661/fft.ipynb
import timeit

print(timeit.timeit('X = tf.signal.rfft(x)', setup='import tensorflow as tf; x = tf.random.normal([50000, 512])', number=10))
print(timeit.timeit('X = numpy.fft.rfft(x)', setup='import numpy.fft; import tensorflow as tf; x = tf.random.normal([50000, 512])', number=10))

24.331672433999984
2.2700748900000463

Any update here?

Could we expect any update on it? CPU would be pretty popular for inference and improving it from 10+times speech difference would be great.

@rryan Are there any updates on this ?
I found myself confronted to this problem again while trying to optimize some of my input data pipelines, which are placed on CPU but still need to perform fft-like operations.
Btw, a hack I hadn't thought about beforehand is to use map_fn if you are using the Fast Fourier Transform operations in batches on CPU in an architecture with multiple cores. Depending on your batch size and architecture it could greatly reduce the computation time.

Actually, using this idea, you can build a much faster FFT for dimension 2 or higher.
Here is an example for 2D:
import multiprocessing

import tensorflow as tf
from tensorflow.python.ops.signal.fft_ops import fft2d, fft

@tf.function
def parallel_fft2d(image):
    partial_fourier_coefs = tf.map_fn(
        fft,
        image,
        parallel_iterations=multiprocessing.cpu_count(),
    )
    partial_fourier_coefs_t = tf.transpose(partial_fourier_coefs)
    fourier_coefs_t = tf.map_fn(
        fft,
        partial_fourier_coefs_t,
        parallel_iterations=multiprocessing.cpu_count(),
    )
    fourier_coefs = tf.transpose(fourier_coefs_t)
    return fourier_coefs

a = tf.cast(tf.random.normal([320, 320]), dtype=tf.complex64)
tf.test.TestCase().assertAllClose(
    fft2d(a),
    parallel_fft2d(a),
    rtol=1e-4,
    atol=1e-3,
)

The speed-up is only visible on non-power of 2 shapes, and in this case, on my machine with 8 cores, you can go from 150ms to 30ms. My guess is the speed-up will be even more significant if you have more cores and higher dimensions.

Any idea on how to use this map_fn function to improve the stft, @zaccharieramzi?

Hi @JPery ,
I seem to notice from your issue here, that you are interested in the stft for audio processing, so we are talking 1D stft.
For 1D applications, unfortunately, the only to currently benefit from the map_fn is to compute the transform in batches and have each element in the batch be transformed separately in parallel (the more cores you have the more acceleration you will get basically up to your batch size).
The idea behind the acceleration for 2D transforms is that you can write the application of the 2D transform as successive applications of the 1D transform, on each dimension.
After reading the code of the stft, there actually is a way to do it for the stft but it's a bit involved.
So my basic understanding of the stft, is that you will compute an fft (or rftt in this case) over multiple segments (or frames) of your input signal.
You could see these segments as multiple elements in a batch, and you can definitely try to transform those in parallel.
Schematically, this would mean replacing this line, by something like:
from functools import partial

return tf.map_fn(
	partial(fft_ops.rfft, fft_length=[fft_length]),
	framed_signals,
	parallel_iterations=multiprocessing.cpu_count(),  # or how many parallel ops you see fit
)

So you could just copy-paste the stft code and replace the final line, you should see a speed-up related to the number of frames you consider and the number of cpus you have at hand.

Thank you for your comment, @zaccharieramzi.
With your implementation I was able to perform up to 10 times faster in my use case.
It works like a charm!

Glad I could help!
I hope this idea can help others to enhance their CPU-Fourier workflow, until this issue is resolved.
It could maybe be directly implemented in TF if the legalities are going to block any further improvement.

@diggerdu,
I have executed your code in the latest version of Tensorflow (2.5) and observed that Tensorflow is much faster compared to Numpy. Please find the Gist of the working code. Thanks!

@rmothukuru the gist you linked does not seem to be a proper benchmark since it only executes variable assignment in a loop, no computation.
Here is an example demonstrating that TF 2.5 is still 10x slower than numpy:
https://colab.research.google.com/gist/Andreas5739738/fc603468829ee0fc7e40a2e27d8a6661/fft.ipynb

I added jax to the comparison, which is also much faster than tensorflow:
seconds (lower is better):
Tensorflow 2.5.0: 23.75205922899977
Numpy:  2.10841278099997
Jax:  1.2626468630001
jax uses pocketfft (https://jax.readthedocs.io/en/latest/_modules/jax/_src/lax/fft.html#fft), which seems to be much faster then Tensorflow's Eigen.
Updated colab: https://colab.research.google.com/gist/Andreas5739738/fc603468829ee0fc7e40a2e27d8a6661/fft.ipynb

We see that you are using old version of Tensorflow which is officially considered as end of life, We recommend that you upgrade to 2.6  version and let us know if the issue still persists in newer versions .Please open a new issue in case you face any errors, we will get you the right help .Thanks!

The situation has still not improved with TF 2.6:
Tensorflow 2.6.0 26.172108803000015
Numpy:  1.4308665990000122
WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
Jax:  1.412082421999969

This is still an issue in current tf-nightly (2.8+):
seconds (lower is better):
Tensorflow 2.9.0-dev20220126 24.188889063000033
Numpy:  2.1835842080000134
WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
Jax:  1.476103615999989

https://colab.research.google.com/gist/Andreas5739738/fc603468829ee0fc7e40a2e27d8a6661/fft.ipynb

Looks like Jax team found the same issue with XLA FFT slowness, and integrated PocketFFT as a workaround: https://github.com/google/jax/issues/2952
Would be great if the pocketfft OP could also be integrated into XLA itself, so that both TF and PocketFFT benefit from the speedup.

Is there any workaround for GPU? Most of my computational cost is just going to compute FFT. PyTorch's implementation is also very fast.

Are there any plans to address this issue any further? It's a huge inconvenience for anyone wanting to perform FFT transformations in a deployed model.

@diggerdu !
I could replicate this issue in 2.11 CPU version. Attached gist  and output for reference.
Colab only includes TensorFlow 2.x; %tensorflow_version has no effect.
2.11.0
 GPUs exists False
seconds (lower is better):
Tensorflow 2.11.0 8.614034000000004
Numpy:  2.022543939000002
WARNING:jax._src.lib.xla_bridge:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
Jax:  1.7004664330000026

Thank you!

I tried to execute the mentioned code on tf-nightly(2.17.0-dev20240403) on CPU and observed that the time take for the execution on tensorflow is lesser than numpy.
Tensorflow 2.17.0-dev20240403 1.0675210610000079
Numpy:  1.7257418959999882
Jax:  2.5296104049999997

Kindly find the gist of it here. Thank you!

Yes, we now use DUCC FFT in all of TF, JAX, XLA.  This is resolved.


Add quantum volume to backend configuration

Add quantum volume to the backend configuration schema. The version remains unchanged since it's an existing field (the schema just never got updated).

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Various minor fixes for blocks-plot

This fixes three issues:

bin/blocks-plot.py was still importing plot from blocks instead of blocks.extras.
plotting multiple experiments in one plot would not show all datapoints when the TrainingLogs have different lengths. The extended unit test now covers this case.
spaces in interactive channel specs are now now stripped: "train_cost, valid_cost, test_cost" will now work although there are spaces behind the comma.

Ups -- my commits were on top of some older version. I rebased onto current master.


Move to deploy?

Deploy now supports Deno.connect, meaning we can use postgres inside deploy and remove docker

But this means we'd have to host the database somewhere else...


[Feature] route trait

Can you support a service with multiple domains, when I use route traits

Can you clarify your use case?

name: nginx services:   nginx:     type: webservice     image: nginx:1.8     port: 80     route:       domain: www.nginx.com
There's an example where I can visit my nginx  through www.nginx.com, Now I'd like to add an extra domain to my nginx as well, route trait don't support it now

In your case, the nginx is not an App, it's API-Gateway. This should use api-gateway as a capability instead of an app. We will consider support api-gateway in the near future.

I just accidentally wrote a nginx。 Actually I need an app with multiple domain

@huolunl Why do your  app need two domain?

We can add K8s nginx as a trait, and set conflictWith field with route trait. So you will be able to use multiple domain route by raw ingress.

We can add K8s nginx as a trait, and set conflictWith field with route trait. So you will be able to use multiple domain route by raw ingress.

This issue can be resolved now, you can define a customized ingress trait by using cue @huolunl
please read the doc:
https://kubevela.io/#/en/platform-engineers/advanced-cue?id=trait-generate-multiple-resources

This issue can be resolved now, you can define a customized ingress trait by using cue @huolunl
please read the doc:
https://kubevela.io/#/en/platform-engineers/advanced-cue?id=trait-generate-multiple-resources


add grabCssFrom

@DavertMik please review this feature!
this is the adapted version of http://webdriver.io/api/property/getCssProperty.html

Looks like your test doesn't correspond to feature, so tests are failing. Please fix.
Also just curious, can you describe your use-case for this feature?

@DavertMik you are right I got confused with grabAttribute...
so there is no real need for this


Swap the order of the CORS and ContextRewrite valves

Closes https://github.com/wso2/product-is/issues/10132

PR builder: https://github.com/wso2/product-is/actions/runs/360998368


serverpod run not failing to start server

Description of issue
First of all, this is the first time I'm doing anything with serverpod, so I might have missed something obvious.
I've followed the Getting Started as documented here https://docs.serverpod.dev/ for version 0.9.10.
Steps to reproduce

Already had Flutter on my machine (version 3.3.1), stable branch.
Installed docker desktop for Mac M1 (version 4.12.0)
Executed dart pub global activate serverpod_cli without problems
Executed serverpod and succesfully saw the help output
Executed serverpod create mypod. This has generated 1 folders with 3 subfolders as expected.
Executed cd mypod/mypod_server -> No issues.
Executed docker-compose up -d --build -> No issues.
Executed serverpod run -> 2 errors, see below.

During step 8, this is the output I got:

View output from image above in text
Output
Starting Serverpod.

• Automatic generate and reload are enabled.

Spinning up serverpod generate (this can take a few seconds).
Waiting for Postgres on localhost:8090.
Waiting for Redis on localhost:8091.
Setup complete. Starting the server.
../../../../.pub-cache/hosted/pub.dartlang.org/serverpod_postgres_pool-2.1.1/lib/postgres_pool.dart:560:28: Error: The method 'PgPool.query' doesn't have the named parameter 'useSimpleQueryProtocol' of overridden method 'PostgreSQLExecutionContext.query'.
  Future<PostgreSQLResult> query(
                           ^
../../../../.pub-cache/hosted/pub.dartlang.org/postgres-2.5.0/lib/src/execution_context.dart:52:28: Context: This is the overridden method ('query').
  Future<PostgreSQLResult> query(String fmtString,
                           ^
../../../../.pub-cache/hosted/pub.dartlang.org/serverpod_postgres_pool-2.1.1/lib/postgres_pool.dart:727:28: Error: The method '_PgExecutionContextWrapper.query' has fewer named arguments than those of overridden method 'PostgreSQLExecutionContext.query'.
  Future<PostgreSQLResult> query(String fmtString,
                           ^
../../../../.pub-cache/hosted/pub.dartlang.org/postgres-2.5.0/lib/src/execution_context.dart:52:28: Context: This is the overridden method ('query').
  Future<PostgreSQLResult> query(String fmtString,

It stays there for at least 5 minutes (I cancelled after that). Any suggestions?
Within Docker, I see that 2 out of 3 containers are running. Not sure if normal:

View logs of that exited Postgress run container
Output
CREATE TABLE
ALTER TABLE
CREATE INDEX
CREATE TABLE
ALTER TABLE
CREATE INDEX
CREATE INDEX
CREATE TABLE
ALTER TABLE
CREATE INDEX
CREATE TABLE
ALTER TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE TABLE
ALTER TABLE
CREATE INDEX
CREATE TABLE
ALTER TABLE
CREATE INDEX
CREATE TABLE
ALTER TABLE
CREATE INDEX
CREATE TABLE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE TABLE
ALTER TABLE
CREATE INDEX
ALTER TABLE
CREATE TABLE
ALTER TABLE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE INDEX
ALTER TABLE

Flutter client
I'm able to run the Flutter mypod example project that got generated from above steps. However as seen in image below, it fails to connect to the server.

When I check the logs of both Docker containers that are still running (redis & postgress), I couldnt find any mentions regarding port 8080, 8081, 8090 or 8091.
I did spot the following ports mentioned:

5432 (postgress container)
6379 (redis container)

I've solved the issue
I can submit a PR with this fix in the serverpod_postgres_pool repo but I still want to check whether I did something wrong on my end. I'll wait for replies.
How to fix
I'm digging a bit deeper and the error message seems to be right, when looking in execution_context.dart I noticed that we do indeed use bool? useSimpleQueryProtocol as a named parameter that we want to override:

However when looking at the source, postgres_pool.dart, there's no such name (the error was spot on :)):

So, I've added that useSimpleQueryProtocol named argument in 2 places within postgres_pool.dart and that seems to do the trick.

Another issue, another day.
I do run into another issue now..something about CPU health check failed. But thats for another day.


wrong chunk sizes in html repr with nonuniform chunks

What happened:
The HTML repr is using the first element in a chunks tuple;
What you expected to happen:
it should be using whatever dask does in this case
Minimal Complete Verifiable Example:

import xarray as xr
import dask

test = xr.DataArray(
    dask.array.zeros(
        (12, 901, 1001),
        chunks=(
            (1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1),
            (1, 899, 1),
            (1, 199, 1, 199, 1, 199, 1, 199, 1, 199, 1),
        ),
    )
)
test.to_dataset(name="a")

EDIT: The text repr has the same issue
<xarray.Dataset>
Dimensions:  (dim_0: 12, dim_1: 901, dim_2: 1001)
Dimensions without coordinates: dim_0, dim_1, dim_2
Data variables:
    a        (dim_0, dim_1, dim_2) float64 dask.array<chunksize=(1, 1, 1), meta=np.ndarray>

that's because we're actively picking the first element of each chunk: https://github.com/pydata/xarray/blob/9c85dd5f792805bea319f01f08ee51b83bde0f3b/xarray/core/formatting.py#L233
Maybe it's time to finalize _repr_inline_ and try to get dask to implement it? Or, if that doesn't work, we could also monkeypatch it onto dask arrays, similar to what pint-xarray is doing with pint quantities.

I agree. We should outsource this to dask

for that to work I guess we first have to improve the documentation on _repr_inline_ (see https://github.com/pydata/xarray/pull/4248#discussion_r466546100)


Wrong syntax {}

@startuml
left to right direction
skinparam packageStyle rectangle
actor customer
actor clerk
rectangle checkout {
customer -- (checkout)
(checkout) .> (payment) : include
(help) .> (checkout) : extends
(checkout) -- clerk
}
@enduml

RP #34


Remove Session Replay from public API

What does this PR do?
Temporary commit to remove Session Replay from the public API. It will be reverted shortly after, needed to do the SDK v2 content merge into develop branch.
Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] Make sure you discussed the feature or bugfix with the maintaining team in an Issue
[ ] Make sure each commit and the PR mention the Issue number (cf the CONTRIBUTING doc)

Codecov Report

Merging #1235 (e27490f) into feature/sdkv2 (7b37814) will increase coverage by 0.07%.
The diff coverage is 100.00%.

@@                Coverage Diff                @@
##           feature/sdkv2    #1235      +/-   ##
=================================================
+ Coverage          82.75%   82.81%   +0.07%
=================================================
  Files                358      358
  Lines              12623    12579      -44
  Branches            2098     2091       -7
=================================================
- Hits               10445    10417      -28
+ Misses              1528     1524       -4
+ Partials             650      638      -12

Impacted Files
Coverage Δ

...oid/src/main/kotlin/com/datadog/android/Datadog.kt
77.36% <ø> (+2.81%)
:arrow_up:

...main/kotlin/com/datadog/android/DatadogEndpoint.kt
0.00% <ø> (ø)

...src/main/kotlin/com/datadog/android/DatadogSite.kt
100.00% <ø> (+13.33%)
:arrow_up:

.../kotlin/com/datadog/android/v2/core/DatadogCore.kt
85.59% <ø> (-0.70%)
:arrow_down:

...atadog/android/core/configuration/Configuration.kt
98.46% <100.00%> (-0.13%)
:arrow_down:

.../main/kotlin/com/datadog/android/plugin/Feature.kt
100.00% <100.00%> (ø)

...rsistence/file/batch/PlainBatchFileReaderWriter.kt
81.54% <0.00%> (-1.54%)
:arrow_down:

...lin/com/datadog/android/rum/internal/RumFeature.kt
92.03% <0.00%> (ø)

...android/log/internal/logger/TelemetryLogHandler.kt
85.71% <0.00%> (ø)

... and 8 more


corrections to the spec

Recently identified bugs in the spec:

Fixing some minor omissions about constraints on package name length and depth of nested messages.
Fixing an omission about the constraints of octal and hexadecimal integer literals. Until recently, decimal literals had the same constraint, but were then treated inconsistently when a negative sign was applied (based on whether the context was a normal or custom option value or a field default value). Instead of describing that behavior, the spec now describes the behavior of the next release of protoc (which will include this fix).
The final change is a straight up error in the spec. I think I wrote the paragraph on extensions in message literals from memory instead of looking at actual code. I even fixed this recently in protoparse, but failed to recall when I wrote this. This fixes that mistake.

@pkwarren, you'll notice that these are all related to recent changes in protocompile (yours and mine). So hopefully you don't mind me tagging you as the reviewer for this repo, too :)


[Bug] BD incompatible with Latest Discord Update.

Describe the Bug
 Wont load or anything and can't move or close the windows and the application think your are in a Browser.
To Reproduce
 Install latest BD version and have fun.
Expected Behavior
 Make the Discord App like on a browser, so can't close or move the window and nothing load about BD.
Discord Version
 Stable 81452 (1c03382)
Additional Context
 Are you guys working on it ?

Please try the latest installers: https://github.com/BetterDiscord/Installer/releases/latest

FIXED. Thanks.


Customer Churn Prediction

Project-Guidance! 🎊🎈
Hi, I am Simran Shaikh. my contribution to open source project for Customer Churn Prediction the most important topic that show that why  the process of analyzing data to identify which customers are likely to cancel a company's products or services .
Here is the screenshots:

Hi there!
I am the Mentor of this project
Although this repo is not part of Hacktoberfest 2024, we have some great alternatives for you. Welcome to SkillShow, a repository focused on skill enhancement!
Guidelines:

Please adhere to the existing guidelines.
No additional rules, thanks!
Check it out here: SkillShow.

Note: Start by creating an issue. Once it’s assigned to you, feel free to open a pull request (PR). Directly opening a PR will lead to an invalid label.
Looking forward to your contributions!

@Kushal997-das  is there any changes ?

@SimranShaikh20 Kindly mention your issue number.

@SimranShaikh20 Kindly mention your issue number.

Added!

@SimranShaikh20  You have created an invalid PR, as your changes affect all folders. Please check and fix this; otherwise, we will have to close it.


/data folder doesn't seem to work

When I add a data folder as I've done here -> https://github.com/UKDataServiceOpen/r-course-starter/tree/master/data
running
# test read in the dataset from /data/test.txt
df <- read.table('data/test.txt', header = FALSE)

causes
Warning message in file(file, "rt"):
“cannot open file 'data/test.txt': No such file or directory”
Error in file(file, "rt"): cannot open the connection
Traceback:

1. read.table("data/test.txt", header = FALSE)
2. file(file, "rt")

meanwhile
# read from web hosted files
df <- read.table("https://s3.amazonaws.com/assets.datacamp.com/blog_assets/test.txt",
                 header = FALSE)

works

I've tried it with a .csv and it does work, I am guessing somewhere only .csvs are allowed through as datasets but this wasn't clear in the docs.
Thanks anyway Ines!

looking at this again I need to load in SPSS .sav files too, so I might have to convert it to .CSV and lose some benefit in this process.


obey int at argument when launching GUI

fixes https://github.com/micahflee/torbrowser-launcher/issues/628

Closing in favor of #642


How to visualize the result?

Hello, thank you for your good works. I tried your code and it works well, may i inquire how to visualize the result as showed in your video result? Thank you very much.

Thanks for your interest in our work. We draw the bounding boxes and trajectories using Open-CV in Python. Specifically, we use the cv2.rectangle() for bounding boxes, cv2.line() for the trajectory, and cv2.addWeighted() to create the transparent coloured boxes. We're not releasing the visualization code, but you should be able to create the same visualization with a combination of these functions.


Drone on Kubernetes

I am running Drone as docker container on top of kubernetes.
Kubernetes pod has floating ip and private ip.
When i do github authentication using the pod public ip, it gets authenticated and github redirects to the drone app. When i make a commit to my github repo, drone detects it, but fails with the following error.
ssh: Could not resolve hostname github.com: Name or service not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Looks like github is unable to find right auth of pod. Did anyone face this issue?
If yes, do you guys think its because of pod floating and private ip?

ssh: Could not resolve hostname github.com: Name or service not known
This is a dns issue. Do a nslookup github.com or use dig github.com from your drone container. If that is not working grab the dns information for the container cat /etc/resolv.conf and try to use a valid dns server.

If yes, do you guys think its because of pod floating and private ip?

Use a kubernetes service for accessing your drone pod.

Is it not the error that you get when github is not available? There's something you can try: pull from https and see what happens, if it works, it's a problem with github, if it does not, it's a problem with your DNS

@mariomarin and @ipedrazas. Its not a DNS issue. I login into the container and did a ping of github.com and its able to ping.
Using Kubernetes service for accessing pod works fine. As part of github authrorization callback url, if i give service ip, it works fine. But when the container tries to clone github, it fails authorization. It is because github authorized kubernetes service ip and not the pod/container ip.
Outgoing traffic goes through vm/server and does not use service, GIT is unable to authorize. Any idea how to solve this issue?

@prashantchitta the git clone is not happening from the drone container. Drone spawns a new container for the build, which in turn spawns a new container for each step in the build process. The git clone happens inside its own container.
so the entire build process is probably happening outside of kubernetes all together and communicating directly with the machines Docker daemon. Any kubernetes / pod setting would therefore not be passed on to the build containers.

@bradrydzewski Wondering if anyone has setup LB(load balancer) entry for Drone? If yes, then how does git auth work? Because incoming traffic to drone and git auth works for LB ip. But, outgoing traffic, such as git clone will not go through LB and as a result, git auth fails. Isnt it?

@bradrydzewski Look at issue # 1305 for more details on what I am trying to do

@prashantchitta unfortunately I'm not sure I can offer much help here since I don't have experience with Kubernetes. I'm a bit unsure how #1305 differs from this issue.

@bradrydzewski May be i should change the subject. Its not releated to Kubernetes. Its about how do you set Load Balancer entry to a drone container. Containers tend to die. So i wanted to create a LB ip so that, when a container dies and a new container comes up, i can point my LB ip to the new container.
The advantage with above approach is, my drone url never changes, and drone auth callback url inside github does not change.

@prashantchitta regarding the git errors, have you attempted to run the git clone plugin on your servers to isolate the issue. You should, for example, be able to run the following command:
docker run -i plugins/drone-git <<EOF
{
    "repo": {
        "clone_url": "git://github.com/drone/drone"
    },
    "build": {
        "event": "push",
        "branch": "master",
        "commit": "427560a52d87ea9ee1a6dce65e9d7161995a8f94",
        "ref": "refs/heads/master"
    },
    "workspace": {
        "root": "/drone/src",
        "path": "/drone/src/github.com/drone/drone"
    }
}
EOF

@prashantchitta create an RDS instance, set the same session secret and define your ELB...
it doesn't work always... and if you haven't tried, creating an ELB in front of a bunch of docker_hosts will not work either.
If you're not using secrets, you just need to set the session secret to the same value. Then the token is always valid (regardless the node the LB uses).
SESSION_SECRET=97638eaa535c0ad1ae468c9b5ad780835446a585
Still, drone is not ready to work the way you want to. I'm looking at ways of working around the limitations (of running apps in kubernetes)

@prashantchitta is this stack overflow relevant?
http://stackoverflow.com/questions/25886862/a-ssh-issue-when-tried-to-rebuild-a-node-js-image-with-drone-io
Also I'm wondering why it is cloning with git+ssh since the system uses git+https by default to clone GitHub repository. Perhaps you could provide more of your build output (including the git clone command)? Does this repository use submodules? Also could you provide your Drone version (check the X-Drone-Version http header)?

closing due to lack of activity and because we have multiple successful kubernetes installations. If we can get more details or get someone else to repeat the issue we can re-open
https://discuss.drone.io/t/example-drone-kubernetes-manifests/31


packaging tweaks for compilation issues.

Remove typings install.  Remove typings.json config in favor of @types/XXX dev dependencies.  Switch to <reference types... instead of <reference path....
Tests pass and this works for my project.  This may fix #69

There's still a ref to typings in Highcharts.ts

Let's try to merge and check what ppl say about this fix in the master branch. Then and will make release and publish.


feat: Add popular collections on AHK to sidebar on explore

Thank you for your contribution to the KodaDot - One Stop Shop for Polkadot NFTs.
👇 __ Let's make a quick check before the contribution.
PR Type

[x] Bugfix
[x] Feature
[ ] Refactoring

Needs QA check

@kodadot/qa-guild please review

Context

[x] Closes #7131
[ ] Requires deployment <snek/rubick/worker>

Did your issue had any of the "$" label on it?

[x] My DOT address: Payout

Screenshot 📸

[x] My fix has changed UI

Copilot Summary

🤖 Generated by Copilot at ce5fd6e
Added ahk chain to popular collections filter and improved its UI. Removed unused code and simplified the display of collection chain names using CSS.

🤖 Generated by Copilot at ce5fd6e

We filter the collections of doom
We choose the ahk chain of gloom
We purge the code of useless waste
We style the names with CSS grace

pay 30 usd

😍 Perfect, I’ve sent the payout
💵 $30 @ 4.27 USD/DOT ~ 7.026 $DOT
🧗 16SjUbGKSdjCdWTy3NNT3JxbRVGGqD4mwkHpc6BD9U2Rp29Z
🔗 0x7b9d2176e45609d0e15e2be00a8781472ea5938f092a8050d23265899734a3b2
🪅 Let’s grab another issue and get rewarded!
🪄 github.com/kodadot/nft-gallery/issues


[libde265] build failure in ARM (Apple M1)

Host Environment

OS: macOS ARM
Compiler: Xcode 13.0

To Reproduce
Steps to reproduce the behavior:
./vcpkg install libde265
Additional context
libde265 uses by default SSE and x86 instructions by default. The portfile requires this for ARM builds:
vcpkg_configure_cmake(
    ...
    OPTIONS
        -DDISABLE_SSE=ON
    ...

Thanks,

@JackBoosY  can you solve this issue as you have solved #20930 ?


Ggarra13 patch 2

Pre-submission checklist:
Please check each of these after submitting your pull request:

[x] Are you including a repo_url? You need to pick the single best GitHub repository for your project, not a GitHub organization.
[x] Is your project open source, and does your project have at least 300 GitHub stars?
[ ] Has this project been used successfully in the production of a film, television program, animated content or other projects within the scope of the ASWF. Was it designed specifically for the entertainment industry?
[x] Have you picked the single best (existing) category for your project?
[x] Does it follow the other guidelines from the new entries section?
[x] Have you included a URL for your SVG or added it to hosted_logos and referenced it there?
[x] Does your logo clearly state the name of the project/product and follow the other logo guidelines?
[x] Does your project name match the text on the logo?
[x] Have you verified that the Crunchbase data for your organization is correct (including headquarters and LinkedIn)?
[ ] ~10 minutes after opening the pull request, the CNCF-Bot will post the URL for your staging server. Have you confirmed that it looks good to you and then added a comment to the PR saying "LGTM"?

ggarra13 Please LGTM if this looks OK.

@ggarra13 Please add an LGTM if https://ggarra13-patch-2--awsf-landscape.netlify.com/selected=mr-viewer looks good to you.

El 4/10/19 a las 15:08, Dan Kohn escribió:

@ggarra13 https://github.com/ggarra13 Please add an LGTM if
https://ggarra13-patch-2--awsf-landscape.netlify.com/selected=mr-viewer
looks good to you.

LGTM.  However, the LICENSE is GPL 2 or 3 to user's choosing and
libraries permitting.
--
Gonzalo Garramuño

The license is determined by GitHub running https://github.com/licensee/licensee.


Default storage driver to "" and let provisioner choose default

The storage driver will be "" by default an the provisioner must choose
their default provisioner.  If the user chosen storage driver is not
supported the provisioner should return an error.
I changed boot2docker to throw an error if something besides aufs is chosen.  I have no clue what boot2docker supports so that may be wrong.

Looks like b2d supports all storage drivers.  I will have to update that.

Updated to allow drivers for b2d.

LGTM

cc @ehazlett

LGTM


Datarace using pinpoint identifyUser

Describe the bug
Using both Amplify.Notifications.Push.identifyUser  and Amplify.Analytics.identifyUser  causes crashes due to datarace.
The documentation is not very clear about which identifyUser I should use so I call both, but in production, a lot of crashes occur.
Steps To Reproduce
Here the code:

let analitycsUserProfile = AnalyticsUserProfile(name: user.username,
                                                email: user.email,
                                                 plan: previousPurchaseStatus?.rawValue,
                                           properties: [
                              "notificationEnabled" : previousNotificationEnabled,
                                             "lang" : Locale.current.identifier,
                                     "lastRateDate" : previousLastRateDate?.timeIntervalSince1970 ?? 0,])
Amplify.Analytics.identifyUser(userId: user.username, userProfile: analitycsUserProfile)

let pushUserProfile = BasicUserProfile(name: user.username,
                                      email: user.email,
                                       plan: previousPurchaseStatus?.rawValue,
                           customProperties: [
                             "notificationEnabled" : previousNotificationEnabled,
                                            "lang" : Locale.current.identifier,
                                            "lastRateDate" : previousLastRateDate?.timeIntervalSince1970 ?? 0,])
try await Amplify.Notifications.Push.identifyUser(userId: user.username, userProfile: pushUserProfile)

### Expected behavior

No crash

### Amplify Framework Version

2.23.0

### Amplify Categories

Analytics

### Dependency manager

Swift PM

### Swift version

5.9.2

### CLI version

12.8.2

### Xcode version

15.1 (15C65)

### Relevant log output

```shell
Thread 14 Crashed:
0   libswiftCore.dylib            	0x00000001a49f9034 swift_isUniquelyReferenced_nonNull_native + 0 (SwiftObject.mm:1375)
1   Whosnext                      	0x000000010362001c specialized Dictionary._Variant.isUniquelyReferenced() + 8 (<compiler-generated>:0)
2   Whosnext                      	0x000000010362001c specialized Dictionary._Variant.setValue(_:forKey:) + 52
3   Whosnext                      	0x0000000103630ef8 specialized PinpointEndpointProfile.setCustomProperty(_:forKey:) + 364 (<compiler-generated>:0)
4   Whosnext                      	0x000000010362f4a8 PinpointEndpointProfile.setCustomProperty(_:forKey:) + 40 (PinpointEndpointProfile.swift:0)
5   Whosnext                      	0x000000010362f4a8 PinpointEndpointProfile.addCustomProperties(_:) + 380
6   Whosnext                      	0x000000010362f01c PinpointEndpointProfile.addUserProfile(_:) + 384 (PinpointEndpointProfile.swift:65)
7   Whosnext                      	0x0000000103642f10 AWSPinpointPushNotificationsPlugin.identifyUser(userId:userProfile:) + 152 (AWSPinpointPushNotificationsPlugin+ClientBehaviour.swift:26)
8   Whosnext                      	0x0000000103644e89 protocol witness for PushNotificationsCategoryBehaviour.identifyUser(userId:userProfile:) in conformance AWSPinpointPushNotificationsPlugin + 1 (<compiler-generated>:0)
9   Whosnext                      	0x000000010278606d PushNotificationsCategory.identifyUser(userId:userProfile:) + 1 (PushNotificationsCategory+ClientBehaviour.swift:13)
10  Whosnext                      	0x000000010259ffd5 AnalyticsManager.updateUser(newUser:) + 1 (AnalyticsManager.swift:187)
11  Whosnext                      	0x000000010259d4a1 closure #1 in closure #1 in AnalyticsManager.startListenTasks() + 1 (AnalyticsManager.swift:150)
12  Whosnext                      	0x00000001025a8d4d partial apply for closure #1 in closure #1 in AnalyticsManager.startListenTasks() + 1
13  Whosnext                      	0x000000010258a579 specialized thunk for @escaping @callee_guaranteed @Sendable @async () -> (@out A) + 1
14  Whosnext                      	0x00000001025a8d91 thunk for @escaping @callee_guaranteed @Sendable @async () -> (@out A)specialized partial apply + 1
15  libswift_Concurrency.dylib    	0x00000001b63f7ac5 completeTaskWithClosure(swift::AsyncContext*, swift::SwiftError*) + 1 (Task.cpp:463)

Is this a regression?
No
Regression additional context
No response
Platforms
iOS
OS Version

iOS 16.6

Device
iPhone / iPad
Specific to simulators
No
Additional context
crashs.zip
3 crashes downloaded from Xcode

Thanks for reporting this, we'll investigate and provide an update as soon as we have more information.

Hi @gbitaudeau! Both Notifications.Push.identifyUser and Amplify.Analytics.identifyUser do the exact same thing, which is to update the local endpoint information and send a updateEndpoint API request to Pinpoint.
This is likely what's causing the race condition, so while I work on a fix you can try removing one invocation as a potential quick fix.

Hi @ruisebas, I will try and let you know the result. Thanks


Map Fields Between Open Connectors and HubSpot

Tutorials: https://developers.sap.com/tutorials/cp-apim-openconnectors-common-resources.html
Hi,
When trying to add common resource template no templates appear to be loading. Screenshot attached for reference.
Thanks

This tutorial will soon be deprecated.  Thank you.


Rearrange graphs

Description
Switching the market view to PC1-left and the supplydemand view to Hub1. Also removing two graphs from the market view (these are duplicates of graphs that already appear on the marketsreserve view)
Close #78
Type of change

[ ] New feature (non-breaking change which adds functionality)
[ ] Optimization (non-breaking, back-end change that speeds up the code)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Breaking change (whatever its nature)

Key checklist

[x] All tests pass (python -m pytest)
[x] Pre-commit hooks run successfully (pre-commit run --all-files)

@pclemow may be a useful reference if you want to move around any graphs in the future

I wonder if we can organise the addition of the plots in a more succinct manner, given that they are all identical and just changing the content, with a loop and some clever maps to get the right row and col. But most certainly not an urgent thing to do!

Yeah good point. When I first wrote GridBuilder the figures were a mix of html.Img objects and plotly figures, so I needed to keep it flexible enough to take different object types. Since this pull request they are all plotly figures so in theory we could simplify everything


Issue with shortcut assistant

Small issue with the shortcut assistance preset. Aurora seems to always think I am holding down alt. has been working fine for the past week, then yesterday it starting doing this, resulting my my, ctrl, tab, f4, v and E always being yellow. not sure why.
Anyone else have this issue and know how to resolve it? Tried reseting the profile and that didn't work, issue still persisted on the default.

More information on this. Seems to happen whenever I first boot it up, or if I open a new application. For example, I launch Hearthstone and the issue occurs. Hit alt and it goes away.

Hm, I'm not sure, I haven't been able to replicate it. I'll try and have a look at it to see if I can see anything out of place.

Could you try this build and let me know if it solves your issue.

I've downloaded and extracted the file, but what am I supposed to do with
it from here? I don't see any executable, do I simply over write the
existing aurora files with this? Where is the existing aurora app stored? I
can't find it locally.
On Wed, May 30, 2018 at 3:53 AM, Simon W<EMAIL_ADDRESS>wrote:

Could you try this build
https://ci.appveyor.com/project/antonpup/aurora/build/0.6.1-dev4-319/artifacts
and let me know if it solves your issue.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/antonpup/Aurora/issues/929#issuecomment-392829504,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AhSZHdbrF36stvlU17tE3bQUjwMjfCjCks5t3W7ngaJpZM4RP-FR
.

Apologies, must have missed that (no coffee yet today). Will check again.
On Thu, May 31, 2018 at 9:26 AM, Simon W<EMAIL_ADDRESS>wrote:

@BeardyMcBeardFace https://github.com/BeardyMcBeardFace I just
downloaded it and there is "Aurora.exe", all you need to do is extract the
zip into a folder and run Aurora.exe.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/antonpup/Aurora/issues/929#issuecomment-393324844,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AhSZHUhYkQYsV6i3M-_ntJ5Lpk4Xpi3iks5t3w51gaJpZM4RP-FR
.

Seems to fix the issue when I load up hearthstone, which was the most
common cause. I'll keep at it and if it acts up again I'll let you know.
Many thanks :)
On Thu, May 31, 2018 at 9:27 AM, Andrew Cockell<EMAIL_ADDRESS>wrote:

Apologies, must have missed that (no coffee yet today). Will check again.
On Thu, May 31, 2018 at 9:26 AM, Simon W<EMAIL_ADDRESS>wrote:

@BeardyMcBeardFace https://github.com/BeardyMcBeardFace I just
downloaded it and there is "Aurora.exe", all you need to do is extract the
zip into a folder and run Aurora.exe.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/antonpup/Aurora/issues/929#issuecomment-393324844,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AhSZHUhYkQYsV6i3M-_ntJ5Lpk4Xpi3iks5t3w51gaJpZM4RP-FR
.

Awesome! I'll close this for now, it can be reopened if you run into the same issues as before again.


Error running test in Internet Explorer 11 (32bits)

Run this test in IE 11 ( 32 bits ) . It opens google.com in IE 11 (32bits) but it can not type react in searchBox.
- description: Navigate to Google
        action: org.getopentest.selenium.NavigateTo
        args:
          url: https://www.google.com

      - description: Type "react" in the search box
        action: org.getopentest.selenium.SendKeys
        args:
          locator: { name: q }
          text: react

Error stack:
Caused by: java.lang.RuntimeException: Failed executing action org.getopentest.selenium.SendKeys with arguments {"locator":{"name":"q"},"text":"react"}
at org.getopentest.base.TestActor.executeActionByDef(TestActor.java:1821)
at org.getopentest.base.TestActor.executeTestSegment(TestActor.java:2073)
... 3 more
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Failed sending keys to element By.name: q
at org.getopentest.base.TestActor.executeActionByClassName(TestActor.java:1626)
at org.getopentest.base.TestActor.executeActionByDef(TestActor.java:1705)
... 4 more

I just tested and it is an issue with old selenium client library in the class path.


Sleuth and Reactor Kafka

Hi,
i use spring-cloud-starter-sleuth and spring-kafka with reactor-kafka.
send and receive kafka messages by org/springframework/kafka/core/reactive/ReactiveKafkaProducerTemplate  and org/springframework/kafka/core/reactive/ReactiveKafkaConsumerTemplate classes.
traceid/spanid does not propagate over kafka messages, seems like SleuthKafkaAspect doesn't work with reactor-kafka.
is there another aspects for working sleuth with reactor-kafka?

Hi!
When you created this issue you had the following template in front of you
**Describe the bug**
Please provide details of the problem, including the version of Spring Cloud that you
are using.

**Sample**
If possible, please provide a test case or sample application that reproduces
the problem. This makes it much easier for us to diagnose the problem and to verify that
we have fixed it.

Could you please provide details that we ask of you in the template?

Describe the bug
test case
1 - run kafka on localhost
docker run --rm --network=local-apps -p 2181:2181 -p 3030:3030 -p 8081-8083:8081-8083 -p 9581-9585:9581-9585 -p 9092:9092 -e ADV_HOST=localhost -e ENABLE_SSL=1 -d --name kafka lensesio/fast-data-dev:latest
2 - send messages to kafka by org.springframework.kafka.core.reactive.ReactiveKafkaProducerTemplate
expected result = all messages in kafka contains traceId/spanId in headers
actual result = headers are empty in all kafka messages
Sample
https://github.com/no0ker/demo
code in this repo sends and receives kafka messages.

We don't instrument reactor kafka at this point. You can file an issue in https://github.com/openzipkin/brave repository so that such support is added and then we'll autoconfigure it in Sleuth

best support ever

I wonder if this won't help https://github.com/spring-cloud/spring-cloud-sleuth/pull/1920 can you check the latest snapshots?

Hi @marcingrzejszczak thx for the replay, I saw you merged it on the 3.1.x branch, but I can't find that artifact published already.

https://repo.spring.io/libs-snapshot/org/springframework/cloud/spring-cloud-sleuth-instrumentation/3.1.0-SNAPSHOT/ and https://repo.spring.io/libs-snapshot/org/springframework/cloud/spring-cloud-dependencies/2020.1.0-SNAPSHOT/ (most likely I made a mistake and 2020.1.0 should be 2021.0.0 but it doesn't matter at this point).

I modified a bit a SampleConsumer and SampleProducer from https://github.com/reactor/reactor-kafka/tree/main/reactor-kafka-samples/src/main/java/reactor/kafka/samples by integrating this code with spring application and it seems that traceid's and spanid's are not generated:
2022-02-07 17:05:44.902  INFO [,,] 75563 --- [-sample-group-1] c.e.poc.SampleConsumer      : Received demo-topic-0@0
Received message: topic-partition=demo-topic-0 offset=0 timestamp=17:05:06:707 CET 07 Feb 2022 key=1 value=Message_1

Should it work in this case too ?

We have the following samples https://github.com/spring-cloud-samples/spring-cloud-sleuth-samples/tree/main/kafka-reactive-consumer and https://github.com/spring-cloud-samples/spring-cloud-sleuth-samples/tree/main/kafka-reactive-producer . Can you check them out and see if you're doing stuff in a similar way?

Thx, examples that you provided examples works well so the issue on my side.


Arrows cannot be rotated automatically

Describe the Bug
For example.
https://reactflow.dev/docs/examples/edges/custom-edge/

↑  I just want this effect, arrows can automatically rotate with the angle of the edge. But after deleting an edge, the arrow does not rotate automatically.

The arrow is fixed there and will not rotate automatically.
Even if I change the orient attribute, the arrow cannot automatically rotate.https://reactflow.dev/docs/examples/edges/markers/

Your Example Website or App
No response
Steps to Reproduce the Bug or Issue
1、Add egde's arrow
2、Set the orient property to "auto"
3、Arrows cannot be rotated automatically
Expected behavior
Arrows can automatically rotate with the angle of the edge
Screenshots or Videos
No response
Platform

OS: [e.g. macOS, Windows, Linux]
Browser: [e.g. Chrome, Safari, Firefox]
Version: [e.g. 91.1]

Additional context
No response

Hey @tjxtjxtjx It's the default SVG marker-end attribute behaviour https://developer.mozilla.org/en-US/docs/Web/SVG/Attribute/marker-end It depends how you draw the edge. We are not going to change how we draw the default edges.


[simd] Update include missing header files

Describe the pull request

What does your PR fix?

Update include missing header files

Which triplets are supported/not supported? Have you updated the CI baseline?

linux, windows

Does your PR follow the maintainer guide?

yes

If you have added/updated a port: Have you run ./vcpkg x-add-version --all and committed the result?

yes
If you are still working on the PR, open it as a Draft: https://github.blog/2019-02-14-introducing-draft-pull-requests/

Thank you for your PR.
After any modification, the number of port-version must be increased, the process is as follows:

Any modification
Modify the port-version in vcpkg.json
Commit the modified file and and simd/vcpkg.json
./vcpkg x-add-version simd (the same PR has been modified multiple times to add --overwrite-version)
Commit versions/s-/simd.json and versions/baseline.json
Push


Fix parsing of nameOfHolder for NFC scan

Current Behaviour
parsing of nameOfHolder is not returning properly
Expected Behaviour
Fix parsing of nameOfHolder to separate givenNames and surname depending on locale defined

Fixed via a33ad1ffd090d9ad8d2b28b1faa70ae97bdd14d8


Excluded Chapters Toggles

Adds 7 new buttons to the Settings Popout for Poptracker, with 7 new Chapter labels. Each button disables locations for the corresponding chapters, using Visibility Rules. Buttons can be added to slot data to automatically update if said information is added to the Apworld.

I should have mentioned in discord to take a look at the emotracker settings popout as it has a LCL marker already that should be able to get hooked into this, as well as some other stuff like dungeon shuffle that AP currently does not support. It was only excluded from poptracker because it was purely a visual feature for the broadcast view and poptrack doesn't support layering images on top of each other.


Typescript?

Is there any way to configure this to also work with typescript?

While Typescript is a superset of JavaScript, it is not one of the goals of this project to support it. However, someone has forked this syntax file into https://github.com/leafgarland/typescript-vim
I haven't try it (since I haven't worked with TS), but you could check it out.


Tests failing after INVALID_OVERRIDE_DIFFERENT_DEFAULT_VALUES warnings were removed

The following tests went from Pass -> MissingCompileTimeError on analyzer-asserts-linux, analyzer-asserts-mac, and analyzer-asserts-win
co19_2/Language/Classes/Abstract_Instance_Members/override_default_value_t01
co19_2/Language/Classes/Abstract_Instance_Members/override_default_value_t02
co19_2/Language/Classes/Abstract_Instance_Members/override_default_value_t03
co19_2/Language/Classes/Abstract_Instance_Members/override_default_value_t04
co19_2/Language/Classes/Abstract_Instance_Members/override_default_value_t05
co19_2/Language/Classes/Abstract_Instance_Members/override_default_value_t06
co19_2/Language/Classes/Instance_Methods/override_different_default_values_t01
co19_2/Language/Classes/Instance_Methods/override_different_default_values_t02
Looks like it's because the warnings were removed in https://github.com/dart-lang/sdk/commit/15be61ed9a0bdd88f6cf3718d51db1e20444116c but are still being used in those tests.

Thanks! I'd like to accept these failures. How do I do that?

I've approved them already. I did it by going down to your change on https://dart-ci.firebaseapp.com, hitting approve and adding this issue as a comment.

Thank you much! https://github.com/dart-lang/co19/commit/17f2b4079f55bb5beb8f8301970089c290590ae4 should tidy those up.

Looks good, thanks!


JS tool to edit JSON colors

I am trying to build a JS tool that can automate changing the colors of any color(able) shape or item or component (or whatever the technical term is in AE, developer here with 0 AE experience).
Understanding the JSON structure is not that straightforward from the research I've done, is there any documentation available that can help with this task?
What are the color properties in the JSON file exactly, and how is there placement designed? What should I look for in the JSON tree?
This tool can be helpful for using animations on Lottie Files, would make things much easier for many developers.
I would appreciate any help on this.

Made some progress.

Just published bodymovin-editor repo. You can find the main code I utilized here, everything else is pretty much just UI. Kindly advise on this.

looks awesome! I'm working on something similar for editing the json files and was also looking for documentation on the format itself. I think that could be really helpful for people who want to build supporting tools for bodymovin. For now I'm just figuring out the structure through trial and error. I can start working a basic form of documentation to get started, then have hernan chime in and fill in the gaps :)

Sorry about the delay, I want to give a more thorough answer during the weekend.
But does this help for starters?
https://github.com/bodymovin/bodymovin/tree/master/docs/json

BTW that tool looks awesome.
And also, if you have a good format for documenting the json, I'll check it out.

oh man not sure how I missed that. Already looks super helpful and thorough. thanks! I'll think about formatting.

@bodymovin Yes that's where I started looking, although this documentation does offer a good idea on how things are structured, it doesn't provide enough details on how to integrate all of the different parts together, this is where I struggled.
So far, I found out that looking into fl and st types in json.layers.shapes.it.type and json.assets.layers.shapes.it.type are the main components responsible for colors (through type.c.k), are there any others that I should be looking for?
In additions, many animations are failing as a result of bad recursion model my guess, structures differ and the tool need to take that into account to cover all cases.
Also, I understand that the color is represented as an array of color vector units, but what's the fourth component? "k": [r, g, b, ?].
I made some great progress today, used the first customized animation in production after editing its colors with this tool (also published it online for everyone to use while the development is being done in the open, feel free to chime in).
Waiting to hear more from you about the subject, thanks.

Just tested 60 random animations from Lottie Files, 67% success rate.

@sonaye it's the alpha value but it is not used.
If animations interpolate colors, instead of a single array, you will get an array of keyframes with a starting values and end value for each.

Made more progress, 98% success rate now on the same set (rarely fails to load). The only thing this tool needs at the moment would be to cover more edge cases, I still need to do more research on where should I be looking in the json, would appreciate some help on this.

@sonaye Nice! do you know any case that is not working so I can check it out?

Sure
https://www.lottiefiles.com/767-crying-emoji-reaction
https://www.lottiefiles.com/175-day-night-cycle
https://www.lottiefiles.com/61-octopus
https://www.lottiefiles.com/629-empty-box
https://www.lottiefiles.com/856-thumbs-up-grey-blue
https://www.lottiefiles.com/187-fish
https://www.lottiefiles.com/159-servishero-loading
Some editing some parts of them (NaN) will cause a crash, and some have missing colors.
And this is the only animation left so far that completely fails to load https://labs.nearpod.com/bodymovin/demo/the_lady/data.json.

Main algorithm is now located here https://github.com/sonaye/bodymovin-editor/blob/0b28f2be3ec1cd4cf66129b0d4d2d349442f24bf/src/utils.js#L19-L54.
And is being utilized here https://github.com/sonaye/bodymovin-editor/blob/0b28f2be3ec1cd4cf66129b0d4d2d349442f24bf/src/app.js#L154-L160.

very cool!
you should check at line 28 if prop.c.a equals 0 or 1.
If it equals 1, it means the color has keyframes.
Eventually it would be cool to display the interpolation values and allow to edit each keyframe value separately.

https://www.lottiefiles.com/download_public/570
can not edit the color of the explosion, just the arrow

@jiangbodev I faced this before with this collection, look for 0,0,0 in the JSON file and change it to match your color (they are unit vectors, so divide by 255), only one should be responsible if you find many, just as a temp. workaround. I am planning to support this eventually but recently I'be been occupied with work, will get to it soon enough though.

thanks @sonaye, another thing I think would be very helpful is to crop the white spaces around the image, take 570 file as example, it has 800*800 size, but the real image inside is very small, it could be better to provide a way to cut off the white spaces around it, to reduce the size.

@jiangbodev I don't think it's possible, it has to do with the original design itself.

Closing this for now, will continue the work in the dedicated repo.

Same for me, tested with Microsoft Edge, Mozilla Firefox, Chrome. The only that open the dialog to save the new json in Chrome

The next update will utilize file-saver which should fix any download issues.

Hi, sorry for interuption, but I am wondering how can I change the bacground color of the loader?

@Aki77777 Chances are the animation has a transparent background, double check on that, meaning that it would work with any background. Colors list is derived from the animation itself, if a color is not there then either the animation doesn't have it or it's an edge case that is not supported yet.

great tool, exactly what I needed.

how can i convert transparent color insted of white?

It is building some improper json files ending abruptly

For ex.
on this animation
https://lottiefiles.com/4761-collecting-money

For ex.
on this animation
https://lottiefiles.com/4761-collecting-money

I looked into the exported json file and it's been cut off causing the error.

Yes but if you extract directly from the website without the tool then it doesn't cut off

Yes but if you extract directly from the website without the tool then it doesn't cut off

Yes, also when I checked the json of the working file, I can't find any rgb or hex color except the #f2f2f2

Made more progress, 98% success rate now on the same set (rarely fails to load). The only thing this tool needs at the moment would be to cover more edge cases, I still need to do more research on where should I be looking in the json, would appreciate some help on this.

Thank you. You've been a great help already.

@sonaye Has filed the repo, i have some improvements to do in this tool, like highlight selected area in animation to change the color, a timeline to pause in exact frame. I love this tool! I Will fork and revive then.
https://github.com/sonaye/lottie-editor

Hi all. I've found an animation I would like to use but I can't edit the colour codes for each square (there are four). The editor doesn't display it at all, so I have to run it locally inside my React project. It's this animation here:
https://lottiefiles.com/4188-cuadros-rojos-giran
I don't use After Effects or Bodymovin' so I have no idea what the encoded values I need to enter are. "k": [                     0.86159620098,                     0.057677313861,                     0.057677313861,                     1                   ],
Does anybody know what the values are if I wanted to use the following colours:
#cfcfcf
#d1461c
Thanks all.

Those values are RGB but from a scale of [0,1]. Convert your hex values to RGB, divide each by 255 and you got the answer ;)


refactor!: license arguments spdx compliant

Main changes

:new: (feat)
:wrench: (chore)
:memo: (docs)
:arrows_counterclockwise: (refactor)
:microscope: (test)
:bangbang: (breaking change)

This PR is a BREAKING CHANGE. The main goal of this PR is to keep the current functionality but making the scanning licenses process SPDX compliant. For that, I've done the following:

Remove the utility functions related to the old approach for the matching process. We were using string or RegEx's passed to failOn to check if any of the packages' licenses matched the value. As we no longer need that, I've removed all the related functions and their tests
Remove the spdx-validator component introduced in the previous PR. The only reason for this component to exist was the wrapper around the spdx-expression-validate to throw an error when the license wasn't SPDX compliant. This was used to check the value of checkLicense option. I've moved this code from the index.js to runner.js, and control the error there. So there is no need to keep the spdx-validator component anymore, we import the SPDX packages on the runner.js and use them
failOn stays as a list and generateOutputOn has been changed to a list for consistency. I've updated its description and the type on the args.js file (the yargs setup), and the documentation.
Remove the args' middleware. The old functions were used to verify if the RegEx values passed to failOn were valid, and to parse the generateOutput licenses. These were meant to be passed as a comma separated list of licenses, which no longer applies.
Refactor the runner logic to use spdx-validator and spdx-expression-validate packages.

As mentioned before, the runner now controls the checkLicenses flow. If a value is passed, it will check it and finish without triggering the scanning process
The list of licenses passed to failOn and generateOutputOn options, if provided, must be a SPDX compliant. Otherwise, an error will be thrown. If all are correct, they are concatenated using the OR operand. The result of this operation is used for the matching process. So if executed with --failOn MIT GPL-1.0 0BSD, the SPDX expression to check the package licenses will be MIT OR GPL-1.0 OR 0BSD
Packages' licenses must also be SPDX. So if any of the package.json entries has a non-compliant license, it won't be taken into account for the scanning process. A warning message will be printed to inform what packages are in this situation. We could potentially include an option to fail if that's the case.

To sum up, the main functionality stays the same. However, the checking process will make use of the SPDX packages. This will provide support for the AND, OR and WITH syntax during the matching process, which previously wasn't possible.
Additional notes
README has been updated accordingly. We've found an issue in spdx-satisfies package. An explanation has been added to this document as well.
I have had some errors with eslint (incompatible packages). So I've reinstalled all the eslint dependencies and replaced the browser option by node on the .eslintrc.json
Context

:ticket: Closes / Relates #53

@nanotower is taking over this PR. As discussed, he will implement an alternative approach which will remove the licenses passed to failOn from the complete SPDX list. Then, will check whether every package satisfies at least one license of the resulting list.

@all-contributors please add @nanotower for code

@all-contributors please add @nanotower for code


Design (and implement?) support for Prometheus metrics in image-based gadgets

The design of the prometheus support is still missing in https://github.com/inspektor-gadget/inspektor-gadget/blob/main/docs/design/002-containerized-gadgets.md#prometheus.
This should be probably done after https://github.com/inspektor-gadget/inspektor-gadget/issues/1598 to apply all the learning from there.

Closing in favor of #1598.


Add option to not log input prompt for requests that fail

Feature request
When there is an error handling a request, TGI logs an error message along with the input prompt. Below is an example where the user exceeded the max input token. For data privacy reason, there should be an option to redact the prompt as I've done below.
2023-11-02T15:33:55.212716Z ERROR HTTP request{otel.name=POST / http.client_ip= http.flavor=1.1 http.host=<model-name> http.method=POST http.route=/ http.scheme=HTTP http.target=/ http.user_agent=Python/3.11 aiohttp/3.8.5 otel.kind=server trace_id=ed599b385f39ecc3f4bdba02254d1a0e}:compat_generate{default_return_full_text=false}:generate{parameters=GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: None, top_k: None, top_p: None, typical_p: None, do_sample: false, max_new_tokens: 1, return_full_text: Some(false), stop: ["\nUser"], truncate: None, watermark: false, details: true, decoder_input_details: false, seed: None, top_n_tokens: None }}:generate{request=GenerateRequest { inputs: "<prompt>":", parameters: GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: None, top_k: None, top_p: None, typical_p: None, do_sample: false, max_new_tokens: 1, return_full_text: Some(false), stop: ["\nUser"], truncate: None, watermark: false, details: true, decoder_input_details: false, seed: None, top_n_tokens: None } }}:generate_stream{request=GenerateRequest { inputs: "<prompt>", parameters: GenerateParameters { best_of: None, temperature: Some(0.01), repetition_penalty: None, top_k: None, top_p: None, typical_p: None, do_sample: false, max_new_tokens: 1, return_full_text: Some(false), stop: ["\nUser"], truncate: None, watermark: false, details: true, decoder_input_details: false, seed: None, top_n_tokens: None } }}: text_generation_router::infer: router/src/infer.rs:110: `inputs` tokens + `max_new_tokens` must be <= 2048. Given: 2254 `inputs` tokens and 1 `max_new_tokens`

Motivation
When TGI is deployed to production, we don't want to log user input prompt for data privacy reason. Logging sensitive data will require adding more restrictions to our logging infrastructure, which we don't want to implement.
Your contribution
I can help test.

Thanks for reporting.


[release/1.1] cherry-pick: Increase reaper buffer size

Cherry-pick #2748
Cherry-pick #2770
This increases the buffer size for process exit subscribers. ~It also
implements a non-blocking send on the subscriber channel.  It is better
to drop an exit even than it is to block a shim for one slow subscriber.~
Signed-off-by: Michael Crosby<EMAIL_ADDRESS>Signed-off-by: Lantao Liu<EMAIL_ADDRESS>Signed-off-by: Phil Estes<EMAIL_ADDRESS>

Codecov Report

Merging #2762 into release/1.1 will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff              @@
##           release/1.1    #2762   +/-   ##
============================================
  Coverage        48.99%   48.99%
============================================
  Files               85       85
  Lines             7603     7603
============================================
  Hits              3725     3725
  Misses            3203     3203
  Partials           675      675

Flag
Coverage Δ

#linux
48.99% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 68a2cbc...e2ac712. Read the comment docs.

LGTM

I'm a bit uncomfortable about this. Init/exec process exit events come only once, and are very important for the container lifecycle. Discarding the event feels unsafe to me, and will cause state mismatch.
I think https://github.com/containerd/containerd/pull/2765 to fix the deadlock seems a better fix.

any opinions @crosbymichael? We already have the events change in 1.2/master, so whatever we decide, if we change anything we should remember to "fix" master as well.

At least for CRI plugin, if the container exit event is discarded, the container won't be stopped any more. Until user restarts containerd, and CRI recovers state by listing/inspecting all the containers.
Our current assumption in CRI is that event is reliable, and we can purely rely on it. If that is not the case, we'll probably need to add periodically relisting to sync state for each container.

We could revert the change for dropping the send and keep the larger buffer size in addition to the #2765 change

We could revert the change for dropping the send and keep the larger buffer size in addition to the #2765 change

SGTM. :) Thanks a lot!

@Random-Liu go ahead and send a patch.  I'm working on a fix for the pause and state issues right now that some have been seeing, the weird bug

Should I update this PR with just the buffer size change?

@Random-Liu updated!

LGTM

Looks like GitHub is a bit too eager hiding comments; here it's hiding "LGTM"'s


null array anti-pattern

if there is no objective, then Solver.getObjectives() returns null.
It would be more convenient to make it returns an empty array to avoid any NPE then require a special treatment in the code.

I changed the initialization to:
 this.objectives = new Variable[0];

and the Solver.setObjectives(Variable... objectives) to:
if(objectives == null){
    this.objectives = new Variable[0];
}else {
    this.objectives = objectives;
}

looks good to me