Vectorise AllTripletsMiner

AllTripletsMiner is implemented with the usage of cycles. Let's replace this implementation with vectorized operations

Hi! I really enjoy your library, and I would like to contribute to it somehow. I've read the Contribution guide, and following your suggestion, I looked at tasks 'good first issue' and here I am :-)
What sort of vectorization do you have in mind for AllTripletsMiner? I've done some research, and tried well-known matrix-based approach for triplets generation.
Specifically, I took implementation of triplets generation from pytorch-triplet-loss and I wrote the same code by means of numpy. Next, in order to perform performance tests, I've looked at the examples in the repository and checked batch sizes for train dataset. The biggest batch size is in the InShop example: it is 600 (30 * 5 * 4). I've generated labels of the size 600, set max_out_triplets equal to sys.maxsize, and run several versions of get_available_triplets: the original one, pytorch based on cpu, pytorch based on gpu and numpy based. The results are the following:
Original                 1.4906s
Pytorch CPU              2.0471s
Pytorch CUDA             1.7806s
Numpy                    2.2817s

But I've got a little improvement by removing redundant lists concatenation in inbatch_all_tri.py:L53 (it could be replaced with itertools.chain) which gives me
Original refactored      1.3124s

By the way, the original get_available_triplets could throw the error ValueError: not enough values to unpack (expected 3, got 0) if labels contains only unique elements. As far as I understand, if a user will use one of oml's sampler this situation couldn't happen, but if the sampler is from pytorch or a custom one, that in some cases this error could arise.

Hi, @dapladoc !
Nice to hear that you are interested in our repo :)

What sort of vectorization do you have in mind for AllTripletsMiner?
Any, the main idea is to get rid of cycles. I checked the implementation in PML, and it's okay, but I also thought about using TORCH.CARTESIAN_PROD, so I need some time to figure out if it's better.

Thank you for your experiments! The difference among implementations is not huge because the batch size is small. Although the batch size you mentioned is the biggest among configs, it may be much bigger. First, one may use resnet instead of vit. Second, one may use a heavy model, but freeze almost all of the layers. Third, one may use a memory bank.
So, could you please perform the same tests, but for batch sizes 10, 100, and 1000 times bigger? I guess you measured the performance averaging N calls of the function. So, I ask you to provide this number of calls as well for better understanding.
Could you please also paste your code here as snippets?
Looking forward for your answer.

As for errors, you are right, the only way to avoid them is to use a sampler from OML.
(Or a miner can skip classes with only 1 example, but I don't like this approach. One of its drawbacks is that effective batch size cannot be fixed.)

I did profiling of different variants of get_available_triplets, and it turned out that the slowest part was random sampling from the list of all triplets. Previously I used random.sample for torch-based and numpy-based variants as it was done in original code, because I thought that the for-loop was a bottleneck.
So, I rewrote torch-based and numpy-based variants of get_available_triplets so that they don't use random.sample anymore, and test their performance by mean of timeit.repeat with 20 runs (number = 1, repeat = 20). Unfortunately I don't have access to a powerful PC, so I couldn't perform tests with a really large batch size, I only tested batch sizes 600, 750, 900, 1050. The code is below. Sorry for the mess in the code and a lot of copy-paste, it was faster to do it that way

timeit code for get_available_triplets variants
import timeit
from collections import defaultdict
from functools import partial
from sys import maxsize
from typing import Any, Iterable, List

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import torch
from tqdm.notebook import tqdm

def timeit_original(n_labels, n_instances, number, repeat):
  setup = f"""
from typing import List, Iterable, Any
from random import sample
import numpy as np
from sys import maxsize
from itertools import product, combinations, chain, islice
import scipy
import torch

def find_value_ids(it: Iterable[Any], value: Any) -> List[int]:
  if isinstance(it, np.ndarray):
      inds = list(np.where(it == value)[0])
  else:  # could be very slow
      inds = [i for i, el in enumerate(it) if el == value]
  return inds

def get_available_triplets(labels: List[int], max_out_triplets: int = maxsize):
  num_labels = len(labels)
  triplets = []
  for label in set(labels):
      ids_pos_cur = find_value_ids(labels, label)
      ids_neg_cur = set(range(num_labels)) - set(ids_pos_cur)

      # (l0, l1, n) and (l1, l0, n) are 2 different triplets
      # and we want both of them
      pos_pairs = list(combinations(ids_pos_cur, r=2)) + list(combinations(ids_pos_cur[::-1], r=2))

      tri = [(a, p, n) for (a, p), n in product(pos_pairs, ids_neg_cur)]
      triplets.extend(tri)

  triplets = sample(triplets, min(len(triplets), max_out_triplets))
  ids_anchor, ids_pos, ids_neg = zip(*triplets)

  return list(ids_anchor), list(ids_pos), list(ids_neg)

n_labels = {n_labels}
n_instances = {n_instances}
labels = np.repeat(np.arange(n_labels, dtype='int'),  n_instances).tolist()
max_out_triplets = sys.maxsize
  """

  stmt = "out = get_available_triplets(labels, max_out_triplets)"""

  times = np.array(timeit.repeat(stmt=stmt, setup=setup, number=number, repeat=repeat)) / number
  return times

def timeit_torch(n_labels, n_instances, number, repeat, device):

  setup = f"""
from typing import List, Iterable, Any
from random import sample
import numpy as np
from sys import maxsize
from itertools import product, combinations, chain, islice
import scipy
import torch

def get_triplet_mask(labels, device, max_out_triplets):
  labels = torch.from_numpy(np.array(labels)).to(device)
  # Check that i, j and k are distinct
  indices_equal = (
      torch.eye(labels.size()[0], requires_grad=False)
      .type(torch.ByteTensor)
      .to(device)
  )  # make a index in Bool
  indices_not_equal = -(indices_equal - 1)  # flip booleans
  i_not_equal_j = torch.unsqueeze(indices_not_equal, 2)
  i_not_equal_k = torch.unsqueeze(indices_not_equal, 1)
  j_not_equal_k = torch.unsqueeze(indices_not_equal, 0)

  distinct_indices = (i_not_equal_j & i_not_equal_k) & j_not_equal_k

  # Check if labels[i] == labels[j] and labels[i] != labels[k]
  label_equal = torch.unsqueeze(labels, 0) == torch.unsqueeze(labels, 1)
  i_equal_j = torch.unsqueeze(label_equal, 2)
  i_equal_k = torch.unsqueeze(label_equal, 1)

  valid_labels = i_equal_j & ~i_equal_k
  # Combine the two masks

  mask = distinct_indices.to(device).bool() & valid_labels
  triplets = torch.nonzero(mask)
  p = torch.ones(len(triplets), device=device, requires_grad=False)
  idx = p.multinomial(
      num_samples=min(len(triplets), max_out_triplets), replacement=False
  )
  triplets = triplets[idx, :].T.cpu().numpy()
  ids_anchor, ids_pos, ids_neg = triplets
  return list(ids_anchor), list(ids_pos), list(ids_neg)

n_labels = {n_labels}
n_instances = {n_instances}
labels = np.repeat(np.arange(n_labels, dtype='int'),  n_instances).tolist()
max_out_triplets = sys.maxsize
  """

  stmt = f"out = get_triplet_mask(labels, '{device}', max_out_triplets)"
  times = np.array(timeit.repeat(stmt=stmt, setup=setup, number=number, repeat=repeat)) / number
  return times

def timeit_numpy(n_labels, n_instances, number, repeat):

  setup = f"""
from typing import List, Iterable, Any
from random import sample
import numpy as np
from sys import maxsize
from itertools import product, combinations, chain, islice
import scipy
import torch

def get_available_triplets_np2(labels: List[int], max_out_triplets: int = maxsize):
  labels = np.array(labels)
  # Check that i, j and k are distinct
  indices_equal = np.eye(len(labels), dtype="bool")
  indices_not_equal = np.logical_not(indices_equal)  # flip booleans
  i_not_equal_j = indices_not_equal[..., None]
  i_not_equal_k = indices_not_equal[:, None, :]
  j_not_equal_k = indices_not_equal[None, ...]

  distinct_indices = (i_not_equal_j & i_not_equal_k) & j_not_equal_k

  # Check if labels[i] == labels[j] and labels[i] != labels[k]
  label_equal = labels[:, None] == labels[None, :]
  i_equal_j = label_equal[..., None]
  i_equal_k = label_equal[:, None, :]

  valid_labels = i_equal_j & ~i_equal_k
  # Combine the two masks

  mask = distinct_indices & valid_labels

  triplets = np.array(np.where(mask))
  random_triplets_ids = np.random.choice(
      np.arange(len(triplets), dtype="int"), min(triplets.shape[1], max_out_triplets)
  )
  triplets = triplets[:, random_triplets_ids]

  # triplets = sample(triplets, min(len(triplets), max_out_triplets))
  # ids_anchor, ids_pos, ids_neg = zip(*triplets)
  ids_anchor, ids_pos, ids_neg = triplets

  return list(ids_anchor), list(ids_pos), list(ids_neg)

n_labels = {n_labels}
n_instances = {n_instances}
labels = np.repeat(np.arange(n_labels, dtype='int'),  n_instances).tolist()
max_out_triplets = sys.maxsize
  """

  stmt = f"out = get_available_triplets_np2(labels, max_out_triplets)"

  times = np.array(timeit.repeat(stmt=stmt, setup=setup, number=number, repeat=repeat)) / number
  return times

# run timeit for pytorch with cuda once to prevent cold start
timeit_torch(2, 2, 1, 1, 'cuda')

number = 1
repeat = 20
n_labels = 30 * 5
results = defaultdict(dict)
cases = {'Pytorch CUDA': partial(timeit_torch, device='cuda'),
       'Pytorch CPU': partial(timeit_torch, device='cpu'),
       'Numpy': timeit_numpy,
       'Original': timeit_original}
for n_instances, (case_name, case_func) in tqdm(list(product([7, 6, 5, 4], cases.items()))):
  print(f'{n_instances}, {case_name}')
  results[n_labels * n_instances'][case_name] = case_func(n_labels, n_instances, number, repeat)

# and some visualizations
# for each batch size make its own plot
dfs = {k: pd.DataFrame.from_dict(v) for k, v in results.items()}
n_rows = n_cols = int(np.ceil(np.sqrt(len(dfs))))
for n, k in enumerate(dfs, 1):
  ax = plt.subplot(n_rows, n_cols, n)
  ax = sns.boxplot(data=dfs[k], ax=ax)
  ax.set_ylabel('Time (s)')
  ax.set_title(str(k))
  ax.tick_params(labelrotation=90)

# make one plot for everything
df_dict = defaultdict(list)
for bs in results:
  for method in results[bs]:
      df_dict['batch size'].extend([bs, ] * len(results[bs][method]))
      df_dict['method'].extend([method, ] * len(results[bs][method]))
      df_dict['time'].extend(results[bs][method])

df = pd.DataFrame.from_dict(df_dict)
ax = sns.boxplot(data=df, y='time', x='batch size', hue='method')

And here are visualizations

It worth mentioning that pytorch and numpy based approaches requires O(n ** 3) space.

Seems like we need to pick Pytorch-based realisation. We can also add the "device" parameter to the init method of a miner to pick whether we want to use CPU or GPU.
I also suggest the following:

Let's estimate the memory in megabytes required for GPU-based realisation for the different batch sizes.
If the number is small, we can set device=gpu as default (device=cpu otherwise).

@DaloroAT what would you say?

Hey @dapladoc !
Nice to see you in our project.
Thank you for your investigations. As expected, CUDA version works faster.
Before we make a decision, could you try to replace the functions from itertools in the current miner with the corresponding functions from torch. Probably, for some batches, it may works better.

@DaloroAT
As I wrote above, the bottleneck is random.sample. List comprehensions (and I guess product is in that category, because I couldn't find it in the table) takes only 3% of all the computational time.
Here is profiling code

 Profiling code
def get_available_triplets(
    labels: List[int], max_out_triplets: int = maxsize
) -> TTripletsIds:
    """
    For each label, the function generates all possible positive and negative pairs of triplets.
    """
    pr = cProfile.Profile()
    pr.enable()
    num_labels = len(labels)

    triplets = []
    for label in set(labels):
        ids_pos_cur = find_value_ids(labels, label)
        ids_neg_cur = set(range(num_labels)) - set(ids_pos_cur)

        # (l0, l1, n) and (l1, l0, n) are 2 different triplets
        # and we want both of them
        pos_pairs = list(combinations(ids_pos_cur, r=2)) + list(
            combinations(ids_pos_cur[::-1], r=2)
        )

        tri = [(a, p, n) for (a, p), n in list(product(pos_pairs, ids_neg_cur))]
        triplets.extend(tri)

    triplets = sample(triplets, min(len(triplets), max_out_triplets))
    ids_anchor, ids_pos, ids_neg = zip(*triplets)
    pr.disable()
    s = io.StringIO()
    sortby = SortKey.CUMULATIVE
    ps = pstats.Stats(pr, stream=s).sort_stats(sortby)
    ps.print_stats()
    print(s.getvalue())
    return list(ids_anchor), list(ids_pos), list(ids_neg)

And here its output
        13686926 function calls (13686910 primitive calls) in 4.195 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    1.991    1.991    3.707    3.707 /home/aron/.pyenv/versions/3.8.15/lib/python3.8/random.py:315(sample)
  4023000    1.203    0.000    1.716    0.000 /home/aron/.pyenv/versions/3.8.15/lib/python3.8/random.py:250(_randbelow_with_getrandbits)
      150    0.457    0.003    0.457    0.003 /home/aron/PycharmProjects/open-metric-learning/dev_notebooks/profile_get_available_triplets.py:51(<listcomp>)
  5640132    0.333    0.000    0.333    0.000 {method 'getrandbits' of '_random.Random' objects}
  4023000    0.180    0.000    0.180    0.000 {method 'bit_length' of 'int' objects}
      150    0.024    0.000    0.024    0.000 {method 'extend' of 'list' objects}
      150    0.001    0.000    0.007    0.000 /home/aron/PycharmProjects/open-metric-learning/dev_notebooks/profile_get_available_triplets.py:14(find_value_ids)
      150    0.006    0.000    0.006    0.000 /home/aron/PycharmProjects/open-metric-learning/dev_notebooks/profile_get_available_triplets.py:26(<listcomp>)
      152    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
        2    0.000    0.000    0.000    0.000 /home/aron/.pyenv/versions/3.8.15/lib/python3.8/abc.py:96(__instancecheck__)
        2    0.000    0.000    0.000    0.000 {built-in method _abc._abc_instancecheck}
     10/2    0.000    0.000    0.000    0.000 /home/aron/.pyenv/versions/3.8.15/lib/python3.8/abc.py:100(__subclasscheck__)
     10/2    0.000    0.000    0.000    0.000 {built-in method _abc._abc_subclasscheck}
        1    0.000    0.000    0.000    0.000 {built-in method math.ceil}
        1    0.000    0.000    0.000    0.000 {built-in method math.log}
        1    0.000    0.000    0.000    0.000 {built-in method builtins.min}
        7    0.000    0.000    0.000    0.000 /home/aron/.pyenv/versions/3.8.15/lib/python3.8/_collections_abc.py:392(__subclasshook__)
        3    0.000    0.000    0.000    0.000 {built-in method builtins.len}
        3    0.000    0.000    0.000    0.000 /home/aron/.pyenv/versions/3.8.15/lib/python3.8/_collections_abc.py:302(__subclasshook__)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

As you can see, list comprehension (as far as I understand including the product) takes just 0.457 sec
  ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     150    0.457    0.003    0.457    0.003 /home/aron/PycharmProjects/open-metric-learning/dev_notebooks/profile_get_available_triplets.py:51(<listcomp>)

But anyway, I'll check the torch replacements for itertools.

By the way, here are the same plots as above, but I got them from google colab (notebook). Numpy version works faster than torch CPU on colab.

Sorry, I checked the bottleneck again. That's true, random.sample is slow.
I've tried a simple hack. Replace
triplets = sample(triplets, min(len(triplets), max_out_triplets))

to
from operator import itemgetter
...
ids = np.random.choice(len(triplets), min(len(triplets), max_out_triplets), replace=False)
triplets = itemgetter(*ids)(triplets)

And it drastically increases performance, listcomp becomes a bottleneck in non-vectorized solution.

Sample from random
         8229218 function calls (8229202 primitive calls) in 3.255 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    1.381    1.381    2.921    2.921 /Users/atarasov/opt/miniconda3/envs/ny/lib/python3.8/random.py:315(sample)
  2389100    1.043    0.000    1.540    0.000 /Users/atarasov/opt/miniconda3/envs/ny/lib/python3.8/random.py:250(_randbelow_with_getrandbits)
  3450924    0.317    0.000    0.317    0.000 {method 'getrandbits' of '_random.Random' objects}
       10    0.315    0.031    0.315    0.031 /Users/atarasov/code/open-metric-learning/shit.py:76(<listcomp>)
  2389100    0.180    0.000    0.180    0.000 {method 'bit_length' of 'int' objects}
       10    0.020    0.002    0.020    0.002 {method 'extend' of 'list' objects}
       10    0.000    0.000    0.000    0.000 /Users/atarasov/code/open-metric-learning/oml/utils/misc.py:13(find_value_ids)
       10    0.000    0.000    0.000    0.000 /Users/atarasov/code/open-metric-learning/oml/utils/misc.py:25(<listcomp>)
       12    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
        2    0.000    0.000    0.000    0.000 /Users/atarasov/opt/miniconda3/envs/ny/lib/python3.8/abc.py:96(__instancecheck__)
        2    0.000    0.000    0.000    0.000 {built-in method _abc._abc_instancecheck}
     10/2    0.000    0.000    0.000    0.000 /Users/atarasov/opt/miniconda3/envs/ny/lib/python3.8/abc.py:100(__subclasscheck__)
     10/2    0.000    0.000    0.000    0.000 {built-in method _abc._abc_subclasscheck}
        1    0.000    0.000    0.000    0.000 {built-in method math.ceil}
        1    0.000    0.000    0.000    0.000 {built-in method math.log}
        1    0.000    0.000    0.000    0.000 {built-in method builtins.min}
        7    0.000    0.000    0.000    0.000 /Users/atarasov/opt/miniconda3/envs/ny/lib/python3.8/_collections_abc.py:392(__subclasshook__)
        3    0.000    0.000    0.000    0.000 {built-in method builtins.len}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        3    0.000    0.000    0.000    0.000 /Users/atarasov/opt/miniconda3/envs/ny/lib/python3.8/_collections_abc.py:302(__subclasshook__)

 Sample from numpy
         65 function calls in 0.447 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       10    0.359    0.036    0.359    0.036 /Users/atarasov/code/open-metric-learning/shit.py:76(<listcomp>)
        1    0.069    0.069    0.069    0.069 {method 'choice' of 'numpy.random.mtrand.RandomState' objects}
       10    0.019    0.002    0.019    0.002 {method 'extend' of 'list' objects}
       10    0.000    0.000    0.000    0.000 /Users/atarasov/code/open-metric-learning/oml/utils/misc.py:13(find_value_ids)
       10    0.000    0.000    0.000    0.000 /Users/atarasov/code/open-metric-learning/oml/utils/misc.py:25(<listcomp>)
        1    0.000    0.000    0.000    0.000 <__array_function__ internals>:177(prod)
        1    0.000    0.000    0.000    0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}
        1    0.000    0.000    0.000    0.000 /Users/atarasov/opt/miniconda3/envs/ny/lib/python3.8/site-packages/numpy/core/fromnumeric.py:2970(prod)
        1    0.000    0.000    0.000    0.000 /Users/atarasov/opt/miniconda3/envs/ny/lib/python3.8/site-packages/numpy/core/fromnumeric.py:69(_wrapreduction)
        1    0.000    0.000    0.000    0.000 {method 'reduce' of 'numpy.ufunc' objects}
       10    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
        1    0.000    0.000    0.000    0.000 {built-in method builtins.min}
        3    0.000    0.000    0.000    0.000 {built-in method builtins.len}
        1    0.000    0.000    0.000    0.000 {built-in method builtins.getattr}
        1    0.000    0.000    0.000    0.000 /Users/atarasov/opt/miniconda3/envs/ny/lib/python3.8/site-packages/numpy/core/fromnumeric.py:70(<dictcomp>)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.000    0.000    0.000    0.000 /Users/atarasov/opt/miniconda3/envs/ny/lib/python3.8/site-packages/numpy/core/fromnumeric.py:2965(_prod_dispatcher)
        1    0.000    0.000    0.000    0.000 {method 'items' of 'dict' objects}

Probably, Colab have numpy with some mkl optimizations.

@dapladoc Could you check memory consumption as @AlekseySh suggested?

Seems like we need to pick Pytorch-based realisation. We can also add the "device" parameter to the init method of a miner to pick whether we want to use CPU or GPU.
I also suggest the following:

Let's estimate the memory in megabytes required for GPU-based realisation for the different batch sizes.
If the number is small, we can set device=gpu as default (device=cpu otherwise).

Great job!
Looks like Ori-Torch is the optimal setup for medium and large batch sizes for both CPU and CUDA.
Probably, Pytorch CUDA is the fastest on small and medium batch sizes, but it produces large memory overhead by creating [len(labels), len(labels), len(labels)] matrix with vectorizing. I suppose 0.06s overhead between Ori-Torch and Pytorch CUDA is nothing when we need to read and transform 600 images.
At the same time, there is no big difference in duration between Ori-Torch CPU and Ori-Torch CUDA, so we can run mining only on the CPU.
I suggest using only Ori-Torch CPU. What do you think @dapladoc @AlekseySh ?
Or you can try to implement another vectorized implementation @dapladoc, because the current Numpy and Pytorch ... versions produce and operate with high dimensional arrays. Probably you can get some ideas from another miner, removing logic with matrix distance.

@DaloroAT Agree with you

Inspiring by the miners I was suggested, I've come up with another (almost) loop-less solution, that doesn't evaluate those 3d arrays. But this solution looks a little bit tricky, it works just as fast as like Ori-Torch but requires a little bit less memory. I called it Pytorch complex in the tables below.

  As result we get an array called neg_groups,
    # such that neg_groups[i] is all the negatives, that come along with
    # the anchor number i.
    # For simplicity, lets assume that
    # idx_anch_pos = [[0, 1],
    #                 [0, 2],
    #                 [1, 3],
    #                 [1, 4]]
    # and idx_anch_neg = [[0, 5],
    #                     [0, 6],
    #                     [1, 7]]
    #                     [1, 8]]
    # We want to get the following correspondence:
    #     0 -> [5, 6]
    #     1 -> [7, 8]
    # We know that idx_neg is sorted lexicographically. So, we need to find indexes of rows
    # where the first column changes its value, lets call it `steps`, and split the second column of
    # the idx_anch_neg (where all the negative indexes are) by `steps`.
    # For the example above, we see that idx_anch_neg[:, 0] changes its values only at row 3. So, we need
    # to split idx_anch_neg[:, 1] by taking first 2 elements in the first chunk, and the last
    # 2 elements in the second chunk
    unique_anch_inds, unique_anch_inds_inverse = torch.unique(idx_anch_neg[:, 0], return_inverse=True)
    steps = torch.nonzero(unique_anch_inds_inverse[1:] != unique_anch_inds_inverse[:-1])[:, 0]
    steps = torch.cat((torch.tensor([0, ]).to(steps), steps + 1, torch.tensor([len(unique_anch_inds_inverse)]).to(steps)))
    chunck_sizes = tuple((steps[1:] - steps[:-1]).cpu().numpy())
    neg_groups = torch.split(idx_anch_neg[:, 1], chunck_sizes)

    # Now for a pair (anchor, positive)we have
    # neg_group[i], where i is the number of the anchor, with all
    # the negatives that come along with that anchor.
    # In order to get all possible triplets that contains (anchor, positive)
    # we just need to repeat it as much as we have negative elements
    # in the neg_groups[i].
    # For example above, we want to generate
    # [0, 1, 5]   | we need to repeat [[0, 1], [0, 2]] as much
    # [0, 1, 6]   | as there are negatives that come along with anchor 0
    # [0, 2, 5]
    # [0, 2, 6]
    # [1, 3, 7]
    # [1, 3, 8]
    # [1, 4, 7]
    # [1, 4, 8]
    n_negs_in_group = [len(g) for g in neg_groups]
    idx_neg = torch.cat(tuple(itemgetter(*idx_anch_pos[:, 0].cpu().tolist())(neg_groups)))

    n_repeats = torch.tensor(tuple(itemgetter(*idx_anch_pos[:, 0].cpu().numpy())(n_negs_in_group))).to(idx_anch_pos)
    idx_anch_pos = idx_anch_pos.repeat_interleave(n_repeats, dim=0)
    idx_anch, idx_pos = idx_anch_pos.T

    random_idx = np.random.choice(len(idx_anch), min(len(idx_anch), max_out_triplets), replace=False)
    random_idx = torch.tensor(random_idx).to(device)
    idx_anch = idx_anch[random_idx]
    idx_pos = idx_pos[random_idx]
    idx_neg = idx_neg[random_idx]

    idx_anch = idx_anch.tolist()
    idx_pos = idx_pos.tolist()
    idx_neg = idx_neg.tolist()
    return idx_anch, idx_pos, idx_neg

I like Ori-Torch CPU or Pytorch complex CPU, and they seem to be good candidates to replace current version of get_available_triplets.

@dapladoc Wow! That's impressive. It's able to reduce memory usage, but without losing performance.
And it's really complex, but well-documented. But still complex...
@dapladoc @DaloroAT what do you guys think, is this gap worth bringing this complexity to the codebase or should we go for Ori-Troch CPU?

@dapladoc I very like your solution. Thank you for documenting your ideas! The idea about groups is excellent.
I printed the shapes of each row to understand the way of calculations. It really tricky, and I found the simplest way to get neg_groups. After that, it becomes possible to simplify the rest of the steps, could you check it?

Refactored version

def refactored_mining(labels, max_out_triplets, device):
    labels = torch.tensor(labels, device=device, requires_grad=False)
    # First we need to get initial ingridients of triplets:
    # the arrays of all (anchor, positive) and (anchor, negavite) pairs
    mask_same_label = labels[:, None] == labels[None, :]

    idx_anch_neg = torch.nonzero(~mask_same_label)
    mask_same_label.fill_diagonal_(False)
    idx_anch_pos = torch.nonzero(mask_same_label)

    # START MODIFY
    # idx_anch_neg and idx_anch_pos are sorted and blocked, so we can get counts directly
    _, neg_counts = torch.unique(idx_anch_neg[:, 0], return_counts=True, sorted=True)
    neg_groups = torch.split(idx_anch_neg[:, 1], neg_counts.tolist())

    _, pos_counts = torch.unique(idx_anch_pos[:, 0], return_counts=True, sorted=True)
    pos_groups = torch.split(idx_anch_pos[:, 1], pos_counts.tolist())

    # calc combinations of pos and neg for each anchor, appending on the left the value of anchor
    idx_anch_pos_neg = [torch.constant_pad_nd(torch.cartesian_prod(pos, neg), (1, 0), anch)
                        for anch, (pos, neg) in enumerate(zip(pos_groups, neg_groups))]

    idx_anch_pos_neg = torch.cat(idx_anch_pos_neg)

    idx_anch, idx_pos, idx_neg = idx_anch_pos_neg[:, 0], idx_anch_pos_neg[:, 1], idx_anch_pos_neg[:, 2]

    # STOP MODIFY

    random_idx = np.random.choice(len(idx_anch), min(len(idx_anch), max_out_triplets), replace=False)
    random_idx = torch.tensor(random_idx).to(device)
    idx_anch = idx_anch[random_idx]
    idx_pos = idx_pos[random_idx]
    idx_neg = idx_neg[random_idx]

    idx_anch = idx_anch.tolist()
    idx_pos = idx_pos.tolist()
    idx_neg = idx_neg.tolist()
    return idx_anch, idx_pos, idx_neg

@DaloroAT your solution is brilliant. Omg, I feel so dumb because of my way to imitate return_counts=True (-:
I created yet another get_available_triplets and called it Pytorch complex simplified (sorry for the naming, I'm out of ideas). It is like Pytorch complex but with your code to obtain neg_groups.

Pytorch complex simplified (without comments for brevity)
def get_available_triplets_torch_complex_simplified(labels, max_out_triplets, device='cpu'):
    num_labels = len(labels)
    labels = torch.tensor(labels, device=device, requires_grad=False)

    mask_same_label = labels[:, None] == labels[None, :]
    idx_anch_neg = torch.nonzero(~mask_same_label)
    mask_same_label.fill_diagonal_(False)
    idx_anch_pos = torch.nonzero(mask_same_label)

    # START CODE BORROWING
    _, neg_counts = torch.unique(idx_anch_neg[:, 0], return_counts=True, sorted=True)
    neg_groups = torch.split(idx_anch_neg[:, 1], neg_counts.tolist())
    # END CODE BORROWING

    n_negs_in_group = [len(g) for g in neg_groups]
    idx_neg = torch.cat(tuple(itemgetter(*idx_anch_pos[:, 0].cpu().tolist())(neg_groups)))

    n_repeats = torch.tensor(tuple(itemgetter(*idx_anch_pos[:, 0].cpu().numpy())(n_negs_in_group))).to(idx_anch_pos)
    idx_anch_pos = idx_anch_pos.repeat_interleave(n_repeats, dim=0)
    idx_anch, idx_pos = idx_anch_pos.T

    random_idx = np.random.choice(len(idx_anch), min(len(idx_anch), max_out_triplets), replace=False)
    random_idx = torch.tensor(random_idx).to(device)
    idx_anch = idx_anch[random_idx]
    idx_pos = idx_pos[random_idx]
    idx_neg = idx_neg[random_idx]

    idx_anch = idx_anch.tolist()
    idx_pos = idx_pos.tolist()
    idx_neg = idx_neg.tolist()
    return idx_anch, idx_pos, idx_neg

by means of timeit I've got the following performance measures

batch size
method
time (s)

1200
Pytorch complex
1.3734

1200
refactored_miner (@DaloroAT)
1.4443

1200
Pytorch complex simplified
1.3219

Next, I've run profiler in order to figure out why the timings looks like that.
Here is function to get profiler information

profile_get_available_triplets
def profile_get_available_triplets(f, labels, max_out_triplets, repeat, device):
    pr = cProfile.Profile()
    pr.enable()
    for i in range(repeat):
      out_c = f(labels, max_out_triplets, device=device)
    pr.disable()
    s = io.StringIO()
    sortby = SortKey.CUMULATIVE
    ps = pstats.Stats(pr, stream=s).sort_stats(sortby)
    ps.print_stats()
    print(s.getvalue())

And here is results of the profiler for batch size 1200, repeat = 20 and device='cuda'.

profile_get_available(get_available_triplets_torch_complex, labels, max_out_triplets, repeat, 'cuda')
        121481 function calls (121461 primitive calls) in 32.127 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       20    0.089    0.004   32.127    1.606 <ipython-input-13-a4693f8eed72>:1(get_available_triplets_torch_complex)
       80   19.335    0.242   19.335    0.242 {method 'tolist' of 'torch._C._TensorBase' objects}
       20    7.323    0.366    7.324    0.366 {method 'choice' of 'numpy.random.mtrand.RandomState' objects}
       20    0.000    0.000    4.368    0.218 /usr/local/lib/python3.7/dist-packages/torch/functional.py:140(split)
       20    0.019    0.001    4.368    0.218 /usr/local/lib/python3.7/dist-packages/torch/_tensor.py:597(split)
       20    4.349    0.217    4.349    0.217 {method 'split_with_sizes' of 'torch._C._TensorBase' objects}
      100    0.383    0.004    0.383    0.004 {built-in method torch.tensor}
       80    0.350    0.004    0.350    0.004 {method 'to' of 'torch._C._TensorBase' objects}
       60    0.105    0.002    0.105    0.002 {method 'cpu' of 'torch._C._TensorBase' objects}
    40/20    0.000    0.000    0.058    0.003 /usr/local/lib/python3.7/dist-packages/torch/_jit_internal.py:413(fn)
       20    0.000    0.000    0.058    0.003 /usr/local/lib/python3.7/dist-packages/torch/functional.py:855(_return_inverse)
       20    0.000    0.000    0.058    0.003 /usr/local/lib/python3.7/dist-packages/torch/functional.py:688(_unique_impl)
       20    0.058    0.003    0.058    0.003 {built-in method torch._unique2}
       40    0.054    0.001    0.054    0.001 {built-in method torch.cat}
       20    0.004    0.000    0.042    0.002 <ipython-input-13-a4693f8eed72>:55(<listcomp>)
    24100    0.007    0.000    0.039    0.000 {built-in method builtins.len}
    24060    0.021    0.000    0.032    0.000 /usr/local/lib/python3.7/dist-packages/torch/_tensor.py:701(__len__)
       60    0.011    0.000    0.011    0.000 {built-in method torch.nonzero}
    24080    0.006    0.000    0.006    0.000 {built-in method torch._C._get_tracing_state}
       20    0.004    0.000    0.004    0.000 {method 'repeat_interleave' of 'torch._C._TensorBase' objects}
    24080    0.003    0.000    0.003    0.000 {method 'dim' of 'torch._C._TensorBase' objects}
    24140    0.002    0.000    0.002    0.000 {built-in method torch._C._has_torch_function_unary}
       20    0.000    0.000    0.001    0.000 <__array_function__ internals>:2(prod)
       20    0.001    0.000    0.001    0.000 {method 'fill_diagonal_' of 'torch._C._TensorBase' objects}
       20    0.000    0.000    0.001    0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}
       20    0.000    0.000    0.001    0.000 /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:2933(prod)
       20    0.000    0.000    0.001    0.000 /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:69(_wrapreduction)
       20    0.001    0.000    0.001    0.000 {method 'reduce' of 'numpy.ufunc' objects}
       20    0.000    0.000    0.000    0.000 /usr/local/lib/python3.7/dist-packages/torch/_tensor.py:713(__iter__)
       40    0.000    0.000    0.000    0.000 {method 'numpy' of 'torch._C._TensorBase' objects}
       20    0.000    0.000    0.000    0.000 {method 'unbind' of 'torch._C._TensorBase' objects}
       20    0.000    0.000    0.000    0.000 {built-in method builtins.getattr}
       20    0.000    0.000    0.000    0.000 /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:70(<dictcomp>)
       20    0.000    0.000    0.000    0.000 {built-in method builtins.min}
       40    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
       20    0.000    0.000    0.000    0.000 {built-in method builtins.iter}
       20    0.000    0.000    0.000    0.000 /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:2928(_prod_dispatcher)
       20    0.000    0.000    0.000    0.000 {method 'items' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

profile_get_available(refactored_mining, labels, max_out_triplets, repeat, 'cuda')
         145401 function calls (145361 primitive calls) in 34.134 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       20    0.085    0.004   34.134    1.707 <ipython-input-13-a4693f8eed72>:145(refactored_mining)
      100   18.983    0.190   18.983    0.190 {method 'tolist' of 'torch._C._TensorBase' objects}
       20    6.924    0.346    6.925    0.346 {method 'choice' of 'numpy.random.mtrand.RandomState' objects}
       40    0.000    0.000    5.046    0.126 /usr/local/lib/python3.7/dist-packages/torch/functional.py:140(split)
       40    0.020    0.000    5.045    0.126 /usr/local/lib/python3.7/dist-packages/torch/_tensor.py:597(split)
       40    5.025    0.126    5.025    0.126 {method 'split_with_sizes' of 'torch._C._TensorBase' objects}
       20    0.081    0.004    2.322    0.116 <ipython-input-13-a4693f8eed72>:164(<listcomp>)
    24000    1.129    0.000    1.129    0.000 {built-in method torch.constant_pad_nd}
    24000    0.062    0.000    1.111    0.000 /usr/local/lib/python3.7/dist-packages/torch/functional.py:1068(cartesian_prod)
    24000    1.030    0.000    1.030    0.000 {built-in method torch.cartesian_prod}
       40    0.356    0.009    0.356    0.009 {built-in method torch.tensor}
       20    0.346    0.017    0.346    0.017 {method 'to' of 'torch._C._TensorBase' objects}
    80/40    0.000    0.000    0.047    0.001 /usr/local/lib/python3.7/dist-packages/torch/_jit_internal.py:413(fn)
       40    0.000    0.000    0.046    0.001 /usr/local/lib/python3.7/dist-packages/torch/functional.py:835(_return_counts)
       40    0.000    0.000    0.046    0.001 /usr/local/lib/python3.7/dist-packages/torch/functional.py:688(_unique_impl)
       40    0.046    0.001    0.046    0.001 {built-in method torch._unique2}
       20    0.015    0.001    0.015    0.001 {built-in method torch.cat}
    24000    0.011    0.000    0.015    0.000 /usr/local/lib/python3.7/dist-packages/torch/_VF.py:25(__getattr__)
       40    0.009    0.000    0.009    0.000 {built-in method torch.nonzero}
    24020    0.004    0.000    0.004    0.000 {built-in method builtins.getattr}
    24000    0.004    0.000    0.004    0.000 {built-in method torch._C._has_torch_function}
       20    0.000    0.000    0.001    0.000 <__array_function__ internals>:2(prod)
       20    0.000    0.000    0.001    0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}
       20    0.000    0.000    0.001    0.000 /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:2933(prod)
       20    0.000    0.000    0.001    0.000 /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:69(_wrapreduction)
       20    0.001    0.000    0.001    0.000 {method 'fill_diagonal_' of 'torch._C._TensorBase' objects}
       20    0.000    0.000    0.001    0.000 /usr/local/lib/python3.7/dist-packages/torch/_tensor.py:713(__iter__)
       20    0.001    0.000    0.001    0.000 {method 'reduce' of 'numpy.ufunc' objects}
       20    0.000    0.000    0.000    0.000 {method 'unbind' of 'torch._C._TensorBase' objects}
       80    0.000    0.000    0.000    0.000 {built-in method builtins.len}
       40    0.000    0.000    0.000    0.000 /usr/local/lib/python3.7/dist-packages/torch/_tensor.py:701(__len__)
       60    0.000    0.000    0.000    0.000 {built-in method torch._C._get_tracing_state}
      200    0.000    0.000    0.000    0.000 {built-in method torch._C._has_torch_function_unary}
       80    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
       60    0.000    0.000    0.000    0.000 {method 'dim' of 'torch._C._TensorBase' objects}
       20    0.000    0.000    0.000    0.000 /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:70(<dictcomp>)
       20    0.000    0.000    0.000    0.000 {built-in method builtins.min}
       20    0.000    0.000    0.000    0.000 {built-in method builtins.iter}
       20    0.000    0.000    0.000    0.000 {method 'items' of 'dict' objects}
       20    0.000    0.000    0.000    0.000 /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:2928(_prod_dispatcher)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

profile_get_available(get_available_triplets_torch_complex_simplified, labels, max_out_triplets, repeat, 'cuda')
         121241 function calls (121221 primitive calls) in 31.007 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       20    0.084    0.004   31.007    1.550 <ipython-input-13-a4693f8eed72>:75(get_available_triplets_torch_complex_simplified)
      100   18.987    0.190   18.987    0.190 {method 'tolist' of 'torch._C._TensorBase' objects}
       20    6.790    0.339    6.791    0.340 {method 'choice' of 'numpy.random.mtrand.RandomState' objects}
       20    0.000    0.000    4.190    0.210 /usr/local/lib/python3.7/dist-packages/torch/functional.py:140(split)
       20    0.008    0.000    4.190    0.210 /usr/local/lib/python3.7/dist-packages/torch/_tensor.py:597(split)
       20    4.182    0.209    4.182    0.209 {method 'split_with_sizes' of 'torch._C._TensorBase' objects}
       60    0.366    0.006    0.366    0.006 {built-in method torch.tensor}
       40    0.348    0.009    0.348    0.009 {method 'to' of 'torch._C._TensorBase' objects}
       40    0.099    0.002    0.099    0.002 {method 'cpu' of 'torch._C._TensorBase' objects}
       20    0.046    0.002    0.046    0.002 {built-in method torch.cat}
       20    0.004    0.000    0.041    0.002 <ipython-input-13-a4693f8eed72>:126(<listcomp>)
    40/20    0.000    0.000    0.039    0.002 /usr/local/lib/python3.7/dist-packages/torch/_jit_internal.py:413(fn)
       20    0.000    0.000    0.039    0.002 /usr/local/lib/python3.7/dist-packages/torch/functional.py:835(_return_counts)
       20    0.000    0.000    0.039    0.002 /usr/local/lib/python3.7/dist-packages/torch/functional.py:688(_unique_impl)
       20    0.039    0.002    0.039    0.002 {built-in method torch._unique2}
    24080    0.007    0.000    0.037    0.000 {built-in method builtins.len}
    24040    0.020    0.000    0.031    0.000 /usr/local/lib/python3.7/dist-packages/torch/_tensor.py:701(__len__)
       40    0.008    0.000    0.008    0.000 {built-in method torch.nonzero}
    24060    0.006    0.000    0.006    0.000 {built-in method torch._C._get_tracing_state}
       20    0.004    0.000    0.004    0.000 {method 'repeat_interleave' of 'torch._C._TensorBase' objects}
    24060    0.003    0.000    0.003    0.000 {method 'dim' of 'torch._C._TensorBase' objects}
    24120    0.002    0.000    0.002    0.000 {built-in method torch._C._has_torch_function_unary}
       20    0.000    0.000    0.001    0.000 <__array_function__ internals>:2(prod)
       20    0.000    0.000    0.001    0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}
       20    0.001    0.000    0.001    0.000 {method 'fill_diagonal_' of 'torch._C._TensorBase' objects}
       20    0.000    0.000    0.001    0.000 /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:2933(prod)
       20    0.000    0.000    0.001    0.000 /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:69(_wrapreduction)
       20    0.001    0.000    0.001    0.000 {method 'reduce' of 'numpy.ufunc' objects}
       20    0.000    0.000    0.000    0.000 /usr/local/lib/python3.7/dist-packages/torch/_tensor.py:713(__iter__)
       20    0.000    0.000    0.000    0.000 {method 'unbind' of 'torch._C._TensorBase' objects}
       20    0.000    0.000    0.000    0.000 {method 'numpy' of 'torch._C._TensorBase' objects}
       20    0.000    0.000    0.000    0.000 /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:70(<dictcomp>)
       20    0.000    0.000    0.000    0.000 {built-in method builtins.getattr}
       20    0.000    0.000    0.000    0.000 {built-in method builtins.min}
       40    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
       20    0.000    0.000    0.000    0.000 {built-in method builtins.iter}
       20    0.000    0.000    0.000    0.000 /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:2928(_prod_dispatcher)
       20    0.000    0.000    0.000    0.000 {method 'items' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

As I see it, the reason why @DaloroAT version is slower is related to

torch.split is called twice
torch.constant_pad_nd
listcomp
torch.cartesian_prod

I'm quite surprised that even my dumb jungling with chunk sizes is faster than the version based on torch. But actually it looks fair, because, for instance, even though the line idx_neg = torch.cat(tuple(itemgetter(*idx_anch_pos[:, 0].cpu().tolist())(neg_groups))) looks ugly, the evaluation of the torch.cat argument costs nothing: no tensor creation occurs here, it is just a collection of references, and after that one big tensor is created. And idx_anch_pos = idx_anch_pos.repeat_interleave(n_repeats, dim=0) looks simpler than a lot of cartesian products.
So, @AlekseySh, @DaloroAT I would say the last version of code is a good candidate to replace get_available_triplets.

By the way, is it important to return lists of indexes from the function? Or can we return tensors and reduce huge amount of time on conversion from torch.Tensor to list?

@dapladoc thank you for the analysis
As for the types: we can get rid of lists, but I guess it should be done in a dedicated PR.

Hey @dapladoc . Sorry for the long reply.
Happy to help you. Suppose we can finally create PR with your last version of mining.
You can keep and use the current function get_available_triplets (as a naive implementation, which everyone understands) for testing the new miner, they have to return the same sets of triplets. For the updated version of the miner you can add the argument device in __init__  with cuda as default.

Hey @dapladoc . Sorry for the long reply.
Happy to help you. Suppose we can finally create PR with your last version of mining.
You can keep and use the current function get_available_triplets (as a naive implementation, which everyone understands) for testing the new miner, they have to return the same sets of triplets. For the updated version of the miner you can add the argument device in __init__  with cuda as default.

Agree with an idea about test.
Are you sure about cuda as default?

Sorry, cpu as default.

Sorry, cpu as default.

that's better


Feature request. Add support for React v17

React v17 no longer needs to import React from 'react'
I was wondering if it is possible to update this extension.
Thanks.

@markhker the problem is, that not all projects are already migrated to v17. I would like to not force most of people to migrate, I will probably make some additional alias or maybe finally find way to implement options and dynamically parse json files to handle semis, single / double colons etc.

Hi,
I don't know whether My issue also related to the latest version, whenever I try useState or useEffect snippet it is not adding it to Import Statement.
It used to work a few weeks back, I am using Auto Import Extension as well, not sure what am I doing wrong there!.
I tried all the recent React versions i.e 16.8, 16.12,....,17.0.01
Vs Code:1.50.1
Can u let me know what I can do to fix it so that it can add automatically whenever I use this Snippets

@markhker They will be with _ prefix for current development. In future I hope I could make options for it.
@srikanth597 It's not related to package but snippets itself. Auto import does not work that way and to resolve this you should look for some solution in VSCode but I couldn't also find this.

Have you removed the _rafce for creating a functional component without importing React? I tried it today and it's not working. Thanks

@prospernglazi looks like I forgot totally to apply importReactOnTop setting. It will be fixed in v4.0.4 in an hour or two. Currently you won't need _ prefix for those, just changing a setting inside your VSCode

@prospernglazi released v4.0.4 Check out settings 👀


Typo: Should be G3, not G5

There is typo on this page:
Microsoft 365 G5 subscription (paid or trial version)
Microsoft 365 G5 subscription + the Microsoft 365 G5 Compliance add-on
Microsoft 365 G5 subscription + the Microsoft 365 G5 Insider Risk Management add-on
The last two lines should says: G3 and not G5.
Scott
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: cb98e0ff-d34c-75bc-0ade-7dc70fc7f4bd
Version Independent ID: 5172471f-e514-bcde-16e0-d8d8090a892a
Content: Get started with insider risk management - Microsoft 365 Compliance
Content Source: microsoft-365/compliance/insider-risk-management-configure.md
Product: microsoft-365-enterprise
GitHub Login: @robmazz
Microsoft Alias: robmazz

@scottyv  - yes, thanks! I will correct now and republish.


Error while mounting app after upgrade to Nuxt 3.4 and ssr = false

Environment

Operating System: Windows_NT
Node Version:     v18.12.1
Nuxt Version:     3.4.0
Nitro Version:    2.3.3
Package Manager<EMAIL_ADDRESS>Builder:          vite
User Config:      ssr
Runtime Modules:  -
Build Modules:    -

Reproduction
https://stackblitz.com/github/stenet/nuxt-34-non-ssr
Start the server and look at the console:

Describe the bug
After updating to Nuxt 3.4, an error occurs at startup when ssr is set to false in nuxt.config.ts.
Additional context
No response
Logs
No response

This is resolved in the edge channel via #20218, and will be included in the next release.

😁 That was fast. I saw the PR but didn't know it was for the same reason.
Thanks Daniel!


Fix bugs in DataAnalyzer

Codecov Report

Merging #260 (b6b3bff) into master (5c04d4e) will not change coverage.
The diff coverage is 0.00%.

@@            Coverage Diff            @@
##             master     #260   +/-   ##
=========================================
  Coverage     55.23%   55.23%
  Complexity      610      610
=========================================
  Files           107      107
  Lines          2799     2799
  Branches        392      392
=========================================
  Hits           1546     1546
  Misses         1105     1105
  Partials        148      148

Impacted Files
Coverage Δ

...rc/main/java/seedu/address/logic/DataAnalyzer.java
0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5c04d4e...b6b3bff. Read the comment docs.


ErrorBoundary missing on root

I've found that there is no function ErrorBoundary() on root.tsx. If a route manually typed in to url that didn't exist it shows an ugly error.

You're right. We definitely should have this. Would you be willing to add something?

Yeah, I can do that.
I see in the Remix docs they have this example though, Epic stack already uses an error-boundary.tsx component I can reuse that


Release/4.0.0

Coverage remained the same at 100.0% when pulling a18a66f52a26f51d885c7f24450b7b44f48f3ecc on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage remained the same at 100.0% when pulling acabcb2c508357079a82b7d0a1555d6328a020d7 on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage remained the same at 100.0% when pulling d28496b2f47380ab32eecdaa076584c9eb886d5e on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage remained the same at 100.0% when pulling d28496b2f47380ab32eecdaa076584c9eb886d5e on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage remained the same at 100.0% when pulling d28496b2f47380ab32eecdaa076584c9eb886d5e on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage remained the same at 100.0% when pulling d28496b2f47380ab32eecdaa076584c9eb886d5e on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage remained the same at 100.0% when pulling 067b683684c493a6ebd61c1066ac3fd8780cf890 on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage remained the same at 100.0% when pulling 674ed6eab8d1225b48bb4fbe4b798830a6ced096 on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage remained the same at 100.0% when pulling f50a32738a1fb1d0e1998fad2cc2af377faddbfe on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage decreased (-0.2%) to 99.822% when pulling 97a091c943ce997f916abee935af7e82f973841a on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage decreased (-0.2%) to 99.822% when pulling 97a091c943ce997f916abee935af7e82f973841a on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage decreased (-0.2%) to 99.822% when pulling 97a091c943ce997f916abee935af7e82f973841a on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage decreased (-0.2%) to 99.822% when pulling 97a091c943ce997f916abee935af7e82f973841a on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage remained the same at 100.0% when pulling 0cc3e309a19fa9e7406d9cccee3c218a251085ca on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage remained the same at 100.0% when pulling 0cc3e309a19fa9e7406d9cccee3c218a251085ca on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage remained the same at 100.0% when pulling 3cdd4e3aca0f12af1ce6f439a14c59d30f33b303 on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage remained the same at 100.0% when pulling ed20d290b1b5d0369beb49fb537f7da705af7c98 on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage remained the same at 100.0% when pulling ed20d290b1b5d0369beb49fb537f7da705af7c98 on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage decreased (-1.3%) to 98.674% when pulling d1f16d4c9d839be592a566ad68153cca2b4ff4fe on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage decreased (-0.1%) to 99.858% when pulling d1f16d4c9d839be592a566ad68153cca2b4ff4fe on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage decreased (-0.1%) to 99.858% when pulling d1f16d4c9d839be592a566ad68153cca2b4ff4fe on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage decreased (-0.1%) to 99.858% when pulling d1f16d4c9d839be592a566ad68153cca2b4ff4fe on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage decreased (-0.3%) to 99.679% when pulling 32acb453e58e097ae9c0a592fb727e2092e84bad on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.

Coverage remained the same at 100.0% when pulling fdb1cb3c9cb8ec8fb2631439c04771d652bba185 on release/4.0.0 into 881c0a62d9879d82790ef40cca0c34c370f1eef2 on master.


--insecure-skip-tls-verification flag no longer works

Type
Incorrect behavior
Operating System
MacOS
OS Version
Monterey 12.5.1
Bug description
I need to use the --insecure-skip-tls-verification flag in order to run RuneLite on this machine from the command line. As of this morning I get this error
java.lang.NullPointerException: null
	at okhttp3.internal.tls.BasicTrustRootIndex.<init>(BasicTrustRootIndex.java:32)
	at okhttp3.internal.platform.Platform.buildTrustRootIndex(Platform.java:303)
	at okhttp3.internal.platform.Platform.buildCertificateChainCleaner(Platform.java:173)
	at okhttp3.internal.tls.CertificateChainCleaner.get(CertificateChainCleaner.java:41)

I noticed in the most recent commit 5ff774fed5a8b075f475bf1d6e9ba857b63c847e that there are some changes around this.
Screenshots or videos
No response
RuneLite version
Runelite Version: Unknown but I presume 1.9.11 (No mention of it in console logs)
Launcher Version: 2.6.1
Logs
client.log
launcher.log

fixed in cfaebe145fbcd7c32439870019d05078b69ed6d3

fixed in cfaebe1

Hero

Is there an ETA when this patch will be released without requiring we build the client ourselves?


Support showing other view modes in Digma embedded Jaeger

Problem: Jaeger currently supports other view modes like statistics and flame-graph. However, the embedded mode in Digma blocks navigation to these views.

Solution: We want to allow the user to access these view modes in Digma while making sure the usercan go back to the Digma augmented trace.

Closed in #54


Sent IMAP Not Saved

Click an email address in colornote.
Compose and send from IMAP account.
Sent mail is NOT in sent folder.
Any chance this can be fixed?
Thx

Same device and version as "Data Loss: Compose from Other App" #1838

Can you provide a debug log for this?

hi, didn't see your request above.
in any case, it seems the drafts were getting saved, in the Drafts folder of a different account.
I fixed it. i think the fix had to do with my 'default account', but i don't recall now.
thx!


Carto transfers not updating slippage bumps

Describe the bug
For example, this bump txn: https://polygonscan.com/tx/0x74259e3159ca7a7b8fdd5c088695184e7a9404bc9a12c6b640c5d1947a6390b0
It's reflected in subgraph but not Carto.
Additional context
We're seeing Carto errors in bindTransfers context.
@oncall

1 TX:
https://connextscan.io/tx/0x3c41dc4382c59bdf62f65f929a2ef6aa9426f2abbb949e5d36220167ab6c705d
Bump:

https://polygonscan.com/tx/0x74259e3159ca7a7b8fdd5c088695184e7a9404bc9a12c6b640c5d1947a6390b0
https://polygonscan.com/tx/0x659b73b5a4b4f34c29d95b308de2256c3ce6b87715db25d7c624e91af4761ef2

2 TX: https://connextscan.io/tx/0xbcee0465ee35ae6989c199e132bb262a0a8455b83745d57b14c5b9d4400df14d
Bump: https://polygonscan.com/tx/0xe7f729913efe11a5c6fd8a37bd5c09396975518b5fe3d70e5e717f6013e24352
3 TX:
https://connextscan.io/tx/0xee2624d208e7bf8cf37efc01244e92db68306d8cf84f67378be68bba743cbac5
Bump:
https://polygonscan.com/tx/0x7cb4f82183a0ee61b2eb01f49f5a605621b5814c0a314619866b31fbfa03287b
4TX:
https://connextscan.io/tx/0x37a5c58f3e96e890d3ecd6751e38704cacd46b44306f44e3ae4dfa7d6de49352
Bump:
https://polygonscan.com/tx/0x639b90329f34fe5178c4c86cfe523ae8e56b240d0f15c81e279d683f1606caac

Working with bware, will push a hotfix as needed, shortly.

Root caused it:
carto poller for fee increases got stuck on polygon, fetching the same 100 from the checkpoint<PHONE_NUMBER> which is likely when we did mass realer fee bumps.

Resolved.
These transfers are no completed.


OnNotificationOpened is not called when app is closed.

Bug Information
OnNotificationOpened is not calling when the app is terminated.
Version Number of Plugin: 2.3.0
Device Tested On Samsung s6 and iPhone 6+
Simulator Tested On:  No
Version of VS: 15.7.3
Version of Xamarin: <IP_ADDRESS>3944
Versions of other things you are using:
Steps to reproduce the Behavior
Expected Behavior
OnNotificationOpened should be called.
Actual Behavior
OnNotificationOpened is not calling.
Code snippet
Body:
{
   "to":"/topics/debug",
   "data":{
   		"priority":"high",
   		"sound":"default",
   		"id": 3,
    	"message":"msg"
   },
   "notification":{
   		"priority":"max",
      "title":"Sai Mobile Apps"
      "body":"msg",
      "sound":"default",
      "content_available":true
   }
}

Screenshots

Remove notification from the payload:
 "notification":{
   		"priority":"max",
      "title":"Sai Mobile Apps"
      "body":"msg",
      "sound":"default",
      "content_available":true
   }

Send this instead:
{
   "to":"/topics/debug",
   "data":{
   		"priority":"high",
   		"sound":"default",
   		"id": 3,
    	"message":"msg"
   }
}


Problem with compat with Genetic animals - 1.18.2

It happened randomly 2 times (1 cow and 1 chicken) but when I catched them with the diamond mob catcher, their gene are reinitialized to defaults. The two mob that changed where spawned when de world loaded.
It was in creative with minecraft 1.18.2 Forge (GeneticAnimals-0_8_25.jar). I was playing in a LAN server (I was not the host)
I wish I had more details on how it happened but except those two random times, I was not able to reproduce the issue with mobs I spawned with the egg.
I will drop server logs tomorrow to help fix the bug :D

latest.zip
Here is the log, I checked it quickly, but I did not find interesting info

The catcher simply creates a new mob of the type, so and modifications to the mob are lost.

I will not be pursuing this enhancement


Fix links on non-default port (and not https)

When I run locally, all the links appear as https://localhost/* regardless of the port I’m using. This fixes that.

Changes Unknown when pulling 84c7123a190d4f4e888b882041bdfc20d232e9ee on andylolz:fix-urls into ** on theodi:master**.

Changes Unknown when pulling cab64f6356704b9213f55c75391a631a0ea8fd77 on andylolz:fix-urls into ** on theodi:master**.

Changes Unknown when pulling 5c7be1b04f19e6548af15630d05932685bd99a3d on andylolz:fix-urls into ** on theodi:master**.


WIP: Ghidra support

WIP, still need table BinSync server.

This will be closed until further development on the core is ready


How to do distributed training with backbone written by myself

My backbone runs normally on a  single card, once the distributed training is carried out, the following error occurs：
File "./train.py", line 120, in main                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 20, in init_dist                                        │
_init_dist_pytorch(backend, **kwargs)                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 34, in _init_dist_pytorch                               │
init_dist(args.launcher, **cfg.dist_params)                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 20, in init_dist                                        │
init_dist(args.launcher, **cfg.dist_params)dist.init_process_group(backend=backend, **kwargs)                                                                 │
│
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 547, in init_process_group                  │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 20, in init_dist                                        │
_init_dist_pytorch(backend, **kwargs)                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 34, in _init_dist_pytorch                               │
_init_dist_pytorch(backend, **kwargs)                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/dist_utils.py", line 34, in _init_dist_pytorch                               │
dist.init_process_group(backend=backend, **kwargs)                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 547, in init_process_group                  │
dist.init_process_group(backend=backend, **kwargs)                                                                                │
_store_based_barrier(rank, store, timeout)                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 547, in init_process_group                  │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 207, in _store_based_barrier                │
time.sleep(0.01)                                                                                │
KeyboardInterrupt                                                                                │
_store_based_barrier(rank, store, timeout)                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 207, in _store_based_barrier                │
_store_based_barrier(rank, store, timeout)                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 207, in _store_based_barrier                │
time.sleep(0.01)                                                                                │
KeyboardInterrupt                                                                                │
time.sleep(0.01)                                                                                │
KeyboardInterrupt                                                                                │
Traceback (most recent call last):                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/runpy.py", line 193, in _run_module_as_main                                                            │
"main", mod_spec)                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/runpy.py", line 85, in _run_code                                                                       │
exec(code, run_globals)                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/launch.py", line 173, in                                       │
main()                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/launch.py", line 169, in main                                          │
run(args)                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/run.py", line 624, in run                                              │
)(*cmd_args)                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 116, in call                                │
return launch_agent(self._config, self._entrypoint, list(args))                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 348, in wrapper      │
return f(*args, **kwargs)                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 238, in launch_agent                            │
result = agent.run()                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/elastic/metrics/api.py", line 125, in wrapper                          │
result = f(*args, **kwargs)                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/elastic/agent/server/api.py", line 700, in run                         │
result = self._invoke_run(role)                                                                                │
File "/home/guozebin/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/elastic/agent/server/api.py", line 828, in _invoke_run                 │
time.sleep(monitor_interval)                                                                                │
KeyboardInterrupt
**I think it should be the problem of initialization, but I don’t know how to solve it**

@Bin-ze Please make sure that the distributed start command is correct and please use the issue template.


Feedback providers should export a VT escape sequence for other Terminals

Summary of the new feature / enhancement
In order to have extra UI or visual changes when feedback providers trigger in PowerShell in other terminals, i.e VSCode terminal, we need way to indicate to the terminal in a general way to notify a feedback provider triggered. We should have a VT escape sequence to notify other terminals, or at least have some logic to determine we are in VSCode and output that sequence when feedback providers trigger.
Proposed technical implementation details (optional)
Output a VT escape sequence with every feedback provider

have some logic to determine we are in VSCode

$env:TERM_PROGRAM -eq 'vscode' tells you that.
More generally, $host.UI.SupportsVirtualTerminal provides this information for any given host('s current configuration).

$env:TERM_PROGRAM -eq 'vscode' tells you that.
More generally, $host.UI.SupportsVirtualTerminal provides this information for any given host('s current configuration).

Neither mechanism gets through SSH automatically to a remote session, whereas the TERM environment variable does get forwarded.
PS> $host.UI.SupportsVirtualTerminal
True
PS> invoke-command -HostName foo -ScriptBLock { $host.UI.SupportsVirtualTerminal }
False

Good point about the commands I mentioned only working locally (not also in remoting sessions).

the TERM environment variable does get forwarded.

That is true if you use the ssh utility, but If you enter an interactive session via Enter-PSSession -HostName foo, TERM is not set.

As an aside: If PowerShell is the default shell for the targeted user, something like
TERM=xterm-mono ssh jdoe@someserver (ditto for TERM=dumb) results in a broken interactive experience at least on macOS; see also (your own issue): https://github.com/PowerShell/PSReadLine/issues/3918

While $host.UI.SupportsVirtualTerminal  is always $false in a remote PowerShell session,
$PSStyle.OutputRendering still defaults to Host in the absence of a TERM variable, and PowerShell commands do produce colored output.
It is debatable whether this is the right default and arguably Enter-PSSession too should set TERM, but, leaving that aside, user code that produces strings for the interactive command-line editing experience could base its decision as to whether to use VT/ANSI escape sequences on $PSStyle.OutputRendering -in 'Host', 'Ansi'


feat: remove support for Python 3.8

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

[X] Make sure to open an issue as a bug/issue before writing your code!  That way we can discuss the change, evaluate designs, and agree on the general idea
[X] Ensure the tests and linter pass
[X] Code coverage does not decrease (if any source code was changed)
[X] Appropriate docs were updated (if necessary)

Fixes #101  🦕
https://github.com/googleapis/langchain-google-firestore-python/issues/101

/gcbrun


Add C docs

Hi, thanks for implementing that library!
Could you please add C documentation (by analogy with e.g. arduino-bme280), so that we have docs nicely rendered at https://mongoose-os.com/docs/api/mgos_onewire_rmt.h.html ?
Thanks!

Done.


[WIP] Restructured for Go standard project layout

I've got Nikhil's work rebased and refactored to account for changes on master.
Currently, it looks like it's failing the e2e-kind test. I'm working on replicating the issue in my own environment to evaluate.

Looks like the binary it's producing is a problem...
./bin/multus
./bin/multus: line 1: syntax error near unexpected token `newline'
./bin/multus: line 1: `!<arch>'

Looks like the build was referencing building something that was not main so I cleared that up, I believe.
However, it's failing during a docker build for the Dockerfile.openshift, with a weird error on copy. Nikhil had reported this, and I can't replicate locally. Needs some more digging.

For example, local run looks like: https://pasteall.org/Nnok


Support reload of instances of systemd --user

Pull Request (PR) description
This change increased the minimum required Puppet version to 6.24.0 or 7.9.0 PUP-5704 to support arrays to the command attribute of the exec type.
Support the calling the systemd --user daemon-reload for a particular user.
Example run:
notify{'junk':
 notify => Systemd::Daemon_reload['user_foobar'],
}

systemd::daemon_reload{'user_steve':
  user => 'steve',
}

This results on a Fedora box:
Notice: /Stage[main]/Main/Notify[junk]/message: defined 'message' as 'junk'
Notice: /Stage[main]/Main/Systemd::Daemon_reload[user_steve]/Package[systemd-container]: Triggered 'refresh' from 1 event
Notice: /Stage[main]/Main/Systemd::Daemon_reload[user_steve]/Exec[systemd-user_steve-systemctl-user-steve-daemon-reload]: Triggered 'refresh' from 1 event

and a journal (debug on) for<EMAIL_ADDRESS>of
Mar 29 10:32:11 fedora systemd[2062]: Created slice background.slice - User Background Tasks Slice.
Mar 29 10:32:11 fedora systemd[2062]: Starting systemd-tmpfiles-clean.service - Cleanup of User's Temporary Files and Directories...
Mar 29 10:32:11 fedora systemd[2062]: Reloading requested from client PID 4379 ('systemctl')...
Mar 29 10:32:11 fedora systemd[2062]: Reloading...
Mar 29 10:32:11 fedora systemd[2062]: Reloading finished in 179 ms.
Mar 29 10:32:11 fedora systemd[2062]: Finished systemd-tmpfiles-clean.service - Cleanup of User's Temporary Files and Directories.

Only recent versions of machinectl support the shell option or actually manage to connect to DBUS.
This Pull Request (PR) fixes the following issues

Fixes #444

@traylenator I think the bump to Puppet >= 7.9 is missing? (And I'm still torn if that's a breaking change or not).

systemd-run --pipe --wait --user --machine<EMAIL_ADDRESS>systemctl --user daemon-reload

is a better command since the return code makes it out.


User flow to request Acces to Dashboard

It's a bit tricky / heavy for content creators to provide access to Dashboards.
When users open a link to a dashboard or slice they don't have access to, here are a few things that could happen:

Error message
Request access button, maybe a field to specify which Admin user should get notified
A row is added in a AccessRequest model
Admins can go to the crud view for AccessRequest and see a list of request, filter to requests meant for them (or not), check a bunch of boxes of what they want to approve, and approve a set

There's a choice on whether the datasource perm should be added to one of the user's role, or if a new role that has the datasource should be added to the user's profile...

it's a wondeful feature if it can complete, then the access management can be more easily..

Is anyone working on this? Would be wonderful to be able to give access to single/multiple dashboards to different users!


out attribute in Kamelet type CRD is in documentation example but not in yaml schema

from https://camel.apache.org/camel-k/1.7.x/kamelets/kamelets-dev.html#_creating_a_simple_kamelet
apiVersion: camel.apache.org/v1alpha1
kind: Kamelet
metadata:
  name: twitter-search-source
  labels:
    camel.apache.org/kamelet.type: "source"
spec:
  definition:
    title: "Timer"
    description: "Produces periodic events with a custom payload"
    required:
      - message
    properties:
      period:
        title: Period
        description: The time interval between two events
        type: integer
        default: 1000
      message:
        title: Message
        description: The message to generate
        type: string
  types:
    out:
      mediaType: text/plain
  flow:
    from:
      uri: timer:tick
      parameters:
        period: "{{period}}"
      steps:
        - set-body:
            constant: "{{message}}"
        - to: "kamelet:sink"

in spec -> types, there is an out tag but there is no out tag defined in crd yaml schema
https://github.com/apache/camel-k/blob/48422c08bc20295d09cb31694adc177726806d6b/helm/camel-k/crds/crd-kamelet.yaml#L271-L426

is it normal? like we can have whatever attributes at this level?

It's "normal" because the types field has been modelled as a map, but it should probably be modelled as a comprehensive struct.

I think it is still a feature that can help users a lot. So putting comment to remove stale status

I'm working on this. Sending a PR soon.


Not working at all

Anyone tried this tutorial which is not working at all?
majorly because of the new web server aiohttp. We need a new startup command to launch it.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 53c67a1a-6151-18a9-b5a4-bc84693592bc
Version Independent ID: 61e09eab-c8f9-097e-d0d7-b75208d457d3
Content: Tutorial to deploy a basic bot - Bot Service
Content Source: articles/v4sdk/bot-builder-tutorial-deploy-basic-bot.md
Service: bot-service
GitHub Login: @mmiele
Microsoft Alias: kamrani

@jinpenglv  Asking a dev for information. Hope to get back to you soon.

Do we have details about what isn't working?  We released the samples for Python some time ago, using aiohttp.  The ARM templates take this into account when deploying the bot.  See "appCommandLine" in the ARM templates.

@jinpenglv - I was able to successfully follow the instructions as written for creating a new Python bot from scratch and deploying it to a new resource group. Please provide additional information.

Closing. Please reopen if you need more help.


Node status filter not working

Node status filter is broken on mainnet
Screencast from 18-01-24 16:22:43.webm

Investigation and Solution:
I see the issue is on mainnet locally, but it works in devnet and ninja, still investigating.

blocked on #2196

For Node 309 it came as a down node with status up nodes

Hello ya Samar, the issue happened originally from gridproxy, it was reported on the issue here https://github.com/threefoldtech/tfgrid-sdk-go/issues/620, it should be fixed and deployed on dev/ninja.
if you need to use it locally you can run the proxy locally on the tag v0.13.11, and add the local server on the dashboard env file as the gridproxy url

verified on https://dashboard.bknd1.ninja.tf/.

verified on https://dashboard.bknd1.ninja.tf/.
searched for more than 10 pages and still working fine

Verified:

https://dashboard.bknd1.ninja.tf/
2.3.0-alpha9


No information on readme about participating

there's no information on readme about participating. It is kind of a bad experience for people willing to apply.

What kind of hackathons you want to participate in. Guild-based or team-based?

I don't know about anything related to it. I just came to this repo and there was no information. Thus, opened this issue (first asked on discord then @Rahma-AlWadhahi recommended to do so). So I don't know anything

I think that it would be great to add some links to the previous Hackathon. For example, people have participated in Local Hack Day, why isn't there any link to that if it happens in different times? Like maybe having a link to the next possible date for the Hackathon to occur?
This way, we will show the people's work as well as get more potential participants in the next edition of the Hackathon whether it is Local Hack Day or any other Hackathon.

@Rahma-AlWadhahi You are right. So we can add previous hackathons and coming dates and timing to Share & build hackathons!

Closing this on @kaiwalyakoparkar's concern (🆙)


[FEATURE] Extend support for Bedrock beyond Anthropic to all other models

Is your feature request related to a problem?
Although it's quite possible that the current implementation of RAG works with all other models supported by Bedrock, we need to ensure it works with more than just Anthropic Claude.
What solution would you like?
A clear and concise description of what you want to happen.
What alternatives have you considered?
A clear and concise description of any alternative solutions or features you've considered.
Do you have any additional context?
Add any other context or screenshots about the feature request here.

Hi @austintlee @ylwu-amzn  - I believe we can make this easier
I’m the maintainer of LiteLLM - we allow you to deploy an LLM proxy to call 100+ LLMs in 1 format - Bedrock, OpenAI, Anthropic etc https://github.com/BerriAI/litellm/tree/main/openai-proxy.
If this looks useful (we're used in production)- please let me know how we can help.
Usage
Bedrock request
curl http://<IP_ADDRESS>:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "model": "bedrock/anthropic.claude-instant-v1",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'

gpt-3.5-turbo request
curl http://<IP_ADDRESS>:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "model": "gpt-3.5-turbo",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'

claude-2 request
curl http://<IP_ADDRESS>:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "model": "claude-2",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'

@ishaan-jaff Yes, conceptually, something like this would be ideal.  However, I am not sure about running a proxy.  What I had in mind is to have this conversion layer in ml-commons itself.  Are you familiar with OpenSearch?  This might be a good use case for the OpenSearch Python extension (@dblock @dbwiddis).  @ylwu-amzn I think of this as an all-in-one HttpConnector (but implemented in Python).  It is certainly an interesting idea and I think it might be easier for people to use it if it's an extension (as in an extension of OpenSearch) as opposed to another thing people have to download and install and manage, although the quickest way to put this in the hands of the users is to have the HttpConnector talk to this proxy as-is.

@ishaan-jaff Thanks for sharing this. The consistent input/output feature definitely helpful.

the quickest way to put this in the hands of the users is to have the HttpConnector talk to this proxy as-is.

Agree with @austintlee , seems this proxy could be some layer between OpenSearch HttpConnector and LLM, which adapts the HttpConnector and LLM.

@austintlee @ylwu-amzn how can we be most helpful here?
sounds like there are two options:

use the litellm package for consistent LLM I/O
use the litellm proxy server for consistent LLM I/O

The first option won't work because we don't have a language binding for Python or to any language other than Java.
The second option works out of the box (in theory).  Maybe you can try out a few examples and one way to contribute or let people know about it is to write a blog post?

#2826 covers this.


[FEATURE] Predefined variables for CPU/Memory usage, etc

Description
CPU and RAM usage variables for cpu and ram usage monitors.
Configuration syntax
In a box you could put like, <cpu></cpu> or something like that and then configuration and label formatting goes elsewhere

If anything, these would be added as "magic variables". For that, we'll first need to implement the #52 lazy evaluation ticket, to avoid having code run unnecessarily.

Wait why do we need to implement #52 first? Isn't the issue wanting Haskell in eww or something?

yea, that issue was incredibly stupidly framed. I updated the description, now it should actually explain what it does ;D

it wasn't stupid.. i was lazy

Well that issue is closed now but im not sure if this is a good idea maybe if its under a feature or something or as part of example eww widgets or smth

Well that issue is closed now but im not sure if this is a good idea maybe if its under a feature or something or as part of example eww widgets or smth

@legendofmiracles has started working on this in https://github.com/legendofmiracles/eww/tree/predefined-variables

Implemented in #117


When using template translation state doesn't work anymore

When i use a template for the state, the translation doesn't work anymore:

This is intended, overwriting displayed state with your own value


Feature: Verbose output

Add a --verbose option. When used, print for example when there's no rule unconfigured (atm. this has no output at all).
We could easily make use npm-log for this.

How should we assign the output to different log levels?
By default, npmlog (I spelled it wrong in the op) comes with these log levels:

silly: I think this will be unused, but could eventually be used to simply log everything.
verbose: Print things like "there's no rule unconfigured" (relates to the original package's issue #24.
info: I think this one should become the default output level.
http: we do not make any http request, so it'll stay unused.
warn & error: obviously as usual, when there's sth that went wrong.

Any other ideas? Did I miss anything?

Sounds good!
Wondering if silly and verbose should be combined.

Yeah I think we can skip silly completely, but verbose is sth. I'd really like to have in.

We might want to address #9 along with this.

Yep

Another thing that came into my mind, when making the tabular output / sorting the rules' names alphabetically:
We probably should also group those rules, like in a batch of rules for eslint (default) and all loaded plugins eslint-plugin-import etc. ...

Go for it.

@ta2edchimp do you think we can close this issue?

Yep.


Specto...Fork - Will not generate RealDebrid Authorize Code

Please follow the guide below

Put an "x" into all boxes [ ] relevant to your issue
Use Preview tab to see how your issue will actually look like

Kodi & Addon

Kodi verison: _____
[ ] Filmy online - Mrknow
[ ] MrknowTV
[X ] Specto
[ ] FanFILM
Version /  2<IP_ADDRESS>

Descryption / Opis
My version of Specto...Fork will not generate the RealDebrid activation code.  When I go into settings, Accounts, and click on the RealDebrid Activate Kodi field to generate the code, the screen clears, then comes back to the TV tab in settings.  No activationcode is ever generated.  I cannot use RealDebrid with Specto...Fork.  I have a RealDebrid account.  This works for Exodus and others.  Any assistance you can provide to help me resolve this problem would be greatly appreciated.  I have Uninstalled / Reinstalled Specto several times and still it does not work.
Donation / Darowizna
This addons are free, but you can always thank the author for ... / Te wtyczki są darmowe, ale zawsze możesz podziękować autorowi za jej stworzenie...

Nobody's is working as far as I know. Same for thevideo and openload.
On Jul 15, 2017 2:11 AM, "steve02usa"<EMAIL_ADDRESS>wrote:

Please follow the guide below

Put an "x" into all boxes [ ] relevant to your issue
Use Preview tab to see how your issue will actually look like

Kodi & Addon

Kodi verison: _____
Filmy online - Mrknow
MrknowTV
[X ] Specto
FanFILM
Version / 2<IP_ADDRESS>

Descryption / Opis
My version of Specto...Fork will not generate the RealDebrid activation
code. When I go into settings, Accounts, and click on the RealDebrid
Activate Kodi field to generate the code, the screen clears, then comes
back to the TV tab in settings. No activationcode is ever generated. I
cannot use RealDebrid with Specto...Fork. I have a RealDebrid account. This
works for Exodus and others. Any assistance you can provide to help me
resolve this problem would be greatly appreciated. I have Uninstalled /
Reinstalled Specto several times and still it does not work.
Donation / Darowizna
This addons are free, but you can always thank the author for ... / Te
wtyczki są darmowe, ale zawsze możesz podziękować autorowi za jej
stworzenie...
[image: paypal]
https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=HEV6TJY83S3Q4&lc=PL&item_name=Donation-Darowizna&currency_code=PLN&bn=PP-DonationsBF%3Abtn_donateCC_LG.gif%3ANonHosted
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/mrknow/filmkodi/issues/920, or mute the thread
https://github.com/notifications/unsubscribe-auth/AUPc7JHBNPZK7NBhv_iRt0Ibgg7sdSp-ks5sOFgKgaJpZM4OY7iz
.

I wouldn't expect a fix anytime soon.
On Jul 15, 2017 2:11 AM, "steve02usa"<EMAIL_ADDRESS>wrote:

Please follow the guide below

Put an "x" into all boxes [ ] relevant to your issue
Use Preview tab to see how your issue will actually look like

Kodi & Addon

Kodi verison: _____
Filmy online - Mrknow
MrknowTV
[X ] Specto
FanFILM
Version / 2<IP_ADDRESS>

Descryption / Opis
My version of Specto...Fork will not generate the RealDebrid activation
code. When I go into settings, Accounts, and click on the RealDebrid
Activate Kodi field to generate the code, the screen clears, then comes
back to the TV tab in settings. No activationcode is ever generated. I
cannot use RealDebrid with Specto...Fork. I have a RealDebrid account. This
works for Exodus and others. Any assistance you can provide to help me
resolve this problem would be greatly appreciated. I have Uninstalled /
Reinstalled Specto several times and still it does not work.
Donation / Darowizna
This addons are free, but you can always thank the author for ... / Te
wtyczki są darmowe, ale zawsze możesz podziękować autorowi za jej
stworzenie...
[image: paypal]
https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=HEV6TJY83S3Q4&lc=PL&item_name=Donation-Darowizna&currency_code=PLN&bn=PP-DonationsBF%3Abtn_donateCC_LG.gif%3ANonHosted
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/mrknow/filmkodi/issues/920, or mute the thread
https://github.com/notifications/unsubscribe-auth/AUPc7JHBNPZK7NBhv_iRt0Ibgg7sdSp-ks5sOFgKgaJpZM4OY7iz
.

Dish lawyers hard at work I see...

Yup. It's cast a dark cloud over the Kodi world. This is an important case
that's being watched by many other players and other significant court
cases of late have been bad news. Hopefully it all blows over.
On Jul 16, 2017 3:22 PM, "uvxcoin"<EMAIL_ADDRESS>wrote:

Dish lawyers hard at work I see...
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/mrknow/filmkodi/issues/920#issuecomment-315630885,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AUPc7Ow1xnIF3ojuFMbPMeFUjkxn_Uj6ks5sOmLTgaJpZM4OY7iz
.

confirm. can't add accounts either

Ditto for me.  No response from the reauthorize link.  Other links on the accounts page work.

Another confirmation. It is not working in Specto. Such a pity as Specto is by far the easiest video add-on to use imho. Until then it is back to Exo.....

I am starting to believe he won't be back. One likes to think it's because
of the Dish court case, but that's all to do with live streams, so not
necessarily urgent for someone like Specto to disappear because of it, but
who knows. I think the time has come for developers to take greater caution
as far as their identities, and perhaps that's what all these disappeared
guys are doing; getting their identities secured. No one wants to end up
owing x number of dollars per infringement, for countless infringements.
That would be a bad day.
On Jul 24, 2017 10:58 PM, "spectofan"<EMAIL_ADDRESS>wrote:

Another confirmation. It is not working in Specto. Such a pity as Specto
is by far the easiest video add-on to use imho. Until then it is back to
Exo.....
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/mrknow/filmkodi/issues/920#issuecomment-317614610,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AUPc7MnrjMDbDPLQNJKS8PqCE9T6yedEks5sRVnwgaJpZM4OY7iz
.


run_job_template action issue

Hi,
Using run_job_template action in my rulebook. It triggers the job but then giving below error where it tries to get the result from the template. I think it is caused because of the double quotes after the job ID. Is anyone else having the same issue?
ansible_rulebook.exception.ControllerApiException: Failed to get from https://<controller_url>/api/v2/jobs/jobID//job_events/. Status: 404, Body: {"detail":"The requested resource could not be found."}
Thanks
Jon

@bzwei

Hi @jonsaydam, seems that this is an issue for https://github.com/ansible/ansible-rulebook
We would need the output of ansible-rulebook --version as well as the rulebook and any other relevant data that allows us to reproduce the error.

@jonsaydam I believe the issue has been fixed. Try to use the latest ansible-rulebook.

The issue is resolved in the new version. Thanks


When we remove the Account lookup on a Contact, system should fill in the Account lookup if it can find an Account that Primary Contact = Current Contact

Create a Contact with either Admin or HH account model. System will automatically create an Account for the Contact.  And mark current Contact as the Primary Contact of the new Created Account.
Now, try to remove the Account lookup of the Contact and click Save. You will see that now the Account Lookup is empty of the Current Contact.
The behavior is wrong here since every Contact must associate with an Account.
The correct behavior is that system should fill in the Account lookup if it can find an Account that Primary Contact = Current Contact

**lurch:add

Tracking W-023054

@weizsforg Why would the user want to clear that field in the first place? Is this really a bug?

Hi @ceiroa , yes, I think it is an unusual use case.
Actually, the reason that I create this bug is because HEDA has the existing code to set Primary Contact for this use case. However, the code has a bug so it does not save the Primary Contact field on Account record. Also, NPSP has the same logic and it saved the Primary Contact on Account record.

**lurch: attach W-035178.

Tracking W-035178


[Bug] [UI] The data quality tab page cannot switch with other pages normally

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened

When I click the data quality tab page, I can't switch to other tabs
What you expected to happen
The tab page can be switched normally
How to reproduce
Click DataQuality tab page ，and click other page
Anything else
No response
Version
dev
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

that's a frontend bug, i want to fix it. @caishunfeng @labbomb


Unable to install from NPM / Source

Hai there.
Im trying to give µWS a go but am struggling to get it installed. I've already compiled >= cmake 3, and compiled libuv 1.0.0 without a problem.
Installing trough npm, and compiling from git yields to the same compiler error for me now though:
In file included from ../src/uWS.cpp:1:0:
../src/uWS.h: In constructor ‘uWS::Server::Server(int, bool, std::string)’:
../src/uWS.h:87:9: warning: ‘uWS::Server::port’ will be initialized after [-Wreorder]
../src/uWS.h:86:22: warning:   ‘bool uWS::Server::master’ [-Wreorder]
../src/uWS.cpp:163:1: warning:   when initialized here [-Wreorder]
../src/uWS.cpp: In lambda function:
../src/uWS.cpp:213:52: error: ‘this’ was not captured for this lambda function
../src/uWS.cpp: In lambda function:
../src/uWS.cpp:217:54: error: ‘this’ was not captured for this lambda function
../src/uWS.cpp: In lambda function:
../src/uWS.cpp:633:41: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
make: *** [Release/obj.target/uws/src/uWS.o] Error 1

Am i doing something really obvious wrong here?

If you want the Node.js addon you simply hit "npm install uws". This install never fails and does not print any output (except for to build_log.txt). Did you find this output in build_log.txt? Are you on Windows?

Totally forgot to mention that, I'm sorry. Yes this is from the build log which I checked because after installing it with npm and requiring it in my code I was greeted with a message that uws failed to compile and there's no precompiled version for my OS. I am on Linux, Debian to be exact.

What major version of Node? Even if the compilation fails it should fallback to a pre compiled binary.

Currently running 5.1.0

Well this should not happen. There are pre-built binaries for 64-bit Linux with Node 5, so in worst case it should just use that binary. Are you on 64-bit? What version of GCC/Clang? What version of NPM?

Yes I am on 64 bit. My npm version is 3.3.12, gcc is 4.7.2, cmake is at 3.2.2. Thank you

Well I just installed uws 0.4.1 on my secondary computer with "npm install uws" and both compilation and fallback works. I guess the problem in this could be your ancient GCC version causing the compilation to fail and also the pre-compiled binary to not be compatible.
This lib is written in C++11, which became fully supported in GCC 4.8. I cannot see anything making sense at this error report, "this" is not even used in the lambda at all so reporting an issue with the capture of "this" is not correct as far as what I can see.
How does your node_modules/uws folder look like? Mine looks like this:
[alexhultman@localhost uws]$ ls
binding.gyp    logo.png      uws_darwin_46.node  uws_linux_46.node
build          package.json  uws_darwin_47.node  uws_linux_47.node
build_log.txt  README.md     uws_darwin_48.node  uws_linux_48.node
LICENSE        src           uws.js

What I'm wondering about is if you have the "uws_linux_47.node" file or not (that would be the pre-compiled binary for Node.js 5.x).

The file indeed does exist
...
ls node_modules/uws/
binding.gyp    logo.png      uws_darwin_46.node  uws_linux_46.node
build          package.json  uws_darwin_47.node  uws_linux_47.node
build_log.txt  README.md     uws_darwin_48.node  uws_linux_48.node
LICENSE        src           uws.js
...
I will try upgrading my gcc and give it another go.

I have gcc 4.9 now, compile and install worked (now via npm too). The only thing I had to change from the docs is doing io.engine.ws = new(require('uws')).Server({ instead of io.engine.ws = new require('uws').Server({ as it would otherwise tell me that class constructors cannot be called without new.
Works now though, sorry for the inconveniences and thank you for your time!

Nice to hear. Don't worry about it, I find these kinds of reports about how it behaves on different set-ups useful. I had no idea it was broken on GCC 4.7 and prior versions - that's good to know :wink: If the docs about Socket.IO is wrong I can update it, I got this snippet from one of the other Socket.IO users and just copy-pasted it.

I found the issue. You might want to consider .loging the error that comes from the try catch, as the issue in my case simply was a missing (outdated) libc6.

What is the reopening about then?

Im sorry im on my phone and must have hit the wrong button.


Add public root method

This adds a public root() method to the shallow and mount wrappers. This is a reasonable public API to have, and chai-enzyme needs this for a couple of things.

@lelandrichardson Can we add some tests for this method?


[CHORE] adding ebpf collector

As proposed on this https://github.com/open-telemetry/opentelemetry-ebpf/issues/186, I'm adding the eBPF collector as part of the collector helm chart.

cc @atoulme

Nice! Thanks for this work.

@nicolastakashi what would you think of making this a new top level chart? @TylerHelmuth is there a process to add a new chart?

@atoulme new charts should be proposed via an Issue.

@nicolastakashi I do not have the context necessary to review this PR.  Can you create an issue (or can we move the issue linked to this PR) to discuss what problem this PR solves?

I suggest making it another helm chart. Later, if we want it to be installable along with the collector, it can be set as a subchart. Or, just provide instructions on how to install it with the collector

Going to open another PR, cause I was struggling with conflicts


Completion of all the pages

Need to complete the following pages ranked in order of importance

[x] Home Page
[x] Events Page
[ ] Sign Up/Login
[x] Teams
[ ] Profile Page
[ ] Contact Us

Remember that we are using Material Design

Sounds good! I'll get back on it tonight after my exam

@madewithsmiles made some changes to the initial issue