Reproduce LVQ and Leanvec speedup

Hi, I am using the cpp example in the documentation with the test dataset svs/data/test_dataset, but LVQ and Leanvec indexes takes longer to complete the search. Can anyone help me to check what may have caused this issue?
running lvq8 vs uncompressed:
//! [Example All]

//! [Includes]
// SVS Dependencies
#include "svs/orchestrators/vamana.h"  // bulk of the dependencies required.
#include "svs/core/recall.h"           // Convenient k-recall@n computation.
#include "svs/extensions/vamana/lvq.h" // LVQ extensions for the Vamana index.

// Alternative main definition
#include "svsmain.h"

// stl
#include <chrono>
#include <map>
#include <string>
#include <string_view>
#include <vector>
//! [Includes]

const std::string save_prefix = "index/";

//! [Helper Utilities]
double run_recall(
    svs::Vamana& index,
    const svs::data::SimpleData<float>& queries,
    const svs::data::SimpleData<uint32_t>& groundtruth,
    size_t search_window_size,
    size_t num_neighbors,
    std::string_view message = ""
) {
    index.set_num_threads(12);
    auto search_start = std::chrono::high_resolution_clock::now();
    index.set_search_window_size(search_window_size);
    auto results = index.search(queries, num_neighbors);
    auto search_end = std::chrono::high_resolution_clock::now();

    double recall = svs::k_recall_at_n(groundtruth, results, num_neighbors, num_neighbors);
    if (!message.empty()) {
        fmt::print("[{}] ", message);
    }
    auto duration = std::chrono::duration<double>(search_end - search_start).count();
    fmt::print("Windowsize = {}, Duration = {}, Recall = {}\n", search_window_size,  duration, recall);
    return recall;
}

const bool DEBUG = false;
void check(double expected, double got, double eps = 0.001) {
    double diff = std::abs(expected - got);
    if constexpr (DEBUG) {
        fmt::print("Expected {}. Got {}\n", expected, got);
    } else {
        if (diff > eps) {
            throw ANNEXCEPTION("Expected ", expected, ". Got ", got, '!');
        }
    }
}
//! [Helper Utilities]

// Alternative main definition
int svs_main(std::vector<std::string> args) {
    //! [Argument Extraction]
    const size_t nargs = args.size();
    if (nargs != 4) {
        throw ANNEXCEPTION("Expected 3 arguments. Instead, got ", nargs, '!');
    }
    const std::string& data_vecs = args.at(1);
    const std::string& query_vecs = args.at(2);
    const std::string& groundtruth_vecs = args.at(3);
    //! [Argument Extraction]

    // Building the index

    //! [Build Parameters]
    auto parameters = svs::index::vamana::VamanaBuildParameters{
        1.2,  // alpha
        64,   // graph max degree
        128,  // search window size
        1024, // max candidate pool size
        60,   // prune to degree
        true, // full search history
    };
    //! [Build Parameters]

    //! [Index Build]
    auto uncompressed_build_start = std::chrono::high_resolution_clock::now();
    size_t num_threads = 12;
    svs::Vamana index = svs::Vamana::build<float>(
        parameters, svs::VectorDataLoader<float>(data_vecs), svs::DistanceL2(), num_threads
    );
    auto uncompressed_build_end = std::chrono::high_resolution_clock::now();
    printf("Uncompressed build time = %.6f\n", std::chrono::duration<double>(uncompressed_build_end - uncompressed_build_start).count());
    //! [Index Build]

    // Searching the index

    //! [Load Aux]
    // Load the queries and ground truth.
    auto queries = svs::load_data<float>(query_vecs);
    auto groundtruth = svs::load_data<uint32_t>(groundtruth_vecs);
    //! [Load Aux]

    //! [Perform Queries]
    auto recall = run_recall(index, queries, groundtruth, 30, 10, "Uncompressed");
    //! [Perform Queries]

    //! [Saving]
    index.save( save_prefix+"example_config",  save_prefix+"example_graph",  save_prefix+"example_data");
    //! [Saving]

    // Reloading a saved index

    //! [Loading]
    // We can reload an index from a previously saved set of files.
    index = svs::Vamana::assemble<float>(
         save_prefix+"example_config",
        svs::GraphLoader( save_prefix+"example_graph"),
        svs::VectorDataLoader<float>( save_prefix+"example_data"),
        svs::DistanceType::L2,
        4 // num_threads
    );

    recall = run_recall(index, queries, groundtruth, 30, 10, "Reload");
    printf("Recall = %f\n", recall);
    //! [Loading]

    // Search using vector compression

    //! [Compressed Loader]
    // Quantization
    size_t padding = 32;
    namespace lvq = svs::quantization::lvq;

    // Wrap the compressor object in a lazy functor.
    // This will defer loading and compression of the LVQ dataset until the threadpool
    // used in the index has been created.
    auto compressor = svs::lib::Lazy([=](svs::threads::ThreadPool auto& threadpool) {
        auto data = svs::VectorDataLoader<float, 128>( save_prefix+"example_data").load();
        return lvq::LVQDataset<8, 0, 128>::compress(data, threadpool, padding);
    });
    index = svs::Vamana::assemble<float>(
         save_prefix+"example_config", svs::GraphLoader( save_prefix+"example_graph"), compressor, svs::DistanceL2()
    );
    //! [Compressed Loader]

    //! [Search Compressed]
    recall = run_recall(index, queries, groundtruth, 30, 10, "Compressed Load");
    printf("Recall = %f\n", recall);
    //! [Search Compressed]

    //! [Build Index Compressed]
    // Compressed building
    auto compressed_build_start = std::chrono::high_resolution_clock::now();
    index =
        svs::Vamana::build<float>(parameters, compressor, svs::DistanceL2(), num_threads);
    auto compressed_build_end = std::chrono::high_resolution_clock::now();
    printf("Compressed build time = %.6f\n", std::chrono::duration<double>(compressed_build_end - compressed_build_start).count());
    recall = run_recall(index, queries, groundtruth, 30, 10, "Compressed Build");
    auto compressed_search_end = std::chrono::high_resolution_clock::now();
    //! [Build Index Compressed]

    //! [Save Index Compressed]
    index.save( save_prefix+"example_compressed_config",  save_prefix+"example_compressed_graph",  save_prefix+"example_compressed_data");
    //! [Save Index Compressed]

    //! [Loading compressed]
    // We can reload an index from a previously saved set of files.
    index = svs::Vamana::assemble<float>(
        save_prefix + "example_compressed_config",
        svs::GraphLoader(save_prefix + "example_compressed_graph"),
        // svs::VectorDataLoader<float>(save_prefix + "example_compressed_data"),
        svs::lib::Lazy([&]() {
            return svs::lib::load_from_disk<svs::quantization::lvq::LVQDataset<
                8, 0>>(save_prefix + "example_compressed_data");
        }),
        svs::DistanceL2(),
        4  // num_threads
    );
    recall = run_recall(index, queries, groundtruth, 30, 10, "Load compressed");
    printf("Recall = %f\n", recall);
    //! [Loading compressed]

    return 0;
}

// Special main providing some helpful utilties.
SVS_DEFINE_MAIN();
//! [Example All]

// ./vamana ../svs/data/test_dataset/data_f32.fvecs ../svs/data/test_dataset/queries_f32.fvecs ../svs/data/test_dataset/groundtruth_euclidean.ivecs

runing leanvec
//! [Example All]

#define SPDLOG_FMT_EXTERNAL 1

//! [Includes]
// SVS Dependencies
// #include "svs/leanvec/leanvec.h"
#include "svs/extensions/vamana/leanvec.h"

#include "svs/core/recall.h"            // Convenient k-recall@n computation.
#include "svs/extensions/vamana/lvq.h"  // LVQ extensions for the Vamana index.
#include "svs/orchestrators/dynamic_vamana.h"  // bulk of the dependencies required.

// Alternative main definition
#include "svsmain.h"

// stl
#include <map>
#include <string>
#include <string_view>
#include <vector>
//! [Includes]

const std::string save_prefix = "index/";

// Alternative main definition
int svs_main(std::vector<std::string> args) {
  //! [Argument Extraction]
  const size_t nargs = args.size();
  if (nargs != 4) {
    throw ANNEXCEPTION("Expected 3 arguments. Instead, got ", nargs, '!');
  }
  const std::string& data_vecs = args.at(1);
  const std::string& query_vecs = args.at(2);
  const std::string& groundtruth_vecs = args.at(3);
  //! [Argument Extraction]

  //! [Build Parameters]
  auto parameters = svs::index::vamana::VamanaBuildParameters{
      1.2,   // alpha
      64,    // graph max degree
      128,   // search window size
      1024,  // max candidate pool size
      60,    // prune to degree
      true,  // full search history
  };
  size_t num_threads = 4;
  //! [Build Parameters]

  auto data_vecs_ = svs::load_data<float>(data_vecs);
  //! [Load Aux]
  // Load the queries and ground truth.
  auto queries = svs::load_data<float>(query_vecs);
  auto groundtruth = svs::load_data<uint32_t>(groundtruth_vecs);
  //   print the dimension
  fmt::print("Data count: {}\n", data_vecs_.size());                    // 10000
  fmt::print("Data dimension: {}\n", data_vecs_.dimensions());          // 128
  fmt::print("Queries count: {}\n", queries.size());                    // 1000
  fmt::print("Queries dimension: {}\n", queries.dimensions());          // 128
  fmt::print("Groundtruth count: {}\n", groundtruth.size());            // 1000
  fmt::print("Groundtruth dimension: {}\n", groundtruth.dimensions());  // 100
  //! [Load Aux]

  std::vector<size_t> ids;
  ids.reserve(10000);
  for (size_t i = 0; i < 10000; i++) {
    ids.push_back(i);
  }

  //! [Search Parameters]
  auto search_params = svs::index::vamana::VamanaSearchParameters();
  search_params.buffer_config(svs::index::vamana::SearchBufferConfig(30));
  //! [Search Parameters]

  bool use_pca = true;  // they only support loading pre-generated OOD matrices

  auto leanvec_data =
      svs::leanvec::LeanDataset<svs::leanvec::UsingLVQ<8>,
                                svs::leanvec::UsingLVQ<8>, 64,
                                svs::Dynamic>::reduce(data_vecs_, num_threads);

  auto loader = svs::lib::Lazy([&leanvec_data]() { return leanvec_data; });

  // Building the index with original vectors
  {
    auto build_start = std::chrono::high_resolution_clock::now();
    auto index = svs::Vamana::build<float>(parameters, loader,
                                           svs::DistanceL2(), num_threads);
    auto build_end = std::chrono::high_resolution_clock::now();
    printf("Build time = %.6f\n",
           std::chrono::duration<double>(build_end - build_start).count());

    index.set_search_parameters(search_params);
    {
      auto search_start = std::chrono::high_resolution_clock::now();
      auto results = index.search(queries, 10);
      auto search_end = std::chrono::high_resolution_clock::now();
      printf("Search time = %.6f\n",
             std::chrono::duration<double>(search_end - search_start).count());

      double recall = svs::k_recall_at_n(groundtruth, results);
      fmt::print("Recall = {}\n", recall);
    }
    //! [Saving]
    index.save(save_prefix + "example_config", save_prefix + "example_graph",
               save_prefix + "example_data");
    //! [Saving]
  }

  // Reloading a saved index
  // We can reload an index from a previously saved set of files.
  {
    auto reloaded = svs::Vamana::assemble<float>(
        save_prefix + "example_config",
        svs::graphs::SimpleBlockedGraph<uint32_t>::load(save_prefix +
                                                        "example_graph"),
        svs::lib::Lazy([&]() {
          return svs::lib::load_from_disk<svs::leanvec::LeanDataset<
              svs::leanvec::UsingLVQ<8>, svs::leanvec::UsingLVQ<8>, 64,
              svs::Dynamic>>(save_prefix + "example_data");
        }),
        svs::DistanceL2(),
        4  // num_threads
    );
    auto search_start = std::chrono::high_resolution_clock::now();
    auto results = reloaded.search(queries, 10);
    auto search_end = std::chrono::high_resolution_clock::now();
    printf("Reload search time = %.6f\n",
           std::chrono::duration<double>(search_end - search_start).count());

    double recall = svs::k_recall_at_n(groundtruth, results);
    fmt::print("Recall = {}\n", recall);
  }

  return 0;
}

// Special main providing some helpful utilties.
SVS_DEFINE_MAIN();
//! [Example All]

// ./leanvec ../svs/data/test_dataset/data_f32.fvecs ../svs/data/test_dataset/queries_f32.fvecs ../svs/data/test_dataset/groundtruth_euclidean.ivecs

Measured search duration
Uncompressed build time = 0.972131
[Uncompressed] Windowsize = 30, Duration = 0.006569277, Recall = 0.8218
[Reload] Windowsize = 30, Duration = 0.006516681000000001, Recall = 0.8218
Recall = 0.821800
[Compressed Load] Windowsize = 30, Duration = 0.029918565, Recall = 0.822
Recall = 0.822000
Compressed build time = 5.918356
[Compressed Build] Windowsize = 30, Duration = 0.035164214, Recall = 0.8215
[Load compressed] Windowsize = 30, Duration = 0.040216505, Recall = 0.8215
Recall = 0.821500

Data count: 10000
Data dimension: 128
Queries count: 1000
Queries dimension: 128
Groundtruth count: 1000
Groundtruth dimension: 100
Build time = 4.987378
Search time = 0.132416
Recall = 0.3101
Reload search time = 0.129005
Recall = 0.3101

Search duration: Leanvec > lvq > uncompressed ?
Build flags:
# building svs
cmake .. -DSVS_EXPERIMENTAL_LEANVEC=1 -DSVS_BUILD_EXAMPLES=YES -DSVS_BUILD_BENCHMARK=YES -DCMAKE_CXX_COMPILER=g++-11 -DCMAKE_INSTALL_PREFIX=../../install
# for the executable
g++-11 -Ofast -lrt -std=c++2a -DHAVE_CXX0X -march=native -fpic -w -fopenmp -ftree-vectorize -ftree-vectorizer-verbose=0 -I. -I$(HOME)/intel/oneapi/mkl/2024.0//include -I$(HOME)/lvq/install/include -DSPDLOG_FMT_EXTERNAL -I$(HOME)/lvq/install/include/eve-2022.9.1 -DSVS_VERSION_MAJOR=0 -DSVS_VERSION_MINOR=0 -DSVS_VERSION_PATCH=3 -DSVS_ENABLE_NUMA=0 -DSVS_CHECK_BOUNDS=1 -DSVS_INITIALIZE_LOGGER=1 vamana.cpp -o vamana -I. -L$(HOME)/lvq/install/lib -l:libfmt.a -l:libspdlog.a -pthread -lm -fopenmp -L$(HOME)/intel/oneapi/mkl/2024.0/lib/ -lmkl_rt

Thanks for your question! These experiments actually make sense in the context of these experiments.
First, the dataset is very small and everything fits in cache. In that situation, compressing vectors by using quantization or dimensionality reduction will barely yield any gains. Increasing the number of vectors is the first step to simulate any real workload.
Second, the dimensionality is relatively low, with only 128 dimensions. Here, dimensionality reduction might not give you noticeable gains.  In most modern use cases, vectors are produced by embedding models that use a high-dimensionality The most common range today is probably 512-1536 dimensions. In this range, you will see a significant improvement from using LeanVec.
Third (and this is related tomy previous point), you are using a random dataset. This statistical distribution has a flat spectra and cannot be easily compressed using linear dimensionality reduction (or can be done while carrying too much loss in search accuracy). The statistical distribution of the vectors produced by embedding models has different characteristics that make it more amenable for reducing the dimensionality.
All in all, if you try a dataset produced by a modern embedding method with more dimensions and containing more vectors, you will then see the right acceleration: LeanVec < LVQ < uncompressed.

Thank you for the detailed explanation. It's great to get some insights directly from the paper author! I really like the idea of LVQ and Leanvec :) Those points make sense, so I tested on the GIST dataset from the ann-benchmark http://ann-benchmarks.com/gist-960-euclidean.hdf5, which is also used in the LVQ and Leanvec paper. I tested for L2 with the original dataset. And I tested MIP after I normalized them to have all vectors with l2 norm = 1 and use brute-force search to generate the true neighbors.

For L2,

Vamana is still faster than LVQ at comparable recall.
Leanvec is performing better than LVQ, but the latency seems to be surprisingly high. On the same machine, I can achieve search time(s): 4.245  recall: 0.991  with hnswlib using the configuration m=32, ef_construction=200 and ef_search=512,

For MIP,

Leanvec < LVQ < Vamana here
Leanvec with MIP gives extremely low recall. I only changed the distance metric and the alpha (1.2->0.,95). The remaining codes are exactly the same.

Using LVQ with MIP seems to be significantly faster than L2, which is not observed on other index libraries like hnswlib.

Hope you could also help me to understand these results.

Thank you @hugy718 for trying out our library and providing feedback. I am attaching two sample scripts below for building index (using fp32) and searching it using fp32, lvq4x8, lvq8, and leanvec for the GIST-1M dataset. With these scripts, for a batch size of 1 on our hardware, we observe the queries per second (qps) of leanvec (~8x) > lvq4x8(~4x) > lvq8(~3x) > fp32(1x) for search window size 32.
The commands used for these experiments are as follows:
numactl -m 1 -N 1 python search_gist.py 1 fp32
numactl -m 1 -N 1 python search_gist.py 1 lvq8
numactl -m 1 -N 1 python search_gist.py 1 lvq4x8
numactl -m 1 -N 1 python search_gist.py 1 leanvec
Please note that the second argument to the search script is the batch size. I conducted the above experiments with a batch size of 1, similar to your runs where a single query is sent at a time, and the index is spawning a single thread. We observe similar trends with higher batch sizes of 10k, where the search utilizes all available threads on a single socket.
Here are a few points to consider:

LVQ/Leanvec performance significantly decreases on systems without AVX512 support. We have validated the above performance trends on Xeon Cascade Lake, Icelake, and newer generations.
Please adjust the threads/max_threads values in the scripts to match the maximum available threads on your system's single socket/node. (Numactl is used to restrict the runs to a single socket to reduce variability between runs.)
The search script executes five iterations for each search window size, and we select the highest qps, similar to the ann/big-ann benchmarks. However, qps from all other runs is also printed. Typically, the first run's qps is lower due to the overhead of thread settings and other initializations.
For MIP experiments, did you normalize the queries as well? In Leanvec, the PCA matrix is generated from the dataset and used for both data and query conversions. If the queries have a different distribution than the dataset, the conversions will not work correctly.

build_gist.py
import os
import sys
import svs
import numpy as np
import time

base_dir='/export/data/gist/'
vecs_file = f'{base_dir}/gist_base.fvecs'
X_query = svs.read_vecs(f'{base_dir}/gist_queries.fvecs')
idx_gt = svs.read_vecs(f'{base_dir}/gist_gtruth.ivecs')
dataset= 'gist'

alpha = 1.2
max_degree = 128
max_candidate = 750
window_size = 200
max_threads = 112

parameters = svs.VamanaBuildParameters(
            graph_max_degree = max_degree,
            prune_to = max_degree - 4,
            window_size = window_size,
            max_candidate_pool_size = max_candidate,
            alpha = alpha,
)

data_loader = svs.VectorDataLoader(
        vecs_file,
        svs.DataType.float32,
        dims = X_query.shape[1]
)

start = time.time()
index = svs.Vamana.build(
        parameters,
        data_loader,
        svs.DistanceType.L2,
        num_threads = max_threads
)
end = time.time();

print(f"Max degree = {max_degree}")
print(f"Build time = {end-start}")

index.search_window_size = 20
search_times = []
for i in range (0, 5):
    start = time.time()
    idx, _ = index.search(X_query, 10)
    end = time.time();
    search_times.append(end-start)
print("qps: {}".format(X_query.shape[0]/min(search_times)))

idx, _ = index.search(X_query, 10)
rec10at10 = svs.k_recall_at(idx_gt, idx, 10, 10)
print(f'rec10at10 = {rec10at10}')

index_dir = f"{base_dir}/{dataset}_M{max_degree}_FP32/"

if not os.path.exists(index_dir):
    os.makedirs(index_dir)

index.save(
        os.path.join(index_dir, "config"),
        os.path.join(index_dir, "graph"),
        os.path.join(index_dir, "data"),
)

search_gist.py
import os
import sys
import svs
import numpy as np
import time
import statistics

base_dir='/export/data/ishwarsi/gist/'

X_query = svs.read_vecs(f'{base_dir}/gist_queries.fvecs')
idx_gt = svs.read_vecs(f'{base_dir}/gist_gtruth.ivecs')
dataset= 'gist'
max_degree = 128

nq = X_query.shape[0]

batch_size = int(sys.argv[1])
count = 10
max_threads = 112

data_type=sys.argv[2]

num_threads = min(max_threads, batch_size)

print(f"Batch size = {batch_size}")
print(f"Number of Threads = {num_threads}")

vecs_file = f'{base_dir}/gist_base.fvecs'
data_loader = svs.VectorDataLoader(
        vecs_file,
        svs.DataType.float32,
)
loader = data_loader

if data_type == 'fp32':
    print(f"Using FP32")
elif data_type == 'fp16':
    print(f"Using FP16")
    vecs_file = f'{base_dir}/gist_base_fp16.fvecs'
    loader = svs.VectorDataLoader(
            vecs_file,
            svs.DataType.float16,
    )
elif data_type == 'lvq4x8':
    B1 = 4    # Number of bits for the first level LVQ quantization
    B2 = 8    # Number of bits for the residuals quantization
    padding = 32
    strategy = svs.LVQStrategy.Turbo
    #strategy = svs.LVQStrategy.Sequential

    compressed_loader = svs.LVQLoader(
            data_loader,
            primary=B1,
            residual=B2,
            strategy=strategy, # Passing the strategy is optional.
            padding=padding # Passing padding is optional.
    )
    print(f"Using LVQ{B1}x{B2}")
    loader = compressed_loader

elif data_type == 'lvq8':
    B1 = 8    # Number of bits for the first level LVQ quantization
    B2 = 0    # Number of bits for the residuals quantization
    padding = 32
    strategy = svs.LVQStrategy.Sequential

    compressed_loader = svs.LVQLoader(
            data_loader,
            primary=B1,
            residual=B2,
            strategy=strategy, # Passing the strategy is optional.
            padding=padding # Passing padding is optional.
    )
    print(f"Using LVQ{B1}x{B2}")
    loader = compressed_loader

elif data_type == 'leanvec':
    primary = svs.LeanVecKind.lvq8
    secondary = svs.LeanVecKind.lvq8
    leanvec_dims = 160

    compressed_loader = svs.LeanVecLoader(
            data_loader,
            leanvec_dims = leanvec_dims,
            primary_kind = primary,
            secondary_kind = secondary,
            alignment = 32
    )
    print(f"Using LeanVec ({primary}, {secondary}) - {leanvec_dims}")
    loader = compressed_loader
else:
    raise BaseException(f'Unknown data_type={data_type}')

#index_dir = f"{base_dir}/"
index_dir = f"{base_dir}/{dataset}_M{max_degree}_FP32/"
index = svs.Vamana(
        f"{index_dir}/config",
        f"{index_dir}/graph",
        loader,
        query_type = svs.DataType.float32,
        distance = svs.DistanceType.L2,
        num_threads = num_threads,
        enforce_dims = True,
)
print(index.experimental_backend_string)

search_windows = [32, 64, 128]
num_batches = int(np.ceil(nq/float(batch_size)))
print(f"Number of queries = {nq}, number of batches = {num_batches}")
for sw in search_windows:

    index.search_window_size = sw

    qps = []

    for i in range (0, 5):
        total_time = float(0)
        idx = np.empty((0, count), int)

        for bb in range(num_batches):
            init_batch = bb*batch_size
            end_batch = min(init_batch + batch_size, nq)

            start = time.time()
            res, _ = index.search(X_query[init_batch:end_batch], count)
            end = time.time();
            total_time += float(end - start)
            idx = np.append(idx, res, axis=0)

        qps.append(nq/total_time)

    rec = svs.k_recall_at(idx_gt, idx, count, count)
    print(f'search_window = {sw}, rec{count}at{count} = {rec} qps(Max) = {max(qps)}')
    #print("qps: Avg {} Max {} StdDev {}".format(statistics.mean(qps), max(qps), statistics.stdev(qps)))
    print(f" Raw QPS values: {qps}")

Thank you @ibhati for providing the codes. I was able to see the performance trend with them for both. I checked that the query set also normalized. However, the previous results that I show did not use the same vamana graph built under fp32. Instead, compression of LVQ8, LVQ4x8 or Leanvec was applied to the index building as well.
def init_loader(args):
    if args.mode == "vamana":
        loader = svs.VectorDataLoader(args.train_path, svs.DataType.float32)
    elif args.mode == "leanvec":
        validate_leanvec_settings(args.D, args.leanvec_dr)
        loader = svs.LeanVecLoader(
            svs.VectorDataLoader(args.train_path, svs.DataType.float32),
            args.D // args.leanvec_dr,
            primary_kind=svs.LeanVecKind.lvq8,
            secondary_kind=svs.LeanVecKind.float16,
        )
    elif args.mode in ["lvq8", "lvq4x8"]:
        B1, B2 = parse_lvq_mode(args.mode)
        loader = svs.LVQLoader(
            svs.VectorDataLoader(args.train_path, svs.DataType.float32),
            primary=B1,
            residual=B2,
            padding=32,
        )
    else:
        raise ValueError(f"Unsupported mode: {args.mode}")
    return loader

...
        # build index
        build_params = svs.VamanaBuildParameters(
            alpha=0.95,
            graph_max_degree=args.R,
            window_size=args.window_size,
            max_candidate_pool_size=1024,
            prune_to=60,
        )

        build_start = time.time()
        index = svs.Vamana.build(
            build_params, loader, svs.DistanceType.MIP, num_threads=args.num_threads
        )
        build_end = time.time()
        print(f"Build time = {build_end - build_start} seconds")
        # create the path
        os.makedirs(os.path.join(build_index_path(args)), exist_ok=True)
        index.save(
            os.path.join(build_index_path(args), "config"),
            os.path.join(build_index_path(args), "graph"),
            os.path.join(build_index_path(args), "data"),
        )
...

Interestingly, I changed the scripts (provided in my response above) slightly to build with fp32, lvq8, and leaves. Subsequently, I used those indices for respective searches, and the performance/recall is very similar to the one built with fp32. Build time is shorter for lvq8 and leanvec. (I used L2 distance, but the results should be similar for MIP). I am including the updated scripts below. Thanks!
build_gist.py
import os
import sys
import svs
import numpy as np
import time

base_dir='/export/data/gist/'
vecs_file = f'{base_dir}/gist_base.fvecs'
X_query = svs.read_vecs(f'{base_dir}/gist_queries.fvecs')
idx_gt = svs.read_vecs(f'{base_dir}/gist_gtruth.ivecs')
dataset= 'gist'

alpha = 1.2
max_degree = 128
max_candidate = 750
window_size = 200
max_threads = 112

data_type=sys.argv[1]

data_loader = svs.VectorDataLoader(
        vecs_file,
        svs.DataType.float32,
)
loader = data_loader

if data_type == 'fp32':
    print(f"Using FP32")
elif data_type == 'fp16':
    print(f"Using FP16")
    vecs_file = f'{base_dir}/gist_base_fp16.fvecs'
    loader = svs.VectorDataLoader(
            vecs_file,
            svs.DataType.float16,
    )
elif data_type == 'lvq4x8':
    B1 = 4    # Number of bits for the first level LVQ quantization
    B2 = 8    # Number of bits for the residuals quantization
    padding = 32
    strategy = svs.LVQStrategy.Turbo
    #strategy = svs.LVQStrategy.Sequential

    compressed_loader = svs.LVQLoader(
            data_loader,
            primary=B1,
            residual=B2,
            strategy=strategy, # Passing the strategy is optional.
            padding=padding # Passing padding is optional.
    )
    print(f"Using LVQ{B1}x{B2}")
    loader = compressed_loader

elif data_type == 'lvq8':
    B1 = 8    # Number of bits for the first level LVQ quantization
    B2 = 0    # Number of bits for the residuals quantization
    padding = 32
    strategy = svs.LVQStrategy.Sequential

    compressed_loader = svs.LVQLoader(
            data_loader,
            primary=B1,
            residual=B2,
            strategy=strategy, # Passing the strategy is optional.
            padding=padding # Passing padding is optional.
    )
    print(f"Using LVQ{B1}x{B2}")
    loader = compressed_loader

elif data_type == 'leanvec':
    primary = svs.LeanVecKind.lvq8
    secondary = svs.LeanVecKind.lvq8
    leanvec_dims = 160

    compressed_loader = svs.LeanVecLoader(
            data_loader,
            leanvec_dims = leanvec_dims,
            primary_kind = primary,
            secondary_kind = secondary,
            alignment = 32
    )
    print(f"Using LeanVec ({primary}, {secondary}) - {leanvec_dims}")
    loader = compressed_loader
else:
    raise BaseException(f'Unknown data_type={data_type}')

parameters = svs.VamanaBuildParameters(
            graph_max_degree = max_degree,
            prune_to = max_degree - 4,
            window_size = window_size,
            max_candidate_pool_size = max_candidate,
            alpha = alpha,
)

start = time.time()
index = svs.Vamana.build(
        parameters,
        loader,
        svs.DistanceType.L2,
        num_threads = max_threads
)
end = time.time();

print(f"Max degree = {max_degree}")
print(f"Build time = {end-start}")

index.search_window_size = 20
search_times = []
for i in range (0, 5):
    start = time.time()
    idx, _ = index.search(X_query, 10)
    end = time.time();
    search_times.append(end-start)
print("qps: {}".format(X_query.shape[0]/min(search_times)))

idx, _ = index.search(X_query, 10)
rec10at10 = svs.k_recall_at(idx_gt, idx, 10, 10)
print(f'rec10at10 = {rec10at10}')

index_dir = f"{base_dir}/{dataset}_M{max_degree}_{data_type}/"

if not os.path.exists(index_dir):
    os.makedirs(index_dir)

index.save(
        os.path.join(index_dir, "config"),
        os.path.join(index_dir, "graph"),
        os.path.join(index_dir, "data"),
)

search_gist.py
import os
import sys
import svs
import numpy as np
import time
import statistics

base_dir='/export/data/gist/'

X_query = svs.read_vecs(f'{base_dir}/gist_queries.fvecs')
idx_gt = svs.read_vecs(f'{base_dir}/gist_gtruth.ivecs')
dataset= 'gist'
max_degree = 128

nq = X_query.shape[0]

batch_size = int(sys.argv[1])
count = 10
max_threads = 112

data_type=sys.argv[2]

num_threads = min(max_threads, batch_size)

print(f"Batch size = {batch_size}")
print(f"Number of Threads = {num_threads}")

vecs_file = f'{base_dir}/gist_base.fvecs'
data_loader = svs.VectorDataLoader(
        vecs_file,
        svs.DataType.float32,
)
loader = data_loader

if data_type == 'fp32':
    print(f"Using FP32")
elif data_type == 'fp16':
    print(f"Using FP16")
    vecs_file = f'{base_dir}/gist_base_fp16.fvecs'
    loader = svs.VectorDataLoader(
            vecs_file,
            svs.DataType.float16,
    )
elif data_type == 'lvq4x8':
    B1 = 4    # Number of bits for the first level LVQ quantization
    B2 = 8    # Number of bits for the residuals quantization
    padding = 32
    strategy = svs.LVQStrategy.Turbo
    #strategy = svs.LVQStrategy.Sequential

    compressed_loader = svs.LVQLoader(
            data_loader,
            primary=B1,
            residual=B2,
            strategy=strategy, # Passing the strategy is optional.
            padding=padding # Passing padding is optional.
    )
    print(f"Using LVQ{B1}x{B2}")
    loader = compressed_loader

elif data_type == 'lvq8':
    B1 = 8    # Number of bits for the first level LVQ quantization
    B2 = 0    # Number of bits for the residuals quantization
    padding = 32
    strategy = svs.LVQStrategy.Sequential

    compressed_loader = svs.LVQLoader(
            data_loader,
            primary=B1,
            residual=B2,
            strategy=strategy, # Passing the strategy is optional.
            padding=padding # Passing padding is optional.
    )
    print(f"Using LVQ{B1}x{B2}")
    loader = compressed_loader

elif data_type == 'leanvec':
    primary = svs.LeanVecKind.lvq8
    secondary = svs.LeanVecKind.lvq8
    leanvec_dims = 160

    compressed_loader = svs.LeanVecLoader(
            data_loader,
            leanvec_dims = leanvec_dims,
            primary_kind = primary,
            secondary_kind = secondary,
            alignment = 32
    )
    print(f"Using LeanVec ({primary}, {secondary}) - {leanvec_dims}")
    loader = compressed_loader
else:
    raise BaseException(f'Unknown data_type={data_type}')

#index_dir = f"{base_dir}/"
index_dir = f"{base_dir}/{dataset}_M{max_degree}_{data_type}/"
index = svs.Vamana(
        f"{index_dir}/config",
        f"{index_dir}/graph",
        loader,
        query_type = svs.DataType.float32,
        distance = svs.DistanceType.L2,
        num_threads = num_threads,
        enforce_dims = True,
)
print(index.experimental_backend_string)

search_windows = [32, 64, 128]
num_batches = int(np.ceil(nq/float(batch_size)))
print(f"Number of queries = {nq}, number of batches = {num_batches}")
for sw in search_windows:

    index.search_window_size = sw

    qps = []

    for i in range (0, 5):
        total_time = float(0)
        idx = np.empty((0, count), int)

        for bb in range(num_batches):
            init_batch = bb*batch_size
            end_batch = min(init_batch + batch_size, nq)

            start = time.time()
            res, _ = index.search(X_query[init_batch:end_batch], count)
            end = time.time();
            total_time += float(end - start)
            idx = np.append(idx, res, axis=0)

        qps.append(nq/total_time)

    rec = svs.k_recall_at(idx_gt, idx, count, count)
    print(f'search_window = {sw}, rec{count}at{count} = {rec} qps(Max) = {max(qps)}')
    #print("qps: Avg {} Max {} StdDev {}".format(statistics.mean(qps), max(qps), statistics.stdev(qps)))
    print(f" Raw QPS values: {qps}")

@hugy718 let me know if you have any further questions on this topic. Thanks!

Thank you for the careful explanations. I still find that building with Inner product with Leanvec yields low recalls. But other than that, I can replicate the performance trends.

For the inner product, we need to regenerate the groundtruths again with MIP. Just normalizing the dataset/queries will not work because, after the leanvec transformation, data is not guaranteed to stay normalized.

Great to hear that you could replicate the performance trends. Regarding the low recall that you observe when building with inner product with LeanVec, one thing to try is to increase the LeanVec dimensionality, as 240 may not be enough in this case. In your results the 240 dimensionality seems to work well for L2 with the original GIST dataset, but this won’t necessary be the case for the normalized dataset using MIP.


Fix tests for JSON parse errors in node 6

The error message has changed from Unexpected end of input to Unexpected end of JSON input, so let's just test that it's a SyntaxError instead.

:+1:


Fixes 'count-to' module not found error.

Module is declared as 'countTo' but loaded as 'count-to'. Should be
var myApp = angular.module('myApp', ['countTo']);

Pulled from elsewhere


Office365Connector action card issue :On submitting information if appplication run slow, it show error message "There was a problem submitting your changes. Try again in a minute."

Bot Info

SDK Platform:
SDK Version:
Active Channels:
Deployment Environment:

Issue Description

Code Example

Reproduction Steps
Run application in debug mode.

Launc offcie365Connector Action Card.
Fill required data and Click Submit button
There should debug point before submit operation .Wait debugger there up to 2 minute

Expected Behavior
On click of Submit  data should save or in case of error user can handle the error.

## Actual Results
When application run slow /in debug mode
A pop up error message is showing
"There was a problem submitting your changes. Try again in a minute."
Even submitted data is getting save .
I want either I can hide this pop up error OR submitted data do not save after having that issue.

Please provide more information such as the code you're using, the SDK version and the SDK Language.

`private static Attachment CreateSampleO365ConnectorCard()
{
#region Multichoice Card
var multichoiceCard = new O365ConnectorCardActionCard(
O365ConnectorCardActionCard.Type,
"Multiple Choice",
"Multiple Choice Card",
new List
{
new O365ConnectorCardMultichoiceInput(
O365ConnectorCardMultichoiceInput.Type,
"CardsType",
true,
"Pick multiple options",
null,
new List
{
new O365ConnectorCardMultichoiceInputChoice("Hero Card", "Hero Card"),
new O365ConnectorCardMultichoiceInputChoice("Thumbnail Card", "Thumbnail Card"),
new O365ConnectorCardMultichoiceInputChoice("O365 Connector Card", "O365 Connector Card")
},
"expanded",
true),
new O365ConnectorCardMultichoiceInput(
O365ConnectorCardMultichoiceInput.Type,
"Teams",
true,
"Pick multiple options",
null,
new List
{
new O365ConnectorCardMultichoiceInputChoice("Bot", "Bot"),
new O365ConnectorCardMultichoiceInputChoice("Tab", "Tab"),
new O365ConnectorCardMultichoiceInputChoice("Connector", "Connector"),
new O365ConnectorCardMultichoiceInputChoice("Compose Extension", "Compose Extension")
},
"compact",
true),
new O365ConnectorCardMultichoiceInput(
O365ConnectorCardMultichoiceInput.Type,
"Apps",
false,
"Pick an App",
null,
new List
{
new O365ConnectorCardMultichoiceInputChoice("VSTS", "VSTS"),
new O365ConnectorCardMultichoiceInputChoice("Wiki", "Wiki"),
new O365ConnectorCardMultichoiceInputChoice("Github", "Github")
},
"expanded",
false),
new O365ConnectorCardMultichoiceInput(
O365ConnectorCardMultichoiceInput.Type,
"OfficeProduct",
false,
"Pick an Office Product",
null,
new List
{
new O365ConnectorCardMultichoiceInputChoice("Outlook", "Outlook"),
new O365ConnectorCardMultichoiceInputChoice("MS Teams", "MS Teams"),
new O365ConnectorCardMultichoiceInputChoice("Skype", "Skype")
},
"compact",
false)
},
        new List<O365ConnectorCardActionBase>
              {
               new O365ConnectorCardHttpPOST(
                    O365ConnectorCardHttpPOST.Type,
                    "Send",
                    "multichoice",
                    @"{""CardsType"":""{{CardsType.value}}"", ""Teams"":""{{Teams.value}}"", ""Apps"":""{{Apps.value}}"", ""OfficeProduct"":""{{OfficeProduct.value}}""}")
             });

        #endregion

        #region Input Card
        var inputCard = new O365ConnectorCardActionCard(
            O365ConnectorCardActionCard.Type,
            "Text Input",
            "Input Card",
            new List<O365ConnectorCardInputBase>
            {
                new O365ConnectorCardTextInput(
                    O365ConnectorCardTextInput.Type,
                    "text-1",
                    false,
                    "multiline, no maxLength",
                    null,
                    true,
                    null),
                new O365ConnectorCardTextInput(
                    O365ConnectorCardTextInput.Type,
                    "text-2",
                    false,
                    "single line, no maxLength",
                    null,
                    false,
                    null),
                new O365ConnectorCardTextInput(
                    O365ConnectorCardTextInput.Type,
                    "text-3",
                    true,
                    "multiline, max len = 10, isRequired",
                    null,
                    true,
                    10),
                new O365ConnectorCardTextInput(
                    O365ConnectorCardTextInput.Type,
                    "text-4",
                    true,
                    "single line, max len = 10, isRequired",
                    null,
                    false,
                    10)
            },
            new List<O365ConnectorCardActionBase>
            {
                new O365ConnectorCardHttpPOST(
                    O365ConnectorCardHttpPOST.Type,
                    "Send",
                    "inputText",
                    @"{""text1"":""{{text-1.value}}"", ""text2"":""{{text-2.value}}"", ""text3"":""{{text-3.value}}"", ""text4"":""{{text-4.value}}""}")
            });
        #endregion

        #region Date Card
        var dateCard = new O365ConnectorCardActionCard(
            O365ConnectorCardActionCard.Type,
            "Date Input",
            "Date Card",
            new List<O365ConnectorCardInputBase>
            {
                new O365ConnectorCardDateInput(
                    O365ConnectorCardDateInput.Type,
                    "date-1",
                    true,
                    "date with time",
                    null,
                    true),
                new O365ConnectorCardDateInput(
                    O365ConnectorCardDateInput.Type,
                    "date-2",
                    false,
                    "date only",
                    null,
                    false)
            },
            new List<O365ConnectorCardActionBase>
            {
                new O365ConnectorCardHttpPOST(
                    O365ConnectorCardHttpPOST.Type,
                    "Send",
                    "dateInput",
                    @"{""date1"":""{{date-1.value}}"", ""date2"":""{{date-2.value}}""}")
            });
        #endregion

        var section = new O365ConnectorCardSection(
            "**section title**",
            "section text",
            "activity title",
            "activity subtitle",
            "activity text",
            "http://connectorsdemo.azurewebsites.net/images/MSC12_Oscar_002.jpg",
            true,
            new List<O365ConnectorCardFact>
            {
                new O365ConnectorCardFact("Fact name 1", "Fact value 1"),
                new O365ConnectorCardFact("Fact name 2", "Fact value 2"),
            },
            new List<O365ConnectorCardImage>
            {
                new O365ConnectorCardImage
                {
                    Image = "http://connectorsdemo.azurewebsites.net/images/MicrosoftSurface_024_Cafe_OH-06315_VS_R1c.jpg",
                    Title = "image 1"
                },
                new O365ConnectorCardImage
                {
                    Image = "http://connectorsdemo.azurewebsites.net/images/WIN12_Scene_01.jpg",
                    Title = "image 2"
                },
                new O365ConnectorCardImage
                {
                    Image = "http://connectorsdemo.azurewebsites.net/images/WIN12_Anthony_02.jpg",
                    Title = "image 3"
                }
            });

        O365ConnectorCard card = new O365ConnectorCard()
        {
            Summary = "O365 card summary",
            ThemeColor = "#E67A9E",
            Title = "card title",
            Text = "card text",
            Sections = new List<O365ConnectorCardSection> { section },
            PotentialAction = new List<O365ConnectorCardActionBase>
             {
                multichoiceCard,
                inputCard,
                dateCard,
                new O365ConnectorCardViewAction(
                    O365ConnectorCardViewAction.Type,
                    "View Action",
                    null,
                    new List<string>
                    {
                        "http://microsoft.com"
                    })

              }
        };

        return card.ToAttachment();
    }

Use above card in application . After implementing above code  .Put debug point in application. Run application using ngrok in teams..Perform above card Send button action . and wait at debug point up to 2 minute .An error message in team comes intermittently "There was a problem submitting your changes. Try again in a minute."

How to call API using O365ConnectorCardHttpPOST ..Is there any card type can be used for  the below potential actions
"potentialAction": [
{
"@type": "ActionCard",
"name": "Approve",
"inputs": [
{
"@type": "TextInput",
"id": "comment",
"isMultiline": true,
"title": "Reason (optional)"
}
],
"actions": [
{
"@type": "HttpPOST",
"name": "Submit",
"target": "https://{Site Address}/api/Approv?ID={{requestid}}",
"body": "={{comment.value}}",
"headers": [
{
"Content-Type": "application/x-www-form-urlencoded"
}
]
}
]
}

@satheeshganji48 I am unclear what you are asking.  Is your question related to the "Send" button timeout that @poonam0025 was referring to?  This sounds like a separate issue altogether.

@nodeomega  How do you extend the timeout?

@nodeomega it seems Async is a solution, it works for me
var o365CardActionAsyncHandler = async function(event, query, callback) {};
connector.onO365ConnectorCardAction(async o365CardActionAsyncHandler);

@nguyendn is your issue related to the send button timing out as well, and secondly that looks like NodeJS, correct?

@nodeomega yes. it related to the send button timing and it is NodeJS.

OK. @poonam0025  and @nguyendn ,   It would appear that there is no way to adjust that 10 second timeout for Invoke actions (the send button).  It would appear that this behavior is by design.
What I am told is that there are a few ways to workaround it however:

Ensure that your bot finishes processing the operation before the 10 second mark
Respond with HTTP 202 immediately but handle processing of the action in the background.

Also, if using IIS, be advised that the operation should complete before the threads time out, as IIS will do that after an extended amount of time.
As there is little more that can be done in terms of the Bot Framework itself in Teams regarding this issue, I will be closing this.  Thank you all for your patience regarding this, and I know this wasn't exactly the answer we were hoping for.


Add keybindings.json example for OS X

Thank you for this extension!
I hope this small contribution will be useful to other VSCode users on a Mac.

Well if we're gonna have this kind of help, then definitely for all of the people. But alt? Isn't it explicitly cmd for mac shortcuts? I mean I personally after getting used to xcode's cmd+arrow use alt as the modifier in windows but does that get translated to mac layout?

Yes, it does! 😃
BTW, these are the default key sequences regarding the ones in the PR:
// …
{ "key": "alt+right",             "command": "cursorWordEndRight",
                                     "when": "editorTextFocus" },
{ "key": "shift+alt+right",       "command": "cursorWordEndRightSelect",
                                     "when": "editorTextFocus" },
{ "key": "alt+left",              "command": "cursorWordStartLeft",
                                     "when": "editorTextFocus" },
{ "key": "shift+alt+left",        "command": "cursorWordStartLeftSelect",
                                     "when": "editorTextFocus" },
// …
{ "key": "alt+backspace",         "command": "deleteWordLeft",
                                     "when": "editorTextFocus && !editorReadonly" },
{ "key": "alt+delete",            "command": "deleteWordRight",
                                     "when": "editorTextFocus && !editorReadonly" },
// …

Cmd+Left and Cmd+Right are analogues, respectively, to Home and End on a PC.
Thank you!


Add clarification about mailbox moves within EXO

Docs Build status updates of commit 85a677d:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

exchange/exchange-ps/exchange/New-MoveRequest.md
:white_check_mark:Succeeded
View (exchange-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit af8af4b:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

exchange/exchange-ps/exchange/New-MoveRequest.md
:white_check_mark:Succeeded
View (exchange-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Use support.js in the playground demo so that we avoid vulcanization with bundle.js

This change is

Reviewed 2 of 2 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


Question: How do i use this program to build a simple Swap program?

I wanted to make a program that cpi's this one to make a simple swap

Also where do i contact y'all to ask noob questions like this one?

Here is an example of a DEX built on top of the AOB https://github.com/Bonfida/dex-v4
For these types of questions https://forum.bonfida.org/ is more adapted than Github issues.


Error on using az ad app update

I tried your example for updating the app roles and received the following error
Property  value cannot be deleted or updated unless it is disabled first.
I failed to find the az cli command that would allow me to disable the app role to allow me to perform this app role update successfully.
Your assistance in addressing this pre-requisite would be greatly appreciated because this is the update I need to perform on my environment.
[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 921aa920-c25f-5227-aabb-49116da38fa7
Version Independent ID: 9b5a53ad-edf8-4d89-f8e6-cde01b7866f6
Content: az ad app
Content Source: latest/docs-ref-autogen/ad/app.yml
Service: active-directory
GitHub Login: @rloutlaw
Microsoft Alias: routlaw

az ad

This is still an issue. To recap:
az ad app update --id $app.appId --app-roles @rolesManifest.json`
Fails if roles exist: ERROR: Permission (scope or role) cannot be deleted or updated unless disabled first.
I tried setting isEnabled: false, in the rolesManifest.json file, but no luck.

I've run into this as well. You need to set the flag in a separate transaction prior to deleting or updating the role/scope.

You need to set the flag in a separate transaction prior to deleting or updating the role/scope.

Cool, so there’s is actually a workaround that doesn’t involve manual fix in Azure portal? Can you please elaborate a bit, cause I can’t get it to work.

interesting enough, my earlier solution leveraged Powershell AZ not AZ CLI. When I used the example code for the command against one of my app registrations that contained an app role, I had no issues. I did have to reference the "manifest.json" file without using the '@' character. My command was
az ad app update --id "{myAppIdGuid}" --app-roles manifest.json

Yes thanks, but that's what I'm doing it and I get the error Permission (scope or role) cannot be deleted or updated unless disabled first.

You need to set the flag in a separate transaction prior to deleting

I'm looking for a (script) way to disable them so I can remove/update then.  I'm trying different permutations of the roles.json file to no avail.

@MartinWickman, thank you for sharing the solution. I am sure it will help other users.
As for the behavior of AD Graph API, you may contact AAD support by creating a support ticket at https://azure.microsoft.com/en-us/support/create-ticket/. The "Removing roles requires the existing role to be 'disabled'" behavior is not related to Azure CLI command itself.

Eventually, I figured it out. My powershell skillz are horrible, but it works.
az ad app update --id $appId --app-roles<EMAIL_ADDRESS>
if (!$?) {
    # Note: Roles are quirky. Updates and adds work fine if the role contains the "id" in the json file.
    # Removing roles requires the existing role to be "disabled". We try to handle that here.

    Write-Output "* Roles failed: trying to fix by deleting all roles and creating again..."
    # Fetch existing roles
    $roles = (az ad app show --id $appId | ConvertFrom-Json ).appRoles
    $roles | ForEach-Object { $_.isEnabled = $false }
    $roles = ConvertTo-Json $roles
    $roles | Out-File "disabled_roles.json"
    # Disable them
    az ad app update --id $appId --app-roles<EMAIL_ADDRESS>    # Add all roles. It should replace all existing roles
    az ad app update --id $appId --app-roles<EMAIL_ADDRESS>    Write-Output "- Roles replaced"
}
else {
    Write-Output "- Roles created"
}

It has a side effect, users lose their role assignment.
I am testing updating the ids in roles.json for the existing roles and using  az rest --method PATCH --uri "https://graph.microsoft.com/v1.0/applications/$($app.objectId)" --headers 'Content-Type=application/json' --body<EMAIL_ADDRESS>


Feature request - shorter wait time

Some game streams only last for 40 minutes to an hour and offer drops for watching for 30 minutes (Warframe is one example - they do it every week with two featured streamers after the weekly dev stream). Sometimes the stream may be longer, say, 1.5 hours, and requires watching for 40 minutes. In both cases mining often starts too late to achieve the goal and get the drop. Would it be possible to reduce the waiting time between checks from 1 hour to something shorter? Or possibly make it a custom setting?
Appreciate this wonderful program a lot, and this would make it perfect - thank you so much for making it in the first place!
EDIT: I checked the behavior in the version built from the latest source - it still persists. Warframeinternational channel is a good example - it should give one full drop per week and progress towards the next one, but it only gives some progress for the first one instead (the system is one drop for 40 minutes between that channel and official Warframe one)

Hello o/
The thing is, the 1 hour refresh cycle is quite arbitrary - campaign start and end times are tracked in real time. That means the miner will automatically reload itself when the campaign starts already, and you can verify that by monitoring closely what it does right at the campaign start time. I'd like to not lower this 1 hour to anything shorter, because whole inventory reloads are expensive network-wise and I don't want to "hammer" Twitch with requests any more often than that. Everything needed to start and track a campaign at a specific time is already implemented and should be working - if it doesn't, that's what should be investigated, not the refresh cycle time being changed.
That being said, to tell what's happening, please provide the output from the miner with the CALL trace enabled, by passing in -vvv at startup. The period it should include is at least 5 minutes before and after an affected campaign starts.

if there's a slight discrepancy (e.g. the campaign becomes active a few seconds or a minute after the stream starts), the miner would wait for an hour to retry looking for a stream with available drops?

Not at all. Notice how the campaign lists it's ACL:

This means those two channels are the only ones able to progress the campaign, and thus they should be added via previous reload to the tracked channels list up to an hour before the campaign even starts. At the start time, the entire inventory and tracked channels list is reloaded yet again, just to make sure nothing's missed, and then everything should work as expected.
In an event the channel is available already when the reload happens, the last step of the reload (channel select) should automatically start watching and progressing it. If the channel isn't immediately available, it's still added as a tracked channel (because of the ACL), and when it goes online, it'll automatically start being watched via the "channel goes online" event. Both of these situations assume that no other user condition stops it from happening, such as selecting a different channel due to the priority list, exclusion list, priority only setting, etc.
There is one case that I am aware of, where the above logic can fail, but it can only happen for campaigns without an ACL, so it doesn't apply here.

According to the log, a channel cleanup happened exactly at 4 AM. The cleanup should've checked the online status of all channels on the list, and subscribed to their status updates. The websocket connection would then be responsible for receiving those updates within the next hour, and if one of the channels would come online, the websocket would then generate an event that would cause a channel switch to happen and start watching as expected.
But that never happened, until a reload doing another channel cleanup happened at 04:59:38 - this isn't even related to inventory being updated, but to websocket never receiving the "channel goes ONLINE" event. And that is/could be possible, because the current websocket system has no confirmations setup for the topics it subscribes to. The topic subscription message is sent, but the feedback message telling the miner it got received properly, is never handled beyond simply ignoring the returning feedback message: https://github.com/DevilXD/TwitchDropsMiner/blob/a19ecebd7b624569441b3ef7c0c13aaf8753b124/websocket.py#L290-L292
During websocket implementation, there was an incentive on my part to create a system where this confirmation would be handled properly, but I was having trouble synchronizing the whole thing, and the websocket was simply getting stuck and preventing the program from, well, working at all. So my solution was to simply postpone this until "later" when a need for this would arise (like it does now), and until then, simply send a request for subscription and "assume" it got through properly.
This thing has a related issue of #110, so when #110 is closed, this one should be closed as well, and vice versa. I'm leaving it open for anyone that possibly ends up with the same issue, so they can find the exact cause.


Dask.distributed setup documentation

I'm considering a rewrite of the dask.distributed cluster setup documentation.  I'm also considering moving that documentation to the main Dask page for visibility (I'd like to slowly move the less detail-oriented documentation over to the main site).
Any concerns or thoughts?

I believe this was fixed in #3014. Ok to close?

Thanks for the reminder


Download retry

HTTP requests should time out and be retried on timeouts or HTTP errors (503 etc.) - DNS retries are out of scope, similar to how curl is used already: https://github.com/flatcar/Flatcar/issues/1028#issuecomment-1727741906
Not a blocker, because we could run a bash loop around it but it's not nice because we don't know if it's a download or verification error, and would wastefully retry on verification errors then

Until then, to distinguish download from verification errors we can also do the following: Download into a  .download folder and only if finished, move to the .unverified folder. This way the caller can check that downloading worked by looking if the file exists in the .unverified folder.


pktdisp: detect dispatcher disconnect, and export error metrics instead of logging

Currently go/lib/pktdisp will produce an infinte amount of logging if the connection to the dispatcher dies. It will also happily log so many errors that it is unable to read from the dispatcher quickly enough, causing the dispatcher to kill the socket. These 2 factors combined is worse than the sum of their parts.

Related to #2181

Partially addressed by #2207

Error part moved to https://github.com/Anapaya/scion/issues/1127.
Reconnecting logic in the rest of the SIG is covered by https://github.com/scionproto/scion/issues/1896.
Closing this since we can use the finer grained issues instead.


Verify Hibernation allowed from Control plane before setting swap file

Currently the hibernation-setup tool would run even if Hibernation was not enabled for the VM during creation time.
This PR verifies the hibernation state from IMDS and only proceeds with setup if enabled from CRP.

Looks good to me.


[Data] : Error inserting closure: UNIQUE constraint failed: gui_closures.funding_txid, gui_closures.funding_index

Hello,
I noticed my data on the LNDg dashboard seemed a bit stale, so I checked the logs and saw this:
Sat Jan 13 00:16:52 2024 : [Data] : Starting data execution...
Sat Jan 13 00:16:55 2024 : [Data] : Updating forwards, have 156690
Sat Jan 13 00:16:55 2024 : [Data] : Added 0 new forwards to DB...
Sat Jan 13 00:16:57 2024 : [Data] : Error inserting closure: UNIQUE constraint failed: gui_closures.funding_txid, gui_closures.funding_index
Sat Jan 13 00:16:57 2024 : [Data] : Data execution completed...sleeping for 20 seconds
Sat Jan 13 00:17:00 2024 : [Rebalancer] : Queue manager is starting...
Sat Jan 13 00:17:00 2024 : [Rebalancer] : Queue currently has 0 items...
Sat Jan 13 00:17:00 2024 : [Rebalancer] : There are currently 0 tasks in progress...
Sat Jan 13 00:17:00 2024 : [Rebalancer] : Queue manager is checking for more work...
Sat Jan 13 00:17:00 2024 : [Rebalancer] : Queue is still empty, stoping the rebalancer...
Sat Jan 13 00:17:00 2024 : [Rebalancer] : Queue manager has shut down...
Sat Jan 13 00:17:03 2024 : [Rebalancer] : Manager and workers have stopped...
Sat Jan 13 00:17:03 2024 : [Rebalancer] : Rebalancer successfully exited...sleeping for 20 seconds
Sat Jan 13 00:17:17 2024 : [Data] : Starting data execution...
Sat Jan 13 00:17:19 2024 : [Data] : Updating forwards, have 156690
Sat Jan 13 00:17:19 2024 : [Data] : Added 0 new forwards to DB...
Sat Jan 13 00:17:21 2024 : [Data] : Error inserting closure: UNIQUE constraint failed: gui_closures.funding_txid, gui_closures.funding_index
Sat Jan 13 00:17:21 2024 : [Data] : Data execution completed...sleeping for 20 seconds

Is there any reason I'm getting this UNIQUE constraint failed? Any way to resolve it?
Running v1.8.0
Thanks!

In this case, the duplicates should be removed and lndg will try to reinsert the row on the next run but other data fields should still be able to process their data without issue.

How can I remove the duplicates?

This happens in the code already after the error, it tries to resolve the issue by deleting the duplicate record.
https://github.com/cryptosharks131/lndg/blob/master/jobs.py#L450-L451
You could also check lncli closedchannels to see if the source list has any duplicates.
If you would like to clear the closures table completely and let lndg rebuild the list, you can go to this page and clear it manually: /reset


Pagination

Is there any plan for pagination support ?

That would be nice, i really need to paginate my posts from prismic....

Me too!!

Does anyone have an idea on how to implement it, I'm ready to work on it, I just need some inspiration.
@vortex6688 @thangngoc89 @mbanting

Hi all I apologize for not being around much. Unfortunately our current projects at work have moved off Prismic so I haven't been keeping up to date. I'd be happy to accept pull requests and even open this project up to more contributors. I'll free up some time over the next couple of weeks to see what can be done here with some of the issues coming in.

@mbanting I'm wondering whats the reason you left Prismic, did you find something better or its was just business decision?
Thanks for sharing!

@david-genger sorry I misrepresented myself. We still have a couple of metalsmith projects using Prismic. It's been very stable and working quite well. We have a support team now managing those sites. The other projects have grown beyond static sites, so we moved off Metalsmith altogether.

Got it, Thanks!

This is an old thread, but still at the top of search results. Pagination options for Metalsmith collections that should work with Prismic...

https://github.com/AlansCodeLog/metalsmith-organizer
https://github.com/blakeembrey/metalsmith-pagination
https://github.com/RobinThrift/metalsmith-paginate


Question about Data augmentation for MoCo.

Dear author, I feel thankful of your great masterpiece, and I really appreciate about your work these days.
Reading your paper with comprehending your code,
I got in my mind about data augmentation.
I found that you give data augmentation on Image, then, did you do same thing on text modalities?
If it is right, then can you check out that line in code?
Thx.

Hi, thanks for your interest in our paper.
No, there is not data augmentation for text input, but we follow simCSE and
consider standard dropout as minimal data augmentation for the text.

Thx. your reply is helpful. Hope you have good days :)


Use consistent scalatest/scalatest versions

For starters scalaTestScalaCheckVersion as 1-14 isn't meant to run with scalaCheckVersion 1.15.1 and scalaTest 3.2.x series does support Scala 2.12

Then after this change, will not be able to run tests against 2.12.*？

Then after this change, will not be able to run tests against 2.12.*？

Scala 2.12 is still fully supported, all of these artifacts versions I picked are also published for 2.12

@mdedetrich Will it compile and test with scala 2.12.x?

@mdedetrich Will it compile and test with scala 2.12.x?

Yes

Just to be clear, nothing whatsoever has changed regarding Scala 2.12 support in any way (or any other Scala version for that matter). All that the PR does is pick specific versions of ScalaTest and its various integrations that happen to be cross compiled for all scala versions we support (i.e. 2.12/2.13/3) rather than having bespoke versions depending on the Scala version.


Update Alpine image to 3.18.5 and reduce installed packages

Removed dbus in favor of using the already provided machine-id OpenRC service. Remove shadow and bash, as the test suite passes without them. Remove the systemd specific signal.

See https://github.com/k0sproject/bootloose/pull/73#issuecomment-1862750250.


Get Started - Create Success Screen

Fixed by https://github.com/liferay-labs-br/liferay-grow/issues/22


Update SNSEvent.Message Coding Keys

Update SNSEvent.Message coding keys to match go models & documentation
Motivation:
When trying to decode an SNSEvent as the lambda's event (using runtime 1.0.0-alpha.1), I am getting:
2023-05-18T14:51:32+0000 warning Lambda : lifecycleIteration=16 lambda handler returned an error: requestDecoding(Swift.DecodingError.keyNotFound(CodingKeys(stringValue: "SigningCertURL", intValue: nil), Swift.DecodingError.Context(codingPath: [CodingKeys(stringValue: "Records", intValue: nil), _JSONKey(stringValue: "Index 0", intValue: 0), CodingKeys(stringValue: "Sns", intValue: nil)], debugDescription: "No value associated with key CodingKeys(stringValue: \"SigningCertURL\", intValue: nil) (\"SigningCertURL\").", underlyingError: nil)))

It seems that the model's coding keys have changed with the proper keys being SigningCertUrl and UnsubscribeUrl  (as opposed to SigningCertURL and UnsubscribeURL).
https://github.com/aws/aws-lambda-go/blob/771b391678d3f54bfa38531774d656f5e0f2ab58/events/sns.go#L28
https://github.com/aws/aws-lambda-go/blob/771b391678d3f54bfa38531774d656f5e0f2ab58/events/sns.go#L30
https://docs.aws.amazon.com/lambda/latest/dg/with-sns.html
Modifications:
Update the coding key raw values for SNSEvent.Message.signingCertURL and SNSEvent.Message.unsubscribeURL
Result:
Lamda handler can now successfully decode SNSEvent in then handle(_:context:) method.

thanks @hiimtmac


Does the plugin work for GraphQL data?

Does it work only in REST API?
Or it works both in REST API and GraphQL?

Hello,
Currently this plugin does not work with GraphQL, but I am looking into adding support for it.
I have managed to handle the response part and am currently investigating on how to modify the entity schemas to remove the attributes keys so no mismatch between the request and response.


ghostscript: fix icc profile validation

Currently, printing color documents fails with the following error:
Oct 05 08:34:21 tp cupsd[16855]: | ./base/gsicc_manage.c:1863: gsicc_verify_device_profiles(): Mismatch of ICC profiles and device color model
Oct 05 08:34:21 tp cupsd[16855]: | ./base/gsicc_manage.c:1990: gsicc_set_device_profile(): Error in device profiles
Oct 05 08:34:21 tp cupsd[16855]: Unrecoverable error: rangecheck in .putdeviceprops

Add the corresponding patch from ghostscript master to fix this.
Motivation for this change
Printing color documents didn't work on my printer (HP Officejet 4620). With this patch, it works again.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Fits CONTRIBUTING.md.

I didn't check whether unstable is affected, too as switching there took to long for me (but guess it is).
As there are some scary fixes on master anyhow, I assume we should bump to 9.26 as soon as it's released on both channels anyhow, but I didn't want to introduce a printing regression in 18.09

cc @vcunat @samueldr

Success on aarch64-linux (full log)
Attempted: ghostscript
Partial log (click to expand)
shrinking RPATHs of ELF executables and libraries in /nix/store/mfw6kyiryxyc2khjxgf78l38vrll9bkj-ghostscript-9.24-man
gzipping man pages under /nix/store/mfw6kyiryxyc2khjxgf78l38vrll9bkj-ghostscript-9.24-man/share/man/
strip is /nix/store/y4ymnvgxygpq05h03kyzbj572zmh6zla-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/mfw6kyiryxyc2khjxgf78l38vrll9bkj-ghostscript-9.24-man
checking for references to /build in /nix/store/mfw6kyiryxyc2khjxgf78l38vrll9bkj-ghostscript-9.24-man...
shrinking RPATHs of ELF executables and libraries in /nix/store/c2136mql4l417wdvfrlgg7cc7437f6mc-ghostscript-9.24-doc
strip is /nix/store/y4ymnvgxygpq05h03kyzbj572zmh6zla-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/c2136mql4l417wdvfrlgg7cc7437f6mc-ghostscript-9.24-doc
checking for references to /build in /nix/store/c2136mql4l417wdvfrlgg7cc7437f6mc-ghostscript-9.24-doc...
/nix/store/yh5afax8jsidcz2354n8rz880z76n8kg-ghostscript-9.24

Success on x86_64-linux (full log)
Attempted: ghostscript
Partial log (click to expand)
shrinking RPATHs of ELF executables and libraries in /nix/store/kaghvzzq67gh8vc9y49l13x53lxwqr03-ghostscript-9.24-man
gzipping man pages under /nix/store/kaghvzzq67gh8vc9y49l13x53lxwqr03-ghostscript-9.24-man/share/man/
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/kaghvzzq67gh8vc9y49l13x53lxwqr03-ghostscript-9.24-man
checking for references to /build in /nix/store/kaghvzzq67gh8vc9y49l13x53lxwqr03-ghostscript-9.24-man...
shrinking RPATHs of ELF executables and libraries in /nix/store/815bzw5b6zrwlhiwvayj9zwwhmrhqh1h-ghostscript-9.24-doc
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/815bzw5b6zrwlhiwvayj9zwwhmrhqh1h-ghostscript-9.24-doc
checking for references to /build in /nix/store/815bzw5b6zrwlhiwvayj9zwwhmrhqh1h-ghostscript-9.24-doc...
/nix/store/900ldasg3lbxg754py2phmay2s3mpqyi-ghostscript-9.24

Success on x86_64-darwin (full log)
Attempted: ghostscript
Partial log (click to expand)
/nix/store/38a415gm0x2h5pl2i0l4x483p7rdyvcg-ghostscript-9.24/bin/ps2pdfwr: interpreter directive changed from "/bin/sh" to "/nix/store/cblfnvb5rmhd2z231mqasn0brzh1hhv4-bash-4.4-p23/bin/sh"
/nix/store/38a415gm0x2h5pl2i0l4x483p7rdyvcg-ghostscript-9.24/bin/ps2ps: interpreter directive changed from "/bin/sh" to "/nix/store/cblfnvb5rmhd2z231mqasn0brzh1hhv4-bash-4.4-p23/bin/sh"
/nix/store/38a415gm0x2h5pl2i0l4x483p7rdyvcg-ghostscript-9.24/bin/ps2ps2: interpreter directive changed from "/bin/sh" to "/nix/store/cblfnvb5rmhd2z231mqasn0brzh1hhv4-bash-4.4-p23/bin/sh"
/nix/store/38a415gm0x2h5pl2i0l4x483p7rdyvcg-ghostscript-9.24/bin/unix-lpr.sh: interpreter directive changed from "/bin/sh" to "/nix/store/cblfnvb5rmhd2z231mqasn0brzh1hhv4-bash-4.4-p23/bin/sh"
gzipping man pages under /nix/store/772rs1sw0s3n2b4855yq4hkdw1bpcm3h-ghostscript-9.24-man/share/man/
strip is /nix/store/df6k4mgdjxciy0f637lryp7c9ln7n1m3-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/772rs1sw0s3n2b4855yq4hkdw1bpcm3h-ghostscript-9.24-man
strip is /nix/store/df6k4mgdjxciy0f637lryp7c9ln7n1m3-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/hph177sqkx42c6vddnr0pcyfi68jvaqn-ghostscript-9.24-doc
/nix/store/38a415gm0x2h5pl2i0l4x483p7rdyvcg-ghostscript-9.24

Success on x86_64-darwin (full log)
Attempted: ghostscript
Partial log (click to expand)
/nix/store/38a415gm0x2h5pl2i0l4x483p7rdyvcg-ghostscript-9.24/bin/unix-lpr.sh: interpreter directive changed from "/bin/sh" to "/nix/store/cblfnvb5rmhd2z231mqasn0brzh1hhv4-bash-4.4-p23/bin/sh"
/nix/store/38a415gm0x2h5pl2i0l4x483p7rdyvcg-ghostscript-9.24/bin/pdf2ps: interpreter directive changed from "/bin/sh" to "/nix/store/cblfnvb5rmhd2z231mqasn0brzh1hhv4-bash-4.4-p23/bin/sh"
/nix/store/38a415gm0x2h5pl2i0l4x483p7rdyvcg-ghostscript-9.24/bin/eps2eps: interpreter directive changed from "/bin/sh" to "/nix/store/cblfnvb5rmhd2z231mqasn0brzh1hhv4-bash-4.4-p23/bin/sh"
/nix/store/38a415gm0x2h5pl2i0l4x483p7rdyvcg-ghostscript-9.24/bin/lprsetup.sh: interpreter directive changed from "/bin/sh" to "/nix/store/cblfnvb5rmhd2z231mqasn0brzh1hhv4-bash-4.4-p23/bin/sh"
gzipping man pages under /nix/store/772rs1sw0s3n2b4855yq4hkdw1bpcm3h-ghostscript-9.24-man/share/man/
strip is /nix/store/df6k4mgdjxciy0f637lryp7c9ln7n1m3-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/772rs1sw0s3n2b4855yq4hkdw1bpcm3h-ghostscript-9.24-man
strip is /nix/store/df6k4mgdjxciy0f637lryp7c9ln7n1m3-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/hph177sqkx42c6vddnr0pcyfi68jvaqn-ghostscript-9.24-doc
/nix/store/38a415gm0x2h5pl2i0l4x483p7rdyvcg-ghostscript-9.24

Success on x86_64-linux (full log)
Attempted: ghostscript
Partial log (click to expand)
shrinking RPATHs of ELF executables and libraries in /nix/store/kaghvzzq67gh8vc9y49l13x53lxwqr03-ghostscript-9.24-man
gzipping man pages under /nix/store/kaghvzzq67gh8vc9y49l13x53lxwqr03-ghostscript-9.24-man/share/man/
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/kaghvzzq67gh8vc9y49l13x53lxwqr03-ghostscript-9.24-man
checking for references to /build in /nix/store/kaghvzzq67gh8vc9y49l13x53lxwqr03-ghostscript-9.24-man...
shrinking RPATHs of ELF executables and libraries in /nix/store/815bzw5b6zrwlhiwvayj9zwwhmrhqh1h-ghostscript-9.24-doc
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/815bzw5b6zrwlhiwvayj9zwwhmrhqh1h-ghostscript-9.24-doc
checking for references to /build in /nix/store/815bzw5b6zrwlhiwvayj9zwwhmrhqh1h-ghostscript-9.24-doc...
/nix/store/900ldasg3lbxg754py2phmay2s3mpqyi-ghostscript-9.24

duplicate of #47903.


WMS with valid XML can't be imported

@viddo (or @saleiva please let us know if this goes elsewhere, thanks)
Please ref S/B 6651888
Expected:
User would be able to use this WMS as a basemap:
XML file url:
http://sit.cittametropolitana.na.it/arcgis/services/poi/MapServer/WMSServer?request=GetCapabilities&service=WMS
WMS being added:
http://sit.provincia.napoli.it/arcgis/services/poi/MapServer/WMSServer
Happening:
"Data download timed out." error
Is this an error on our side or from where user's getting the url?

"Data download timed out." error

Where does this error appear/happen? Can you provide more detailed steps-to-reproduce?

WMS being added:
http://sit.provincia.napoli.it/arcgis/services/poi/MapServer/WMSServer

This URL doesn't work, testing with curl yields curl: (6) Could not resolve host: sit.provincia.napoli.it.
According to our WMS proxy the XML file is not a valid capabilities doc (returns "error": "not a valid capabilities document"), I'd need @CartoDB/systems to know the reason since I don't have access to the logs

I think there's a confusion here.
WMS is not a supported format for importing data to CartoDB. It can be used to add WMS layers as a custom basemap, but it cannot be used as a URL to import.

User would be able to use this WMS as a basemap:

@iriberri :point_up:, but @ztephm can probably clarify if the user was trying to import or adding a basemap

According to the support thread, it's been always an import error. Adding WMS basemaps does not give any error ID neither :-)

We have studied this issue again just focusing in the WMS basemap approach, and the Capabilities of the WMS are announcing 3 possible projections:
CRS="CRS:84"
CRS="EPSG:4326"
CRS="EPSG:102113"  info here
CRS:84 is equivalent to EPSG:4326 but changing the axis from Lat-Long to Long-Lat.

if can help same thing happens with this WMS:
http://sit.regione.campania.it:80/arcgis/services/cartografia_base/MapServer/WMSServer
with this error code:
9c6fd585-ec56-4903-b595-9c13751e1d75
i've tested this WMS in Qgis and arcgis and works fine i can see the layers.

She tried importing so that's why I sent the error. We're clarifying with her that it needs to be used via 'Change basemap' but in the meantimej it also fails that way. @dani checked it out & XML looked ok?

if i import as basemap all is fine for this WMS:
http://sit.regione.campania.it:80/arcgis/services/cartografia_base/MapServer/WMSServer
i can see and import layers as basemap.
I hoped that i could import also datas.

@nicoali, unfortunately WMS is not of our supported formats, as we are focused on importing vector data. You can check more details about this here: http://docs.cartodb.com/cartodb-editor.html#supported-file-formats
As the WMS seems to be working, I'm closing the issue. Please, don't hesitate to ask us any other doubt or question you might have in our support channel or directly opening a new issue in GitHub if you think it's due to a technical issue.


image from TGA RGBA -> BC7 getting a weird transformation

I honestly have no idea what is going on. I am using the latest windows binary with commands:
./cuttlefish.exe -i INPUT.tga -j -m -f BC7 -Q highest -v -o OUTPUT.dds

Here are the files if you want to take a look.
bin.zip

I believe this is a bug with the viewer you're using. I tried both RenderDoc and Visual Studio and both view OUTPUT.dds correctly. For example, see this screenshot from RenderDoc:

I installed IrfanView based on your screenshot and was able to see the same error you did. However, given that two other tools display the .dds correctly, I'm certain that the bug here lies with IrfanView.

Taking a quick look I'm not sure if there's a corrupt size. However, the width (4121) isn't a multiple of the BC7 block size (4x4), so most likely both IrfanView and the engine aren't handling this properly. The texture data will always be a multiple of the block size, but any extra pixels are discarded when displaying. For example, if IrfanView decodes 4124 pixels each line and displays as 4121 pixels (without discarding the remainder) that would explain the skew. The game engine may similarly be attempting to incorrectly round the size.
Either way these are bugs with the IrfanView and the engine, but the easiest way to avoid them is to make sure your image size is a multiple of 4.


Polish models in raw/execsum.py

These being part of the raw models, I feel like the goal should be as much consistency as possible with the source data, even at the expense of readability and Python or Django code style conventions.
For each model:

[x] Make sure the class name matches the table name
[x] Make sure the model class subclasses BaseRMPModel
[x] Set the verbose_name (and verbose_name_plural, if necessary) model meta option to table name in the tlkpRS_TableNames.csv included in the source data.
[x] Set our custom source_file attribute
[x] For each field:

[x] Set the help_text keyword argument to the field name in the tlkpRS_FieldNames.csv included in the source data.

[x] Same for verbose_name

[x] Set the model field class to the proper CopyFrom... subclass.
[x] Make sure the field name matches the source file's column name, or pass our custom source_column kwarg, if necessary.
[x] Pass whatever other arguments necessary for the file to be imported without any errors.

Renamed this module as tbl.py to match the prefix of the .csv files.
I thought about dropping these prefixes from the model class names, but then some of them clashed with processed model names (e.g., we already have a Facility processed model). Could be fixed if we decide to split the raw and processed models into separate Django apps in the project. But it isn't super necessary.


[BUG] Unable to add media to gallery when "Show more" has been clicked

Describe the bug
For notes that are large enough to produce a "Show more" icon, when the "Show more" is clicked it is not possible to add media to gallery.
To Reproduce
Steps to reproduce the behavior:

Go to:  nostr:nevent1qqsrec6u7pxf7a80f8h374m2xmnlr5up3edw5k2jh0w4fk70xfcer2qppemhxue69uhkummn9ekx7mp0qgsv4zwtz8cuwh2mvc3zdrl5853g365t9j6mn25edlul7uz0eyzt0zcrqsqqqqqp0ghja3
Click "Show more"
Click on any image in the post
Only option available is "Copy URL"

Expected behavior
"Add media to Gallery" to be present. Note: If you click an image before clicking "Show more" you are able to add it to the gallery
Video and Screenshots

Device (please complete the following information):

Phone Brand/Model Pixel 7 Pro
Android Version 34
App Version 89.7

@believethehype

Interessting, Copy node id to clipboard is also missing in this Case.

You're right. I tested it and the save to gallery is missing.
I also realized from your post that I have been downloading images instead of saving it to my gallery.
The download button says saved to your gallery which I thought meant saved to my Nostr profile gallery
I guess they mean saved to my phone gallery.


chore(deps): update acorn

What kind of change does this PR introduce?
fix #13664
Did you add tests for your changes?
No need
Does this PR introduce a breaking change?
No
What needs to be documented once your changes are merged?
No

Thanks


Add Abdoulaye NDOYE to the contributors list

Added my name and the link of my github account to the contributors list

Félicitation @NDOY3M4N ton premier PR 🙌🏾🎉


Document use of fixture scripts and adding makefile entry

This aims to resolve points 2 and 3 on issue #31.
Type of change

[ ] Bug fix (non-breaking change that fixes an issue)
[x] New feature (non-breaking change that adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality not to work as expected)
[ ] This change requires a documentation update
[x] This is a documentation update

Action Checklist

[x] Work on a single issue/concept (if there are multiple separate issues to address, please use a separate pull request for each)
[ ] Fork the pyani-plus repository under your own account (please allow write access for repository maintainers)
[x] Set up an appropriate development environment (please see CONTRIBUTING.md for this repository)
[x] Create a new branch with a short, descriptive name
[x] Work on this branch

[x] style guidelines have been followed (please see CONTRIBUTING.md for this repository)
[x] new code is commented, especially in hard-to-understand areas
[x] corresponding changes to documentation have been made
[ ] tests for the change have been added that demonstrate the fix or feature works

[x] Test locally with pytest-plus -v non-passing code will not be merged
[x] Rebase against origin/master
[x] Check changes with linting/formatting tools before submission (please see CONTRIBUTING.md for this repository)
[x] Commit branch
[x] Submit pull request, describing the content and intent of the pull request
[x] Request a code review
[x] Continue the discussion at Pull requests section in the pyan-plus repository

Rebasing locally (not sure why GitHub thinks it cannot be rebased safely)...

Rebasing locally (not sure why GitHub thinks it cannot be rebased safely)...

Looking at the changes, it all seems quite benign and unlikely to cause any conflicts.


Infinite modal from vscode.dev

open incognito mode and go to https://insiders.vscode.dev/github/microsoft/vscode
Try to close out of the modals.
They appear infinitely :bug:

You can't use Remote Repos without signing into GitHub

That makes sense to me, but I was hoping that there was a message of some sort there instead of this infinite loop? not sure if this has been considered before.


Fix some typo

Signed-off-by: xianlubird<EMAIL_ADDRESS>

@thomastaylor312

Merging since this is just a doc typo fix.


Allow using Azure for embedding with the JS client + fix docker-compose for running tests

Description of changes
Add the possibility to use Azure for embeddings with the JS client (same ability as the python client)
Summarize the changes made by this PR.

New functionality

allow using Azure for embedding with the JS client (just like the Python client)

Small fix in docker-compose file for test : set the chromadb port with env var.

Test plan

[x] built and ran tests locally.

Documentation Changes
Docs were updated (very similar to the existing python docs)

Our underlying impl has changed and so this PR is not landable as is.
That being said - we'd still like to add this functionality and that is now tracked in this issue.


Merge release/2.2 branch to master

Includes aspnet/FileSystem's master branch.

This means, by the way, that https://github.com/aspnet/FileSystem/pull/334 and https://github.com/aspnet/FileSystem/pull/335 would have to be closed and moved to this repo. Are you okay with that?

Ok, Artak closed those PRs. Merging this PR now.


k tutorial fun_untyped has parsing ambiguity

it is (both env based and substitution based) having difficulties parsing int + int, such as 1+2
[Warning] Inner Parser: Parsing ambiguity. Arbitrarily choosing the first.
1: Exp ::= Exp "+" Exp
#token("#Int", "1")(.KList) + #token("#Int", "2")(.KList)
2: Exp ::= Exp Exp
#token("#Int", "1")(.KList) #token("#Int", "2")(.KList)
program get stuck when we have 1+2+3

That's a nice one. So the problem is the language has Exp Exp and Exp + Exp, but also positive integers. So the ambiguity report is correct if you write 1+2. But if you write 1+ 2 (with space) then, this parses non ambiguously.
I'm assigning @grosu, since he is the owner of this language.


Question

it is possible to use native interfaces, native google maps plugin and URLImage.createToStorage?

Native interfaces: Yes, may require a bit of tweaking when using external libs, and remember it is not a Silverlight project.
Native google maps plugin: I didn't test it, I believe it is not supported but the openstreetmap fallback should work.
URLImage: works.


executionObservables.switchLatest() doesn't work after an error produce

        action = Action<Int, Int>.init {
            if $0 % 2 == 0 {
                return .just(0)
            } else {
                return Observable<Int>.error(NSError(domain: "", code: 0, userInfo: nil))
            }
        }

        _ = action.executionObservables.switchLatest().subscribe(onNext: { _ in
            print("next")
        })
        _ = action.errors.subscribe(onNext: { _ in
            print("error")
        })

        _ = action.execute(0) // print next
        _ = action.execute(0) // print next
        _ = action.execute(1) // print error
        _ = action.execute(0) // do nothing

The executionObservables.switchLatest() doesn't work any more after the action produce a error.
is this the design to be or a bug?

@scubers Neither. In fact, executionObservables did not stop working. It's switchLatest observable subscription that got disposed (because it got terminating (error) event from executionObservable).


Added Multiple Notifications

#182
Added Multiple Notifications in toast_notification.js. It will send random quotes.
Before

After

merged


SPARQL extension: Property value configuration

All property values, which are entities (people, places, and things as opposed to numbers and dates ) will be created (or reused if existing) as page references and their matching Wikidata item URI will be inserted as value of same as:: attribute, the same way as the property.

This results in the creation of a lot of pages and blocks that have little or no utility in the graph. It would be nice to be able to specify that certain attributes are not brought into the graph or are just brought in as data without creating a new page (f a page exists, go ahead and make a reference).

Use options:
Create page reference for Wikidata
[ ] Properties
[ ] Values
Currently, it is both, which could remain the default, but one or both can be switched off to avoid bulking the graph.
So, I'd suggest, configuring the default in the setting and having these two as options in the pop-up.

This could also be another pop-up (or wizard page) after the extension gets the data saying "We found these attributes, what do you want to do with them?" and for each one, [ ] Link to page [ ] No link [ ] Discard. Reading defaults from the configuration. This would actually be quite helpful as there's currently no way of knowing what data is being added to your graph until after the extension creates all of the blocks and pages (~8 blocks for each attribute value).

Audit: Won't do - Could revisit if interest resurfaces


[FEATURE] Create support page

Detailed description
Suggested idea on https://trello.com/c/NDxD6XZg/70-documentation-for-a-bug-report
Context
It can help Dev Protocol followers to report their concern
Possible implementation
Idea and it can be copy design: https://tailwindui.com/components/marketing/sections/header#component-547517d4dc48e1bf0ccab051ebb0dcba
Additional information
if you have suggestions, feel free to comment below
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Is there a mockup for this?

Is there a mockup for this? What will be the contents? And what will be the new url subpath (https://devprotocol.xyz/support)?

will try to do it on figma, I'll ping you once it's done

Can't edit well. I think u can continue this, let me handle the content, you can use the content there

Hi @vinzvinci can you please share where the contact links are to point to make it easier for contributors?

We used 3 email @krshkodes for communications
For general issue<EMAIL_ADDRESS>For security issue<EMAIL_ADDRESS>For community related issue<EMAIL_ADDRESS>


Handle unexpected interview_metadata

Ran into an error when interview_metadata["main_interview_key"} was set to a dictionary. Definitely not expected use for us, but it shouldn't error out if that's the case.
This PR adds a check to make sure the interview key is a string like we expect it to be. If it's not, we fallback to the other methods (just checking the only entry in interview_metadata like we do in the next check works in this case).

That's fair. Even if we're deprecating this variable, I think it'll be a while before this code can be removed.

Am I missing the reason it was a dictionary in the interview you found in the wild?

It was in a MLH interview was just finishing up development. Given it was a simpler fix than #777 (which also makes me think there could be other metadata block issues), I figured it could be merged sooner, but we don't have to fix it.


Running helloworld-minimal-sample says "Cannot find module 'vscode'"

>>> node extension.js
  throw err;
  ^

Error: Cannot find module 'vscode'
Require stack:
- /Users/kkrithika/Documents/Projects/Salesforce/research/vscode-extension-samples/helloworld-minimal-sample/extension.js
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:956:15)
    at Function.Module._load (node:internal/modules/cjs/loader:804:27)
    at Module.require (node:internal/modules/cjs/loader:1028:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/Users/kkrithika/Documents/Projects/Salesforce/research/vscode-extension-samples/helloworld-minimal-sample/extension.js:3:16)
    at Module._compile (node:internal/modules/cjs/loader:1126:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1180:10)
    at Module.load (node:internal/modules/cjs/loader:1004:32)
    at Function.Module._load (node:internal/modules/cjs/loader:839:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/kkrithika/Documents/Projects/Salesforce/research/vscode-extension-samples/helloworld-minimal-sample/extension.js'
  ]
}

>>> node --version
v16.17.1

Please ask your question on StackOverflow. We have a great community over there. They have already answered thousands of questions and are happy to answer yours as well. See also our issue reporting guidelines.
Happy Coding!


Update widgets.py

Problem
Example for widgets usage for export doesn't work.
Solution
Usage has to instantiate class, and correct namespace is widgets, not widget.
Acceptance Criteria
Only updated documentation

Coverage remained the same at 96.74% when pulling a5d78e8ad9c988a33beec784975cbfe27d646ce8 on renatoch:patch-1 into cd06c59e1c11b3e6d0b62b525e83e36bc31db7a7 on django-import-export:master.


Cross button (top rightIcon) in the input component not working.

Clicking on the cross button (rightIcon) in the input section must delete the contents of the input section, but it does nothing.
I tried to fix it, but I am not able to do so. Any help?

could I get a chance to work on it? I think I can fix it

could I get a chance to work on it? I think I can fix it

Yeah sure! Go for it.

thanks for giving me chance.

I am getting this error and I am unable to resolve it. Please help me resolve this error

Well, The module you are tryna import is missing....

hey @dikshantsharma12 you will find your ans here https://github.com/react-native-elements/playground/issues/3#issuecomment-797278137

thanks, I got the solution

thanks, I got the solution

Hey @dikshantsharma12  did you fix the problem?

I had some dependencies issue so, my project was not starting so I had updated some dev dependencies but after that, now all the icons are not loading and I tried so much to fix them but still, they are not loading.

Hey buddy, do you want me to work on this issue and help you out?
On Fri, Mar 26, 2021 at 10:05 AM Dikshant sharma @.***>
wrote:

I had some dependencies issue so, my project was not starting so I had
updated some dev dependencies but after that, now all the icons are not
loading and I tried so much to fix them but still, they are not loading.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/react-native-elements/playground/issues/70#issuecomment-807928948,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQH4KHRU6TL3OO6TORMX3LLTFQFKZANCNFSM4ZGBZZMQ
.

@Uyadav207 @ARJUN-SABU @ankushdutt @pranshuchittora please look into PR #104
Fix
https://user-images.githubusercontent.com/68142366/112758858-07d0d300-900e-11eb-8e21-84c62887bb0f.mov


Better way to include javascript

Hello,
I'm still playing with the module and I noticed that the javascript is included inside the emptyForm_forms_flash div.
I think you should find a better way to execute some JS when form is submitted (custom JS function, form reset, etc....) for several reasons :

It's not a good practice to include javascript in the middle of the DOM.
If you want to theme your form and don't want to include that empty div "emptyForm_forms_flash" you lost some functionnalities, which should never happens.

So I think that the javascript addition should be done in a separate way and the emptyForm_forms_flash div should only be used as a wrapper for the alert message.
Thanks

Hello @Alex360hd
You're right, it's not the best option.
For an advanced use, it's possible to use the javascript api of October and run your custom code when ajaxError event are triggered.


nanoboyadvance: init at 1.8.1

Description of changes
Closes https://github.com/NixOS/nixpkgs/issues/301114
@keenanweaver

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 301259 run on x86_64-linux 1

1 package built:

nanoboyadvance


Limit the number of transactions in flight in benchmark

I tried running this with a factor of 10 and got 52 snapshots with a confirmation avg. of over 4s. I suspect our current benchmark is not the right approach as it does not represent any realistic client pattern, and it keeps the node busy with useless messages. In order to get some useful and reliable metrics from this benchmark we should think more about the transactions' dependencies.
We are currently generating a sequence of transactions which is mostly linear, so there's no point in submitting any transaction after a transaction has been rejected because they will probably also be invalid. This means the code of the submitter could be simplified, eg. simply wait for transactions to be confirmed before submitting next one.
Alternatively, we could also explore generating transactions which are not dependent. This can easily be done by splitting the initial random UTXO set and generating sequence of transactions from each fragment. This way we could load the server with meaningful work to do, and plot what happens to latency when the interdependency degree of transactions increase.


Accessing the method addCatalog through ParsleyValidator is deprecated.

Getting the following messages with the latest version
parsley.js:93Accessing the method addCatalog through ParsleyValidator is deprecated. Simply call window.Parsley.addCatalog(...)
parsley.remote.js:93 Accessing the method addCatalog through ParsleyValidator is deprecated. Simply call window.Parsley.addCatalog(...)

The bundled locale files need to be updated, in which case you can ignore this message. Or maybe you are calling ParsleyValidator.addCatalog yourself?

I updated the local files from latest rc version, are you sure they are updated on the release branch

Sorry, I wasn't clear, the bundled local files that we provide right now need to be updated by us, sorry. In the meantime, you can ignore the message.

I see can you keep this up and update the issue when they have been update. I will update the parsley.js meteor package, by the way glad to get your support and make my package the meteor package I will maintain it - https://atmospherejs.com/almogdesign/parsley.js