Using multiple queues within OpenMP threads slows down SYCL

Hi,
I'm trying to optimize a piece of code by using multiple GPUs. To achieve this I first enumerate the devices on the host, create one queue for each device and then within a for loop access the "correct" queue by using omp_get_thread_num(). The problem is that using multiple queues slows down the entire computation by almost half. Using the same code with only one GPU (and thus one queue) gives the expected performance.
Since the code is not mine to share I have tried to condense the relevant bits:
const std::vector<sycl::device> devices = enumerate_devices();
std::vector<sycl::queue> queues;
for (auto device : devices) {
  queues.push_back(sycl::queue{device});
}
omp_set_num_threads(queues.size());
#pragma omp parallel for
for (...) {
  auto Q = queues[omp_get_thread_num()];
  // Each iteration will allocate quite a bit of memory
  // then run quite a few kernels
  // Lastly, free device memory again
}

I get the correct results in both instances, but using multiple devices uses the same total amount of time as using a single device, where kernel compute and memory handling takes roughly twice as much time as with a single queue.
I'm wondering if this is a bad approach and if there are any suggestions for using multiple GPUs with hipSYCL (I have looked at the multi device queue, but it seems more difficult than the above with memory handling).

Thanks for the quick reply.
There is probably something I don't understand correctly about SYCL or haven't communicated clearly in the example.
In the outer for loop (included in the example above), I need to submit multiple kernels, but there are also multiple Q.wait() to ensure previous tasks have been completed (i.e. the memory they were touching are ready to read), this is the reason for trying to use OpenMP to utilize multiple devices.

Ok. Is the purpose of Q.wait() to ensure that data is ready to be read by other kernels, or by the host after a copy from device to host?
Are your queues a SYCL in-order queue or a (default) out-of-order queue?

Yeah, the reason for this strange question is that I have multiple nested for loops where I didn't want to return the events to wait for. So I made it as async as possible, but the outer loop still required some Q.wait() calls to ensure proper ordering (as some of the kernels could be submitted async while others have data dependencies.
I think a re-write is probably better and then I can see.
Just to be clear, your recommendation for using multiple queues is to skip the OpenMP stuff in my original example, keep everything asynchronous inside the "outer" loop and use one device per iteration (something like: auto Q = queues[i % queues.size()];)?

Just to be clear, your recommendation for using multiple queues is to skip the OpenMP stuff in my original example, keep everything asynchronous inside the "outer" loop and use one device per iteration (something like: auto Q = queues[i % queues.size()];)?

Yes, it's probably better to start from a simple setup :-)
Additionally:

First of all, try to avoid waiting on the host in the first place, if you can. It's never efficient. If there are dependencies between kernels, express them using handler::depends_on(event) or use an in-order queue. This will allow dependencies to be expressed on the level of the GPU backend, independently from the host. If you use the buffer-accessor model, dependencies are set up automatically between kernels.

Use OMP_PROC_BIND if OpenMP is involved

Great, thanks for the quick response and guidance!

(Sorry for reviving this, but I think this follow up is relevant)
So I was able to rewrite the code and in the process sped it up by a considerable amount. It is still slower than the manually optimized CUDA version, but it is now probably close enough, when using a single GPU. However, I still want to utilize multiple GPUs and I'm not quite sure if my code can use multiple queues as discussed above.
My new pseudo-code:
const std::vector<sycl::device> devices = enumerate_devices();
std::vector<sycl::queue> queues;
for (auto device : devices) {
  queues.push_back(sycl::queue{device});
}
for (...) {
  auto Q = queues[index % queues.size()];
  // Create buffers to access input on device
  sycl::buffer<float, 1> buf_a{...};
  ...
  // Run the main computation
  Q.submit([&](sycl::handler& h) {
     sycl::accessor acc_a{buf_a, h, sycl::write_only, sycl::no_init};
    ...
  });
  // Copy out result
  Q.submit([&](sycl::handler& h) {
     sycl::accessor acc_a{buf_a, h, sycl::read_only};
     h.copy(acc_a, dest);
  });

I managed to convert to only buffers and accessors and with only two submits I think both should order correctly (at least I get the correct output) and internally within the loop everything is asynchronously. However, using multiple GPUs the performance stays identical to a single GPU which leads me to believe that the loop iterations are not run concurrently. If I were to hazard a guess I would think the creation of the buffers within the loop is the culprit (during destruction), but I'm not sure.
I was wondering if my analysis above is correct and whether or not there is anything I can do to utilize the additional GPUs in the current setup? At the moment, moving the buffer creation outside the loop iteration would require quite a bit more GPU memory and would complicate the code a lot which I'm hesitant to do. So I was hoping there is an alternative to using multiple GPUs.

buffer creation is not cheap (so maybe better to do outside the loop), but also not blocking. The issue most likely is the buffer destructor because buffer destructors may wait for all operations working on it to complete when they die. (Avoids memory being released while kernels are still running, and necessary for potential write-backs in case the buffer was constructed with a host pointer as argument).
So yes, better construct buffers outside the loop to avoid serialization.
You don't have to use buffers though, in the pointer-based USM model you can also use multiple GPUs just fine. Just make sure to use an in-order queue or depends_on to order kernel execution.

Thanks Aksel!
One follow up then, when moving the buffer creation outside they would have to comprise much more data, but I see that sycl::accessors can be given an offset. The question is then, will the runtime only load the limited data or will the whole buffer be moved to the device and just access limited to the offset + range? To follow up on this again, since each loop iteration then would need to share the buffers, but each iteration only access a unique range, will they be able to run concurrently?
And lastly, depending on the answers to the above, what is guaranteed by the standard and what is specific to hipSYCL? E.g. will the code be portable to other SYCL implementations with the same assumptions on concurrent access to the buffers?

Quick update.
I updated my code again and moved the buffer creation outside the loop. Even though the accessor pattern has offsets, meaning each loop iteration should be independent, using multiple queues still doesn't speed up the performance.
(I know this is now way outside the bounds of the original question, which was answered, so feel free to close this)

Hi,
just a quick list of things that might be of interest to you.
By default, using a buffer in a kernel, always assumes the whole buffer may be accessed.
You may however use the hipSYCL-specific buffer page extension:
https://github.com/illuhad/hipSYCL/blob/55bf4e7b5281ff4ec437a094ce6e342f8b36fc7b/doc/extensions.md#hipsycl_ext_buffer_page_size
https://github.com/illuhad/hipSYCL/blob/develop/doc/runtime-spec.md#data-state-tracking-and-pages
It's used like so:
sycl::buffer<int, 1> buff{size, sycl::property::buffer::hipSYCL_page_size<1>{page_size}};
The effect is documented in this presentation (slide 9 onwards): https://www.iwocl.org/wp-content/uploads/23-iwocl-syclcon-2021-alpay-slides.pdf
Alternatively, it might make sense to have a list of preallocated buffers, one for each device?

Thanks for the pointers @fodinabor !
That really helped me better understand how buffers work. The only trouble now is that I tried to add sycl::property::buffer::hipSYCL_page_size<1>{} to my one output buffer (which is the only non sycl::ready_only buffer access) and I got an error:
[hipSYCL Error] from /dev/shm/ebbuild/hipSYCL/0.9.3/GCC-11.2.0-CUDA-11.4.1/hipSYCL-0.9.3/src/runtime/dag_direct_scheduler.cpp:148 @ operator()(): dag_direct_sc
heduler: Could not obtain data update sources when trying to materialize implicit requirement
============== hipSYCL error report ==============
hipSYCL has caught the following undhandled asynchronous errors:

   0. from /dev/shm/ebbuild/hipSYCL/0.9.3/GCC-11.2.0-CUDA-11.4.1/hipSYCL-0.9.3/src/runtime/dag_direct_scheduler.cpp:148 @ operator()(): dag_direct_scheduler: C
ould not obtain data update sources when trying to materialize implicit requirement

One follow up then, when moving the buffer creation outside they would have to comprise much more data

Why? You could just have std::vector<sycl::buffer<...>> outside of the loop and then index into that vector using the current device index, similarly to what you do with queue. buffers would stay the same size.
Buffers are built to manage one unit of information on multiple devices (i.e. automatically transferring data in between devices if the buffer is accessed somewhere else9, but they are not necessarily built to decompose that information.
The access offset and range are a hint to the runtime that not the entirety of the buffer has to be copied in case data transfers are required. It (as far as the SYCL specification is concerned) does not lead to concurrent execution of kernels using disjoint regions.
In principle, this can be achieved using subbuffers, but those are only partially supported in hipSYCL because they require a contiguous memory region which severely limits their usefulness beyond 1D allocations.
As @fodinabor points out, in hipSYCL specifically, the runtime is able to track memory at a user configurable granularity, which also allows kernels to run concurrently.
But as I've said, I don't understand why you need to deal with this additional complexity and don't maintain a list of smaller buffers, as also suggested by @fodinabor.

That really helped me better understand how buffers work. The only trouble now is that I tried to add sycl::property::buffer::hipSYCL_page_size<1>{} to my one output buffer (which is the only non sycl::ready_only buffer access) and I got an error:

AFAIR, there are some limitations with regards to scattering/gathering buffers again. You might have to manually regather the individual chunks living on different devices, e.g. using multiple invocations (one for each device) of the handler::update() extension.

One last question and then we can close this.
I tried to convert to USM buffers as that was easier when it comes to copying the input buffers per loop iterations, but that resulted in an async error which I don't quite understand. However, I now have a simpler version using buffers where the output buffer is unique per queue while the read only input are shared buffers. I would think that this version would be able to support multiple GPUs, but it seems that even though the shared buffers are accessed read only they end up blocking. My question is simply, is it a correct assumption that shared buffers block even though they are accessed read only?
const std::vector<sycl::device> devices = enumerate_devices();
std::vector<sycl::queue> queues;
std::vector<sycl::buffer<float, 1>> out_bufs;
for (auto device : devices) {
  queues.push_back(sycl::queue{device});
  out_bufs.push_back(sycl::buffer{const size});
}
// Create shared buffers
sycl::buffer<float, 1> buf_inp{input_ptr, num_elements};
for (...) {
  auto Q = queues[index % queues.size()];
  auto out_buf = out_bufs[index % queues.size()];
  // Run the main computation
  Q.submit([&](sycl::handler& h) {
     sycl::accessor acc_a{out_buf, h, sycl::write_only, sycl::no_init};
     sycl::accessor acc_inp{buf_inp, h, sycl::read_only};
    ...
  });
  // Copy out result
  Q.submit([&](sycl::handler& h) {
     sycl::accessor acc_a{out_buf, h, sycl::read_only};
     h.copy(acc_a, dest);
  });
}

My question is simply, is it a correct assumption that shared buffers block even though they are accessed read only?

Yes. Even if you only read from an allocation, you don't want it to be freed while there are still kernels operating on it, do you? :-) So even in this case the destructor will block.
There is an extension in hipSYCL that allows to disable this, namely async buffers: https://github.com/illuhad/hipSYCL/blob/develop/doc/explicit-buffer-policies.md

With the USM version (and both in order queue and not) I get the following error:

CUDA error code 1 is CUDA_ERROR_INVALID_VALUE which happens if you feed something into the API which is an incorrect value. Where does inp_a come frome? Is the indexing correct?
You could also just use a USM pointer for the output, and then use handler::copy() or handler::memcpy() to copy the content to some host pointer to obtain the results. It's possible to mix USM and buffer-accessor model, but not necessary, and mixing them might be ultimately confusing.

I'm sorry for taking so much of your time, but I'm really interested in understanding why my code isn't working properly with multiple queues. The code works and calculates the correct result, if I use two devices I can see that work is being submitted to both, however, no kernel executes concurrently with another (verified with Nsight Systems). I have now gotten permission to share the core SYCL code:

#include <SYCL/sycl.hpp>
#include <chrono>
#include <cmath>
#include <iostream>
#include <stdlib.h>
#include <string>
#include <vector>

#include "matched_filter_SYCL.h"

#define STABILITY_THRESHOLD 0.000001f

void println(const std::string &txt) { std::cout << txt << std::endl; }
float elapsed_sec(std::chrono::time_point<std::chrono::steady_clock> t1, std::chrono::time_point<std::chrono::steady_clock> t2) { return std::chrono::duration_cast<std::chrono::microseconds>(t2 - t1).count() / 1.0e6; }
float elapsed_ms(std::chrono::time_point<std::chrono::steady_clock> t1, std::chrono::time_point<std::chrono::steady_clock> t2) { return std::chrono::duration_cast<std::chrono::microseconds>(t2 - t1).count() / 1.0e3; }

// Helper struct which holds a collection of device pointers, which makes it
// easier to store a group of pointers in vectors
struct DevicePtrs {
    float* templates;
    float* sum_sq_templates;
    float* data;
    float* weights;
    int* moveouts;
    float* cc_sum;
    sycl::event prev_evt;
};

std::vector<sycl::device> enumerate_devices(const int n_samples_template,
					    const int n_samples_data,
					    const int n_stations,
					    const int n_components,
					    const int n_corr) {
    std::vector<sycl::device> devices;
    // Estimate roughly the amount of device memory we need and filter out
    // devices with too little memory
    const int network_size = n_components * n_stations;
    const int total_memory = sizeof(float) *
	(network_size * n_samples_template + // Storage for 'templates'
	network_size + // Storage for 'sum_square_templates'
	network_size * n_samples_data + // Storage for 'data'
	network_size + // Storage for 'weights'
	network_size + // Storage for 'moveouts'
	n_corr); // Approximate storage for 'cc_sum'
    println("------------------------------");
    println("Performing device discovery");
    for (auto device : sycl::device::get_devices(sycl::info::device_type::gpu)) {
	const auto dev_mem = device.get_info<sycl::info::device::global_mem_size>();
	const auto dev_name = device.get_info<sycl::info::device::name>();
	if (dev_mem >= total_memory) {
	    println("\tUsing device: " + dev_name);
	    devices.push_back(device);
	} else {
	    std::cerr << "\033[0;33mWARN: " << dev_name <<
		" has too little memory, found " << dev_mem <<
		" require " << total_memory << " bytes \033[0m\n";
	}
    }
    return devices;
}

extern "C" { // needed for C-style symbols in shared object compiled by syclcc
void matched_filter(float *templates,
                    float *sum_square_templates,
                    int *moveouts,
                    float *data,
                    float *weights,
                    int step,
                    int n_samples_template,
                    int n_samples_data,
                    int n_templates,
                    int n_stations,
                    int n_components,
                    int n_corr,
                    float *cc_sum, // output variable
                    int normalize) {
    const auto devices = enumerate_devices(n_samples_template, n_samples_data,
					   n_stations, n_components, n_corr);
    std::vector<sycl::queue> queues;
    if (devices.size() > 0) {
	for (auto device : devices) {
	    queues.push_back(sycl::queue{device});
	}
    } else {
	queues.push_back(sycl::queue{sycl::default_selector{}});
	std::cerr << "\033[0;33mWARN: Did not find any SYCL enabled GPUs, using default device\033[0m\n";
    }
    println("------------------------------");
    println("step               : " + std::to_string(step));
    println("n_samples_template : " + std::to_string(n_samples_template));
    println("n_samples_data     : " + std::to_string(n_samples_data));
    println("n_templates        : " + std::to_string(n_templates));
    println("n_stations         : " + std::to_string(n_stations));
    println("n_components       : " + std::to_string(n_components));
    println("n_corr             : " + std::to_string(n_corr));
    println("------------------------------");

    const auto alloc_start = std::chrono::steady_clock::now();
    // Allocate device memory per queue/device
    const size_t network_size = static_cast<size_t>(n_stations * n_components);
    std::vector<struct DevicePtrs> device_data;
    // Create one output buffer per queue/device so that each loop iteration
    // below can be run independent
    for(auto Q : queues) {
	// Allocate memory per queue/device
	device_data.push_back(
	    {
		sycl::malloc_device<float>(network_size * n_samples_template, Q),
		sycl::malloc_device<float>(network_size, Q),
		sycl::malloc_device<float>(network_size * n_samples_data, Q),
		sycl::malloc_device<float>(network_size, Q),
		sycl::malloc_device<int>(network_size, Q),
		sycl::malloc_device<float>(static_cast<size_t>(n_corr), Q),
	    }
	);
	// 'data' never changes so we upload it right away
	auto queue_mem = device_data[device_data.size() - 1];
	queue_mem.prev_evt = Q.copy<float>(data, queue_mem.data, network_size * n_samples_data);
    }
    auto dt = elapsed_ms(alloc_start, std::chrono::steady_clock::now());
    println("Device allocation : " + std::to_string(dt) + " ms");
    println("------------------------------");

    const auto loop_start = std::chrono::steady_clock::now();
    for (int t = 0; t < n_templates; ++t) {
	const auto queue_index = t % queues.size();
	auto Q = queues[queue_index];
	auto device_mem = device_data[queue_index];

        println("##############################");
	println("Chosen device (" + std::to_string(queue_index) +
	 ")  : " + Q.get_device().get_info<sycl::info::device::name>());
	println("------------------------------");
        const auto t1 = std::chrono::steady_clock::now();
        const int network_offset = t * network_size;
        const int cc_sum_offset = t * n_corr;

        int min_moveout = 0;
        int max_moveout = 0;
        for (int ch = 0; ch < (n_stations * n_components); ++ch) {
            if (moveouts[network_offset + ch] < min_moveout) min_moveout = moveouts[network_offset + ch];
            if (moveouts[network_offset + ch] > max_moveout) max_moveout = moveouts[network_offset + ch];
        }
        const int start_i = static_cast<int>(std::ceil(std::abs(min_moveout)
						       / static_cast<float>(step)) * step);
        const int stop_i = 1 + (n_samples_data - n_samples_template - max_moveout);
	const sycl::range i_range{static_cast<size_t>(stop_i - start_i) / step};
        println("Kernel range : " + std::to_string(i_range.size()));
        println("------------------------------");

	// Copy iteration data to our current queue
	const auto temp_evt = Q.copy<float>(templates + network_offset * n_samples_template,
					    device_mem.templates, network_size * n_samples_template,
					    device_mem.prev_evt);
	const auto sum_sq_evt = Q.copy<float>(sum_square_templates + network_offset,
					      device_mem.sum_sq_templates, network_size,
					      device_mem.prev_evt);
	const auto weight_evt = Q.copy<float>(weights + network_offset,
					      device_mem.weights, network_size,
					      device_mem.prev_evt);
	const auto move_evt = Q.copy<int>(moveouts + network_offset,
					  device_mem.moveouts, network_size,
					  device_mem.prev_evt);
	// First calculate network correlation with all data residing on device
	// this will ensure that all tasks enqueued are run asynchronously
	const auto compute_evt = Q.submit([&](sycl::handler& h) {
	    // Create dependencies on the data movement above
	    h.depends_on({temp_evt, sum_sq_evt, weight_evt, move_evt});

	    h.parallel_for(i_range, [=](auto& idx) {
		const int index = idx[0] * step + start_i;
		float cc_sum = 0.0;
		for (int station = 0; station < n_stations; station++) {
		    for (int comp = 0; comp < n_components; comp++) {
			const int comp_offset = station * n_components + comp;
			const int temp_offset = comp_offset * n_samples_template;
			const int data_offset = comp_offset * n_samples_data + device_mem.moveouts[comp_offset];
			float mean = 0.0;
			float numerator = 0.0;
			float sum_square = 0.0;
			// Check if we need to continue
			if (device_mem.weights[comp_offset] == 0.0) {
			    continue;
			}
			if (normalize) {
			    for (int i = 0; i < n_samples_template; i++) {
				mean += device_mem.data[index + data_offset + i];
			    }
			    mean /= static_cast<float>(n_samples_template);
			}
			for (int i = 0; i < n_samples_template; i++) {
			    const float sample = device_mem.data[index + data_offset + i] - mean;
			    numerator += device_mem.templates[temp_offset + i] * sample;
			    sum_square += sample * sample;
			}
			const float denominator = std::sqrt(
					device_mem.sum_sq_templates[comp_offset] * sum_square);
			if (denominator > STABILITY_THRESHOLD) {
			    cc_sum += (numerator / denominator) * device_mem.weights[comp_offset];
			}
		    }
		}
		device_mem.cc_sum[idx[0]] = cc_sum;
	    });
	});
	// Copy result from device memory to output buffer once the above
	// compute kernel is done
	device_mem.prev_evt = Q.copy(device_mem.cc_sum, cc_sum + cc_sum_offset + start_i,
				     i_range.size(), compute_evt);

	const auto t2 = std::chrono::steady_clock::now();
        dt = elapsed_ms(t1, t2);
        println("Total time : " + std::to_string(dt) + " ms");
        println("------------------------------");
        println("##############################");
    }
    dt = elapsed_ms(loop_start, std::chrono::steady_clock::now());
    println("All loop iterations : " + std::to_string(dt) + " ms");
    println("------------------------------");
    // Free memory per queue
    for (int i = 0; i < queues.size(); i++) {
	auto Q = queues[i];
	// Wait for last submission of this queue
	Q.wait();
	auto dev_mem = device_data[i];
	sycl::free(dev_mem.templates, Q);
	sycl::free(dev_mem.sum_sq_templates, Q);
	sycl::free(dev_mem.data, Q);
	sycl::free(dev_mem.weights, Q);
	sycl::free(dev_mem.moveouts, Q);
    }
} // matched_filter
} // extern C

This version uses only USM pointers, as you can see, and uses multiple queues to submit. Naively it seems that this should work, the queues should be able to run in parallel as there are no dependencies between queues nor loop iterations (the for (int t = 0; t < n_templates; ++t))

I don't see anything immediately; does it execute concurrently if you increase problem size? For short-running kernels, I can imagine that the execution of one iteration of the loop and processing all the submits might take longer than the actual operations on device. In this case, they would appear to run sequentially.
What happens if you turn the queue into an in-order queue?

Maybe that could be the problem, but I don't have a larger problem size at the moment.
From the println debugging I can see that All loop iterations : 0.310000 ms, so job submission seem to end quite quickly, but then it takes another 5 seconds before the jobs are done. Based on the Nsight trace I see that each kernel runs for around 300+ms:

Changing to in-order queue changes nothing (I tested by changing  the queues.push_back(sycl::queue{device}) to queues.push_back(sycl::queue{device, sycl::property::queue::in_order()})

From the println debugging I can see that All loop iterations : 0.310000 ms, so job submission seem to end quite quickly

It offloads submission to a worker thread to process things in the background, so you cannot really draw this conclusion unfortunately. A submit() will return pretty much immediately to the main thread.
Are you sure that we actually see kernels running here? Are the blue bars kernels or memory transfers? The CUDA API trace looks suggestively like it spends most of its time actually doing data transfers, not kernels (but I don't know from the top of my head how nsight visualizes things, and I don't have it here right now to verify).
You do data copies in every iteration, which might have a substantial impact. Is it really necessary to copy back every iteration?
I know that with a single device, NVIDIA typically can only do data transfers concurrently if any involved host pointers are from page-locked ("pinned") host memory. Such memory can be obtained in SYCL from sycl::malloc_host(). I don't know if this also applies to multi-device concurrency, but it might be worth a try - of course, even better would be to just eliminate unnecessary data transfers.

Thank you so much @illuhad, the point about CUDA not being able to concurrently copy data to host pointers turned out to be the culprit in this scenario. I changed it to store everything in device memory until the end and now it is working (running kernels in parallel on two GPUs)!
Thank you both so much for the help, I have learned a lot!


Continuous error display for new version

Hello,
first of all thank you for the nice Extension.
I have just downloaded the new version 2.24.1 and installed.
However, I consistently get the red error icon displayed.
The old Versio still works very well for me, have the first reinstalled! 2.24.0
Greetings
Here is the error message

Just see, the automatic rewards do not work either

I am also receiving this error

Me too

same here

yh same

use Microsoft Edge

alr

use Microsoft Edge

No Thanks

use Microsoft Edge

I am using Edge, and still get the error. 2.24.0 works fine, 2.24.1 does not.

Uncaught (in promise) ParseJSONFailed: Fail to parse the received json document. Has MSR updated its json structure? InnerEx: TypeError: Cannot read properties of null (reading 'promotionType')

Context

extension://bgodbankkcableilakajbiaceeeflpef/_generated_background_page.html

Stack Trace

status/DailyRewardStatus.js:208 (DailyRewardStatus._parsePunchCards)
status/DailyRewardStatus.js:194 (DailyRewardStatus._parseDetailedUserStatus)
status/DailyRewardStatus.js:58 (DailyRewardStatus.update)
background.js:89 (async checkDailyRewardStatus)
background.js:68 (async doBackgroundWork)

Seems to be in this line 208:
           const promoTypes = card.parentPromotion.promotionType.split(',');

Hi All,
Thanks a lot for reporting. I just managed to get back from a holiday. I've pushed a fix for this error. All should be resolved now!


Added lozenge-bounce.html

Hey, I added a new animation with lozenge shapes. :smiley:

I will transfer the ownership to @tobiasahlin. Feel free to delete if you don't want the changes! :smile_cat:

@niklasravnsborg Closing this for now, since it's outdated, but feel free to update to match new way of contributing :heart:


No error, uses CPU instead of GPU.

I understand it's difficult to impossible to diagnose without any error or log, but I really don't have one to provide. I've tried installing the OpenCL version (I primarily use the DirectML version because it actually works for me) but it only uses my CPU and without providing any errors or anything in logs. I'm running on a fresh installation, no additional models downloaded or anything. I've tried selecting GPU to auto and my 7900xt.
The only possible thing I could think of is that this may be a driver issue? (I use 23.11.1 so that Davinci Resolve works on my system). I read through changelogs, and optimizations for AI have been made since, but as far as I could tell that doesn't necessarily mean an update is required.

Are you sure you're using the latest UVR version?
You can optionally try out the latest Roformer patch:
https://github.com/Anjok07/ultimatevocalremovergui/releases/tag/v5.6
I assume you have GPU Conversion checked.
Alternatively, try out copying DirectML.dll from your Windows installation into Ultimate Vocal Remover\torch_directml but it worked on some old drivers when GPU Conversion was crashing all the separations.

Bold of both of us to assume I had GPU conversion checked, apparently. I somehow managed to have that off. Thank you.


Add support for header-based versioning

Is there a plan to support header based api versioning? There are some folks out there that prefer to use the headers to provide the version information mainly for 2 reasons (see Troy Hunt post ):

the URL to the resource should not change
headers are already used for similar purposes, as describing how the data should be returned

Some months ago, I've created my own versioning system taking inspiration by MVC Versioning WebSite and essentially adding the support for header-based versioning (with a HeaderVersionRangeValidator) which is working pretty well (I've added also with Swagger support).
Since it would be nice to use a standardized facility for this, the one that you're building here, I was wandering if there are plans in this direction that will allow me to replace my custom code.

Doesn't the API-Version-Reader do what you are required to do in the latest version?
It is one line to configure and you can then let the clients send a specific header for an API version request, with just a change to the reader configured you support also the header and query versioning very fast.

@Duranom exactly right.
@timotyweis, interesting that you site that particular link. It's one of the first questions asked here. Although it's been closed for a while, issue #42 talks about this in detail with a possible implementation. Succinctly, the issue is that there is no standard, reliable format used in media types that warrants creating an IApiVersionReader implementation out-of-the-box. If the community comes to a consensus on what this media type would look like, I'm happy to entertain including it. From all the methods I've seen, there are at least two common, but distinct way of using media types for versioning. I agree with stable URLs, which is one reason why the query string is the default. The query portion of the URL is technically not part of the path, which is what identifies a resource. I hope that helps.

@Duranom thank you for pointing it out. I'll try to replace my own implementation then
@commonsensesoftware thank you for the info, the thread in the #42 issue is really interesting.
Do you plan to support Swagger by providing proper operation filters? Could be an interesting extensions (clearly on both sides)...

Re-opening this issue so that it can be linked to the forthcoming PR.
After quite a bit of thought, there is a way to make this work standard and out-of-the-box using a media type parameter. The feature will provide a new MediaTypeApiVersionReader which will use a media type parameter to extract the version from.  The default parameter name will be "v", but it can be changed. The value of the parameter will be extracted from the Accept or Content-Type header depending on the type of request being made. If both headers are present, then the Content-Type is preferred as it is the least ambiguous.
The typical setup will look like:
options.ApiVersionReader = new MediaTypeApiVersionReader();

Versioned requests can now be matched with this request:
GET /api/helloworld HTTP/1.1
host: localhost
accept: application/json;v=2.0

POST /api/helloworld HTTP/1.1
host: localhost
content-type: application/json;v=2.0
content-length: 25

{"message":"Hello world"}

If the Accept header has qualities, the first media type with a defined version and the highest quality will be matched. It's possible that the media type support could vary by API version. This seems to be an unlikely or, at least, uncommon scenario. Complex handling for this type of media type negotiation will have to be achieved by developer extension.


We merged your local files with your environment's files 10 times, but your local and environment's files still don't match.

Error ID
5e8151e2-b798-49f7-9252-88336d10d798
What happened?
I cannot seem to determine what started to cause the issue but this is the error message that I get when trying to run ggt dev : We merged your local files with your environment's files 10 times, but your local and environment's files still don't match..
It seems like I am able to both pull and push. However I cannot have my ggt dev watcher live.
What did you expect to happen?
Being able to run ggt dev as usual.
Steps to reproduce
No response
Anything else?
Application  musc-intime
Environment  development
Branch  feature/customer-payment-method-improvements

So strange, I'm not sure why this happened 🤔
Please re-open if it happens again.


Performance regression querying changes using _doc_ids filter

When upgrading from v1.7.1 to v2.2.0 I noticed our replication was taking longer. I investigated further and found the problem was specifically in relation to the initial request for changes. We use the _doc_ids filter so we can replicate only certain documents and this has been stable and performant on the 1.x versions.
Steps to Reproduce
Use this node script to create a database and fill it with 1 million docs and then query it for specific IDs.
Context
In my testing using the above script I got responses in 1 to 2ms on v1.7.1 and 2500 to 2600ms on 2.2.0.
In our production database with real world data and about 8 million docs it takes less than a second on v1.7.1 and around 40 seconds on v2.2.0.
This has affected real world performance for users trying to replicate their data.
Your Environment

Version used: 2.2.0 (from 1.7.1)
Operating System: Ubuntu 1804
Link to your project: https://github.com/medic/medic-webapp/

What I've tried

changing clustering from n=3 q=8 to n=1 q=1 - no improvement
changing the changes_doc_ids_optimization_threshold at 1, 100, and very large - no improvement
using a very large seq_interval parameter - no improvement
using a mango selector - worse performance

ok, I think I get this finally. Before 2.0, we had an optimization for _doc_ids and _design filter;
commit bfa0a8900163edd4f85c7bbf5b595de9885cfbf9
Author: Filipe David Borba Manana<EMAIL_ADDRESS>Date:   Tue Sep 20 22:55:29 2011 +0000

    Efficient implementation of builtin filters

    Currently, the builtin changes filters "_doc_ids" and "_design"
    are not very efficient because they fold the entire seq btree
    and then filter the values by document ID.
    This implementation avoids that by doing direct lookups against
    the id btree, and then, for continuous changes requests, it
    just listens for database update events and does partial seq
    btree folds.

    COUCHDB-1288

The clustered code for _changes does not use it.

we pass {default,main_only} down as the filter, so the optimization does not fire.
It's not yet clear if this can be restored given how the clustered version of changes feed has to work, but at least this is progress.

so fabric_rpc:changes always calls couch_db:fold_changes .

@garethbowen This is fixed by #1771 and will appear in CouchDB 2.3.0.

Awesome. Thanks for the quick turnaround!

I ran the test script above against 2.3.0-RC1 and got responses in 4 to 14ms which is slightly slower than 1.7.1 (1 to 2ms) and massively faster than 2.2.0 (2500 to 2600ms). Great work @janl !


refactor remove unused code in profile

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run feut


Clarify language around "max" CLI flags

I've attempted to be more explicit and consistent with the language used to explain and differentiate between the two CLI flags: --max-results and --results-per-call. These are understandably a bit confusing and https://github.com/twitterdev/search-tweets-python/issues/37 reminded me that we could try to be more specific.

Thanks for the clarification, and no problem. I'll make those changes and add another commit.

Perhaps we can roll #39 into this? How would you feel about that, @jrmontag? We can also do that in a different PR.

@binaryaaron I've reverted that change and move the text into the correct file. Let me know if you'd like to see any other changes.

@jrmontag @binaryaaron could you guys follow up on this?


Performance of GPT-3.5

Hello, thanks for the exciting work. I have a question.
Is MINDACT (w/ GPT-3.5) reported in this paper the result of using both ranking LM and GPT?
If so, has it been tested on GPT-3.5 alone?
I want to know how GPT-3.5 with only Demotions 3 ICT performs on MIND2WEB dataset.
Looking forward to your reply

By the way, have you tried the test results on GPT3.5 with only three context instances? I'm not sure if I made myself clear.

Yes, the results in the Mind2Web paper are using the ranking LM. Without the ranking LM, GPT-3.5 (or also 4 for this matter) can barely complete any task because the raw HTML is too noisy and complex for text-only LLMs. Check newer results in the SeeAct paper (https://arxiv.org/pdf/2401.01614.pdf). With additional visual input, it's becoming possible to use GPT-4V alone without a ranking LM.

@ysu1989
Thank you for your reply.  I'll take a look at your latest work. It's really interesting.
I have understood how Mind2Web performs an element selection on a page.
But I still have another doubt, that is, how Mind2Web realizes a multi-step task, which requires jumping between multiple web pages. When a task needs to operate multiple different pages continuously, how to achieve it?
Thank you for your reply


running podman as sudo cause umask of 0000 inside of container. why?

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
In trying to figure out some permissions issues involved with attempting to share a local folder between two containers: one running rootful and one running rootless, I discovered some strange behavior with umask.
Running podman rootless, the default umask is always 0022.
Running sudo podman, the default umask is always 0000.
Running sudo docker, the default umask is always 0022.
I would expect the same result in all cases but it isn't that way.
Steps to reproduce the issue:

podman run --rm -it alpine:latest /bin/sh / # umask 0022

sudo podman run --rm -it alpine:latest /bin/sh / # umask 0000

sudo docker run --rm -it alpine:latest /bin/sh / # umask 0022
Describe the results you received:
seeing default umask of 0000 when running sudo podman
Describe the results you expected:
expected to get default umask of 0022
Output of podman version:
Version:            1.8.0
RemoteAPI Version:  1
Go Version:         go1.13.6
OS/Arch:            linux/amd64

Output of podman info --debug:
Version:            1.8.0
RemoteAPI Version:  1
Go Version:         go1.13.6
OS/Arch:            linux/amd64
[core@localhost mnt]$ podman info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.13.6
  podman version: 1.8.0
host:
  BuildahVersion: 1.13.1
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.10-2.fc31.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.10, commit: 6b526d9888abb86b9e7de7dfdeec0da98ad32ee0'
  Distribution:
    distribution: fedora
    version: "31"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree:<PHONE_NUMBER>
  MemTotal:<PHONE_NUMBER>
  OCIRuntime:
    name: runc
    package: runc-1.0.0-102.dev.gitdc9208a.fc31.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: 96f6022b37cbe12b26c9ad33a24677bec72a9cc3
      spec: 1.0.1-dev
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 1
  eventlogger: journald
  hostname: localhost.localdomain
  kernel: 5.5.5-200.fc31.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.0-20.1.dev.gitbbd6f25.fc31.x86_64
    Version: |-
      slirp4netns version 0.4.0-beta.3+dev
      commit: bbd6f25c70d5db2a1cd3bfb0416a8db99a75ed7e
  uptime: 22h 0m 20.5s (Approximately 0.92 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - quay.io
store:
  ConfigFile: /var/home/core/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.5-2.fc31.x86_64
      Version: |-
        fusermount3 version: 3.6.2
        fuse-overlayfs: version 0.7.5
        FUSE library version 3.6.2
        using FUSE kernel interface version 7.29
  GraphRoot: /var/home/core/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 3
  RunRoot: /run/user/1000/containers
  VolumePath: /var/home/core/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):
podman-1.8.0-2.fc31.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):
Fedora CoreOS

I am not able to reproduce, I am using the same version of runc and podman as reported above.
Is your reproducer just podman --runtime /usr/bin/runc run --rm -ti alpine sh -c 'umask?
Are you sure you are not using crun as the runtime when running as root?  There was an issue with an older crun version but that should be fixed now

Okay.  If I use that --runtime parameter I get the expected results but for some reason if I just use:
sudo podman run --rm -ti alpine sh -c 'umask'
with no --runtime=/usr/bin/runc, I still get 0000.
I'm not really familiar though with that parameter.  What else would my Fedora CoreOS system be using instead of runc and how might I find out?
Thanks!

Can you make a container without --runtime and provide the output of podman inspect for that container? It should contain the name of the OCI runtime in use.

Also podman info which should show the version.

Sure.  I created a container using sudo podman --rm -ti alpine /bin/sh and then switched to another session and inspect it and got the following (sorry the formatting is so bunched up):
[     {         "Id": "74340be6f373e5febf23b28e2fb88983dc0e42dd203b1d0c3af8dba01d2e2d61",         "Created": "2020-03-17T01:30:48.786055569+08:00",         "Path": "/bin/sh",         "Args": [             "/bin/sh"         ],         "State": {             "OciVersion": "1.0.1-dev",             "Status": "running",             "Running": true,             "Paused": false,             "Restarting": false,             "OOMKilled": false,             "Dead": false,             "Pid": 3311,             "ConmonPid": 3307,             "ExitCode": 0,             "Error": "",             "StartedAt": "2020-03-17T01:30:49.300816462+08:00",             "FinishedAt": "0001-01-01T00:00:00Z",             "Healthcheck": {                 "Status": "",                 "FailingStreak": 0,                 "Log": null             }         },         "Image": "e7d92cdc71feacf90708cb59182d0df1b911f8ae022d29e8e95d75ca6a99776a",         "ImageName": "docker.io/library/alpine:latest",         "Rootfs": "",         "Pod": "",         "ResolvConfPath": "/var/run/containers/storage/overlay-containers/74340be6f373e5febf23b28e2fb88983dc0e42dd203b1d0c3af8dba01d2e2d61/userdata/resolv.conf",         "HostnamePath": "/var/run/containers/storage/overlay-containers/74340be6f373e5febf23b28e2fb88983dc0e42dd203b1d0c3af8dba01d2e2d61/userdata/hostname",         "HostsPath": "/var/run/containers/storage/overlay-containers/74340be6f373e5febf23b28e2fb88983dc0e42dd203b1d0c3af8dba01d2e2d61/userdata/hosts",         "StaticDir": "/var/lib/containers/storage/overlay-containers/74340be6f373e5febf23b28e2fb88983dc0e42dd203b1d0c3af8dba01d2e2d61/userdata",         "OCIConfigPath": "/var/lib/containers/storage/overlay-containers/74340be6f373e5febf23b28e2fb88983dc0e42dd203b1d0c3af8dba01d2e2d61/userdata/config.json",         "OCIRuntime": "crun",         "LogPath": "/var/lib/containers/storage/overlay-containers/74340be6f373e5febf23b28e2fb88983dc0e42dd203b1d0c3af8dba01d2e2d61/userdata/ctr.log",         "LogTag": "",         "ConmonPidFile": "/var/run/containers/storage/overlay-containers/74340be6f373e5febf23b28e2fb88983dc0e42dd203b1d0c3af8dba01d2e2d61/userdata/conmon.pid",         "Name": "busy_kalam",         "RestartCount": 0,         "Driver": "overlay",         "MountLabel": "system_u:object_r:container_file_t:s0:c633,c704",         "ProcessLabel": "system_u:system_r:container_t:s0:c633,c704",         "AppArmorProfile": "",         "EffectiveCaps": [             "CAP_CHOWN",             "CAP_DAC_OVERRIDE",             "CAP_FSETID",             "CAP_FOWNER",             "CAP_MKNOD",             "CAP_NET_RAW",             "CAP_SETGID",             "CAP_SETUID",             "CAP_SETFCAP",             "CAP_SETPCAP",             "CAP_NET_BIND_SERVICE",             "CAP_SYS_CHROOT",             "CAP_KILL",             "CAP_AUDIT_WRITE"         ],         "BoundingCaps": [             "CAP_CHOWN",             "CAP_DAC_OVERRIDE",             "CAP_FSETID",             "CAP_FOWNER",             "CAP_MKNOD",             "CAP_NET_RAW",             "CAP_SETGID",             "CAP_SETUID",             "CAP_SETFCAP",             "CAP_SETPCAP",             "CAP_NET_BIND_SERVICE",             "CAP_SYS_CHROOT",             "CAP_KILL",             "CAP_AUDIT_WRITE"         ],         "ExecIDs": [],         "GraphDriver": {             "Name": "overlay",             "Data": {                 "LowerDir": "/var/lib/containers/storage/overlay/5216338b40a7b96416b8b9858974bbe4acc3096ee60acbc4dfb1ee02aecceb10/diff",                 "MergedDir": "/var/lib/containers/storage/overlay/d4803b384b193b91034673be26c6998583bc9941b37c4daece641d3f6364f2f8/merged",                 "UpperDir": "/var/lib/containers/storage/overlay/d4803b384b193b91034673be26c6998583bc9941b37c4daece641d3f6364f2f8/diff",                 "WorkDir": "/var/lib/containers/storage/overlay/d4803b384b193b91034673be26c6998583bc9941b37c4daece641d3f6364f2f8/work"             }         },         "Mounts": [],         "Dependencies": [],         "NetworkSettings": {             "Bridge": "",             "SandboxID": "",             "HairpinMode": false,             "LinkLocalIPv6Address": "",             "LinkLocalIPv6PrefixLen": 0,             "Ports": [],             "SandboxKey": "/var/run/netns/cni-a7335410-6bcd-57b1-b6a1-0ab27a41a22f",             "SecondaryIPAddresses": null,             "SecondaryIPv6Addresses": null,             "EndpointID": "",             "Gateway": "<IP_ADDRESS>",             "GlobalIPv6Address": "",             "GlobalIPv6PrefixLen": 0,             "IPAddress": "<IP_ADDRESS>",             "IPPrefixLen": 16,             "IPv6Gateway": "",             "MacAddress": "fe:59:44:d0:3f:73"         },         "ExitCommand": [             "/usr/bin/podman",             "--root",             "/var/lib/containers/storage",             "--runroot",             "/var/run/containers/storage",             "--log-level",             "error",             "--cgroup-manager",             "systemd",             "--tmpdir",             "/var/run/libpod",             "--runtime",             "crun",             "--storage-driver",             "overlay",             "--storage-opt",             "overlay.mountopt=nodev,metacopy=on",             "--events-backend",             "journald",             "container",             "cleanup",             "--rm",             "74340be6f373e5febf23b28e2fb88983dc0e42dd203b1d0c3af8dba01d2e2d61"         ],         "Namespace": "",         "IsInfra": false,         "Config": {             "Hostname": "74340be6f373",             "Domainname": "",             "User": "",             "AttachStdin": false,             "AttachStdout": false,             "AttachStderr": false,             "Tty": true,             "OpenStdin": true,             "StdinOnce": false,             "Env": [                 "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",                 "TERM=xterm",                 "container=podman",                 "HOSTNAME=74340be6f373",                 "HOME=/root"             ],             "Cmd": [                 "/bin/sh"             ],             "Image": "docker.io/library/alpine:latest",             "Volumes": null,             "WorkingDir": "/",             "Entrypoint": "",             "OnBuild": null,             "Labels": null,             "Annotations": {                 "io.container.manager": "libpod",                 "io.kubernetes.cri-o.Created": "2020-03-17T01:30:48.786055569+08:00",                 "io.kubernetes.cri-o.TTY": "true",                 "io.podman.annotations.autoremove": "TRUE",                 "io.podman.annotations.init": "FALSE",                 "io.podman.annotations.privileged": "FALSE",                 "io.podman.annotations.publish-all": "FALSE",                 "org.opencontainers.image.stopSignal": "15"             },             "StopSignal": 15,             "CreateCommand": [                 "podman",                 "run",                 "--rm",                 "-ti",                 "alpine",                 "/bin/sh"             ]         },         "HostConfig": {             "Binds": [],             "ContainerIDFile": "",             "LogConfig": {                 "Type": "k8s-file",                 "Config": null             },             "NetworkMode": "default",             "PortBindings": {},             "RestartPolicy": {                 "Name": "",                 "MaximumRetryCount": 0             },             "AutoRemove": true,             "VolumeDriver": "",             "VolumesFrom": null,             "CapAdd": [],             "CapDrop": [],             "Dns": [],             "DnsOptions": [],             "DnsSearch": [],             "ExtraHosts": [],             "GroupAdd": [],             "IpcMode": "",             "Cgroup": "",             "Cgroups": "default",             "Links": null,             "OomScoreAdj": 0,             "PidMode": "",             "Privileged": false,             "PublishAllPorts": false,             "ReadonlyRootfs": false,             "SecurityOpt": [],             "Tmpfs": {},             "UTSMode": "",             "UsernsMode": "",             "ShmSize": 65536000,             "Runtime": "oci",             "ConsoleSize": [                 0,                 0             ],             "Isolation": "",             "CpuShares": 1024,             "Memory": 0,             "NanoCpus": 0,             "CgroupParent": "",             "BlkioWeight": 0,             "BlkioWeightDevice": null,             "BlkioDeviceReadBps": null,             "BlkioDeviceWriteBps": null,             "BlkioDeviceReadIOps": null,             "BlkioDeviceWriteIOps": null,             "CpuPeriod": 0,             "CpuQuota": 0,             "CpuRealtimePeriod": 0,             "CpuRealtimeRuntime": 0,             "CpusetCpus": "",             "CpusetMems": "",             "Devices": [],             "DiskQuota": 0,             "KernelMemory": 0,             "MemoryReservation": 0,             "MemorySwap": 0,             "MemorySwappiness": -1,             "OomKillDisable": false,             "PidsLimit": 4096,             "Ulimits": [                 {                     "Name": "RLIMIT_NOFILE",                     "Soft": 1048576,                     "Hard": 1048576                 },                 {                     "Name": "RLIMIT_NPROC",                     "Soft": 1048576,                     "Hard": 1048576                 }             ],             "CpuCount": 0,             "CpuPercent": 0,             "IOMaximumIOps": 0,             "IOMaximumBandwidth": 0         }     } ]

and here is sudo podman info which I see produces different output from the podman info I provided before.
`
sudo podman info
host:
BuildahVersion: 1.13.1
CgroupVersion: v1
Conmon:
package: conmon-2.0.10-2.fc31.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.0.10, commit: 6b526d9888abb86b9e7de7dfdeec0da98ad32ee0'
Distribution:
distribution: fedora
version: "31"
MemFree:<PHONE_NUMBER>
MemTotal:<PHONE_NUMBER>
OCIRuntime:
name: crun
package: crun-0.12.1-1.fc31.x86_64
path: /usr/bin/crun
version: |-
crun version 0.12.1
commit: df5f2b2369b3d9f36d175e1183b26e5cee55dd0a
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
SwapFree: 0
SwapTotal: 0
arch: amd64
cpus: 1
eventlogger: journald
hostname: localhost.localdomain
kernel: 5.5.5-200.fc31.x86_64
os: linux
rootless: false
uptime: 10m 57.84s
registries:
search:

docker.io
registry.fedoraproject.org
registry.access.redhat.com
registry.centos.org
quay.io
store:
ConfigFile: /etc/containers/storage.conf
ContainerStore:
number: 3
GraphDriverName: overlay
GraphOptions:
overlay.mountopt: nodev,metacopy=on
GraphRoot: /var/lib/containers/storage
GraphStatus:
Backing Filesystem: xfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "true"
ImageStore:
number: 12
RunRoot: /var/run/containers/storage
VolumePath: /var/lib/containers/storage/volumes
`

"--runtime", "crun" from podman inspect
So this is properly wierd. Inspect says we're creating containers with crun, but the default runtime listed in podman info is runc. I'm not sure how this is.
Regardless, I think this is probably a runc vs crun change.

I posted after you wrote that last comment but it seems sudo podman causes the use of crun and podman without sudo uses runc.

Just to confirm what you guys are saying about "crun".
podman run --runtime=/usr/bin/crun --rm -ti alpine sh -c 'umask'

produces 0000.

/etc/containers/libpod.conf doesn't exist.
/usr/share/containers/libpod.conf:
# libpod.conf is the default configuration file for all tools using libpod to
# manage containers

# Default transport method for pulling and pushing for images
image_default_transport = "docker://"

# Paths to look for the conmon container manager binary.
# If the paths are empty or no valid path was found, then the `$PATH`
# environment variable will be used as the fallback.
conmon_path = [
            "/usr/libexec/podman/conmon",
            "/usr/local/libexec/podman/conmon",
            "/usr/local/lib/podman/conmon",
            "/usr/bin/conmon",
            "/usr/sbin/conmon",
            "/usr/local/bin/conmon",
            "/usr/local/sbin/conmon",
            "/run/current-system/sw/bin/conmon",
]

# Environment variables to pass into conmon
conmon_env_vars = [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
]

# CGroup Manager - valid values are "systemd" and "cgroupfs"
cgroup_manager = "systemd"

# Container init binary
#init_path = "/usr/libexec/podman/catatonit"

# Directory for persistent libpod files (database, etc)
# By default, this will be configured relative to where containers/storage
# stores containers
# Uncomment to change location from this default
#static_dir = "/var/lib/containers/storage/libpod"

# Directory for temporary files. Must be tmpfs (wiped after reboot)
tmp_dir = "/var/run/libpod"

# Maximum size of log files (in bytes)
# -1 is unlimited
max_log_size = -1

# Whether to use chroot instead of pivot_root in the runtime
no_pivot_root = false

# Directory containing CNI plugin configuration files
cni_config_dir = "/etc/cni/net.d/"

# Directories where the CNI plugin binaries may be located
cni_plugin_dir = [
               "/usr/libexec/cni",
               "/usr/lib/cni",
               "/usr/local/lib/cni",
               "/opt/cni/bin"
]

# Default CNI network for libpod.
# If multiple CNI network configs are present, libpod will use the network with
# the name given here for containers unless explicitly overridden.
# The default here is set to the name we set in the
# 87-podman-bridge.conflist included in the repository.
# Not setting this, or setting it to the empty string, will use normal CNI
# precedence rules for selecting between multiple networks.
cni_default_network = "podman"

# Default libpod namespace
# If libpod is joined to a namespace, it will see only containers and pods
# that were created in the same namespace, and will create new containers and
# pods in that namespace.
# The default namespace is "", which corresponds to no namespace. When no
# namespace is set, all containers and pods are visible.
#namespace = ""

# Default infra (pause) image name for pod infra containers
infra_image = "k8s.gcr.io/pause:3.1"

# Default command to run the infra container
infra_command = "/pause"

# Determines whether libpod will reserve ports on the host when they are
# forwarded to containers. When enabled, when ports are forwarded to containers,
# they are held open by conmon as long as the container is running, ensuring that
# they cannot be reused by other programs on the host. However, this can cause
# significant memory usage if a container has many ports forwarded to it.
# Disabling this can save memory.
#enable_port_reservation = true

# Default libpod support for container labeling
# label=true

# The locking mechanism to use
lock_type = "shm"

# Number of locks available for containers and pods.
# If this is changed, a lock renumber must be performed (e.g. with the
# 'podman system renumber' command).
num_locks = 2048

# Directory for libpod named volumes.
# By default, this will be configured relative to where containers/storage
# stores containers.
# Uncomment to change location from this default.
#volume_path = "/var/lib/containers/storage/volumes"

# Selects which logging mechanism to use for Podman events.  Valid values
# are `journald` or `file`.
# events_logger = "journald"

# Specify the keys sequence used to detach a container.
# Format is a single character [a-Z] or a comma separated sequence of
# `ctrl-<value>`, where `<value>` is one of:
# `a-z`, `@`, `^`, `[`, `\`, `]`, `^` or `_`
#
# detach_keys = "ctrl-p,ctrl-q"

# Default OCI runtime
runtime = "crun"

# List of the OCI runtimes that support --format=json.  When json is supported
# libpod will use it for reporting nicer errors.
runtime_supports_json = ["crun", "runc"]

# List of all the OCI runtimes that support --cgroup-manager=disable to disable
# creation of CGroups for containers.
runtime_supports_nocgroups = ["crun"]

# Paths to look for a valid OCI runtime (runc, runv, etc)
# If the paths are empty or no valid path was found, then the `$PATH`
# environment variable will be used as the fallback.
[runtimes]
runc = [
            "/usr/bin/runc",
            "/usr/sbin/runc",
            "/usr/local/bin/runc",
            "/usr/local/sbin/runc",
            "/sbin/runc",
            "/bin/runc",
            "/usr/lib/cri-o-runc/sbin/runc",
            "/run/current-system/sw/bin/runc",
]

crun = [
                "/usr/bin/crun",
                "/usr/sbin/crun",
                "/usr/local/bin/crun",
                "/usr/local/sbin/crun",
                "/sbin/crun",
                "/bin/crun",
                "/run/current-system/sw/bin/crun",
]

# Kata Containers is an OCI runtime, where containers are run inside lightweight
# Virtual Machines (VMs). Kata provides additional isolation towards the host,
# minimizing the host attack surface and mitigating the consequences of
# containers breakout.
# Please notes that Kata does not support rootless podman yet, but we can leave
# the paths below blank to let them be discovered by the $PATH environment
# variable.

# Kata Containers with the default configured VMM
kata-runtime = [
    "/usr/bin/kata-runtime",
]

# Kata Containers with the QEMU VMM
kata-qemu = [
    "/usr/bin/kata-qemu",
]

# Kata Containers with the Firecracker VMM
kata-fc = [
    "/usr/bin/kata-fc",
]

# The [runtimes] table MUST be the last thing in this file.
# (Unless another table is added)
# TOML does not provide a way to end a table other than a further table being
# defined, so every key hereafter will be part of [runtimes] and not the main
# config.

~/.config/containers/libpod.conf doesn't exist for either core user or root user.
It is odd to me that the default runtime in the one file is crun when that is not what rootless podman uses.

@vrothberg @baude Rootless should be basing its configuration for some things (runtime being among them) on root's libpod.conf, but it doesn't seem like that's happening here?

Just to confirm what you guys are saying about "crun".
podman run --runtime=/usr/bin/crun --rm -ti alpine sh -c 'umask'

produces 0000.

can you tell me the output for /usr/bin/crun --version?  Could you try crun-0.13?

can you tell me the output for /usr/bin/crun --version? Could you try crun-0.13?

$ /usr/bin/crun --version
crun version 0.12.1
commit: df5f2b2369b3d9f36d175e1183b26e5cee55dd0a
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL

I'm not sure how to upgrade crun on Fedora CoreOS.  I will see if I can figure that out but would appreciate any direction if you have it.

Just FYI.  There doesn't appear to be a way to upgrade crun on Fedora CoreOS.  Or at least I don't know how so I cannot try this with crun-0.13

@wdouglascampbell Do you still have this issue?  @dustymabe When does CoreOS get updated?

@wdouglascampbell Do you still have this issue? @dustymabe When does CoreOS get updated?

Thanks!  I just checked again and yes, this is resolved now.  It seems a recent CoreOS update deployed crun 0.13 that is indeed working.


Temporary queues example

How do we create a temporary queue and bind this to an exchange?
Thanks

It depends the broker you are using. Prefix of the destination is significant (like /topic for durable queue) Moreover options to broker queue creation can generally be sent via headers.
Look at https://www.rabbitmq.com/stomp.html section destination for example

@pantonis if you find your answer could you right it down here so users with the same question wont be frustrated?
Also might helper me to improve the doc.


Binding Ctrl-C to "copy" prevents using Ctrl-C for cancelling a command

Description of the new feature/enhancement
Right now when you add the following to your keybindings:
{
                "command" : "copy",
                "keys" :
                [
                    "ctrl+c"
                ]
            },

It prevents Ctrl-C from terminating processes, even when there's no current text selection.
If I remove the keybinding (or change it to "ctrl+shift+c"), then I get my Ctrl-C for terminating processes back, but lose the ability to copy with Ctrl-C when text is selected.
Compare this with the current command prompt, where pressing Ctrl-C when no text is selected cancels/terminates whatever process is running, while pressing Ctrl-C with text selected copies the text.
My guess is that keybindings probably need to have an extra additional option for "this keybinding is only active while text is selected".

This is fixed in 0.3,
"copy" is now bound to "ctrl+shift+c"
(see #2014, # 2014, #1093, and possibly others...)

I don't want to use Ctrl-Shift-C for copy; I'd like to use Ctrl-C for copy when I have text selected, as I can in both the native Windows terminal and ConEmu (as well as Ctrl-C being "copy" in every other Windows application).

ctrl+shift+c still does not work.
Ctrl+c is a standard is another thing... And one of the most important thing we can expect from shell environment. There should be a way to extract info...
Version: 0.3.2142.0

The copy key handler should be returning false if there was no active selection.

Oh hey, #2285 is totally a dup of this issue, but that thread has a bit more details in it, so I'm gonna close this one as a dupe of that one.


[docs] Demo SearchIcons crashes

[ ] I have searched for similar issues in this repository and believe that this is not a duplicate.

Steps to Reproduce

Visit https://mui.com/components/material-icons/?query=image
Click "Image"
demo SearchIcons crashes

Your Environment

Tech
Version

MUI
v5.2.5

netlify deploy
https://61c098e520eae9000736c0df--material-ui-docs.netlify.app

Browser
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.93 Safari/537.36

Hi @dagumak. It was fixed in #30319 and should work now in the new docs version. Please hard refresh the docs page.


Bump to dbt-core 1.3.0

Bump to dbt-core 1.3.0

I will take a look at the different release notes, and come back with a PR.


Variable value is correctly interpolated into the warning text

Fixes issue #176
Anyway to get visual confirmation?

All right, merging, thx.


ASSETS-20895 remove rendition data from action results

For https://jira.corp.adobe.com/browse/ASSETS-20895 Minimize activation response size of workers (COGS impact)
For workers using sdk: removed rendition and source data from activation results.
related pipeline pr
Result before code changes:

Result after code changes:

dependent-module-tests are failing. which I am looking more into, but are unrelated.

verified that dependent-test failure not related to this change.


[Bug]: Widget error states does not have different styles for different selection states

Is there an existing issue for this?

[X] I have searched the existing issues

Description
Widget error states does not have different styles for different selection states

Steps To Reproduce

Hover over a widget in error state
Select a widget in error state
Observe both states have the same design style

Public Sample App
No response
Environment
Deploy Preview
Severity
High (Blocker to building or releasing)
Issue video log
No response
Version
cloud

@tbrizitsky can we have designs for this so it can be part of widget polish?

This is as per the design


esp_http_client_open() causes huge memory consumption (IDFGH-5864)

`
// Free heap after calling esp_http_client_open() (reading 512 bytes once)
D (57423) SYSTEM_MNGR: (55) free heap: internal 25136 - external 4122820
// Free heap after calling esp_http_client_cleanup()
D (58425) SYSTEM_MNGR: (56) free heap: internal 98268 - external 4124496
`
I am running esp-idf version 4.2.2
Is this a bug? If no, is there a possibility to use external SPIRAM instead for this?

@ac-frg Please help check

Before esp_http_client_open, how much free heap is there?
Do this http connection is over TLS, if yes, is it two-way authentication?
For how to use SPIRAM, you can refer to https://docs.espressif.com/projects/esp-idf/en/v4.2.2/esp32/api-guides/external-ram.html

Free heap: internal 98268 - external 4124496
Yes


Which version of go_buildpack is suitable to push the app?

Hi,
I have tried to push the app using go_buildpack v 1.7.10 but the push failed and from the logs, I found out that,  only supported versions are as follows:
-----> Checking Godeps/Godeps.json file.
DEPENDENCY MISSING IN MANIFEST: go 1.7
It looks like you're trying to use go 1.7.
Unfortunately, that version of go is not supported by this buildpack.
The versions of go supported in this buildpack are:
- 1.6.3
- 1.6.2
- 1.5.4
- 1.5.3

I tried with an older version having go 1.6.3 version in go buildpack. Below error prevented me a successful push:
file:///tmp/buildpacks/21466bd6544f9c7c1e98de2a3425342f/dependencies/https___pivotal-buildpacks.s3.amazonaws.com_concourse-binaries_godep_godep-v14-linux-x64.tgz
-----> Checking Godeps/Godeps.json file.
-----> Resource https://storage.googleapis.com/golang/go1.7.linux-amd64.tar.gz is not provided by this buildpack. Please upgrade your buildpack to receive the latest resources.
Failed to compile droplet
-----> Installing go1.7... Exit status 223
Staging failed: Exited with status 223

FAILED

Appreciate the suggestions.
Thanks,
Chandan

Try pushing the app with newer version of the buildpack: cf push yourapp -b https://github.com/cloudfoundry/go-buildpack.


Fix mesh plot transparency not being honored.

Ensure all actors receive the opacity setting.
Resolves #19141
Type of change

[X] Bug fix

Checklist:

[X] I have commented my code where applicable.~~
[X] I have updated the release notes.~~
- [ ] I have made corresponding changes to the documentation.
- [ ] I have added debugging support to my changes.
- [ ] I have added tests that prove my fix is effective or that my feature works.
- [ ] I have confirmed new and existing unit tests pass locally with my changes.
- [ ] I have added new baselines for any new tests to the repo.
[X] I have NOT made any changes to protocol or public interfaces in an RC branch.~~

@BradWhitlock  sorry for the re-review request. I did something wrong with my previous branch trying to resolve conflicts for the merge.

Thanks @BradWhitlock !


Pgp/mime

current merge

this schould be intern, my github just didnt change the path correctly, sry about that


Function egg, 2FA, bidprices

Hey Guys,
Thanks a lot for developing this API, it really is a gift from above.
To further improve the security, I was thinking to implement 2FA (à la: https://github.com/pladaria/degiro) and some way of pulling bid prices (ideas?). However the egg thingy prevents me from doing so...
Relatively new to python, but why did you use an egg structure in this context?

Bid prices are not given when calling the API but maybe a work-around with beautiful soup could somewhat solve the issue, donno

Unfortunately, the original author is unresponsive, so they are not accepting Pull Requests with additional features.
Somebody may have to take over, fork this project and start publishing it under a different name in PyPI.
2FA
You could notice that 2FA has been contributed by @pforero here: #12
If you want, you can very easily install his version that has 2FA using this command:
pip install -U git+https://github.com/pforero/DegiroAPI.git@totp

egg
As of the "egg", I'm not sure what you're talking about. You need to explain what your problem with the "egg" is.
My wild guess would be that you installed this package from PyPI (Python repository) and then you attempted to edit the files in ...\Lib\site-packages\degiroapi.
If you want to make your own edits in this project, you should instead:

Clone this project, or download and extract the source
In the folder that contains "setup.py", run pip install -U -e .
Now, when you make changes in your source code, they take an immediate effect, no need to rerun pip install again.

Bid prices
What do you mean exactly? Price of share in the last trade? Or actually the current best Bid and best Ask?

Wow! Great response! Thanks man!
I will definitely check the 2FA version out! As for the egg, I meant a Python Egg (first time seeing it),  and I did indeed try to make some changes to the init file to achieve 2FA. But I couldn't, so it bothered me. But that's not an issue anymore with the 2FA version you referenced. Need to look into the Egg thingy a bit deeper tho...
I do indeed mean the current best bid and best ask. At the moment I'm using the last trade price, since its the only price we can consult via the API. However best bid and best ask price would be more useful. When looking at the Degiro UI the best bid and ask prices are provided. So I was wondering if it was possible to scrape them from the UI and use them that way. Probably not the pentacle of efficiency, so ideas for a more elegant solution are greatly appreciated :)

https://packaging.python.org/discussions/wheel-vs-egg/
Wheel is superior. But you don't need to get into that too much. Both are just packaging standards. All you need to do is to install each package using pip - either from PyPI, or directly from source as I described.
If you do the editable/debug install (parameter -e) you can forget about wheels and eggs. It will just work.
As for the bid/ask data:
I highly discourage you from trying to scrape the data from the UI. The UI is very dynamic, so Beautiful Soup will only see an almost empty page. You'd have to automate a full-blown browser instance with something like Selenium, and that's an overkill.
Our best bet is to find which API end-points Degiro uses to get the data. But that seems to be quite a challenge; see https://github.com/lolokraus/DegiroAPI/issues/6#issuecomment-653456750. I tried to find the source using Chrome DevTools, with no success. Apparently, it could be seen using tools like Wireshark but I haven't tried that myself.
BTW, are you sure the bid/ask data is so important to you? Aren't they delayed by 15-30 minutes anyway?

P.S.: Do you know if retrieving the bid/ask works in the library that you linked (https://github.com/pladaria/degiro)? If is does, then we may be able to ~steal~ borrow the solution from here: https://github.com/pladaria/degiro/blob/b3fcb2ab12f02e5c74ac3db917c344806ba22ff9/src/index.js#L80-L162

Yep I installed the 2FA version. Works just perfect!
The bid and ask prices are real prices you can get for buying and selling a stock. The last trade price is just indicative in terms of the current bid and ask price. The current best bid or ask could be laying way above or below this price. btw they are real time.
Consider the liquidity of the stock.  For liquid stocks (take Amazon) with tight spreads (0.09%), you can go off the last trade price and most of the time you will find a buyer/seller for your order. However, for less liquid stocks, with a bigger spread and not a deep orderbook you might overpay/get underpaid substantially. These profits go out of your pocket, into a market maker's.
Same occurs with big market events like covid and the vaccines (even liquid stocks become illiquid). So, if you want the BEST prices at ALL times, you should use current bid and ask prices.
It also allows you to easily develop your own market making or liquidity provider bot. Be it a naïve one :)

P.S.: Do you know if retrieving the bid/ask works in the library that you linked (https://github.com/pladaria/degiro)? If is does, then we may be able to ~steal~ borrow the solution from here: https://github.com/pladaria/degiro/blob/b3fcb2ab12f02e5c74ac3db917c344806ba22ff9/src/index.js#L80-L162

I dont know if it works. Looks fine. I kept away from Java for as long as I could. Seems the time has come to face the beast.
Will look into it when i find the time

Getting Bid-/Ask-Data (on a near realtime basis), can be done. I did it the following way:
First, collect all vwdIDs for the securities of interest (with the product_info() function of degiroapi) and put them into a list.
Then I call the following method of my trader-class to subscribe for the Bid/Ask/Last-Data:
    def get_vwd_session_and_initialise_data_request(self, listOfvwdIds):
        self.https_session = requests.Session()

        response = self.https_session.post(self.__VWD_SESSION_REQUEST_URL, params=self.vwd_parameters, data=self.vwd_payload, headers=self.vwd_headers)
        logger.info(response.text)
        self.vwd_session_id = response.json()['sessionId']

        request_string = ''
        for vwdId in listOfvwdIds:
            request_string = request_string + 'req({0}.LastPrice);req({0}.BidPrice);req({0}.AskPrice);req({0}.ClosePrice);req({0}.LastTime);req({0}.BidVolume);req({0}.AskVolume);req({0}.LastDate);'.format(vwdId)
        self.vwd_payload2 = '{{"controlData":"{0}"}}'.format(request_string)

        return self.https_session.post(self.__LATEST_PRICE_DATA_URL + self.vwd_session_id, data=self.vwd_payload2, headers=self.vwd_headers), self.vwd_session_id, self.vwd_payload2

After this call you can poll the requested information in an infinite loop with this code:
def get_exchange_data(self, vwdIds):
        # das new_tape_entry muss ein dict von dicts sein, so dass für  jede ISIN ein eigener tape-entry aufgebaut wird.
        # beim anfügen an das tape muss der tape_entry auf den 2-stufigen spaltenindex umformatiert werden.
        new_tape_entry = {}

        try:
            response = self.https_session.get(self.__LATEST_PRICE_DATA_URL + self.vwd_session_id, headers=self.vwd_headers)

            for i in response.json():
                if i['m'] == 'a_req':
                    # save id for priceKey
                    self.data_keys[i['v'][1]] = [i['v'][0].split(".")[-1], vwdIds[i['v'][0].split(".")[0]]]
                if i['m'] == 'us' or i['m'] == 'un':
                    new_tape_entry[(self.data_keys[i['v'][0]][1], self.data_keys[i['v'][0]][0])] = i['v'][1]
        except:
            logger.warning('https_session.get() failed inside get_exchange_data')

        return new_tape_entry

This method was "borrowed" from the Degiro-website. They use a similar approach. There is no push-API available, but you can pull every second. For my trading strategies that's more than fast enough.
In general, I support the suggestion from Jakub-CZ to fork the project as the author is pretty unresponsive...

Hi, nice reverse engineering job,
If you are looking for a python library which handle real-time data, I have made one :
https://github.com/Chavithra/degiro-connector
It is documented and I plan to maintain it for a while.

Hi, nice reverse engineering job,
If you are looking for a python library which handle real-time data, I have made one :
https://github.com/Chavithra/degiro-connector
It is documented and I plan to maintain it for a while.

There is now a new Degiro library every day :)
Stay safe guys, do a proper code base analysis before you pass your Degiro credentials to any library that is not your own.

There is now a new Degiro library every day :)
Stay safe guys, do a proper code base analysis before you pass your Degiro credentials to any library that is not your own.

Awesome! I have some troubles configuring @rob1181's solution for my own (I'm not that savvy). I'll definitely give @Chavithra's library a go.
Thanks for the contributions guys! :)

Awesome! I have some troubles configuring @rob1181's solution for my own (I'm not that savvy). I'll definitely give @Chavithra's library a go.
Thanks for the contributions guys! :)

There is now a new Degiro library every day :)
Stay safe guys, do a proper code base analysis before you pass your Degiro credentials to any library that is not your own.

what's your recommendation right now including 2FA and live Bid and ask?


Please don't be Arch-specific for the build system

Hello,
Even if this builds an Arch image, it doesn't have to be that this only works on an Arch system to build.
For example I launch the scripts on a Debian machine. First attempt:
bash: ./1_compile.sh: /usr/bin/sh: bad interpreter: No such file or directory
What is wrong with the normal "/bin/sh"?
Second attempt:
./consts.sh: line 61: pacman: command not found
Please install 'riscv64-linux-gnu-gcc'

Sorry I got no "pacman" on my machine, meanwhile /usr/bin/riscv64-linux-gnu-gcc-12 is happily present and ready to be executed. Why not just check for presence of the compiler, not of an Arch package of a compiler.
Feels very petty and hostile, like "f u if you use a different distro", we will specifically hardcode our own little incompatible ways so this doesn't work on your system.

I've fixed the first error by making it distro independent.
Regarding the second, I cannot (and will not) maintain different distros, if this sounds or feels hostile, deal with it - if I remove that check, people will complain that the script isn't working because they are missing packages.
Nevertheless, I'm open for PRs, so feel free to add a check for Debian.


getChildRoutes signature suggestion (resolve, reject, location)

Bikeshedding a bit here, but I've found myself forgetting the signature of getChildRoutes several times as well as that the first argument of the resolve callback being for any error.
Anyone else interested in changing it on the next major rev to be closer to Promise?
getChildRoutes(resolve, reject, location)

It's really standard for this sort of callback to take error as the first arg. Look through e.g. the Node docs.

Also, while you won't find much promise love around here, that's not the reason we don't use them. Its important for all of our async stuff to be able to also operate synchronously for server rendering :trophy:


dak.nan_to_num() not Implemented

I am implementing some corrections that involves dividing the result from one look-up table and another. Sometimes it is the case that there is division by zero, and the nan values need to be converted back to 0. I would like to ask that this unsupported call be supported.

Is dak.fill_none perhaps the one you want?
For nan_to_num, the current code looks like
    # return map_partitions(
    #     ak.nan_to_num,
    #     array,
    #     output_partitions=1,
    #     copy=copy,
    #     nan=nan,
    #     posinf=posinf,
    #     neginf=neginf,
    #     highlevel=highlevel,
    #     behavior=behavior,
    # )
    raise DaskAwkwardNotImplemented("TODO")

Would you mind uncommenting the block, to see if this already works? You could also call map_partitions directly.

It doesn't seem that dak.fill_none will replace the nan values.
I tried uncommenting this block but I get a long error and I'm unsure how to remedy this.
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File /depot/cms/top/fstrug/conda_environments/TOPCoffea/lib/python3.10/site-packages/awkward/_dispatch.py:39, in named_high_level_function.<locals>.dispatch(*args, **kwargs)
     38 with OperationErrorContext(name, args, kwargs):
---> 39     gen_or_result = func(*args, **kwargs)
     40     if isgenerator(gen_or_result):

TypeError: nan_to_num() got an unexpected keyword argument 'output_partitions'

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
Cell In[6], line 12
      9 scale_factor_den = evaluator["lo_QCD"](events['gen_ttbar_mass'], np.cos(compute_scattering_angle(events)))
     11 scale_factor = (scale_factor_num / scale_factor_den)
---> 12 scale_factor = dak.nan_to_num(scale_factor).compute()

File /depot/cms/top/fstrug/conda_environments/TOPCoffea/lib/python3.10/site-packages/dask_awkward/lib/structure.py:610, in nan_to_num(array, copy, nan, posinf, neginf, highlevel, behavior, attrs)
    599 @borrow_docstring(ak.nan_to_num)
    600 def nan_to_num(
    601     array: Array,
   (...)
    608     attrs: Mapping[str, Any] | None = None,
    609 ) -> Array:
--> 610     return map_partitions(
    611         ak.nan_to_num,
    612         array,
    613         output_partitions=1,
    614         copy=copy,
    615         nan=nan,
    616         posinf=posinf,
    617         neginf=neginf,
    618         highlevel=highlevel,
    619         behavior=behavior,
    620     )
    621     raise DaskAwkwardNotImplemented("TODO")

File /depot/cms/top/fstrug/conda_environments/TOPCoffea/lib/python3.10/site-packages/dask_awkward/lib/core.py:2107, in map_partitions(base_fn, label, token, meta, output_divisions, traverse, *args, **kwargs)
   2099         arg_lens_for_repackers.append(1)
   2101 fn = ArgsKwargsPackedFunction(
   2102     base_fn,
   2103     arg_repackers,
   2104     kwarg_repacker,
   2105     arg_lens_for_repackers,
   2106 )
-> 2107 return _map_partitions(
   2108     fn,
   2109     *arg_flat_deps_expanded,
   2110     *kwarg_flat_deps,
   2111     label=label,
   2112     token=token,
   2113     meta=meta,
   2114     output_divisions=output_divisions,
   2115 )

File /depot/cms/top/fstrug/conda_environments/TOPCoffea/lib/python3.10/site-packages/dask_awkward/lib/core.py:1943, in _map_partitions(fn, label, token, meta, output_divisions, *args, **kwargs)
   1935 lay = partitionwise_layer(
   1936     fn,
   1937     name,
   1938     *args,
   1939     **kwargs,
   1940 )
   1942 if meta is None:
-> 1943     meta = map_meta(fn, *args, **kwargs)
   1945 hlg = HighLevelGraph.from_collections(
   1946     name,
   1947     lay,
   1948     dependencies=deps,
   1949 )
   1951 if len(dak_arrays) == 0:

File /depot/cms/top/fstrug/conda_environments/TOPCoffea/lib/python3.10/site-packages/dask_awkward/lib/core.py:2467, in map_meta(fn, *deps)
   2463 def map_meta(fn: Callable | ArgsKwargsPackedFunction, *deps: Any) -> ak.Array | None:
   2464     # NOTE: fn is assumed to be a *packed* function
   2465     #       as defined up in map_partitions. be careful!
   2466     try:
-> 2467         meta = fn(*to_meta(deps))
   2468         return meta
   2469     except Exception as err:
   2470         # if compute-unknown-meta is False then we don't care about
   2471         # this failure and we return None.

File /depot/cms/top/fstrug/conda_environments/TOPCoffea/lib/python3.10/site-packages/dask_awkward/lib/core.py:1903, in ArgsKwargsPackedFunction.__call__(self, *args_deps_expanded)
   1901     len_args += n_args
   1902 kwargs = self.kwarg_repacker(args_deps_expanded[len_args:])[0]
-> 1903 return self.fn(*args, **kwargs)

File /depot/cms/top/fstrug/conda_environments/TOPCoffea/lib/python3.10/site-packages/awkward/_dispatch.py:38, in named_high_level_function.<locals>.dispatch(*args, **kwargs)
     35 @wraps(func)
     36 def dispatch(*args, **kwargs):
     37     # NOTE: this decorator assumes that the operation is exposed under `ak.`
---> 38     with OperationErrorContext(name, args, kwargs):
     39         gen_or_result = func(*args, **kwargs)
     40         if isgenerator(gen_or_result):

File /depot/cms/top/fstrug/conda_environments/TOPCoffea/lib/python3.10/site-packages/awkward/_errors.py:85, in ErrorContext.__exit__(self, exception_type, exception_value, traceback)
     78 try:
     79     # Handle caught exception
     80     if (
     81         exception_type is not None
     82         and issubclass(exception_type, Exception)
     83         and self.primary() is self
     84     ):
---> 85         self.handle_exception(exception_type, exception_value)
     86 finally:
     87     # Step out of the way so that another ErrorContext can become primary.
     88     if self.primary() is self:

File /depot/cms/top/fstrug/conda_environments/TOPCoffea/lib/python3.10/site-packages/awkward/_errors.py:95, in ErrorContext.handle_exception(self, cls, exception)
     93     self.decorate_exception(cls, exception)
     94 else:
---> 95     raise self.decorate_exception(cls, exception)

TypeError: nan_to_num() got an unexpected keyword argument 'output_partitions'

This error occurred while calling

    ak.nan_to_num(
        <Array-typetracer [...] type='## * float64'>
        output_partitions = 1
        copy = True
        nan = 0.0
        posinf = None
        neginf = None
        highlevel = True
        behavior = None
    )


QuerySet does not appear to trigger change detection properly

Bevy version
0.5
What you did
From a user's system on Discord:

pub fn check_tasks(
    mut commands: Commands,
    sim_params: Res<SimParams>,
    physical_object_query: Query<&PhysicalObject>,
    physical_object_id_query: Query<Entity, With<PhysicalObject>>,
    mut resource_carrier_query: Query<&mut ResourceCarrier>,
    mut resource_storage_query: Query<(&mut ResourceStorage, Entity)>,
    mut tasks_queries: QuerySet<(Query<&mut TaskQue>, Query<Entity, Changed<TaskQue>>)>,
    mut ev_check_task_event: EventReader<CheckTaskEvent>,
) {
    let creature_ids: Vec<Entity> = tasks_queries
        .q1()
        .iter()
        .chain(
            ev_check_task_event
                .iter()
                .map(|CheckTaskEvent(creature_id)| *creature_id),
        )
        .collect();
    println!("Something was updated for {}", creature_ids.len());

    for creature_id in creature_ids {
        if let Ok(mut task_que_component) = tasks_queries.q0_mut().get_mut(creature_id) {
            if let Some(task) = task_que_component.0.front() {
                println!("Something was updated");
                if try_accomplish_task(
                    &mut commands,
                    &sim_params.world_rect,
                    &physical_object_query,
                    &physical_object_id_query,
                    &mut resource_carrier_query,
                    &mut resource_storage_query,
                    &creature_id,
                    &task,
                ) {
                    task_que_component.0.pop_front();
                    *task_que_component = TaskQue(task_que_component.0.clone());
                    println!("dereferenced");
                }
            }
        }
    }
}

TaskQue contains a standard VecDeque, and the method called is pop_front.
What you expected to happen
The TaskQue component should be marked as changed.
What actually happened
The TaskQue component is not marked as changed.
Additional information
This needs a minimal reproduction still.
        if let Ok(mut task_que_component) = tasks_queries.q0_mut().get_mut(creature_id) {

I think that QuerySet is the most likely candidate, but get_mut may be responsible instead.
Even if pop_front doesn't trigger change detection, setting an entirely new value certainly should.

What is a TaskQue?
Or more specifically, what are the definitions of it's methods?

Wait, what's the purpose of even using a QuerySet here?
Can't you just do
mut tasks_queries: Query<(Entity, &mut TaskQue), Changed<TaskQue>>,

I can't seem to reproduce this, even with the following test:
#[test]
    fn or_query_set_system() {
        // Regression test for issue #762
        fn query_system(
            mut ran: ResMut<bool>,
            set: QuerySet<(
                Query<(), Or<(Changed<A>, Changed<B>)>>,
                Query<(), Or<(Added<A>, Added<B>)>>,
            )>,
        ) {
            let changed = set.q0().iter().count();
            let added = set.q1().iter().count();

            assert_eq!(changed, 1);
            assert_eq!(added, 1);

            *ran = true;
        }

        let mut world = World::default();
        world.insert_resource(false);
        world.spawn().insert_bundle((A, B));

        run_system(&mut world, query_system.system());

        assert!(*world.get_resource::<bool>().unwrap(), "system ran");
    }

Is it possible that try_accomplish_task is returning false, which would then cause change detection to not be triggered?

I did a minimal repro and certainly saw the behavior but after adding a bunch of tracing found that change tracking was triggering as expected. However change tracking tracks "changed since last system run" which seems to be exclusive of changes made by the system itself.
is_changed() is roughly last_change_tick < change_tick which in the context of the above program you will just get last_change_tick == change_tick.

However change tracking tracks "changed since last system run" which seems to be exclusive of changes made by the system itself.

Correct: that's the intended behavior for it. See #2681, which would cause it to be inclusive of the changes made by the system itself. Does the user's code rely on that behavior? It's simple to add: we're just short on compelling reasons to do so.

Certainly, it is looking for Changed<TaskQue> to "trigger" based on the modification to Query<&mut TaskQue>. In fact this cyclical relationship appears to be the reason they are using a QuerySet.
As an alternative sending a CheckTaskEvent using an EventWriter should be mostly equivalent in this snippet. Of course there isn't enough here to know if that is an acceptable workaround.

Indeed, using CheckTaskEvent is how I ended up notifying about the update. It's just the Changed<> query felt more declarative, but more magical, it was hard to believe that all sorts of changes are detected as Changed. In the end imperative but simple solution won over declarative "gray" magic. That's for the explanation!


compilation fails with ./prepare.sh

http://imgur.com/It3tCxd
compliation fails due to lack of <ffi.h>
I'm not sure how to fix

very sorry, didn't see you could attach images

Nevermind, closing issue.
Solution:
[code] apt-get install python-cffi[/code]

#30 should fix this.
For now, you can just apt-get install libffi-dev.


Fix Cron Expression Parsing Issue with Invalid Characters

When parsing cron expressions, the system correctly accepts and processes valid expressions like 15 10 * * 1 (which runs at 10:15 AM every Monday). However, when encountering invalid cron expressions such as X Y 5 6 7, the system incorrectly displays the time as aN:0X AM. This is not the expected behavior, and the cron expression parser should handle invalid characters properly or reject them with an appropriate error message.

Please see the FAQ.  The cron is expected the be valid.


[ENH] Extend support to python 3.11

Python 3.11 was release some months ago (24 Oct 2022) and we should try to include it as supported version.
Waiting for:

https://github.com/numba/numba/issues/8304

One of the main hold-ups for this is numba, which seems to be close to supporting 3.11 (see https://github.com/numba/numba/issues/8304)

Nów waiting for the release


Add a share widget to the Project site

A share widget would be lovely as an addition to the site. Feel free to add your own creativity to add one. Expanding buttons with hover stylings are more preferred!

Can you please elaborate on it more?

@chehak123, for any issues/discussions, join the Anyxz Gitter channel & mention me over there!

Can I work on this issue?

@Kerveyfelix Sure! Go for it!

@Neilblaze Please review my pr #24. I have added 3 share widgets.

is the issue still open? I would like to work on it.

@Kerveyfelix is working on this @kumarshobhit . Feel free to grab any other issue :)

How about re-designing share button like this to occupy less space?

@KerinPithawala umm no, think of something different, something more compact.

How about adding the share buttons in footer?

We're talking about making the thing compact right? It won't make any change if we switch the position of those icons to the footer. Make it a shrinkable one, for example when a user hovers over the share icon, multiple options/buttons appear (maybe with a hover effect). Applying creativity is up to you :)

Okay. Will make a toggle like sharing widget


Add PMD Suppression for Music Plugin Changes

Resolve the unhelpful PMD errors breaking the build despite being a minor issue. This patch simply adds a SuppressWarning annotation to the two problematic methods which were altered within the net.runelite.client.plugins.music.MusicPlugin class in commit 56055a07d48f1da72c2ad1c906cba594432577f8. The suppression approach should be temporary, so a TODO comment was added to make it easier to find the annotations for a more permanent fix in the future.

We elected to revert the commit which caused the PMD errors in the first place, done here: c7956bfff3fcc470dba2276c36c5865f1992719f. We may yet remove the feature as today's update broke it in ways which are seemingly not so easily fixed, but if we do we'll do so in a way which doesn't break CI.


Tests: misk.web.ssl.PemSslClientServerTest.failsIfServerIsUntrusted() is flaky

This test is flaky and apparently only reproducible on CircleCI. We think the fix might be difficult and interesting.
See lines 78-88 in PemSslClientServerTest.kt.
This test failed recently with:
java.util.concurrent.TimeoutException: Timeout waiting for the services to stop. The following services have not stopped: {STOPPING=[JettyService [STOPPING]]}

this might've started happening after I bumped the jetty version

Tracking upstream: https://github.com/eclipse/jetty.project/issues/3929

Closing, this was resolved


Refactor modules through wrapper

Lot of common in md/asciidoc modules, wrapper to complete/implement inferable functions.
Related #17

#82


sentencepiece 0.2.0

A bunch of things have changed upstream, so we should be able to use abseil properly now (even though it symlinks to the third_party folder...). Also address #47 and use libprotobuf-lite (I didn't know -lite at the time and had thought it was another sentencepiece-ism).

This is running into https://github.com/google/sentencepiece/issues/978 on unix and needs https://github.com/conda-forge/libprotobuf-feedstock/pull/210 on windows.


node-red Error saving flows: EROFS:

Hello,
suddenly comes the Failure on deploying:
"Error: EROFS: read-only file system, open '/home/root/.node-red/flows_iot2000.json' at Error (native)"
before it has worked for weeks good
Tahnk you

Have you checked that the filesystem hasn't actually been remounted read-only due to corruption? It is normal for Linux to remount broken filesystems read-only to prevent further damage.
Also please check the contribution guide, questions should be asked on the Google Group or Slack as we like to keep issues for identified defects.

Wolfgang are you using the neopixel node? I have seen that it would bring the filesystem to a hold (readonly)

this Problem apears since I have nodes with Console :"mount -a" and "umount -a"
to copy files to USB-Stick
I have tried to solve the Problem with reinstall of Yocto-Linux-Image but the Failure is still coming by deploying even if I delete the exec-Nodes
the filesystem /.node-red has 755 permission actually, and it can't be changed
Thanks a lot

umount -a unmount ALL filesystems, this is not a good idea and probably should only be used as part of the system shutdown scripts.
This is not a Node-RED problem.

oh, thank you

Closing as it isn't a NR issue.
If you check out our contribution guidelines you'll see we prefer to handle this sort of thing on the mailing list, slack team or you could ask a question on Stack Overflow and tag it node-red. That way the whole Node-RED user community can help, rather than rely on the core development team.
Thanks!


src/server.js: Don't write to an ended stream.

If a stream has ended, it will be marked as not writable.
Writing to such a stream will throw an 'This socket has been
ended by the other party' error.

Can a test be added for this?