nvidia-smi: No running processes found

$ python neural_gpu_trainer.py --problem=bmul
...
modprobe: ERROR: could not insert 'nvidia_375_uvm': Invalid argument
E tensorflow/stream_executor/cuda/cuda_driver.cc:509] failed call to cuInit: CUDA_ERROR_UNKNOWN
What related GitHub issues or StackOverflow threads have you found by searching the web for your problem?
http://stackoverflow.com/questions/33970755/tensorflow-not-using-gpu
Environment info
Operating System:
Ubuntu 14.04 LTS
Installed version of CUDA and cuDNN:
(please attach the output of ls -l /path/to/cuda/lib/libcud*):
-rw-r--r-- 1 root root 179466 Mar 27 16:01 /usr/local/cuda/lib/libcudadevrt.a
lrwxrwxrwx 1 root root     16 Mar 27 16:01 /usr/local/cuda/lib/libcudart.so -> libcudart.so.7.0
lrwxrwxrwx 1 root root     19 Mar 27 16:01 /usr/local/cuda/lib/libcudart.so.7.0 -> libcudart.so.7.0.28
-rwxr-xr-x 1 root root 303052 Mar 27 16:01 /usr/local/cuda/lib/libcudart.so.7.0.28
-rw-r--r-- 1 root root 546514 Mar 27 16:01 /usr/local/cuda/lib/libcudart_static.a
If installed from binary pip package, provide:
$ conda install -c jjh_cio_testing tensorflow-gpu

A link to the pip package you installed:

The output from python -c "import tensorflow; print(tensorflow.__version__)".
ubuntu@ip-10-0-1-131:~/cuda/lib64$ python -c "import tensorflow; print(tensorflow.version)"
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.7.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.7.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.7.5 locally
1.0.1

If installed from source, provide

The commit hash (git rev-parse HEAD)
The output of bazel version

If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)
ubuntu@ip-10-0-1-131:~/models/neural_gpu$ python neural_gpu_trainer.py --problem=bmul
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.7.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.7.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.7.5 locally
Generating data for bmul.
cut 1.20 lr 0.100 iw 0.80 cr 0.30 nm 64 d0.1000 gn 4.00 layers 2 kw 3 h 4 kh 3 batch 32 noise 0.00
Creating model.
Creating backward pass for the model.
WARNING:tensorflow:Tried to colocate gpu0/gradients/gpu0/Gather_2_grad/Shape with an op target_embedding/read that had a different device: /device:GPU:0 vs /device:CPU:0. Ignoring colocation property.
Created model for gpu 0 in 6.46 s.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
modprobe: ERROR: could not insert 'nvidia_375_uvm': Invalid argument
E tensorflow/stream_executor/cuda/cuda_driver.cc:509] failed call to cuInit: CUDA_ERROR_UNKNOWN
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:158] retrieving CUDA diagnostic information for host: ip-10-0-1-131
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:165] hostname: ip-10-0-1-131
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] libcuda reported version is: 346.46.0
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:363] driver version file contents: """NVRM version: NVIDIA UNIX x86_64 Kernel Module  346.46  Tue Feb 17 17:56:08 PST 2015
GCC version:  gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3)
"""
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:193] kernel reported version is: 346.46.0
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:300] kernel version seems to match DSO: 346.46.0
Created model.  ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GRID K520           Off  | 0000:00:03.0     Off |                  N/A |
| N/A   37C    P0    35W / 125W |     10MiB /  4095MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+--------------------------------------------
Logs or other output that would be helpful
(If logs are large, please upload as attachment or provide link).

http://stackoverflow.com/questions/42984743/nvidia-smi-has-failed-because-it-couldnt-communicate-with-the-nvidia-driver

ubuntu@ip-10-0-1-131:~/cuda/lib64$ cd ~/NVIDIA_CUDA-7.0_Samples/1_Utilities/deviceQuery
ubuntu@ip-10-0-1-131:~/NVIDIA_CUDA-7.0_Samples/1_Utilities/deviceQuery$ ./deviceQuery
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
modprobe: ERROR: could not insert 'nvidia_375_uvm': Invalid argument
cudaGetDeviceCount returned 30
-> unknown error
Result = FAIL

ubuntu@ip-10-0-1-131:~/cuda/lib64$ cd ~/NVIDIA_CUDA-7.0_Samples/1_Utilities/deviceQuery
ubuntu@ip-10-0-1-131:~/NVIDIA_CUDA-7.0_Samples/1_Utilities/deviceQuery$ ./deviceQuery
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

modprobe: ERROR: could not insert 'nvidia_375_uvm': Invalid argument
cudaGetDeviceCount returned 30
-> unknown error
Result = FAIL

It seems you are using a mix of cuda 7 and cuda 7.5? Please try CUDA 8.

I tried CUDA 8.  I got errors.  Do you know if CUDA 8 works on g2.2xlarge instances with the NVIDIA GK104GL [GRID K520]?
$ lspci | grep -i nvidia
00:03.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K520] (rev a1)

On Mar 31, 2017, at 10:40 AM, Andrew Selle<EMAIL_ADDRESS>wrote:
It seems you are using a mix of cuda 7 and cuda 7.5? Please try CUDA 8.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/tensorflow/tensorflow/issues/8877#issuecomment-290778962, or mute the thread https://github.com/notifications/unsubscribe-auth/AC9i20n-sqbbFXCVQyJ-roJPglota2VFks5rrTqGgaJpZM4Mv8Hb.

CUDA 7.5 requires driver version 352.xx or newer but you seem to be using an older version, can you update the driver?

Yes.
This is what I installed.  Do you know the URI to the correct installer?
$sudo wget http://developer.download.nvidia.com/compute/cuda/7_0/Prod/local_installers/cuda_7.0.28_linux.run
$sudo chmod 700 ./cuda_7.0.28_linux.run
$sudo ./cuda_7.0.28_linux.run
$sudo update-initramfs -u
$sudo reboot

On Mar 31, 2017, at 10:52 AM, Jonathan J. Helmus<EMAIL_ADDRESS>wrote:
CUDA 7.5 requires driver version 352.xx or newer but you seem to be using an older version, can you update the driver?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/tensorflow/tensorflow/issues/8877#issuecomment-290782021, or mute the thread https://github.com/notifications/unsubscribe-auth/AC9i2503VfzlTfScnXMqC9cfXsyJRvRAks5rrT1lgaJpZM4Mv8Hb.

Ubuntu has a page on how to install the NVidia drivers, https://help.ubuntu.com/community/BinaryDriverHowto/Nvidia.  You can check the version using:
cat /proc/driver/nvidia/version

Does this mean the installation of NVIDIA-Linux-x86_64-367.57 failed?
ubuntu@ip-10-0-1-135:~/nvidia-graphics-drivers$ sudo ./NVIDIA-Linux-x86_64-367.57-no-compat32.run
Unable to load: nvidia-installer ncurses v6 user interface
Using: nvidia-installer ncurses user interface
-> Detected 8 CPUs online; setting concurrency level to 8.
-> License accepted.
-> Installing NVIDIA driver version 367.57.
-> Running distribution scripts
executing: '/usr/lib/nvidia/pre-install'...
-> done.
-> The distribution-provided pre-install script failed!  Are you sure you want to continue? (Answer: Continue installation)
-> Would you like to register the kernel module sources with DKMS? This will allow DKMS to automatically build a new module, if you install a different kernel later. (Answer: Yes)
-> Installing both new and classic TLS OpenGL libraries.
-> Installing both new and classic TLS 32bit OpenGL libraries.
-> Will install GLVND GLX client libraries.
Looking for install checker script at ./libglvnd_install_checker/check-libglvnd-install.sh
executing: '/bin/sh ./libglvnd_install_checker/check-libglvnd-install.sh'...
Checking for libglvnd installation.
Checking libGLdispatch...
Checking libGLdispatch dispatch table
Checking call through libGLdispatch
All OK
libGLdispatch is OK
Checking for libGLX
libGLX is OK
Checking entrypoint library libOpenGL.so.0
Checking call through libGLdispatch
Checking call through library libOpenGL.so.0
All OK
Entrypoint library libOpenGL.so.0 is OK
Checking entrypoint library libGL.so.1
Checking call through libGLdispatch
Checking call through library libGL.so.1
dlopen("libGL.so.1") failed: libGL.so.1: cannot open shared object file: No such file or directory
-> An incomplete installation of libglvnd was found. Do you want to install a full copy of libglvnd? This will overwrite any existing libglvnd libraries. (Answer: Install and overwrite existing files)
Will install libglvnd libraries.
-> Searching for conflicting files:
-> done.
-> Installing 'NVIDIA Accelerated Graphics Driver for Linux-x86_64' (367.57):
executing: '/sbin/ldconfig'...
/sbin/ldconfig.real: /usr/local/lib/libcudnn.so.6.5 is not a symbolic link
-> done.
-> Driver file installation is complete.
-> Installing DKMS kernel module:
-> done.
ERROR: Unable to load the 'nvidia-drm' kernel module.
ERROR: Installation has failed.  Please see the file '/var/log/nvidia-installer.log' for details.  You may find suggestions on fixing installation problems in the README available on the Linux driver download page at www.nvidia.com.
~
ubuntu@ip-10-0-1-135:~/nvidia-graphics-drivers$ ls -l /usr/local/lib/dnn
-rwxr-xr-x 1 root root 11172416 May  7  2015 /usr/local/lib/libcudnn.so
-rwxr-xr-x 1 root root 11172416 May  7  2015 /usr/local/lib/libcudnn.so.6.5
-rwxr-xr-x 1 root root 11172416 May  7  2015 /usr/local/lib/libcudnn.so.6.5.48
-rw-r--r-- 1 root root 11623922 May  7  2015 /usr/local/lib/libcudnn_static.a
ubuntu@ip-10-0-1-135:~/nvidia-graphics-drivers$ cat /proc/driver/nvidia/version
cat: /proc/driver/nvidia/version: No such file or directory

On Mar 31, 2017, at 11:02 AM, Jonathan J. Helmus<EMAIL_ADDRESS>wrote:
cat /proc/driver/nvidia/version

I successfully installed the NVIDIA 367.57 driver and now 'nvidia-smi' is working:
ubuntu@ip-10-0-1-70:~/NVIDIA_CUDA-7.0_Samples/1_Utilities/deviceQuery$ nvidia-smi
Sat Apr  1 17:30:19 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.57                 Driver Version: 367.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GRID K520           Off  | 0000:00:03.0     Off |                  N/A |
| N/A   41C    P0    77W / 125W |   3800MiB /  4036MiB |     92%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0     19821    C   python                                        3798MiB |
+-----------------------------------------------------------------------------+
ubuntu@ip-10-0-1-70:~/NVIDIA_CUDA-7.0_Samples/1_Utilities/deviceQuery$


FontIterator name is not good

Consider the following code sample:
const fontManager = navigator.fonts;
const fontIterator = await fontManager.query();
const fontIterator2 = fontIterator[Symbol.asyncIterator]();

const { value, done } = await fontIterator2.next();

In JavaScript, "iterator" has a very specific meaning, which is an object with a next() method which returns a { value, done } tuple. (Or a promise for such a tuple, in the case of async iterators.)
That is, per the JavaScript spec and ecosystem's definitions, fontIterator2 is an iterator. fontIterator is not an iterator; it is an iterable.
So the interface would be better named FontIteratable or Fonts or something like that. (Although IMO it could probably be eliminated entirely... I'll file another issue for that.)

I'm quite confused about iterables vs iterators.
As implemented, navigator.fonts.query() returns an iterable object.
const iterable = navigator.fonts.query()
> {Symbol(Symbol.asyncIterator): ƒ}

However, when creating an interator out of it yields the iterator as defined, i.e. FontIterator:
const iterator = iterable[Symbol.asyncIterator]()
> FontIterator {}

This seems to have the next() method, which returns a promise returning the { value, done } tuple.
As such, the interface name seems appropriate, but fontManager returns an iterable.
Does the comment still stand RE: FontIterator?

Ah, I see. I got confused because the spec and Chromium don't match each other. So yes, it does sound like in Chromium the FontIterator object is an iterator, not an iterable.
To fix the spec/implementation mismatch, I see a few steps:

Decide on the API you want programmers to use. This is partially what #33 is about.

Specify that using Web IDL + proper prose. This is something myself and @jakearchibald can help with.

Implement that in Chromium. This will be a bit trickier than usual because Chromium's bindings code does not have proper async iterator support, so you'll need hacks like an FontIterator interface with a next() method, which will not exist in the spec. (In spec-land, Web IDL will automatically generate your iterators whenever you use async iterable<>.)


accessibility for web app

Is there a nice way to make the web app accessible??? What did Kodu do?

when cursor changes, read the tile
menus
short cuts?

@tballmsft use accessibility.setLiveRegion to force the screen reader to read a piece of text.

The work is done on the web app side, the editor needs to call this api.


in aggregation query how I can use collation options?

Laravel-mongodb Version: 3.8
PHP Version: ^7.3
Database Driver & Version: MongoDB 6.0.8

Description:
How I can use collation option in aggregation query? In aggregation query options not working below is example:
$results = (new KeywordData)
->options([
    'collation' => [
        'locale' => 'en',
        'strength' => 2
    ]
])
->raw(function ($collection) use($keywords) {
    $query = [
        [
            '$match' => [
                '$text' => [
                    '$search' => $keyword,
                ],
            ]
        ]
    ];
    return $collection->aggregate($query);
});

Hello, the function Collection::aggregate accepts an array of options as 2nd parameter.
$results = (new KeywordData)
->raw(function ($collection) use($keywords) {
    $query = [
        [
            '$match' => [
                '$text' => [
                    '$search' => $keyword,
                ],
            ]
        ]
    ];

    $options = [
        'collation' => [
            'locale' => 'en',
            'strength' => 2
        ]
    ];

    return $collection->aggregate($query, $options);
});


feat: add pubsub batch publishing sample

Closes #978.
cc @hongalex
This is my first time working with the samples, so please forgive any nonsense contained below.

Looks good, but I'll leave David to comment on PHP specific things I might have missed.


ifndef indentation issue

tabbed conditionals seem to be unsupported in make (or at least the make version i have locally). when using tabs to indent the ifndef for AKSCONFIG i get the following error.
AKSCONFIG=abc make sv-cluster

Makefile:41: *** "Must set AKSCONFIG". Stop

in order to not start an argument about "tabs vs spaces" and about "how many spaces", i removed the tabs. the makefile syntax seems to be fine with it.

What this PR does
Before this PR:
After this PR:
Jira:
Link to demo recording:
Special notes for your reviewer

Checklist
This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

[ ] PR: The PR description is expressive enough and will help future contributors
[ ] Code: Write code that humans can understand and Keep it simple
[ ] Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
[ ] Upgrade: Impact of this change on upgrade flows was considered and addressed if required
[ ] Deployment: The deployment process was considered and addressed if required
[ ] Testing: New code requires new unit tests.
[ ] Documentation: Is the documentation updated? Either in the doc located in focus area, in the README or in the code itself.
[ ] Customers: Is this change affecting customers? Is the release plan considered?

Ah trying to fix this in #112 - it's not intended to be all the way to the left


Actix Websocket Server

Do an example actix websocket server streaming something into the system and streaming something out.

Decided cba to do both actix and axum so gonna stick with the devil I know


Build for Python 3.8 on Ubuntu 18.04

Description
I need to build pycoral and tflite for for Python 3.8 on Ubuntu 18.04.
My application runs on a Jetson Nano (which only supports Ubuntu 18.04) and it requires Python 3.8.
If i install the pre-built wheels for Python 3.8 i get a dependency issue with GLIBC 2.29, which is not available in Ubuntu 18.04.
When i try to build this repo from source, and specify that i want to build for Python 3.8 it defaults to Ubuntu 20.04 which is a no go for me.
I edited the scripts/build.sh to only build for python 3.8, and changed python 3.8 to build inside the ubuntu:18.04 image, but the build only outputs wheels for python 3.6. (eg tflite_runtime-2.5.0.post1-cp36-cp36m-linux_aarch64.whl)
Why do the scripts not respect my chosen python version?
Click to expand!
Issue Type
Build/Install
Operating System
Ubuntu
Coral Device
USB Accelerator
Other Devices
No response
Programming Language
Python 3.8
Relevant Log Output
No response

Hello @roflcoopter can you please check whether your default python version is 3.6 or 3.8.
https://github.com/google-coral/pycoral/blob/master/Makefile#L17

Thanks for your response!
Well, the script build.sh builds inside a docker container so my hosts python version wouldn't matter right?
I modified the script to launch an ubuntu:18.04 image as base but that produces python 3.6 binaries.
Tried modifying the Dockerfile as well to include python 3.8 but that results in dependency issues during build.
Is there a way to build without using docker?

We don't have any steps to build with out docker.
I think, Docker image ubuntu:18.04 itself has python 3.6. Can you try to build the ubuntu image with python 3.8 and pass that image here: https://github.com/google-coral/pycoral/blob/master/scripts/build.sh#L44

Managed to do it!
I modified scripts/build.sh to launch an ubuntu:18.04 image when Python 3.8 was selected.
Then i modified the libcoral/docker/Dockerfile and added these lines to install Python 3.8 and all required libraries
RUN add-apt-repository ppa:deadsnakes/ppa
RUN DEBIAN_FRONTEND=noninteractive apt install -y \
  python3.8 \
  python3-pip \
  python3.8-dev \
  python3.8-full \
  libpython3.8-dev \
  libpython3.8-dev:armhf \
  libpython3.8-dev:arm64
RUN python3.8 -m pip install cython numpy

ENV CPLUS_INCLUDE_PATH="$CPLUS_INCLUDE_PATH:/usr/include/python3.8/:/usr/include/aarch64-linux-gnu/python3.8/"

And lastly i modified https://github.com/google-coral/pycoral/blob/master/Makefile#L15 to:
PYTHON ?= $(shell which python3.8)


changes to allow for multi-uav support

Modifications to allow simulating multi-UAVs, the following modifications were made:

Mavlink class can now take the mavlink_fcu_url as a ros parameter
ardrone and iris launch files expect this argument to be provided, otherwise they will default to: udp://localhost:14565@localhost:14560
A new multi_uav.launch file was added to demonstrate this capability

@TarekTaha This is a great feature. However, I noticed that this requires rotors_simulator package. After I built rotors_simulator from source, and run roslaunch px4 multi_uav.launch, I get the following error:
while processing /home/mzahana/ros_ws/src/rotors_simulator/rotors_gazebo/launch/spawn_iris.launch:
Invalid roslaunch XML syntax: [Errno 2] No such file or directory: u'/home/mzahana/ros_ws/src/rotors_simulator/rotors_gazebo/launch/spawn_iris.launch'
The traceback for the exception was written to the log file

So, I looked into the rotors_gazebo/launch/ folder, but I did not find the spawn_iris.launch file.
Also, if you check the rotors_gazebo repository , you won't find  spawn_ardrone.launch or spawn_iris.launch
So, I guess that is why it does not work.
Does the launch files of the models have to be from rotors_gazebo? I guess it is better to default to models inside the px4 package to make sure all is self-contained, no?
Thanks.

@mzahana This multi-uav support does not work anymore due to various changes in the Firmware.
In preparation for MBZIRC, we decided to shift to sitl_gazebo and we had to fiddle a bit with the settings in order to enable multi-uav support.
Please check this repository : https://github.com/kuri-kustar/kuri_mbzirc_sim and in particular : https://github.com/kuri-kustar/kuri_mbzirc_sim/blob/master/launch/mavros_posix_sitl_px4_ch3.launch https://github.com/kuri-kustar/kuri_mbzirc_sim/tree/master/config https://github.com/kuri-kustar/kuri_mbzirc_sim/tree/master/urdf/iris
Basically, we had to modify the way the Firmware talks to the sitl_gazebo, and modify the ports used to talk to each drone.
Let me know if you need any help.

@TarekTaha  Thanks for the explanation. I think I see the complete picture now and I know what to modify/setup.
Basically,

each vehicle has a seperate sdf model that includes a unique mavlink udp port for gazebo to communicate with px4 app
startup file rsC for each vehicle, where a mavlink instance is srarted at the same port in the previous step. Another mavlink instance can be started on a unique port to communicate with mavros
a launch file that runs gazebo with a world model, loads each vehicle (spawn),  runs px4 app with corresponding rsC startup file for each vehicle, runs mavros for each vehicle.

I will do some testing and issue a PR. I will also do a documentation for the steps as there are no clear documentation for that feature.
Thanks and let me know if you have comments.

@mzahana correct. It's not the most elegant or general solution, but it works. It would be great if you can generalize the solution so that all the ports are picked up from the launch file or yaml file, and spawns multiple instances of the UAV model with the new port.
Just one thing to take care of, we had to play with a lot of port numbers to find ones that work.

I created PR #7235 which tries to generalize the approach as much as possible. Noe need for multiple SDF for each vehicle.
This PR https://github.com/PX4/sitl_gazebo/pull/101 is required too.
Feedback is appreciated.


Edited README

Made some grammar/spelling changes, and added a link at the end for how to integrate lumina into an iOS application.

LGTM!


between and unlessBetween do not update the expression property

I've just realized that the methods between() and unlessBeteen() do not store the value in the schedule's expression property. I guess this is because the artisan schedule:run command is executed every minute so we only need to evaluate if we are between that time interval (calling the method inTimeInterval()). Still is there any other reason for not storing the "hours" value in the schedule's expression property?
https://github.com/laravel/framework/blob/23b8d992ea1f35b346a62024ebeb4af6969b161a/src/Illuminate/Console/Scheduling/ManagesFrequencies.php#L29-L32
$schedule
    ->command('inspire')
    ->cron('*,30 3-7 * * 1,2');

dd($schedule->events()[0]->getExpression());
// artisan schedule:run -> "*,30 3-7 * * 1,2" (correct)

$schedule
    ->command('inspire')
    ->weekly()
    ->days(1,2)
    ->between('3:00', '7:00')
    ->everyThirtyMinutes();

dd($schedule->events()[0]->getExpression());
// artisan schedule:run -> "0,30 0 * * 1,2 *" (incorrect)

The reason behind this question is that I was thinking about the idea of implementing an artisan command to list the scheduled tasks and their cron expression in the terminal, something like schedule:list: What do you think @themsaid
+-------------------+-----------------+-----------------+----------------+---------------------+
| Expression        | Command         | Prevent Overlap | On Maintenance | Task Output         |
+-------------------+-----------------+-----------------+----------------+---------------------+
| *,30 3-7 * * 1,2  | database:backup | true            | false          | app/logs/backup.log |
| 0 3,15 * * * *    | Inspire         | false           | false          |                     |
+-------------------+-----------------+-----------------+----------------+---------------------+

This is intended. The "between" is an adjutant to the expression - not a replacement.
It would be seriously difficult to modify an existing expression and magically add the expected times into it. There are too many variables.
The options are to either hard code your cron expression with the times, or use the dynamic properties of the scheduler.

Thanks @laurencei for your reply. I agree with you about modifying an existing expression, but what about overwriting the "hours" value?
I mean, in the example above, let's say we have ->between('3:00', '7:00') that adds '3-7' to the expression, and later on  we call ->hourlyAt(17). The final results could just be '17' overwriting the value... The days work in the same way right now, you can't concatenate two days ->mondays()->turesdays() = '2' (instead of '1,2').
My point here is to find a way to "store" the full expression of the task so we can output its value if we want to build a table, or read the value for whatever reason...
Thanks again @laurencei

The problem is that works for hours - but not minutes.
There is no way in the expression to store ->between('3:30', '7:30')

@laurencei point is a concern, there's no way we can consider minutes that's why it's just used as an extra condition not included in the actual cron expression. I suggest that you just ignore this edge case and show the cron expression as is, it'll be hard to list extra conditions on the event in the table so you may consider this case of them.

I've updated the title to reflect the discussion. As a reference, here's a sample output of schedule:list implemented in the PR. Opinions needed.

Well since there are no plans to change this at the moment I'm going to close the issue, but please feel free to open a PR with your proposed change or use https://github.com/laravel/internals to discuss a change proposal.
Thanks :)


feat(db) query semaphore for Postgres connector

Summary
This commit introduces an optional semaphore locking mechanism within the Postgres connector :query() method. This provides a solution to the longstanding problem of OpenResty not providing a ceiling on the number of connections that can be established to a remote TCP host. As Postgres is sensitive about the number of connections opened to a given server, this provides a workaround solution until a native connection cap implementation is made available for OpenResty.
This commit introduce two new config options, 'pg_semaphore_max' and 'pg_semaphore_timeout'. The former defines the number of resources allocated to a ngx.semaphore instance that is bound to a Postgres connector object. This effectively defines the maximum number of concurrent queries to a Postgres server that can be issued from a given worker process (the more natural number of maximum number of queries issued by a given Kong node can then be extrapolated by multiplying this value by the number of Kong worker processes - using the native ngx semaphore mechanism seems the more appropriate solution vs. creating a node-wide semaphore (via some messy shared dict approach) that provided a slightly more human-natural accounting).
The default 'pg_semaphore_max' value of 0 results in no semaphore object created in the connector, and no lock acquisition performed before executing a query. This leaves the existing behavior in place.
Full changelog

add semaphore object to postgres connector
post/wait on semaphore resources within :query()
add unit and integration tests for semaphore resource acquisition timeout behavior (returning nil, err)

Looks great! Thanks @p0pr0ck5!

OpenResty <IP_ADDRESS> ships with connection queuing for cosockets. Have you given any thoughts as to what those configuration properties will become if/once we switch our underlying implementation?

I had thought about just going with this approach for this patch series. That would require updating pgmoon and lua-cassandra to leverage that new API. I think ultimately it's a good long-term goal, but I'm not sure what the timeline would be to add that. Once upon a time we had a pgmoon fork IIRC- not sure we want to go down that route again?

Should we issue a warning when pg_semaphore_max > lua_socket_pool_size? There is no value afaict in configuring Kong as such, but the performance penalty will be visible.

There is a Kong-configurable value for this. However, I'm not sure issuing a warning is warranted or correct. I'm not sure what the performance penalty would be, apart from needing to open up new TCP connections. While there is overhead there, I don't know that I would call it a penalty, especially in the context of the queries that Kong runs (we have seen cluster_events inserts take several hundred milliseconds to execute on a loaded server - this is a pathological case and was solved in part by #4550, but the few-milliseconds overhead of opening a new TCP connection is not of the same magnitude). These are advanced options for operators that need to tune Kong, and issuing a warning might just cause more confusion? The way to silence that warning would be to adjust lua_socket_pool_size, which would affect all socket pools by default, not just the postgres driver pool.

That would require updating pgmoon and lua-cassandra to leverage that new API.

I see no reason why either of those projects would reject a patch for connection queuing support in OpenResty >= <IP_ADDRESS>! Our pgmoon fork has always been a temporary one, for at times it has been necessary to ship patches faster than they were merged upstream. We can go down the same road again if needed.

There is a Kong-configurable value for this.

I know, hence my comment. There is more overhead than a few ms when you take, e.g. low bandwidth networks or TLS-enabled database connections. A warning could also be a notice log. But I'm just as fine with not issuing any kind of log, especially when considering that the property affects all socket pools.

I see no reason why either of those projects would reject a patch for connection queuing support in OpenResty >= <IP_ADDRESS>!

Yes, I don't think that these projects shouldn't be patched to use this, only that the approach here is a first step. We can push this out and then keep the same pg_max_concurrent_queries config value, just with a different implementation, if/when pgmoon uses native queueing support.

@p0pr0ck5 Oh, something else: I think this should go to next. Mind rebasing your branch?

@thibaultcha done, thanks!

Leaving a comment here to express my thoughts on the discussion regarding error handling in case of semaphore resource acquisition failure.
500 vs 503
The current proxy implementation combined with this patch will make Kong return HTTP 500 errors when a semaphore resource cannot be acquired. The semantics of a 500 error (read: "an unexpected error") don't really apply to this particular failure:

Said failure is (at least theoretically) predictable: there exist a predetermined number of clients allows to connect concurrently to PostgreSQL. Trying to open more connections will produce a very much expected error.
Additionally, we should convey to the Kong clients that the request failed at this given point in time, but that said request may succeed if retried, hence why "HTTP 503".

At this point, I think that this point wasn't a disagreement topic in our 1-1 discussions @p0pr0ck5 and @hishamhm.
However, there are valid implementation concerns:

A "catch-all" if no_semaphore_resource then ngx.exit(503) in the DAO is bad separation of concerns.
Updating all proxy-path DAO error handling code to return 503 on this exception isn't really an option either.

DAO abstraction layer for the proxy path
This conversation has strengthened my opinion that there should exist an abstraction layer to use the DAO on the proxy path (or other parts of the codebase). There exist at least 3 behavorial differences when using the DAO in the proxy path:

Proxy, Admin API, and migrations/CLI have different consistency requirements for C* read queries (availability vs consistency tradeoffs)
Proxy and Admin API have different "global" options, such as options.nulls
Proxy/Admin API and other users of the DAO have different error handling policies (e.g. proxy handles "semaphore resource acquisition" by responding with HTTP 503)

Path forward for this PR
Let's merge this patch as-is and consider the above concerns outside of the scope of this particular PR. Those concerns are a pressing topic though, particularly because of the point stressed out in this comment (on returning 500 not being an ideal situation).

With the above said, I will pull this and merge it now!

Merged!


Error in GroudingDINO label

Hello,
It seems there is a minor issue in using GroundingDINO.  I get the following message: Perhaps, a simple check may solve the
issue.
Labeling People/thenewyorker_its-what-each-person-needs.jpg: 100%|█████████████████| 114/114 [00:31<00:00,  3.67it/s]
Traceback (most recent call last):
File "/media/csverma/M2Disk/Projects/CompVis/ObjectDetection/AutoDistill/GroundingDINO_labels.py", line 10, in
base_model.label(input_folder=folder_name, output_folder='mldata')
File "/media/csverma/M2Disk/Projects/CompVis/ObjectDetection/AutoDistill/autodistillenv/lib/python3.11/site-packages/autodistill/detection/detection_base_model.py", line 58, in label
split_data(output_folder)
File "/media/csverma/M2Disk/Projects/CompVis/ObjectDetection/AutoDistill/autodistillenv/lib/python3.11/site-packages/autodistill/helpers.py", line 66, in split_data
shutil.move(os.path.join(images_dir, file + ".jpg"), train_images_dir)
File<EMAIL_ADDRESS>line 823, in move
raise Error("Destination path '%s' already exists" % real_dst)
shutil.Error: Destination path 'mldata/train/images/images2.jpg' already exists

Hello there! Autodistill does not currently delete a directory if the one you have already specified contains images. We should add a check for this; this is on my radar and something I want to action on in the future.
For now, make sure the output_folder you specify is a directory that does not exist. Let me know if you have any questions!


No Data generated by report

I get this issue as well. I am using SimpleCov with RubyMine IDE.

and this what the coverage report looks like:

Really need some help

Need the actual command that is running the test - it's the first line of the output window, but is truncated. My guess is it's loading up your code before running the file in question.


exec: "/usr/local/bin/jenkins.sh": permission denied

That error occurs when trying tio run de container when building the image. Any idea? I'm a newbie with Docker.

@gcaracuel chmod 0755 *.sh


Adding @nothingmuch 's splitting options

Adding this spreadsheet of splitting options which @nothingmuch created

I like the concept, but should be rebased atop master

I was unable to rebase onto master as I am not using an ssh clone via command line; only using GitHub broswer interface at this stage (appears I don't have sufficient access rights to ssh). Have create a new pull request for the Google spreadsheet link to be added to the docs page. < https://github.com/chainsplit/chainsplit-www/pull/14 >
Sorry if this is not ideal, I am a bit limited in just using the browser.


Revise project level resource quota for storage

Since project level resource quota was introduced, there were some changes done to resource quota on k8s side. Revise the https://kubernetes.io/docs/concepts/policy/resource-quotas/#storage-resource-quota to see if/how we can enable resource quota for storage.

Currently we support:

requests.storage
persistentvolumeclaims

requests.ephemeral-storage is still in alpha on kubernetes side, so we'll wait for it to become official till we put it in.

Would it be possible to allow requests.storage per class at the project level as detailed here?
https://kubernetes.io/docs/concepts/policy/resource-quotas/#storage-resource-quota

Would it be possible to allow requests.storage per class at the project level as detailed here?
https://kubernetes.io/docs/concepts/policy/resource-quotas/#storage-resource-quota

Would it be possible to allow requests.storage per class at the project level as detailed here?
https://kubernetes.io/docs/concepts/policy/resource-quotas/#storage-resource-quota

@AntonSmolkov thanks for the reply!  We are currently using that, but I'd love to see being able to set requests.storage based upon the storage class:
.storageclass.storage.k8s.io/requests.storage
see:  https://kubernetes.io/docs/concepts/policy/resource-quotas/#storage-resource-quota
Example issue we run into:  we have 2 classes -- NFS (slow) and FlexOS (fast).  We'd like to be able to assign a project/namespaces X amount of NFS and X amount of FlexOS.  As it stands (unless I'm gaping on something) any class used counts towards one quota.


Scaledpsd

This new cone was just added in MOI v1.6. This allows simplifying the MOI wrapper of many SDP solver but it also allows passing scaled PSD cone data directly to the solver.

Codecov Report
Patch coverage: 72.31% and project coverage change: -1.24 :warning:

Comparison is base (50b0c6e) 82.22% compared to head (3853da5) 80.99%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #131      +/-   ##
==========================================
- Coverage   82.22%   80.99%   -1.24%
==========================================
  Files          37       39       +2
  Lines        2701     2799      +98
==========================================
+ Hits         2221     2267      +46
- Misses        480      532      +52

Impacted Files
Coverage Δ

src/cones/coneops_defaults.jl
15.55% <0.00%> (-7.18%)
:arrow_down:

src/equilibration.jl
97.95% <ø> (ø)

src/precompile.jl
0.00% <0.00%> (ø)

src/settings.jl
4.44% <0.00%> (-3.56%)
:arrow_down:

src/variables.jl
77.64% <ø> (+11.64%)
:arrow_up:

src/version.jl
28.57% <0.00%> (ø)

src/Clarabel.jl
55.55% <42.85%> (-44.45%)
:arrow_down:

src/utils/mathutils.jl
80.98% <50.00%> (+3.77%)
:arrow_up:

src/types.jl
85.71% <70.00%> (-0.65%)
:arrow_down:

src/MOI_wrapper/MOI_wrapper.jl
85.27% <91.66%> (+0.30%)
:arrow_up:

... and 16 more

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Thanks for this - scaled PSD cone through MOI is indeed much easier on our side.


[BugFix] bugfix_get_empty_tableNames_in_dropDb (#13797)

What type of PR is this：

[x] BugFix
[ ] Feature
[ ] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Which issues of this PR fixes ：
Problem:
This Problem was introduced by commit bdb4da1. This commit use Collections.unmodifiableSet instead of new HashSet to reduce the memory usage of fe. But the problem here is that Collections.unmodifiableSet is just return the view of nameToTable.keySet(). Before use the TableNames in recycleDatabase, nameToTable had been dropped and also the view of it would become empty, which cause this problem.
Solution:
We new the empty HashSet first in dropDb and add the table name one by one after remove the element in nameToTable.
Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] This pr will affect users' behaviors
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

[x] I have checked the version labels which the pr will be auto backported to target branch

[FE PR Coverage Check]
:disappointed: fail : 6 / 9 (66.67%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/server/LocalMetastore.java
3
6
50.00%
[482, 483, 2815]

:large_blue_circle:
com/starrocks/catalog/Database.java
3
3
100.00%
[]

run starrocks_admit_test

run starrocks_admit_test

run #39455

run 39455

run starrocks_fe_unittest

[FE PR Coverage Check]
:disappointed: fail : 6 / 9 (66.67%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/server/LocalMetastore.java
3
6
50.00%
[482, 483, 2815]

:large_blue_circle:
com/starrocks/catalog/Database.java
3
3
100.00%
[]

[FE PR Coverage Check]
:disappointed: fail : 1 / 4 (25.00%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/service/FrontendServiceImpl.java
0
2
00.00%
[260, 716]

:large_blue_circle:
com/starrocks/server/LocalMetastore.java
1
2
50.00%
[482]

run starrocks_admit_test

@Mergifyio backport branch-2.5


SPARKC-109: Fix column aliases being ignored when fetching RDDs of Cassa...

...ndraRow.

LGTM
BTW I wonder whether we should address native aliasing in CQL in a separate ticket. WDYT?

All tests passed on my local machine


"Disable Internet Explorer 11 as a standalone browser" step-by-step directions are incorrect

See Disable Internet Explorer 11 as a standalone browser
Step 3 states:
Go to Computer Configuration/Administrative Templates/Windows Components/Internet Explorer
But it SHOULD state:
Go to Computer Configuration/Policies/Administrative Templates/Windows Components/Internet Explorer
The rest of the document appears correct. Just wanted to let you know!

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: b557e6ee-bb46-7b9c-108b-c78829c6d8b4
Version Independent ID: dd39c398-eb44-ab64-69ee-d860147072df
Content: Disable Internet Explorer 11
Content Source: edgeenterprise/edge-ie-disable-ie11.md
Product: microsoft-edge
GitHub Login: @dan-wesley
Microsoft Alias: shisub

@shrout1 Thanks!  I'm not able to reproduce.  Currently with my device the path is Computer Configuration/Administrative Templates/Windows Components/Internet Explorer

I realize that I was looking at gpmc.msc not gpedit.msc. Would be nice if there was a way to call out the difference, but it's not the end of the world. Just trying to help the next guy!


Wishlist of next functions to build/migrate from react-use

New Features
We have been using react-use, but now got into trouble with typescript. So we're looking into migrating to something else. So basically we are using these from react-use which has not yet been implemented in react-hookz. So this is basically a wishlist from our side 😸

useInterval
usePreviousDistinct
useWindowSize
useHover
useCustomCompareEffect
useCopyToClipboard

What is the new or updated feature that you are suggesting?
Why should this feature be included?

useCustomCompareEffect already exists, btw - https://react-hookz.github.io/web/?path=/docs/lifecycle-usecustomcompareeffect--example

We made custom hooks for the ones that did not exist. Thank you

@bboydflo added hook that can help with that useIntervalEffect

@bboydflo added hook that can help with that useIntervalEffect

Thank you!


BufferingLimit and UseBuffering and (Video)SyncStreamMode properties support added

Summary: BufferingLimit and UseBuffering and VideoSyncStreamMode properties support added
Type: Feature
Test Plan: UT/ CT
Jira: RIALTO-639

Sanity tests for ['RPI4 IPSTB'] added.
https://jenkinsrdkm.cmf.code.rdkcentral.com/job/test-components-flow/60664/

Sanity tests for ['RPI4 IPSTB'] added.
https://jenkinsrdkm.cmf.code.rdkcentral.com/job/test-components-flow/60677/

build-brcm-refboard3 Console Log uploaded to Gist URL: https://gist.github.com/rdkcmf-jenkins/07147d7e6ebf30942bf96f56d63c7836

build-brcm-refboard3 Console Log uploaded to Gist URL: https://gist.github.com/rdkcmf-jenkins/b74eda3d3e338203ff43e5bdcae18964

build-brcm-refboard3 Console Log uploaded to Gist URL: https://gist.github.com/rdkcmf-jenkins/c06968024306cfeefef8ed003c7c6e22

build-brcm-refboard3 Console Log uploaded to Gist URL: https://gist.github.com/rdkcmf-jenkins/9edb4f805e2aec2635462f514d937d87

Sanity tests for ['RPI4 IPSTB'] added.
https://jenkinsrdkm.cmf.code.rdkcentral.com/job/test-components-flow/60871/

build-brcm-refboard3 Console Log uploaded to Gist URL: https://gist.github.com/rdkcmf-jenkins/46a2fe1b1db52893a828a8b482d5e37e


docs: translate to simplified Chinese

translate to simplified Chinese

Thanks, however we are currently leaving the translation to browser plugins


Observation dependent variables interfaces

The dependent variable filtering option is now implemented. @DominicDirkx  I have not created a similar dependent variable-based observation set splitter, because I could not really identify an immediate use case for this. But just let me know if you think otherwise :)

Cool, thanks! I don't really see (for now) much of a use of a set-splitter, let's leave it for now.
I see you commented out some of the test cases in unitTestObservationsProcessing.cpp, I assume to speed up your local compilation and run (?) Except for adding this back, is there anything that you want to add before we can merge this?

I see you commented out some of the test cases in unitTestObservationsProcessing.cpp
Yes, I realised that too, sorry! They will be back with the next commit

Except for adding this back, is there anything that you want to add before we can merge this?
nothing feature-wise, but I thought I spent some time adding some proper comments this afternoon, because that's really missing

For now, I only added comments to the new dependent variables stuff but I realised that the whole observations.h file could use a lot more comments in general (also for the new parser/filter/splitter functionalities). I don't think that it is necessary for this PR though(?) but something to keep in mind

Merged :)


Request : foreverPins

I have a request : always show the pins with values, even when user don't drag it. (poke @jkwiecien who have made a good work with temporaryPins)

can do by attribute
custom:temporaryPins = "false"

I haven't tested this, does it work?

custom:temporaryPins = "false" means that you never want display pins ? no ?
Already tried

I get it !
temporaryPins are canceled in createPins
wich is call in setRangePinsByValue
Same issue with formater that i fix here https://github.com/oli107/material-range-bar/pull/35


Use DLIB_CASSERT to avoid unused variable warning in release compilation

After updating to latest dlib I got a warning about unused variable, so I changed the DLIB_ASSERT to DLIB_CASSERT. I am guessing we always want to check the mode we're in.

Yeah, good call.


Tests on macOS timeout on 10.12+

TravisCI recently updated their default image for macOS, which causes the build to hang and time out.
The source of the timeout is coming from gps and it is most likely related to handling external VCS. My initial tests suggest either gps/source_manager_test.go or gps/vcs_source_test.go are responsible when unable to call a VCS. Could be that we're not using requireBins() everywhere we might need it.
I was able to recreate a similar timeout by removing hg from my Mac running 10.13.1 (although this can't be the direct cause in Travis, as the image has both hg and bzr).

So, after digging into today I have some insight (but not much).
Here are some of my attempts to understand this problem with Travis:

Added a call to requireBins() in gps/source_manager_test.go to try and fix my local, no hg timeout.
Still locks.
Decided to turn off parallelism locally and run the test with and without the new calls to requireBins().
It worked. Figured it was parallelism.
Disabled parallelism in Travis for all tests.
Problem still occurs.
Discovered the test causing the problem in Travis (with and without parallelism):
github.com/golang/dep/gps/manager_test.go:939
Disabled the test TestUnreachableSource
Passes
I ran the test TestUnreachableSource by itself to see if it was the cause:
Locks

So, in Travis, the issue is TestUnreachableSource locking. But locally, I have a completely separate (but similar) locking issue when hg isn't installed. Instead of TestUnreachableSource, the problem comes from TestSourceManager_InferConstraint.

Okay, I think I understand why my local command is hanging. It appears to be attempting to call:
git ls-remote<EMAIL_ADDRESS>in the hg test in TestSourceManager_InferConstraint. This would hang forever because of:
$ git ls-remote<EMAIL_ADDRESS>The authenticity of host 'bitbucket.org (<IP_ADDRESS>)' can't be established.
RSA key fingerprint is SHA256:zzXQOXSRBEiUtuE8AikJYKwbHaxvSc0ojez9YXaGp1A.
Are you sure you want to continue connecting (yes/no)?

On Travis its the same issue: https://travis-ci.org/arbourd/dep/jobs/311157141#L92
git ls-remote<EMAIL_ADDRESS>is being called, and I assume, hanging on SSH protocol stuff.

Looks like we missed a spot on our latest passthrough to turn off interactive prompts: https://github.com/golang/dep/pull/1357.

p.s. Nice sleuthing! 👍

The ssh authenticity warning cannot be suppressed, but<EMAIL_ADDRESS>should be added to known_hosts in Travis already. There might be an issue with their newer Mac VMs where they accidentally removed it.


Add ColorGains Setting

Worth noting this setting should disable AWB... Question is do I need to reflect this in the UI (likely)

ColourGains might need more code than I thought. Default values are as always min, max and default but they are a single float number. Picamera2 (or rather libcamera at the end) expects 2 floating point numbers for Red and Blue.
So a decision needs to be made. Do I collect the values for the ColourGains from the feed and use them as default. Zero the floating point numbers or not apply any ColourGains and allow the user to set them (basically exclude them from the default values)

yo that would be nice for people using night vision cameras, to correct some of the redness when using. The infrared, that makes its way into the camera makes the image quite a bit red. I would appreciate it if you could implement it.


Snapshot CRDs are referenced two charts

I've stumbled over a pretty wired issue: The snapshot-controller and the snapshot-validation-webhook chart both import the "external snapshot CRDs" within their Makefile. When declaring both charts as a dependency in Chart.yaml and run helm template an error is returned.
Steps to reproduce:

Create a Chart.yaml in an empty directory:

apiVersion: v2
name: snapshot-test
description: A Helm chart for Piraeus Operator
type: application
version: 0.1.0
appVersion: 0.1.0
dependencies:
  - name: "snapshot-controller"
    version: "1.9.1"
    repository: https://piraeus.io/helm-charts/
  - name: "snapshot-validation-webhook"
    version: "1.8.2"
    repository: https://piraeus.io/helm-charts/

Run helm dep up to pull the dependencies.
Generate the template and write them to an output dir:

$ helm template --include-crds --output-dir test .
wrote test/snapshot-test/charts/snapshot-controller/crds/groupsnapshot.storage.k8s.io_volumegroupsnapshotclasses.yaml
wrote test/snapshot-test/charts/snapshot-controller/crds/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
wrote test/snapshot-test/charts/snapshot-controller/crds/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
wrote test/snapshot-test/charts/snapshot-controller/crds/snapshot.storage.k8s.io_volumesnapshots.yaml
Error: open test/snapshot-test/charts/snapshot-validation-webhook/crds/groupsnapshot.storage.k8s.io_volumegroupsnapshotclasses.yaml: no such file or directory

I will also file a bug in the Helm Github repo, but from my tests I'm pretty sure this issue is caused by importing the same CRDs in both charts.
The question here is: Is this intended or should the CRDs only exist in the snapshot-controller chart?

Filed an issue in Helm repo as well: https://github.com/helm/helm/issues/12513

The issues has been fixed in the Helm 3.13.0 release.

As expected ArgoCD is complaining about the duplicate CRDs in the manifest.
So the initial question remains: Do both charts need to bundle the CRDs or can be they removed (at least in one of the charts)?

Is there a way to disable CRD rendering in ArgoCD?
If not, we could also switch to providing the CRDs as a default-enabled normal template, like cert-manager does it with installCRDs=true :thinking:

@WanzenBug yes, but in Argo CD it is an "all or nothing toggle": either install CRDs or not. And of course it makes sense that Argo CD installs the CRDs on the first deployment. (You don't want to do it manually, instead Argo CD should handle all for you, also updates, this is what GitOps is about!)
Actually the problem is not limited to CRDs. The problem is that the same resources are declared within the same Argo CD application twice.
Here is my "meta chart":
apiVersion: v2
name: linstor
description: A Helm chart for Piraeus Operator
type: application
version: 0.1.0
appVersion: 0.1.0
dependencies:
  - name: "piraeus"
    version: "1.10.7"
    repository: "file://../../external-charts/piraeus-operator/charts/piraeus"
  - name: "snapshot-controller"
    version: "1.9.1"
    repository: https://piraeus.io/helm-charts/
    condition: snapshot-controller.enabled
  - name: "snapshot-validation-webhook"
    version: "1.8.2"
    repository: https://piraeus.io/helm-charts/
    condition: snapshot-validation-webhook.enabled
  - name: "piraeus-ha-controller"
    version: "1.1.4"
    repository: "https://piraeus.io/helm-charts/"
    condition: piraeus-ha-controller.enabled

Argo CD accepts a resource only once. As long as the snapshot-controller and the snapshot-validation-webhook chart define the same resource (="(namespace +)? kind + name") , they can not be deployed together. As a workaround you could deploy e.g. the snapshot-validation-webhook separately, but technically it doesn't make much sense to deploy one without the other chart, right?
If both charts rely on the CRDs then it maybe makes more sense to merge both charts in a "2.0" version and add a toggle for the controller and the webhook to switch them on/off individually.
I'm happy to create a PR, just let me know which solution makes more sense to you.

Yeah, merging it makes sense. I'm still thinking it would also make sense to not use the Helm ./crd directory, as that has a lot of known limitations, and we don't actually need them to be installed first.


Lab3 - Add Letter frequency and sort letters functions

Fixes #544
Review of colleague's PR #339
Changes proposed in this PR:

Concurrent implementation Letter frequency letters(s string) map[rune]int

Standard Letter frequency implementation letterFreq(s string) map[rune]int

Function that returns a sorted slice of strings with elements "{key}:{val}"

Provide comprehensive tests and benchmark for standard and Concurrent Letter frequency functions

Could you also update your commit message to adhere to the "Recipe for great commit message" as outlined in the course slides: https://go-course.org/gocourse-2019-05-melbourne.slide#31
I let it slide on lab 2, but since you need to address comments on this commit, it's time to get that right. Thanks.

I've updated the commit message and add benchmark data showing the speed improvement.

The speed I expected when implementing this Concurrent version of Letter frequency was not achieved, the benchmarks show that it is about 4 times slower than the standard synchronous version. Which is fine by me. I learned so much more about the go language and goroutines than any of the labs so far. I would like to keep the implementation as-is, assuming it meets the code owners criteria.


Remove sudo for Blackfire config when it's not needed

name: 🐞 Bug Fix
about: Bugs about Blackfire integration
labels: bug

A Pull Request should be associated with an Issue.

If you're fixing a bug, adding a new feature or improving something please provide the details in Issues,
so that the development can be pointed in the intended direction.

Related issue: #247 #248

Further notes in Contribution Guidelines
Thank you for your contribution.

Description
This PR removes sudo when it's not needed for Blackfire configuration.

In case this PR introduced TypeScript/JavaScript code changes:

[ ] I have written test cases for the changes in this pull request
[ ] I have run npm run format before the commit.
[ ] I have run npm run lint before the commit.
[ ] I have run npm run release before the commit.
[ ] npm test returns with no unit test errors and all code covered.

In case this PR edits any scripts:

[x] I have checked the edited scripts for syntax.
[ ] I have tested the changes in an integration test (If yes, provide workflow link).

Thank you @shivammathur

I can confirm that it works :-)


TPCH - Why are we slow on Query 1?

DuckDB is substantially faster on Query 1, which is pretty much read_parquet, a bunch of elemwise operations, and then a groupby-aggregation.  I'm surprised (pleasantly) that there's more opportunity here.
This surprises me because I would have thought that we were mostly just bound by parquet reading.  Apparently not?  I'd be curious to see what a careful profiling reading would say.
This should be easy today with the data generation script in tests/tpch/generate.py and then just normal operation with LocalCluster.
cc @fjetter @phofl

Some comments with regards to the read_parquet speed:
I played around a little bit with repeated reads from s3 and the performance goes up quite significantly if you read the same dataset over and over again:
https://cloud.coiled.io/clusters/293998/information?account=dask-engineering&tab=Metrics
CPU also goes up when we get increased bandwidth, so this is one of the bottlenecks here. This is a more general observation, I don't think that it explains this specific difference.
Only looking at query 1 on these two clusters:
Dask:
https://cloud.coiled.io/clusters/294197/information?account=dask-benchmarks&tab=Metrics&computation=7ea8db5e-834c-4ba0-b636-ce61cfadfc7f&sinceMs=1697636488636&untilMs=1697636703866&worker+network_variation=Cluster+Total+Data
DuckDB:
https://cloud.coiled.io/clusters/294212/information?account=dask-benchmarks&tab=Metrics&sinceMs=1697637161975&untilMs=1697637321975&cluster+network_variation=Cluster+Total+Data
DuckDB ends up only receiving half the data that we query from s3

Maybe we should look at s3fs parameters?
Or are we using pyarrow readers in this case?
Worth mentioning to the arrow folks?  Joris?

Spark is closer to DuckDB than us, so that's definitely something we should look into.
I'll look into this


win: implement uv_try_write() for pipes (take 2: it was pr #888)

I've taken @bnoordhuis code from https://github.com/libuv/libuv/pull/888 and have modernized the tests and tried to fix the issues it had.

It seems the new tests are failing on build-cross-qemu-alpha. Should we block the PR on that?

Feedback addressed. @vtjnash PTAL. Thanks!

PR updated. Let me know if I got it right this time. Thanks

@bnoordhuis @santigimeno
I've been investigating this issue in Node and saw this PR that would potentially fix that issue.
According to the log, this PR was merged and reverted for some reason. Do you think this PR could be updated so that it can be merged to fix the related issue?

This was reverted because it was breaking node.js ipc tests, but I didn't have the time to investigate. Please, feel free to take this up. Thanks!

I had forgotten this existed. I surmised it should work this way (https://github.com/libuv/libuv/pull/4470), but also haven't had time to investigate


docker_sd: Deduplicate containers attached to multiple networks

Proposal
I have several containers attached to multiple Docker networks (e.g. frontend, backend). At the moment docker_sd creates a scraping target for each container on all networks. As some services are only attached to one network, it's not feasible to use relabelling to drop all containers in a certain network (or only keep those).
It would be nice if docker_sd could be configured to only use the first network, for example or find some other way to avoid duplicate targets.

This is interesting. Can you workaround the issue by taking multiple labels in input_labels and use regexes ? e.g. can you find which container is attached to which network based on the container name?
Are you using docker swarm?

Can you workaround the issue by taking multiple labels in input_labels and use regexes ? e.g. can you find which container is attached to which network based on the container name?

The container network name is stored in __meta_docker_network_name, so I tried setting a label prometheus.network on the container, but AFAIK there's no way to compare __meta_docker_network_name and the resulting __meta_docker_container_label_prometheus_network during relabeling, as the RE2 engine doesn't support back references.

Are you using docker swarm?

No I'm using docker_sd, but I guess this is also the case for docker_swarm_sd.

Indeed, you would need to hardcode them:
- source_labels: [__meta_docker_network_name,__meta_docker_container_label_prometheus_network]
  regex: frontend;frontend
  target_label: __tmp_docker_keep
  replacement: keep
- source_labels: [__meta_docker_network_name,__meta_docker_container_label_prometheus_network]
  regex: backend_backend
  target_label: __tmp_docker_keep
  replacement: keep
- source_labels: [__tmp_docker_keep]
  regex: keep
  action: keep

or, shorter
- source_labels: [__meta_docker_network_name,__meta_docker_container_label_prometheus_network]
  regex: `frontend;frontend|backend;backend`
  target_label: __tmp_docker_keep
  replacement: keep

The issue with the approach you are proposing is that I do not know what are the guarantees from Docker that the networks are ordered in fixed order between API calls, and how it behaves when you attach/detach to networks, then restarts the daemon, etc.

I'm not sure about the behavior of the Docker API either, it would just be nice to have some built-in way to avoid duplicate targets.

okay. I think we could have a networks array where we take a list of networks. For each container, we add its target on that network. If not network is matched, we dismiss the target. Does it sound like a solution?

Ok, I'm not sure I understand you correctly. Given the following container setup (and assuming all containers have Prometheus metric endpoints), how can we make sure the "api" container doesn't end up twice in the target list?

container: frontend
networks: frontend

container: api
networks: frontend, backend

container: database
networks: backend

We could implement/explore with an alternative mode, like we do for kubernetes, where one target has the addresses over all the networks.
I have tagged this issue as P3.

Currently, for each container we retrieve we loop through each of its networks and furthermore each of its ports, adding a target for each network+port combination.
https://github.com/prometheus/prometheus/blob/9e88c3bb4dd7b01456639df06df07053d5626445/discovery/moby/docker.go#L186-L205
I think we could just provide a configuration setting to break after the first network iteration, allowing target(s) to be added for only one network per container (there could be multiple ports).

If we break, we should order them. We should also have one label for each other network address.

If we break, we should order them.

Using what method?

We should also have one label for each network address (without port).

So when we break and only add a target for the first network, we should add extra labels to that target with the other network addresses (like __meta_docker_network_other_frontend=<IP_ADDRESS>)

I think Docker, Docker Swarm and Kubernetes SD should ideally behave the same way, as it boils down to "software running containers attached to (multiple) networks". Not saying, that the current Kubernetes implementation is the way to go, but if we come to a decision, it should apply to all three SD methods.

Kubernetes implementation is the way to go, but should be behind a flag if possible.

If we break, we should order them.

Using what method?

We should also have one label for each network address (without port).

So when we break and only add a target for the first network, we should add extra labels to that target with the other network addresses (like __meta_docker_network_other_frontend=<IP_ADDRESS>)

I think it's no need order multiple networks by default, just get the first network will be ok.
People can order the networks in docker-compose config file.
Kubernetes meet the same promblem when pod has multiple networks and it just shows the first network as default network.
The command look as below:
nsenter -n -t19032 -F -- ip -o -4 addr show dev eth0 scope global

For anyone curios, I solved the network deduplication using the keepequal action in the relabel configs. I also use an extra couple relabel configs to deduplicate the port (if you have a container that publishes multiple ports). I am also able to override the port when the default 80 is not the port I want. With these relabel configs I am able to monitor 13 containers using just a few simple labels on each docker swarm service and double checking the prometheus UI it is truly only picking up one target on the port and network I expect.
relabel configs with comments:
    relabel_configs:
      # Only keep containers that should be running.
      - source_labels: [__meta_dockerswarm_task_desired_state]
        regex: running
        action: keep
      # Only keep containers that have a `prometheus-port` label.
      - source_labels: [__meta_dockerswarm_service_label_prometheus_port]
        regex: .+
        action: keep
      # get the port from this target
      - source_labels: [__address__]
        regex: '[^\s]+:(\d+)'
        target_label: __tmp_found_port
      # keep only if this is the port we're looking for
      - source_labels: [__meta_dockerswarm_service_label_prometheus_port]
        target_label: __tmp_found_port
        action: keepequal
      # limit to one network
      - source_labels: [__meta_dockerswarm_service_label_prometheus_network]
        target_label: __meta_dockerswarm_network_name
        action: keepequal
      # replace address with prometheus-replacement-port if available
      - source_labels: [__address__, __meta_dockerswarm_service_label_prometheus_replace_port]
        separator: ':'
        regex: '([^:]+):[^:]+:(\d+)'
        replacement: "${1}:${2}"
        target_label: __address__
      # override address if an override is provided
      - source_labels: [__meta_dockerswarm_service_label_prometheus_address]
        regex: '([^\s]+)'
        target_label: __address__
      # override scheme if an override is provided
      - source_labels: [__meta_dockerswarm_service_label_prometheus_scheme]
        regex: '([^\s]+)'
        target_label: __scheme__

Hello from the bug scrub: please see the #10490 for latest status.


page media resize with a value from page header doesn't seem to work

Been trying to pass a page.header value to a page.media resize command and have unexpected results and wonder if it is a bug or there is a correct way of passing it
if I do this
{% for key, item in page.header.intro %}
  {{dump(evaluate_twig(page.media[item.image].resize(50,50))) }}
{% endfor %}

it creates a reduced size image of the correct dimensions  in the cache as expected
but if I do
{% for key, item in page.header.intro %}
  {{dump(evaluate_twig(page.media[item.image].resize(item.imagesize))) }}
{% endfor %}

all I get is the original image.
Is there a certain way I need to format / to pass the size from the page header imagesize ( I understand from an error it is expecting a int / string ) but cant figure why it isn't passing correctly. Is it I just need to evaluate / filter the string ?
---
title: intro
intro:
  - image: wedding-catering-nottingham.jpg
    offset: 'x=25 y=50'
    imagesize: 10,10

  - image: hot-buffet-catering.jpg
    offset: 'x=-25 y=0'
    imagesize: 150,150

  - image: wedding-catering-nottingham.jpg
    header: Cold Buffets
    text: something about disasters
    url:
---

I have tried alternatives like '150,150' and '150, 150'
Wonder if @mahagr will have the answer straight away, with you currently working on the media stuff.. Thank you

Function resize expects two parameters while you are passing in a string with value '150, 150'
Try something like:
{% set sizes = explode(',', item.imagesize) %}
{{ page.media[item.image].resize(sizes[0], sizes[1]) | raw }}

@pamtbaau .Thank you so much, now you have said it, makes sense. Ahhh I was thinking the function / api was expanding / exploding the string
Whilst thanking you, can I throw one more at you, could I use a page.header and send a command like imagesize: .resize(300,300)  or another full command and {{ page.media[item.image].item.imagesize | raw }} or is that going to cause me issues.
Many thanks again with that solution and the understanding behind it :)

@spamhater-grav, For the sake of other users seeking for answers it might be better to not burying new unrelated questions inside existing questions. The title then doesn't match the content.
Preferably you should start a new question here, or, even better, on your preferred forum.

Fair comment thank you @pamtbaau  will do, its a strong start point and will give me a start point . Solution / Explanation to initial question. Thank you.


Failed to parse html

Describe the bug
Vite is failing to build due to a failure to parse the index.html file.
Reproduction
Create a new vanilla JS project with yarn create vite repro --template vanilla.
Replace the content of the index.html with the following:
<!DOCTYPE html>

foo <=> baz;

Try to build the project or run the dev server and the following error appears:
yarn run v1.22.10
warning package.json: No license field
$ vite build
vite v2.6.14 building for production...
✓ 0 modules transformed.
[vite:build-html] Unable to parse {"file":"repro/index.html","line":5,"column":6}
3  |
4  |

5  |  foo <=> baz;
   |       ^
6  |

7  |
file: repro/index.html
error during build:
Error: Unable to parse {"file":"repro/index.html","line":5,"column":6}
3  |
4  |

5  |  foo <=> baz;
   |       ^
6  |

7  |
    at traverseHtml (repro/node_modules/vite/dist/node/chunks/dep-e0fe87f8.js:21240:15)
    at async Object.transform (repro/node_modules/vite/dist/node/chunks/dep-e0fe87f8.js:21305:17)
    at async ModuleLoader.addModuleSource (repro/node_modules/rollup/dist/shared/rollup.js:22152:30)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

System Info
System:
    OS: macOS 12.0.1
    CPU: (8) arm64 Apple M1
    Memory: 232.83 MB / 16.00 GB
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 17.0.1 - /opt/homebrew/bin/node
    Yarn: 1.22.10 - /opt/homebrew/bin/yarn
    npm: 8.1.0 - /opt/homebrew/bin/npm
  Browsers:
    Brave Browser: <IP_ADDRESS>
    Chrome: 95.0.4638.69
    Firefox: 92.0.1
    Safari: 15.1
  npmPackages:
    vite: ^2.6.4 => 2.6.14

Used Package Manager
yarn
Logs
No response
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to https://github.com/vuejs/vue-next instead.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

< or > is not valid as child element in HTML. You need to escape these, see. Maybe the error message here can be improved.

OK thanks. It was working in the browser and other similar tools so I assumed it would be OK. I'll escape the characters.

OK thanks. It was working in the browser and other similar tools so I assumed it would be OK. I'll escape the characters.

Browser tends to be forgiving on syntax error.

opening to possibly get bet error message


locale/ko: update some spacing and awkward phrases

There was some spacing problems like '일분' (oneminute) and little awkward pharses, so fixed them. tests included.

@kyungw00k what do you think?
@hibiyasleep you should rebase on latest develop. Check https://github.com/moment/moment/blob/develop/CONTRIBUTING.md for more info.

@hibiyasleep "지난주" doesn't have a problem because "지난주" is a word itself. Also, as far as we have "n개월" as sequences I think it's better to use "1개월" not "한 달". Finally I think "방금 전에" sounds more natural than "잠시 전에".

@leehankyeol :+1:

@leehankyeol '방금 전' is fine, but '방금 후' is weird.
// locale/ko.js:39
        relativeTime : {
            future : '%s 후',
            past : '%s 전',
            s : '방금',

@hibiyasleep it was a long time ago, but 1) did you fix all comments in https://github.com/moment/moment/pull/2177#issuecomment-96501182
2) you need to make the changes on top of latest develop and also properly update the tests (now it looks like you're adding a new test file). https://github.com/moment/moment/blob/develop/CONTRIBUTING.md

Closing for now. Please rebase on latest develop.


Check fly-forward setting of EKF for multicopters.

@priseborough What do we need to obey when using the mag offset estimation on multicopters in the test harness?

@Zefz this will be relevant for you: You should set ekf->inhibitMagStates to true if these conditions are met:

The airframe type is a multicopter
ekf->accNavMag is below some threshold - because only if the copter is accelerating the mag offset is observable.

It would be great if you would be interested to look into this, as its relevant for your use case.

@Zefz have you noticed this issue?

I can take a look into this. Seems like a simple enough fix.


TypeError: Cannot read property '_location' of null

Since version 3.22.0, probably because of fix https://github.com/getsentry/raven-js/pull/1180 I'm receiving the following error while running tests:
TypeError: Cannot read property '_location' of null
    at Window.get location [as location] (/home/circleci/repo/node_modules/jsdom/lib/jsdom/browser/Window.js:148:79)
    at Raven._getHttpData (/home/circleci/repo/node_modules/raven-js/src/raven.js:1873:17)
    at Raven._send (/home/circleci/repo/node_modules/raven-js/src/raven.js:1970:23)
    at Raven.captureMessage (/home/circleci/repo/node_modules/raven-js/src/raven.js:652:10)
    at _api2.default.studiesandme.getPatient.then (/home/circleci/repo/src/containers/UserData/index.js:32:31)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)

For running tests I'm using create-react-app as follows: react-scripts test --env=jsdom --runInBand. As I googled jsdom might have something to do with this issue (https://github.com/jsdom/jsdom/issues/1798).

Hello, thanks for reporting the issue.
We are in the process of working on the next major release of our SDK.
Because of that, we had to put working on the current version (except major or security bugs) on hold.
We'll try to get back to all the reports as soon as possible, so please be patient.
Thanks for your understanding,
Cheers!

Initial frame is coming from jsdom, not our SDK. We are very defensive about accessing global object
var _window =
  typeof window !== 'undefined'
    ? window
    : typeof global !== 'undefined' ? global : typeof self !== 'undefined' ? self : {};

The error originates here https://github.com/jsdom/jsdom/blob/master/lib/jsdom/browser/Window.js#L204-L206 and there's not much we can do about fixing it.


Add nbqa-yapf

Adds a new hook definition for yapf.
I'm a big fan of nbqa! It's truly an amazing tool. 💯

Awesome, thanks! As well as this, it would be good add a simple test, and to show a little example in the docs - do you have time to do these too? No worries if not, I can do it next week

I'm a big fan of nbqa! It's truly an amazing tool. 100

Very kind of you, mind if I include this as a README testimonial?

@all-contributors please add @bdice for ideas and code

Awesome, thanks! As well as this, it would be good add a simple test, and to show a little example in the docs - do you have time to do these too? No worries if not, I can do it next week

I'm taking a look at tests and docs now. For testing, I'm going to just copy the tests for black and adjust the output to whatever yapf should produce.

I'm a big fan of nbqa! It's truly an amazing tool. 100

Very kind of you, mind if I include this as a README testimonial?

😄 Sure! Maybe a more descriptive testimonial would be: "nbqa is a clean, easy to use, and effective tool for notebook code style. Formatting and readability makes a huge difference when rendering notebooks in a project's documentation!" Feel free to use either statement. 🥳

Awesome - yeah, I think that the test test_black_works_with_leading_comment would be a good one to copy, just make a short notebook in tests/data and check that nbqa yapf test/data/<your new notebook> --nbqa-diff looks as you'd expect it to

@MarcoGorelli Thanks again for the advice/help! I pushed changes to docs/tests. I copied the black tests, with a couple of exceptions:

The test for failing when provided invalid syntax seems to do nothing with yapf. I don't know if this is a bug or an expected behavior of yapf, but I just deleted the test.
The test for successive runs checks modification times. I couldn't get the modification times to change in the before/after on my system, so I altered the test for yapf to compare the raw file contents and ensure that the first pass modifies the notebook and the second pass does not.

Hey @bdice ,
thanks, much appreciated that you looked into this and made extensive tests. This actually uncovers an issue. If we use two newlines between cell separators, then yapf wants to remove one.
For example. if we have t.py as
# %%NBQA-CELL-SEP
import os

# %%NBQA-CELL-SEP
def hello_world():
    print('hello world')

hello_world()

# %%NBQA-CELL-SEP

then we will get
$ yapf t.py --diff
--- t.py	(original)
+++ t.py	(reformatted)
@@ -9,5 +9,4 @@

 hello_world()

-
 # %%NBQA-CELL-SEP

But, if instead we use single new lines as separators, e.g.
# %%NBQA-CELL-SEP
import os

# %%NBQA-CELL-SEP
def hello_world():
    print('hello world')

hello_world()

# %%NBQA-CELL-SEP

then we get
$ yapf t.py --diff
--- t.py	(original)
+++ t.py	(reformatted)
@@ -1,5 +1,6 @@
 # %%NBQA-CELL-SEP
 import os
+

 # %%NBQA-CELL-SEP
 def hello_world():

So...I'm a bit stuck about what to do here

@MarcoGorelli It looks like the statements hello_world() and # %%NBQA-CELL-SEP are being treated differently than def / class blocks. It looks like yapf prefers to leave at most one newline between statements like hello_world(), x = 5, or # My comment and two newlines between method/class definitions (and two newlines after the imports at the top).
Can you help me understand why this is an issue and what effect it has? Is it changing the number of trailing newlines in notebook cells in an unexpected way?

Sure - the issue is that nbqa converts a notebook to a temporary Python file, with two newlines and a special comment to indicate separation between cells. The issue is that like this we end up in a situation in which yapf wants to either add or remove some newlines, and so nbqa yapf tests/data/notebook_for_testing.ipynb --in-place will always return "mutation detected".
The only workaround I can think of it to apply yapf to each cell one-by-one - this'd be quite a departure from how other tools are treated though, but I can't think of other options. Will this about it anyway

Aha, I see. Perhaps there is an option in the yapf configuration that would stop this? Perhaps BLANK_LINES_BETWEEN_TOP_LEVEL_IMPORTS_AND_VARIABLES=2?

Great, yeah - if this is somewhat configurable then we could set that as the default in https://github.com/nbQA-dev/nbQA/blob/master/nbqa/config/default_config.toml , thanks!

@MarcoGorelli It seems like the option for configuring BLANK_LINES_BETWEEN_TOP_LEVEL_IMPORTS_AND_VARIABLES hasn't been released yet: https://github.com/google/yapf/issues/347#issuecomment-677006212
I'm not sure what to do about this, I will defer to your expertise if you have further ideas for how to handle it. Do we try the cell-by-cell approach or just add a warning to the docs section on yapf?

Hey @bdice - I think my preference would be to either wait for the next release, or try the cell-by-cell approach (I'm about to start a new job though so don't really have the resources for the latter at the moment - I can review a PR if you put one together, else I hope to be able to get back to this later)

Hey @bdice - looks like yapf 0.31.0 is out! So I think we can add this option and require this as the minimum version for yapf (similarly to how we require a minimum version for isort)

@MarcoGorelli Thanks! Unfortunately my guess was incorrect -- BLANK_LINES_BETWEEN_TOP_LEVEL_IMPORTS_AND_VARIABLES does not control that setting with blank lines around comments. I looked through the yapf settings and source, but I don't see how to change that behavior.
Likewise, I'm not sure that I have time to change to a "cell by cell" approach. I prefer using black and this PR is mostly intended to be a minor convenience for a project I contribute to that uses yapf (and to give something back to nbqa because the tool is wonderful).
In practice, the pre-commit hook I have set up here appears to work as desired, and it's fine with us to always mutate the notebooks on commit.
https://github.com/glotzerlab/hoomd-examples/blob/3402c93a1842e5818d86dbc39a2b2d0912c84326/.pre-commit-config.yaml#L59-L69

and to give something back to nbqa because the tool is wonderful

That's very kind of you :pray:
I'm also very short on time in this period (just started a new job) so let's close for now, we can always come back to this later

Reopening as I have an idea for how to get this through without adding too much complexity

Reopening as I have an idea for how to get this through without adding too much complexity

Nice! Let me know if you’d like a helping hand or review.

Sure, thanks! A review would be helpful if you have time :)
cc @girip11 @s-weigand if you have comments

I will review today

Codecov Report

Merging #547 (f8abdf7) into master (1c57a52) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##            master      #547   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           14        14
  Lines          781       797   +16
  Branches       115       117    +2
=========================================
+ Hits           781       797   +16

Impacted Files
Coverage Δ

nbqa/__main__.py
100.00% <100.00%> (ø)

nbqa/output_parser.py
100.00% <100.00%> (ø)

nbqa/replace_source.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1c57a52...f8abdf7. Read the comment docs.

merging for now as I'm pretty keen on making a new release before tomorrow, can always revert if it anyone has objections / causes bugs anywhere

@MarcoGorelli Thank you again for your guidance and help! 😄

@bdice I forgot to mention, if you wanted to mention yapf in the README (alongside other tools), that'd be welcome!


chore: Use slim image in Dockerfile

SUMMARY
Use slim image for python and nodeJS. Final slim python image will be 500 MB less.
Use slim image for modeJS no reduce downloading time.
Remove installing npm 7 because nodeJS 16 already has npm 8 which the same like npm 7
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

Thanks for the PR. I would advise splitting this PR in two:

First one changes from the full images to the slim ones
Second one bumps the main Python version from 3.8 to 3.9. In practice this would mean bumping the versions on CI so that 3.8 is replaced by 3.9 and 3.9 is replaced by 3.10.

Finally to get started with deprecating 3.8, we would need to notify the mailing list about the upcoming deprecation well in advance (3-6 months ahead of happening).

@villebro @zhaoyongjie
It's good practice to use an image as light as possible to reduce the number of vulnerabilities and downloading time.
Sure, after that, users need to install some system or third-part tools or manually build superset from full python image.

@betodealmeida @dpgaspar ^^

@EugeneTorap Thanks for a very useful change. Here is AWS ECR scan reports for comparison:
Before change:
Critical - 2
High - 19
Medium - 108
After Change:
Critical - 1
High - 6
Medium - 50
IMO, Its a big improvement. Bumping python to 3.9 should further reduce vulnerabilities as most of the remaining critical and high vulnerabilities are Linux related.


package com.unity3d.* does not exist

Hello, I am following the setup instructions within the readme. When I try to build the app I get the following error:
`
C:\src\flutter.pub-cache\hosted\pub.dartlang.org\flutter_unity_widget-0.1.6+5\android\src\main\java\com\rexraphael\flutterunitywidget\UnityView.java:9: error: pack
age com.unity3d.player does not exist
import com.unity3d.player.UnityPlayer;
^
C:\src\flutter.pub-cache\hosted\pub.dartlang.org\flutter_unity_widget-0.1.6+5\android\src\main\java\com\rexraphael\flutterunitywidget\UnityView.java:13: error: can
not find symbol
private UnityPlayer view;
^
symbol:   class UnityPlayer
location: class UnityView
C:\src\flutter.pub-cache\hosted\pub.dartlang.org\flutter_unity_widget-0.1.6+5\android\src\main\java\com\rexraphael\flutterunitywidget\UnityView.java:19: error: can
not find symbol
public void setUnityPlayer(UnityPlayer player) {
^
symbol:   class UnityPlayer
location: class UnityView
C:\src\flutter.pub-cache\hosted\pub.dartlang.org\flutter_unity_widget-0.1.6+5\android\src\main\java\com\rexraphael\flutterunitywidget\UnityUtils.java:10: error: pa
ckage com.unity3d.player does not exist
import com.unity3d.player.UnityPlayer;
^
C:\src\flutter.pub-cache\hosted\pub.dartlang.org\flutter_unity_widget-0.1.6+5\android\src\main\java\com\rexraphael\flutterunitywidget\UnityUtils.java:23: error: ca
nnot find symbol
private static UnityPlayer unityPlayer;
^
symbol:   class UnityPlayer
location: class UnityUtils
C:\src\flutter.pub-cache\hosted\pub.dartlang.org\flutter_unity_widget-0.1.6+5\android\src\main\java\com\rexraphael\flutterunitywidget\UnityUtils.java:30: error: ca
nnot find symbol
public static UnityPlayer getPlayer() {
^
symbol:   class UnityPlayer
location: class UnityUtils
C:\src\flutter.pub-cache\hosted\pub.dartlang.org\flutter_unity_widget-0.1.6+5\android\src\main\java\com\rexraphael\flutterunitywidget\UnityUtils.java:55: error: ca
nnot find symbol
unityPlayer = new UnityPlayer(isAR ? activity : activity.getApplicationContext());
^
symbol: class UnityPlayer
C:\src\flutter.pub-cache\hosted\pub.dartlang.org\flutter_unity_widget-0.1.6+5\android\src\main\java\com\rexraphael\flutterunitywidget\UnityUtils.java:79: error: ca
nnot find symbol
UnityPlayer.UnitySendMessage(gameObject, methodName, message);
^
symbol:   variable UnityPlayer
location: class UnityUtils
Note: C:\src\flutter.pub-cache\hosted\pub.dartlang.org\flutter_unity_widget-0.1.6+5\android\src\main\java\com\rexraphael\flutterunitywidget\FlutterUnityViewFactory
.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
8 errors
FAILURE: Build failed with an exception.

What went wrong:
Execution failed for task ':flutter_unity_widget:compileDebugJavaWithJavac'.

Compilation failed; see the compiler error output for details.

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

Get more help at https://help.gradle.org

BUILD FAILED in 4s
Running Gradle task 'assembleDebug'...
Running Gradle task 'assembleDebug'... Done                         4.9s
Gradle task assembleDebug failed with exit code 1
`

same error here anyone solved it yet


Custom numerical integrator for MPC predictions

Hello @nicolapiccinelli, I'd like to use your library, but I need to employ a custom numerical integrator to make the MPC predictions along the prediction horizon. Can you give me some info about the routine now used to do this? I see there is an RK4 class, but I didn't find anywhere its usage in the provided examples.

Hello, @sara9915. Are you referring to the integration of the model dynamics?

Yes, exactly

Ok, the library is handling things differently between linear and non-linear MPC. Which one are you interested in?

Non-linear MPC

If you already have a discrete-time model, the library is simply propagating the state x(k+1) = f(x(k),u(k)). Otherwise, if you provide a continuous time model, the integration is performed using the sample time's central difference.  In this case, the integration exploits the prediction horizon to implement the integration scheme.

I got it, thank you!