torchrec: Super slow allreduce in multi-node multi-gpu setting

I am using torchrec version. With a single node and 8 A100 GPUs the training speed is roughly 18 it/s; however, when using 2 nodes (each with 8 A100 GPUs again), the training speed reduces to about 3 it/s. The 2 nodes are connected through a 100Gb/s NIC.
Upon further investigation using the pytroch profiler, it turns out the slowness is coming from the all reducing during sparse_data_dist:
Using single node, the sparse_data_dist takes 28ms:

Using two nodes with TREE allreduce sparse_data_dist takes 246ms:

Using two nodes with RING allreduce sparse_data_dist takes 254ms::

I wonder if there are any best practices to optimize the allreduce time with torchrec? As an example knob, horovod allows to treat sparse tensors as dense tensors during all reduce, since the number of NCCL CC calls are less when allreducing sparse tensors.

The slowdown you report seems reasonable considering that 8 A100 GPUs on a single computer are connected via NVLink or the PCIe bus, and both have considerably more bandwidth than a 100Gb/s NIC which you used for 2 nodes. One note I would like to add is that the training speed is calculated using the formula: average it/s * local batch size * number of GPUs across all nodes. So your speed drop in percentage is not 3/18 but 6/18.
I think you may be confusing allreduce and Torchrec's KJTAllToAll. Allreduce would be used in the context of averaging gradients where there is model replication (data parallelism). It's not used for model parallel parts such as sharded embedding tables, because there aren't any replicated trainable parameters to need their gradients to be averaged and made the same on all replicas.
Torchrec uses a sharding planner to pick the sharding plan it considers best. Regarding of NCCL collective algorithms like Ring and Tree, the NCCL repo would be a better place to ask or read about the tradeoffs of different settings. I think prior issues may cover the topic, such as this issue.

Hi @samiwilf, thanks for the insights.
The reason I mentioned allreduce  is that when you zoom into the profile under sparse_data_dist name scope, there are a bunch of Allreduce calls:

Some Send/Recives can be seen as well, but I was wondering what are those allreduce calls?
Another question that I had regarding the sparse_data_dist was that based on the code, it seems that this part is the initial feature distribution among ranks for the next steps which is supposed to be pipelined. However, from the profiling results, it seems that it happens right between the forward pass and the backward pass of the current step which make things look sequential rather than pipeline:

Am I missing something here? I would appreciate any insight on how to map the profile to the pipeline code?

I would recommend performing an ablation study of the code and profiling iteratively to pinpoint which parts of the model or training loop correspond to the parts of the profile in question. You could try switching to TorchRec's nightly build, since sparse_data_dist has been replaced with start_sparse_data_dist and wait_sparse_data_dist. It may provide a more granular and informative profile. Lastly, although TorchRec is used for the embedding tables, the bottom and top MLPs are still ordinary PyTorch DDP modules. They use allreduce for gradient averaging.


Notifications Demo

re: 3357

Coverage increased (+0.05%) to 27.49% when pulling 61d85da7dea6b85185047a4f76d978d22ef94077 on rjstires:notifications-demo into b9687cff4c6c0306404fddee24aa482c8790f9e5 on linode:develop.


ssl.SSLZeroReturnError

The APNs service keep getting this error since yesterday, anyone knows what's the reason? Thanks

It is common behavior for the APNs service to close connections on their end at will.
Over the last 6 months, 2/3 of our push-attempts failed on first try.
That is why you should check for common SSL-errors like that and retry in such cases with a fresh connection.
After the retries, all the 2nd push-attempts went through for us.

It is common behavior for the APNs service to close connections on their end at will.
Over the last 6 months, 2/3 of our push-attempts failed on first try.
That is why you should check for common SSL-errors like that and retry in such cases with a fresh connection.
After the retries, all the 2nd push-attempts went through for us.

Thanks for your sharing. I only get this error from yesterday.

ssl.SSLZeroReturnError

ssl.SSLZeroReturnError is a subclass of ssl.SSLError, maybe that's the reason you haven't get this error yet.

I have add code to capture ssl.SSLError and retry, but still got ssl.SSLError from one of the service. I can confirm the error report from the second try for push notification.

Close the underlaying HTTP2 connection before the second try seems fixed the issue, no ssl.SSLError anymore.


API-425, API-416, API-426: Initial version

This version only supports .Net45.  We'll create a new ticket to add other individual platforms.

ooooo

@marchello2000 you may want to take a look too in case we want changes to copyrights, etc.

:+1:

@josephdwyer also, we want to make sure versioning in AssemblyInfo is handled automatically.

@marchello2000 wanted to make sure you had a chance to review my last commits before merging.

Thanks @kirkplangrid looks good to me! And @abhikp just confirmed, MIT license is GTG!

@marchello2000 last thing, what should the file header be for each file.  Is what we have right now OK?  Is:

Copyright (c) 2015 PlanGrid, Inc. All rights reserved.

Appropriate for the MIT license?  @josephdwyer pointed out that concern, and I'm curious too.

(i.e. what happens if someone does a PR and we contribute their code?  I Am Not a Lawyer, etc.

Ah, didn't notice/understand that. My impression (from my limited exposure to this in the past life) that this is correct attribution. But... I am not a lawyer :) So... I will check on that. In the mean time it's good since we don't expect anyone to contribute. Btw, the way the repo is setup they can't just push to master, they have to PR the changes, right? I am not sure how this works

@marchello2000 yeah, as a public repo, it's standard Github operating procedures.  Which means that we can do what we will, but non-plangrid fold will be forced to fork and submit PRs in the usual open-source fashion.

Gotcha, thanks!


[FEATURE] Aligning social icons horizontally for mobile screens.

Detailed description

Context
Want to work under GSSoC'23.
Possible implementation
No response
Additional information
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

assigned the issue @ShreyaDhiman24


tikv-ctl region-properties doesn't work on rawkv

Bug Report

What version of TiKV are you using?
master
What operating system and CPU are you using?

Not related
Steps to reproduce

For rawkv, call tikv-ctl region-properties on any region
What did you expect?
print useful info
What did happened?
print write cf info whose all field are 0 except min_ts

/type bug
/severity Minor


uninitialized constant Celluloid::Logger

celluloid 0.17.2
reel 0.5.0
require 'reel/rack/server'
uninitialized constant Celluloid::Logger
/Users/kostya/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/reel-rack-0.2.2/lib/reel/rack/server.rb:9:in `<class:Server>'
/Users/kostya/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/reel-rack-0.2.2/lib/reel/rack/server.rb:8:in `<module:Rack>'
/Users/kostya/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/reel-rack-0.2.2/lib/reel/rack/server.rb:7:in `<module:Reel>'
/Users/kostya/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/reel-rack-0.2.2/lib/reel/rack/server.rb:6:in `<top (required)>'

That symbol needs to be changed to Celluloid::Internals::Logger in the gem.

:+1:

Hitting this issue as well.

@pulkit4tech do you have a bit of time to update the constant to Celluloid::Internals::Logger and post a PR?
@tmm1 @jlyonsmith for now, you can do Celluloid::Logger = Celluloid::Internals::Logger and let us know if you run into anything else like that.

Sure @digitalextremist, Currently I am out of station, but will make a PR for it very soon :smiley:

can you release gem with this fix?


Java frontend

this fork of java2python will output Rusthon's syntax.
https://github.com/rusthon/java2python

the option --rusthon can now be given to j2py to output Rusthon syntax.
https://github.com/rusthon/java2python/commit/04d02c1593eafb82a55cc55eae8673e6b41cde07

This fork of j2c will repackage it as a command line tool.  j2c is a more direct way to translate to c++, it should be an option where java2python fails.
https://github.com/rusthon/j2c

testing translating the deeplearning4java framework in this fork.
https://github.com/rusthon/deeplearning4j

java2python using the old Antlr may not support new libraries like deeplearning4j, the hacks i put into java2python to get it to work with deeplearning4j may not be safe. also generates ugly Runtime._exec(hacks).
JavaLang
a better solution looks like JavaLang by Chris Thunes which parses java7 into an AST.
forked here:
https://github.com/rusthon/javalang

java in the markdown should be compiled by default using javac and make g++ link to jvm in the final exe.
xml in the markdown tagged with @giws will trigger wrappers to be generated with Giws (reverse Swig):
https://github.com/opencollab/giws

this fixes linking to libjvm.so https://github.com/rusthon/Rusthon/commit/37c6d1d9e5a744158ee3e5f4ad8605bb633fef76
but now there is this error when running the test:
g++ -O3 -fprofile-generate -march=native -mtune=native /tmp/rusthon-c++-build.cpp -o /tmp/rusthon-c++-bin -pthread -std=c++11 -ljvm -I/usr/lib/jvm/java-7-oracle/include -I/usr/lib/jvm/java-7-oracle/include/linux -L/usr/lib/jvm/java-7-oracle/jre/lib/amd64 -L/usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/
running: /tmp/rusthon-c++-bin
Could not get the Class mymodule/MyJavaClass
Exception in thread "main" java.lang.NoClassDefFoundError: mymodule/MyJavaClass
Caused by: java.lang.ClassNotFoundException: mymodule.MyJavaClass
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

hello world now works using giws.
https://github.com/rusthon/Rusthon/commit/36c0df7f83e2bd68922172cf6e68c3df992c5059
new syntax
this enables the JavaVM
import jvm

this creates a new Java Object
a = jvm(SomeJavaClass())

new fork of Giws to support properties being worked on here:
https://github.com/rusthon/giws


Add latitude and longitude in regencies table

Add latitude and longitude in regencies table.
The coordinate was taken from Google Map API.
https://maps.googleapis.com/maps/api/geocode/json?address=[nama kota/kabupaten]&region=id

Hi Kovloq,
Do you get the lat/lng information using script? If yes, it would be better if you could share your script.

Hi Edward,
The lat/lng information was obtained using Google Geocode written in PHP script. I use limit when selecting a table to prevent timeout in PHP.
fecd0b0
The Geocode function provided by Google is not accurate enough. So, you need to find the lat/lng manually by using maps.google.com. Some of the cases are:

The information when obtained the information from "Kabupaten Bekasi" and "Kota Bekasi" are the same.
Sometimes the function return blank. No lat/lng information is returned.
More cases might be happen in other table as well. Those cases were only happened when I find the lat/lng in regencies table only.

Sorry for slow response. Have no free time for this project lately.
Hmm, the script can't be automated here and really hard to reproduce a new data. And also I think the lat/lng data may not be accurate here. What do you think?

Yes, you right the lat/lng data is not accurate.
It's better not to merge this branch into master branch. Just leave the master branch like this is OK.

found this data
data.go.id
but it quite old

@int2k As you said, the data is quite old. And looks like not up to date.
If someone needs the latitude-longitude data, can consider the link above.


underling a text

Is your feature request related to a problem? Please describe.
In call for speaker section we have coding resources, online webinars, Doubt support, Guidance when we hover on it if they get underlined then it looks nice
Describe the solution you'd like.
we can set css as text-decoration into underline and we can solve the issue and also while hovering over it making back ground into light color helps our webpage to look good
Describe alternatives you've considered.
.
Add any other context or screenshots about the feature request here.
.

please assign this issue to me.

issue raised before already


Docs: console doc fixes WIP

Type of change

Documentation

Description
Console docs updates to reflect new console UI
Checklist

[ ] Update/write design documentation in ./documentation/design
[ ] Write tests and make sure they pass
[X] Update documentation
[ ] Check RBAC rights for Kubernetes / OpenShift roles
[ ] Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
[ ] Reference relevant issue(s) and close them after merging
[ ] Update CHANGELOG.md

Can one of the admins verify this patch?

@k-wall, Could you please review for technical accuracy, and also add the new console screenshot?  Thanks!

@jenmalloy sorry I intended to submit a PR against yours, but pushed to it by mistake,
I updated the existing screenshot.  I also include screenshots of the address/conns - perhaps it is worth including these into ref-view-message-connection-stats-table

@k-wall, looks good to me. I think you will need to approve so I can merge?


[HOW-TO] debug dequeue timer

Hello,
I have a fairly complex app running flask, picamera2 and autofocus on boot.
I am often getting this message
[0:05:43.417007419] [1582]  WARN V4L2 v4l2_videodevice.cpp:2007 /dev/video0[18:cap]: Dequeue timer of 1000000.00us has expired!
[0:05:44.308115388] [1582] ERROR RPI pipeline_base.cpp:1374 Camera frontend has timed out!
[0:05:44.308464398] [1582] ERROR RPI pipeline_base.cpp:1375 Please check that your camera sensor connector is attached securely.
[0:05:44.308506534] [1582] ERROR RPI pipeline_base.cpp:1376 Alternatively, try another cable and/or sensor.

I have checked and it definitely isn't a cable issue (happening on multiple pi's and cameras). What does this message mean exactly? I am trying to work out the best way to debug the issue because I can't make it repeatable
Thanks,
Sam

Hi, thanks for the report.
Obviously that shouldn't be happening. It might be worth double-checking what kind of Pi you are using, what version of the OS (Bullseye or Bookworm), that the software is as up-to-date as it can be, and what kind of camera this is. Also, just to rule things out, how long is the cable, and can you confirm there are no obvious sources of interference nearby? Finally, are you using a Raspberry Pi power supply (it's well known that many third party "cheap" power supplies don't supply the current/voltage they claim).
Out of interest, does the camera keep running, or is it then locked up?
One thing you can try is to locate the rpi_apps.yaml file. Usually I think this is in /usr/share/libcamera/pipeline/rpi/vc4 for most Pis (replace vc4 by pisp for a Pi 5). In that file you can add the line:
    "camera_timeout_value_ms":    5000000,

to push the timeout up to 5s. If there's nothing much wrong but only some marginal timings, then this will help. Though I have my doubts.
@naushir Anything else you can think of to investigate?

Some users have reported this very issue when using using a 3rd party sensor with IR emitters on the sensor board.  During daytime operation things are fine, but at night time, when the IR emitters are turned on, the power draw causes the camera to stop streaming and throw this error.  Could it be the same problem here?

Hello,
A quick update. Increasing the camera_timeout_value to 5s fixes the issue.
I have tried to explore this a little further. If I reset the camera timeout value so it is 'default' I do not receive the camera dequeue message when I do not run autofocus whilst boot. I do get the dequeue message when running the autofocus described here Autofocus - issue 915 in a thread alongside a mjpeg stream/ webserver.

Hi again, I'm afraid I've got a bit confused as to exactly what isn't working now. Would you be able to summarise that again?
If you're able to create a short self-contained example that demonstrates the problem then we could try to reproduce it here. Thanks!

I have a very similar issue. Using a wide-angle v3 camera on a Pi Zero 2W, and with the rpi_apps.yaml already set timeout set to 10s, I still get a very similar error to that above (an extra zero in the timeout and different line numbers for pipeline_base.cpp, so probably a libcamera different version).
[331:02:32.983494193] [1293]  INFO Camera camera_manager.cpp:297 libcamera v0.0.5+83-bde9b04f
[331:02:33.173252533] [1295]  INFO RPI vc4.cpp:437 Registered camera /base/soc/i2c0mux/i2c@1/imx708@1a to Unicam device /dev/media2 and ISP device /dev/media0
[331:02:33.173427897] [1295]  INFO RPI pipeline_base.cpp:1101 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
Stream configuration adjusted
[331:02:33.183831827] [1293]  INFO Camera camera.cpp:1033 configuring streams: (0) 640x480-YUV420 (1) 2304x1296-SBGGR10_CSI2P
[331:02:33.184642710] [1295]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx708@1a - Selected sensor format: 2304x1296-SBGGR10_1X10 - Selected unicam format: 2304x1296-pBAA
[331:43:48.332848098] [1295]  WARN V4L2 v4l2_videodevice.cpp:2007 /dev/video0[13:cap]: Dequeue timer of 10000000.00us has expired!
[331:43:50.413275084] [1295] ERROR RPI pipeline_base.cpp:1333 Camera frontend has timed out!
[331:43:52.697810696] [1295] ERROR RPI pipeline_base.cpp:1334 Please check that your camera sensor connector is attached securely.
[331:43:54.127755142] [1295] ERROR RPI pipeline_base.cpp:1335 Alternatively, try another cable and/or sensor.
ERROR: Device timeout detected, attempting a restart!!!
With the timeout increased from 1s to 10s I get far fewer timeouts than I used to, but they still happen, and seemingly at random times. The camera application might run for days, or just minutes before it happens. They are not restricted to very dark environments as another issue covers.
I have a second, nearly identical setup, but with a v2 NOIR camera and no timeout set in rpi_apps.yaml which seems to run much more reliably.
Various possibly useful details:
$  libcamera-hello --list-cameras
Available cameras
0 : imx708_wide [4608x2592] (/base/soc/i2c0mux/i2c@1/imx708@1a)
Modes: 'SBGGR10_CSI2P' : 1536x864 [30.00 fps - (0, 0)/0x0 crop]
2304x1296 [30.00 fps - (0, 0)/0x0 crop]
4608x2592 [30.00 fps - (0, 0)/0x0 crop]
$ grep timeout /usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml
# Set the timeout for camera response, in ms
"camera_timeout_value_ms": 10000,
$ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 11 (bullseye)"
NAME="Raspbian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
$ dpkg-query -l | grep libcam
ii  libcamera-apps-lite                  1.2.1-1                          armhf        libcamera-apps-lite
ii  libcamera0:armhf                     0~git20230720+bde9b04f-1         armhf        complex camera support library
ii  python3-libcamera                    0~git20230720+bde9b04f-1         armhf        complex camera support library (Python 3 bindings)
Last few lines of /proc/cpuinfo
Hardware        : BCM2835
Revision        : 902120
Serial          :<PHONE_NUMBER>49a90f
Model           : Raspberry Pi Zero 2 W Rev 1.0
Official RPi PSU, normal (15cm?) camera ribbon cable
Application code running is the streaming utility at https://github.com/Squeack/libcamera-broadcaster, which does not explicitly set auto-focus on or off, so it is whatever the default is. The current view is from a window over the garden, so autofocus is unlikely to trigger anyway.

@lurch Do you want to see if you can reproduce this? I think I'm seeing that we need a Pi Zero 2 W, Bullseye lite, a v3 camera, and then install and run https://github.com/Squeack/libcamera-broadcaster. Does that sound right?

Or I can run any debug-augmented code you might want to try?

Thanks for the offer, but I'm not quite sure what I'd actually want to debug, except for "everything" (which would probably change all the timings and make everything fail differently, or even work). So we probably just need to try it first and see if we get the error.
Presumably we can just run the server directly from the command line, connect a remote machine to view the camera, and that should be it. Anything else we would need to do to set it up?

Just copy the files from github and follow the installation instructions in readme.md should be enough. The video stream should then be visible from any modern browser going to http://server:2050/camera.html
If it makes a difference, I am currently running node v12.22.12 and npm v7.5.2, but that should be unimportant.

One difference from the github code is that I changed the last line of ws+https-server.js to
new Broadcaster({width: 640, height: 480, framerate: 20, hdr:1, port:8080, mode:'1536:864'});

Adding the mode parameter makes it use the full FOV instead of clipping to the centre region. The mode is camera model specific so needs to be set to different values for v2 and v3 cameras.

I too have been dealing with the camera timeout issue. I'm posting from mobile, but can add more information if needed.
I have tried the following Pi's:
Pi 3 A+
Pi 4 B
I have tried the following cameras:
Raspberry Pi Camera Module V3
Raspberry Pi Camera Module V3 NoIR
I have tried the following OS systems and their latest, stable version:
Bullseye
Bookworm
Running libcamera-hello --list-cameras, both cameras on both Pis are able to be detected.
However, any other libcamera command I use causes the error (libcamera-hello, libcamera-jpeg, etc.)
I tried a combination of all of the above with each other and continued to get the same error that everyone here has mentioned. I tried setting my camera timeout to be 5s and even 10s. Sadly, this did not work. I have installed fresh images of each OS multiple times on the Pis, ran apt update and apt upgrade, this did not fix the issues either. Both cameras had the same error on both Pis and both OS (please note, I tried each OS and camera in different combinations on each Pi).
I even tried turning on Glamor for thr Pi3, changing the gpu-memory on the Pi3 to 256, and both things did not help :/ so I ended up buying a Pi4B thinning it was a memory issue, but alas, it was not.
Reseating each camera multiple times didn't work either.
I also tried running libcamera with debug switched on and noticed in the error logs that the camera module component cannot be opened. (I can post that exact message later today or tomorrow if needed).
Any help or tips on how else to fix this would be much appreciated. I feel like I'm going crazy lol

@Squeack I've reseated the cable on both sides of the camera module multiple times and the timeout error continues to persist :/

@davidplowman should I open a new issue for my previous comment? It's related to this issue's original content and I kind of just jumped in on the convo.

No problem, I'm fine to leave everything here for now. There seem to be enough reports that we need to have a go at reproducing this, so I'm hoping we'll get round to it during next week. Then we'll see where that takes us!

I've now managed to get @Squeack 's software running on a Pi Zero 2 W with Bullseye Lite and a Camera Module 3 (it gives impressively low-latency streaming, at least compared to the last time I did anything like this), so I guess now I just need to wait and see if I get the timeout error... :slightly_smiling_face:

If it helps, I ran the camera-bug-report tool and uploaded the error log .txt files to this issue
I can upload them here if that's easier too :)

@Squeack I've now been running your example application (with the one-line tweak you suggested) continuously for a full week, and haven't seen a single error. My terminal window still says:
pipi@bullseye32-lite:~/libcamera-broadcaster $ node ws+http-server.js
Creating new broadcaster
(node:7149) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
[0:56:33.186845626] [7156]  INFO Camera camera_manager.cpp:297 libcamera v0.0.5+83-bde9b04f
[0:56:33.335562173] [7157]  INFO RPI vc4.cpp:437 Registered camera /base/soc/i2c0mux/i2c@1/imx708@1a to Unicam device /dev/media3 and ISP device /dev/media0
[0:56:33.335692111] [7157]  INFO RPI pipeline_base.cpp:1101 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
Stream configuration adjusted
[0:56:33.345661780] [7156]  INFO Camera camera.cpp:1033 configuring streams: (0) 640x480-YUV420 (1) 2304x1296-SBGGR10_CSI2P
[0:56:33.346340166] [7157]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx708@1a - Selected sensor format: 2304x1296-SBGGR10_1X10 - Selected unicam format: 2304x1296-pBAA

and my web-browser window is still successfully streaming video.
I started with a clean Bullseye-Lite image (2023-12-05-raspios-bullseye-armhf-lite.img.xz) on a PiZero 2W with a Camera Module 3, did a full apt-upgrade, and apt-installled git, nodejs & npm (with the latter installed with --no-install-recommends to avoid pulling in a bunch of GUI-related packages).
So unless you can come up with a more easily-reproducible problem, I'm not sure what else we can do here.

Hi again, I'm afraid I've got a bit confused as to exactly what isn't working now. Would you be able to summarise that again?
If you're able to create a short self-contained example that demonstrates the problem then we could try to reproduce it here. Thanks!

Hi David, sorry for not replying sooner. Increasing the timeout to 10 seconds has stopped all the errors in my code. I'm on quite a short deadline so have accepted 10 seconds as the solution and am implementing that on each install.
I'm happy to send you a copy of the whole code for debugging but probably won't have time to cut it down to the core picam2 elements myself. I'm creating a custom camera to sell so would have to send the code privately, sorry if that's a hassle,
Thanks,
Sam

@lurch Thank you for taking a look. Increasing the timeout through 1s, 5s, 10s has made the error less common, so I might just have to increase it further and just accept the occasional rare pause in the video.

Hi, thanks for the report.
Obviously that shouldn't be happening. It might be worth double-checking what kind of Pi you are using, what version of the OS (Bullseye or Bookworm), that the software is as up-to-date as it can be, and what kind of camera this is. Also, just to rule things out, how long is the cable, and can you confirm there are no obvious sources of interference nearby? Finally, are you using a Raspberry Pi power supply (it's well known that many third party "cheap" power supplies don't supply the current/voltage they claim).
Out of interest, does the camera keep running, or is it then locked up?
One thing you can try is to locate the rpi_apps.yaml file. Usually I think this is in /usr/share/libcamera/pipeline/rpi/vc4 for most Pis (replace vc4 by pisp for a Pi 5). In that file you can add the line:
    "camera_timeout_value_ms":    5000000,

to push the timeout up to 5s. If there's nothing much wrong but only some marginal timings, then this will help. Though I have my doubts.
@naushir Anything else you can think of to investigate?

Hi!
Sorry to sound stupid but can someone give me some instructions on how to action this?
I have a raspi4 connected to cam 3. I have this setup on voron 3d printer using klipper.
I know how to loginto SSH using Putty but can't execute that command and change the timeout timer in .yml
Do I need to just plug the SD into PC and open some file or? Sorry never used this kind of commands before.
thanks for your help.

If you can log in with ssh (via Putty for example), you should be able to run something like
sudo nano /usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml

assuming that's where the file is for you. You'll need root privileges to edit it (hence sudo), and nano is a simple text editor.
You might be able to plug the SD card into a PC and do it there, but the PC would have to recognise the filesystem. I think a PC running Linux probably would, but a Windows PC probably wouldn't (it would probably recognise the "boot" partition, but not the "root" partition which is where most files are).

Just share my experience.
I had the same issue on Raspberry Pi Camera Module 3 NoIR model on a Raspberry Pi 5 on the cam1 and cam0.
Then I got a Raspberry Pi Camer Module 3 NoIR Wide model. It throw “error:XDG_RUNTIME_DIR not set in the environment” error when I used a 20cm cable. But it worked fine after I changed to another 50cm cable.

I am having similar timeout issue after connecting rpi 3b+ to rev1.3 camera with 2m cable. If I increase the timeout, it just hangs.
Should i file another issue because for some reason it runs completely fine with legacy mode and raspistill/raspivid?

Hi, I think you would have to reproduce the problem with a ~30cm length cable. They're just not designed to be much longer than that, and there's no reason why different software would behave the same. If it has worked in the past with a 2m cable, that is in large part probably luck and it's quite likely there's nothing we can do to make it work reliably.

Yea, that is the thing, it works with original 15cm cable in both legacy and libcamera modes, but with 2m cable only legacy can get images so i thought it might be some software issue. libcamera-hello can recognize the camera but no image received. I dont have other cables to reproduce with like 30cm so I will stick with legacy stuff for now.
I am happy to provide any details if that will help with development, including remote access to pi.

Mostly I think people find the cables are pretty reliable up to about 50cm. You can get some quite nice shielded gold coloured cables for the narrow camera connector like you have on a Pi Zero 2W, but if you really need 2m then that's obviously not enough. Even when it appears to work I would regard 2m cables as high suspect - they'll be prone to all manner of interference, random corruptions even if you're not seeing a lock-up.

It sounds like this problem may occur for a number of reasons some of which may be outside the control of the programmer.
Is there a way of detecting the timeout from with the program (c++ in my case) so that it can make some sensible decision and keep running or at least degrade gracefully? I'm currently using a latch to block the main thread until the callbacks are done and if the camera times out and even one frame is missed then the main thread remains blocked.

The next version of Picamera2 (already on the next branch in this repo) has some extra timeout handling. When you do a straightforward "capture", you can put in a simple timeout value (rather than just block/don't block). Realistically, when that happens, the only choice you have is to carry on waiting (if you are doing the capture asynchronously), or simply to clear everything out, possibly shut the camera down, then cross your fingers and hope for better luck if start over. We'll be releasing that soon-ish (I'd guess 2 or 3 weeks).
In the C++ code you can obviously do all the same stuff - though I don't think anything has really been done for you there.


No Output for Winget running as system from "C:\program files\windowsapps\Microsoft.DesktopAppInstaller_1.18.2691.0_x64__8wekyb3d8bbwe"

On about 10% of our desktop estate winget.exe produces no console output when running as the Local System Account from "C:\program files\windowsapps\Microsoft.DesktopAppInstaller_1.18.2691.0_x64__8wekyb3d8bbwe"
Winget runs as expected when run as a standard user.  This is only happening on about 10% of our devices.  The other 90% winget.exe runs as expected for the local system account when run from "C:\program files\windowsapps\Microsoft.DesktopAppInstaller_1.18.2691.0_x64__8wekyb3d8bbwe"

Windows: Windows.Desktop v10.0.19044.2251
System Architecture: X64
Package: Microsoft.DesktopAppInstaller v1.18.2691.0
Logs: %LOCALAPPDATA%\Packages\Microsoft.DesktopAppInstaller_8wekyb3d8bbwe\LocalState\DiagOutputDir
Links
Privacy Statement   https://aka.ms/winget-privacy
License Agreement   https://aka.ms/winget-license
Third Party Notices https://aka.ms/winget-3rdPartyNotice
Homepage            https://aka.ms/winget
Windows Store Terms https://www.microsoft.com/en-us/storedocs/terms-of-sale

We need to update https://github.com/microsoft/winget-cli/tree/master/doc/troubleshooting with the information regarding running WinGet using the system context. The issue is that we are distributed as an MSIX package which is registered "per user" upon login.
Calling WinGet remotely using the system context requires a separate package and is leveraging the COM API developed for Intune and third-party MDM integration.

We need to update https://github.com/microsoft/winget-cli/tree/master/doc/troubleshooting with the information regarding running WinGet using the system context. The issue is that we are distributed as an MSIX package which is registered "per user" upon login.
Calling WinGet remotely using the system context requires a separate package and is leveraging the COM API developed for Intune and third-party MDM integration.

Morning @denelon ,
Thanks for replying.  I am using Winget through an Intune app so it is executing as "system" but executed locally.  This is working correctly on circa 1500 machines but is failing to produce winget output as system on circa 250 of the machines.  I appreciate that as msix it installs per user but i have been executing it from ""C:\program files\windowsapps\Microsoft.DesktopAppInstaller_1.18.2691.0_x64__8wekyb3d8bbwe"\winget.exe"  when running it as system.
The problem is if i cant get it working consistently for the whole estate I wont be able to use it as a solution as it would be inconsitent.  I have had a look at the new Intune App Store and while it does show promiss I think it still has some growing up to do, hence why I would prefer to use winget.
Many thanks,
Scrat

Have you been able to determine if there are any similarities between the two sets of machines? OS version, App Installer Version, or anything else? I'm wondering if it has to do with whether there has been an interactive logon doing the actual registration or if the App Execution Alias is enabled or disabled.

Morning,
Machines are running the same windows 10 version and CU.  App installer vesions are also the same.  I have tried  the Winget preview 1.4.x versions but no change.  I have tried toggling the Execution Alias but again no change.

I've been experiencing this myself and am yet to determine the circumstances. Have had it on Windows 10 and 11 machines, both of which are 22H2, both of which have worked previously also.
The only thing of relevance I can probably add here is that winget.exe exits with code -1073741515 (0xC0000135) after showing nothing to the console.
I could probably build a debug version of winget and see if I trigger some kind of assertion? It'd be good if debug versions were built and had PDBs available though.

We need to update https://github.com/microsoft/winget-cli/tree/master/doc/troubleshooting with the information regarding running WinGet using the system context. The issue is that we are distributed as an MSIX package which is registered "per user" upon login.
Calling WinGet remotely using the system context requires a separate package and is leveraging the COM API developed for Intune and third-party MDM integration.

Morning @denelon ,
Thanks for replying. I am using Winget through an Intune app so it is executing as "system" but executed locally. This is working correctly on circa 1500 machines but is failing to produce winget output as system on circa 250 of the machines. I appreciate that as msix it installs per user but i have been executing it from ""C:\program files\windowsapps\Microsoft.DesktopAppInstaller_1.18.2691.0_x64__8wekyb3d8bbwe"\winget.exe" when running it as system.
The problem is if i cant get it working consistently for the whole estate I wont be able to use it as a solution as it would be inconsitent. I have had a look at the new Intune App Store and while it does show promiss I think it still has some growing up to do, hence why I would prefer to use winget.
Many thanks,
Scrat

How are you calling it? I'm doing something like this 7zip example....
#region CODE
$winget = $null
$DesktopAppInstaller = "C:\Program Files\WindowsApps\Microsoft.DesktopAppInstaller_*_x64__8wekyb3d8bbwe"
$SystemContext = Resolve-Path "$DesktopAppInstaller"
if ($SystemContext) { $SystemContext = $SystemContext[-1].Path }
$UserContext = Get-Command winget.exe -ErrorAction SilentlyContinue
if ($UserContext) { $winget = $UserContext.Source }
elseif (Test-Path "$SystemContext\AppInstallerCLI.exe") { $winget = "$SystemContext\AppInstallerCLI.exe" }
elseif (Test-Path "$SystemContext\winget.exe") { $winget = "$SystemContext\winget.exe" }
else { return $false }
if ($null -ne $winget) { $winget }
# Logs $(env:LOCALAPPDATA)\Packages\Microsoft.DesktopAppInstaller_8wekyb3d8bbwe\LocalState\DiagOutputDir
& "$winget" install --id "7zip.7zip" --exact --silent --accept-source-agreements --accept-package-agreements | Out-String
#endregion

Hey,
I have the same Problem as @Scrat01 on an fresh and patched installed windows 11. winget is installed and i can use it normaly as an local user. but when i try to use it from remote (via C:\Program Files\WindowsApps\Microsoft.DesktopAppInstaller_1.20.2201.0_x64__8wekyb3d8bbwe\winget.exe ) in system context no output is printed.
Does anybody has an workaround for this? I also tried the ps7 winget.client module with no success.

Experienced similar issue. After deploying the Visual C++ Redistributables the WinGet starts normally as SYSTEM user.

@kugelbit, this is the solution.

has something changed? I am still having this issue on DesktopAppInstaller_1.21.3482.0_x64__8wekyb3d8bbwe and MS Visual C++ 14.38.33130 and I get no output from winget when running as system...

has something changed? I am still having this issue on DesktopAppInstaller_1.21.3482.0_x64__8wekyb3d8bbwe and MS Visual C++ 14.38.33130 and I get no output from winget when running as system...

It is still an issue, I have been troubleshooting this the whole night before I finally found this issue.
After manually installing Visual C++ Redistributables it is working like a charm. Still weird, as it is working just fine as normal user.
@rozmansi thanks! Your comment saved my weekend!

Happened to us too right now, installing vcredist fixed the problem

After manually installing Visual C++ Redistributables it is working like a charm. Still weird, as it is working just fine as normal user.

winget is supposed to use the packaged version of the VCLibs, which gets installed as a dependency. This works fine as a normal user.
But if you are running winget as admin (and I'm assuming as system too), there is an OS bug that causes the DLL search path to not include the package dependencies, so it cannot find the VCLibs and crashes very early in execution. Manually installing the VCLibs redistributables makes it work because those are in the global DLL search path, so winget can find them.


Racon for low heterozygote diploid genome

Hello,
is Racon expected to work for such genome?
I ran, using PE illumina mapped with bwa mem2
racon -t 48 illumina.fq mapped.sam raw.fasta > racon.polished.fasta
the resulting fasta has 3 less contigs. However, the number of Het sites is increased:
Before racon: 1150321
After racon: 1208714
Any idea of why this might be so?
My genome is expected to have a heterozygosity of 2~3 %. I realise Racon might not be suitable for such case. Is Racon able to handle diploid sites?
Thank you.
EDIT: I realise it might be a duplicate of issue #135

Hello,
are the 3 missing contigs small? Sometimes due to lack of coverage some smaller contigs drop out after polishing, but you can retain them with option -u (--include-unpolished). Which assembler did you use to get the raw assembly? How are you getting the information about Het sites? Can you explain how this issue is a duplicate of #135?
Best regards,
Robert

Oh I am really sorry it was issue #134
yes it seems they are small ones.
Here are more information about the assembly:
The assembly was obtained using Flye with ONT (60 x coverage) and PacBio (140 x coverage) reads, then scaffolded with a 3C scaffolder (instagraal).
For the Het sites, I mapped reads with bwa mem2 and called variants with deepvariants.
I think the sequencing quality of the illumina reads I used is all right (see the plot)

Can you please copy the commands you used in running Racon with Illumina data (both mapping and Racon step)?

./bwa-mem2 mem -t 48 -p raw.fasta illumina.fq > mapped.sam
racon -t 48 illumina.fq mapped.sam raw.fasta > racon.polished.fasta

Looks alright. Not sure what to tell you here :/ You can try with minimap2 -ax sr and check whether it is better or not.

okay I will do that. I will report to you (I am in a bottleneck situation on the cluster, so it might take a few days).

Hello, so I did the polishing with minimap2 and the same is happening
before: 1435144
after: 1847183

Not sure what to tell you here.

Well, it's a notoriously hard-to-assemble genome I am dealing with. So I am  not so surprised that "regular" tools give unexpected results.
If I find the cause I will let you know.

Did you by any chance try Pilon in the same setting?

That's the next step


Submodules update

Signed-off-by: Slawomir Jankowski<EMAIL_ADDRESS>

Make sure the tests work without protobuf installed.

Can we update test framework only, because OCTF is going to be updated in another pull request.

Please update only test-framework submodule, unless octf is needed also :) . Almost newest OCTF submodule is already on master.
(Unless needed we try to update OCTF submodule periodically or when we want a change to be introduced. I think benefits of this are that we don't need to constantly update submodules, and we can focus more on validating the specific submodule update when it is introduced)


Fix strange naming of kiss emoji

The namimg of "kiss: woman, man" emoji (http://unicode.org/emoji/charts/full-emoji-list.html#1f469_200d_2764_fe0f_200d_1f48b_200d_1f468), ":kiss::man:" is very strange.
I think that I should fix it.

@JuanitoFatas wdyt?

Awesome, thanks! @yantene Would you like to be a collaborator of this project? 😊

@JuanitoFatas I'd love to join! If there are any essential points I should know, please tell me. :hatching_chick:

@JuanitoFatas I'd love to join! If there are any essential points I should know, please tell me. 🐣

invitation sent! ✉

@JuanitoFatas I've got the invitaion. Thank you! :smile:


Add speech to text pipeline

This change adds the speech-to-text pipeline to the AI Runner

I did a review of this PR.  Initial comments below.
My tests included: speech-mp3-lowbitrate.mp3 (worked), speech-aac-lowbitrate.m4a (worked), vp8-opus.webm (worked), vp8-vorbis.webm (worked), vp9-vorbis.webm (worked), h264 and h265 variants did not work for some reason in mp4 files.  Pulling the audio out of the mp4 files were successfully processed.

Communicate processing was not successful when input is in wrong format or failure happens

I was getting an error on some transcoded files that were intentionally in different input format.  The results right now is a null/empty text response: {"chunks":null,"text":""}.  I think should be an error response code or at least an error in the response to provide error or "ok" if no error.  The last line in the traceback below is where transformers is trying to use ffmpeg in container to parse the file I believe.
Interesting enough is ffmpeg 4.2.7 is installed in the runner container!
File "/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/transformers/pipelines/pt_utils.py", line 186, in __next__ processed = next(self.subiterator)             ^^^^^^^^^^^^^^^^^^^^^^ File "/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 362, in preprocess inputs = ffmpeg_read(inputs, self.feature_extractor.sampling_rate)          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/transformers/pipelines/audio_utils.py", line 41, in ffmpeg_read raise ValueError( ValueError: Soundfile is either not in the correct format or is malformed. Ensure that the soundfile has a valid audio file extension (e.g. wav, flac or mp3) and is not corrupted. If reading from a remote URL, ensure that the URL is the full address to **download** the audio file.

Add the download cmd to dl_checkpoints.sh

huggingface-cli download openai/whisper-large-v3 --include "*.safetensors" "*.json" --cache-dir models

I've updated the runner to handle these errors.
The ai-runner responds with a 400 Bad Request and logs 2024-07-02 15:52:18,181 INFO:     <IP_ADDRESS>:38806 - "POST /speech-to-text HTTP/1.1" 400 Bad Request
The gateway logs I0702 11:50:40.632077 2527064 ai_process.go:561] clientIP=<IP_ADDRESS> request_id=dd12552c Error submitting request cap=31 modelID=openai/whisper-large-v3 try=6 orch=https://<IP_ADDRESS>:8936 err=speech-to-text container returned 400
The orchestrator logs 2024/07/02 11:50:40 ERROR speech-to-text container returned 400 err="{\"detail\":{\"msg\":\"Error processing audio file: Soundfile is either not in the correct format or is malformed. Ensure that the soundfile has a valid audio file extension (e.g. wav, flac or mp3) and is not corrupted. If reading from a remote URL, ensure that the URL is the full address to **download** the audio file.\"}}"
Attached is 400 Bad Request response from Swagger UI.

I've updated the runner to handle these errors.
The ai-runner responds with a 400 Bad Request and logs 2024-07-02 15:52:18,181 INFO:     <IP_ADDRESS>:38806 - "POST /speech-to-text HTTP/1.1" 400 Bad Request
The gateway logs I0702 11:50:40.632077 2527064 ai_process.go:561] clientIP=<IP_ADDRESS> request_id=dd12552c Error submitting request cap=31 modelID=openai/whisper-large-v3 try=6 orch=https://<IP_ADDRESS>:8936 err=speech-to-text container returned 400
The orchestrator logs 2024/07/02 11:50:40 ERROR speech-to-text container returned 400 err="{\"detail\":{\"msg\":\"Error processing audio file: Soundfile is either not in the correct format or is malformed. Ensure that the soundfile has a valid audio file extension (e.g. wav, flac or mp3) and is not corrupted. If reading from a remote URL, ensure that the URL is the full address to **download** the audio file.\"}}"
Attached is 400 Bad Request response from Swagger UI:

I was getting an error on some transcoded files that were intentionally in different input format.  The results right now is a null/empty text response: {"chunks":null,"text":""}.  I think should be an error response code or at least an error in the response to provide error or "ok" if no error.  The last line in the traceback below is where transformers is trying to use ffmpeg in container to parse the file I believe.

I've added error handling to the AI Runner if the model experiences any error while processing the file. It specifically checks for
"invalid soundfile" and returns these as 400 Bad Request.

Is seed a parameter for the speech-to-text pipeline? I was not seeing it on Whisper, maybe keeping it in case a model would use it?

That's correct, I don't see a purpose for the seed parameter with this model. I think it's unlikely this pipeline will need it.

Add the download cmd to dl_checkpoints.sh

huggingface-cli download openai/whisper-large-v3 --include "*.safetensors" "*.json" --cache-dir models

I've added this to dl_checkpoints.sh


Speed To Detect very SLOW

First, Thank you for your source, It has helped me a lot.
But speed to detect the object in an image is slow,
How can faster do it such as use GPU or custom list object, Because I need counting vehicle from the camera.
Please help!

+1

This has GPU support:
https://github.com/wang-xinyu/tensorrtx
Some benchmarks at the page end.

Hi,
There is already an open issue about that here: https://github.com/dotnet/machinelearning/issues/5597 Unfortunatly, and as explained here, this doesn't come from this code but from the ML.Net/Onnx step.
You can use the GPU. If I remember well you will need to use the following package: https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime.Gpu/
I did some tests in the past and it worked, but you might have error messages that are linked to nuget packages conflicts. You will need to uninstall the cpu package.

Try updating to NL.Net v1.6.0, they have improved image handling and it's much faster.
I've updated the repos with https://github.com/BobLd/YOLOv4MLNet/commit/5cab380bf969a03ed33b9e5569746e844ade4ef7

Hi,
There is already an open issue about that here: dotnet/machinelearning#5597 Unfortunatly, and as explained here, this doesn't come from this code but from the ML.Net/Onnx step.
You can use the GPU. If I remember well you will need to use the following package: https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime.Gpu/
I did some tests in the past and it worked, but you might have error messages that are linked to nuget packages conflicts. You will need to uninstall the cpu package.

Hi,
I have nvidia rtx 2080 super i haved try to do that in past but it not work for me,
Are you know which CUDA and cuDNN version i need?

From this recent article, I see that they use

Install CUDA v10.1
Install cuDNN v7.6.4 for CUDA 10.1

But I haven't tried yet

Tnx for your answer!
The Model Builder recognized the GPU but how can it help me with your code?

I found that the inference speed is a lot faster on .Net Core 5.0. It took around ~8000ms on .NET Framework 4.7.2 and ~600s on Core.


Updated paths to cmdlets' reference documentation

Paths to cmdlet documentation in "Find the cmdlets for the DLP reports" referenced an old repository and resulted in "Error 404 Not found" when viewed from Git, updated with the new PowerShell repository paths:

Line 88 updated: from (/powershell/exchange/connect-to-scc-powershell) to (exchange/docs-conceptual/connect-to-scc-powershell.md)
Line 92 updated: from (/powershell/module/exchange/get-dlpdetailreport) to (exchange/exchange-ps/exchange/Get-DlpDetailReport.md)
Line 93 updated: from (/powershell/module/exchange/get-dlpdetectionsreport) to (exchange/exchange-ps/exchange/Get-DlpDetectionsReport.md)
Line 94 updated: from (/powershell/module/exchange/get-dlpsidetectionsreport) to (exchange/exchange-ps/exchange/Get-DlpSiDetectionsReport.md)
Line 98 updated: from (/powershell/exchange/connect-to-exchange-online-powershell) to (exchange/docs-conceptual/connect-to-exchange-online-powershell.md)
Line 102 updated: from (/powershell/module/exchange/get-dlpdetailreport) to (exchange/exchange-ps/exchange/Get-DlpDetailReport.md)
Line 103 updated: from (/powershell/module/exchange/get-maildetaildlppolicyreport) to (exchange/exchange-ps/exchange/Get-MailDetailDlpPolicyReport.md)

these changes result in broken links, editing them out


deploy coverage report to codacy

Codecov Report

Merging #17 (1ee7981) into main (ee76216) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main      #17   +/-   ##
=======================================
  Coverage   87.33%   87.33%
=======================================
  Files          22       22
  Lines        4304     4304
=======================================
  Hits         3759     3759
  Misses        545      545

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ee76216...1ee7981. Read the comment docs.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 87.337%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
3759

Relevant Lines:
4304

💛  - Coveralls


Use Animated.View for TabBar root container

This PR simply replaces the root container of the TabBar component from View to Animated.View.
This way, properties like backgroundColor can be dynamically animated based on the active tab.
I don't think there are any side effects to doing this, but let me know if I'm missing something.
Cheers!

LGTM


Method to add distance to the trajectory DataFrame

Similar to the add_direction and add_speed methods, it will be useful to add add_distance as a method that computes distance between successive points of the trajectory. A distribution of distance, looking at large and small distance segments could be useful to get an understanding of the datasets.
Ref: https://movingpandas.readthedocs.io/en/master/trajectory.html
@anitagraser Since, we have a method to add_speed, I am assuming the distance calculation should already be there somewhere. I am happy to make a PR in that case to add_distance to the trajectory DataFrame.

That's correct. Distance calculation is already implemented. This feature should therefore be straightforward to implement. Please provide unit tests with the PR.


add blog.rmhogervorst.nl :memo: :tada:

Adds roel's R-tefacts to the bloglist. I could nominate myself right? or isn't that awesome? :tada:

Hey Roel, thanks for the submission and apologies for the delay!
For what it's worth, I think the twitter link on you site goes to the wrong place and the security link at the bottom of this page (http://rmhogervorst.nl/cleancode/about/) results in a 404. Confusingly, I can only navigate to that page on my phone - the link to 'about' doesn't seem to exist on the full size version of the site. Maybe something to do with the theme?
Anyway, only minor stuff - thanks again for the submission!

thank you so much for that! I recently changed my twitter handle. The about section is weird though. I'll look into it!


Markdown starters

Each assignment we will be required to make a review ticket. Most assignments provided will have a Hacks section. A Hack is a term for coding, a fun term.

Short Description
Code Fragments
Screen Captures are nice for Hacks

This text below is something called Markdown. This is a heading, inside of code scaffolding.
Investing in Your Technical Future XXXYYY

This is emphasis

Explore the Computer Science Pathway at Del Norte High School and invest in your technical skills. All Del Norte CompSci classes are designed to provide a real-world development experience. Class time includes tech talks (lectures), peer collaboration, communication with teachers, critical thinking while coding, and creativity in projects. Grading is focused on time invested, participation with peers, and engagement in learning.

Sample of bullets

Introduction to concepts and requirements by the teacher

Project-based learning with teacher support, performing Agile/Scrum development

Coding, frontend, backend, devops, version control and algorithmic thinking

Creativity, research, design, data structures, and utilizing ChatGPT

Performing team work, team communication and collaboration, peer reviews/grading

Focus on tehnical communications through project presentations and student led teaching

Grades are on projects, learnt concepts, and live reviews between student(s) and teacher


Remove 1.46 from build file

Remove 1.46 and 1.55 and add 1.53 and 1.60 to build script.  Also add comments and minor fix to expect message.

While you are deleting 46, could you add 53 and 60 which both appear to be used but are not in build.rs?

While you are deleting 46, could you add 53 and 60 which both appear to be used but are not in build.rs?

Yes sure.  Although I'd also like to make this file more maintainable.  Maybe we could add some comments about how this file works?  For example, were does RUSTC env get passed in?

Yes sure. Although I'd also like to make this file more maintainable.

Well, I'd kinda prefer we not encourage people to use this file, so I don't think it should be in the README. I guess we could give it a smal section in CONTRIBUTING.md. But I'm unsure how much it'd help to have it documented better. The problem is that the numbers here need to be in sync with the uses of rust_v_1_* in the codebase and they need to be manually synced. (I guess we could write a CI/lint script that could check.)

Maybe we could add some comments about how this file works? For example, where does RUSTC env get passed in?

This file is build.rs, which cargo compiles and runs before building the rest of the codebase. When it outputs lines of the form cargo:rustc-cfg=X this tells cargo to add RUSTFLAGS=--cfg=X when it is compiling everything else. This allows us to check the current toolchain versions and set rustflags dynamically based on that.

Well, I'd kinda prefer we not encourage people to use this file, so I don't think it should be in the README. I guess we could give it a smal section in CONTRIBUTING.md.

I wasn't suggesting adding it to CONTRIBUTING.md, only looking to add some comments to the file itself.  For example, I can see that env::var_os(RUSTC) is used to discover what rustc the compiler is resolved to use.  This defaults to rustc it looks like if nothing is set.  Is this env ever set?  Also, I think just env::var is good enough since the compiler name should always be valid UTF8? but I guess it doesn't hurt to leave this as is.

This file is build.rs, which cargo compiles and runs before building the rest of the codebase. When it outputs lines of the form cargo:rustc-cfg=X this tells cargo to add RUSTFLAGS=--cfg=X when it is compiling everything else. This allows us to check the current toolchain versions and set rustflags dynamically based on that.

Thanks, I vaguely remember this now from the last time I looked at the build script.  The Cargo book does a pretty good job of documenting this actually: https://doc.rust-lang.org/cargo/reference/build-scripts.html#rustc-cfg

While you are deleting 46, could you add 53 and 60 which both appear to be used but are not in build.rs?

done here: https://github.com/rust-bitcoin/rust-bitcoin/pull/2125/commits/3c05a91606020240097cecf7cec3bfbe823777e1
I also added some comments to the file which hopefully make it quicker to to read:
https://github.com/rust-bitcoin/rust-bitcoin/pull/2125/commits/4ee47c0ff3f36606cd18418a1984ab7d59e723a8

Is this env ever set?

This is settable by the user and it's the same variable cargo already uses. So if a user overrides rustc by setting RUSTC both cargo and our script will do the right thing.

I guess we could write a CI/lint script that could check.

I'd like to see that if anyone has the time.

I agree with @tcharding, also because I am still not understanding what this PR is proposing an why.

This is settable by the user and it's the same variable cargo already uses. So if a user overrides rustc by setting RUSTC both cargo and our script will do the right thing.

@Kixunil thanks, I came to the same realization after reading https://doc.rust-lang.org/cargo/reference/environment-variables.html.

I agree with @tcharding, also because I am still not understanding what this PR is proposing an why.
@vincenzopalazzo this PR started primarily to sync the cfg attributes that are defined in the buildscript with the cfg attributes defined in the codebase.  If you run git grep "rust_v_1_

bitcoin/src/blockdata/script/borrowed.rs:#[cfg(rust_v_1_53)]
bitcoin/src/blockdata/script/borrowed.rs:#[cfg(rust_v_1_53)]
bitcoin/src/blockdata/script/mod.rs:#[cfg(any(not(rust_v_1_60), target_has_atomic = "ptr"))]
bitcoin/src/blockdata/script/mod.rs:#[cfg(any(not(rust_v_1_60), target_has_atomic = "ptr"))]
bitcoin/src/blockdata/script/push_bytes.rs:    #[cfg(rust_v_1_53)]
bitcoin/src/blockdata/script/push_bytes.rs:    #[cfg(rust_v_1_53)]
bitcoin/src/consensus/encode.rs:#[cfg(any(not(rust_v_1_60), target_has_atomic = "ptr"))]
bitcoin/src/lib.rs:    #[cfg(all(not(feature = "std"), not(test), any(not(rust_v_1_60), target_has_atomic = "ptr")))]
internals/build.rs:            println!("cargo:rustc-cfg=rust_v_1_{}", version);
internals/src/serde.rs:        #[cfg(rust_v_1_55)]
internals/src/serde.rs:        #[cfg(not(rust_v_1_55))]

And in the build script, we currently have compile time flags for v_1_46 and v_1_55 https://github.com/rust-bitcoin/rust-bitcoin/blob/master/internals/build.rs#L30.  This means that 1_46 is never used while 1_53 and 1_60 will never be compiled in.  It looks like I made a mistake in this PR by removing 1_55 which I'll add back.  There should be a sanity check for this during compile time so we don't have these mismatches, but for now it's just re-syncing the versions.

NACK to all these "what" comments (comments are mainly useful when they say "why" code does something not what it does, we can see what it does by looking at the code).

@tcharding  I'm fine with not including the comments with this PR if they are noise.

adcaa2cdea28e8e9b4359db9114e028d395c4d43 still has a lot of "what" comments which slow down the code because it forces the reader to interpret them and check that the code matches. But neither the human reader or the compiler learns anything new from them.

dropped https://github.com/rust-bitcoin/rust-bitcoin/commit/adcaa2cdea28e8e9b4359db9114e028d395c4d43 since it's not helpful to others.

In d6024bd2351247a4b33d236fa8cec08491653b5c can you explain where the "expect message should show what you expect" convention comes from? I have never heard of this, it makes the resulting error messages confusing, and I couldn't find it after a quick web search.

@apoelstra I believe the convention is valid for internal invariants. E.g. write!(vec, "foo").expect("vec never errors") however in this case it's used as a quick "hack" parsing outside information since I considered it pointless to do proper error handling. As such the error message should really be what it was originally.

can you explain where the "expect message should show what you expect" convention comes from?
@apoelstra see: https://doc.rust-lang.org/std/option/enum.Option.html#recommended-message-style

let item = slice.get(0)
    .expect("slice should not be empty");

Really I'm fine either way.  It would be nice to have a standard format for the project.

@yancyribbens the doc you reference links to another doc which says:

There are two common styles for how people word expect messages. Using the message to present information to users encountering a panic (“expect as error message”) or using the message to present information to developers debugging the panic (“expect as precondition”).

This is exactly what I said above. Parse error is intended for the user because it triggered from outside. (And could happen if someone tried to wrap rustc in a way that breaks --version.)

I've gotta say, these are pretty poor docs (and they wouldn't even be necessary if the format of the expect panic message wasn't stupid).
But thanks for the citation.
I agree we should just "pick something and be consistent", and in my own code I've always taken the "choose a message that would provide a reasonable error message" except for cases where I'm really certain will never trigger, and which I'm only avoiding unwrap_unchecked because I don't want to argue about the presence of unsae blocks.

We are discussing the same thing over here https://github.com/rust-bitcoin/hex-conservative/pull/31.

Yeah, I'm not a fan of the docs here.  Also, it really takes a high amount of cognition to decide how to format a simple error message which is annoying.  Although, if we are to follow the docs suggestions, I agree that @Kixunil had it right to begin with since in this particular case, it's a parse error that should be presented to the user.  I'll drop this the expect commit.

it really takes a high amount of cognition to decide how to format a simple error message which is annoying.

I think that the only places where an error might be presented to the user are build scripts and examples, so that should help.

FYI we're getting "dependencies could be updated" warnings. Someone could look into updating Cargo-recent.lock when there's time.

@Kixunil we know that there are several MSRV breaks in Cargo-recent.lock and that we can't update it to the latest versions. Personally I think we should just drop the warnings.
But sure, I can update it to the most recent versions that work, since we haven't done that in a while.
(For reference, you added these warnings in https://github.com/rust-bitcoin/rust-bitcoin/pull/1764 and @tcharding commented that "we'd probably have to override it somehow in case of MSRV breaks".)

While you are deleting 46, could you add 53 and 60 which both appear to be used but are not in build.rs?

You've thrown him a curve ball here @apoelstra, there are now some problems with this PR:

internals/build.sh uses all versions used in the workspace (53 and 60 are only used in bitcoin not in internals)
bitcoin/build.sh only uses those versions used in the crate (53 and 60)

Can we be uniform and either use all versions in both build.rs files, or just use the ones needed by the actual crate.

@apoelstra I don't remember if we decided whether Cargo-recent.lock should be MSRV-compliant but I think it shouldn't (or at least we should have one that isn't). We can do MSRV testing with -minimal.
@tcharding I don't think we need to check all those versions - just MSRV and stable. The reason is the gates strictly add things so none of them are active in MSEV and all of them are on stable. Unless you're worried that we make off-by-one mistake (we're copying those from the doc/rust releases, so unlikely) or that there will be interactions between the activated features that make some combinations wrong - unlikely in practice IMO. And if you're worried about these things I'd just test (cargo check) on every version in MSRV..=STABLE.

@Kixunil Cargo-recent.toml is supposed to be "the latest version that we've tested and confirm works". If stuff breaks MSRV then it's confirmed not to work.
@tcharding yeah, I think being uniform is the best/simplest thing. I'm tempted to dynamically get the list using grep (or maybe to store the list in a flat file that we regenerate in CI using grep?)

@apoelstra since IIRC my original idea was to make it non-MSRV would it make sense to add a third one?

Sure, we could add a third one, but I wouldn't be testing it locally because it'd increase my machine workload in order to test dep versions that I don't actually care to test.
I'm not sure what the value would be beyond our existing CI run that just runs cargo update and blithely tests whatever versions crates.io gives us on the day of the test.

Similar to testing with nightly, we would get the chance of reporting any bugs found in other crates early.

I dropped the commit to add 53 and 60 to internals/build.rs.  Thanks @tcharding for pointing that out, I didn't realize that there was a few of theses build.rs files and that they are workspace specific, but that makes sense.  Nice curve ball @apoelstra :)

I'm tempted to dynamically get the list using grep (or maybe to store the list in a flat file that we regenerate in CI using grep?)

This is a good idea I think.

I'm tempted to dynamically get the list using grep..

The problem is 1) it slows down the build ever so slightly since grep runs each build, and I guess windows and other dev stacks may not have grep installed.  Related to the conversation here https://github.com/rust-bitcoin/rust-bitcoin/pull/2144

Why not just loop over all possible versions in each build.rs file greater than the msrv like:
for version in &[msrv..99] {
    println!("cargo:rustc-cfg=rust_v_1_{}", version);
}
```

There's no consequence afaik for supplying versions that don't exists.  Then we don't need to grep or anything.   Maybe we add something to CI to check for stale flags that are in the codebase less than MSRV.

@yancyribbens that's what I suggested in https://github.com/rust-bitcoin/rust-bitcoin/pull/2144#discussion_r1374352559
I think we should do it. Note that your 99 is wrong, it has to be the current version.

@Kixunil oops, sorry I didn't read the code snippet and instead read just read the part about running grep.  Anyway, good idea :) - yes lets do that.

Note that your 99 is wrong, it has to be the current version.

Ah ok, I didn't test trying higher numbers than the current compiler version, although I can see why that would be the case.  That's annoying kind of since that means this needs to be updated every time a new compiler drops I guess.

Or maybe we can just auto update this file when a new compiler drops.

No, the version must be read from rustc stdout. Only MSRV needs to be updated when we change it.
Not emitting numbers higher than the current version is the entire point of that build script!

Not emitting numbers higher than the current version is the entire point of that build script!

Ha, good to know.

No, the version must be read from rustc stdout. Only MSRV needs to be updated when we change it.

Yes, but the file that echos to stdout could be auto built by CI for example.. right?

No, the version must be read from rustc stdout. Only MSRV needs to be updated when we change it.

Ok, I see what you mean now.

Closing this PR in favor of: https://github.com/rust-bitcoin/rust-bitcoin/pull/2146

Ha, good to know.

It'd be helpful if you invested more time into understanding the purpose of any code before you try to change it. I believe it can minimize noise and confusion. Of course feel free to ask if you can't figure things out.

It'd be helpful if you invested more time into understanding the purpose of any code before you try to change it. I believe it can minimize noise and confusion.

IMO a few high level comments would help in this file, although I know my opinions are not always the same here as others.

Of course feel free to ask if you can't figure things out.

Now that you say, I was going to ask this on a related PR, but I guess I can ask it here.  Since we can probably pull out the check for the minor numbers https://github.com/rust-bitcoin/rust-bitcoin/pull/2146 here as you commented, could we just remove all parsing and version checking?  For example, why do we care if the major version number starts with 1?

Maybe a comment that explains the purpose of the code would be better than a comment explaining the steps.
We need to know the minor version and checking major is just a sanity check that we're talking to rustc. Exiting early if something is messed up is much better for debugging.

checking major is just a sanity check that we're talking to rustc

I think that's the responsibility of cargo.  I suppose someone could do something silly and set RUSTC env to something else on their system, but that's not something we should care to check imo.

That check is super cheap though.

That check is super cheap though.

cheap computationally doesn't mean it should be done.  Although it's already setup that way so I wont try to change it :)

We definitely should be sanity checking that a string is shaped the way we expect before we try extracting data from it.

We definitely should be sanity checking that a string is shaped the way we expect before we try extracting data from it.

I guess my confusion is what makes 2.x.x an insane version for example.

cheap computationally

Not just computationally. It's a single obvious line in the code.

I guess my confusion is what makes 2.x.x an insane version for example.

Rust uses semver so 2.x.y literally means incompatible with 1.u.v.

nevermind, the fact that we are starting from MSRV minor version means we are being specific about all version 1.u.v greater.


Add header, Setup routes, Styling updates

Adds header, puts pages on routes, contract details update

I think DLC returns should be moved to the Contracts tab.

Also IMO it would be nice to go directly to the contract tabs if you have DLCs in your wallet.
What do you think about moving the Balance into the header here? I think we would only display Total Balance on the header. This gives me an idea of how much money I have while looking at the contracts tab to create a DLC, especially when we have htis nice little wallet icon here

This LGTM, are we good for merge on this PR?

Yes


Developers should get confirmation that code analysis has run but no defects were found

Feedback from Kasperi Summanen:
He could not tell if code analysis ran or not. He was not seeing any results - his suggestion is to get a clear statement that code analysis ran but no defects were found.

Did the SARIF Results Pane open, yet it was in the empty state (just an open log button)? Did the tab for the pane show a count? If that's the case, we have seen a similar bug (but couldn't repro again).
@GabeDeBacker If there are truly zero defects, how is that communicated to the user?

@jeffersonking The user was not sure the SARIF viewer was even downloaded. So we can infer that the pane did not open. However, I will follow-up for more specific detail by tomorrow afternoon (in time for Triage).

Customer Follow Up: SARIF only opened if there was issues. There were no clear messages in the build log.
[2:58 PM] Kasperi Summanen
It SARIF opened only if there was issues.
[3:03 PM] Malavika Pande
Got it, so SARIF didn't open at all
[3:03 PM] Malavika Pande
if there were no issues that is
[3:04 PM] Kasperi Summanen
Yep nor anything clear msg in the build log

@mpande98 This is likely better handled by the WAVE extension as they have explicit knowledge of the code analysis run. The SARIF extension, in this context, does not see the forest from the trees. It sees a list of logs, but does not know they came from the same "WAVE run".

A better experience: SARIF Viewer opens anyway, and there's a message that says "No defects were found" (or something along these lines). This should also exist in Visual Studio.

Pop out window is possible. We can have a persistent message at the top.

11/2 Notes

We have log files that represent all these conditions
Most of the Visual Studio items are unblocked
In VS Code, the status bar can be cluttered
For colored status bar, have an alternative (i.e text to accompany it)

Note that one of the really lousy experiences in this area is that if things are working, you get a Refresh results button, e.g:

But in many cases, you get a nearly empty panel without a Refresh results and the only way to trigger a refresh is to use SARIF: Clear State which triggers a teardown of the enter window's UI which is really painful...


docs(README,site): add documentation about createInstance()

Merges #367
Fixes #260

Looks good, I'm gonna add the CoffeeScript version of the docs then merge it in :+1:


Update counterparty details

Scope of changes
This PR gives users the ability to update a counterparty.
Note: I encountered a bug where the country didn't always appear in the modal, after I'd previously opened one, but never saw any error messages. I attempted to debug, but still couldn't sort out the issue. This didn't always happen, so that made it challenging to sort out. I'm hoping it's due to the servers being slow and causing a loading issue here. However, I can't confirm that without testing in different environment.
Type of change

[ ] bug fix
[x] new feature
[ ] documentation
[ ] other (describe)

Acceptance criteria
https://www.awesomescreenshot.com/video/30950506?key=9c01493c0ae6c31e8df940a4b7746345
Author checklist

[x] I have manually tested the change and/or added automation in the form of unit tests or integration tests
[ ] I have updated the dependencies list
[ ] I have added new test fixtures as needed to support added tests
[ ] I have added or updated the documentation
[x] Check this box if a reviewer can merge this pull request after approval (leave it unchecked if you want to do it yourself)

Reviewer(s) checklist

[ ] Any new user-facing content that has been added for this PR has been QA'ed to ensure correct grammar, spelling, and understandability.
[ ] To the best of my ability, I believe that this PR represents a good solution to the specified problem and that it should be merged into the main code base.

@bbengfort In the counterparty list partial, we display the edit button if the source is user and the user has edit permission.

@bbengfort In the counterparty list partial, we display the edit button if the source is user and the user has edit permission.

Awesome! Glad we already did that!


Cut/Copy/Paste buttons for CupertinoTextField do not respect SafeArea

Steps to Reproduce
I have this code in an otherwise empty project:
import 'package:flutter/cupertino.dart';

void main() {
  runApp(CupertinoApp(
    home: SafeArea(
      child: Column(
        children: [
          SearchBar(),
          Expanded(
            child: Container(),
          ),
        ],
      ),
    ),
  ));
}

class SearchBar extends StatelessWidget {
  final TextEditingController controller;
  final FocusNode focusNode;

  SearchBar({
    @required this.controller,
    @required this.focusNode,
  });

  @override
  Widget build(BuildContext context) {
    return DecoratedBox(
      decoration: BoxDecoration(
        color: CupertinoColors.lightBackgroundGray,
        borderRadius: BorderRadius.circular(10.0),
      ),
      child: Padding(
        padding: const EdgeInsets.symmetric(
          horizontal: 4.0,
          vertical: 8.0,
        ),
        child: Row(
          children: [
            Icon(
              CupertinoIcons.search,
              color: Color(0xff808080),
            ),
            Expanded(
              child: CupertinoTextField(
                controller: controller,
                focusNode: focusNode,
                cursorColor: CupertinoColors.activeBlue,
              ),
            ),
            GestureDetector(
              onTap: () {
                controller.clear();
              },
              child: Icon(
                CupertinoIcons.clear_thick_circled,
                color: Color(0xff000000),
              ),
            ),
          ],
        ),
      ),
    );
  }
}

Start app on a device with a notch (I used an XS Max sim).
Tap into the text field.

Double-tap on the string of characters to highlight a selection.
Notice that the cut/copy/paste tooltips are obscured by the notch.

Logs
 $ flutter doctor -v
[✓] Flutter (Channel stable, v1.0.0, on Mac OS X 10.13.6 17G3025, locale en-US)
    • Flutter version 1.0.0 at /Users/redbrogdon/Library/flutter
    • Framework revision 5391447fae (3 weeks ago), 2018-11-29 19:41:26 -0800
    • Engine revision 7375a0f414
    • Dart version 2.1.0 (build 2.1.0-dev.9.4 f9ebf21297)

[✓] Android toolchain - develop for Android devices (Android SDK 27.0.3)
    • Android SDK at /Users/redbrogdon/Library/Android/sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-27, build-tools 27.0.3
    • ANDROID_HOME = /Users/redbrogdon/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1024-b01)
    • All Android licenses accepted.

[✓] iOS toolchain - develop for iOS devices (Xcode 10.0)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 10.0, Build version 10A255
    • ios-deploy 1.9.4
    • CocoaPods version 1.5.3

[✓] Android Studio (version 3.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 27.1.1
    • Dart plugin version 173.4700
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1024-b01)

[✓] IntelliJ IDEA Community Edition (version 2018.3.1)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin version 31.1.4
    • Dart plugin version 183.4733

[✓] Connected device (1 available)
    • iPhone XS Max • 33CDC891-E0C9-4EA2-AB0C-D7D0DACB0834 • ios • iOS 12.0 (simulator)

• No issues found!

Since it's technically the same solution, I'm gonna dupe into #3945


Code blocks: Colour contrast issues

This issue is from the accessibility audit of the Design System website by DAC in July 2024.
DAC's description

Page: Password input and Type scale
WCAG reference: 1.4.6 Contrast (Enhanced) (Level AAA)
Issue ID: DAC_Colour_Contrast_Text_03 and DAC_Colour_Contrast_Text_04

Red on grey
The colour contrast on the red text against the grey background failed to meet the expected ratio to pass WCAG 2.2 AAA. The expected ratio for colour contrast with text should be at least 7:1, however the colour contrast ratio for text within this element is 4.51:1. This may be problematic for visually impaired users.
Foreground: #D13118
Background: #F3F2F1
Font Size: 16px
Ratio: 4.51:1

Current code:

use
autocomplete to securely create and enter passwords

Green on grey
The colour contrast on the green text against the grey background failed to meet the expected ratio to pass WCAG 2.2 AAA. The expected ratio for colour contrast with text should be at least 7:1, however the colour contrast ratio for text within this element is 5.84:1. This may be problematic for visually impaired users.
Foreground: #00703C
Background: #F8F8F8
Font Size: 19px
Ratio: 5.84:1

Current code:

@include govuk-font($size:
19);

[…]

@include govuk-font($size 19, $weight: bold,
$tabular: true);

[…]

@include govuk-font-size($size:
19);

[…]

DAC's proposed solution
For sites to meet AAA, they must comply with WCAG 2.2 checkpoint 1.4.6 for colour contrast.
As far as contrast ratio is concerned, it must at least be:

if text is not bold and its size is less than 18pt/24px/1.5em/150%: 7:1 for AAA level
if text is not bold and its size is at least 18pt/24px/1.5em/150%: 4.5:1 for AAA level
if text is bold and its size is less than 14pt/19px/1.2em/118%: 7:1 for AAA level
if text is bold and its size is at least 14pt/19px/1.2em/118%: 4.5:1 for AAA level

Thoughts on solution
We could take inspirations from other syntax highlighting projects, like ericwbailey/a11y-syntax-highlighting or mpchadwick/pygments-high-contrast-stylesheets.
Related issues

The PR in which colours for code highlighting were added was https://github.com/alphagov/govuk-design-system/pull/85
https://github.com/alphagov/govuk-design-system/issues/4011

Additional instances
This will appear wherever there are code blocks. But they would only need fixing once in the CSS file for code highlighting for all of them.
Needed roles
Designer

@querkmachine from today (16th Oct) stand up, @hazalarpalikli confirmed that this could work, so we are ready to merge it once you are back.

Just wanna re-emphesise @selfthinker's comment before we go ahead and merge: https://github.com/alphagov/govuk-design-system/pull/4064#issuecomment-2401809928

@Izabela-16 @querkmachine @owenatgov Leaving this comment here as well:
Hi both,
I’ve been giving this some thought:
I’m fully on board with not using bold for the text. It might unintentionally make it seem like we’re placing more emphasis on certain parts, and could even come across as if we're "shouting" at the users. I think it’s best to keep the text unbolded for a more balanced and approachable tone.
I have some concerns around using orange and red together. I left this comment in the slack earlier:
I noticed that we’re using both orange and red in the code text, but the two colours appear quite similar, which makes it hard to distinguish between them. Since they’re meant to convey different information, it might be confusing. Also the fact that i couldn’t even differentiate the two when i first reviewed this.
So instead we can try replacing orange with purple (#4c2c92) ?

@Izabela-16 just had a chat with Owen, we are removing this ticket from this cycle as it might need some more thinking.

Following https://github.com/alphagov/govuk-design-system/pull/4064 as an attempt to fix this, we've decided we're not going to resolve this issue.


Adds some types to src/runtime/debug and src/runtime/keymgmt

Improves type coverage by a whopping 0.27 percentage points! Worth 0.27 points.


JSON-LD

I am interested in adding JSON-LD support.
If anybody else is interested: Add your name in a comment or contact me.

It's been more than a year, so closing.


The end of the video is cut if recording until auto stop

Description
Record for all allowed time. Very tiny amount of the video at the end is cut off. Maybe a second or even less. On my project for a 15s video sometimes almost 2s.
Steps to reproduce
It is visible on the example (audio/video). When replaying recorded video it stops at 9s (not sure if always but I get that).
Additional Information
Both chrome and firefox seems to be affected.

@ignl can you reproduce this with a pure RecordRTC-only example?

This is more like a countdown issue. In webrtc sample you have to manually stop recording (https://www.webrtc-experiment.com/RecordRTC/) which works ok for chrome (and cuts a lot maybe 2-3s for firefox). Videojs-record seems to work ok for me when you stop manually however using countdown it seems to stop just a liiittttle bit too early. It's subtle but noticeable.

Maybe https://github.com/videojs/video.js/issues/3369 is also related.

Yeah it's very likely. It also seems that it is OK now after I updated to new version of video.js. Ticket is not closed but maybe they already solved that problem.

@ignl what version of video.js are you using now?


insert is not working like in<EMAIL_ADDRESS>

I wanted to create a definition list like this:

foo

bar

baz

quaax

lorem

ipsum

from this data:
{
  "foo": "bar",
  "baz": "quaax",
  "lorem": "ipsum"
}

I found this answer on stack overflow an saw, that there is a insert function which does exactly what I need without an unneccassary sort.
I use d3 in version 4 and there that insert function just appending the element.
I tried that code, working in version 3, but not in version 4

Copied from the 1.0.0 release notes:
In 3.x, selection.append would always append the new element as the last child of its parent. A little-known trick was to use selection.insert without specifying a before selector when entering nodes, causing the entering nodes to be inserted before the following element in the update selection. In 4.0, this is now the default behavior of selection.append; if you do not specify a before selector to selection.insert, the inserted element is appended as the last child. This change makes the general update pattern preserve the relative order of elements and data. For example, given the following DOM:

a

b

f

And the following code:
var div = d3.select("body").selectAll("div")
  .data(["a", "b", "c", "d", "e", "f"], function(d) { return d || this.textContent; });

div.enter().append("div")
    .text(function(d) { return d; });

The resulting DOM will be:

a

b

c

d

e

f

Thus, the entering c, d and e are inserted before f, since f is the following element in the update selection. Although this behavior is sufficient to preserve order if the new data’s order is stable, if the data changes order, you must still use selection.order to reorder elements.

If you have further questions, please use Stack Overflow tag d3.js to ask for help. Although I make an effort to assist everyone that asks, I am not always available to provide help promptly or directly. Stack Overflow provides a better collaborative forum for self-help: tens of thousands of D3-related questions have already been asked there, and some answered questions may be relevant to you.
When asking for help, please include a link to a live example that demonstrates the issue, preferably on bl.ocks.org. It is often impossible to debug from code snippets alone. Isolate the issue and reduce your code as much as possible before asking for help. The less code you post, the easier it is for someone to debug, and the more likely you are to get a helpful response.
If you have a question about D3’s behavior and want to discuss it with other users, also consider the d3-js Google Group or joining the d3-js Slack.
Thank you! 🤗

@mbostock thanks for your answer.
I already looked for a solution on stackoverflow and found the answer that insetr would append a sibling and was just wondering that it didn't work anymore in version 4.
for more details see this collection on codepen.
I wrote 2 examples: one with version 3 and one with verison 4.
So the only way in version 4 to display connected data entries is to sort them after appending, right?

You are trying to append to the enter selection twice, which is not a good idea. The idea of the data-join is that there is a one-to-one correspondence between your data and your DOM elements. In your case you want a one-to-two correspondence, with each datum corresponding to both a DT and a DD element. The way to fix that is to create a wrapper DIV containing the DT and DD for each datum.
var div = d3.select("#list")
  .selectAll("div")
  .data([
    {key: "foo", value: "bar"},
    {key: "baz", value: "quaax"},
    {key: "lorem", value: "ipsum"}
  ])
  .enter().append("div");

div.append("dt")
    .text(function(d) { return d.key; });

div.append("dd")
    .text(function(d) { return d.value; });

Live example: https://bl.ocks.org/mbostock/492d99ad44b663a92a690513f873c82e
If you really want to avoid the wrapper DIV, then another way to do this would be to represent your data as six elements rather than three, and pass a function to selection.append to create the right type of DOM element for each key and value:
d3.select("#list")
  .selectAll("dt,dd")
  .data([
    {type: "dt", value: "foo"},
    {type: "dd", value: "bar"},
    {type: "dt", value: "baz"},
    {type: "dd", value: "quaax"},
    {type: "dt", value: "lorem"},
    {type: "dd", value: "ipsum"}
  ])
  .enter().append(function(d) { return document.createElement(d.type); })
    .text(function(d) { return d.value; });

ahhh, omg, I was so blind! :D
Yes, I wanted to avoid the wrapping div.
Thanks a lot. Your second example is exactly what I need.
I never thought of extending the data or using a function in .append()!


[Mac OS] platform realtime-protection exclusions for Microsoft products

Documentation Improvement Request
Can we have matching documentation added for Mac OS platform Microsoft Apps regarding improving their performance when used in tandem with microsoft defender or other similar anti-virus software?
Example Apps:

Microsoft Teams
Visual Studio Code
Microsoft OneNote
Microsoft Outlook

Right now only windows has documentation on recommended path exclusions for some microsoft products like Microsoft teams. Microsoft Teams is shown as excluded from scanning.
See the example document here: https://docs.microsoft.com/en-us/microsoftteams/troubleshoot/teams-administration/include-exclude-teams-from-antivirus-dlp
matching github doc
Here are some working examples of mdatp exclusions that would improve mac OS performance for Microsoft Apps:
Microsoft Teams
/Applications/Microsoft Teams.app/Contents/Frameworks/Microsoft Teams Helper.app/Contents/MacOS/Microsoft Teams Helper
/Applications/Microsoft Teams.app/Contents/MacOS/Teams

Microsoft OneNote
/Applications/Microsoft OneNote.app/Contents/MacOS/Microsoft OneNote

Microsoft Visual Studio Code
/Applications/Applications/Visual\ Studio\ Code.app/Contents/MacOS/Electron

Hello @FrederickEngelhardt. Can you please tell us to which article(s) your feedback applies? This information will help us route the issue to the correct writer/team. Thanks!

Hello @FrederickEngelhardt. Can you please tell us to which article(s) your feedback applies? This information will help us route the issue to the correct writer/team. Thanks!

This doc should have mac references too:

https://github.com/MicrosoftDocs/OfficeDocs-Support/blob/public/Teams/teams-administration/include-exclude-teams-from-antivirus-dlp.md

@denisebmsft

To make it easier for you to submit feedback on articles on learn.microsoft.com, we're transitioning our feedback system from GitHub Issues to a new experience.
As part of the transition, this GitHub Issue will be moved to a private repository. We're moving Issues to another repository so we can continue working on Issues that were open at the time of the transition. When this Issue is moved, you'll no longer be able to access it.
If you want to provide additional information before this Issue is moved, please update this Issue before December 15th, 2023.
With the new experience, you no longer need to sign in to GitHub to enter and submit your feedback. Instead, you can choose directly on each article's page whether the article was helpful. Then you can then choose one or more reasons for your feedback and optionally provide additional context before you select Submit.
Here's what the new experience looks like.
Note: The new experience is being rolled out across learn.microsoft.com in phases. If you don't see the new experience on an article, please check back later.
First, select whether the article was helpful:

Then, choose at least one reason for your feedback and optionally provide additional details about your feedback:

Article was helpful
Article was unhelpful

Finally, select Submit and you're done!


Updated Plugin.prototype.destroy

Added functionality to destroy plugin instance on an element, by calling $(element).logosDistort('destroy');

good catch. I will add these changes to the coffee script version, (which is the main source of everything)


[WIP] QF-3857 Added dark theme

However, doesn't show as advertised.
For example, for the superhero dark theme, it is advertised in the documentation as looking like this.
If I use the integrated jazzmin UI tweaker, I see this, which is ok (note the grey buttons on the right which is not nice in any case):

However, If I set "dark_mode_theme": "superhero" and my browser to dark mode, I see this, which is rather unpalatable:

To be discussed, but the jazzmin dark themes don't seem so stable/ready yet.

Task linked: QF-3857 Jazzmin dark theme

As stated in the PR opening, jazzmin admin dark mode is not yet production-ready, so giving up for now.
To be reopened if something changes on the jazzmin side ;-)


Problems with the go version on the buildpack

I'm using the buildpack https://github.com/revel/heroku-buildpack-go-revel but I got problems witth the GORM library because of this line https://github.com/jinzhu/gorm/blob/master/association.go#L233 I think it is because the go version 1.3, how can I update the version on the buildpack ?
remote: -----> Restarting applications
-----> Using u1000 to run an application
-----> Using u1000 to run an application
-----> Fetching custom buildpack
-----> Revel app detected
-----> Using go1.3
-----> Running: go get -tags heroku ./...
remote: # github.com/jinzhu/gorm
remote: ../../../github.com/jinzhu/gorm/association.go:233: syntax error: unexpected range, expecting {
remote: ../../../github.com/jinzhu/gorm/association.go:243: non-declaration statement outside function body
remote: ../../../github.com/jinzhu/gorm/association.go:244: syntax error: unexpected }

Solved using godeps

:+1: thanks for sharing your fix @mrkaspa!


Add error_trace to the Jobs.RunsGetOutput response

Hi,
JobsApiClient.RunsGetOutput returns only notebook output, error message and Run metadata. Error trace is missing there which is possible to retrieve via Databricks Jobs API.

Hi Krzysztof, thanks for opening the issue. I’m on vacation this week. I’ll take a look into this when I come back.

Error trace should be available in v2 of the package:
https://github.com/Azure/azure-databricks-client/blob/e4ebc4b48ef4f4c82ecc541846a46ff8dbca9fea/csharp/Microsoft.Azure.Databricks.Client/Models/Run.cs#L70-L71
Does this work for you?

Closing for non-response.


rps_counters_v1 endpoint

r @kriskowal @Raynos
Adds an endpoint for rps counters!! they're great!

:ok:


Add GS to 'Projects using React-Grid-Layout' section, README.md

Add Goldman Sachs to 'Projects using React-Grid-Layout' section
Thanks for submitting a pull request to RGL!
Please reference an open issue. If one has not been created, please create one along with a failing
example or test case.
Please do not commit built files (/dist) to pull requests. They are built only at release.

Thanks!
There is some pretty high-quality documentation on that page; are you able to get permission from your end for us to use the graphics in the README here? It could improve the experience for all users of RGL.

Hi Sam,
I don't actually work at Goldmans - I was also looking at their design
guidelines as I'm at a different bank but also doing react dev work and as
you've noticed, it's got some useful docs.
Was reassured to see they use it (and it prompted me to start looking at
using it too)
So I couldn't provide any permission as it's not my content.
On Thu, 13 Jan 2022 at 13:22, Samuel Reed @.***> wrote:

Thanks!
There is some pretty high-quality documentation on that page; are you able
to get permission from your end for us to use the graphics in the README
here? It could improve the experience for all users of RGL.
—
Reply to this email directly, view it on GitHub
https://github.com/react-grid-layout/react-grid-layout/pull/1645#issuecomment-1012131822,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AESON2N745LUDV5R5NJ3GK3UV3GZ5ANCNFSM5L3MQJOA
.
You are receiving this because you authored the thread.Message ID:
@.***>


Add dnsforge.de DNS provider

Description
Resolves: #2019
Check List

[x] I understand that by not opening an issue about a software/service/similar addition/removal, this pull request will be closed without merging.

[x] I have read and understand the contributing guidelines.

[x] The project is Free Libre and/or Open Source Software

Netlify preview for the mainly edited page:

Code repository of the project (if applicable):

I don't think we decided to add this in #2019 so I will close until we come to a proper conclusion.


Start using human-panic

https://github.com/rust-cli/human-panic

It mentions "issue or email" but it doesn't seem to put email in the report.  It puts authors if set but authors isn't set by this.