failed to resolve: use of undeclared crate or module microbit

I have a micro:bit v2.21 and using Linux Kubuntu, i created a new rust project using cargo and modified the files like this:
Cargo.toml
[package]
name = "test_0"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
microbit-v2 = { version = "0.13.0", optional = true }
cortex-m = { version = "0.7.7", features = ["critical-section-single-core"] }
cortex-m-rt = "0.7.3"
panic-halt = "0.2.0"
defmt-rtt = "0.4.0"
defmt = "0.3.2"

[features]
v2 = ["microbit-v2"]

default = ["defmt-default"]

# do NOT modify these features
defmt-default = []
defmt-trace = []
defmt-debug = []
defmt-info = []
defmt-warn = []
defmt-error = []

main.rs
#![no_std]
#![no_main]

use defmt_rtt as _;
use panic_halt as _;

use cortex_m_rt::entry;

use microbit::{
    board::Board,
    display::blocking::Display,
    hal::{prelude::*, Timer},
};

#[entry]
fn main() -> ! {
    if let Some(board) = Board::take() {
        let mut timer = Timer::new(board.TIMER0);
        let mut display = Display::new(board.display_pins);

        #[allow(non_snake_case)]
        let letter_I = [
            [0, 1, 1, 1, 0],
            [0, 0, 1, 0, 0],
            [0, 0, 1, 0, 0],
            [0, 0, 1, 0, 0],
            [0, 1, 1, 1, 0],
        ];

        let heart = [
            [0, 1, 0, 1, 0],
            [1, 0, 1, 0, 1],
            [1, 0, 0, 0, 1],
            [0, 1, 0, 1, 0],
            [0, 0, 1, 0, 0],
        ];

        #[allow(non_snake_case)]
        let letter_R = [
            [0, 1, 1, 0, 0],
            [0, 1, 0, 1, 0],
            [0, 1, 1, 0, 0],
            [0, 1, 0, 1, 0],
            [0, 1, 0, 1, 0],
        ];

        #[allow(non_snake_case)]
        let letter_u = [
            [0, 0, 0, 0, 0],
            [0, 0, 0, 0, 0],
            [0, 1, 0, 1, 0],
            [0, 1, 0, 1, 0],
            [0, 1, 1, 1, 0],
        ];

        #[allow(non_snake_case)]
        let letter_s = [
            [0, 0, 0, 0, 0],
            [0, 0, 1, 1, 0],
            [0, 1, 0, 0, 0],
            [0, 0, 1, 0, 0],
            [0, 1, 1, 1, 0],
        ];

        #[allow(non_snake_case)]
        let letter_t = [
            [0, 0, 1, 0, 0],
            [0, 1, 1, 1, 0],
            [0, 0, 1, 0, 0],
            [0, 0, 1, 0, 0],
            [0, 0, 1, 0, 0],
        ];
        loop {
            display.show(&mut timer, letter_I, 1000);
            display.show(&mut timer, heart, 1000);
            display.show(&mut timer, letter_R, 1000);
            display.show(&mut timer, letter_u, 1000);
            display.show(&mut timer, letter_s, 1000);
            display.show(&mut timer, letter_t, 1000);
            display.clear();
            timer.delay_ms(250_u32);
        }
    }

    panic!("End");
}

When i run cargo embed --target thumbv7em-none-eabihf, i get that error:
error: could not compile `test_0` due to 3 previous errors
error[E0433]: failed to resolve: use of undeclared crate or module `microbit`
 --> src/main.rs:9:5
  |
9 | use microbit::{
  |     ^^^^^^^^ use of undeclared crate or module `microbit`

error[E0432]: unresolved imports `microbit::board::Board`, `microbit::display::blocking::Display`, `microbit::hal::Timer`
  --> src/main.rs:10:5
   |
10 |     board::Board,
   |     ^^^^^^^^^^^^
11 |     display::blocking::Display,
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^
12 |     hal::{prelude::*, Timer},
   |                       ^^^^^

error: aborting due to 2 previous errors

Some errors have detailed explanations: E0432, E0433.
For more information about an error, try `rustc --explain E0432`.
       Error Failed to run cargo build: exit code = Some(101).

Have you tried running cargo embed --target thumbv7em-none-eabihf --features v2?
Looks like the docs use --features v2 in the example cargo run command.

@curt-wise does the thumbs-up mean that it worked for you? If so I think we should close this issue 🙂

Have you tried running cargo embed --target thumbv7em-none-eabihf --features v2? Looks like the docs use --features v2 in the example cargo run command.

Yes i know that the docs state that. But i want to make it work using only:
cargo embed --target thumbv7em-none-eabihf
and preferably using only:
cargo embed
Any idea how i might do that ?

@curt-wise Features you can read up on cargo features but as I understand it you would have to create your own version of the crate with all conditional compilation for "v2" removed.
See here on how to set a default target.

I wanted to create my own project from scratch but could not do that. Like doing cargo new <project_name> and add the microbit v2 crate in cargo.toml. I wanted to do that instead of cloning the repository and modifying in it.

possibly have to remove the optional:
[dependencies]
microbit-v2 = { version = "0.13.0", optional = true }
remove from Cargo.toml:
[features]
v2 = ["microbit-v2"]
try to compile. if compilation doesn't work, check errors, may have to add this to your .cargo/config if you get linker errors:
[target.'cfg(all(target_arch = "arm", target_os = "none"))']
rustflags = [ "-C", "link-arg=-Tlink.x",
]
should be able to create a new project with cargo new and flash with cargo embed --target thumbv7em-none-eabihf


feat(reporter) local & remote Endpoint improvements

Fixes #55
In all the Spans generated by kong:

The localEndpoint is now { serviceName = "kong" }
The remoteEndpoint has changed like so:

If a Kong Service was found when proxying,

If that service has a name then remoteEndpoint will have { serviceName = "<service.name>"}.
Otherwise remoteEndpoint = { serviceName = "<service.id>" } where id is a uuid.

In addition to the serviceName, the remoteEndpoint can have ipv4/ipv6 and port attributes. These have remained unchanged from before:

On the request (root) span,  ipv4/v6 and port are the consumer's forwarded ip and port
On each balancer try span, ipv4/v6 and port are what the balancer attempted
On the proxy span, ipv4/v6 and port are first successful combination the balancer attempted

I've got some concerns about this PR, quoting my comment in #55:

For all the spans created by the plugin, the localEndpoint is now { serviceName = "kong" }, and nothing else (kong doesn't know its own IP address). It seems a bit ... unuseful?
When a span is tagged error: false it appears red on the UI. Should I remove the error tag and only use it when there's an error?
On the usual "API gateway" scenario, kong proxies traffic between a "downstream consumer" and an "upstream service", generating a root "request" span, with one proxy span and 1 or more balancer tries. These (now) all have the same serviceName, when Kong has identified a service to proxy to. But the ip4/port values can be very different: on the root span they point to the consumer, while on the other spans they point to the upstream service ips/ports. I think it is weird that they have the same serviceName but such different ips/ports.


Feature/mkdocs

This updates the documentation to support mkdocs, fixes long line wrapping, and cleans up the links.

Thanks for pushing through on these doc updates. I'm going to merge this.
For clarity, are both methods for building docs (build_docs.sh and npm run docs) needed? Are they for different purposes?

The npm run docs uses the preferred docker method for doing the build on systems that do not have mkdocs installed.  The build_docs.sh is for building the docs on a system that already has mkdocs installed.  I ran into this because I was doing dev within a docker container and couldn't run the docker-compose because that would be doing docker within docker.
Thanks for merging!


Not Working youtube downloads in my user bot please fix it

Checklist

[ ] I'm reporting a broken site support issue
[ ] I've verified that I'm running youtube-dl version 2021.06.06
[ ] I've checked that all provided URLs are alive and playable in a browser
[ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[ ] I've searched the bugtracker for similar bug reports including closed ones
[ ] I've read bugs section in FAQ

Verbose log

PASTE VERBOSE LOG HERE

Description

WRITE DESCRIPTION HERE

Duplicate. See #29979.


OpenVPN client export: support OpenVPN Connect client

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

[x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
[x] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Is your feature request related to a problem? Please describe.
OpenVPN client export already supports a variety of different OpenVPN clients (e.g. Viscosity) with distinct configuration packages. OpenVPN Connect seems to be a new client which requires yet another format of exported configuration. As this client seems to be the official client by openvpn.net Website it may gain much more popularity in future, thus support for its configuration format would be beneficial to both opnsense Administrators and OpenVPN Connect users.
Describe the solution you like
Add another option to the dropdown in "Export type". The format required by OpenVPN Connect is nothing entirely new. The export generated needs the same .p12 file as included in the "Archive" option in combination with the configuration file as generated by the "File Only" option.
Describe alternatives you considered
Currently the only alternative is to first download the "Archive" format, then the "File Only" format and manually recombine the two to a new archive. This is not very admin friendly as you first need to find out that this would be a working solution.
Additional context
n/a

The file option is intended for the standard OpenVPN client (which is more or less the same as the command line version if I'm not mistaken) and ships the certificate in the <cert> section in standard pem format, which usually works out of the box.

"OpenVPN Connect client" is NOT command line openvpn. It's a GUI client for Windows and other OS. Please click the link in my ticket for more information.

yes, which underneath uses the same client if I'm not mistaken and accepts an ovpn profile (which is our plain file format). It's not that I don't know their client, I've used it before on other platforms where Viscosity isn't supported....

If the VPN server requires user certificates then the p12 certificate store from the archive and the plain text file are both needed. The plain file alone is not enough. We trialed&errored a while to find out how to configure that client correctly.

Maybe there's something else specific in your use case, but the file version should contain the same certificate information (only can't carry a password as that's part of the p12 format).
https://github.com/opnsense/core/blob/58186f8c4ed345d0f7a830045b84355ea2247984/src/opnsense/mvc/app/library/OPNsense/OpenVPN/ArchiveOpenVPN.php#L80-L88
https://github.com/opnsense/core/blob/58186f8c4ed345d0f7a830045b84355ea2247984/src/opnsense/mvc/app/library/OPNsense/OpenVPN/PlainOpenVPN.php#L182-L190
What sort of issues are you facing when using the plain file directly (without wrapping it into a new archive, which shouldn't be needed)?


Add Clubstep x Money Machine by 30th century homer alternative

Song Name
Clubstep x Money Machine
Artist Name
30th century homer alternative
Source
Youtube
Youtube Link
FTkHz5Pbm_0
Direct File Link
No response
Song ID
13
Start Offset [ms]
0
Extra
Created from Auto Nong.
Created for level: 450920

bla

accept


OpenCVSharp does not seem to work with Raspberry PI Camera Module 3

I have been successfully using OpenCVSharp with Camera module 2.1 . I recently bought a new Camera module v3 and the code throws the following errors:
[<EMAIL_ADDRESS>global cap_gstreamer.cpp:2839 handlemessage opencv | gstreamer warning: embedded video playback halted; module v4l2src0 reported: failed to allocate required memory.

Hi, i have some troubles with raspberry pi4 using opencvsharp and i think you can help me. i just connected camera module v2 by
capture=videocapture(-1);
and seems like connected well, but when i try to show in picturebox like
Mat frame;
capture.read(frame);
picturebox1.Image=Bitconverter.toBitmap(frame);
the picturebox is just empty.
was the connecting wrong? or is there any sample code for camera module V2?
Help me plz,, I just want to show video and draw contours in real time.


Thanks for great app. Could you add support for DirectML/ONNX/OpenVino acceleration and LCM/SD-Turbo/SDXS/Hyper SD/SDXL-Lightning models, please?

...

I have no plan to support others backends besides GGML (at least in the short term). GGML has support for GPU acceleration and in the latest version the Vulkan performance has improved.
Regarding distilled models (LCM/Lightining/Turbo/Hyper) it should  be possible to run them, but the quality may not be the best due to missing sampling methods. I do plan to add more methods when I have the time.

Thanks for answer. Good luck in developing of this nice project.


Potential null check issue in code compiled for the web

It looks like there is something weird going on with using null check patterns on nullable strings in Dart 3.3.1 (compiled for web).
I'm running a Flutter 3.19.3 web app in Google Chrome Canary 125.0.6386.0.
      print('query: $query');
      if (query case final query?) {
        print('$query is not null');
      } else {
        print('$query is null');
      }

Output
query: mystring
null is null

The code behaves as expected when compiled for MacOS.
[✓] Flutter (Channel stable, 3.19.3, on macOS 14.3.1 23D60 darwin-arm64, locale en-DE)
    • Flutter version 3.19.3 on channel stable at /Users/max/fvm/versions/3.19.3
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision ba39319843 (3 weeks ago), 2024-03-07 15:22:21 -0600
    • Engine revision 2e4ba9c6fb
    • Dart version 3.3.1
    • DevTools version 2.31.1

[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.2)
    • Android SDK at /Users/max/Library/Android/sdk
    • Platform android-34, build-tools 33.0.2
    • Java binary at: /Users/max/Applications/Android Studio Hedgehog 2023.1.1.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.7+0-17.0.7b1000.6-10550314)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 15.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 15E204a
    • CocoaPods version 1.15.2

[✓] Chrome - develop for the web
    • CHROME_EXECUTABLE = /Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary

[✓] Android Studio (version 2023.1)
    • Android Studio at /Users/max/Applications/Android Studio Hedgehog 2023.1.1.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.7+0-17.0.7b1000.6-10550314)

[✓] VS Code (version 1.87.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.84.0

[✓] Connected device (3 available)
    • Max’iPad (mobile)               • 00008101-001264C02690801E • ios            • iOS 17.3.1 21D61
    • macOS (desktop)             • macos                     • darwin-arm64   • macOS 14.3.1 23D60 darwin-arm64
    • Chrome (web)                • chrome                    • web-javascript • Google Chrome 125.0.6386.0 canary

[✓] Network resources
    • All expected network resources are available.

#### General info

- Dart 3.3.1 (stable) (Wed Mar 6 13:09:19 2024 +0000) on "macos_arm64"
- on macos / Version 14.3.1 (Build 23D60)
- locale is en-DE

#### Project info

- sdk constraint: '>=3.3.1 <4.0.0'
- dependencies: auto_size_text, bloc, collection, cupertino_icons, dynamic_color, equatable, firebase_core, floating_action_bubble, flutter, flutter_bloc, flutter_svg, flutter_vector_icons, google_fonts, google_geocoding_api, google_maps_flutter, hydrated_bloc, oauth2, reorderable_grid_view, swipeable_page_route, uuid, very_good_analysis
- dev_dependencies: flutter_lints, flutter_test
- elided dependencies: 2

Thank you so much for reporting the issue @maxluchterhand1!
I believe this is a duplicate of https://github.com/dart-lang/sdk/issues/54559. Which was fixed recently. The fix landed in 70d3adade20205a073b30967f1018e6fcf8de17e, which is already in the main, dev, and beta releases of 3.4.0.

Note that while the fix rolls out, you can workaround the problem by introducing a new variable name and not shadowing the outer variable. For example:
void main() {
  final query = "myquery";
  print('query: $query');
  if (query case final query?) {
    print('A $query is not null');
  } else {
    print('B $query is null');
  }

  if (query case final query2?) {
    print('C $query2 is not null');
  } else {
    print('D $query is null');
  }
}

Prints:
query: myquery
B null is null
C myquery is not null


Error compiling on Debian 11

I believe I have the requirements correct.  I'm thinking it might be a python issue
root@PM03:~/jbdtool-main# make
gcc -Wall -g -DDEBUG=1 -DBLUETOOTH  -o jbdtool main.o jbd_info.o jbd.o parson.o list.o utils.o cfg.o daemon.o module.o ip.o serial.o bt.o can.o -lgattlib -lgobject-2.0 -lgio-2.0 -lglib-2.0 -lpthread
/usr/bin/ld: warning: libpython3.5m.so.1.0, needed by /usr/lib/gcc/x86_64-linux-gnu/10/../../../../lib/libgattlib.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: main.o: in function main': /root/jbdtool-main/main.c:940: undefined reference to mqtt_new'
/usr/bin/ld: /root/jbdtool-main/main.c:944: undefined reference to mqtt_connect' /usr/bin/ld: /root/jbdtool-main/main.c:945: undefined reference to mqtt_disconnect'
/usr/bin/ld: /root/jbdtool-main/main.c:1246: undefined reference to mqtt_connect' /usr/bin/ld: /root/jbdtool-main/main.c:1247: undefined reference to mqtt_pub'
/usr/bin/ld: /root/jbdtool-main/main.c:1248: undefined reference to mqtt_disconnect' /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/10/../../../../lib/libgattlib.so: undefined reference to PyGILState_Release'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/10/../../../../lib/libgattlib.so: undefined reference to PyEval_CallObjectWithKeywords' /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/10/../../../../lib/libgattlib.so: undefined reference to PyGILState_Ensure'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/10/../../../../lib/libgattlib.so: undefined reference to `Py_BuildValue'
collect2: error: ld returned 1 exit status
make: *** [Makefile:72: jbdtool] Error 1

In the makefile, set MQTT=yes.   You'll need the paho-mqtt-c library: https://github.com/eclipse/paho.mqtt.c
for the gattlib (bluetooth) errors, I'll have to check that out.
You could always grab it from github and compile it yourself: https://github.com/labapart/gattlib
Here's the command I use to build it (note the no python interface):
rm -rf build; mkdir -p build && cd build || exit 1
cmake -DGATTLIB_BUILD_EXAMPLES=NO -DGATTLIB_SHARED_LIB=NO -DGATTLIB_BUILD_DOCS=NO -DGATTLIB_PYTHON_INTERFACE=NO ..
make -j 9 && make install

I though I had installed the two dependencies correctly.  I don't see a CMakeLists.txt file, but i'm not super familiar with the process vs make
root@PM03:~/jbdtool-main# rm -rf build; mkdir -p build && cd build || exit 1
cmake -DGATTLIB_BUILD_EXAMPLES=NO -DGATTLIB_SHARED_LIB=NO -DGATTLIB_BUILD_DOCS=NO -DGATTLIB_PYTHON_INTERFACE=NO ..
make -j 9 && make install
CMake Error: The source directory "/root/jbdtool-main" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.
make: *** No targets specified and no makefile found.  Stop.

My apologies - the commands I gave were what I use to build gattlib.
Generally, you want to build paho, then build gattlib, then jbdtool.
Here's the commands I use to build paho:
cd to directory where you cloned it
m -rf build; mkdir -p build && cd build
cmake -DPAHO_WITH_SSL=TRUE -DPAHO_BUILD_SHARED=TRUE -DPAHO_BUILD_STATIC=TRUE -DPAHO_ENABLE_TESTING=FALSE -DPAHO_BUILD_SAMPLES=TRUE ..
make && make install
Here's the commands I use to build gattlib:
cd to directory where you cloned it
m -rf build; mkdir -p build && cd build
cmake -DGATTLIB_BUILD_EXAMPLES=NO -DGATTLIB_SHARED_LIB=NO -DGATTLIB_BUILD_DOCS=NO -DGATTLIB_PYTHON_INTERFACE=NO ..
make && make install
then I build jbdtool simply by running make (make sure MQTT=yes and BLUETOOTH=yes in Makefile)
I'll need to fix it so that if MQTT isn't enabled it doesn't try and use it...
What platform you want this for? d11 x86_64?  I'll see if I can build a static version for you tomorrow if you're still stuck.

Hi, yes it is for x86_64.  I attempted to rebuild the dependencies using your commands.  Same issue I'm afraid.  If you could build a static version that would be great.  Thanks!
This is my first JBD BMS. So far I like ti much better than the Daly BMS it's replacing

ok give this a try
jbdtool_static.zip

your compiled version worked thanks.


Let docker run bind also to a host not only a ip

when you do docker run -p host.box:10080:80 you get
Cannot start container cbf350dc593e5b27e05a7cef06dd73f890a7d18671e5fadf38fd52065f4bf73a: Bad parameter: invalid host ip host.box

but when you would supply the ip to the command it would work i am voting for implaminting host usage at that place because then i can keep my run files same and it will always bind to right ip.

The problem with this idea is that host names don't necessarily have a one-to-one mapping to IP addresses. A hostname can map to multiple IP addresses. As such, this kind of thing would have to be tailored to the end user's setup.
A simple solution would be:
docker run -p $(getent ahostsv4 host.box | awk '{ print $1 }'):10080:80

ya that true thats true but imagin some one that uses this in his run command will probally have dns setuped for this situation so every host will have local the right response that he expects (the public main ip to use)

in my case i have on every box a host.box entry so that it knows its ip via that dns entry i use it in many scripts but it would in this case simply enable me to do one run command on 1000 of boxes and bind the container to the right ip its more well as at present i transform the run command for every host only because the leak of dns ability in that point

and your suggest command is realy well realy good but thats not fig compatible.

Perhaps this makes more sense to be implemented in Fig then; there's a PR for Fig to support environment variables in fig.yml;  https://github.com/docker/fig/pull/76#issuecomment-63954578
thinking something like this;
DOCKER_HOST_BASE=$(getent ahostsv4 host.box | head -n 1 | awk '{ print $1 }') fig up

not good i will edit fig to simply dns resolve your right with that but every where where i can supply i ip i should be able to supply a host that generaly the right way-

I can see binding to a eth device instead of an ip, but by host seems a bit off since host can be external.

i can see that your a good guy cpuguy83 then do binding to a device that solves my needs too and i patched fig as work around till this day comes :dash:

Binding to an interface also has issues. A single interface can have multiple addresses. :-/

phemmer how ?

@phemmer That's ok, it would be equivalent to multiple -p's. Though, ip's would have to be checked at each container start.

@frank-dspeed ip addr add <IP_ADDRESS>/32 dev eth0; ip addr add <IP_ADDRESS>/32 dev eth0
@cpuguy83 but what happens if an address is added or removed while docker is running? I think such a feature would be misleading, as then users would expect their container to pick up the new address (which it might, but that means docker has to monitor IP addresses on the interfaces).
I'm not opposed to dynamic detection of the bind address, but I don't think this is docker's responsibility. If fig is in use, this sounds like fig's job.

@phemmer your wrong ip addr add will not result in a new ip added to the device it will not show up in ifconfig and the ip if you bind to a interface is not importent all ips of that interface will work you bind to a interface. adding removing ips don't matters then.

@frank-dspeed It doesn't show up in ifconfig because ifconfig was deprecated years ago. It doesn't support a lot of modern features in the linux kernel. Some distros have even stopped including it by default.
Secondly, you cannot bind to an interface. See man 2 bind. You specify an address, not an interface. You can bind to <IP_ADDRESS>, which means all addresses on the system, but there is no way to specify all addresses on an interface.
Additionally, if you bind to one of those addresses, you will not receive traffic for other addresses on the interface.

This can be closed i think i will do my own version of docker-compose now :D


Fix links in resource metrics pipeline page

This PR intentionally consists of two commits: the first one fixes the links in the page to avoid using absolute paths or redirection paths; the second one wraps the long lines in the page for ease of change tracking.

/lgtm

Preview of updated page:
https://deploy-preview-32257--kubernetes-io-main-staging.netlify.app/docs/tasks/debug-application-cluster/resource-metrics-pipeline/
/lgtm

Approving since there's a second lgtm from @RA489
/approve


Initializing and returning a custom dtype array/record from njit causes garbled data

Reporting a bug

[x] I have tried using the latest released version of Numba (most recent is
visible in the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
[x] I have included a self contained code sample to reproduce the problem.
i.e. it's possible to run as 'python bug.py'.

Hi,
I found the following issue: Numba allows initializing a custom numpy.dtype array/record with numpy.zeros() inside a numba.njit function, but the values are sometimes non-zero.
Unlike numpy.zeros(), numpy.ones() and numpy.full() throw a Numba.TypingError ("No implementation of function Function(<function ones at 0x00000182EACAD9D0>) found for signature: ..." etc.).
The Supported NumPy features page states that "numpy.zeros() (only the 2 first arguments)" is supported, but I think this only applies for simple dtypes like numpy.float64.
I discovered the issue when a program that should have been deterministic was randomly crashing or getting stuck.
Version information: 64-bit Python 3.8.10 and Numba 0.54.1 on Windows 10.
import numba
import numpy as np

point_dtype = np.dtype([
    ("x", np.float64),
    ("y", np.float64),
    ("z", np.float64)
])

@numba.njit()
def zeros_array():
    # [(0., 0., 0.)]
    point = np.zeros((1,), dtype=point_dtype)
    return point

@numba.njit()
def zeros_array2():
    # [(1., 2., 3.)]
    point = np.zeros((1,), dtype=point_dtype)
    point[0]["x"] = 1.0
    point[0]["y"] = 2.0
    point[0]["z"] = 3.0
    return point

@numba.njit()
def zeros_array_print():
    # [(0., 0., 0.)]
    # -9.868309920863368e+148
    point = np.zeros((1,), dtype=point_dtype)
    print(point)
    print(point[0]["x"])

@numba.njit()
def zeros_array_print2():
    # [(0., 0., 0.)]
    # 0.0
    # [(0., 0., 0.)]
    point = np.zeros((1,), dtype=point_dtype)
    print(point)
    print(point[0]["x"])
    print(point)

@numba.njit()
def zeros_record():
    # (-9.86830992e+148, -9.86830992e+148, -9.86830992e+148)
    point = np.zeros((1,), dtype=point_dtype)[0]
    return point

@numba.njit()
def ones_array():
    # numba.TypingError
    point = np.ones((1,), dtype=point_dtype)
    return point

@numba.njit()
def ones_record():
    # numba.TypingError
    point = np.ones((1,), dtype=point_dtype)[0]
    return point

@numba.njit()
def full_array():
    # numba.TypingError
    point = np.full((1,), 0, dtype=point_dtype)
    return point

@numba.njit()
def full_record():
    # numba.TypingError
    point = np.full((1,), 0, dtype=point_dtype)[0]
    return point

def main():
    za = zeros_array()
    print(za)
    print()

    za2 = zeros_array2()
    print(za2)
    print()

    zeros_array_print()
    print()

    zeros_array_print2()
    print()

    zr = zeros_record()
    print(zr)
    print()

    try:
        oa = ones_array()
        print(oa)
    except numba.TypingError as e:
        print(e)
    print()

    try:
        or_ = ones_record()
        print(or_)
    except numba.TypingError as e:
        print()
    print()

    try:
        fa = full_array()
        print(fa)
    except numba.TypingError as e:
        print(e)
    print()

    try:
        fr = full_record()
        print(fr)
    except numba.TypingError as e:
        print()
    print()

if __name__ == "__main__":
    main()

From #5853:

The issue is that the structured type has its reference count decremented while there's still something pointing at one of its members.

That explains why zeros_array_print2 works and zeros_array_print doesn't.
I would have expected zeros to behave like ones and full do and throw a TypingError, but I guess those are different.
Closing this as a duplicate, thanks.

I would have expected zeros to behave like ones and full do and throw a TypingError, but I guess those are different.

IIRC np.zeros goes via memset(3), whereas np.ones is doing np.empty and then assigning a 1 into each element, the typing error is coming from the setitem associated with this assignment.
Thanks for closing this as a duplicate.


Add Shortest Distance in a Binary Maze

This problem uses Dijkstra algorithm to find the shortest path between source and destination cell. The path can be selected if its value is 1.
I would like to work on this issue under SSOC'23.

Greetings, I want to work on this issue. This problem can be solved using bfs. I want to work under SSOC'23

assign to me

please assign  this issue to me
2

It was shown in my quine as tasked to me.
On Fri, 23 Jun, 2023, 7:17 am Kumar Laxmikant, @.***>
wrote:

Closed #806 https://github.com/Kumar-laxmi/Algorithms/issues/806 as not
planned.
—
Reply to this email directly, view it on GitHub
https://github.com/Kumar-laxmi/Algorithms/issues/806#event-9612590189,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A7FPFQWVEPATPFG2WXRFTYLXMTYSVANCNFSM6AAAAAAYWRN7QM
.
You are receiving this because you commented.Message ID:
@.***>


[Python][C++] S3FileSystem delete_dir() regression in PyArrow 14

Describe the bug, including details regarding any error messages, version, and platform.
I think I have found a regression for S3FileSystem in PyArrow 14. Using the delete_dir() method with PyArrow 13 will remove a "directory" object and all its children, i.e. it will delete everything with the directory prefix. However when I switch to PyArrow 14 this no longer works, after calling delete_dir() the object still exists (I am checking with get_file_info() != FileType.NotFound). The issue also seems to affect delete_dir_contents(). The delete_dir() method still works on empty directories, and if I remove all the objects inside a directory and then call delete_dir() it will also work.
Could this be something to do with the AWS call for removing objects by prefix? In the native APIs for GCP / Azure this is not possible and objects need to be listed and deleted in batches, but for S3 delete by prefix is available so I'm guessing the S3 FileSystem implementation uses this feature? These low level details are hidden by the Arrow FS abstraction though, so while delete_dir() is not working there is no effective way to work around without writing an implementation in the native AWS SDK - I'd prefer not to do that for the sake of one call!
I have seen this issue on macOS with Python 3.12 and 3.10, also on Ubuntu with Python 3.11. In all cases the Arrow 13 version worked and the Arrow 14 version didn't. I haven't tried Arrow 14 on Windows but I do know Arrow 13 used to work on Windows. Also I'm assuming this is in the C implementation so it could affect more than just the Python component, but I have only tested with Python.
Please let me know if there's anything I can do to help diagnose this issue. If it is a regression and it's possible to fix with a 14.0.1 that would be amazing but I don't know how possible that is.
Component(s)
Python

Could you provide a minimal script to reproduce this problem?
If you can build PyArrow on local, could you try git bisect to detect the commit that is related to this problem?
See also: https://arrow.apache.org/docs/developers/python.html

I have created a script to reproduce the problem, the script and outputs for both version 13 and version 14 are below which show the differences. It appears to be an issue when the parent dir contains explicit sub dir objects.
In version 14, an empty dir is deleted ok and so is a dir containing blobs, but a dir containing other dirs is not. It only seems to be an issue when the sub dirs are created explicitly - if you put a blob in the dir with extra slashes in the blob name, the dir and content is deleted fine. Also if you explicitly delete the problematic sub dirs, then you can delete the parent dir.
I also had a look into the code. It does seem the S3 implementation is listing dir contents and then deleting batches of objects in DoDeleteDirContentsAsync(). So I guess either the listing part of the deleting part isn't working right when the dir contains explicit sub dir objects.
That's as far as I got before breakfast! Script and results are below. Is this enough for you to find and fix the issue or do you still need the bisection? Realistically I won't have time to set up all the tooling etc for that until the weekend.
import pyarrow.fs as pa_fs
import uuid

s3_args = {
    "region": "<region>",
    "access_key": "<access_key_id>",
    "secret_key": "<secret_access_key>"
}

s3_bucket = "<bucket>"

s3fs = pa_fs.S3FileSystem(**s3_args)

test_dir = f"test_{uuid.uuid4()}"
s3fs.create_dir(s3_bucket + "/" + test_dir)
s3fs.delete_dir(s3_bucket + "/" + test_dir)
dir_info = s3fs.get_file_info(s3_bucket + "/" + test_dir)
print(f"Single dir deleted: [{dir_info.type == pa_fs.FileType.NotFound}]")

test_dir = f"test_{uuid.uuid4()}"
s3fs.create_dir(s3_bucket + "/" + test_dir)
with s3fs.open_output_stream(s3_bucket + "/some_blob.dat") as stream:
    stream.write(b"Some data")
s3fs.delete_dir(s3_bucket + "/" + test_dir)
dir_info = s3fs.get_file_info(s3_bucket + "/" + test_dir)
print(f"Dir with content deleted: [{dir_info.type == pa_fs.FileType.NotFound}]")

test_dir = f"test_{uuid.uuid4()}"
s3fs.create_dir(s3_bucket + "/" + test_dir)
s3fs.create_dir(s3_bucket + "/" + test_dir + "/sub_dir")
s3fs.delete_dir(s3_bucket + "/" + test_dir)
dir_info = s3fs.get_file_info(s3_bucket + "/" + test_dir)
print(f"Dir with sub dir deleted: [{dir_info.type == pa_fs.FileType.NotFound}]")

test_dir = f"test_{uuid.uuid4()}"
s3fs.create_dir(s3_bucket + "/" + test_dir)
s3fs.create_dir(s3_bucket + "/" + test_dir + "/sub_dir")
s3fs.delete_dir(s3_bucket + "/" + test_dir + "/sub_dir")
s3fs.delete_dir(s3_bucket + "/" + test_dir)
dir_info = s3fs.get_file_info(s3_bucket + "/" + test_dir)
print(f"Dir deleted after subdir removed explicitly: [{dir_info.type == pa_fs.FileType.NotFound}]")

test_dir = f"test_{uuid.uuid4()}"
s3fs.create_dir(s3_bucket + "/" + test_dir)
with s3fs.open_output_stream(s3_bucket + "/sub_dir/some_blob.dat") as stream:
    stream.write(b"Some data")
s3fs.delete_dir(s3_bucket + "/" + test_dir)
dir_info = s3fs.get_file_info(s3_bucket + "/" + test_dir)
print(f"Dir deleted with blob in implicit subdir: [{dir_info.type == pa_fs.FileType.NotFound}]")

test_dir = f"test_{uuid.uuid4()}"
s3fs.create_dir(s3_bucket + "/" + test_dir)
s3fs.create_dir(s3_bucket + "/" + test_dir + "/sub_dir/")
with s3fs.open_output_stream(s3_bucket + "/sub_dir/some_blob.dat") as stream:
    stream.write(b"Some data")
s3fs.delete_dir(s3_bucket + "/" + test_dir)
dir_info = s3fs.get_file_info(s3_bucket + "/" + test_dir)
print(f"Dir deleted with blob in explicit subdir: [{dir_info.type == pa_fs.FileType.NotFound}]")

Results with PyArrow 13:
Single dir deleted: [True]
Dir with content deleted: [True]
Dir with sub dir deleted: [True]
Dir deleted after subdir removed explicitly: [True]
Dir deleted with blob in implicit subdir: [True]
Dir deleted with blob in explicit subdir: [True]

Results with PyArrow 14:
Single dir deleted: [True]
Dir with content deleted: [True]
Dir with sub dir deleted: [False]
Dir deleted after subdir removed explicitly: [True]
Dir deleted with blob in implicit subdir: [True]
Dir deleted with blob in explicit subdir: [False]

Also just to mention this does work as expected for the local and GCS implementations, only the S3 implementation is affected.

I'm seeing the same problem using PyArrow 14. 😢

This was the only significant PR merged between 13.0 and 14.0 that could possibly affect this behavior: https://github.com/apache/arrow/pull/35440

I am trying to reproduce it using our test infrastructure with minio. However, I haven't been able to do so, and so it might be specific to actual AWS S3, and not reproducible with minio.
I ran your reproducer script above using pyarrow 14.0 and a minio server running, and I get:
Single dir deleted: [True]
Dir with content deleted: [True]
Dir with sub dir deleted: [True]
Dir deleted after subdir removed explicitly: [True]
Dir deleted with blob in implicit subdir: [True]
Dir deleted with blob in explicit subdir: [True]

And I also tried to mimic the reproducer into a test function that can be dropped into test_fs.py using our test infrastructure (containing the two failure cases):
def test_delete_dir_with_explicit_subdir(fs, pathfn):
    skip_fsspec_s3fs(fs)

    d = pathfn('directory/')
    nd = pathfn('directory/nested/')

    # deleting dir with explicit subdir
    fs.create_dir(d)
    fs.create_dir(nd)
    fs.delete_dir(d)
    dir_info = fs.get_file_info(d)
    assert dir_info.type == FileType.NotFound

    # deleting dir with blob in explicit subdir
    d = pathfn('directory2')
    nd = pathfn('directory2/nested')
    f = pathfn('directory2/nested/target-file')

    fs.create_dir(d)
    fs.create_dir(nd)
    with fs.open_output_stream(f) as s:
        s.write(b'data')

    fs.delete_dir(d)
    dir_info = fs.get_file_info(d)
    assert dir_info.type == FileType.NotFound

But this passes for all tested filesystems.

I can reproduce the issue with an actual AWS S3 bucket. Enabling some debug logging, and comparing the output using pyarrow 13.0 vs 14.0 using the example of "Dir with subdir deleted" (s3fs.delete_dir(s3_bucket + "/" + test_dir)).
With pyarrow 13.0:

Listing with test_dir/ as prefix (GET with list-type), which returns a ListBucketResult
Listing with sub_dir/ as prefix
POST /?delete (passing both test_dir/ as test_dir/sub_dir/ as objects to delete with a Delete xml object
POST /?delete with only the test_dir/sub_dir/ (no idea why this is done a second time)
DELETE /test_dir/

With pyarrow 14.0:

HEAD /test_dir
Listing with test_dir/ as prefix
POST /?delete with specifying test_dir/sub_dir as single object to delete (without trailing /, is that the difference?)
DELETE /test_dir/

(and there is no indication that this last DELETE failed, but in practice it did not delete the test_dir, and neither the sub_dir was deleted)
In case this is useful for someone more familiar with S3 / the AWS SDK / our bindings.

POST /?delete (passing both test_dir/ as test_dir/sub_dir/ as objects to delete with a Delete xml object
POST /?delete with only the test_dir/sub_dir/ (no idea why this is done a second time)

I suspect it's because test_dir/sub_dir/ as the prefix leads to itself appearing in the listing response.

POST /?delete with specifying test_dir/sub_dir as single object to delete (without trailing /, is that the difference?)

Yes, this would be the issue because an empty object with key test_dir/sub_dir/ is created to indicate that the test_dir/sub_dir directory exists (something that CreateEmptyDir will do, so DeleteDir has to undo.

So before we had a custom WalkForDeleteDirAsync, after https://github.com/apache/arrow/pull/35440 DoDeleteDirContentsAsync uses a FileSelector / GetFileInfo, and that returns the paths for directories without trailing slash?

I don't know exactly what introduced the issue, but from your trace, it's clear that the DELETE for test_dir/sub_dir/ is missing. test_dir/sub_dir and test_dir/sub_dir/ are different object from S3's point of view. It could be that test_dir/sub_dir/ is present in the listing, but when the request for deletion is sent, the test_dir/sub_dir key is used instead.

If I manually ensure that the directory paths get a / appended when iterating over the file paths in DoDeleteDirContentsAsync, with the following quick patch:
--- a/cpp/src/arrow/filesystem/s3fs.cc
+++ b/cpp/src/arrow/filesystem/s3fs.cc
@@ -2408,7 +2408,11 @@ class S3FileSystem::Impl : public std::enable_shared_from_this<S3FileSystem::Imp
                   std::vector<std::string> file_paths;
                   for (const auto& file_info : file_infos) {
                     DCHECK_GT(file_info.path().size(), bucket.size());
-                    file_paths.push_back(file_info.path().substr(bucket.size() + 1));
+                    auto file_path = file_info.path().substr(bucket.size() + 1);
+                    if (file_info.IsDirectory()) {
+                      file_path = file_path + kSep;
+                    }
+                    file_paths.push_back(file_path);
                   }
                   scheduler->AddSimpleTask(
                       [=, file_paths = std::move(file_paths)] {

Then all the reproducer cases from above now work (all print True). However, not fully sure that is the correct fix? (are we sure the path never already has a trailing slash?).
It is the FileSelector that returns the paths without a / for directories:
In [18]: s3fs.get_file_info(pa_fs.FileSelector(s3_bucket + "/test_dir/", recursive=True))
Out[18]:
[<FileInfo for 'test-bucket-joris/test_dir/sub_dir': type=FileType.Directory>,
 <FileInfo for 'test-bucket-joris/test_dir/sub_dir/some_blob.dat': type=FileType.File, size=9>]

So assuming for a moment that FileSelector always does that (and we use FileSelector within DeleteDirContents right now), we can maybe assume we can always add a / to the path for directories (and also must do that to keep to the contract that "folders" in S3 are objects with a trailing /).

Opened a PR with the patch I showed above -> https://github.com/apache/arrow/pull/38845

@jorisvandenbossche I think this is the right fix. FileInfo is probably normalizing paths to "files" and "directories" to never have a trailing / like a real filesystem API would do, but to ask S3 to delete a directory (a concept they don't have), the client has to ask for the deletion of the empty object with a key ending with / as that is the convention used to represent the existence of an empty directory on an object store like S3.

This fix worked for us, thank you.


Codeformer has disappeared

@TheLastBen hey great work still.  Just noticed something that Codeformer is no longer an option for face restoration and the Restore Faces options is completely gone.
EDIT
Never mind just did a refresh and removed some extensions and working again

@TheLastBen hey great work still. Just noticed something that Codeformer is no longer an option for face restoration and the Restore Faces options is completely gone.
EDIT Never mind just did a refresh and removed some extensions and working again

Hi, one question then the SD interface already gives you the option to restore face? I have the same problem running SD and not recognizing codeformer. I hope you can help me.

I've got this going on too.

Try the latest notebook

Here, the error persists:
"Warning: CodeFormer not found at path /content/gdrive/MyDrive/sd/stablediffusion/src/CodeFormer/inference_codeformer.py
Warning: BLIP not found at path /content/gdrive/MyDrive/sd/stablediffusion/src/BLIP/models/blip.py"
and the 'restore faces' option has disappeared

Now these errors keep appearing one after the other:
Warning: CodeFormer not found at path /content/gdrive/MyDrive/sd/stablediffusion/src/CodeFormer/inference_codeformer.py
Warning: BLIP not found at path /content/gdrive/MyDrive/sd/stablediffusion/src/BLIP/models/blip.py
2023-07-20 22:38:44,851 - ControlNet - INFO - ControlNet v1.1.233
ControlNet preprocessor location: /content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/downloads
2023-07-20 22:39:23,209 - ControlNet - INFO - ControlNet v1.1.233
Loading weights [4c86efd062] from /content/gdrive/MyDrive/sd/stable-diffusion-webui/models/Stable-diffusion/model.ckpt
*** Failed reading extension data from Git repository (sd-webui-controlnet)
Traceback (most recent call last):
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/extensions.py", line 63, in do_read_info_from_repo
commit = repo.head.commit
File "/usr/local/lib/python3.10/dist-packages/git/refs/symbolic.py", line 217, in _get_commit
obj = self._get_object()
File "/usr/local/lib/python3.10/dist-packages/git/refs/symbolic.py", line 210, in _get_object
return Object.new_from_sha(self.repo, hex_to_bin(self.dereference_recursive(self.repo, self.path)))
File "/usr/local/lib/python3.10/dist-packages/git/objects/base.py", line 85, in new_from_sha
oinfo = repo.odb.info(sha1)
File "/usr/local/lib/python3.10/dist-packages/git/db.py", line 43, in info
hexsha, typename, size = self._git.get_object_header(bin_to_hex(binsha))
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/gitpython_hack.py", line 18, in get_object_header
ret = subprocess.check_output(
File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.10/subprocess.py", line 503, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "/usr/lib/python3.10/subprocess.py", line 1152, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/lib/python3.10/subprocess.py", line 2004, in _communicate
self._check_timeout(endtime, orig_timeout, stdout, stderr)
File "/usr/lib/python3.10/subprocess.py", line 1196, in _check_timeout
raise TimeoutExpired(
subprocess.TimeoutExpired: Command '['git', 'cat-file', '--batch-check']' timed out after 2 seconds

It even generates the Gradio link, but the "restore faces" option no longer appears, and even if it manages to generate an image, it produces an error and no longer works.
To use the updated notebook, wouldn't you just need to click on the link? https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/main/fast_stable_diffusion_AUTOMATIC1111.ipynb
What can I do?

I'll check it out

Fixed in the latest notebook

It worked, thank you very much! And congratulations on your excellent work @TheLastBen


Question about FP16 training

In line 399-400 of https://github.com/mit-han-lab/efficientvit/blob/master/efficientvit/models/nn/ops.py ,
it seems that we convert back to FP32 during training. Why would this be the case? Does training that part with FP16 significantly cause low accuracy?

checkout #15


Is any plan for 0.10.0 release?

Hi all,
I'm hoping the 0.10.0 final release. The new aggregation functions in 0.10.0 looks pretty good.
Is any time plan for it ?

#1690 is blocking release of 0.10.0. Presumably work on this has been delayed due to the holiday season but hopefully we should see progress on it again soon.


JavaScript: highlight keyword after dot

Keyword after dot should not be highlighted as a keyword. This is already appeared in .catch issue #1526. Current found issue is in the next code:
Array.from()
Array.forEach()

Where from and forEach highlighted as keyword and function respectively.

I can see the problem with from and other keywords but what's wrong with forEach? It's not a keyword AFAIK.

This is an example of different behavior. Highlighting should be the same in both cases.

Agreed.


Bitwise ops

This is a PR to add bitwise operations for scalars (and, or, xor, left shift and right shift).
This PR can be reviewed before the RNG pull request.
@breuleux @abergeron

Hi @breuleux @abergeron I added all suggested changes.
For relay, I added left shift and right shifts, but I did not find operations for bitwise and, or and xor.

@notoraptor I see. Seems like an odd omission. Maybe Relay's logical_and and logical_or are actually bitwise?

@breuleux Updated, I merged Int and UInt into Integral.
Relay logical_and and logical_or seem to be boolean operations. Following code fails asking for an entry to be a boolean:
from tvm import relay

def test_relay():
    x = relay.var('x', dtype='int32', shape=())
    y = relay.var('y', dtype='int32', shape=())
    z = relay.logical_and(x, y)
    f = relay.Function([x, y], z)
    mod = relay.Module({})
    fname = relay.GlobalVar('f')
    mod[fname] = f
    e = relay.create_executor(mod=mod)
    c = e.evaluate(mod[fname])
    print(c(7, 5))

if __name__ == '__main__':
    test_relay()

@notoraptor Okay, good to know. We'll have to wait until these operations are available in Relay.

@breuleux updated!


Improve speed by reduce stat syscall

https://github.com/petdance/ack/issues/149

https://github.com/petdance/ack3/issues/44


What-if function

As we have history now we can proceed with what-if. Keep in mind #4 as it's might be related to that

Too soon to create issues


Poisson regression exposure as offset

I am trying to model data where my outcome is also defined by a varying exposure. This means that each sample measures a count over a different amount of time in days. In traditional Poisson models, this can be handled by an offset with constant coefficient of 1. For details see:
https://en.wikipedia.org/wiki/Poisson_regression#.22Exposure.22_and_offset
Is this somehow possible to incorporate in Lightgbm?

@pford221 Yes, this is definitely an option and I have thought of it. I will continue to play around with it. However, there seem to be some differences in how GBT utilizes the weight compared to e.g., a GLM model, see: https://stats.stackexchange.com/questions/325008/why-is-xgboost-giving-me-seriously-biased-predictions-with-small-nrounds
I am also wondering whether lightgbm has a similar functionality as xgboost to set a base margin.

@psinger you can use init_score, which is set per-sample.

@guolinke Thanks!


Wait for new hotplug attachment pod to be ready

What this PR does / why we need it:
Before deleting the old attachment pod, wait for the new attachment pod to be ready, so k8s will not detach the volume from the node since there will always be a pod using the volume from its perspective. This should prevent issues with certain CSI drivers that don't respect the device being in use before detaching it from the node. This will now break csi drivers that don't allow multiple pods on the same node to use the volume (this is a bug in that driver).
Fixed issue where when adding or removing a volume the existing volumes would still have the UID of the old attachment pod in the VMI status which caused errors to appear in the virt-handler logs about not being able to find the device or image. This can be observed with SyncFailed events on the VMI
Fixed issue where the cleanup would attempt to remove a volume that was already gone causing errors to appear in the virt-handler log. This can be observed with SyncFailed events on the VMI
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
Special notes for your reviewer:
Release note:

Ensure new hotplug attachment pod is ready before deleting old attachment pod

Looks like I have some flaky tests, investigating

/hold
feel free to remove hold when flaky test is sorted out

/retest-required
/hold cancel

/test pull-kubevirt-e2e-k8s-1.26-sig-compute

/test pull-kubevirt-e2e-k8s-1.26-sig-storage
/test pull-kubevirt-e2e-k8s-1.27-sig-storage

Agreed, I ran through a bunch of tests and I believe you are right not sure what we were testing there, but already exists cannot happen.

/test pull-kubevirt-e2e-kind-1.27-vgpu

/cherrypick release-1.0

Similar Design like this pr #6728

Yes, with one difference, we ensure the new pod is in running state before deleting the old pod. IIRC that PR didn't do that. https://github.com/kubevirt/kubevirt/pull/10567 is an improvement adding a rate limit to how fast we switch attachment pods.


Implemented fix for min/max for Floats, Doubles and Items Sorted Views.

Also fixed a problem I discovered in the deserialization for Boolean quantiles, but it would also have shown up with other types. The data was ok, but the max value would have been incorrect.
Also fixed some problems with some of the tests.
Along the way I also replaced "back-slash-n" with platform independent "LS" in the files I touched.
Also moved some functions like getCDF(..) and getPMF(...) into parent interfaces as default functions. Now the three parent interfaces for Doubles SV, Floats SV and Items SV are all code-parallel.

Changes by file:
/kll/
KllDirectCompactItemsSketch

Fix for getMaxItem

KllDoublesSketch

Propagation of new SV constructor signature

KllFloatsSketch

Propagation of new SV constructor signature

KllHeapDoublesSketch

Removed //TODO

KllItemsSketch

Fixed a problem in toString(...)

KllMemoryValidate

add code doc for better understanding

/quantiles/
DoublesSketch

Propagation of new SV constructor signature

ItemsUtil

Removed obsolete code comment

/quantilescommon/
DoublesSketchSortedView

Revised the main constructor to include Min/Max fix.

Added separate test constructor also with Min/Max fix

Removed unneeded class members for min/max

FloatsSketchSortedView

Revised the main constructor to include Min/Max fix.

Added separate test constructor also with Min/Max fix

Removed unneeded class members for min/max

GenericSortedView

Moved getCDF() and getPMF() from ItemsSketchSortedView to here as an interface default implementation.  This is consitent with how floats and dobles are implemented.

IncludeMinMax

A new class that consolidates all new include Min and Max code for floats, doubles and item in one place.

ItemsSketchSortedView

Revised the main constructor to include Min/Max fix.

Added separate test constructor also with Min/Max fix

Removed unneeded class members for min/max

/req/
ReqSketch

Propagation of new SV constructor signature

/test/...
/kll/
KllDirectCompactItemsSketch

Minor restructuring and added diagnostic print statements

KllMiscDoublesTest

Redesigned a test to accommodate new min/max adjustments

Apply "/n" to "LS" corrections

KllMiscFloatsTest

Redesigned a test to accommodate new min/max adjustments

Apply "/n" to "LS" corrections

KllMiscItemsTest

Redesigned a test to accommodate new min/max adjustments

Apply "/n" to "LS" corrections

/quantilescommon/
IncludeMinMaxTest

tests for the new IncludeMinMax class

We changed the behavior of the sorted view to ensure the min and max values are always represented, which, we believe, will provide a more practical information to users of these distributions without distorting the resulting data. -- @jmalkin


option for ctrl/cmd modifier when clicking link

Requires https://github.com/JunoLab/atom-ink/pull/251.

@pfitzseb is it possible/easy to suppress underline highlighting on link-hover when without ctrl/cmd modifier ?
Currently the highlight is invoked always on link-hover and could be misleading I guess.


feat(EMI-1778): list privacy popover

This PR resolves EMI-1778 and EMI-1786
Description
This implements the list privacy eye icon popover and fixes some copy issues.
https://github.com/artsy/eigen/assets/50849237/75124963-f62f-4d8b-97d3-01ecb1a0b005

PR Checklist

[x] I have tested my changes on iOS and Android.
[x] I hid my changes behind a feature flag, or they don't need one.
[x] I have included screenshots or videos, or I have not changed the UI.
[x] I have added tests, or my changes don't require any.
[x] I added an app state migration, or my changes do not require one.
[x] I have documented any follow-up work that this PR will require, or it does not require any.
[x] I have added a changelog entry below, or my changes do not require one.

To the reviewers 👀

[ ] I would like at least one of the reviewers to run this PR on the simulator or device.

Changelog updates
Changelog updates

Cross-platform user-facing changes

artwork list privacy popover

iOS user-facing changes

Android user-facing changes

Dev changes

Need help with something? Have a look at our docs, or get in touch with us.

This PR contains the following changes:

Cross-platform user-facing changes (artwork list privacy popover)

  Generated by :no_entry_sign: dangerJS against 60144ad90bfd7e30333c9526f9e78aa82e95192f


Remove validation of Community Progress report excel dates

As an analyst
I want to be able to upload the community progress report even if the dates are not filled in
so that there aren't unnecessary validation errors
Description:
Currently if all the dates are not filled out in the community progress report then the user can't upload the report successfully.
This ticket is to remove the validation on the dates being filled out.  If there is validation currently on the date fields being formatted correctly then keep the validation on formatting.
Acceptance Criteria

[ ] user can upload a community progress report even if dates are null without getting validation errors

Hey team! Please add your planning poker estimate with Zenhub @AntBush @marcellmueller @rafasdc

Just flagging that without completing this card, the first Community Progress Reports will be blocked from being uploaded.


Drafted API top-level documentation (FOR DISCUSSION)

This PR is not for merging, it's for discussion. We've talked a bit about top-level documentation. But what would it contain? Have a look and comment and what you think it should contain.

Good starting point. I think we probably need to describe what the sources are and how they relate to the existing SA forms, similarly how do the summaries relate to those forms? Basically how does someone familiar with the current SA process start using this?


[Enhancement]: Adding Navbar to the Homepage for better user experience

Do you want to have the enhancement of existing game ? 😀 Describe yourself..
I want to add Navbar with features like home, contact us, contributors,user manual, and feedback
Describe the solution you'd like
Solution steps - for this task I will use HTML & CSS with its required properties to create a Navbar and will also add hover effect to make it more attractive.Please assign me this task @kun
Select program in which you are contributing
No response
Code of Conduct

[X] I follow CONTRIBUTING GUIDELINE of this project.

Hey @HarshitGupta1511 !
Thank you for raising an issue 💗
You can self assign the issue by commenting /assign in comment 😀
Make sure you follow CODE OF CONDUCT and CONTRIBUTING GUIDELINES 🚀
Don’t Forget to ⭐ our GameZone🎮
Make sure you join our Discord🕹

/assign

/assign

Hey @thisismevri ! , This issue is already assigned to @HarshitGupta1511. 💗
You can work on other issues 🚀
If there is anything wrong you can contact us on Discord🕹👀

Hey @HarshitGupta1511 ! Thank you so much for your raising the issue💗
It’s all yours, you can come anytime again and make some contributions! 🚀
Alone, we can do little, but together we can do so much! 😇


Cyrillic is not supported

Terms

[X] Before you openning this issue, i have checked if the issue has already been reported.

Bagisto Version(s) affected
v2.1.2
Issue Description
I can't create the category with a Russian name.
EN - Organization and storage of things
RU - Организация и хранение вещей
Creating products in Russian also does not work correctly (URLs).
Preconditions
When you create the new category with EN text it works, but Russian does not.
Steps to reproduce

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected Result
Russian text
Actual Result
When I change the language for text in Category
EN - Reloading page
RU - Screenshot
For Chinese the same - Undefined array key "zh_CN"
Screenshots

No response
Additional context
No response

Thank you for the reply.
You can find my problem here.
https://disk.yandex.ru/i/NHayFEYMG-FijA

@plyaskin
The issue to which you are referring in the video has already been fixed. However, upon checking your concern, we need to discuss the slug-related issue in the Category for different locales. We will discuss this and inform you accordingly.
issue fixed in following PR -> https://github.com/bagisto/bagisto/pull/9613

ALready fixed in the master

@jitendra-webkul It is still there - the Cyrillic Slug is corrupted "website.com/-".

Will look into it

Hi @jitendra-webkul
I'm not sure that it should save Cyrillic or another language in the URL address, because some URLs will not opened in this case...
It will be better to save it in English.

You will still have the option to save in the English.

But also customers will have errors with other languages... and a lot of reports to you.
Exclude the possibility of an error.

Could you confirm if the URL address supports Hindi, Hebrew, Arabic or Cyrillic symbols?
Did you try to open the product for 19 available locales in the browser with a localised slug?
I'm not sure that this URL will be opened and will not back an error like "The page not found"

Tested with Hindi, Hebrew, Arabic, now it's working.


fix(invidious): update border colors and remove scrollbar changes

🔧 What does this fix? 🔧
Updates the border colors from crust to surface1/2, removes scrollbar changes which went beyond just color customization.
🗒 Checklist 🗒

[x] I have read and followed Catppuccin's contributing guidelines.
[x] I have updated the version appropriately in the ==UserStyle== header of the catppuccin.user.css file.

While we are on this userstyle, I cant think of any reason you would ever not want styleVideoPlayer enabled? We don't have an option for it on the YouTube userstyle, I would opt for removing it altogether (it defaults to on anyway).


Show warning message in standard way if the SMS content is empty when sending

We need to show warning message by using standard way.

I think we can add Integration Test for this scenario.

Fixed with #102

@embbnux Looks like Integration Test is not added.

There are two verfications on text empty before, one is browser required feature and one is standard way.
There are Integration Test test on text empty with standard way.
But Integration Test don't test browser way. Now the browser way is removed.

ok. It makes sense.


Gray Creeper Heads

Minecraft v1.19.2
Forge v43.1.52
Forgery v3.1.3

When the option "Foliage Creepers" is enabled, creeper heads render as gray in the inventory and when placed.
Debug Log

the 1.19.2 implementation of foliage creepers is super jank because mojang changed stuff and i was loosing my patiance i need to re do it at some point

i'm not getting gray creeper heads though.

please try without any other mods installed, see if it's a conflict?

Hmm, I'll try without other mods. I should also mention this was on a server

This is odd, I reloaded the game, and now the heads are rendering normally.

I reloaded the game & server with the same mods, and once again the head is gray

1.19 foliage creepers was a disaster, should be fixed in the next release

Thanks!


Exclude GA in DEV_MODE

Story
No response
Description
It would be nice to exclude GA data reporting when we are developing the website in DEV_MODE on localhost. This way we won't capture localhost data in the GA data.

@iskerrett I filtered this on GA I think, is this still an issue @iskerrett ?

localhost still shows up in GA so it is still an issue.


Installing Autodock Vina for use in Chimera?

I've downloaded the mac os file "vina_1.2.3_mac_x86_64" and put it into a folder on my mac. I've set the Autodock Vina to run as a local file and pointed Chimera to the file I downloaded, but when I try to run it I get an error; "vina_1.2.3_mac_x86_64" is not an executable". Am I misunderstanding the instructions or does Vina 1.2.3 no longer run inside Chimera?

Maybe you need to give it permission to execute:
chmod +x vina_1.2.3_mac_x86_64

Thanks for this. My mac will now run it (after I overrode the non-verified developer bar), but it fails with a "no output" error. Are there some instructions that show non-specialist users how to run the newer versions of Vina on macs?

We have information on how to run vina from the command line, which identical to vina v1.1.2 for many use cases: https://autodock-vina.readthedocs.io/en/latest/docking_basic.html

Is Chimera raising the "no output" error?
Does vina print anything when executed directly in the command line?

Yes, the "no output" error occurs in Chimera. Does this mean that the newer versions of Vina cannot be run in Chimera on a mac? Version 1.1.2 and the windows version of 1.2.3, run in Chimera.
I'm sorry I don't have any experience of running Vina from the terminal on a mac. The instructions you cite say that an additional software suite is also required, ADFR? But there are warning messages about that not running in some versions of mac os? Is there somewhere I find instructions on the syntax etc for use in running Vina from the terminal?

I don't know if the newer version of Vina can run in Chimera.
ADFR has nice tools for receptor preparation, but it is possible to prepare receptors in others ways, with MGLTools or OpenBabel. So it is possible to run Vina without ADFR.
The docs I shared contain the instructions for running Vina from the terminal.


Question with an address input form

Is it possible to render a question that asks for a postal address?
Something like this (screenshot attached):

Hi,
There is a width property, that should make it work. It has the default value is 100%, but it can be 50%. Unfortunately, it doesn't work. I have just checked it. I will try to fix it on this week.
There is no ability to show question label under a question. Do you need this functionality? Do you need it on the survey level or page level?
Thank you,
Andrew

Having a separate label under a question would be a nice feature.
What do you mean by "on the survey level or page level" ?

Survey level: title will be placed under questions for entire survey.
Page level: title will be placed under questions for a specific page. You will have to set the property for every page, in case you want to change it for all questions in the survey.

I will try to implement the same layout in a demo. I will keep you inform. I will likely do it on the next week, The width property will not make a trick. It should be something like, "keep on the same line" property.
Thank you,
Andrew

The first feature is implemented: https://github.com/andrewtelnov/surveyjs/issues/64

FYI: I am working on several questions in one row (single line). I hope to show something in couple days.
Thank you,
Andrew

The fun is starting when the question visibility is changed :)

https://github.com/andrewtelnov/surveyjs/issues/65
I have to change the survey visual editor now to finish with the feature.

It has been implemented. The example


Can't add package to XCode project

axmol version: 2.1.3
devices test on:
developing environments: XCode 15.4
Steps to Reproduce:

Create a new project (axmol new -p com.new.www -d ./ -l cpp Coloring -i)
Create an XCode project for the project (axmol build -p ios -a arm64 -c)
Try to add a package dependency to the project, specifically: https://github.com/tikhop/ASN1Swift
XCode stuck on "Preparing to validate..."

If I create a new random project in XCode I can add the package with no problem.

Just tested on axmol 2.1.4 and the problem persists.

I don't know how packages work in Xcode, but if they integrate into your Xcode project, then they will be gone the next time Xcode project is regenerated (e.g. after adding new source files, etc). This is not how Axmol workflow works. You should treat generated Xcode project as "readonly". You'll probably want to look into something that can be integrated from command line like CocoaPods.

Oh...
So it's not like the XCode project Cocos2d-x made where I edited it directly?

So it's not like the XCode project Cocos2d-x made where I edited it directly?

Cocos2d-x v3.17+ and V4 both use cmake, which generate the Xcode project, and Axmol also uses the cmake build system, so nothing has changed in that regard.  If you're referring to v2 of Cocos2d-x. then yes, a lot has changed.  Anything you want to add to the Xcode project would need to be done via cmake or CocoaPods.
The ASN1Swift project you linked has an example of how to add it via CocoaPods.

To expand on this: yes, you should not edit Xcode project directly, and do not add it to your source control. Basically, you want to have an easy runnable two steps setup:

Generate Xcode project - axmol build -p ios -a arm64 -c or similar.
Modify Xcode project - your custom automated step that makes the changes to the Xcode project. This could be done using CocoaPods or something else.

I was a bit wrong to say "treat generated Xcode project as "readonly"". You can certainly modify it, but it must be automated, cause you will lose changes the second that you modify CMake files, or CMake runs ZERO_CHECK task for some reason. It also hardcodes paths, so you would not be able to build that project on another machine, or if you moved your project to another folder.
Also, be aware that if you're running build in Xcode after editing CMake files (or if CMake reruns ZERO_CHECK task for some other reason), then Xcode project will be regenerated during the build, and Xcode will notice it and it will cancel the build to reload it. Then you need to execute your step 2, since your Xcode project changes will be gone, and then restart the build.

So it's not like the XCode project Cocos2d-x made where I edited it directly?

Cocos2d-x v3.17+ and V4 both use cmake, which generate the Xcode project, and Axmol also uses the cmake build system, so nothing has changed in that regard. If you're referring to v2 of Cocos2d-x. then yes, a lot has changed. Anything you want to add to the Xcode project would need to be done via cmake or CocoaPods.
The ASN1Swift project you linked has an example of how to add it via CocoaPods.

I am working with cocos 3.10 which created an XCode project when I created the project and I've been working on it directly all the time and it worked fine. I thought I could do the same with axmol.

To expand on this: yes, you should not edit Xcode project directly, and do not add it to your source control. Basically, you want to have an easy runnable two steps setup:
1. Generate Xcode project - `axmol build -p ios -a arm64 -c` or similar.

2. Modify Xcode project - your custom _automated_ step that makes the changes to the Xcode project. This could be done using CocoaPods or something else.

I was a bit wrong to say "treat generated Xcode project as "readonly"". You can certainly modify it, but it must be automated, cause you will lose changes the second that you modify CMake files, or CMake runs ZERO_CHECK task for some reason. It also hardcodes paths, so you would not be able to build that project on another machine, or if you moved your project to another folder.
Also, be aware that if you're running build in Xcode after editing CMake files (or if CMake reruns ZERO_CHECK task for some other reason), then Xcode project will be regenerated during the build, and Xcode will notice it and it will cancel the build to reload it. Then you need to execute your step 2, since your Xcode project changes will be gone, and then restart the build.

I have never worked directly with CMake. I created my project with cocos 3.10 and it created an XCode project and I've done all my work through XCode and it worked great.
Is there a way to work with axmol the same way?

I am working with cocos 3.10 which created an XCode project when I created the project and I've been working on it directly all the time and it worked fine. I thought I could do the same with axmol.

When the cmake build system was added to Cocos2d-x v3, the intention was never to use it in the way you are describing, and you would not be doing yourself any favors by attempting to continue working in this way.  The fact that it worked without issues is purely by chance.
Embracing the cmake build system should make things simpler, and less time consuming, especially if you wish to support multiple platforms for your application.
@halx99 I feel that this should be moved to the discussion section, since it's not an issue with Axmol.


Firmware/src/drivers/distance_sensor/pga460/CMakeFiles/drivers__pga460.dir/build.make:62: recipe for target 'Firmware/src/drivers/distance_sensor/pga460/CMakeFiles/drivers__pga460.dir/pga460.cpp.o' failed make[2]: *** [Firmware/src/drivers/distance_sensor/pga460/CMakeFiles/drivers__pga460.dir/pga460.cpp.o] Error 1 CMakeFiles/Makefile2:7339: recipe for target 'Firmware/src/drivers/distance_sensor/pga460/CMakeFiles/drivers__pga460.dir/all' failed

@lioneleeedadad What is the question? I don't see anything in the description

I can't do catkin_make .. and this error is appearing

there is an error in Firmware !

If you are doing catkin_make you clearly are not following the setup process on the readme of this repo.
Could you try following the readme?
In order for me to help, you need to "describe" your problem in the description. as detailed as possible- what you are trying to do, how you tried and what is not working.

I followed the setup process how I now cloned the rqt reconfigure and other packages related to rqt and then I'm trying to do that catkin_make and I'm recieving output
cc1plus: error: assuming signed overflow does not occur when assuming that (X + c) < X is always false [-Werror=strict-overflow]
compilation terminated due to -Wfatal-errors.
cc1plus: all warnings being treated as errors
Firmware/src/drivers/distance_sensor/pga460/CMakeFiles/drivers__pga460.dir/build.make:62: recipe for target 'Firmware/src/drivers/distance_sensor/pga460/CMakeFiles/drivers__pga460.dir/pga460.cpp.o' failed
make[2]: *** [Firmware/src/drivers/distance_sensor/pga460/CMakeFiles/drivers__pga460.dir/pga460.cpp.o] Error 1
CMakeFiles/Makefile2:10238: recipe for target 'Firmware/src/drivers/distance_sensor/pga460/CMakeFiles/drivers__pga460.dir/all' failed
make[1]: *** [Firmware/src/drivers/distance_sensor/pga460/CMakeFiles/drivers__pga460.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

Also this issue is related to PX4 avoidance package thats why I'm asking here .. So please help me :)

This is much more clear.
So to make this clear. This is not an issue with the avoidane package itself, but a problem with your installation.
I see a few problems from your log:

where is your firmware located? It shouldn't be in the catkin_ws Please follow the installation instructions precisely on the devguide: http://dev.px4.io/en/setup/dev_env_linux.html
You shouldn't be using catkin_make as the avoidance package builds on catkin build You cannot use both in the same workspace. You need to clean the workspace before you do any builds
I am curious if you were able to run any ros node with your system? If not, to verify everything is fine try following one of the ROS examples so that at least you are sure everything is alright with ROS-and rqt reconfigure

My Firmware is in catkin_ws/src/

@lioneleeedadad Exactly in shouldn't be there

I will try to add it in its right place and tell you what happened

@lioneleeedadad Please use discuss.px4.io for help instead of opening new issues. We will close and lock issues which do not include information and do not follow standard guidelines.


Adds free function tests for serializer

Submission Checklist

[x] Run unit tests: ./runTests.py src/test/unit
[x] Run cpplint: make cpplint
[x] Declare copyright holder and open-source license: see below

Summary
For the refactoring of transform_inits() in the compiler we actually need the free functions to be in the serializer. See the discussion here for more info.
Intended Effect
Allow the serializer to perform the free'ing of constrained variables
How to Verify
Tests were added to do the same checks as the deserializer's free tests
./runTests.py ./src/test/unit/io/serializer_test.cpp

Side Effects
Documentation
Documentation for each of the free functions in serializer
Copyright and Licensing
Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company): Steve Bronder
By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:

Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

@rok-cesnovar or @t4c1 do one of yinz have time to look this over?

I can take a look.

@t4c1 made the above fix ready for another look!


Vent Device Type

I am looking at a way to add Keen Home to HomeBridge. Is there such thing as a Vent Device type? It operates like a dimmer bulb and also has a temperature sensor in it.
I'd like to work on adding this https://github.com/sibartlett/homebridge-wink3 but can't seem the best device type to add it as.

Have you taken a look at the Fanv2 or the HeaterCooler Service?

Yea, but they both have power on states, the vents are more just open or close.

How about the Slat service?


--configfile flag is broken in combination with Kubernetes and GS buckets

Snakemake version

snakemake=6.0.5
python-kubernetes=12.0.1

Describe the bug
Excpected:
Executing a Snakemake workflow using Kubernetes and GS buckets with --configfile.
Content of config file is available in snakemake workflow as config.
Actual:
Snakemake can't find the config file and is shutting down. It is working when providing the parameters with the --config flag.
Logs

Snakemake log
Building DAG of jobs...
Unable to retrieve additional files from git. This is not a git repository.
Using shell: /usr/bin/bash
Provided cloud nodes: 1
Job counts:
        count   jobs
        1       all
        1       cat_dub
        1       fastqc
        3
Select jobs to execute...

[Sat Mar 13 14:48:46 2021]
rule cat_dub:
    input: bucket-name/path/to/some.fastq
    output:  bucket-name/output/sample.fastq,  bucket-name/output/somefile.txt
    jobid: 2

                cat  bucket-name/path/to/some.fastq >  bucket-name/output/sample.fastq
                touch  bucket-name/output/somefile.txt

Get status with:
kubectl describe pod snakejob-id
kubectl logs snakejob-id
[Sat Mar 13 14:48:56 2021]
Error in rule cat_dub:
    jobid: 2
    output:  bucket-name/output/sample.fastq,  bucket-name/output/somefile.txt
    shell:

                cat  bucket-name/path/to/some.fastq >  bucket-name/output/sample.fastq
                touch  bucket-name/output/somefile.txt

        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
    jobid: snakejob-id

For details, please issue:
kubectl describe pod snakejob-id
kubectl logs snakejob-id
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /home/user/kubertest/.snakemake/log/2021-03-13T144842.529963.snakemake.log

Error message with kubectl logs {snakejob}
Traceback (most recent call last):
  File "/opt/conda/envs/snakemake/lib/python3.9/site-packages/snakemake/io.py", line 1564, in _load_configfile
    with open(configpath) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/user/kubertest/config.yml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/snakemake/bin/snakemake", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/envs/snakemake/lib/python3.9/site-packages/snakemake/__init__.py", line 2592, in main
    success = snakemake(
  File "/opt/conda/envs/snakemake/lib/python3.9/site-packages/snakemake/__init__.py", line 505, in snakemake
    overwrite_config.update(load_configfile(f))
  File "/opt/conda/envs/snakemake/lib/python3.9/site-packages/snakemake/io.py", line 1594, in load_configfile
    config = _load_configfile(configpath)
  File "/opt/conda/envs/snakemake/lib/python3.9/site-packages/snakemake/io.py", line 1589, in _load_configfile
    raise WorkflowError("{} file {} not found.".format(filetype, configpath))
snakemake.exceptions.WorkflowError: Config file /home/user/kubertest/config.yml not found.

Minimal example

Snakefile
rule all:
        input:
                zip = "output/sample_fastqc.zip",
                html = "output/sample_fastqc.html"

rule cat_dub:
        input:
                a =  config['input'],
                c = config['random'],
        output:
                b = "output/sample.fastq"
        threads: 1
        shell:
                """
                cat {input.a} > {output.b}
                touch {input.c}
                """

rule fastqc:
        input:
                a = rules.cat_dub.output.b
        output:
                zip = "output/sample_fastqc.zip",
                html = "output/sample_fastqc.html"
        threads: 1
        conda: "envs/fastqc.yml"
        shell:
                "fastqc {input.a} "

config.yaml
input: 'path/to/some.fastq'
random: 'output/somefile.txt'

envs/fastqc.yaml
name: fastqc
channels:
  - bioconda
dependencies:
  - fastqc

Execute with --configfile flag -> failed
snakemake  --default-remote-provider 'GS' --default-remote-prefix 'bucket-name' --cores 1 --use-conda --kubernetes --configfile config.yml

Execute with --config flag -> works
snakemake  --default-remote-provider 'GS' --default-remote-prefix 'bucket-name' --cores 1 --use-conda --kubernetes --config input="path/to/some.fastq" random="output/somefile.txt"

Additional context
It seems that snakemake is trying to read the config file in the Kubernetes pod, but it fails because it is not listed in the bucket or the pod. I would expect the same behavior from the --config and --configfile flag.

Not sure if I should open a new issue or post in this one. Did anyone ever find a way to get this working on kubernetes?
I am getting this error using snakemake==7.12.1. My job is failing if I use --kubernetes and --configfile but not if I use the --config flag. Error message is a little different and I can't find the pod logs (not sure the pod is starting)
Logs
Get status with:
kubectl describe pod snakejob-c3426f86-5dbc-58ab-925f-25e009e1838b
kubectl logs snakejob-c3426f86-5dbc-58ab-925f-25e009e1838b
[Wed Aug 24 15:58:05 2022]
Error in rule r1:
    jobid: 1
    output: mybucket/testfile.txt
    shell:
        echo config['test'] > mybucket/testfile.txt
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
    jobid: snakejob-c3426f86-5dbc-58ab-925f-25e009e1838b

Unknown pod snakejob-c3426f86-5dbc-58ab-925f-25e009e1838b. Has the pod been deleted manually?

Minimal example
command:
snakemake --default-remote-provider GS --default-remote-prefix mybucket --jobs 1 --kubernetes --configfile config.yaml

Snakefile:
# # Still fails using `--configfile` if I set config['test'] in the snakefile
# config["test"] = "hello

rule all:
    input:
        f = "testfile.txt"

rule r1:
    output: "testfile.txt"
    shell:
        "echo config['test'] > {output}"

config.yaml:
test: hello

context

My one addition to this thread is that the job still fails with the --configfile flag even if I set config["test"] = "hello" directly at the top of the Snakefile.
config.yaml is tracked in git if that matters

My workaround at the moment is to manually load the contents of the config at the top of my snakemake:
with open("config.yaml", "r") as f:
    config = yaml.safe_load(f)

Hello everybody,
thanks for highlighting this issue, I'm experiencing the exact same problem with --kubernetes and --configfiles using a bare metal cluster with a custom S3 implementation and previously to this thread I was not aware about the two conflicting flags.
@peter-fsdx's workaround about manually reading the yaml file works, thank you.


Remove unneeded byte[] allocation from TypeCodec

The last allocation remaining in TypeCode.DecodeDouble() after @alprema improvements can be removed using a conversion struct. The benchmark now runs 70% faster than the baseline instead of 54%.
TypeCode.DecodeFloat() was also updated to benefit from the same enhancements.

I signed the contribution license agreement. Is the label supposed to be automatically removed?

Yes, it should have happened automatically, but you've triggered a bug. Should be fixed now.

Those changes result from benchmarks that we ran using our read and write use cases.
Both are time series related (supposedly the most common pattern on Cassandra), and some of the changes we made were quite trivial, which makes me wonder how the driver is benched on Datastax's side. Do you do stress testing? On what kind of data? Do you compare the results to other drivers (our benchmarks ran between 2 and 3 times faster on the Java driver)?
More transparency on the performance of the driver would be appreciated.

I created a micro benchmark of the 2 decoding strategies, the one that is proposed here and the current one.
https://gist.github.com/jorgebay/445d9ba9c0ad371c5796
It seems that there isn’t a significant improvement between the two. In terms of mean, both methods show similar values. In median, using physical layout shows some improvement. I think the readability of the code should not be compromised to deliver this small performance improvement (if any).
About driver benchmarking, we are working on it, it is high in our priority. We will share the results when we have them. We are now focusing on Cassandra 3.0 support.
As we discussed on the mailing list, there are 2 features that are implemented on the java driver that are not yet implemented on the C# side that significantly affects throughput:

JAVA-562 / CSHARP-244: Message coalescing can increase throughput from ~ 50% to ~ 100%. I've implemented it in a dev branch and the results are consistent with those figures (around 80% more throughput with coalescing enabled and tcp no delay ON). It reduces the payload overhead (tcp + ip headers) and the amount of sys calls. You can pull it and compare results.
Buffer pooling / CSHARP-270: Netty 4 provides a buffer pooling implementation. Buffer pooling can reduce CPU usage, memory fragmentation and GC pressure.

Hi,
I just ran your code as is and the result is about 15% in favor of the physical layout, changing the inner iteration count to 1000 to give more time for GC to factor in gives about 47% difference (benchmarking such code with 2000 iterations is clearly not representative).
Finally, removing Linq from the benchmark loop (as follows) gives about 80% difference.
                    for (var j = 0; j < values.Length; ++j)
                        DecodeFunc(values[j]);

instead of
                    values.Select(DecodeFunc).Sum();

(at those speeds, Linq accounts for about half the time of the measurement).
I understand your concern about readability of the code, but when writing serialization code that is called millions of times per seconds (and that is a bottleneck to a very common use case), performance can be chosen over readability.
I am at your disposal if you need more details on my results / tests.

Showing that a Linq iteration of a couple of values affects the results of the microbenchmark is exactly what my point is: the improvement is small and gets diminished when paired with the rest of the tasks that need to be done within the driver. (ie: thread (Task) scheduling; managing streamIds / inflight requests).
Anyway, it's a valid possible improvement and I created the ticket CSHARP-357 for it were we can continue discussing and plan for it.
We should create 3 structures for (2 bytes, 4 bytes and 8 bytes) all numeric types, not just double and float.


Add Florianópolis (SC) spider

Yay, first lines of code in this awesome project!
Some doubts:
1. About powers field
Florianópolis seams to have an unique gazette for all powers (for example, in this issue we can see in the table of contents that it covers the executive and the legislative). This I hardcoded powers=None – is that ok or is there a better way to record that?
UPDATE Thanks for the tip, @brunolellis : )
2. About is_extra_edition fields
I browsed Florianópolis portal a bit and I've found no extra issues so far. Thus I also hardcoded is_extra_edition=False – is that ok or is there a better way to ponder on that?
UPDATE I've found one, now I know how to parse it.
3. How to know if the spider is working?
I ran it manually as I haven't seen anything in the logs of docker-compose run:
$ docker-compose run --rm processing bash
root@processing # cd data_collection/gazette
root@processing # scrapy runspider spiders/sc_florianopolis.py

Was there a better way to test it?

também tive essa dúvida
Entra na pasta data_collection para ter o comando crawl disponível
/mnt/code/data_collection# scrapy crawl  sc_florianopolis

Na live o Irio fazia um import no ipdb e fazia os debugs direto no console.

Cheers, @brunolellis – fixed :tada:

@cuducos

How to know if the spider is working?

How I personally do it at this moment:
$ docker-compose run processing sh

$ cd data_collection
$ scrapy crawl <spider_name>


mongodb support

Is it possible to add xray tracing to https://github.com/mongodb/mongo-go-driver ?

Hi @iain-raidiam
MongoDB instrumentation is currently not on X-Ray SDK's roadmap and I can't give an ETA for it at the moment.
As a potential workaround, you can use OpenTelemetry Go as it already has the MongoDB instrumentation. Using OpenTelemetry, you can still send traces to X-Ray since now we have a first class support.
More info on AWS Distro for OpenTelemetry Go: https://aws-otel.github.io/docs/getting-started/go-sdk