Increasing payload size for microROS publisher

Describe the bug
I'm trying to publish a ros_msg of type sensor_msgs__msg__LaserScan from mu olimex-stm32-e407 dev. board.
I'm only interested in sending the .ranges & header.stamp for now, meaning the intensities is only a pointer to 1 value.
System information:

OS: NuttX running on Olimex-stm32-e407
ROS 2 Dashing
microROS Dashing branch

I'm using the micro ROS publisher example to publish the new type: (Note the ROS handling is in its own class, but i dont think that matters)
#include <sensor_msgs/msg/laser_scan.h>
...
rv_ = rcl_publisher_init(&publisher_,
                            &node_,
                            ROSIDL_GET_MSG_TYPE_SUPPORT(sensor_msgs, msg, LaserScan),
                            "bea_lzr_scan",
                            &pub_ops);

To allocate memory for my laser_scan msg i do:
const uint8_t LASER_MSG_SIZE = 120;   //or the size im testing
...
laser_scan_msg_ = *sensor_msgs__msg__LaserScan__create(); //this happens in the initializer list
laser_scan_msg_.ranges.data = (float *)calloc(LASER_MSG_SIZE, sizeof(float));
laser_scan_msg_.ranges.capacity = LASER_MSG_SIZE;

I fill the msg with dummy data like so:
laser_scan_msg_.ranges.size = 0;
size_t i;
for (i = 0; i < laser_scan_msg_.ranges.capacity; i++) // every datapoint is 2 bytes
{
   laser_scan_msg_.ranges.data[i] =
       1.0;
   laser_scan_msg_.ranges.size++;
 }

The breakpoint of LASER_MSG_SIZE where the publisher fails is: 111
Ideally, I would like to send 400 floats of 4 bytes
During debugging I've narroved it down to to an error coming from:
mcu_ws/uros/rmw_microxrcedds/rmw_microxrcedds_c/src/rmw_publish.c::49
--> mcu_ws/eProsima/Micro-XRCE-DDS-Client/src/c/core/session/session.c::766
--> mcu_ws/eProsima/Micro-XRCE-DDS-Client/src/c/core/session/stream/output_reliable_stream.c::49
It's something with the lenght, block_size or remaining blocks.
So my question:
Is there anywhere I can increase the allowed payload size for RMW?

Hi @krm-mir,
The payload size is calculated at building time, when this file is generated:
https://github.com/micro-ROS/rmw-microxrcedds/blob/80a305b05dec7ff0954c91726db78418fbf07661/rmw_microxrcedds_c/src/config.h.in#L26
As you can see this is related with the RMW_UXRCE_STREAM_HISTORY, which you can modify on the CMakeFile and the RMW_UXRCE_MAX_TRANSPORT_MTU which is a value manage by Micro-XRCE-DDS and you can modify it on the next file: Micro-XRCE-DDS Config.
To clarify, the MTU on Micro-XRCE-DDS is the maximum size of data that can be sent on each packet. This value can be modified but having in mind that RMW_UXRCE_STREAM_HISTORY*RMW_UXRCE_MAX_TRANSPORT_MTU should be at least 2048.

@jfm92
Thanks for the response.
I've been playing around with the history and MTU and i still get memory leaks when I try to publish 200-400 floats in the Laser_scan_msg. Do you think 400 floats is simple to much for the RAM of the Olimex board? When i bumped up (history = 8, MTU = 2048) I couldn't build because of SRAM overflow - maybe an indication of to much memory usage?

Hello @krm-mir, that is right, setting a history of 8 and a MTU of 2048 will allocate static buffers of ~16 Kb, and depends on your configuration they can overflow your SRAM at linking time. For publishing it will work with a history of 1.
200 messages of aprox 4 Bytes is ~ 800 Bytes. I think that this should work, let us try a demo app with such big payload.
Are you sure that laser_scan_msg_ = *sensor_msgs__msg__LaserScan__create(); is allocationg the element? Please try the following:
sensor_msgs__msg__LaserScan laser_scan_msg_;
laser_scan_msg_.ranges.data = (float *)calloc(LASER_MSG_SIZE, sizeof(float));
laser_scan_msg_.ranges.capacity = LASER_MSG_SIZE;

I still get an error when preparing the output stream (same place as in my initial comment).
A small sidetrack while debugging:
Is the output stream fixed to reliable?
https://github.com/micro-ROS/rmw-microxrcedds/blob/80a305b05dec7ff0954c91726db78418fbf07661/rmw_microxrcedds_c/src/rmw_publish.c#L49-L52

For now the Micro-XRCE-DDS RMW is fixed to reliable while something about rmw specific configuration API is proposed in the RCL.
You can check this branch where publishing and subscribing are done using best effort: https://github.com/micro-ROS/rmw-microxrcedds/tree/test/best_effort
Remember that this branch is experimental and far from being stable.

@pablogs9  & @jfm92
I've experimented with another application, which publishes an increasing array of floats from the Olimex board. I can send an array of floats until the size = 123. Then I either get a memory leak or the app crashes:

I'm gonna put my float publisher on hold for now. Du you want to close the issue or keep it open for future discussion?

Please keep it open, we will be back to this issue soon or later.
Thank you so much for the research!

Hi @pablogs9 , I've come around to running some test on this issues again.
I'm still using the Olimex-stm32-e407 board but switched to microROS on FreeRTOS (dashing).
My test is basically the same - keep publishing an array of float32 and increase the size of the array after every publish.
I can publish until size = 122.

(screenshot of ros2 topic echo command)
Since i can replicate the same error (with same result) on two different OS, could this mean that it's related to microROS or fastRTPS?
regards

@julibert can this be related to fragmentation? are the new fragmentation branches used in micro-ROS build system?

@krm-mir can you show us what is the agent output on the working and not working publications? Please use -v6 to set the max verbosity and share us the trace

@pablogs9
Here the output:

Hi @krm-mir,
This PR in Micro XRCE-DDS Client should fix this issue. Please, update Micro CDR and Micro XRCE-DDS Client repos and rebuild your app. I hope it helps you.

@julibert
I've updated the repos and rebuild. I'm not sure if its related, but I run into another error:
firmware/mcu_ws/eProsima/Micro-XRCE-DDS-Client/src/c/core/session/stream/input_reliable_stream.c:217:11: error: 'ucdrBuffer' {aka 'struct ucdrBuffer'} has no member named 'offset'          ub->offset);
and
firmware/mcu_ws/eProsima/Micro-XRCE-DDS-Client/src/c/core/session/stream/output_reliable_stream.c:269:11: error: 'ucdrBuffer' {aka 'struct ucdrBuffer'} has no member named 'offset'          ub->offset);
when rebuilding.
I also tried with a fresh build of micro-ros-build --> create_firmware -> configure -> buildfirmware.

@julibert
I've reopened the issue after running the same tests in FreeRTOS and NuttX. Results are much better, but not near the paylaod size of eg a camera frame.
Both tests use the laser_scan msg type  with only the frame_id changed to “LZR_data” and the ranges increased.
NuttX results

360floats + header.
FreeRTOS resutls

371floats + header.

Hi @krm-mir, could you share the Agent output again?

@julibert
I believe the issue could be a segmentation fault (or something similar), since the agent doens't croak. Here are the agent outputts:
NuttX

...

I can't make the NuttX debugging work atm to trace the error - sorry.
 I reckon there's some issue with the buffer?
Below is the application source code:
#include <allocators.h>

#include <rcl/rcl.h>
#include <rcl_action/rcl_action.h>
#include <rcl/error_handling.h>

#include <rmw_uros/options.h>

#include <stdio.h>
#include <unistd.h>

// application specific includes
#include <std_msgs/msg/float32.h>
#include <cmsis_os.h>

// #define USE_FLOAT_ONLY

#ifdef USE_FLOAT_ONLY
#include <custom_interfaces/msg/float_array.h>
#include <std_msgs/msg/float32_multi_array.h>
#else
#include <sensor_msgs/msg/laser_scan.h>
#endif /* USE_FLOAT_ONLY */
//application defines
#define PAYLOAD_SIZE 400 // 111 is max

void appMain(void *argument)
{
    rcl_init_options_t options = rcl_get_zero_initialized_init_options();

    rcl_init_options_init(&options, rcl_get_default_allocator());

    // Optional RMW configuration
    // rmw_init_options_t *rmw_options = rcl_init_options_get_rmw_init_options(&options);
    // RCCHECK(rmw_uros_options_set_client_key(0xBA5EBA11, rmw_options))

    rcl_context_t context = rcl_get_zero_initialized_context();
    rcl_init(0, NULL, &options, &context);

    // create node
    rcl_node_options_t node_ops = rcl_node_get_default_options();

    rcl_node_t node = rcl_get_zero_initialized_node();
    rcl_node_init(&node, "payload_node", "", &context, &node_ops);

    // create publisher
    rcl_publisher_options_t publisher_ops = rcl_publisher_get_default_options();

    // SET RELIABILITY QOS:
    publisher_ops.qos.reliability = RMW_QOS_POLICY_RELIABILITY_RELIABLE;
    // payload_publisher_ops.qos.reliability = RMW_QOS_POLICY_RELIABILITY_BEST_EFFORT;
    rcl_publisher_t publisher = rcl_get_zero_initialized_publisher();
#ifdef USE_FLOAT_ONLY
    rosidl_message_type_support_t *type_support = ROSIDL_GET_MSG_TYPE_SUPPORT(custom_interfaces, msg, FloatArray);
#else
    rosidl_message_type_support_t *type_support = ROSIDL_GET_MSG_TYPE_SUPPORT(sensor_msgs, msg, LaserScan);
#endif /* USE_FLOAT_ONLY */
    rcl_publisher_init(&publisher,
                       &node,
                       type_support,
                       "/payload_test",
                       &publisher_ops);

// allocate space for msg
#ifdef USE_FLOAT_ONLY
    custom_interfaces__msg__FloatArray msg;
    custom_interfaces__msg__FloatArray__init(&msg);
    msg.data.capacity = PAYLOAD_SIZE;
    msg.data.size = 1;
    msg.data.data = (float *)calloc(1, sizeof(float));
    msg.data.data[0] = 0.0;
#else
    const char HEADER_ID[] = "LZR_data";
    float ranges[PAYLOAD_SIZE];
    sensor_msgs__msg__LaserScan msg;
    sensor_msgs__msg__LaserScan__init(&msg); // important!!!
    msg.header.frame_id.data = HEADER_ID;
    msg.header.frame_id.size = sizeof(HEADER_ID);
    msg.header.frame_id.capacity = sizeof(HEADER_ID);
    msg.ranges.data = (float *)calloc(PAYLOAD_SIZE, sizeof(float));
    msg.ranges.capacity = PAYLOAD_SIZE;
    msg.ranges.size = 0;
    msg.ranges.data = &ranges;
    // for (size_t i = 0; i < PAYLOAD_SIZE; i++)
    // {
    //     msg.ranges.data[i] = (float)i * 1.0 + 1.0;
    //     msg.ranges.size++;
    // }
#endif /* USE_FLOAT_ONLY */
    size_t array_size = 357;

    osDelay(5000);
    rcl_ret_t rc;
    do
    {
        rcl_ret_t rc = rcl_publish(&publisher, (const void *)&msg, NULL);

        if (rc != RCL_RET_OK)
        {
            break;
        }

        if (array_size < PAYLOAD_SIZE)
        {
#ifdef USE_FLOAT_ONLY
            free(msg.data.data);
            msg.data.data = (float *)calloc(array_size, sizeof(float));
            msg.data.size = 0;
            msg.data.capacity = array_size;
            for (size_t i = 0; i < array_size; i++)
            {
                msg.data.data[i] = (float)i;
                msg.data.size++;
            }
#else
            free(msg.ranges.data);
            msg.ranges.data = (float*)calloc(array_size, sizeof(float));
            msg.ranges.size = 0;
            msg.ranges.capacity = array_size;
            for (size_t i = 0; i < array_size; i++)
            {
                msg.ranges.data[i] = (float)i;
                msg.ranges.size++;
            }

#endif /* USE_FLOAT_ONLY */
            array_size++;
        }

            osDelay(500);
        }
        while (true);

        sensor_msgs__msg__LaserScan__fini(&msg);
        rcl_publisher_fini(&publisher, &node);
        rcl_node_fini(&node);

        vTaskDelete(NULL);
    }

Hope it makes sense.

Hi @krm-mir,
I have a couple of questions:

What are your rmw-microxrcedds and Micro XRCE-DDS configurations? RMW_UXRCE_STREAM_HISTORY & CONFIG_UDP_TRANSPORT_MTU.
Do you try the same application for host? Maybe then you can debug/trance the error.

RMW_UXRCE_MAX_HISTORY=2
CONFIG_UDP_TRANSPORT_MTU=512
(totally forgot the MTU).
Using CONFIG_UDP_TRANSPORT_MTU=1024 (History unchanged), so MTU*HISTORY >= 2048:
At a LaserScan.msg containing 741 (float32) + header (total msg size is 24040bit) the agent output is:
0000: 81 00 00 00 0A 01 05 00 77 04 00 00 80
[1588752160.036618] debug    | DataWriter.cpp     | write                    | [** <<DDS>> **]        | client_key: 0x00000006, len: 3028, data:
0000: A5 A5 A5 A5 A5 A5 A5 A5 09 00 00 00 4C 5A 52 5F 64 61 74 61 00 80 F8 43 A5 A5 A5 A5 A5 A5 A5 A5
0020: A5 A5 A5 A5 A5 A5 A5 A5 A5 A5 A5 A5 A5 A5 A5 A5 A5 A5 A5 A5 E6 02 00 00 00 00 00 00 00 00 80 3F
0040: 00 00 00 40 00 00 40 40 00 00 80 40 00 00 A0 40 00 00 C0 40 00 00 E0 40 00 00 00 41 00 00 10 41
0060: 00 00 20 41 00 00 30 41 00 00 40 41 00 00 50 41 00 00 60 41 00 00 70 41 00 00 80 41 00 00 88 41
0080: 00 00 90 41 00 00 98 41 00 00 A0 41 00 00 A8 41 00 00 B0 41 00 00 B8 41 00 00 C0 41 00 00 C8 41
...
0B40: 00 80 30 44 00 C0 30 44 00 00 31 44 00 40 31 44 00 80 31 44 00 C0 31 44 00 00 32 44 00 40 32 44
0B60: 00 80 32 44 00 C0 32 44 00 00 33 44 00 40 33 44 00 80 33 44 00 C0 33 44 00 00 34 44 00 40 34 44
0B80: 00 80 34 44 00 C0 34 44 00 00 35 44 00 40 35 44 00 80 35 44 00 C0 35 44 00 00 36 44 00 40 36 44
0BA0: 00 80 36 44 00 C0 36 44 00 00 37 44 00 40 37 44 00 80 37 44 00 C0 37 44 00 00 38 44 00 40 38 44
0BC0: 00 80 38 44 00 C0 38 44 00 00 39 44 00 40 39 44 00 00 00 00
[1588752160.037209] debug    | UDPv4AgentLinux.cpp | recv_message             | [==>> UDP <<==]        | client_key: 0x172387A4, len: 13, data:
0000: 81 00 00 00 0B 01 05 00 75 04 77 04 80
[1588752160.037245] debug    | UDPv4AgentLinux.cpp | send_message             | [** <<UDP>> **]        | client_key: 0x172387A4, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 78 04 00 00 80
[1588752160.037324] debug    | UDPv4AgentLinux.cpp | send_message             | [** <<UDP>> **]        | client_key: 0x172387A4, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 78 04 00 00 80
[1588752160.534734] debug    | UDPv4AgentLinux.cpp | recv_message             | [==>> UDP <<==]        | client_key: 0x172387A4, len: 20, data:
0000: 81 80 78 04 03 01 04 00 01 CC 00 65 03 01 04 00 01 CD 00 53
[1588752160.535137] debug    | ProxyClient.cpp    | delete_object_unlock     | object deleted         | client_key: 0x172387A4, object_id: 0x0006
[1588752160.535207] debug    | ProxyClient.cpp    | delete_object_unlock     | object deleted         | client_key: 0x172387A4, object_id: 0x0005
[1588752160.535270] debug    | UDPv4AgentLinux.cpp | send_message             | [** <<UDP>> **]        | client_key: 0x172387A4, len: 14, data:
0000: 81 80 07 00 05 01 06 00 01 CC 00 65 00 00
[1588752160.535281] debug    | UDPv4AgentLinux.cpp | send_message             | [** <<UDP>> **]        | client_key: 0x172387A4, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 79 04 00 00 80
[1588752160.535300] debug    | UDPv4AgentLinux.cpp | send_message             | [** <<UDP>> **]        | client_key: 0x172387A4, len: 14, data:
0000: 81 80 08 00 05 01 06 00 01 CD 00 53 00 00
[1588752160.535429] debug    | UDPv4AgentLinux.cpp | recv_message             | [==>> UDP <<==]        | client_key: 0x172387A4, len: 13, data:
0000: 81 00 00 00 0B 01 05 00 78 04 78 04 80
[1588752160.535546] debug    | UDPv4AgentLinux.cpp | send_message             | [** <<UDP>> **]        | client_key: 0x172387A4, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 79 04 00 00 80
[1588752160.535986] debug    | UDPv4AgentLinux.cpp | recv_message             | [==>> UDP <<==]        | client_key: 0x172387A4, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 08 00 00 00 80
[1588752160.536312] debug    | UDPv4AgentLinux.cpp | recv_message             | [==>> UDP <<==]        | client_key: 0x172387A4, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 09 00 00 00 80
[1588752160.536624] debug    | UDPv4AgentLinux.cpp | recv_message             | [==>> UDP <<==]        | client_key: 0x172387A4, len: 12, data:
0000: 81 80 79 04 03 01 04 00 01 CE 00 42
[1588752160.536674] debug    | ProxyClient.cpp    | delete_object_unlock     | object deleted         | client_key: 0x172387A4, object_id: 0x0004
[1588752160.536787] debug    | UDPv4AgentLinux.cpp | send_message             | [** <<UDP>> **]        | client_key: 0x172387A4, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 7A 04 00 00 80
[1588752160.536799] debug    | UDPv4AgentLinux.cpp | send_message             | [** <<UDP>> **]        | client_key: 0x172387A4, len: 14, data:
0000: 81 80 09 00 05 01 06 00 01 CE 00 42 00 00
[1588752160.537437] debug    | UDPv4AgentLinux.cpp | recv_message             | [==>> UDP <<==]        | client_key: 0x172387A4, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 0A 00 00 00 80
[1588752160.537733] debug    | UDPv4AgentLinux.cpp | recv_message             | [==>> UDP <<==]        | client_key: 0x172387A4, len: 20, data:
0000: 81 80 7A 04 03 01 04 00 01 CF 00 35 03 01 04 00 01 D0 00 23
[1588752160.537955] debug    | ProxyClient.cpp    | delete_object_unlock     | object deleted         | client_key: 0x172387A4, object_id: 0x0003
[1588752160.537998] debug    | ProxyClient.cpp    | delete_object_unlock     | object deleted         | client_key: 0x172387A4, object_id: 0x0002
[1588752160.538053] debug    | UDPv4AgentLinux.cpp | send_message             | [** <<UDP>> **]        | client_key: 0x172387A4, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 7B 04 00 00 80
[1588752160.538095] debug    | UDPv4AgentLinux.cpp | send_message             | [** <<UDP>> **]        | client_key: 0x172387A4, len: 14, data:
0000: 81 80 0B 00 05 01 06 00 01 D0 00 23 00 00
[1588752160.538125] debug    | UDPv4AgentLinux.cpp | send_message             | [** <<UDP>> **]        | client_key: 0x172387A4, len: 14, data:
0000: 81 80 0A 00 05 01 06 00 01 CF 00 35 00 00
[1588752160.538435] debug    | UDPv4AgentLinux.cpp | recv_message             | [==>> UDP <<==]        | client_key: 0x172387A4, len: 13, data:
0000: 81 00 00 00 0B 01 05 00 7A 04 7A 04 80
[1588752160.538545] debug    | UDPv4AgentLinux.cpp | send_message             | [** <<UDP>> **]        | client_key: 0x172387A4, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 7B 04 00 00 80
[1588752160.539058] debug    | UDPv4AgentLinux.cpp | recv_message             | [==>> UDP <<==]        | client_key: 0x172387A4, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 0A 00 00 01 80
[1588752160.539184] debug    | UDPv4AgentLinux.cpp | send_message             | [** <<UDP>> **]        | client_key: 0x172387A4, len: 14, data:
0000: 81 80 0A 00 05 01 06 00 01 CF 00 35 00 00
[1588752160.539317] debug    | UDPv4AgentLinux.cpp | recv_message             | [==>> UDP <<==]        | client_key: 0x172387A4, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 0C 00 00 00 80
[1588752160.539329] debug    | UDPv4AgentLinux.cpp | recv_message             | [==>> UDP <<==]        | client_key: 0x172387A4, len: 12, data:
0000: 81 80 7B 04 03 01 04 00 01 D1 00 12
[1588752160.539344] debug    | ProxyClient.cpp    | delete_object_unlock     | object deleted         | client_key: 0x172387A4, object_id: 0x0001
[1588752160.539406] debug    | UDPv4AgentLinux.cpp | send_message             | [** <<UDP>> **]        | client_key: 0x172387A4, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 7C 04 00 00 80
[1588752160.539416] debug    | UDPv4AgentLinux.cpp | send_message             | [** <<UDP>> **]        | client_key: 0x172387A4, len: 14, data:
0000: 81 80 0C 00 05 01 06 00 01 D1 00 12 00 00
[1588752160.539918] debug    | UDPv4AgentLinux.cpp | recv_message             | [==>> UDP <<==]        | client_key: 0x172387A4, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 0C 00 00 00 80
[1588752160.540365] debug    | UDPv4AgentLinux.cpp | recv_message             | [==>> UDP <<==]        | client_key: 0x172387A4, len: 13, data:
0000: 81 00 00 00 0A 01 05 00 0D 00 00 00 80

I'm actually way within my needs in terms of payload size - the goal was to send 400 float32 data points + header from a Laser scanner.

I have not tried running the payload_test app on host, as I'm aiming towards microROS on embedded only. If you want one, I'll gladly try it out.

@krm-int Hi! Thank you for your interest in micro-ROS. Notice that we’ll discuss the recent contributions and developments in the next Embedded WG meeting. You’re invited to bring your micro-ROS related issue to this platform. We’ll be very happy to have you join us and discuss your questions with the community.

Probably related: https://github.com/micro-ROS/micro_ros_setup/issues/187#issuecomment-702143940
@krm-int did you achieve something regarding this issue?

Solved here: https://github.com/micro-ROS/rosidl_typesupport_microxrcedds/pull/25


How to copy a blob from NFSv3-enabled storage account

Query/Question
I try to copy a blob from a storage account that has NFSv3 enabled.
            var source = "https://source.blob.core.windows.net/container1/file1.bin?sv=2020-10-02&st=2021-11-23T08%3A53%3A18Z&se=2021-12-01T08%3A53%3A00Z&sr=b&sp=r&sig=8Bt3I1guQe0VXjMjeOGOhPHv5tS2%2BH8gpx00bWhCnnE%3D";
            var target = "https://target.blob.core.windows.net/container2/file2.bin?sv=2020-08-04&st=2021-11-23T08%3A54%3A13Z&se=2021-12-01T08%3A54%3A00Z&sr=c&sp=racwl&sig=WZULxHkbM6yo4s%2FIw%2FKH7UNpaf4AQVxnhxIUGq0nv88%3D";

            var blobClient = new Azure.Storage.Blobs.BlobClient(new Uri(target));
            var result = await blobClient.StartCopyFromUriAsync(new Uri(source));

The operation fails. Diagnostic logs show the following error:
StatusCode
409

OperationName
GetBlockList

StatusCode
409

StatusText
BlobOperationNotSupportedForBlobCreatedByNfsV3

I can download the blob without an issue:
            var blobClient = new Azure.Storage.Blobs.BlobClient(new Uri(source));
            await blobClient.DownloadToAsync(@"C:\Temp\MyFile");

Also, azcopy copy works fine.
.\azcopy_windows_amd64_10.12.2\azcopy.exe" copy 'source' 'target'

How can I copy the blob without downloading and re-uploading?
Environment:

Azure.Storage.Blobs 12.10.0

.NET SDK (reflecting any global.json):
 Version:   5.0.403
 Commit:    bf9d168ac2

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19043
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\5.0.403\

Host (useful for support):
  Version: 5.0.12
  Commit:  7211aa01b3

It happens only for blobs created by NFS. I can copy blobs created using the storage SDK without an issue.

Hi, could you double check that the StartCopyFromUriAsync actually causes the following error you posted.
 But your claim is that the StartCopyFromUri command caused this error. I looked in our internals, and we do not call GetBlockList under the hood either (and there's no reason for us to either).

@amnguye
You're right, I found another error:
CallerIpAddress
<IP_ADDRESS>

OperationName
GetBlobProperties

AuthenticationType
SAS

StatusCode
403

StatusText
AuthorizationFailure

Apparently, I need to include the range <IP_ADDRESS>/23 to the firewall somehow.

This other error is a GetProperties Error, please provide an error related to your original issue. (StartCopyFromUriAsync).

Here are all requests from the diagnostic logs:

Uri
File Uploaded By
TimeGenerated [UTC]
OperationName
AuthenticationType
StatusCode
StatusText
CallerIpAddress
UserAgentHeader

file1.txt
Storage Explorer
11/23/2021, 8:08:45.705 PM
GetBlobProperties
SAS
403
AuthorizationFailure
<IP_ADDRESS>:62272
azsdk-net-Storage.Blobs/12.10.0 (.NET 6.0.0-rtm.21522.10; Microsoft   Windows 10.0.22000)

file1.txt
Storage Explorer
11/23/2021, 8:08:45.847 PM
GetBlobProperties
SAS
200
Success
99.128.xxx.xxx:62273
azsdk-net-Storage.Blobs/12.10.0 (.NET 6.0.0-rtm.21522.10; Microsoft   Windows 10.0.22000)

file1.txt
Storage Explorer
11/23/2021, 8:08:45.996 PM
GetBlockList
SAS
200
Success
99.128.xxx.xxx:37504
services_xstore_transport_HTTP2/1.0

file1.txt
Storage Explorer
11/23/2021, 8:08:46.031 PM
GetBlob
SAS
206
Success
99.128.xxx.xxx:37504
services_xstore_transport_HTTP2/1.0

file2.txt
NFSv3
11/23/2021, 8:10:23.353 PM
GetBlobProperties
SAS
403
AuthorizationFailure
<IP_ADDRESS>:9344
azsdk-net-Storage.Blobs/12.10.0 (.NET 6.0.0-rtm.21522.10; Microsoft   Windows 10.0.22000)

file2.txt
NFSv3
11/23/2021, 8:10:23.494 PM
GetBlobProperties
SAS
200
Success
99.128.xxx.xxx:9345
azsdk-net-Storage.Blobs/12.10.0 (.NET 6.0.0-rtm.21522.10; Microsoft   Windows 10.0.22000)

file2.txt
NFSv3
11/23/2021, 8:10:23.623 PM
GetBlockList
SAS
409
BlobOperationNotSupportedForBlobCreatedByNfsV3
99.128.xxx.xxx:37505
services_xstore_transport_HTTP2/1.0

I have 2 files:
file1.txt - uploaded using Storage Explorer, copy works fine for it.
file2.txt - created using NFSv3, copy does not work.

According to your logs, it says the BlobOperationNotSupportedForBlobCreatedByNfsV3 was caused by the an application by the user agent services_xstore_transport_HTTP2/1.0 not azsdk-net-Storage.Blobs/12.10.0 (.NET 6.0.0-rtm.21522.10; Microsoft Windows 10.0.22000) (which is our SDK user agent).
Also I already confirmed that the StartCopyFromUri API calls into the Copy Blob From Url which is not the same as the GetBlockList operation.
Please confirm whether or not the application you are writing send the user agent string as services_xstore_transport_HTTP2/1.0 and that you are altering the user agent string to be this header value.

@amnguye
I executed the API request manually without the .NET SDK and I can confirm that every import request produces 3 lines in the diagnostics log.
PUT https://target.blob.core.windows.net/container2/file2_sftp.txt?sv=2020-08-04&st=2021-11-23T08%3A54%3A13Z&se=2021-12-01T08%3A54%3A00Z&sr=c&sp=racwl&sig=WZULxHkbM6yo4s%2FIw%2FKH7UNpaf4AQVxnhxIUGq0nv88%3D HTTP/1.1
Host: target.blob.core.windows.net
x-ms-copy-source: https://source.blob.core.windows.net/container1/file2.txt?sv=2020-10-02&st=2021-11-23T19%3A54%3A54Z&se=2021-12-01T19%3A54%3A00Z&sr=b&sp=r&sig=wv0KVfbESRqdQpdnbqMFwYTbMQUD1gd%2FuoEC8Cs4Z0g%3D
x-ms-version: 2020-10-02
Accept: application/xml
x-ms-client-request-id: d464c562-6754-40ca-aa30-4a4b97100d73
x-ms-return-client-request-id: true
User-Agent: Fiddler
Content-Length: 0

| TimeGenerated   [UTC]      | OperationName     | AuthenticationType | StatusCode | StatusText                                     | CallerIpAddress     | UserAgentHeader                     |
|----------------------------|-------------------|--------------------|------------|------------------------------------------------|---------------------|-------------------------------------|
| 11/23/2021, 8:55:08.833 PM | GetBlobProperties | SAS                | 403        | AuthorizationFailure                           | <IP_ADDRESS>:2048 | Fiddler                             |
| 11/23/2021, 8:55:08.969 PM | GetBlobProperties | SAS                | 200        | Success                                        | 99.128.xxx.xxx:2049 | Fiddler                             |
| 11/23/2021, 8:55:09.301 PM | GetBlockList      | SAS                | 409        | BlobOperationNotSupportedForBlobCreatedByNfsV3 | 99.128.xxx.xxx:9984 | services_xstore_transport_HTTP2/1.0 |

Maybe Azure creates multiple log items per log request or it passes my IP address for additional internal requests.
services_xstore_transport_HTTP2/1.0 is not my user agent. I assume it belongs to the internal Azure copy worker.

Could you get me the details of that one request that you sent then (e.g. the full request made, and the full response made). Please also redact any credentials or signatures.

@amnguye
Please see above, I added the response to my previous comment.

Okay I see what's happening now. Obviously Copy Blob, e.g. StartCopyFromUri is not supported by NFSv3 currently.
The reason azcopy (and storage explorer since it also uses azcopy) works is because they're using Put Blob From Url. In order to do copies with NFSv3. the API needs to change to using Put Blob From Url
The SDK API to call this method is BlockBlobClient.SyncUploadFromUri. Please use this one instead.

@amnguye
Unfortunately, it does not work. SyncUploadFromUriAsync uses the same API with additional x-ms-requires-sync header.
Request:
PUT https://target.blob.core.windows.net/targetcontainer/file2_sftp.txt?sv=2020-08-04&st=2021-11-23T08%3A54%3A13Z&se=2021-12-01T08%3A54%3A00Z&sr=c&sp=racwl&sig=xxxx HTTP/1.1
Host: iwim2dev.blob.core.windows.net
x-ms-requires-sync: true
x-ms-copy-source: https://source.blob.core.windows.net/sourcecontainer/file3.txt?sv=2020-10-02&st=2021-11-23T22%3A41%3A37Z&se=2021-12-01T22%3A41%3A00Z&sr=b&sp=r&sig=xxxx
x-ms-version: 2020-10-02
Accept: application/xml
x-ms-client-request-id: 10b16214-892d-47c7-9e7c-f6b1e776e84e
x-ms-return-client-request-id: true
User-Agent: azsdk-net-Storage.Blobs/12.10.0 (.NET 6.0.0-rtm.21522.10; Microsoft Windows 10.0.22000)
Content-Length: 0

Response:
HTTP/1.1 409 Copy source blob has been modified.
Content-Length: 226
Content-Type: application/xml
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 35cc9539-301e-0009-0bbd-e0ff0d000000
x-ms-client-request-id: 10b16214-892d-47c7-9e7c-f6b1e776e84e
x-ms-version: 2020-10-02
x-ms-error-code: CannotVerifyCopySource
Date: Tue, 23 Nov 2021 22:57:18 GMT

<?xml version="1.0" encoding="utf-8"?><Error>CannotVerifyCopySource<Message>Copy source blob has been modified.
RequestId:35cc9539-301e-0009-0bbd-e0ff0d000000
Time:2021-11-23T22:57:19.2748708Z</Message></Error>

The log contains the same error BlobOperationNotSupportedForBlobCreatedByNfsV3.

Here's a workaround that works:
            var sourceClient = new Azure.Storage.Blobs.BlobClient(sourceUri);
            var destClient = new Azure.Storage.Blobs.BlobClient(destUri);

            using (var sourceStream = await sourceClient.OpenReadAsync())
            {
                await destClient.UploadAsync(sourceStream);
            }

Thanks for providing a request and response that uses the SDK.
Though that request looks like it's sending a BlobBaseClient.SyncCopyFromUri request rather than a BlockBlobClient.SyncUploadFromUri request. I can depict this because x-ms-requires-sync is a required header on the Copy Blob From Url (e.g. SyncCopyFromUri) rest call and not on the Put Blob From URL (e.g. SyncUploadFromUri) Rest call. While the Put Blob From URL rest call requires the x-ms-blob-type: BlockBlob header and value, and your request did not have this value.
I just ran both these APIs and they both send the headers expected (e.g. BlockBlobClient.SyncUploadFromUri has the x-ms-blob-type: BlockBlob header and value, but not the x-ms-requires-sync header, and the BlobBaseClient.SyncCopyFromUri has x-ms-requires-sync header but not the x-ms-blob-type: BlockBlob` header and value)

The workaround would work if you have the capacity to download the blob and re-upload the blob, and that has already been confirmed that in your original post when you said you were able to download the blob.
Does not solve the question of  "How can I copy the blob without downloading and re-uploading?"

Perhaps SyncUploadFromUri can't work across different storage accounts and falls back to SyncCopyFromUri.

Does not solve the question of "How can I copy the blob without downloading and re-uploading?"

As far as understand, I need to use the workaround for now. Maybe later copying of NFSv3 blobs will be added to the API.

Perhaps SyncUploadFromUri can't work across different storage accounts and falls back to SyncCopyFromUri.

This is incorrect. Please send me a code snippet with your updated code using the BlockBlobClient.SyncUploadFromUri
SyncUploadFromUri works across different storage account, and does not state otherwise in the documenation, and I also have tested this across my storage accounts.

@amnguye
Thank you! It works.
            var sourceUri = new Uri("https://source.blob.core.windows.net/sourcecontainer/tenant7/file3.txt?sv=2020-10-02&st=2021-11-23T22%3A41%3A37Z&se=2021-12-01T22%3A41%3A00Z&sr=b&sp=r&sig=xxxx");
            var destUri = new Uri("https://target.blob.core.windows.net/targetcontainer/file1644.txt?sv=2020-08-04&st=2021-11-23T08%3A54%3A13Z&se=2021-12-01T08%3A54%3A00Z&sr=c&sp=racwl&sig=xxxx");

            var blockBlobClient = new Azure.Storage.Blobs.Specialized.BlockBlobClient(destUri);
            var restult = await blockBlobClient.SyncUploadFromUriAsync(sourceUri);


Update CONTRIBUTING.md to be consistent with ci

This updates the developer instructions in CONTRIBUTING.md to be consistent with ci.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 52.42%. Comparing base (09360ac) to head (b3182ce).

Additional details and impacted files

@@           Coverage Diff           @@
##             main      #86   +/-   ##
=======================================
  Coverage   52.42%   52.42%
=======================================
  Files          15       15
  Lines         599      599
=======================================
  Hits          314      314
  Misses        285      285


Add the "Python 3.6 + Anaconda 5.2 EXPERIMENTAL!" mode of PythonTutor (recent feature)

Hello there @kikocorreoso,
I noticed that your extension only recognizes the following mode of using http://PythonTutor.com/:
'python2', 'python3', 'java', 'javascript', 'typescript', 'ruby', 'c', 'c++'
But a recent new feature is "Python 3.6 + Anaconda 5.2 EXPERIMENTAL!".
It's very useful because it provides a teaching/debugging environment that supports lots of scientific packages, including numpy, scipy, etc (of course, no displaying of matplotlib, it gives this error and is a known missing features):

ImportError: libGL.so.1: cannot open shared object file: No such file or directory

Do you think you could update your extension and allow a "anaconda" mode?
By reading the code of this extension, it would probably be easy!

I can try to hack it myself, I don't think it would give any issue, but I won't be able to publish the package to pypi etc, so it would be simpler if you could try it.
If you don't have time but are willing to merge a pull request, I can try to hack it myself, send a PR and let you merge and republish your package to pypi!

Thanks in advance! Regards from France, @Naereen

Using this code as an example, I get the following iframe:
<iframe width="800" height="500" frameborder="0" src="http://pythontutor.com/iframe-embed.html#code=%23%20https%3A//docs.scipy.org/doc/scipy/reference/tutorial/integrate.html%0A%0Ax%20%3D%204%0Aprint%28%22Hello%20world!%22%20*%20x%29%0A%0Aimport%20numpy%20as%20np%0A%0AI%20%3D%20np.sqrt%282/np.pi%29*%2818.0/27*np.sqrt%282%29*np.cos%284.5%29%20-%204.0/27*np.sqrt%282%29*np.sin%284.5%29%29%0A%0Aprint%28I%29%0A%0Aimport%20scipy.integrate%20as%20integrate%0Aimport%20scipy.special%20as%20special%0Aresult%20%3D%20integrate.quad%28lambda%20x%3A%20special.jv%282.5,x%29,%200,%204.5%29%0Aprint%28result%29%0A%0AI%20%2B%3D%20np.sqrt%282*np.pi%29%20*%20special.fresnel%283/np.sqrt%28np.pi%29%29%5B0%5D%0Aprint%28I%29%0A%0Aprint%28abs%28result%5B0%5D-I%29%29%0A&codeDivHeight=400&codeDivWidth=350&cumulative=false&curInstr=0&heapPrimitives=nevernest&origin=opt-frontend.js&py=py3anaconda&rawInputLstJSON=%5B%5D&textReferences=false"> </iframe>

The interesting part is py=py3anacondain the URL!
From what I can read on the code
https://github.com/kikocorreoso/tutormagic/blob/66fb98a757d70eededfb8c713d446cefcc2f5e6f/tutormagic.py#L140
https://github.com/kikocorreoso/tutormagic/blob/66fb98a757d70eededfb8c713d446cefcc2f5e6f/tutormagic.py#L173
it should be easy to add this py3anaconda language code in the "whitelist" that you wrote!
I'll try to hack this on my local installation of the magic, and reload my notebook.

For reproducibility, here is a ~1min GIF showing what I did before trying to add support for this py3anaconda environment:

Here is the durty hack I just wrote:
tutormagic.py: 140,141c140,141
<             if args.lang[0].lower() in ['python2', 'python3',
<                                         'java',
---
>             if args.lang[0].lower() in ['python2', 'python3', 'py3anaconda',
>                                         'java',
152c152
<                     "'typescript', 'ruby', 'c', 'c++'".format(args.lang[0]))
---
>                     "'typescript', 'ruby', 'c', 'c++', 'py3anaconda'".format(args.lang[0]))
177a178,179
>         elif lang == "py3anaconda":
>             url += "py=py3anaconda"

And here is a demo showing that it works!

I was not aware of the py36anaconda option. From the pythontutor.com web page it says 'unsopported'. I can include it here from your PR #11 but take into account this could break your code at any moment (from 3rd parties I don't manage, pythontutor.com).
As a general advice, you shouldn't rely on 'unsupported' stuff.

Hi again,
Well I think in the context of PythonTutor, unsupported means that the claim

Get live help for free in the Python tutoring Discord chat room
is not covering py3anaconda.

As a general advice, you shouldn't rely on 'unsupported' stuff.
I agree, but if I were rigorous, I would take it to the next level: especially for teaching activities, we shouldn't rely on closed-source online stuff! Imagine I build all my course using some visualizations on PythonTutor, and it's unreachable (or taken down) the week when I'm teaching... or just if I don't have WiFi in the room where I'm teaching...

I'll keep in mind the tiny lolviz Jupyter IPython extension, as it allows basic but nice illustrations of simple data-structures and pointers. It's far from close to PyTutor but well it can help.

As a general advice, you shouldn't rely on 'unsupported' stuff.

I agree, but if I were rigorous, I would take it to the next level: especially for teaching activities, we shouldn't rely on closed-source online stuff! Imagine I build all my course using some visualizations on PythonTutor, and it's unreachable (or taken down) the week when I'm teaching... or just if I don't have WiFi in the room where I'm teaching...

When I started this package pythontutor was opensource. I don' know why now it isn't anymore 😞

I'll keep in mind the tiny lolviz Jupyter IPython extension, as it allows basic but nice illustrations of simple data-structures and pointers. It's far from close to PyTutor but well it can help.

I didn't know about lolviz. It seems quite nice!!!! This extension mixed with other of my proof-of-concept jupyter extension called pizarra-nb could be quite interesting. pizarra-nb is not polished and was a hack I made as a proof of concept but I see potential there.

I think PythonTutor's author was tired of maintaining as open source, as his website is used by millions of users I can expect the number of issues to be large.
And the last documentation I read on v5-unity and v4-cokapi indicates that all the serverside setup is difficult to reproduce (I read  something like: "all this is hardwired for my own server, I can't help if you want to reproduce it").
I completely understand the desire to just remove it from GitHub.

Regarding lolviz, it could even be nicer if https://github.com/parrt/lolviz/issues/4 and https://github.com/parrt/lolviz/issues/27 get merged!

Hm your https://github.com/kikocorreoso/pizarra-nb is awesome!
I've been using and helping develop the more complete https://tableaunoir.github.io/ tool for the last 6 months, but pizarra-nb is interesting as it is right in the notebook.
Did you know about RISE which allows to turn classical notebook in live slideshow (live as in I can type and execute code, in any kernel) ? RISE has a tiny and unknown feature that is it includes a chalkboard and a "pen" tool to write on any slide! See https://rise.readthedocs.io/en/stable/customize.html#enable-chalkboard-capabilities

@kikocorreoso any update?

It is merged and you can install it from PyPI using pip.
Thanks.

Oh great, thanks @kikocorreoso !
I'll try it out by upgrading from pip.


Script to help reduce generated box size

This script disables hibernation, removes the sleepimage and deletes
any swapfiles. This reduces uneeded space on disk. Swapfiles will be
re-genereated as needed. If the box is for VMware, it will also make
VMware shrink the disk image via VMware tools. On a Yosemite box
made today this resulted in a reduction from ~14G to ~7G

Thank you! I've been wanting to look into making the box size smaller. Do you know if the same dynamic pager LaunchDaemon path is used for earlier versions of OS X?

facepalm, why didn't I think of the backwards compatibility.
I looked around and found some articles going back to 10.5 (e.g http://hints.macworld.com/article.php?story=20071121070305908) that mention this path, so we should be good. I don't have older install images to test against though.

That's ok, I can have a check tomorrow against some older VMs. The paging system definitely exists, just a question of whether the paths are consistent.
Thanks!

Checked all versions of OS X I support with VMware, all good!
I looked at the space saved, and after compressing to a .box file with the default options, the savings on Yosemite was about 1.5GB on the compressed file, plus the obvious time savings having to compress smaller .vmdks.


unzip: CVE-2019-13232

Motivation for this change
Fixes #64663.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[X] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[X] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[X] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

@GrahamcOfBorg build unzip

@GrahamcOfBorg build darcs

It broke haskellPackages.zip-archive -> darcs -> hydra

@GrahamcOfBorg build haskellPackages.zip-archive

@volth can this be a false positive for the test? The CVE fixed by that patch is actually preventing a zip bomb.

The error is also zip bomb related
https://logs.nix.ci/?key=nixos/nixpkgs.64909&attempt_id=cc70c8f9-1d57-4b64-8073-42691767eeda
There is no new version of haskellPackages.zip-archive, so it has to be patched to disable the test.
We may need a Haskell guru for that.
@peti ? @srhb ?

The error invalid zip file with overlapped components (possible zip bomb) is provided by the security patch, it detects a potential zip bomb and fails in case it does. Ideally we would patch the test and provide it to the upstream, as this CVE patch will get into upstream zip sooner or later. Also, we can look at what Debian does, as they shipped it already too.

Filled an issue with upstream https://github.com/jgm/zip-archive/issues/57

This broke unpacking of some(all?) lua rocks.
nix-build '<nixpkgs>' --no-out-link --show-trace -A luaPackages.luasocket for example.


Added python script

What type of PR is this? (check all applicable)

[x] 🚀 Added Name
[x] ✨ Feature
[x] 🌟 stared the repo
[ ] 🐛 Grammatical Error
[ ] 📝 Documentation Update
[x] 🚩 Other

Description
Added python script
Add Link of GitHub Profile
https://github.com/Attupatil/

can you help me to resolve this conflict?


Reconciler should know when dependent resource already exist

When the AbstractDependentResource already exist prior to creating the primary resource, the reconciler should know about it to be able to handle the situation and update the primary resource status accordingly.
For eg., the reconciler may set a CONFLICT status to let the user knows the observed spec conflict with an existing resource.
I have pushed a commit to demonstrate the use case on mysql-schema sample : https://github.com/scrocquesel/java-operator-sdk/tree/schema-already-exist
Be aware that, on startup, when match is true, the dependent resource alone cannot tell if there is a possible conflict or if it is what it should be from a prior reconciliation, actual and desired just match. Only the reconciler can tell apart if it saves the previous observed state.
For eg.
return context.get(BUILT_SCHEMA, Schema.class).map(s -> {
      updateStatusPojo(schema, context.getMandatory(MYSQL_SECRET_NAME, String.class),
          context.getMandatory(MYSQL_SECRET_USERNAME, String.class));
      log.info("Schema {} created - updating CR status", schema.getMetadata().getName());
      return UpdateControl.updateStatus(schema);
    }).orElseGet(() -> context.get(EXISTING_SCHEMA, Schema.class).map(s -> {
      if (schema.getStatus().getStatus() != "CREATED")
        schema.getStatus().setStatus("CONFLICT");
      log.info("Schema {} already exists - updating CR status", schema.getMetadata().getName());
      return UpdateControl.updateStatus(schema);
    }).orElse(UpdateControl.noUpdate()));

Sorry, I forgot to mention that currently there is no way to set (EXISTING_SCHEMA, Schema.class) in the context.

@scrocquesel is this issue still relevant? not sure if I follow, so you need to know if the resource existed on beginning of reconciliation. This kinda contradicts with the Kubernetes / Controller philosophy of level based event handling. That you should not work based on the history but just the current state.  What is the goal you want to achieve with this?
BTW we were discussion some more inf returned from dependentResource.reconcile(), to denote what exactly happened during the reconciliation. Like if the resource is created, updated or deleted during the reconciliation. Just want to understand you use case more.

see also: https://github.com/java-operator-sdk/java-operator-sdk/pull/1030
(this is for standalone mode)

ReconcileResult.noOperation with a not empty resource is enough. I guess the objective is that the manager save this result in the context for non standalone mode.

@csviri would it be possible to add a get/setReconcileResult on ManagedDependentResourceContext like
private final ConcurrentHashMap<DependentResource<?, ?>, ReconcileResult<?>> reconcileResults =
      new ConcurrentHashMap<>();

  /**
   * Retrieve a reconcile result of the specified dependent resource implementation.
   *
   * @param dependentResource the dependent resource implementation
   * @return an Optional containing the reconcile result or {@link Optional#empty()} if no such
   *         object exists
   */
  @SuppressWarnings("unchecked")
  public <R, T extends DependentResource<R, ?>> Optional<ReconcileResult<R>> getReconcileResult(
      DependentResource<R, ?> dependentResource) {
    return Optional.ofNullable(reconcileResults.get(dependentResource))
        .map(r -> (ReconcileResult<R>) r);
  }

  /**
   * Set a reconcile result of the specified dependent resource implementation.
   *
   * @param dependentResource the dependent resource implementation
   * @param reconcileResult the reconcile result to add to the context associated with the specified
   *        dependent resource
   */
  public <R, T extends DependentResource<R, ?>> void setReconcileResult(
      DependentResource<R, ?> dependentResource, ReconcileResult<R> reconcileResult) {
    reconcileResults.put(dependentResource, reconcileResult);
  }

so the controller can actually put the dependent reconcile result to be retrived from the reconciler ?
dependents.forEach(dependent -> context.managedDependentResourceContext()
    .setReconcileResult(dependent, dependent.reconcile(resource, context)));

I think there is an ongoing discussion to have a better API around the untyped get/put methods but I don't remember where.

Yes, this should be possible, especially with after other PR with named managed dependent resources.
https://github.com/java-operator-sdk/java-operator-sdk/pull/1058
cc @metacosm

This is already implemented.


apis: WireDetails: fix wrong key for vpc name

这个 PR 实现什么功能/修复什么问题:
apis: WireDetails: fix wrong key for vpc name

是否需要 backport 到之前的 release 分支:
NONE
/area region
/cc @ioito @zexi

Codecov Report

Merging #6615 into release/3.1 will increase coverage by 0.01%.
The diff coverage is 0.00%.

@@              Coverage Diff               @@
##           release/3.1   #6615      +/-   ##
==============================================
+ Coverage         7.48%   7.50%   +0.01%
==============================================
  Files              585     585
  Lines            98953   98954       +1
==============================================
+ Hits              7411    7422      +11
+ Misses           90970   90959      -11
- Partials           572     573       +1

Flag
Coverage Δ

#aFlag
7.50% <0.00%> (+0.01%)
:arrow_up:

Impacted Files
Coverage Δ

pkg/cloudcommon/policy/resources.go
0.00% <ø> (ø)

pkg/cloudprovider/objectstore.go
2.51% <0.00%> (-0.01%)
:arrow_down:

pkg/compute/models/buckets.go
1.24% <0.00%> (ø)

pkg/cloudcommon/cronman/cronman.go
54.85% <0.00%> (+5.33%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 94fe13d...290a32b. Read the comment docs.


Only indent the first and last level

I am using vim ver 7.4 on Ubuntu 16.04 and just like the instruction I move all the after, glyph, doc to the .vim dir but it only indent the last level of my code at first and then first level after I set the indentLine_showFirstIndentLevel value from 0 to 1.
If my code have 5 level then it only show level 1 and 5, nothing is happen in level 2, 3, 4.
Someone pls tell me what to do to fix this problem

Make sure they are spaces but not tabs.

I don't have any tabs in my code so I'm sure my code are completely space only and it still not work
/home/nguyen/ピクチャ/Screenshot from 2017-10-10 09-45-31.png

echo &shiftwidth
echo &tabstop

What is the output of them?

I don't understand code so I was searching for the shiftwidth and tabstop, and I found them in line 115, 133, 199

Enter vim
Type :
Type echo &shiftwidth
Type enter
to see the output.

Oh it is 8

You need set the value of shiftwidth to what you want, for example 4.
Add set sw=4 in your vimrc.

Can u explain the set sw=4 for me pls. What will happen if I change the value of sw into 4 or 8 or 3 or any others number

:h shiftwidth

You can change the value and see the indent level.

Oh so it indent the level by the number of white space that I have used. So, I just have to change the value of sw or the number white  space that I always use to go to the new level in my code and it will be ok.
Thank you. I really appreciate your help.  It work just fine now


Self referencing models: maximum recursion depth exceeded

Checks

[x] I added a descriptive title to this issue
[x] I have searched (google, github) for similar issues and couldn't find anything
[x] I have read and followed the docs and still think this is a bug

Bug
Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":
             pydantic version: 1.7.3
            pydantic compiled: True
                 install path: ~/.venvs/repository-pattern/lib/python3.7/site-packages/pydantic
               python version: 3.7.3 (default, Jul 25 2020, 13:03:44)  [GCC 8.3.0]
                     platform: Linux-4.19.0-5-amd64-x86_64-with-debian-10.6
     optional deps. installed: ['typing-extensions']

An RecursionError: maximum recursion depth exceeded while calling a Python object error was raised when accessing self referencing attribute from related models.
Given the following model definition:
from typing import Optional

from pydantic import BaseModel

class A(BaseModel):
    b: Optional["B"] = None

class B(BaseModel):
    a: Optional[A] = None

A.update_forward_refs()

a = A()
b = B()
a.b = b
b.a = a

aa = A()
ba = B()
aa.b = ba
ba.a = aa

When running the following code:
>> type(a.b.a)
<class '__main__.A'>
>> a.b.a
>> type(aa.b.a)
<class '__main__.A'>
>> aa.b.a
*** RecursionError: maximum recursion depth exceeded while calling a Python object

You see that the Recursion error is raised.
I know this error is related with #1370, but as it's not a self referencing model I thought it was better to open another issue. I don't mind merging it with the other if you feel like it.
It can also be related to #659, #524 and #1345.
I guess until the related issues are not solved, this one is not going to be either, but can anyone help me understand why a.b.a works and aa.b.a doesn't? It's driving me crazy xD

Have you checked to see if there is any name mangling or reserved variables?

I came across something similiar, although I'm not sure if I should open a new issue for this:
from pydantic import BaseModel

class Foo(BaseModel):
    other: 'Foo' = None

f = Foo()
f.other = f

print(f == f) # crashes

The stack trace:
Traceback (most recent call last):
  File "test.py", line 9, in <module>
    print(f == f) # crashes
File "pydantic/main.py", line 874, in pydantic.main.BaseModel.__eq__
  File "pydantic/main.py", line 435, in pydantic.main.BaseModel.dict
  File "pydantic/main.py", line 833, in _iter
  File "pydantic/main.py", line 708, in pydantic.main.BaseModel._get_value
  File "pydantic/main.py", line 435, in pydantic.main.BaseModel.dict
  File "pydantic/main.py", line 833, in _iter
  File "pydantic/main.py", line 708, in pydantic.main.BaseModel._get_value
  File "pydantic/main.py", line 435, in pydantic.main.BaseModel.dict
  File "pydantic/main.py", line 833, in _iter
  File "pydantic/main.py", line 708, in pydantic.main.BaseModel._get_value
  [...]
    File "/usr/lib/python3.8/abc.py", line 98, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison

Pydantic version:
             pydantic version: 1.9.0
            pydantic compiled: True
                 install path: /home/samuel/projects/foo/venv/lib/python3.8/site-packages/pydantic
               python version: 3.8.10 (default, Nov 26 2021, 20:14:08)  [GCC 9.3.0]
                     platform: Linux-5.13.0-28-generic-x86_64-with-glibc2.29
     optional deps. installed: ['typing-extensions']

I guess until the related issues are not solved, this one is not going to be either, but can anyone help me understand why a.b.a works and aa.b.a doesn't? It's driving me crazy xD

With 1.9.0 it seems both fail.

pydantic doesn't (yet) support recursive models.
As far as I know this is a duplicate of #659.

I'm not sure it's a duplicate, because I'd like to use it without using SQLAlchemy

It's all the same route cause - we currently have no cycle breaking feature. It's possible but far from trivial to implement and not a priority right now.

This is fixed in pydantic-core, recursion cycles will be safely caught in pydantic v2.
See https://github.com/samuelcolvin/pydantic-core/pull/134.

@samuelcolvin I noticed this mentioned about the v2 release: "Plan is to have all this done by the end of October, definitely by the end of the year.”. Do you have any further insight as to how far away v2 is? I'd love to use Pydantic, but this issue is a deal-breaker so v2 is a requirement for my use-case.

Still aiming for the end of the year. Pretty certain I'll have an alpha release by then, less certain I'll have V2 itself out.

I've found a case which I think breaks the fix in PR134 (discovered on pydantic 2.3.0):
from typing import TypeAlias
from pydantic import BaseModel

# adapted from https://stackoverflow.com/a/76646986
JSONable: TypeAlias = dict[str, "JSONable"] | list["JSONable"] | str | int | float | bool | None

class ArbitraryData(BaseModel):
    data: JSONable

raises: RecursionError: maximum recursion depth exceeded in __instancecheck__

Got the same problem as @quantology -- we should probably reopen this

i had a similar issue but was using pydantic version inside of dspy project (that also uses v2 but old one).  i removed dspy functionality upgraded to latest 2.7.* then it fixed.

I've run into this problem while trying to type JSON data like @quantology. I haven't really found a workaround, but I did find out that pydantic includes a JsonValue which is basically a fancier version of the JSONable type definition above. I don't get the recursion error when using JsonValue.

Should be reopened

Can someone confirm if a fix for recursive type definition is planned or not ?
While working for recursive BaseModel definition, as of version 2.9.3, any attempt to get json_schema from  recursive type definition  like   Foo = Union[int,'Foo']  crash with recursion limit error.


Reduce diagnosticPublishDelay default

Issue Type: Bug
Problem
python.analysis.diagnosticPublishDelay defaults to 1000 milliseconds. This feels like quite a long time in practice, especially while waiting for an error you just fixed to be clear
Proposal
Reduce the default value. Unless there's some reason this needs to be so high, a lower default provide the majority of users who will never change the default with a more responsive editing experience. Advanced users can still increase it if they want/need to
JS/TS users a default delay of around 200ms. If we get another document update before we finish computing the errors, we cancel the ongoing diagnostics request and queue up a new one
Extension version: 2020.4.68060-dev
VS Code version: Code - Insiders 1.44.0-insider (efd7548df65de8e775015b67863190e6bd4934fc, 2020-03-13T06:01:51.669Z)
OS version: Darwin x64 19.3.0

You can set it to the desired value. Actually, it is set to 1 sec by user requests: people did not want to see squiggles right away and error list updating and possibly flickering on every type.

Yes I understand it is configurable. But I suspect few people are changing this and the current default makes it seem like server is slow recomputing diagnostics. Some delay while the document is changing is good but 1sec feels surprisingly long in practice

Just to provide perspective, microsoft/vscode-python#2270 was the original issue. Back then, the only way to get diagnostics errors (syntax, lints) was from pylint or similar on save.
Given we've had real live syntax checking for a while, perhaps we should consider lowering this as I expect many users are more comfortable with seeing info earlier. May be useful to go see what other languages (TS, C#, etc) are doing in their LS/analysis implementations.

Perhaps 400 ms?

Sure, though I also wouldn't be opposed to 200ms if that's the debounce other languages are using. I haven't had time to try and find that out for myself.

0.5.46


External test data and setup

There are some functions that can only be tested using data (data can be created but would be really tedious). We need a new repo with several data sets for different scenarios and this has to be set up accordingly.

I don't have all the information right now but I know there are tools for this.

related to #293


missing entries

loading from a large 610000 words dictionary and then shuffling around some 25000 items, the search in trie fails constantly to find items, even if those items are definitely supposed to be there.
To test this: https://gist.github.com/dragosperca/2d5e2f04816c639c4e7d
Is the trie build process that fails or the search itself?

The words have to be sorted, are they?

yes, if you look within the gist, there's a dropbox link to download the words.txt file
words are sorted
it's only then they are shuffled to emulate some sort of randomness in the search

Is it failing on all entries or only a subset?

constantly failing on the same subset.
What I did was shuffle the entire dictionary and then search the whole trie again. Using the gist provided shows random results because of the slice
some printscreens:
https://www.dropbox.com/s/nt56seatmoncujr/SnappyApp SnappyApp%2C Today at 15.00.17.png?dl=0
https://www.dropbox.com/s/ozzl6vl5bkl99tq/console.png?dl=0

Yeah, looks like there's some bug. II can't reproduce with my own dictionaries but definitely can with yours. I'll take a look next weekend.

thank you, much appreciated

It's actually an impressive one. The encoding of the equivalence class is incorrect. I shouldn't be doing string(t.Child.id) but instead strconv.Itoa(int(t.Child.id)) since I don't want to interpret the ID as a rune. This is causing some nodes with different equivalence classes to collide. It doesn't reproduce with most dictionaries, but it does with yours. You actually need a pretty big one or a very carefully crafted test.

fantastic, thanks Mathieu, it works very well now!


Removes threading from the binance client.

Uses consistent async calls from within the library. This will
allow the ctrl+c control flow of the library to work again as
well as be a cleaner implementation for requests.
Switched out usage of requests with treq, which is the twisted
port of requests.
Uses twisted.internet.defer.inlineCallbacks to facilitate pseudo
blocking IO with yield statements.
--
I realize that having blocking IO will protect some users from
self-harm, but internally it should be avoided, running the
entire client in a thread if the client is using threads.
See: reactor.callInThread documentation.

Coverage decreased (-8.2%) to 28.239% when pulling 0479aad554c1c9d9456f20ee6b2fd26bc509bac8 on jdcumpson:remove-threading into 45f869b5636e34ae2d9fbd3928f62cd827e8f267 on sammchardy:master.

This is also a big semantic change, so it's fair not to put it in - if it does, recommended major release. If not, I hope some one learns something :)

Hey thanks @jdcumpson I'll give it a test soon and look to include this.
How different is treq to requests? Binance requests need to be formatted fairly particularly with ordered params and signatures etc, will need to test that to make sure it's not going to introduce any issues.
Also have requests setup to handle extra params coming in to specify proxy servers or a timeout so if we did change it would need to support that too.


Laravel v8 Support

Any plans to support Laravel v8?

Also after Laravel 8 support if possible.

Appears Laravel 8 support was added:
#14


day_night_mode

Hi,
I have added a new category and a couple of links to it, please accept the PR and add hacktoberfest-accepted label.

Please accept this merge request. I will very grateful to you.


mwc713 should be able to send to http and not just https

This needs to be implemented via commandline and API. Hotbit can only withdraw to https and mwcmqs right now. Some users requested http.

fixed


Normalize JSO to composable immutable objects

Just cleaning up and moving all the CSO's into styles/jso. Might pull this apart later, but seems easier to manage in a single file.
#138191285

          1 Message

:book:
Nice, more deletions than insertions :red_circle:

  Generated by :no_entry_sign: danger

Yeah it's definitely a little weird but sort of makes sense after a bit. Nice and succinct.


proposal: testing(benchmark): add a flag to report metrics for each run

I would like the package to be able to report metrics for each run, not just an aggregate.
I want to be able to analyze and graph the data using external tools, so I need the runtime etc. of each iteration of the benchmark. Using the built-in benchmarking is great but I need to see the distribution of individual iterations.

But the reason the benchmark tool suppresses the intermediate benchmark runs is they are not statistically significant. Even the final benchmark number is not accurate in and of itself, it should be combined with flags like -count and tools like benchstat to account for warmup and cpu throttling issues.

As @davecheney said, -count is the way to get multiple runs. You can also drop -benchtime to be lower than the default 2s if you want many short runs.
There is no way to get the distribution of every iteration in a particular run, because the benchmark functions themselves do that loop from 0 to b.N.

No change in consensus, so declined.
— rsc for the proposal review group


empty Inverse deformation field

Hello, I am facing a strange behavior, I want to invert a deformation field so I could do a reverse transformation on registered image pair, but the function returns empty field...
see also how to use simpleitk get inverse displacement field

import SimpleITK as sitk
import numpy as np

deform = np.zeros((150, 200, 2), dtype=float)
disp_field = sitk.GetImageFromArray(deform, isVector=True)

deform = sitk.GetArrayFromImage(disp_field)
print deform.shape
deform_inv = sitk.GetArrayFromImage(sitk.InverseDisplacementField(disp_field))
print deform_inv.shape

gives following:
(150, 200, 2)
(0, 0, 2)

for SimpleITK Version: 1.1.0 (ITK 4.13), Compiled: Mar 26 2018 15:04:35

You are not setting any of the parameters, and the size is defaulting to 0:
 sitk.InverseDisplacementField?
Signature: sitk.InverseDisplacementField(*args, **kwargs)
Docstring:
InverseDisplacementField(Image image1, VectorUInt32 size, VectorDouble outputOrigin, VectorDouble outputSpacing, unsigned int subsamplingFactor=16) -> Image

If you use the object oriented interface you can use the SetReferenceImage method.

Thanks for such quick response, but should not be there a warning message that you did not set some mandartory parameters or set some reasonable default values?
sitk.InverseDisplacementField(disp_field,
                              size=disp_field.GetSize(),
                              outputOrigin=(0.0, 0.0),
                              outputSpacing=(1.0, 1.0))

I agree this is not very good behavior, and may be un expected.
But it did the exact thing it was specified to do. It was just unexpected that the default size was 0. The other filters to invert displacement fields, InvertDisplacementFieldImageFilter and IterativeInverseDisplacementFieldImageFilter, may meet you expectations as the input and output domain always match. So it is a "feature" and not a :bug: of that the ITK filter was written so that you have to specify the output domain for the inversion :smile:

Thanks for the clear illustration of the problem. That does not look like good results. IT looks like you have some large deformations there that may contain folds, and may not be 1:1 mapping, so the assumption of the method may be violated.
I'd recommend posting the to the general ITK community at https://discourse.itk.org. This question is about details about the algorithm, and there are several other alternative algorithms to invert displacements field that may be better.

yes, I agree that there are some folds and I cannot simply use an interpolation from scipy for warping output image, but it is what I got from the demons registrations and somehow the DisplacementFieldTransform and ResampleImageFilter is able to handle it and gives me right warped images.

I was using FastSymmetricForcesDemonsRegistrationFilter

Please mode the topic to the wider ITK community.

OK, and what channel do you recommend? For example, I do not see an option to add the issue in Jira

I'd recommend posting to the general ITK community at https://discourse.itk.org. You do not appear to have encountered a bug, but need help choosing the appropriate method and understand their requirements.


.NET monikerization and automation

.NET docs team has already extended the ability for .NET moniker. The next step is to have .NET docs to onboard to our automation system as other languages. Use the issue as an epic to track the work.
Basic steps:

[ ] Batch update on docs-ref-service readmes into right monikers.
[ ] Update the docs metadata script for putting service readme to right place.
[ ] Update the toc script and split unified to latest/preview reference.
[ ] Update the JoinTOCPlugin Link
[ ] Add toc automation work for .NET docs.
[ ] Add service level readme automation.

Notes: Work here needs to keep compatibility for current work.
This is the implementation detail issue to resolve https://github.com/Azure/azure-sdk/issues/1857

Package/Service level readme will go to moniker folder
Proposed:

Preview: api/overview/azure/preivew
Latest: api/overview/azure/latest

Where they are currently:

Current: api/overview/azure

ToC:
Proposed:

Preview: docs-ref-toc/reference-preview.yml
Latest: docs-ref-toc/reference-latest.yml
Unified: docs-ref-toc/reference-unified.yml

Where they are currently:

Current Path: docs-ref-toc/top_level_toc.yml

What we don't need to touch

Package json/CSV: They have supported the moniker already
API yaml/xml: They have supported the moniker already.

What workflow we need to change: (We will support proposed path and current path in parallel for a while, and then remove the old one)

Update-DocsMsMetadata.ps1 (docindex, release, nightly): Update the place to check in the readmes.
Fetch the namespace and check in the metadatc for toc update.
New toc automation code.
Update the sync-toc.py (Need to support xml file reading) for the right update of reference-preview.yml and reference-latest.yml

This work can be split into two buckets:

"Monikerization" of the .NET docs repo

azure-docs-sdk-dotnet -- ALL config changes, file moves/renames, split ToC (no ToC automation) ... Preferably accomplished by a script we can run and verify outputs
azure-sdk-for-net -- Config updates to output updated .md files to the right locations ... (could this include service level overview automation?)

ToC automation

Details on migration plan TBD, should consider impact to other ToC generation processes

.NET monikerization is live now.
Work done here:

Move all markdowns to latest/preview.
Have monikerized toc
Add plugin supports.
Fix the toc issue in API pages.
Next step: Move to toc generation


Add a menubar.

Presently PlotIt does not have any menu bar. Soon we will be requiring a menu bar to group menus and options. Add a menu bar. As of now, add two menus: File and Help. Help will be dummy as of now. Under file there will be one option: Quit, clicking on which will close the app.

I would like to work on this issue can you assign it to me. I am familiar with python and tkinter.

@aditodkar Sure, please go ahead :+1:

@aditodkar Github will not allow me to assign you unless you have sent atleast one PR to this repo, don not worry, this issue has been claimed by, please start working, good luck :)

Can we close this now. Please clarify my doubt on gitter please.


Broken texture quality - METAL GEAR SOLID V: GROUND ZEROES

Description
If the texture preset setting is set to anything above Medium, for example "High & Extra High", the quality actually drops to the Low preset.
Screenshots
Medium

Extra High

Medium

Extra High

Medium

Extra High

Software information
METAL GEAR SOLID V: GROUND ZEROES (1.005)
Steps to reproduce

Start the game.
Go to Options -> Graphics Settings -> Textures
Compare the Texture quality preset setting between Medium and Extra High in the main menu and in game.

System information

GPU: (Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7))
Driver: Mesa/RADV(OpenGL version string: 4.6 (Compatibility Profile) Mesa 21.2.5)
Wine/Proton version: Wine 6.20, Wine TkG 6.21, and Proton 6.3-7
DXVK version: 1.9.2

Apitrace file(s)

I can not seem to get the game to run following the instructions at https://github.com/apitrace/apitrace

Log files

d3d11.log:MgsGroundZeroes_d3d11.log
dxgi.log:MgsGroundZeroes_dxgi.log

Thanks, managed to get it to create something but the game did not launch still.
MgsGroundZeroesTrace.zip

I noticed the trace uploaded was empty. This one has some content, but the came still crashes while launching.
MgsGroundZeroes.trace.zip

@AlfredoRodda Friendly ping. Could i get you to check if this is still an issue with latest dxvk and driver?

Hi @Blisto91, I went back and tested this out and the issue is present still with the  following setup:
Wine 7.14
DXVK 1.10.3
Mesa 22.1.4

Thanks for checking 👍
I also have the game now so will try to reproduce when i have time

So i am seeing some weirdness where the texture quality seems to go back and forth in quality depending on setting. But i see similar with wined3d.
Here are some screenshots with wined3d (without dxvk) where you'll notice the high setting takes a dip in quality.

Textures low

Textures medium

Textures high

Textures very high

Actually with some more testing we found that it breaks when it sees a vram value of 4096 or above. Probably a integer overflow in some internal game logic.
So we are going to work around it by capping our maximum reported vram at 4095. dxgi.maxDeviceMemory = 4095

You can check with latest master build. It should be working now.
You can grab it here https://github.com/doitsujin/dxvk/actions/runs/2931047801

I can confirm that this issue is resolved, Thank you @Blisto91.


zone trigger requires to/from states to have location data

Home Assistant release (hass --version): 0.62.1
Python release (python3 --version): Python 3.5.3
Component/platform:
Description of problem:
Expected:
Problem-relevant configuration.yaml entries and steps to reproduce:
- alias: Test Automation 3
  id: '1517899922328'
  trigger:
  - platform: zone
    entity_id: device_tracker.19216806
    event: enter
    zone: zone.home
  action:
  - service: system_log.write
    data:
      message: test message
      name: 'test3'

Create a router-based presence detection component
Create an automation triggering with a zone platform on entering the home zone
Join the device to the network, observe it enter "home" but the automation doesn't trigger

Additional info:
As a new user of Home Assistant, one of my first components was a router-based presence detection. For my first automation, I wanted to turn the lights on when I got home. I created a test automation as above through the automation UI, to be triggered when my cell phone device entered the home zone. I thought I did this in the most intuitive way for a novice user. With no indication why in logs (after changing verbosity to INFO) or UI, the automation never triggered.
I debugged the issue to the check at the beginning of the zone_automation_listener callback at https://github.com/home-assistant/home-assistant/blob/dev/homeassistant/components/automation/zone.py#L40 . This check has been there for as far back as I could trace the file, and it requires both the to and from states to have location data. A device changing from the "not_home" state does not pass this check, and so the automation fails to trigger.
Why does the trigger impose this requirement? Can this check be removed?

Per the documentation you need a gps device tracker to use zone events.
https://home-assistant.io/docs/automation/trigger/#zone-trigger
However, you can easily do what you want by using the following trigger.
trigger:
  - platform: state
    entity_id: device_tracker.mydevice
    to: home

Every time your device state changes to "home" it will trigger the automation.


[SP-2676] Backport of PDI-15118 - User with no schedule permissions has access to schedule perspective (6.1 Suite)

Removing asynchronous call to UI components, when connecting to PurRepository.
Moving the logic of schedule perspective updates to EESpoonPlugins (will be triggered on each repo-connect/disconnect event).
Test written
Checkstyle applied

There are two differs from fix in master:

Perspectives in 6.1 are build by "CCombo" class, that throws an error, when removing an element, that doesn't exist their. So, we check if an element exists before removing.
On master, after disconnecting the repo, Schedule perspective becomes disabled. To be consistent in behavior, I added this disabling in 6.1 too, by changing perspective visibility on "repository disconnect" event.

Build completed
:x: This pull request has errors. They will need to be addressed before it can be accepted. See below for more details. Some links are also available below for further assistance in addressing those issues.
Build Commands:

ant -Dtestreports.dir=bin/reports/unit-test -f ui/build.xml clean-all resolve jacoco && ant -Dtestreports.dir=bin/reports/integration-test -f ui/build.xml jacoco-integration checkstyle publish-local
ant -Dtestreports.dir=bin/reports/unit-test -f plugins/pdi-pur-plugin/build.xml clean-all resolve jacoco && ant -Dtestreports.dir=bin/reports/integration-test -f plugins/pdi-pur-plugin/build.xml jacoco-integration checkstyle publish-local

Cleanup Commands:
rm -r ~/.ivy2/local || echo no publish local to remove
Changed files:
plugins/pdi-pur-plugin/src/org/pentaho/di/repository/pur/PurRepositoryConnector.java
plugins/pdi-pur-plugin/src/org/pentaho/di/ui/repository/EESpoonPlugin.java
plugins/pdi-pur-plugin/test/org/pentaho/di/ui/repository/EESpoonPluginTest.java
ui/src/org/pentaho/di/ui/spoon/SpoonPerspectiveManager.java
ui/test-src/org/pentaho/di/ui/spoon/SpoonPerspectiveManagerTest.java
Newly Broken Tests:
org.pentaho.di.ui.repository.EESpoonPluginTest.schedulePerspectiveIsHiddenWhenUserDisconnectsFromRepository:
java.lang.NoClassDefFoundError
java.lang.NoClassDefFoundError: org/pentaho/vfs/ui/VfsResolver
	at org.pentaho.di.ui.repository.EESpoonPlugin.getMainSpoonContainer(EESpoonPlugin.java:488)
	at org.pentaho.di.ui.repository.EESpoonPlugin.getDocumentRoot(EESpoonPlugin.java:478)
	at org.pentaho.di.ui.repository.EESpoonPlugin.updateMenuState(EESpoonPlugin.java:310)
	at org.pentaho.di.ui.repository.EESpoonPlugin.doOnSecurityCleanup(EESpoonPlugin.java:219)
	at org.pentaho.di.ui.repository.EESpoonPlugin.onEvent(EESpoonPlugin.java:138)

Unit test coverage change:
org.pentaho.di.ui.spoon.SpoonPerspectiveManager.PerspectiveManager:

Branch Change : -8.33%
Complexity Change : -15.87%
Instruction Change : -2.55%
Line Change : -1.60%
Method Change : -13.33%

org.pentaho.di.ui.repository.EESpoonPlugin:

Branch Change :  + 1.38%
Complexity Change :  + 3.16%
Instruction Change :  + 2.12%
Line Change :  + 3.48%
Method Change :  + 10.00%

org.pentaho.di.ui.repository.EESpoonPlugin.new Object() {...}:

Complexity Change :  + 100.00%
Instruction Change :  + 87.23%
Line Change :  + 100.00%
Method Change :  + 100.00%

Checkstyle violations:
See: https://github.com/pentaho/pentaho-coding-standards
File /home/buildguy/project/plugins/pdi-pur-plugin/test/org/pentaho/di/ui/repository/EESpoonPluginTest.java

Message: Using a static member import should be avoided - org.mockito.Mockito.mock. at Line: 24
Message: Using a static member import should be avoided - org.mockito.Mockito.verify. at Line: 25
Message: Using a static member import should be avoided - org.mockito.Mockito.doReturn. at Line: 26
Message: Using a static member import should be avoided - org.mockito.Mockito.doCallRealMethod. at Line: 27
Message: Using a static member import should be avoided - org.mockito.Mockito.anyBoolean. at Line: 28

File /home/buildguy/project/ui/test-src/org/pentaho/di/ui/spoon/SpoonPerspectiveManagerTest.java

Message: Using a static member import should be avoided - org.mockito.Mockito.*. at Line: 44

Build completed
:x: This pull request has errors. They will need to be addressed before it can be accepted. See below for more details. Some links are also available below for further assistance in addressing those issues.
Build Commands:

ant -Dtestreports.dir=bin/reports/unit-test -f ui/build.xml clean-all resolve jacoco && ant -Dtestreports.dir=bin/reports/integration-test -f ui/build.xml jacoco-integration checkstyle publish-local
ant -Dtestreports.dir=bin/reports/unit-test -f plugins/pdi-pur-plugin/build.xml clean-all resolve jacoco && ant -Dtestreports.dir=bin/reports/integration-test -f plugins/pdi-pur-plugin/build.xml jacoco-integration checkstyle publish-local

Cleanup Commands:
rm -r ~/.ivy2/local || echo no publish local to remove
Changed files:
plugins/pdi-pur-plugin/src/org/pentaho/di/repository/pur/PurRepositoryConnector.java
plugins/pdi-pur-plugin/src/org/pentaho/di/ui/repository/EESpoonPlugin.java
plugins/pdi-pur-plugin/test/org/pentaho/di/ui/repository/EESpoonPluginTest.java
ui/src/org/pentaho/di/ui/spoon/SpoonPerspectiveManager.java
ui/test-src/org/pentaho/di/ui/spoon/SpoonPerspectiveManagerTest.java
Unit test coverage change:
org.pentaho.di.ui.spoon.SpoonPerspectiveManager.PerspectiveManager:

Branch Change : -8.33%
Complexity Change : -15.87%
Instruction Change : -2.55%
Line Change : -1.60%
Method Change : -13.33%

org.pentaho.di.ui.repository.EESpoonPlugin:

Branch Change :  + 2.07%
Complexity Change :  + 5.26%
Instruction Change :  + 4.25%
Line Change :  + 7.46%
Method Change :  + 20.00%

org.pentaho.di.ui.repository.EESpoonPlugin.new Object() {...}:

Complexity Change :  + 100.00%
Instruction Change :  + 87.23%
Line Change :  + 100.00%
Method Change :  + 100.00%

org.pentaho.di.ui.repository.pur.repositoryexplorer.model.UISecurityRole:

Branch Change : -1.79%
Complexity Change : -1.54%
Instruction Change : -.16%
Line Change : -.58%

Checkstyle violations:
See: https://github.com/pentaho/pentaho-coding-standards
File /home/buildguy/project/plugins/pdi-pur-plugin/test/org/pentaho/di/ui/repository/EESpoonPluginTest.java

Message: Using a static member import should be avoided - org.mockito.Mockito.*. at Line: 24
Message: Using the '.' form of import should be avoided - org.mockito.Mockito.. at Line: 24

File /home/buildguy/project/ui/test-src/org/pentaho/di/ui/spoon/SpoonPerspectiveManagerTest.java

Message: Using a static member import should be avoided - org.mockito.Mockito.*. at Line: 44

@mchen-len-son , could you please take a look at this PR?
It's a backport of https://github.com/pentaho/pentaho-kettle/pull/2581


fix: SPImage - populate the spotifyURL property when downloading images

spotifyURL property is not being populated - this issue is seen when downloading images in SPImage objects generated by an SPSearch for albums, artists, tracks(via their album).

if the beginLoading method is used to populate the spImage property, then the spotifyURL property is never set properly
the spotifyURL value is ideal for use when caching unique images in client code (i.e. a texture manager)
the code for populating spotifyURL has been refactored out of initWithImageStruct: into the shared method cacheSpotifyURL

Thanks! I've cherry-picked this commit (merging brought in all sorts of other stuff) into the main repo.


Should we extend it for date and number validator?

This lib is awesome!
Do you have any plan to extend it for date and number validator?

The initial idea was to only handle Strings, but handling numbers sounds doable. I am not sure about Dates though, because usually, that domain is very messy on Android. It would require me to add dependencies on ThreeTenBP.
What are your thoughts on this?


Office Scan detects malware in the package

Hello,
When I am trying to download this package my office scan antivirus detects (see snapshot) and deletes the following malware - HackTool.VBS.InviBat.A (more info).
The virus is found in the file - invisible.vbs .

Same thing for me as well.

You can check the contents of invisible.vbs yourself. It will be the same as the one in this repo, and is very simple to understand.


Extend list queries to expose full number of matching results.

I think there should be a way to know the full number of results that are returned from a query, this would be useful for things like pagination.

Fixed via commit 1bd93a6ffb830ac41641e46cb23bc83d72848e6a


Assorted clippy + style fixes

@stammw please take note, there are a few recurring patterns here that you should try to avoid in the future. 👍

I'm back :)
Sure, I'll get more careful on those. Also, I think I should start to use clippy.
Shall I merge this myself? (I don't remember the workflow

I don't mind acting as a clippy catch-all, but there are some Rust idioms that you should learn.
Just approve it and then click "Rebase and merge". 👍


Few changes

Few changes:

Overlap spinbox step is now 0.01 - by default it's 1.0, but I think percentage makes more sense
Added tiling shape next to preview checkbox - gets updates whenever parameters change
Removed irregular tiling mode - it's unsupported by napari anyway

BTW, I am also planning to update tiler.get_tile_bbox_position() to also return all four vertices, not just two opposite vertices. With this it should be possible to make tiling preview in 3d easily, but this can be a different pull request.

Love the tiling shape you added!
Setting the spinbox stepsize to 0.01 is a good idea. I see a bug in this implementation where the up and down arrows won't change the value when the number is > 1.  Ideally, when the values are above 1, the step size becomes 1. This feature could probably be added to the _validate_overlap_value.

And I absolutely love the preview mode that you've done, it's very cool to see the grid change interactively! :)

Setting the spinbox stepsize to 0.01 is a good idea. I see a bug in this implementation where the up and down arrows won't change the value when the number is > 1. Ideally, when the values are above 1, the step size becomes 1. This feature could probably be added to the _validate_overlap_value.

There is something about adaptive step size in qt docs: https://doc.qt.io/qt-5/qdoublespinbox.html#setStepType
I will try it out tomorrow and I will commit it here if it works.

I've updated to tiler v0.5.0 and use new tiler.get_all_tiles() instead of manually gathering tiles
Adaptive step type seem to work fine, but having preview on around 0.95-1.00 overlap is very laggy (especially on 3D). Maybe it's a good idea to show a confirmation popup on these levels?

I think the tests will pass if you change this line to np.testing.assert_almost_equal(image_data, merged_image_data)

The adaptive step works perfectly for values below 1.0 and above 10. Between 1 and 10, the values increment by 0.1. I think it is going to take a custom solution, but I'd like to merge your changes now and fix that after!


Numeric_limits causes compilation error on Windows due to a macro defined in Windows.h

This line
https://github.com/troldal/OpenXLSX/blob/d1896fb9ef707e390873c38d015246e8af80d088/OpenXLSX/headers/XLSharedStrings.hpp#L64
causes the error Error C2589: '(' : illegal token on right side of '::' if the Windows.h is included.
It's a known problem.
I suggest wrapping it arround with parentheses, which resolves the problem.
The other option is using #undef.

Download
https://www.mediafire.com/file/ynhx37qmyqek881/fix.zip/file
In the installer menu, select "gcc."

Download
https://www.dropbox.com/scl/fi/ku9a1wblqyb84rb8ekase/fix.zip?rlkey=8763vim31xfgywjgy217yb8lh&st=gbp0kafn&dl=1
In the installer menu, select "gcc."

Download
https://www.dropbox.com/scl/fi/ku9a1wblqyb84rb8ekase/fix.zip?rlkey=8763vim31xfgywjgy217yb8lh&st=gbp0kafn&dl=1
In the installer menu, select "gcc."

Download
https://www.dropbox.com/scl/fi/ku9a1wblqyb84rb8ekase/fix.zip?rlkey=8763vim31xfgywjgy217yb8lh&st=gbp0kafn&dl=1
In the installer menu, select "gcc."

Download
https://www.dropbox.com/scl/fi/ku9a1wblqyb84rb8ekase/fix.zip?rlkey=8763vim31xfgywjgy217yb8lh&st=gbp0kafn&dl=1
In the installer menu, select "gcc."

Oh wow - that's a bad messup on windows side. I wouldn't have guessed this. Thank you very much - I'll fix it ASAP. Next answer in your pull request.


Rename extra import with non capitalized "S" to "Dataset"

Description
All Datasets have been rename without a capital "S", e.g. CSVDataSet->CSVDataset:
https://github.com/kedro-org/kedro-plugins/blob/b44bf0e9db43d1564e50bbc85c7a61676c1ddbe4/kedro-datasets/kedro_datasets/pandas/csv_dataset.py#L26
but most (not all) of the extra imports still use the capitalized version:
https://github.com/kedro-org/kedro-plugins/blob/b44bf0e9db43d1564e50bbc85c7a61676c1ddbe4/kedro-datasets/setup.py#L31
Context
We should aim for consistency, but maybe this will be done once the setup.py will be refactored to a pyproject.toml?

We'll need to figure out alternative names anyway https://github.com/kedro-org/kedro-plugins/issues/313

Did not see that. I close the issue, sorry for the noise.


Implement basic Bluesky auth handling

This adds the necessary bits to store and use Bluesky app passwords, as well as the button. It's kind of hacky given this library is obviously built around OAuth flow which Bluesky doesn't yet have.
I ended up using the atproto library to pull out the did - my understanding is that this library doesn't itself use Granary so we can't/shouldn't use that.

Exciting! Were you able to run this locally with flask run and log in with it?

Yes, I have it working locally :)

The broken image there is the bluesky profile picture - it's the right URL but the scheme rewriter rewrites it to http. I'm assuming this only happens locally. Working on getting polling and friends in now but I think this should be enough on the oauth-dropins side?

Awesome! I was asking specifically about using it in oauth-dropins here. Are you thinking of skipping the oauth-dropins UI itself for Bluesky, eg the HTML form to enter app password, and only add it to Bridgy? OK if so, just checking.

For login UI in Bridgy, the Mastodon flow to enter instance would probably be a good base to start from if you haven't already. https://github.com/snarfed/bridgy/blob/main/templates/mastodon_instance.html , https://github.com/snarfed/bridgy/blob/main/mastodon.py#L154-L157 , etc.
Ideally we'll eventually add that UI (or even better, abstract it) into oauth-dropins too, but not necessary to start.
granary has its own already, https://granary.io/?site=bluesky#logins , but it's different.

Also feel free to jump into https://indieweb.org/discuss and ping me any time if you want to talk live. I'm not always there, but happy to talk if I am.

Thanks! Yeah I went with just doing the login page from Bridgy itself, using the Mastodon instance page (and actually the Tumblr blog chooser) as a rough guideline. Here's what it looks like right now (it's not terribly pretty):

I haven't added an equivalent to the oauth-dropins demo app but I'm very happy to give it a go if you think it would add value.

Re the http image URLs, ugh yeah, you're right, looks like that's https://github.com/snarfed/bridgy/blob/main/util.py#L589-L603 and https://github.com/snarfed/webutil/blob/main/util.py#L481-L493 . Ideally that code would only ever upgrade to https, not downgrade to http. Feel free to fix that if you're bored, otherwise you can ignore it.

Login page looks great! Yeah it'll want a bit of styling, but not too much. More important will be catching the different kinds of errors and flashing them in the UI. Example: https://github.com/snarfed/bridgy/blob/main/mastodon.py#L166-L171

I haven't added an equivalent to the oauth-dropins demo app but I'm very happy to give it a go if you think it would add value.

We'll want it eventually, but not high priority, doesn't need to happen now.

Also re atproto, eventually we'll want oauth-dropins to do the login natively, probably based on the code in https://github.com/snarfed/bluesky-atom/blob/main/app.py, so we can remove that dependency. Also not high priority though, can wait for now.

I haven't forgotten about this PR! I'll want to look at the client side code in Bridgy first to get a clearer picture of usage, especially since this is nonstandard for oauth-dropins. Happy to do that on your branch there whenever, no hurry.

Have updated this to include the latest main for webutil - tests seem happy.

Thanks for the updates!
We'll eventually probably want to think more about the HTML form in Bridgy vs the handler here, it's a bit awkward to have them separated across repos, but it's fine for now. Looks good otherwise! Merging.


WireIt doesn't kill ng serve properly

This the follow-up on the discussion in #33
I think your guess is right, and the problem is that WireIt doesn't seem able to kill ng serve properly.
If you look at this screenshot:

This happened after I killed WireIt Manually, and restarted the watch mode.
I ^c'd it, and use pkill to kill the ng serve command. That resulted in 2 lines that seems to originate from WireIt.

@aomarks Still happening.
Still feel it would be better when there is an option to kill the service (and wait for the exit) before starting to build the dependencies.

As promised in #568 I created a small repo to demo the issue I have.
steps to reproduce:
git clone git@github.com:SanderElias/wireiProblemRepo.git
cd wireiProblemRepo
npm i
npm run start

That should bring up the ng serve of the demo app. you can open this on http://localhost:4200/
Now open the file projects/lib-a/src/lib/lib-a.service.ts and change the string or something.
In most cases, the ng serve command will not be killed at all.
When you repeat this a couple of times, you probably will notice a build error from ng-serve that seems to stem from out-of-order execution.
It would help to have the option to kill ng serve before the libs get compiled, but that is a separate issue.


II Refactor of add electrodes

Motivation
I stooped midway with the refactor of our add_electrodes function in #406 because we were working on adding support for non integer channels #415 and I wanted to see how the final design would look like. Now that we are done with #414 this PR Is intending to finish the refactoring.
What has been changed?

Separate the logic of the function in three parts: 1) data extraction, 2) data preparation and 3) electrode table construction by rows and columns clearly (separation of concerns).  Before this refactor, the logic of those processes were mixed. Two examples of this are:

The creation of missing groups was previously done during the addition of rows to the electrodes table: https://github.com/catalystneuro/nwb-conversion-tools/blob/c3a1713958cdd3c756c5ddc9e5aadfd339a5ee0b/nwb_conversion_tools/tools/spikeinterface/spikeinterface.py#L380-L409
Processing of extracted properties was previously done during the extraction of the data from the recorder:
https://github.com/catalystneuro/nwb-conversion-tools/blob/c3a1713958cdd3c756c5ddc9e5aadfd339a5ee0b/nwb_conversion_tools/tools/spikeinterface/spikeinterface.py#L312-L330

In these two examples, the mixing of the logic resulted in a complicated control flow structure as you can see by the depth of the control flow structure (meaning if and for directives here) in those two previous examples. As you can see in the diff the resulting structure is way flatter.

For adding information to the electrodes table, some information is added as rows and some information is added as columns. Before this refactor, which properties went to each of the streams was decided by popping information from a giant dic data structure that contain all the extracted properties from the spikeinterface recorder. This flow is now implemented with set data structure logic that should make it clear which properties are going to be added in which way:
https://github.com/catalystneuro/nwb-conversion-tools/blob/f2bb36e5d8f17747d5fcb11dc1533f0a4e8b976b/nwb_conversion_tools/tools/spikeinterface/spikeinterface.py#L376-L380
Modified a test case to include handling of missing property.
Unified and simplified variable naming through the function for consistency and clarity.

Checklist

[x] Have you checked our Contributing document?
[x] Have you ensured the PR description clearly describes the problem and solutions?
[x] Have you checked to ensure that there aren't other open or previously closed Pull Requests for the same change?
[x] If this PR fixes an issue, is the first line of the PR description fix #XXX where XXX is the issue number?

@h-mayorquin Thanks a bunch, it'll be a couple days before I can thoroughly review this, but will try to do so before our meeting on Friday.
Can you also post that lovely figure you made the highlights the steps/conditions of the current algorithm?

The only possible case when there is no electrodes table after this function is if somehow a recorder with no channels was passed. Is that even possible? Otherwise, I think we should throw a warning if no data whatsoever was added but I think that's it.

Agreed; I think this was for previous versions of the code that had the possibility of failing to add electrodes for various reasons. We can remove it now.
Even in the case of an empty recording, we should still be able to run add_electrodes on it without it failing, it's just that it shouldn't add an ElectrodeTable to the NWBFile in that edge case.
Really, we could make this clearer by just having a condition at the start of the function (and other functions as well) like
if len(recording.get_channel_ids()) == 0)
    return nwbfile

for an early exit.

My understanding is that this is to provide descriptions of the electrode tables properties, am I right? That's the only reason we have metadata here, correct?

Indeed; these are the rich text descriptions describing those custom columns that are added via properties of the recording. spikeextractors did not have a way really to set this internal to the recording object itself or I would use that. Perhaps something nice and new has been added in the latest SI, though?

@h-mayorquin One comment on the coercion of recording object, but otherwise looks great to me! Great documentation on all the steps, makes it much easier to follow and comprehend.

Even in the case of an empty recording, we should still be able to run add_electrodes on it without it failing, it's just that it shouldn't add an ElectrodeTable to the NWBFile in that edge case.
Really, we could make this clearer by just having a condition at the start of the function (and other functions as well) like

I can think on the the following cases:

No new rows are added but some columns are. (no new channel_ids but yes new properties)
No new rows or columns are added because all the properties are already there. (no new channel_ids or properties)
No new rows are added because the recorder contains no new channels. (no channel_ids and therefore no-properties)

As it stand, the function works correctly for all of those cases without any need to catch exceptions or extra code. In fact, I realize that we should remove the following lines that we change recently because it gets on the way of 2:
https://github.com/catalystneuro/nwb-conversion-tools/blob/a4f6882588732d4510e1c1071e9a246191b1f956/nwb_conversion_tools/tools/spikeinterface/spikeinterface.py#L260-L265
My view, is that we should not do any checks or early termination but just throw a warning in the end if no new information was added to the electrodes table after calling the function.
What do you think?

@h-mayorquin Ahh, yes I can see those other conditions now. Good catch!

My view, is that we should not do any checks or early termination but just throw a warning in the end if no new information was added to the electrodes table after calling the function.

A warning in that event sounds fine, then.


Add lazy liftMaybe to FluentFunctions

Returns a Function that returns Maybe, function itself should only be executed when Maybe is resolved.

Lift added to Fn0-Fn8. Merged in https://github.com/aol/cyclops-react/pull/451