A compilation bug in latex

Type: Bug
In latex,If you start with a bracket in "\begin{equation}\begin{aligned}\end{aligned}\end{equation}'', the content inside the bracket is compiled as blank.

VS Code version: Code 1.80.2 (2ccd690cbff1569e4a83d7c43d45101f817401dc, 2023-07-27T20:40:28.909Z)
OS version: Windows_NT x64 10.0.22621
Modes:
Extensions (9)

Extension
Author (truncated)
Version

latex-workshop
Jam
9.13.3

vscode-language-pack-zh-hans
MS-
1.80.2023071209

python
ms-
2023.12.0

vscode-pylance
ms-
2023.7.40

jupyter
ms-
2023.6.1101941928

jupyter-keymap
ms-
1.1.2

jupyter-renderers
ms-
1.0.17

vscode-jupyter-cell-tags
ms-
0.1.8

vscode-jupyter-slideshow
ms-
0.1.5

This issue is caused by an extension, please file it with the repository (or contact) the extension has linked in its overview in VS Code or the marketplace for VS Code. See also our issue reporting guidelines. If you don't know which extension is causing the problem, you can run Help: Start extension bisect from the command palette (F1) to help identify the problem extension.
Happy Coding!


Python client support

Changes necessary to run Python client test suite.
basho/riak-python-client#434

https://bugs.python.org/issue8813

Resolved by #4


state 'expanded' set in useControlledState prevents toggling

Describe the bug
Version: 7.0.0-beta.27
I implemented a tree table and it should be initially expanded to a specific path.
When setting the expand state with the useControlledState option, it's not possible to collapse or expand the rows anymore. As an alternative I tried to use the initialState.expanded option, but the initial state resets, when the data is updated.
To Reproduce
Steps to reproduce the behavior:
I replicated both of the problems in a codesandbox
useControlledState example: https://codesandbox.io/s/sparkling-dust-cc8ny (App.js, line:37-46, 59)

Add useControlledState option to Table options and set the expanded state.

initialState example: https://codesandbox.io/s/busy-cherry-mvlhj (App.js, line: 48, 153-161)

Add initialState: { expanded: ["0", "0.0"] } to table options
Add a timeout to simulate data loading

Expected behavior
It should be possible to initially expand rows and still be able to expand/collapse them.
Desktop (please complete the following information):

OS: iOS
Browser: chrome
Version 78.0.3904.108

Try adding getResetExpandedDeps: false to your table options :) That should keep the data-change from clobbering your expanded state.
Check out more here: https://github.com/tannerlinsley/react-table/blob/master/docs/faq.md#how-do-i-stop-my-table-state-from-automatically-resetting-when-my-data-changes

This discussion happening right now is relevant to this issue: https://github.com/tannerlinsley/react-table/issues/1729

Does anybody know what happened to this FAQ:
https://github.com/tannerlinsley/react-table/blob/master/docs/faq.md#how-do-i-stop-my-table-state-from-automatically-resetting-when-my-data-changes

@robgwin Here you go: https://react-table.tanstack.com/docs/faq#how-do-i-stop-my-table-state-from-automatically-resetting-when-my-data-changes

Is it possible to have useControlledState to control the "expanded" key and toggling both working together? Once I set the expanded key in ControlledState then those toggling doesn't work just like @Lukas742  mentioned. Is it expected?

I'm also having issues with this. If I use the useControlledState option for expanded rows, I can persist the expanded state when the data updates, but have to wait for a re-render (as documented) for the expansion to happen rather than having the toggle action trigger the re-render.
Using websockets for data updates, redux/reselect for state management, and react-table v7.3.2 with filtering / sorting / expanding rows / subComponents / editable cells all enabled.
When new data comes in, I don't want the current sort / filter / expanded rows to reset, only the applicable data to be updated.
Is there an actual working example of the FAQ referenced being implemented? It doesn't appear to work at present.


feat: add deno deploy

PR: Thank you!
Reason

[ ] Fix: Wrong information
[ ] Fix: Typo error
[ ] Fix: License violation
[x] Add: New Item

Type

[ ] Node.js Hosting
[ ] VPS
[ ] Static hosting

Name
Deno Deploy
Link
https://deno.com/deploy
Thank you for improving project
This helps others find VPS or hosting of their choise

Thank you for contribution @iamfarishirfan


Changing the 'note_off' property doesn't affect the 'duration_sec' property

I noticed something. Not sure if it is a bug. I have a PerformedPart. When I change the note_off property of a note, then the duration_sec property of the corresponding note_array() stays the same.
Here is an example of how I encountered this phenomenon:
print(performedPart.notes[0])
print(performedPart.note_array()[0])

performedPart.notes[0]['note_off'] += 1.3

print(performedPart.notes[0])
print(performedPart.note_array()[0])

I print the values of the first note before and after changing the note_off value. I can see that this value has changed accordingly. The note_array() shows two types of duration, but not a note_off value. It shows the duration_sec and the duration_tick. Now in my output I see that the value of duration_sec has not changed, but the value of duration_tick has changed as expected.
Am I missing something or is this a bug or is this intentional?
I noticed this because of using the Piano Roll. After changing the note_off value, the new created piano roll shows the same as before. So it seems the piano roll uses this duration_sec value, that hasn't changed.
For now I found a workaround. For changing the length of a note I now use the note_array()'s property duration_sec instead of note_off. And after the change I overwrite my notes of the PerformedPart with the changed note_array().
Like so:
performanceNoteArray = performedPart.note_array()
performanceNoteArray[0]['duration_sec'] += 1.3
performedPart = pt.performance.PerformedPart.from_note_array(performanceNoteArray)

But I'm still curious if it should be possible to get a correct result in the piano roll just with changing note_off
Thanks a lot,
Tim

I also noticed another related thing.
When I set the note_on and note_off values of notes of a performedPart by hand, and when I then try to compute the piano roll, I get the following error message: (coming from file partitura/utils/music.py, line 1316)
raise ValueError("Note durations should be >= 0!")

But as I set the note_on and note_off values by myself, I know that they should result in positive durations for the notes. Here is how I did set the values:
note_on_val = 0.0
note_off_val = 1.0
for note in performedPart.notes:
        note['note_on'] = note_on_val
        note['note_off'] = note_off_val
        note_on_val += 1.0
        note_off_val += 1.0

So it looks like changing properties of performedPart.notes doesn't effect the note_array() that is generated to compute the piano roll (at least I guess that the piano roll uses the note_array() and not the .notes property)

Hi!
sorry for the late answer! This behavior is bit confusing. Here's a script that explains the main cases:
import partitura as pt
import numpy as np
performedPart = pt.load_performance_midi(pt.EXAMPLE_MIDI)[0]

# when creating a note array,
# and the SOUND_OFF property is available,
# it's used for duration_sec -> no change due to NOTE_OFF
print("changes in 'note_off' don't show up in note array, if there is a 'sound_off'")
print("original 'note_off': ",performedPart.notes[0]['note_off'])
print("same note in note array ",performedPart.note_array()[0])
performedPart.notes[0]['note_off'] += 1.0
print("changed 'note_off': ",performedPart.notes[0] )
print("note in note array ",performedPart.note_array()[0])
print("dtypes of note array 'onset_sec', 'duration_sec','onset_tick','duration_tick'")

print("-----------------------------")
# when creating a note array,
# and the NOTE_OFF_TICK property is UNavailable,
# NOTE_OFF is used for duration_sec -> change due to NOTE_OFF
for n in performedPart.notes:
    n.pop('note_on_tick', None)
    n.pop('note_off_tick', None)
print("delete the NOTE_OFF_TICK and the the tick values are inferred from 'note_off' and not 'sound_off'.")
print("the 4th value in note array ('duration_ticks') is higher than before, and inconsistent with the other duration")
print("same note in note array ",performedPart.note_array()[0])
print("dtypes of note array 'onset_sec', 'duration_sec','onset_tick','duration_tick'")

# for manipulations of the performedpart: change both 'sound_off' and 'note_off', for consistency delete all ticks information

print("-----------------------------")
# when creating a performedpart from note_array,
# only the times in seconds are used...
print("when creating the a performedpart from note_array, times in seconds are used")
na_to_change = np.copy(performedPart.note_array())
print("original note in note array ",na_to_change[0])
print("original 'note_off' in performedpart ", pt.performance.PerformedPart.from_note_array(na_to_change).notes[0]["note_off"])
print("change 'duration_tick'")
na_to_change[0]["duration_tick"] = 17
print("changed note in note array ", na_to_change[0])
print("UNCHANGED 'note_off' in performedpart ", pt.performance.PerformedPart.from_note_array(na_to_change).notes[0]["note_off"])
print("change 'duration_sec'")
na_to_change[0]["duration_sec"] = 100.
print("changed note in note array ", na_to_change[0])
print("CHANGED 'note_off' in performedpart ", pt.performance.PerformedPart.from_note_array(na_to_change).notes[0]["note_off"])

# for manipulations of the note_array: change both values in seconds, 'onset_sec' / 'duration_sec', tick times are discarded

The main points are the comments underneath the tests:

for manipulations of the performedpart: change both 'sound_off' and 'note_off', for consistency delete all ticks information
for manipulations of the note_array: change both values in seconds, 'onset_sec' / 'duration_sec', tick times are discarded

The next release will address the negative duration issue, the rest remains as is for now. I hope you can find a workaround!

Alright, thank you! This is very helpful :)


Débug outside cluster

Someone could write a walkthrough on how to debug Keda outside the cluster using vscode.

aminebizid- I have hard a hard time doing this due to the metrics adapter. I could not get a version of it running stand alone outside the cluster. Without this, KEDA is debuggable as a stand alone app. So a workaround is to comment the adapter setup portion in main.go and start it up.

Could you show us an example of vscode launch.json

example of launch.json for debugging Keda controller
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Launch",
            "type": "go",
            "request": "launch",
            "mode": "auto",
            "program": "${workspaceFolder}/cmd/manager/main.go",
            "env": {"OPERATOR_NAME": "keda-operator", "WATCH_NAMESPACE": ""},
            "args": ["--kubeconfig=/path/to/config/file"]
        }
    ]
}


Sub-Task 12.3: Visual indication for unfollowing a user

Task: Once the user confirms the unfollow request, a display somewhere on the screen should indicate the action that was done.

Deleting issues, linked to US#12


Can't create new user account

I tried to register for Roadmap stage as I can't login currently. This threw an error - see below.

A similar issue exists in DMPonline. See screengrab

This is not fixed on roadmap staging. I get this error

@stephaniesimms can you test out on dmponline-test? I've used up all email accounts I have already ;-)

@sjDCC I just tested both roadmap stage and dmponline-test with no errors creating new accounts. I used the Sign Up route w/no Shib pathways. It worked with and without choosing a org. Although I did get an error message about an invalid email address that is in fact valid (screenshot below)

Ok, have re-tried and managed to create an account without error. I suspect the fault was that I tried to register a new account with an email address I had shared a plan with and not validated or followed up. When I checked the backend there was a stem account for s.jones@arts that is showing to have not accepted terms yet.
@stephaniesimms Could you maybe try to replicate that and see if you get the same error? If not, I'm happy to close out. It could just be something random that was wrong with my email in the system as we're trialling it so much?

@sjDCC let me try to reproduce the error before closing. because we def don't want users to get this ugly error message.

@sjDCC I confirmed your suspicions by testing on Roadmap-stage. I invited a user to co-author a plan. Then I tried to create a new account for that user (instead of accepting the email invitation) and received the error below. This is a bug - somehow we need to allow the user to follow either path to create an account.

@vyruss do you want to pull this back to pending or start a new ticket? We definitely need a better fix than that error page when someone who has an unvalidated stem account tries to sign up.
At minimum we need an error message to say their email address is already taken, but if they don't have the email invite to follow the relevant link they're stuck, so a better workflow / solution is needed. Ideally allow them to just set a password for the unvalidated account?

Can you close this and describe the issue in another ticket please?

Done. See ticket #322


[brightcove] add support for brightcove in page embed(fixes #6824)(fixes #5946)

the same error happen here again (https://github.com/rg3/youtube-dl/pull/6785#issue-105191077) if someone has access to the site and can run the tests and check if they need to be changed

as i said:

cannot download extracted videos urls
i can only access the json request
so i just guess that urls like that(streaming_src in api json output):
ttp://brightcove06-f.akamaihd.net/929656772001/201509/,1022/929656772001_4463689346,1238/929656772001_4463679919,2898/929656772001_4463667480,3238/929656772001_4463679901,3898/929656772001_4463667481,3898/929656772001_4463667493,898/929656772001_4463667510,001_Popcorn-Time--English.mp4.csmil/bitrate=2?nn=795285514001&primaryToken=1442406600_89d3546e357d8b62ebff54243b5f8f18&pubId=929656772001&videoId=4463358922001
are f4m but as you get an error i think they are smil manifests that link to other links(like in theplatform).
if you can confirm what is in the file or send the file content so i can fix it.

the NerdistIE can be removed because they use the brightcove in page embed

it should work now.

i fixed some mistakes i made before(copy and paste) in the test and field extraction.

i changed the code it should work in python 2.6.

Thanks.


2D localization mode have roll pitch and Zaxis drit

@carlosmccosta the localization topic also seems have Zaxis drift.

Hello,
For 3 DoF localization (x, y, yaw), you must use the dynamic_robot_localization_system.launch, which is configured for using 2D lidars and 2D occupancy grids.
drl assumes that your lidar is mounted parallel to the floor and after transforming the lidar data to the odom TF, it will set the points Z coordinate to zero.
If your lidar is not parallel to the floor, you will not be able to use 3 DoF localization algorithms and maps, because they all assume sensors parallel to the floor to ensure that when the sensor changes place within the map, the measurements remain consistent.
Have a nice day,

@carlosmccosta
hello,
I do use the [dynamic_robot_localization_system.launch],and below is the launch parameter.

<!-- published pose -->
<arg name="pose_stamped_publish_topic" default="localization_pose" />

<!-- frame ids -->
<arg name="map_frame_id" default="map" />
<arg name="odom_frame_id" default="odom" />
<arg name="base_link_frame_id" default="laser" />
<arg name="sensor_frame_id" default="laser" />

<!-- main configurations -->
<arg name="use_ukf_filtering" default="false" />
<arg name="nodes_namespace" default="super_slam" />
<arg name="nodes_respawn" default="true" />
<arg name="localization" default="localization" />
<arg name="pcl_verbosity_level" default="ALWAYS" /> <!-- VERBOSE | DEBUG | INFO | WARN | ERROR | ALWAYS || ALWAYS -> no console output -->
<arg name="ros_verbosity_level" default="FATAL" /> <!-- DEBUG | INFO | WARN | ERROR | FATAL -->
<arg name="yaml_configuration_filters_filename" default="$(find super_slam)/yaml/configs/filters/filters_large_map_2d.yaml" />
<arg name="yaml_configuration_pose_estimation_filename" default="$(find super_slam)/yaml/configs/empty.yaml" />
<arg name="yaml_configuration_tracking_filename" default="$(find super_slam)/yaml/configs/pose_tracking/cluttered_environments_dynamic_fast_2d.yaml" />
<arg name="yaml_configuration_recovery_filename" default="$(find super_slam)/yaml/configs/pose_recovery/recovery_slam_2d.yaml" />
<arg name="yaml_configuration_covariance_filename" default="$(find super_slam)/yaml/configs/covariance_estimation/covariance_estimation.yaml" if="$(arg use_ukf_filtering)" />
<arg name="yaml_configuration_covariance_filename" default="$(find super_slam)/yaml/configs/empty.yaml" unless="$(arg use_ukf_filtering)" />

<!-- reference map data -->
<arg name="reference_pointcloud_filename" default="" />
<arg name="reference_pointcloud_preprocessed_save_filename" default="" />
<arg name="reference_pointcloud_available" default="true" />
<arg name="reference_pointcloud_type_3d" default="false" />
<arg name="reference_pointcloud_type" default="2D" unless="$(arg reference_pointcloud_type_3d)"/>
<arg name="reference_pointcloud_type" default="3D" if="$(arg reference_pointcloud_type_3d)" />
<arg name="reference_pointcloud_update_mode" default="NoIntegration" if="$(arg reference_pointcloud_available)"/> <!-- NoIntegration | FullIntegration | InliersIntegration | OutliersIntegration -->
<arg name="reference_pointcloud_update_mode" default="FullIntegration" unless="$(arg reference_pointcloud_available)"/>
<arg name="reference_pointcloud_keypoints_filename" default="" />
<arg name="reference_pointcloud_keypoints_save_filename" default="" />
<arg name="reference_pointcloud_descriptors_filename" default="" />
<arg name="reference_pointcloud_descriptors_save_filename" default="" />
<arg name="reference_costmap_topic" default="" if="$(arg reference_pointcloud_type_3d)" />
<arg name="reference_costmap_topic" default="/map" unless="$(arg reference_pointcloud_type_3d)" />
<arg name="reference_costmap_topic_updated" default="/map_updated" />
<arg name="reference_pointcloud_topic" default="reference_pointcloud_updated" if="$(arg reference_pointcloud_type_3d)" /> <!-- OctoMap is configured to use topic 'reference_pointcloud_update' -->
<arg name="reference_pointcloud_topic" default="" unless="$(arg reference_pointcloud_type_3d)" />

<!-- sensor data from point cloud topic -->
<arg name="ambient_pointcloud_topic" default="ambient_pointcloud" />
<arg name="ambient_pointcloud_topic_disabled_on_startup" default="false" />
<arg name="imu_topic" default="/imu/data" />

<!-- initial pose setup -->
<arg name="robot_initial_pose_in_base_to_map" default="false" /> <!-- recommendation: use false to allow direct publish of tf between odom -> map -->
<arg name="robot_initial_pose_available" default="false" />
<arg name="robot_initial_x" default="0.0" />
<arg name="robot_initial_y" default="0.0" />
<arg name="robot_initial_z" default="0.0" />
<arg name="robot_initial_roll" default="0.0" />
<arg name="robot_initial_pitch" default="0.0" />
<arg name="robot_initial_yaw" default="0.0" />

<!-- laser scan assembler -->
<arg name="use_laser_assembler" default="true" />
<arg name="assemble_lasers_on_map_frame" default="false" /> <!-- if false -> assembles lasers on odom frame -->
<arg name="laser_assembly_frame" default="$(arg map_frame_id)" if="$(arg assemble_lasers_on_map_frame)" />
<arg name="laser_assembly_frame" default="$(arg odom_frame_id)" unless="$(arg assemble_lasers_on_map_frame)" />
<arg name="laser_assembly_motion_estimation_source_frame_id" default="" />
<arg name="laser_assembly_motion_estimation_target_frame_id" default="" />
<arg name="laser_scan_topics" default="/scan" />
<arg name="min_range_cutoff_percentage_offset" default="2.00" />
<arg name="max_range_cutoff_percentage_offset" default="0.95" />
<arg name="laser_assembler_remove_invalid_measurements" default="true" />
<arg name="number_of_tf_queries_for_spherical_interpolation" default="4" />
<arg name="number_of_scans_to_assemble_per_cloud" default="1" />
<arg name="timeout_for_cloud_assembly" default="0.5" />
<arg name="dynamic_update_of_assembly_configuration_from_twist_topic" default="" />
<arg name="dynamic_update_of_assembly_configuration_from_odometry_topic" default="/odom" />
<arg name="dynamic_update_of_assembly_configuration_from_imu_topic" default="/imu/data" />
<arg name="min_number_of_scans_to_assemble_per_cloud" default="1" />
<arg name="max_number_of_scans_to_assemble_per_cloud" default="1" />
<arg name="min_timeout_seconds_for_cloud_assembly" default="0.15" />
<arg name="max_timeout_seconds_for_cloud_assembly" default="0.15" />
<arg name="max_linear_velocity" default="0.0001" /> <!-- when the linear velocity of the robot is equal or greater than this value, the number of scans to join and assembly timeout will be set to their minimum values specified above -->
<arg name="max_angular_velocity" default="0.0001" /> <!-- when the angular velocity of the robot is equal or greater than this value, the number of scans to join and assembly timeout will be set to their minimum values specified above -->

<!-- tf configurations -->
<arg name="publish_tf_map_odom" default="true" />
<arg name="publish_initial_pose" default="true" />
<arg name="publish_pose_tf_rate" default="10.0" />
<arg name="tf_lookup_timeout" default="0.1" />
<arg name="publish_last_pose_tf_timeout_seconds" default="-330.0" /> <!-- set to negative value for publishing a tf only when drl estimates a new pose -->
<arg name="invert_tf_transform" default="false" />
<arg name="invert_tf_hierarchy" default="false" />
<arg name="transform_pose_to_map_frame_id" default="true" />

<!-- map update -->
<arg name="use_octomap_for_dynamic_map_update" default="false" />
<arg name="use_octomap_only_to_build_occupancy_grid" default="false" />
<arg name="octomap_pointcloud_topic" default="aligned_pointcloud" />  <!-- ambient_pointcloud | aligned_pointcloud | aligned_pointcloud_outliers | aligned_pointcloud_inliers -->
<arg name="octomap_file" default="" />
<arg name="octomap_sensor_frame_id" default="$(arg sensor_frame_id)" />
<arg name="octomap_resolution" default="0.02" />
<arg name="octomap_minimum_amount_of_time_between_ros_msg_publishing" default="5.0" />
<arg name="octomap_minimum_number_of_integrations_before_ros_msg_publishing" default="10" />
<arg name="octomap_override_sensor_z" default="false" if="$(arg reference_pointcloud_type_3d)" />
<arg name="octomap_override_sensor_z" default="true" unless="$(arg reference_pointcloud_type_3d)" />
<arg name="octomap_override_sensor_z_value" default="0.0" />
<arg name="octomap_cloud_out_topic" default="reference_pointcloud_updated" />
<arg name="initial_2d_map_file" default="" />

<!-- ==================================================== localization system ============================================= -->
<!-- <node pkg="super_slam" type="localization" name="$(arg localization)" respawn="$(arg nodes_respawn)" ns="$(arg nodes_namespace)" clear_params="true" output="screen" launch-prefix="gdbserver localhost:1337" > -->
<node pkg="super_slam" type="localization" name="$(arg localization)" respawn="$(arg nodes_respawn)" ns="$(arg nodes_namespace)" clear_params="true" output="screen" >
	<param name="pcl_verbosity_level" type="str" value="$(arg pcl_verbosity_level)" />
	<param name="ros_verbosity_level" type="str" value="$(arg ros_verbosity_level)" />
	<param name="message_management/tf_timeout" type="double" value="$(arg tf_lookup_timeout)" />
	<param name="frame_ids/map_frame_id" type="str" value="$(arg map_frame_id)" />
	<param name="frame_ids/odom_frame_id" type="str" value="$(arg odom_frame_id)" />
	<param name="frame_ids/base_link_frame_id" type="str" value="$(arg base_link_frame_id)" />
	<param name="frame_ids/sensor_frame_id" type="str" value="$(arg sensor_frame_id)" />
	<param name="initial_pose/robot_initial_pose_in_base_to_map" type="bool" value="$(arg robot_initial_pose_in_base_to_map)" />
	<param name="initial_pose/robot_initial_pose_available" type="bool" value="$(arg robot_initial_pose_available)" />
	<param name="initial_pose/position/x" type="double" value="$(arg robot_initial_x)" />
	<param name="initial_pose/position/y" type="double" value="$(arg robot_initial_y)" />
	<param name="initial_pose/position/z" type="double" value="$(arg robot_initial_z)" />
	<param name="initial_pose/orientation_rpy/roll" type="double" value="$(arg robot_initial_roll)" />
	<param name="initial_pose/orientation_rpy/pitch" type="double" value="$(arg robot_initial_pitch)" />
	<param name="initial_pose/orientation_rpy/yaw" type="double" value="$(arg robot_initial_yaw)" />
	<param name="subscribe_topic_names/pose_topic" type="str" value="$(arg pose_topic)" />
	<param name="subscribe_topic_names/pose_stamped_topic" type="str" value="$(arg pose_stamped_topic)" />
	<param name="subscribe_topic_names/pose_with_covariance_stamped_topic" type="str" value="$(arg pose_with_covariance_stamped_topic)" />
	<param name="subscribe_topic_names/ambient_pointcloud_topic" type="str" value="$(arg ambient_pointcloud_topic)" />
	<param name="subscribe_topic_names/ambient_pointcloud_topic_disabled_on_startup" type="bool" value="$(arg ambient_pointcloud_topic_disabled_on_startup)" />
	<param name="subscribe_topic_names/reference_costmap_topic" type="str" value="$(arg reference_costmap_topic)" />
	<param name="subscribe_topic_names/reference_pointcloud_topic" type="str" value="$(arg reference_pointcloud_topic)" />
	<param name="publish_topic_names/pose_with_covariance_stamped_publish_topic" type="str" value="$(arg pose_with_covariance_stamped_publish_topic)" />
	<param name="publish_topic_names/pose_stamped_publish_topic" type="str" value="$(arg pose_stamped_publish_topic)" />
	<param name="reference_pointclouds/reference_pointcloud_filename" type="str" value="$(arg reference_pointcloud_filename)" />
	<param name="reference_pointclouds/reference_pointcloud_preprocessed_save_filename" type="str" value="$(arg reference_pointcloud_preprocessed_save_filename)" />
	<param name="reference_pointclouds/reference_pointcloud_type" type="str" value="$(arg reference_pointcloud_type)" />
	<param name="reference_pointclouds/reference_pointcloud_available" type="bool" value="$(arg reference_pointcloud_available)" />
	<param name="reference_pointclouds/reference_pointcloud_update_mode" type="str" value="$(arg reference_pointcloud_update_mode)" />
	<param name="keypoint_detectors/reference_pointcloud/reference_pointcloud_keypoints_filename" type="str" value="$(arg reference_pointcloud_keypoints_filename)" />
	<param name="keypoint_detectors/reference_pointcloud/reference_pointcloud_keypoints_save_filename" type="str" value="$(arg reference_pointcloud_keypoints_save_filename)" />
	<param name="initial_pose_estimators_matchers/feature_matchers/reference_pointcloud_descriptors_filename" type="str" value="$(arg reference_pointcloud_descriptors_filename)" />
	<param name="initial_pose_estimators_matchers/feature_matchers/reference_pointcloud_descriptors_save_filename" type="str" value="$(arg reference_pointcloud_descriptors_save_filename)" />
	<rosparam command="load" file="$(arg yaml_configuration_filters_filename)" subst_value="true" />
	<rosparam command="load" file="$(arg yaml_configuration_pose_estimation_filename)" subst_value="false" />
	<rosparam command="load" file="$(arg yaml_configuration_recovery_filename)" subst_value="true" />
	<rosparam command="load" file="$(arg yaml_configuration_tracking_filename)" subst_value="true" />
	<rosparam command="load" file="$(arg yaml_configuration_covariance_filename)" subst_value="true" />
</node>

<include file="$(find super_slam)/launch/ukf.launch" ns="$(arg nodes_namespace)" if="$(arg use_ukf_filtering)" >
	<arg name="planar_pose_estimation" default="false" if="$(arg reference_pointcloud_type_3d)" />
	<arg name="planar_pose_estimation" default="true" unless="$(arg reference_pointcloud_type_3d)" />
	<arg name="tracking_state_reset_topic" default="initial_pose_with_covariance" />
	<arg name="sensor_timeout" default="$(arg publish_last_pose_tf_timeout_seconds)" />
	<arg name="map_frame_id" default="$(arg map_frame_id)" />
	<arg name="odom_frame_id" default="$(arg odom_frame_id)" />
	<arg name="base_link_frame_id" default="$(arg base_link_frame_id)" />
	<arg name="odometry_topic" default="$(arg dynamic_update_of_assembly_configuration_from_odometry_topic)" />
	<arg name="imu_topic" default="$(arg imu_topic)" />
	<arg name="nodes_respawn" default="$(arg nodes_respawn)" />
</include>

<!-- map to odom publisher -->
<include file="$(find pose_to_tf_publisher)/launch/pose_to_tf_publisher.launch" if="$(arg publish_tf_map_odom)" >
	<arg name="publish_rate" default="$(arg publish_pose_tf_rate)" />
	<arg name="tf_lookup_timeout" default="1.5" />
	<arg name="publish_last_pose_tf_timeout_seconds" default="$(arg publish_last_pose_tf_timeout_seconds)" />
	<arg name="pose_with_covariance_stamped_topic" default="$(arg pose_with_covariance_stamped_topic)" /> <!-- rviz topic -->
	<arg name="pose_stamped_topic" default="$(arg pose_stamped_publish_topic)" />
	<arg name="odometry_topic" default="" />
	<arg name="map_frame_id" default="$(arg map_frame_id)_drl" if="$(arg use_ukf_filtering)" />
	<arg name="map_frame_id" default="$(arg map_frame_id)" unless="$(arg use_ukf_filtering)" />
	<arg name="odom_frame_id" default="$(arg odom_frame_id)" />
	<arg name="base_link_frame_id" default="$(arg base_link_frame_id)" />
	<arg name="publish_initial_pose" default="$(arg publish_initial_pose)" />
	<arg name="initial_pose_in_base_to_map" default="$(arg robot_initial_pose_in_base_to_map)" />
	<arg name="initial_x" default="$(arg robot_initial_x)" />
	<arg name="initial_y" default="$(arg robot_initial_y)" />
	<arg name="initial_z" default="$(arg robot_initial_z)" />
	<arg name="initial_roll" default="$(arg robot_initial_roll)" />
	<arg name="initial_pitch" default="$(arg robot_initial_pitch)" />
	<arg name="initial_yaw" default="$(arg robot_initial_yaw)" />
	<arg name="invert_tf_transform" default="true" if="$(arg use_ukf_filtering)" />
	<arg name="invert_tf_transform" default="$(arg invert_tf_transform)" unless="$(arg use_ukf_filtering)" />
	<arg name="invert_tf_hierarchy" default="true" if="$(arg use_ukf_filtering)" />
	<arg name="invert_tf_hierarchy" default="$(arg invert_tf_hierarchy)" unless="$(arg use_ukf_filtering)" />
	<arg name="transform_pose_to_map_frame_id" default="false" if="$(arg use_ukf_filtering)" />
	<arg name="transform_pose_to_map_frame_id" default="$(arg transform_pose_to_map_frame_id)" unless="$(arg use_ukf_filtering)" />
	<arg name="parameters_namespace" default="super_slam" />
	<arg name="nodes_respawn" default="$(arg nodes_respawn)" />
</include>

<!-- laser assembler -->
<include file="$(find laserscan_to_pointcloud)/launch/laserscan_to_pointcloud_assembler.launch" ns="$(arg nodes_namespace)" if="$(arg use_laser_assembler)" >
	<arg name="laser_scan_topics" default="$(arg laser_scan_topics)" />
	<arg name="number_of_tf_queries_for_spherical_interpolation" default="$(arg number_of_tf_queries_for_spherical_interpolation)" />
	<arg name="number_of_scans_to_assemble_per_cloud" default="$(arg number_of_scans_to_assemble_per_cloud)" />
	<arg name="timeout_for_cloud_assembly" default="$(arg timeout_for_cloud_assembly)" />
	<arg name="remove_invalid_measurements" default="$(arg laser_assembler_remove_invalid_measurements)" />
	<arg name="dynamic_update_of_assembly_configuration_from_twist_topic" default="$(arg dynamic_update_of_assembly_configuration_from_twist_topic)" />
	<arg name="dynamic_update_of_assembly_configuration_from_odometry_topic" default="$(arg dynamic_update_of_assembly_configuration_from_odometry_topic)" />
	<arg name="dynamic_update_of_assembly_configuration_from_imu_topic" default="$(arg dynamic_update_of_assembly_configuration_from_imu_topic)" />
	<arg name="min_range_cutoff_percentage_offset" default="$(arg min_range_cutoff_percentage_offset)" />
	<arg name="max_range_cutoff_percentage_offset" default="$(arg max_range_cutoff_percentage_offset)" />
	<arg name="min_number_of_scans_to_assemble_per_cloud" default="$(arg min_number_of_scans_to_assemble_per_cloud)" />
	<arg name="max_number_of_scans_to_assemble_per_cloud" default="$(arg max_number_of_scans_to_assemble_per_cloud)" />
	<arg name="min_timeout_seconds_for_cloud_assembly" default="$(arg min_timeout_seconds_for_cloud_assembly)" />
	<arg name="max_timeout_seconds_for_cloud_assembly" default="$(arg max_timeout_seconds_for_cloud_assembly)" />
	<arg name="max_linear_velocity" default="$(arg max_linear_velocity)" />
	<arg name="max_angular_velocity" default="$(arg max_angular_velocity)" />
	<arg name="target_frame" default="$(arg laser_assembly_frame)" />
	<arg name="recovery_frame" default="$(arg odom_frame_id)" if="$(arg assemble_lasers_on_map_frame)" />
	<arg name="recovery_frame" default="" unless="$(arg assemble_lasers_on_map_frame)" />
	<arg name="initial_recovery_transform_in_base_link_to_target" default="$(arg robot_initial_pose_in_base_to_map)" />
	<arg name="base_link_frame_id" default="$(arg base_link_frame_id)" />
	<arg name="motion_estimation_source_frame_id" default="$(arg laser_assembly_motion_estimation_source_frame_id)" />
	<arg name="motion_estimation_target_frame_id" default="$(arg laser_assembly_motion_estimation_target_frame_id)" />
	<arg name="recovery_to_target_frame_transform_initial_x" default="$(arg robot_initial_x)" />
	<arg name="recovery_to_target_frame_transform_initial_y" default="$(arg robot_initial_y)" />
	<arg name="recovery_to_target_frame_transform_initial_z" default="$(arg robot_initial_z)" />
	<arg name="recovery_to_target_frame_transform_initial_roll" default="$(arg robot_initial_roll)" />
	<arg name="recovery_to_target_frame_transform_initial_pitch" default="$(arg robot_initial_pitch)" />
	<arg name="recovery_to_target_frame_transform_initial_yaw" default="$(arg robot_initial_yaw)" />
	<arg name="nodes_respawn" default="$(arg nodes_respawn)" />
</include>

<!-- OctoMap for dynamic map update -->
<group if="$(arg use_octomap_for_dynamic_map_update)" ns="$(arg nodes_namespace)" >
	<include file="$(find super_slam)/launch/octo_map.launch" >
		<arg name="octomap_file" default="$(arg octomap_file)" />
		<arg name="cloud_in" default="$(arg octomap_pointcloud_topic)" />
		<arg name="map_frame_id" default="$(arg map_frame_id)" />
		<arg name="base_frame_id" default="$(arg base_link_frame_id)" />
		<arg name="sensor_frame_id" default="$(arg octomap_sensor_frame_id)" />
		<arg name="override_sensor_z" default="$(arg octomap_override_sensor_z)" />
		<arg name="override_sensor_z_value" default="$(arg octomap_override_sensor_z_value)" />
		<arg name="resolution" default="$(arg octomap_resolution)" />
		<arg name="minimum_amount_of_time_between_ros_msg_publishing" default="$(arg octomap_minimum_amount_of_time_between_ros_msg_publishing)" />
		<arg name="minimum_number_of_integrations_before_ros_msg_publishing" default="$(arg octomap_minimum_number_of_integrations_before_ros_msg_publishing)" />
		<arg name="cloud_out_topic" default="$(arg octomap_cloud_out_topic)" />
		<arg name="map_out_topic" default="$(arg reference_costmap_topic_updated)" if="$(arg use_octomap_only_to_build_occupancy_grid)" />
		<arg name="map_out_topic" default="$(arg reference_costmap_topic)" unless="$(arg use_octomap_only_to_build_occupancy_grid)" />
		<arg name="nodes_respawn" default="$(arg nodes_respawn)" />
	</include>

	<group if="$(arg reference_pointcloud_available)" >
		<node name="map_server" pkg="map_server" type="map_server" args="$(arg initial_2d_map_file)" respawn="$(arg nodes_respawn)" clear_params="true" output="screen" unless="$(arg reference_pointcloud_type_3d)" >
			<param name="frame_id" type="str" value="$(arg map_frame_id)" />
			<remap from="map" to="initial_2d_map" />
		</node>
	</group>
</group>

Hello,
Check if the roll and pitch angles between the TFs odom and laser are 0 degrees.
The 2d icp configured in drl assumes that the laser is parallel to the ground and uses the pcl transformation_estimation_2D
Check your ROS parameter server (rosparam get / -p) to confirm that the 2d icp was correctly loaded.
Looking at your rviz screenshot it seems that the laser is tilted and not parallel to the ground.
Have a nice day :)

hello @carlosmccosta
this is my param,please check if anything wrong,I need a consecutive pose for navigation so I subscribed the tf (map->laser),I also found that the tf have x y yaw drift and increase the drift with time,but the pose from topic doesn't have drift,and the tf drift will be clean when drl received /initpose topic.
dynamic_robot_localization:
drl_localization_node:
cloud_analyzers:
angular_distribution_analyzer:
number_of_angular_bins: 270
compute_inliers_angular_distribution: true
compute_outliers_angular_distribution: true
filters:
ambient_pointcloud:
1_voxel_grid:
downsample_all_data: false
filter_limit_field_name: z
filter_limit_max: 2.0
filter_limit_min: -1.0
filtered_cloud_publish_topic:
leaf_size_x: 0.02
leaf_size_y: 0.02
leaf_size_z: 0.02
save_leaf_layout: false
5_random_sample:
filtered_cloud_publish_topic: ambient_pointcloud_filtered
invert_sampling: false
number_of_random_samples: 2500
reference_pointcloud:
1_voxel_grid:
downsample_all_data: false
filter_limit_field_name: z
filter_limit_max: 1.8
filter_limit_min: -0.001
filtered_cloud_publish_topic:
leaf_size_x: 0.02
leaf_size_y: 0.02
leaf_size_z: 0.02
save_leaf_layout: false
frame_ids:
base_link_frame_id: laser
map_frame_id: map
odom_frame_id: odom
sensor_frame_id: laser
initial_pose:
orientation_rpy:
pitch: 0.0
roll: 0.0
yaw: 0.0
position:
x: 0.0
y: 0.0
z: 0.0
robot_initial_pose_available: false
robot_initial_pose_in_base_to_map: false
initial_pose_estimators_matchers:
feature_matchers:
reference_pointcloud_descriptors_filename:
reference_pointcloud_descriptors_save_filename:
keypoint_detectors:
reference_pointcloud:
reference_pointcloud_keypoints_filename:
reference_pointcloud_keypoints_save_filename:
localization_point_type: PointXYZINormal
message_management:
tf_timeout: 0.1
outlier_detectors:
euclidean_outlier_detector:
aligned_pointcloud_inliers_publish_topic: /aligned_pointcloud_inliers
aligned_pointcloud_outliers_publish_topic: /aligned_pointcloud_outliers
max_inliers_distance: 0.06
pcl_verbosity_level: ALWAYS
publish_topic_names:
pose_stamped_publish_topic: localization_pose
pose_with_covariance_stamped_publish_topic: localization_pose_with_covariance
reference_pointclouds:
reference_pointcloud_available: true
reference_pointcloud_filename:
reference_pointcloud_preprocessed_save_filename:
reference_pointcloud_type: 2D
reference_pointcloud_update_mode: NoIntegration
ros_verbosity_level: FATAL
subscribe_topic_names:
ambient_pointcloud_topic: ambient_pointcloud
ambient_pointcloud_topic_disabled_on_startup: false
pose_stamped_topic: initial_pose_stamped
pose_topic: initial_pose
pose_with_covariance_stamped_topic: /initialpose
reference_costmap_topic: /map
reference_pointcloud_topic:
tracking_matchers:
ignore_height_corrections: false
point_matchers:
iterative_closest_point_2d:
correspondence_randomness: 50
display_cloud_aligment: false
euclidean_fitness_epsilon: 0.0001
match_only_keypoints: false
max_correspondence_distance: 0.2
max_number_of_ransac_iterations: 140
max_number_of_registration_iterations: 140
maximum_number_of_displayed_correspondences: 0
maximum_optimizer_iterations: 150
ransac_outlier_rejection_threshold: 0.06
rotation_epsilon: 0.002
transformation_epsilon: 0.0001
use_reciprocal_correspondences: false
tracking_recovery_matchers:
point_matchers:
iterative_closest_point:
convergence_time_limit_seconds: 0.75
correspondence_randomness: 50
display_cloud_aligment: false
euclidean_fitness_epsilon: 1.0e-06
match_only_keypoints: false
max_correspondence_distance: 1.75
max_number_of_ransac_iterations: 200
max_number_of_registration_iterations: 300
maximum_number_of_displayed_correspondences: 0
maximum_optimizer_iterations: 100
ransac_outlier_rejection_threshold: 0.06
rotation_epsilon: 0.002
transformation_epsilon: 1.0e-06
use_reciprocal_correspondences: false
transformation_validators:
euclidean_transformation_validator:
max_new_pose_diff_angle: 0.45
max_new_pose_diff_distance: 0.45
max_outliers_percentage: 0.7
max_root_mean_square_error: 0.045
max_transformation_angle: 0.45
max_transformation_distance: 0.45
transformation_validators_tracking_recovery:
euclidean_transformation_validator:
max_new_pose_diff_angle: 0.8
max_new_pose_diff_distance: 0.8
max_outliers_percentage: 0.7
max_root_mean_square_error: 0.045
max_transformation_angle: 0.8
max_transformation_distance: 0.8
laserscan_to_pointcloud_assembler_jason_2342_5905162112553559908:
base_link_frame_id: laser
dynamic_update_of_assembly_configuration_from_imu_topic:
dynamic_update_of_assembly_configuration_from_odometry_topic: /odom
dynamic_update_of_assembly_configuration_from_twist_topic:
enforce_reception_of_laser_scans_in_all_topics: true
include_laser_intensity: true
initial_recovery_transform_in_base_link_to_target: false
laser_frame:
laser_scan_topics: /scan
max_angular_velocity: 0.0001
max_linear_velocity: 0.0001
max_number_of_scans_to_assemble_per_cloud: 1
max_range_cutoff_percentage_offset: 0.95
max_timeout_seconds_for_cloud_assembly: 0.15
min_number_of_scans_to_assemble_per_cloud: 1
min_range_cutoff_percentage_offset: 2.0
min_timeout_seconds_for_cloud_assembly: 0.15
motion_estimation_source_frame_id:
motion_estimation_target_frame_id:
number_of_scans_to_assemble_per_cloud: 1
number_of_tf_queries_for_spherical_interpolation: 4
pointcloud_publish_topic: ambient_pointcloud
recovery_frame:
recovery_to_target_frame_transform_initial_pitch: 0.0
recovery_to_target_frame_transform_initial_roll: 0.0
recovery_to_target_frame_transform_initial_x: 0.0
recovery_to_target_frame_transform_initial_y: 0.0
recovery_to_target_frame_transform_initial_yaw: 0.0
recovery_to_target_frame_transform_initial_z: 0.0
remove_invalid_measurements: true
target_frame: odom
tf_lookup_timeout: 0.15
timeout_for_cloud_assembly: 0.5
pose_to_tf_publisher_node_jason_2342_2149551555346644467:
base_link_frame_id: laser
discard_older_poses: true
float_topic:
float_update_field:
float_update_field_orientation_in_degrees: false
initial_pitch: 0.0
initial_pose_in_base_to_map: false
initial_roll: 0.0
initial_x: 0.0
initial_y: 0.0
initial_yaw: 0.0
initial_z: 0.0
invert_tf_hierarchy: false
invert_tf_transform: false
map_frame_id: map
odom_frame_id: odom
odometry_topic:
pose_stamped_topic: localization_pose
pose_with_covariance_stamped_topic: /initialpose
poses_filename:
publish_initial_pose: true
publish_last_pose_tf_timeout_seconds: -3330.0
publish_rate: 3000.0
tf_lookup_timeout: 1.5
tf_time_offset: 0.0
transform_pose_to_map_frame_id: true
transform_tf_message_source:
transform_tf_message_target:

Hello,
What is your TF chain configuration from map_frame_id to base_link_frame_id ?
From your previous launch files, I assumed that it was map -> odom -> laser
If your odom is not accurate, you can disable its usage within drl by:

Setting the odom_frame_id to a value equal to the base_link_frame_id
Since a TF frame can only have one parent in the TF tree, for bypassing the odom_frame_id, it is necessary to:

Set to true the flag invert_tf_transform
Set to true the flag invert_tf_hierarchy
This way, the TF and pose_stamped_publish_topic will be published directly from base_link_frame_id to map_frame_id (the pose is specified in the map_frame_id coordinate system)
You can check the drl pose and TF in rviz with the fixed frame configured to map and can also check in the console with:

rostopic echo /dynamic_robot_localization/localization_pose
rosrun tf tf_echo map laser _rate:=30 _precision:=4

I rechecked one of my validation testes from a rosbag (freiburg2_pioneer_slam1.launch) and the latest commit of drl in the noetic-devel branch is performing as expected and without drift, as shown in this video:
https://www.youtube.com/watch?v=jAJ5wiN-mJ8
Have a nice day,

hello @carlosmccosta
I found another problem from drl,my odom is accurate so I read the map->laser tf to navigation,and the pose will jump to far a way if the map changed alot and the initial_pose_estimation is set,so I disabled the initial_pose_estimation.but some time it wouldn‘t be able to find  it self in the map even the environment is not so bad,I also tune some param from pose_recovery file,so does the initial_pose_estimation will help this situation？or how to tune the initial_pose_estimation so the pose will not jump to far?

@carlosmccosta
below is the param I set
initial_pose_estimation_2d***********************************
normal_estimators:
reference_pointcloud:
display_normals: false
display_occupancy_grid_pointcloud: false
flip_normals_using_occupancy_grid_analysis: true
occupancy_grid_analysis_radius_resolution_percentage: 4.0
normal_estimator_sac:
model_type: 'SACMODEL_LINE'
method_type: 'SAC_RANSAC'
inlier_distance_threshold: 0.06
max_iterations: 50
sac_termination_probability: 0.99
optimize_coefficients: true
min_model_radius: -1.0
max_model_radius: -1.0
random_samples_max_k: 0
random_samples_max_radius: 0.06
minimum_inliers_percentage: 0.5
ambient_pointcloud:
compute_normals_when_tracking_pose: false
compute_normals_when_recovering_pose_tracking: false
compute_normals_when_estimating_initial_pose: true
display_normals: false
normal_estimator_sac:
model_type: 'SACMODEL_LINE'
method_type: 'SAC_RANSAC'
inlier_distance_threshold: 0.06
max_iterations: 50
sac_termination_probability: 0.99
optimize_coefficients: true
min_model_radius: -1.0
max_model_radius: -1.0
random_samples_max_k: 0
random_samples_max_radius: 0.065
minimum_inliers_percentage: 0.5
curvature_estimators:
reference_pointcloud:
principal_curvatures_estimation:
colorize_pointcloud_with_curvatures: true
search_k: 0
search_radius: 0.06
curvature_type: 'CURVATURE_TYPE_MEAN'
update_normals_with_principal_component_directions: false
ambient_pointcloud:
principal_curvatures_estimation:
colorize_pointcloud_with_curvatures: true
search_k: 0
search_radius: 0.06
curvature_type: 'CURVATURE_TYPE_MEAN'
update_normals_with_principal_component_directions: false
keypoint_detectors:
reference_pointcloud:
sift_3d:
min_scale: 0.015
number_octaves: 4
number_scales_per_octave: 8
min_contrast: 0.0035
sift3d_keypoints_cloud_publish_topic: 'reference_keypoints'
intrinsic_shape_signature_3d:
salient_radius: 0.14
non_max_radius: 0.08
normal_radius: 0.08
border_radius: 0.0
threshold21: 0.975
threshold32: 0.975
min_neighbors: 5
angle_threshold: 1.57
iss3d_keypoints_cloud_publish_topic: 'reference_keypoints'
ambient_pointcloud:
    compute_keypoints_when_tracking_pose: false
    compute_keypoints_when_recovering_pose_tracking: false
    compute_keypoints_when_estimating_initial_pose: true
    sift_3d:
        min_scale: 0.015
        number_octaves: 4
        number_scales_per_octave: 8
        min_contrast: 0.0035
        sift3d_keypoints_cloud_publish_topic: 'ambient_keypoints'

intrinsic_shape_signature_3d:
salient_radius: 0.14
non_max_radius: 0.08
normal_radius: 0.08
border_radius: 0.0
threshold21: 0.975
threshold32: 0.975
min_neighbors: 5
angle_threshold: 1.57
iss3d_keypoints_cloud_publish_topic: 'ambient_keypoints'
initial_pose_estimators_matchers:
feature_matchers:
display_feature_matching: false
save_descriptors_in_binary_format : true
keypoint_descriptors:
feature_descriptor_k_search: 0
feature_descriptor_radius_search: 0.07
fpfh:
number_subdivisions_f1: 11
number_subdivisions_f2: 11
number_subdivisions_f3: 11
pfh:
use_internal_cache: true
maximum_cache_size: 33554432
shot:
lrf_radius: 0.5
shape_context_3d:
minimal_radius: 0.2
point_density_radius: 0.4
unique_shape_context:
minimal_radius: 0.1
point_density_radius: 0.2
local_radius: 2.5
esf:
feature_descriptor_radius_search: 0.25
    matchers:
        max_correspondence_distance: 0.3
        max_number_of_registration_iterations: 2500
        sample_consensus_initial_alignment_prerejective:
            convergence_time_limit_seconds: 3.0
            similarity_threshold: 0.6
            inlier_fraction: 0.75
            inlier_rmse: 0.2
            number_of_samples: 32
            correspondence_randomness: 5

sample_consensus_initial_alignment:
min_sample_distance: 0.5
number_of_samples: 3
correspondence_randomness: 15
point_matchers:
    iterative_closest_point:

iterative_closest_point_2d:
iterative_closest_point_non_linear:
iterative_closest_point_generalized:
        convergence_time_limit_seconds: 2.0
        max_correspondence_distance: 10.0
        transformation_epsilon: 0.0001
        euclidean_fitness_epsilon: 0.0001
        max_number_of_registration_iterations: 350
        max_number_of_ransac_iterations: 350
        ransac_outlier_rejection_threshold: 0.05
        match_only_keypoints: false
        display_cloud_aligment: false
        maximum_number_of_displayed_correspondences: 0
        rotation_epsilon: 0.002
        correspondence_randomness: 50
        maximum_optimizer_iterations: 100
        use_reciprocal_correspondences: false

transformation_validators_initial_alignment:
euclidean_transformation_validator:
max_transformation_angle: -1.0
max_transformation_distance: 3
max_new_pose_diff_angle: -1.0
max_new_pose_diff_distance: 3
max_root_mean_square_error: -1.0
max_outliers_percentage: 0.5

cluttered_environments_dynamic_fast_2d*
tracking_matchers:
ignore_height_corrections: false
point_matchers:
iterative_closest_point:
    iterative_closest_point_2d:

iterative_closest_point_non_linear:
iterative_closest_point_generalized:
        max_correspondence_distance: 0.05
        transformation_epsilon: 0.0001
        euclidean_fitness_epsilon: 0.0001
        max_number_of_registration_iterations: 140
        max_number_of_ransac_iterations: 140
        ransac_outlier_rejection_threshold: 0.05
        match_only_keypoints: false
        display_cloud_aligment: false
        maximum_number_of_displayed_correspondences: 0
        rotation_epsilon: 0.002
        correspondence_randomness: 50
        maximum_optimizer_iterations: 150
        use_reciprocal_correspondences: false

outlier_detectors:
euclidean_outlier_detector:
max_inliers_distance: 0.05
aligned_pointcloud_outliers_publish_topic: '/aligned_pointcloud_outliers'
aligned_pointcloud_inliers_publish_topic: '/aligned_pointcloud_inliers'
cloud_analyzers:
compute_inliers_angular_distribution: true
compute_outliers_angular_distribution: true
angular_distribution_analyzer:
number_of_angular_bins: 270
transformation_validators:
euclidean_transformation_validator:
max_transformation_angle: 0.3
max_transformation_distance: 0.1
max_new_pose_diff_angle: 0.3
max_new_pose_diff_distance: 0.1
max_root_mean_square_error: 0.1
max_outliers_percentage: 0.66
transformation_validators_tracking_recovery:
euclidean_transformation_validator:
max_root_mean_square_error: 0.1
max_outliers_percentage: 0.66

recovery_slam_2d******************
tracking_recovery_matchers:
point_matchers:
iterative_closest_point:
iterative_closest_point_2d:
iterative_closest_point_non_linear:
iterative_closest_point_generalized:
        convergence_time_limit_seconds: 0.75
        max_correspondence_distance: 1.0
        transformation_epsilon: 0.000001
        euclidean_fitness_epsilon: 0.000001
        max_number_of_registration_iterations: 300
        max_number_of_ransac_iterations: 200
        ransac_outlier_rejection_threshold: 0.04
        match_only_keypoints: false
        display_cloud_aligment: false
        maximum_number_of_displayed_correspondences: 0
        rotation_epsilon: 0.002
        correspondence_randomness: 50
        maximum_optimizer_iterations: 100
        use_reciprocal_correspondences: false

transformation_validators_tracking_recovery:
euclidean_transformation_validator:
max_transformation_angle: 0.5
max_transformation_distance: 0.2
max_new_pose_diff_angle: 0.5
max_new_pose_diff_distance: 0.2
max_root_mean_square_error: 0.045
max_outliers_percentage: 0.66

filters_large_map_2d.yaml***************
localization_point_type: 'PointXYZINormal'
filters:
reference_pointcloud:
1_voxel_grid:
leaf_size_x: 0.02
leaf_size_y: 0.02
leaf_size_z: 0.02
filter_limit_field_name: 'z'
filter_limit_min: -0.001
filter_limit_max: 1.8
filtered_cloud_publish_topic: ''
downsample_all_data: false
save_leaf_layout: false
ambient_pointcloud:
1_voxel_grid:
leaf_size_x: 0.02
leaf_size_y: 0.02
leaf_size_z: 0.02
filter_limit_field_name: 'z'
filter_limit_min: -1.0
filter_limit_max: 2.0
filtered_cloud_publish_topic: ''
downsample_all_data: false
save_leaf_layout: false
2_approximate_voxel_grid:
leaf_size_x: 0.025
leaf_size_y: 0.025
leaf_size_z: 0.025
filtered_cloud_publish_topic: ''
downsample_all_data: false
3_radius_outlier_removal:
radius_search: 2.0
min_neighbors_in_radius: 6
invert_removal: false
filtered_cloud_publish_topic: ''
4_crop_box:
box_min_x: -20.0
box_min_y: -20.0
box_min_z: -20.0
box_max_x: 20.0
box_max_y: 20.0
box_max_z: 20.0
box_translation_x: 0.0
box_translation_y: 0.0
box_translation_z: 0.0
box_rotation_roll: 0.0
box_rotation_pitch: 0.0
box_rotation_yaw: 0.0
invert_selection: false
filtered_cloud_publish_topic: ''
    5_random_sample:
        number_of_random_samples: 2500
        invert_sampling: false
        filtered_cloud_publish_topic: 'ambient_pointcloud_filtered'

6_statistical_outlier_removal:
number_of_neighbors_for_mean_distance_estimation: 6
standard_deviation_multiplier: 2.0
invert_selection: false
filtered_cloud_publish_topic: ''
7_covariance_sampling:
number_of_samples: 100
filtered_cloud_publish_topic: ''
8_radius_outlier_removal:
radius_search: 1.0
min_neighbors_in_radius: 2
invert_removal: false
filtered_cloud_publish_topic: ''

*****covariance_estimation.yaml
registration_covariance_estimator:
error_metric: 'PointToPlane3D'
correspondance_estimator: 'CorrespondenceEstimation'
correspondence_distance_threshold: 0.1
sensor_std_dev_noise: 0.015
use_reciprocal_correspondences: false
filtered_reference_cloud_publish_topic: 'covariance_reference_cloud'
filtered_ambient_cloud_publish_topic: 'covariance_ambient_cloud'
random_sample:
number_of_random_samples: 300
invert_sampling: false
filtered_cloud_publish_topic: ''

Hello,
If drl is rejecting its pose estimation in your typical use case / environment, then you may need to increase the tolerance to outliers, by specifying to drl that the map data might be X% different in relation to the sensor data.
This is controlled with max_inliers_distance (in meters) and max_outliers_percentage (range 0 to 1 | note that currently this is an outlier_ratio -> 1 equals to 100%)
It is possible to control how far the tracking and recovery algorithms corrections translate and rotate the robot pose with max_transformation_distance | max_transformation_angle and for recovery recovery_2d.yaml#L27 | recovery_2d.yaml#L26
For the initial pose algorithms, currently it not applied this jump limit to allow the robot to start lost and then find itself in the map.
To ensure that ICP runs fast, confirm that you have the look up table enabled with the correspondence_estimation_approach set to CorrespondenceEstimationLookupTable and with its map_cell_resolution equal to your map, otherwise the icp might stop its convergence after reaching its time limit defined in convergence_time_limit_seconds
If your robot is moving very fast, you may need to increase max_correspondence_distance
Have a nice day :)

hello @carlosmccosta
my odom is accurate，in odom topic I set 0 to angular_velocity and oritation in X Y axis,and set ignore_height_corrections: true, and it doesnt have roll pitch and Zaxis drit,but it still have x y pose drift and yaw drift,you can see the video below,with the increasement of running time ,the drift also increase,and the drift can be clear by send a initialpose topic ,seem the initialpose will triggle another program that recalculate the pose in the map and clear some drift,what I did is send initialpose every 60s ,but I think it can be fix in code which I can not found it,by the way I disabled the pose_estimation module,so maybe the initpose triggled the pose recovery module and cleared the drift?
https://drive.google.com/file/d/18HqVuU20vCfN-7rdiI9YQJneGD024vOo/view?usp=share_link
this video been record after the robot running 1 hours

Hello,
When you send an initial pose, the matching algorithm changes from yaml_configuration_tracking_filename to yaml_configuration_recovery_filename
Assuming you have not changed the yamls loaded, you can compare the configurations here:

cluttered_environments_dynamic_large_map_2d.yaml
recovery_2d.yaml

You should not resend initial poses to fix this issue. Instead, you should adjust the configuration of drl.
Namely, try to increase these 3 parameters:

max_correspondence_distance
convergence_time_limit_seconds
max_number_of_registration_iterations

You can also try and change the tracking algorithm from icp 2d to the standard icp, by commenting this line and uncommenting this line and then changing transformation_estimation_approach to TransformationEstimationSVD
Lastly, if it still has drift, you can disable the look up tables and use the standard k-d trees for correspondence estimation (they are slower, but you can try then and see if the problem persists). For that, change the correspondence_estimation_approach to CorrespondenceEstimation
Have a nice day,

CorrespondenceEstimation

hello @carlosmccosta
I have copy [cluttered_environments_dynamic_large_map_2d.yaml]
and [recovery_2d.yaml] to the drl and change the [correspondence_estimation_approach] in cluttered_environments_dynamic_large_map_2d.yaml to CorrespondenceEstimation,as you can see the link below,the problem still exist,I think this problem is not about the parameter,the point clould topic /dynamic_robot_localization/ambient_pointcloud and laserscan topic still drift slowly to map seems the drift are connect with the running distance,but /dynamic_robot_localization/aligned_pointcloud always aligned with map.maybe some TF code didn`t wrote properly?
https://drive.google.com/file/d/1AA0dVNP_Xgvrz4QMpX9hlSJ_GixUyJhv/view?usp=share_link

CorrespondenceEstimation

@carlosmccosta
I found out that the /dynamic_robot_localization/aligned_pointcloud frame id is map,however the /dynamic_robot_localization/ambient_pointcloud frame id is odom,but the /dynamic_robot_localization/localization_detailed frame is map,

CorrespondenceEstimation

@carlosmccosta I found out that the /dynamic_robot_localization/aligned_pointcloud frame id is map,however the /dynamic_robot_localization/ambient_pointcloud frame id is odom,but the /dynamic_robot_localization/localization_detailed frame is map,

@carlosmccosta
looks like the accumulation error of odom has been add to loc pose,I set , the drift problem seems to be fixed, but it came out a new problem,when I send initpose topic,the pose seems very unstable,and is very hard to let laser aligned with map at startup. and the pose is not stable as before when the robot is static.
https://drive.google.com/file/d/1Bf7CYHju0hmNpTgOpUr5S3t4NAqcuFPK/view?usp=share_link

Hello,
The ambient_pointcloud topic should be assembled in the odom frame and not in the map frame, because laser -> odom provides a smooth tf chain, while laser -> map has discrete small jumps that are necessary and introduced to correct the odom drift over time. These jumps distort the point cloud assebly when the spherical interpolation is enabled. As such, the point cloud assemby should not be done on the map frame, otherwise, these distortions will cause major oscilations in the alignment performed by drl.
The aligned_pointcloud is the result of the alignment performed by drl and is on map frame, and as such, is the one you should be analyzing in rviz. You should not evaluate the ambient_pointcloud because that is the input for drl and not its result. And of course, since the ambient_pointcloud is on odom frame, it will show on rviz with the odometry drift. That is working as is supposed to.
These frames names are this way on purpose and follow the standard ros tf for mobile robotics:
https://www.ros.org/reps/rep-0105.html
https://navigation.ros.org/setup_guides/transformation/setup_transforms.html
Looking at your videos, the aligned_pointcloud is correctly overlapping the map. So drl was working correctly all along !
Keep in mind that every localization system has oscillations (which is different than drift) in its pose estimation due to lidar noise and also due to sliding of the alignment within the walls. Namely, if your map has walls with more than one pixel thickness in the occupancy grid, the alignment algorithms may oscillate. This is normal and if you want to minimize this oscillation, you need to have maps with 1 pixel thickness walls.
For inspecting the results of drl, set your rviz world frame to map and inspect the aligned_pointcloud overlap with the occupancy grid / map.
For checking the drl pose, you have the topics localization_pose and localization_detailed
Alternatively, you can check the pose between map and base_link or base_footprint
In drl_configs.yaml you can check the subscribe / input topics and the publish / output topics of drl
Have a nice day,

Hello,
The ambient_pointcloud topic should be assembled in the odom frame and not in the map frame, because laser -> odom provides a smooth tf chain, while laser -> map has discrete small jumps that are necessary and introduced to correct the odom drift over time. These jumps distort the point cloud assebly when the spherical interpolation is enabled. As such, the point cloud assemby should not be done on the map frame, otherwise, these distortions will cause major oscilations in the alignment performed by drl.
The aligned_pointcloud is the result of the alignment performed by drl and is on map frame, and as such, is the one you should be analyzing in rviz. You should not evaluate the ambient_pointcloud because that is the input for drl and not its result. And of course, since the ambient_pointcloud is on odom frame, it will show on rviz with the odometry drift. That is working as is supposed to.
These frames names are this way on purpose and follow the standard ros tf for mobile robotics: https://www.ros.org/reps/rep-0105.html https://navigation.ros.org/setup_guides/transformation/setup_transforms.html
Looking at your videos, the aligned_pointcloud is correctly overlapping the map. So drl was working correctly all along ! Keep in mind that every localization system has oscillations (which is different than drift) in its pose estimation due to lidar noise and also due to sliding of the alignment within the walls. Namely, if your map has walls with more than one pixel thickness in the occupancy grid, the alignment algorithms may oscillate. This is normal and if you want to minimize this oscillation, you need to have maps with 1 pixel thickness walls.
For inspecting the results of drl, set your rviz world frame to map and inspect the aligned_pointcloud overlap with the occupancy grid / map. For checking the drl pose, you have the topics localization_pose and localization_detailed Alternatively, you can check the pose between map and base_link or base_footprint
In drl_configs.yaml you can check the subscribe / input topics and the publish / output topics of drl
Have a nice day,

hello @carlosmccosta
I tried all the param you suggest,please watch this video,this is the robot ran 1 hours and stoped,the pose of localization_pose localization_detailed and the map->base_link is x=10.29 y=-15.39,then I send a initalpose at x=10.29 y=-15.39,and the pose of X Y axis has changed over 13cm after initpose to x = 10.42 y=-15.25 which is the correct pose to the real world,the map is 0.02 res,and I sure this is not because of oscillations .
https://drive.google.com/file/d/18HqVuU20vCfN-7rdiI9YQJneGD024vOo/view


Using off-site images

It would be cool if, when adding image, we would have a simple text box (below upload field for example) to paste URL of an image which is already online. I think that would play nicely with #321 (Changing an Inserted Image) and we could easily add off-site images, instead of downloading them and uploading them again.

@avramovic - agreed although I don't know if I'll be able to complete that at the same time as #321 so will keep this open as a separate enhancement.


cli/verifier: add Envoy config dump parser

Description:
Adds config parser utility. This is borrowed
from osm-health with some tweaks.
Part of #4634
Co-authored-by: Delyan Raychev<EMAIL_ADDRESS>Co-authored-by: Jon Huhn<EMAIL_ADDRESS>
Affected area:

Functional Area

New Functionality
[X]

CLI Tool
[X]

Please answer the following questions with yes/no.

Does this change contain code from or inspired by another project? Yes

Did you notify the maintainers and provide attribution? Yes (co-authors)

Is this a breaking change? no

Has documentation corresponding to this change been updated in the osm-docs repo (if applicable)? n/a

Codecov Report

Merging #4646 (a4e9302) into main (30e5362) will decrease coverage by 0.07%.
The diff coverage is 61.47%.

@@            Coverage Diff             @@
##             main    #4646      +/-   ##
==========================================
- Coverage   68.97%   68.89%   -0.08%
==========================================
  Files         213      219       +6
  Lines       14861    15077     +216
==========================================
+ Hits        10250    10387     +137
- Misses       4558     4635      +77
- Partials       53       55       +2

Flag
Coverage Δ

unittests
68.89% <61.47%> (-0.08%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

cmd/cli/version.go
42.47% <0.00%> (ø)

cmd/osm-controller/osm-controller.go
15.31% <0.00%> (ø)

pkg/envoy/ads/grpc.go
0.00% <0.00%> (ø)

pkg/envoy/ads/stream.go
8.12% <0.00%> (ø)

pkg/smi/client.go
73.45% <ø> (+0.26%)
:arrow_up:

pkg/trafficpolicy/trafficpolicy.go
97.46% <ø> (ø)

cmd/cli/verify_connectivity.go
42.10% <42.10%> (ø)

pkg/cli/verifier/envoy_config_parser.go
50.98% <50.98%> (ø)

pkg/cli/verifier/connectivity_pod_to_pod.go
56.52% <56.52%> (ø)

pkg/cli/verifier/namespace.go
73.52% <73.52%> (ø)

... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 95a898f...a4e9302. Read the comment docs.


TypeError: '>' not supported between instances of 'NoneType' and 'int'

Hi @phoenixding,
I ran into this issue (please see the attached screenshot) when running scdiff2.
After googling what may cause this type of error, it seems to be related when the script is migrated from python 2 to python 3.
Could it be the case or if I missed something?

Best,
Che-Wei

@Hsu-Che-Wei
Thanks for reporting this issue.
It seems that the "self.ncores" (# of CPU cores used) in your running is specified as None.
There might be an extra space after the parameter --ncores.
Can you share the running log so that I better understand the issue that you enountered? Thanks!

scdiff2_log.txt
Here is the running log.

@Hsu-Che-Wei
Thanks! I checked the detailed log.
Yes, you are right.  The upgraded python no longer allows comparison between None and integer, which causes the issue that you observed.  I have fixed the issue, the code should be working now.
Please upgrade the tool to the latest version.


add the introduction of quality gates, especially for the new jenkins feature

Briefly introduce the quality gates: hound, travis and jenkins.
Introduce new jenkins command in details, explaine how to test interdependent pull requests.
This pr should be merged after new jenkins feature go online(undetermined).

Good write up. :+1:  after comments fix.

👍

@amymullins : Are you OK with the latest doc?

For me, this doc is :+1: now.

@RackHD/corecommitters , this PR has been over 40 days .
any chance to merge it in , if no major issue ?

@panpan0000 though the new feature can be used now, it hasn't been online, so in my opinion this PR shouldn't be merged until it online.

https://github.com/RackHD/on-build-config/pull/16 has been merged. also merge the docs


loss.mean() or loss.sum()

when i use SGD to optimize the model, should i change loss.sum() to loss.mean()?

this is unrelated to ASL loss

when i use SGD to optimize the model, should i change loss.sum() to loss.mean()?

From my experience that using mean for SGD is far better and the loss is far stable than when using sum. But when using Adam then both sum and mean behave similarly. So depend on the optimizer, you should choose the suitable method

Hi, I also have a similar question. Is it possible that using loss.sum() will result in NaN? I train my network on Open Image Dataset, which has more than 10k labels, and I got NaN loss. So I changed it to loss.mean(), and the loss is not NaN, but quite small.

look for bugs of yours. look again. and again.
NaN usually means you have a bug in your code.
you can use mean(), no problem with it, but it won't make the detector learn better during training if you have a bug.
i have been training on open images with sum() all the time

Hi, I also have a similar question. Is it possible that using loss.sum() will result in NaN? I train my network on Open Image Dataset, which has more than 10k labels, and I got NaN loss. So I changed it to loss.mean(), and the loss is not NaN, but quite small.

This could be many things, usually bugs in computing gradient. You can use torch.autograd.set_detect_anomaly(True) to detect where the gradient goes wrong

a NaN can occur after awhile of training under certain conditions, such as when a classification does not have enough examples in the training data and is using loss.mean() (it may also require mixed precision).
It occurs when you calculate probabilities from the logits and the sigmoid goes to -inf.
You can "fix" it by clamping the gradients/logits/whatever themselves with a minimum value, but that would end up with that particular label never being able to be learned.
The best solution is to fix your dataset by adding more instances of the problem classes if possible, and using loss.sum() instead of mean.

a NaN can occur after awhile of training under certain conditions, such as when a classification does not have enough examples in the training data and is using loss.mean() (it may also require mixed precision).
It occurs when you calculate probabilities from the logits and the sigmoid goes to -inf.
You can "fix" it by clamping the gradients/logits/whatever themselves with a minimum value, but that would end up with that particular label never being able to be learned.
The best solution is to fix your dataset by adding more instances of the problem classes if possible, and using loss.sum() instead of mean.

Could you write an example whose loss goes to -inf? Thanks.


Async filter vout array

Removes the unnecessary garbage from the vout array before performing the original logic on it

Travis osx job 920.1 finished, you can download the results here.

Travis linux job 920.2 finished, you can download the results here.


Valve position setting for TS0601

Hi, first of all thank you for your great work!
I want to combine my valve with SmartThermostat. The integration allows to choose a heater that gets values from 0-100. I thought that would be great to use number.XXX_number_valve_position entity for that purpose but unfortunately changing valve position parameter does nothing. How can I make use of valve position entity? Should I put a valve in some mode that takes percentage value?
Valve model: _TZE200_rtrmfadk
Quirk used: ts0601_trv_rtitek.py
Current behavior:

changing valve position parameter between 0-100 doesn't trigger any action on valve

Expected behavior:

setting valve position entity to 50% should place the valve pin in 50% of it's range

If the TRV doesn't support that, there's nothing we could do about it.
Short answer: you can't.

If the TRV doesn't support that, there's nothing we could do about it. Short answer: you can't.

Yeah, that's obvious. I just can't tell if it's valve not supporting it or quirk issue.

Valve.
Commands are sent for sure.
Only Saswell or Siterwell supported that.

@jacekk015 I have thermostatic valve heads _TZE200_b6wax7g0 and _TZE200_cwnjrr72. The valve position is the result of the thermostat working, not something I want to set. Is it possible to somehow change the number on the sensor? Or maybe valve?

Is it possible to somehow change the number to the sensor? Or maybe valve?

I don't see any point of doing that.
Number is easily accessible in HA.
What you wanna to achieve??

sensor... if I want to display this value, I have to use custom settings,
otherwise sliders will be shown instead of values. In addition, these
sliders are not functional.
valve... this is what it was created for :D

wt., 6 lut 2024 o 22:33 Jacek Kończewski @.***>
napisał(a):

Is it possible to somehow change the number to the sensor? Or maybe valve
https://www.home-assistant.io/integrations/valve/?
I don't see any point of doing that.
Number is easily accessible in HA.
What you wanna to achieve??
—
Reply to this email directly, view it on GitHub
https://github.com/jacekk015/zha_quirks/issues/41#issuecomment-1930787492,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABDKAG2WKYSREACHRCG3PJ3YSKOUPAVCNFSM6AAAAABAMY5RJSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZQG44DONBZGI
.
You are receiving this because you commented.Message ID:
@.***>

p.s. could you show me how to change (in .py) so that something instead of in Controls, appears in Configuration?

I have to use custom settings, otherwise sliders will be shown instead of values
Use Helpers


Running >>git status produces spurious indents in WezTerm

Describe the bug
Running >>git status produces spurious indents in WezTerm. The problem does not occur in iTerm2.
To Reproduce

Install WezTerm

Use following zinit setup in .zshrc:

### Added by Zinit's installer
if [[ ! -f $HOME/.zinit/bin/zinit.zsh ]]; then
    print -P "%F{33}▓▒░ %F{220}Installing %F{33}DHARMA%F{220} Initiative Plugin Manager (%F{33}zdharma/zinit%F{220})…%f"
    command mkdir -p "$HOME/.zinit" && command chmod g-rwX "$HOME/.zinit"
    command git clone https://github.com/zdharma/zinit "$HOME/.zinit/bin" && \
        print -P "%F{33}▓▒░ %F{34}Installation successful.%f%b" || \
        print -P "%F{160}▓▒░ The clone has failed.%f%b"
fi

source "$HOME/.zinit/bin/zinit.zsh"
autoload -Uz _zinit
(( ${+_comps} )) && _comps[zinit]=_zinit

# Load a few important annexes, without Turbo
# (this is currently required for annexes)
zinit light-mode for \
    zinit-zsh/z-a-rust \
    zinit-zsh/z-a-as-monitor \
    zinit-zsh/z-a-patch-dl \
    zinit-zsh/z-a-bin-gem-node

### End of Zinit's installer chunk

# Two regular plugins loaded without investigating.
zinit light zsh-users/zsh-autosuggestions
zinit light zdharma/fast-syntax-highlighting

# Plugin history-search-multi-word loaded with investigating.
zinit load zdharma/history-search-multi-word

# Load the pure theme, with zsh-async library that's bundled with it.
zinit ice pick"async.zsh" src"pure.zsh"
zinit light sindresorhus/pure

Expected behavior
New lines should not have spurious indents
Screenshots

Versions:

$ZSH_VERSION == 5.8
Operating system name and version (uname -a): Darwin reportamans-Mac-mini.attlocal.net 20.4.0 Darwin Kernel Version 20.4.0: Fri Mar  5 01:14:02 PST 2021; root:xnu-7195.101.1~3/RELEASE_ARM64_T8101 arm64

Nevermind, this does not always happen. Not sure why it happened then...


[FLINK-2920] [tests] Apply JMH on KryoVersusAvroMinibenchmark class.

JMH is a Java harness for building, running, and analysing nano/micro/milli/macro benchmarks.Use JMH to replace the old micro benchmarks method in order to get much more accurate results.Modify the KryoVersusAvroMinibenchmark class and move it to flink-benchmark module.

Looks good to merge. Thanks!

I will merge this PR.


Globed crashes when trying to login

2024-06-30_01-20-14.log

this should be fixed in the latest release (v1.4.4)


Add discord bot

Adds an npm workspace so we share only one node_modules and adds a discord bot ^-^

A GHA workflow for atleast test-compiling the bot would be nice too :)

No, docker images are not a requirement, PR it later. None of the rest of the project has it qwq

None of the rest of the project has it qwq

An obvious mistake.


Unable to hide TPUs

Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
No
Source
binary
TensorFlow version
2.12
Custom code
No
OS platform and distribution
Kaggle Notebooks
Mobile device
No response
Python version
3.8
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current behavior?
Unable to hide TPUs from TensorFlow. The consequence of this is that if we want to use JAX along with TensorFlow, only one of them will be able to initialize the TPU system, and the other will fail. We won't be able to use tfds, tf.image or any TF operation per se if we can't hide TPUs from being used by TF. I want all these operations to run on CPU only, and leverage JAX for TPU. Here is the code to test it on a TPU machine:
import tenorflow as tf

tf.config.set_visible_devices([], device_type="TPU_SYSTEM")
tf.config.set_visible_devices([], device_type="TPU")

print(tf.config.list_logical_devices())

# output:
# [LogicalDevice(name='/device:CPU:0', device_type='CPU'),
#  LogicalDevice(name='/device:TPU_SYSTEM:0', device_type='TPU_SYSTEM'),
#  LogicalDevice(name='/device:TPU:0', device_type='TPU'),
#  LogicalDevice(name='/device:TPU:1', device_type='TPU'),
#  LogicalDevice(name='/device:TPU:2', device_type='TPU'),
#  LogicalDevice(name='/device:TPU:3', device_type='TPU'),
#  LogicalDevice(name='/device:TPU:4', device_type='TPU'),
#  LogicalDevice(name='/device:TPU:5', device_type='TPU'),
#  LogicalDevice(name='/device:TPU:6', device_type='TPU'),
# LogicalDevice(name='/device:TPU:7', device_type='TPU')]

This also doesn't work:
physical_devices = tf.config.list_physical_devices()
tf.config.set_visible_devices(physical_devices[0], 'CPU')

Standalone code to reproduce the issue
import tenorflow as tf

tf.config.set_visible_devices([], device_type="TPU_SYSTEM")
tf.config.set_visible_devices([], device_type="TPU")

print(tf.config.list_logical_devices())

# This also doesn't work:
physical_devices = tf.config.list_physical_devices()
tf.config.set_visible_devices(physical_devices[0], 'CPU')

### Relevant log output

_No response_

@AakashKumarNain Please follow this guide to know more about the usability of TPU using TF. I tried to replicate the issue as mentioned above, and didn't face the error reported. I was able to run the code as expected. Could you please have a look at this gist and confirm the same. In case you are using multiple TPUs you can also detect the TPUs as below;
import tensorflow as tf
print("Tensorflow version " + tf.__version__)

try:
  tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection
  print('Running on TPU ', tpu.cluster_spec().as_dict()['worker'])
except ValueError:
  raise BaseException('ERROR: Not connected to a TPU runtime; please see the previous cell in this notebook for instructions!')

tf.config.experimental_connect_to_cluster(tpu)
tf.tpu.experimental.initialize_tpu_system(tpu)
tpu_strategy = tf.distribute.TPUStrategy(tpu)

Thank you!

@sushreebarsa I am not asking How to use TPUs with TF?. I have different problem on hand. I have a TPU cluster but I don't want TF to use it. I don't want TF to use any of the TPUs for processing. I want to hide the TPUs and move everything to CPU only. Please go through the issue again for the details

@sachinprasadhs any suggestions?

tf.config.set_visible_devices device_type works for CPU or GPU, other devices will be left unaltered.
For TPUs, you need to configure the logical devices as something like below.
logical_devices = tf.config.list_logical_devices('TPU')
for logical_device in logical_devices:
    tf.config.set_logical_device_configuration(
        logical_device,
        [tf.config.LogicalDeviceConfiguration(), ]
    )

Thanks for the answer but this doesn't work. Error below
ValueError: Unrecognized device: LogicalDevice(name='/device:TPU:0', device_type='TPU')

Could you please attach the outcome of tf.config.list_logical_devices()

I have already done that in the code snippet at the top

Any updates on this?

Team is currently looking into this.
Meantime, can you please check by calling context._reset_context() to remove the TPUs from the logical device list.


How to implement custom directives in SPQR V0.9.8?

How to implement custom directives in SPQR V0.9.8? Please provide some good examples.

Depending on the kind of directives you have in mind. Custom query directives are not yet supported. The others you might add via GraphQLSchemaGenerator#withSchemaProcessors and then register a directive.
But what use-case are you trying to implement?
I'm asking because directives might easily not be what you want, as in SPQR you can have access to much richer options.

I said it wrong (now corrected). Custom query directives are doable. It's the directives on schema elements (types etc) that you can't easily do yet.
That said, you can always add a custom TyoeMapper and add directives there.

We actually are trying to filter/omit/modify the results of a data fetcher of a particular field using custom directives, given like in graphql-java project tutorial. But since we are using SPQR V0.9.8, Seems it is not straight forward.
And am also a beginner in SPQR. So please provide a good example to achieve this feature using custom directives or using any other feature of SPQR for me better understanding. Thanks.

Sorry, by mistake I closed

Can you point me to the tutorial you're referring to so I see if a similar thing can be done in SPQR?

Following is the url which I have referred.
3 examples
Authorization
Date Formatting
Chain of directives
We may end up using any/combination of the above like scenarios
https://graphql-java.readthedocs.io/en/latest/sdldirectives.html?highlight=directive

A gentle reminder, any update on my request?

Posted in #171:

This is how we do in graphql-java project. How can it be achieved via SPQR?

static GraphQLSchema buildSchema() {

    String sdlSpec = "" +
            "type Query {\n" +
            "    dateField : String @dateFormat \n" +
            "}";

    TypeDefinitionRegistry registry = new SchemaParser().parse(sdlSpec);

    RuntimeWiring runtimeWiring = RuntimeWiring.newRuntimeWiring()
            .directive("dateFormat", new DateFormatting())
            .build();

    return new SchemaGenerator().makeExecutableSchema(registry, runtimeWiring);
}

You can register an additional directive as such:
generator.withSchemaProcessors(
    (schemaBuilder, buildContext) -> schemaBuilder.additionalDirective(...));

But (according to my current understanding), this only makes sense for query directive (something the client sends as a part of the query, like @skip or @deffered).
Directives like @dateFormat it simply make no sense in SPQR: they're there to help you when paring SDL and mapping it to your code. In SPQR, there's no SDL and you start from your code.
E.g. @dateFormat is used to tell you that you need to provide date formatting to a specific field when mapping it to Java. In SPQR the field is generated from a method, so the method must already know what format it should return.
Still, sometimes field or type level directives can be useful in instrumentations. E.g. if you want to intercept field resolution and inspect the authentication directives.
In that case, I'd suggest you simply use annotations.
public class BookService {

      @Auth(roles= {"Admin"}) //example annotation
      public addBook(Book book) { /*insert a Book into the DB */ }
}

As each GraphQLFieldDefinition is backed by a Java methods (or a field), you can get the underlying objects in you interceptor or wherever:
GraphQLFieldDefinition field = ...;
Operation operation = Directives.getMappedOperation(field).get();

//Multiple methods can be hooked up to a single GraphQL operation. This gets the @Auth annotations from all of them
Set<Auth> allAuthAnnotations = operation.getResolvers().stream()
                .map(res -> res.getExecutable().getDelegate()) //get the underlying method
                .filter(method -> method.isAnnotationPresent(Auth.class))
                .map(method -> method.getAnnotation(Auth.class))
                .collect(Collectors.toSet());

Or, to inspect only the method that can handle the current request:
Auth auth = .getApplicableResolver(argumentNames).getExecutable().getDelegate().getAnnotation(Auth.class);

Then you can inspect you annotations as you wish,
Of course, there's no real need to actually implement authentication this way, as you're probably using a framework like Spring or Guice or JAX-RS, that already has a way to intercept all methods and implement security. So you can just use that instead. Much simpler and safer. E.g. for Spring Security, just keep using it as normal:

0.9.9 is out, with full directive support. Meta-annotate any annotation with io.leangen.graphql.annotations.types.GraphQLDirective and it well get mapped as directive on the target element.
To register a client directive, use GraphQLSchemaGenerator#withAdditionalDirectives(AnnotatedType...).
To inject a directive value, annotated a parameter with io.leangen.graphql.annotations.GraphQLDirective (notice it's not the same directive as above).


[requires.io] dependency update on master branch

Codecov Report

Merging #193 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff          @@
##           master   #193   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files           9      9
  Lines         267    267
=====================================
  Hits          267    267

Coverage remained the same at 100.0% when pulling 4c4bdfddc8fdbe160d449094cc9a4d8806adcb7c on requires-io-master into bd32f7dd327e8475e7bdc967e007ed8623d0fed2 on master.


Update Realtime-ingestion.md

Added note about sharding and firehose configurations as discussed here.

@estliberitas Thanks for the contrib! Do you mind signing our CLA?
http://druid.io/community/cla.html

@fjy done ;)


Update Example Project

Updating the example project dependencies from:
"react": "17.0.2",
"react-native": "0.66.4",

to
"react":  "18.2.0",
"react-native": "0.73.6"

👍🏼


fix AppleBackend

common parts are not duplicated
All Submissions:

[x] Have you checked to ensure there aren't other open Pull Requests for the same update/change?
[x] Have you added an explanation of what problem you are trying to solve with this PR?
[ ] Have you added information on what your changes do and why you chose this as your solution?
[ ] Have you written new tests for your changes?
[ ] Does your submission pass tests?
[ ] This project follows PEP8 style guide. Have you run your code against the 'flake8' linter?

@c00kiemon5ter Hi, I have a problem and I am not sure how to solve it so that it fits SATOSA logic.
You wrote that a backend should not transform attributes. The problem here (apart from many others) is that "Sign in with Apple" is similar to OIDC, but not fully compliant. Instead of a name claim in id_token or userinfo they supply a user dictionary like<EMAIL_ADDRESS>In the current state, this Apple backend puts the name sub-dictionary into data.attributes, so I have to have this in internal_attributes.yml (because this backend uses the openid mappings, not its own):
# ...
name:
    openid: [name]
givenname:
    openid: [given_name, name.firstName]
surname:
    openid: [family_name, name.lastName]
# ...

This works for Apple backend but fails for standard OIDC providers (e.g. Google), there is an uncaught exception in attribute mapping when it is trying to use the standard name claim (string) as a dictionary (because of name.firstName in config):
[2022-11-29 11:11:21] [ERROR]: [urn:uuid:0eb72947-5f36-421b-a45e-bd515e6050e1] Uncaught exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/satosa/base.py", line 240, in run
    resp = self._run_bound_endpoint(context, spec)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/satosa/base.py", line 180, in _run_bound_endpoint
    return spec(context)
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/satosa/backends/openid_connect.py", line 221, in response_endpoint
    internal_resp = self._translate_response(all_user_claims, self.client.authorization_endpoint)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/satosa/backends/openid_connect.py", line 240, in _translate_response
    internal_resp.attributes = self.converter.to_internal("openid", response)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/satosa/attribute_mapping.py", line 100, in to_internal
    attribute_values = self._collate_attribute_values_by_priority_order(external_attribute_name,
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/satosa/attribute_mapping.py", line 119, in _collate_attribute_values_by_priority_order
    attr_val = self._get_nested_attribute_value(attr_name, data)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/satosa/attribute_mapping.py", line 160, in _get_nested_attribute_value
    d = d.get(key)
        ^^^^^
AttributeError: 'str' object has no attribute 'get'

I can imagine at least two possible solutions (both of them may be implemented):

Change Apple backend so that it does not use openid attribute mappings but its own apple mappings. Most of them will be duplicated from openid, but the name claims will be different and there will be no problem. This just means changing self.converter.to_internal("openid", response) in Apple backend to self.converter.to_internal("apple", response).
Change attribute mapping so that it does not fail when trying nested attribute mapping on a non-nested attribute (string).

What do you suggest?

@c00kiemon5ter any thoughts on this?

@c00kiemon5ter Hi, can you please merge this?

@c00kiemon5ter Hi, is there anything blocking this PR? I am using it in production, it works.

re https://github.com/IdentityPython/SATOSA/pull/427#issuecomment-1330474635
How did you end up with d (data) being a string? It should have been a dict.

OK, I have figured it out; thanks for the comment above.
I have rebased on top of current master and updated the code with 2e3a7823, adding new test cases and fixing the traversal of the internal data.


tests: new test-integration scripts

Description
This PR is a re-write of the integration tests command-line interface to only use a few bash scripts to generate the environment and run the tests.
The scripts are written with readability in mind, to facilitate other developers to make changes as the environment requirements evolve, or new features and fixes are requested.
Changes

Added a new suite of shell scripts that emulate what a human admin would do to run the tests.
Added a simple command-line interface that exposes the original options to run the tests.
Added a formatted man page with descriptions and examples of how to use the tests.
Updated the relevant section in the repository README.md file.
Removed the zendro-env dependency.
Bumped package.json to version 0.5.5.

Usage
# Use the already existing command to run the integration tests via npm
npm run test-integration [-- OPTIONS ]

# For more information about the available options, and some examples
npm run test-integration -- -h

Spec
A (somewhat outdated) document outlining the design decisions made by @asishallab can be found in the gitbook workspace assigned to this PR.

I'm thinking that the most glaring DRY violations, like "heading" debug output logs or the code to source testenv_constants.sh could be moved to helpers provided by testenv_constants.sh itself?

The models folders are now defined in the constants file.
DRYed up the code a little bit. Nothing fancy, but there was some duplications in logic and output messages that I could avoid.

If you want to do a little bit of more clean up. I think files from the patches folder: individual_validate_joi.patch and individual_validate_sequelize_patch are outdated.

If you want to do a little bit of more clean up. I think files from the patches folder: individual_validate_joi.patch and individual_validate_sequelize_patch are outdated.

Thank you for the suggestion, @vsuaste. I have removed the unused patches.


ID token with workload identity federation

Hello togehter,
I want to access an GCP Cloud Endpoint with workload identity federation from AWS. I have a generated impersonated key file and your first example with list buckets worked well.
For Cloud endpoints and gRPC there is the need to use GOOGLE_ID_TOKEN instead of ACCESS_TOKENS.
My code looks like this. I used the getIdTokenClient function.
`
async function main() {
const url = "https://my-cloud-endpoint-url-endpoint-pimf6a67fa-ew.a.run.app/v1/status"

const auth = new GoogleAuth({
    keyFile: process.env.GOOGLE_APPLICATION_CREDENTIALS,
    scopes: 'https://www.googleapis.com/auth/cloud-platform',
    projectId: 'my-project',
});
const targetAudience = "this-is-my-target-audience"
const client = await auth.getIdTokenClient(targetAudience);

const res = await client.request({ url });
console.log(res.data); }

`
The error message is:
Error: Cannot fetch ID token in this environment, use GCE or set the GOOGLE_APPLICATION_CREDENTIALS environment variable to a service account credentials JSON file.     at GoogleAuth.getIdTokenClient
The problem is that I have an access token. In my case I need an ID Token. I was wondering if there is a possibility to get an ID Token before I request my endpoint.
( It works in python: Python google.auth has an impersonated_credentials object with an ID Token functionality.
    creds = impersonated_credentials.IDTokenCredentials(         target_creds,         target_audience=audience,         include_email=True     )
Anyway, I need a solution for nodeJs.
Thanks

@jalbinge the person who implemented this feature is currently away for a few days, I'll loop them in regarding your question as soon as they're back.

Is there any workaround for this? I'm also facing same error.

I'm also having issues running into this error

@bojeil-google any thoughts on this one, or is there someone we can point folks towards?

@xil222 could you help out with this one? Is this a known limitation of the current workload identity implementation?

Hey there,
be aware that I am not very familiar with nodeJs, but here is an example code which worked for me:
// Make a request to a protected Cloud Run service.
const {GoogleAuth} = require('google-auth-library');
const https = require('https')
const axios = require('axios');
async function getGcpIdToken(access_token) {
    const data = JSON.stringify(
        {
            "audience": "https://your-audience-endpoint-pimf6a67fa-ew.a.run.app",
            "includeEmail": "true"
        }
    )
   return await axios({
            method: 'post',
            url: 'https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/account-id.iam.gserviceaccount.com:generateIdToken',
            data: data,
            headers: {
                'Content-Type': 'application/json',
                'Content-Length': data.length,
                'Authorization': `Bearer ${access_token}`
            }
        }) .then( (result) => {
        return result.data.token;
    });
}
async function getStatus(id_token){
    axios({
            method: 'get',
            url: 'https://your-endpoint-url',
            headers: {
                'Content-Type': 'application/json',
                'Authorization': `Bearer ${id_token}`
            }
        }
    ) .then( (result) => {
        console.log(result)
        return result;
    });
}
async function main() {
    const url = "https://your-endpoint-url"
    const auth = new GoogleAuth(
        { scopes: 'https://www.googleapis.com/auth/cloud-platform' }
    )
    const client = await auth.getClient();
    const access_token = (await client.getAccessToken()).token
    getGcpIdToken(access_token).then((id_token)=>getStatus(id_token))
}
main().catch(console.error);

Hey @jalbinge this looks like a duplicate feature request of https://github.com/googleapis/google-auth-library-nodejs/issues/1305
I didn't dive into the details of the provided snippet but it looks like it should work.

@bojeil-google let me know if you can pitch in here, worst case can have someone on our team start ramping up on WIF.

@bojeil-google let me know if you can pitch in here, worst case can have someone on our team start ramping up on WIF.

Hey @bcoe, I mentioned above that this looks like a duplicate of the other issue. @xil222 is no longer on the team. I can hold on this or the duplicate issue until we can find the right owner for it.

@bojeil-google there seems to be some variation in the environment and nuance of two issues. Shall we keep both open for now, even if the same documentation or implementation work addresses both?

@bojeil-google there seems to be some variation in the environment and nuance of two issues. Shall we keep both open for now, even if the same documentation or implementation work addresses both?

Ok, sounds good @bcoe. Let's keep both open.

Any further development to the issue. Or do we still need to do the workaround and manually get the token for an impersonated service account


How to add plugins?

How to add plugins?

Check the closed issues. It has been discussed in multiple issues.


Duplicate objects are allowed

Duplicate objects with same keys(names) are accepted and added at the end instead of replacing existing ones.
RFC 7159: says

The names within an object SHOULD be unique.

IMO this is a useful feature to support. While properly formatted JSON "should" have unique keys, there is nothing wrong with accepting JSON with duplicate keys and letting the client code decide which key(s) to use.

Because it's a "SHOULD" in the RFC, I don't think there's any harm in allowing this.  I would accept a PR for an alternative to json_object_push that replaces existing keys, providing the original functionality is unchanged.

Yes. I agree. To support both, I've added another API to add just Unique keys. Will test and initiate a PR.