finish trajectory service crashes

finish trajectory service called with
rosservice call /finish_trajectory "stem: "full_traj'" gives
F0215 12:31:23.674871 21269 hybrid_grid.h:384] Check failed: new_bits <= 8 (9 vs. 8)
[FATAL] [1487183483.675242597]: F0215 12:31:23.000000 21269 hybrid_grid.h:384] Check failed: new_bits <= 8 (9 vs. 8)
This works for smaller trajectories with 2-3 GB bag files. This was with a 9GB bag file with two VLP-16 and a xsens MTI-g 700 imu. The same bag file when split into two works for finishing trajectory and is able to write both the ply and the png files.
*** Check failure stack trace: ***
@     0x7f2bd3d655cd  google::LogMessage::Fail()
@     0x7f2bd3d67433  google::LogMessage::SendToLog()
@     0x7f2bd3d6515b  google::LogMessage::Flush()
@     0x7f2bd3d67e1e  google::LogMessageFatal::~LogMessageFatal()
@           0x5d2c70  cartographer::io::XRayPointsProcessor::Insert()
@           0x5d3294  cartographer::io::XRayPointsProcessor::Process()
@           0x5cf1b1  cartographer::io::PlyWritingPointsProcessor::Process()
@           0x51aee6  cartographer_ros::WriteAssets()
@           0x50e6f0  cartographer_ros::MapBuilderBridge::WriteAssets()
@           0x4ef35a  ZN5boost6detail8function21function_obj_invoker2IZN16cartographer_ros12_GLOBAL__N_13RunEvEUlRN21cartographer_ros_msgs24FinishTrajectoryRequest_ISaIvEEERNS5_25FinishTrajectoryResponse_IS7_EEE4_bS9_SC_E6invokeERNS1_15function_bufferES9_SC
@           0x4f87e0  ros::ServiceCallbackHelperT<>::call()
@     0x7f2bd0e25811  ros::ServiceCallback::call()
@     0x7f2bd0e78cf0  ros::CallbackQueue::callOneCB()
@     0x7f2bd0e7a0f3  ros::CallbackQueue::callAvailable()
@     0x7f2bd0ed2691  ros::SingleThreadedSpinner::spin()
@     0x7f2bd0eb772b  ros::spin()
@           0x4f0dde  cartographer_ros::(anonymous namespace)::Run()
@           0x4ed514  main
@     0x7f2bcf8d1830  __libc_start_main
@           0x4eef99  _start
@              (nil)  (unknown)
Launch file

----
urdf

Configuration file
include "map_builder.lua"
options = {
map_builder = MAP_BUILDER,
map_frame = "map",
tracking_frame = "base_link",
published_frame = "base_link",
odom_frame = "odom",
provide_odom_frame = true,
use_odometry = false,
use_laser_scan = false,
use_multi_echo_laser_scan = false,
num_point_clouds = 2,
lookup_transform_timeout_sec = 0.2,
submap_publish_period_sec = 0.3,
pose_publish_period_sec = 2e-2
}
TRAJECTORY_BUILDER_3D.scans_per_accumulation = 2
-- No point of trying to SLAM over the points on your car.
TRAJECTORY_BUILDER_3D.laser_min_range = 2
-- These were just my first guess: use more points for SLAMing and adapt a bit for the ranges that are bigger for cars.
--change both the high and low to 8 and 400 from 5 and 250
TRAJECTORY_BUILDER_3D.high_resolution_adaptive_voxel_filter.max_length = 8.
TRAJECTORY_BUILDER_3D.high_resolution_adaptive_voxel_filter.min_num_points = 400.
TRAJECTORY_BUILDER_3D.low_resolution_adaptive_voxel_filter.max_length = 8.
TRAJECTORY_BUILDER_3D.low_resolution_adaptive_voxel_filter.min_num_points = 400.
-- The submaps felt pretty big - since the car moves faster, we want them to be
-- slightly smaller. You are also slamming at 10cm - which might be aggressive
-- for cars and for the quality of the laser. I increased 'high_resolution',
-- you might need to increase 'low_resolution'. Increasing the
-- '*num_iterations' in the various optimization problems also trades
-- performance/quality.
--sri changed both high and low to 0.15 and rotation weight to 1
TRAJECTORY_BUILDER_3D.submaps.num_laser_fans = 80
TRAJECTORY_BUILDER_3D.submaps.high_resolution = .25
TRAJECTORY_BUILDER_3D.submaps.low_resolution = .25
--TRAJECTORY_BUILDER_3D.ceres_scan_matcher.rotation_weight = 1
MAP_BUILDER.use_trajectory_builder_3d = true
MAP_BUILDER.num_background_threads = 7
MAP_BUILDER.sparse_pose_graph.optimization_problem.huber_scale = 5e2
--MAP_BUILDER.sparse_pose_graph.optimization_problem.rotation_weight = 1
-- Trying loop closing too often will cost CPU and not buy you a lot. There is
-- little point in trying more than once per submap.
MAP_BUILDER.sparse_pose_graph.optimize_every_n_scans = 100
MAP_BUILDER.sparse_pose_graph.constraint_builder.sampling_ratio = 0.03
MAP_BUILDER.sparse_pose_graph.optimization_problem.ceres_solver_options.max_num_iterations = 100
-- Reuse the coarser 3D voxel filter to speed up the computation of loop closure
-- constraints.
MAP_BUILDER.sparse_pose_graph.constraint_builder.adaptive_voxel_filter = TRAJECTORY_BUILDER_3D.high_resolution_adaptive_voxel_filter
-- This is probably the most important change: lower the matching score for
-- adding constraints.
--changed the angular search to 90 deg from 30 deg
MAP_BUILDER.sparse_pose_graph.constraint_builder.min_score = 0.4
MAP_BUILDER.sparse_pose_graph.constraint_builder.fast_correlative_scan_matcher_3d.linear_xy_search_window = 20.
MAP_BUILDER.sparse_pose_graph.constraint_builder.fast_correlative_scan_matcher_3d.linear_z_search_window = 5.
MAP_BUILDER.sparse_pose_graph.constraint_builder.fast_correlative_scan_matcher_3d.angular_search_window = math.rad(90.)
return options

urdf file

This indicates that your HybridGrid became too big. What is the area you are trying to cover here?
Two possible solutions:

Change the hybrid grid template parameters to allow for larger maps in the source code. This will require more memory and cost performance.
SLAM at a higher resolution. If your high resolution is 25 cm, your low resolution should probably be a meter or so.

Tried the following:
high resolution at 0.25 and low resolution at 1m (using only a single laser and using both lasers)
high resolution at 0.5 and low resolution at 0.5m (using only a single laser and using both lasers
All of them fail at the finish trajectory rosservice call with the same error at the end. The bag files are ~1079 seconds long. I can run it successfully upto 700 seconds.
The trajectory that I am mapping is approximately 10 km long. All of the above work if the trajectory is ~5km long ie I can split into two separate runs of 5km each and they work
Could you let me know how I should go about changing the hybrid grid parameters?

The above are the two split trajectories. My bag files are split every second. So I can run the first 10 bag files and it works, as soon as I add the 11th bag file rosservice /finishtrajectory gives the above error

Also tried 0.25m and 2.0m f for high resolution and low resolution using only a single LIDAR. Still get the same error

Could you make your data and configuration available?

Closing for inactivity. Please speak up if this is still an issue for you - in which case I'd require a repro data set.

Hi I had same problem, hopefully resolution will fix it.  The dataset that I have that is reproducing this issue is http://kitti.is.tue.mpg.de/kitti/raw_data/2011_10_03_drive_0034/2011_10_03_drive_0034_sync.zip  with calibration file http://kitti.is.tue.mpg.de/kitti/raw_data/2011_10_03_calib.zip.   I used kitti2bag to turn it into bag files.

As @SirVer said this is due to the submaps getting too large, the size here being in voxels. You can improve this by increasing the voxel size (resolution), reducing the max_range or high_resolution_max_range, or reducing num_range_data. If you need more help, please provide your full configuration so that we can reproduce.

Thank You, this weekend I will organize what I have and upload it so you can inspect and reproduce what I'm doing.   Using Kitti Dataset, 2011_10_03 drive 34, it has 5 loop closures (outside road data) however with my current configuration it seems like there is some drift and it is not closing the loops.   In the case of the first loop, the road has drifted slightly above the plane of where it should be and slightly to the left.  When it approaches the second loop on what should be the same road, its maping a slightly offset road on a slightly different plane.  When all is done however, I am unable to create ply file or png files with asset writer with same above error.  I've tried numerous different adjustments.  At one point I watched asset writer climb to 48gb memory (my workstation had 16gb ram w/ 32gb of swap) and then it would die, i since increased the swap to 128gb, and now i'm back at the error above with map growing to large.   I will uploaded and provide full details this weekend.  Thank You!!
-Nathan Obert/ George Mason University

Hi, I have uploaded my configurations ( I have changed then around a bunch of different ways kinda trial and error). Any advice you can provide would be awesome.
I put the files up on github at:  https://github.com/nobertgmuedu/gmukitti
We are using Kitti Dataset, with Cartographer ROS.  I am using a modified kitti2bag to convert the kitti datafiles to bag files (provided on github).  It basically adds compression setting, ditches the images, and also ditches the TF frame, which allows me to configure robot with URDF instead.  I hosted a bag file on amazon S3 at https://s3.amazonaws.com/kittigmu/kitti_2011_10_03_drive_0027_synced.bag .
Obviously the launch files, configurations, etc are all on github as well, should be ready to run with minimal if any hassle.
Here is a quick overview of the route, https://github.com/nobertgmuedu/gmukitti/blob/master/kitti_2011_10_03_drive_0027_synced.png there should be 5 loop closures (i marked green, purple, blue, red, and yellow) and near the end it pretty much totally gets lost running in circles.  To be fair I think this is one of the most complex Kitti files in their entire dataset.   The black arrows are where the car "should" be traveling, instead of the big loop you see it doing at the end.
This is the same dataset that the asset writing it crashing on too.   With the smaller resolution it does better on some of the loop closure, like the blue dots "nearly" touch.
Lastly, here is a youtube video of the same dataset being navigated with other software for comparison sakes: https://www.youtube.com/watch?v=KYvOqUB_odg
Anyways, any feedback/suggestions on getting KITTI working better with Cartographer are more than welcome.
Thanks!
-Nathan

That's an interesting approach to life-long mapping in the linked video - do not add keyframes in visited areas, but do it when exploring new areas. Perhaps Cartographer's submap/node trimming could be expanded to behave like this, instead of always doing FIFO discarding? The planned functionality of being able to trim trajectories in the middle sounds like a way towards this. Sorry for going off topic here, perhaps an issue could be opened for this if there isn't one already.

@nobertgmuedu I took some time looking into this issue now. Specfically I looked at  https://s3.amazonaws.com/kittigmu/kitti_2011_10_03_drive_0027_synced.bag and I can repro your quality issues. Some observations:

The car in the bag starts out driving - this violates Cartographer's zero velocity initial state and therefore it takes a while until the SLAM starts tracking.
The original IMU in the car had 100 Hz frequency, the Kitti dataset only contains "synchronized" data to the velodyne (full revolutions) which reduces this to 10 Hz. That is not enough to capture corners and strong accelerations, so the SLAM drifts whenever the car de- or accelerates or turns corners. This dataset would work better with pure point cloud matching, but Cartographer does not do that - we always use IMU in 3d. Full revolutions are also not ideal, though Andreas invested some time to "unwarp" the data using imu interpolation. The closest to this is enabling the real_time_correlative_scan_matcher which makes Cartographer quite slow.
Running Cartographer, you see that the car is constantly dropping, after a while it is at z=-100m. This indicates that either the URDF is wrong and there is an angle between the IMU and the lidar or that the IMU has a bias.

I think LOAM would do much better on the data, Cartographer really requires a fast IMU. If that is given, it can do quite well on this kind of setup - for example #259. Sorry I cannot help much more here.

1. Change the hybrid grid template parameters to allow for larger maps in the source code. This will require more memory and cost performance.

Hi,
Could you help how to change the hybrid grid template parameters?


mysql_jdbc: 8.1.0 -> 8.2.0

Description of changes
https://dev.mysql.com/doc/relnotes/connector-j/en/news-8-2-0.html
https://github.com/mysql/mysql-connector-j/compare/8.1.0...8.2.0
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Does someone know why the attribute of this package is not named mysql-connector-java (instead of mysql_jdbc)?


Enhancing The footer UI and add Data links

Add GSOC ext and HacktoberFest Label

Can you assign me this under Hacktoberfest

I want to take this up this issue if you are not solving...


Cannot find "Hardware submission page"

Cannot find "Hardware submission page"
[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7135005c-e7aa-19a2-7cfe-5003696da66a
Version Independent ID: dba3259f-cb98-c2ab-9c56-860a731310d9
Content: Hardware submissions - Windows drivers
Content Source: windows-driver-docs-pr/dashboard/hardware-certification-submissions.md
Product: windows-hardware
Technology: windows-devices
GitHub Login: @tedhudek
Microsoft Alias: tedhudek

@tedhudek . did you logged Microsoft partner page using your organization credentials..

Sorry, I have found the problem. After I complete the hardware registration, I can already display the options.

Glad you got it fixed!

BTW you can go directly to the Hardware submission page by going to https://partner.microsoft.com/en-us/dashboard/hardware/Search.


When a worker node is blocked, the all alluxio cluster cannot provide services for presto queries.

Alluxio Version:
2.5.0-1
Describe the bug
When a worker node is blocked, the all alluxio cluster cannot provide services for presto queries.
when openMs long time.many half close.
To Reproduce
when query many big data.
Expected behavior
Urgency
High
Are you planning to fix it
Additional context
2022-03-27 11:20:30,760 WARN  BlockReadHandler - Getting buffer for remote read took longer than 10000 ms. reader: alluxio.worker.block.io.StoreBlockReader location: /home/presto/alluxiodata/data/ssd/alluxioworker/5194830053377 bytes: 1048576 openMs: 299901 transferMs: 5 durationMs: 299906
2022-03-27 11:25:38,421 WARN  BlockReadHandler - Getting buffer for remote read took longer than 10000 ms. reader: alluxio.worker.block.io.StoreBlockReader location: /home/presto/alluxiodata/data/ram/alluxioworker/786582994944 bytes: 1048576 openMs: 291017 transferMs: 1 durationMs: 291018
2022-03-27 11:30:38,435 WARN  BlockReadHandler - Getting buffer for remote read took longer than 10000 ms. reader: alluxio.worker.block.io.StoreBlockReader location: /home/presto/alluxiodata/data/ssd/alluxioworker/4690792153088 bytes: 1048576 openMs: 598047 transferMs: 2 durationMs: 598049
2022-03-27 11:52:40,588 WARN  ShortCircuitBlockReadHandler - Exception occurred processing read request block_id:<PHONE_NUMBER>680
promote: false
.: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2022-03-27 11:54:55,591 WARN  ShortCircuitBlockReadHandler - Exception occurred processing read request block_id:<PHONE_NUMBER>048
promote: false
.: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2022-03-27 11:54:55,591 WARN  ShortCircuitBlockReadHandler - Exception occurred processing read request block_id:<PHONE_NUMBER>672
promote: false
.: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2022-03-27 11:56:25,593 WARN  ShortCircuitBlockReadHandler - Exception occurred processing read request block_id:<PHONE_NUMBER>832
promote: false
.: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2022-03-27 12:17:33,277 WARN  ShortCircuitBlockReadHandler - Exception occurred processing read request block_id:<PHONE_NUMBER>1264
promote: false
.: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2022-03-27 12:17:33,280 WARN  BlockReadHandler - Getting buffer for remote read took longer than 10000 ms. reader: alluxio.worker.block.io.StoreBlockReader location: /home/presto/alluxiodata/data/ssd/alluxioworker/5206104342529 bytes: 1048576 openMs: 3112319 transferMs: 3 durationMs: 3112322
2022-03-27 12:17:33,280 WARN  ShortCircuitBlockReadHandler - Exception occurred processing read request block_id:<PHONE_NUMBER>3696
promote: false
.: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2022-03-27 12:17:33,282 WARN  BlockReadHandler - Exception occurred while sending data for read request BlockReadRequest{chunkSize=1048576, end=134217728, id=5206104342529, openUfsBlockOptions=ufs_path: "hdfs://masters/hive/warehouse/xxx.db/yyy_table/dt=2020-03-01/000000_0"
offset_in_file: 134217728
block_size: 134217728
maxUfsReadConcurrency:<PHONE_NUMBER>
mountId: 1
no_cache: false
, promote=false, sessionId=7243546879341406138, start=0, positionShort=false}.: io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
2022-03-27 12:17:33,282 WARN  UnderFileSystemBlockStore - Key (block ID:<PHONE_NUMBER>529, session ID<PHONE_NUMBER>341406138) is not found when cleaning up the UFS block.
2022-03-27 12:17:33,282 WARN  UnderFileSystemBlockStore - Key (block ID:<PHONE_NUMBER>529, session ID<PHONE_NUMBER>341406138) is not found when releasing the UFS block.
2022-03-27 12:18:10,633 WARN  ShortCircuitBlockReadHandler - Exception occurred processing read request block_id:<PHONE_NUMBER>6464
promote: false
.: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2022-03-27 12:18:10,633 WARN  ShortCircuitBlockReadHandler - Exception occurred processing read request block_id:<PHONE_NUMBER>7216
promote: false

@beinan can you take a look?

Hello @xinip , many thanks for reporting this bug to us!  Can I have more context about this bug?

are you using colocated deployment (deploy presto worker and alluxio worker on the same machine) and enabled short-circuit?
How did you know an alluxio node is blocked? "the all alluxio cluster cannot provide services for presto queries." do you mean all the presto query failed?  does "alluxio fs ls /" work for you?

Hello @beinan
1.alluxio.user.file.short.circuit.enabled=true
2.when alluxio  cannot provide services,all presto query blocked , I had not "alluxio fs ls /"

2022-04-01 20:25:08,832 WARN  BlockReadHandler - Getting buffer for remote read took longer than 10000 ms. reader: alluxio.worker.block.io.StoreBlockReader location: /home/presto/alluxiodata/data/ssd/alluxioworker/5189662670848 bytes: 1048576 openMs: 299948 transferMs: 3 durationMs: 299951
2022-04-01 20:27:17,592 WARN  BlockReadHandler - Getting buffer for remote read took longer than 10000 ms. reader: alluxio.worker.block.io.StoreBlockReader location: /home/presto/alluxiodata/data/ssd/alluxioworker/5203134775296 bytes: 1048576 openMs: 28749 transferMs: 3 durationMs: 28752
2022-04-01 20:33:55,586 WARN  ShortCircuitBlockReadHandler - Exception occurred processing read request block_id:<PHONE_NUMBER>33
promote: false
.: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2022-04-01 20:34:40,593 WARN  ShortCircuitBlockReadHandler - Exception occurred processing read request block_id:<PHONE_NUMBER>040
promote: false
.: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2022-04-01 20:35:25,593 WARN  ShortCircuitBlockReadHandler - Exception occurred processing read request block_id:<PHONE_NUMBER>45
promote: false
.: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2022-04-01 20:37:55,585 WARN  ShortCircuitBlockReadHandler - Exception occurred processing read request block_id:<PHONE_NUMBER>92
promote: false
.: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2022-04-01 21:21:07,229 WARN  BlockReadHandler - Getting buffer for remote read took longer than 10000 ms. reader: alluxio.worker.block.io.StoreBlockReader location: /home/presto/alluxiodata/data/ssd/alluxioworker/5190501531648 bytes: 1048576 openMs: 298977 transferMs: 3 durationMs: 298980
2022-04-01 21:26:10,728 WARN  BlockReadHandler - Getting buffer for remote read took longer than 10000 ms. reader: alluxio.worker.block.io.StoreBlockReader location: /home/presto/alluxiodata/data/ssd/alluxioworker/14793947742208 bytes: 1048576 openMs: 299997 transferMs: 3 durationMs: 300000
2022-04-01 21:33:55,584 WARN  ShortCircuitBlockReadHandler - Exception occurred processing read request block_id:<PHONE_NUMBER>408
promote: false
.: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2022-04-01 21:34:40,585 WARN  ShortCircuitBlockReadHandler - Exception occurred processing read request block_id:<PHONE_NUMBER>832
promote: false
.: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2022-04-01 21:35:25,586 WARN  ShortCircuitBlockReadHandler - Exception occurred processing read request block_id:<PHONE_NUMBER>833
promote: false
.: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2022-04-01 21:35:25,586 WARN  ShortCircuitBlockReadHandler - Exception occurred processing read request block_id:<PHONE_NUMBER>1632
promote: false
.: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close

com.facebook.presto.spi.PrestoException: Timeout waiting for response after 300000ms. clientClosed: false clientCancelled: false serverClosed: false (LocalFileDataReader{request=block_id:<PHONE_NUMBER>9888
promote: false
, address=WorkerNetAddress{host=datanode106, containerHost=, rpcPort=29999, dataPort=29999, webPort=30000, domainSocketPath=, tieredIdentity=TieredIdentity(node=datanode106, rack=null)}})
at com.facebook.presto.hive.GenericHiveRecordCursor.advanceNextPosition(GenericHiveRecordCursor.java:231)
at com.facebook.presto.hive.HiveRecordCursor.advanceNextPosition(HiveRecordCursor.java:175)
at com.facebook.presto.$gen.CursorProcessor_20220401_132648_445579.process(Unknown Source)
at com.facebook.presto.operator.ScanFilterAndProjectOperator.processColumnSource(ScanFilterAndProjectOperator.java:252)
at com.facebook.presto.operator.ScanFilterAndProjectOperator.getOutput(ScanFilterAndProjectOperator.java:244)
at com.facebook.presto.operator.Driver.processInternal(Driver.java:418)
at com.facebook.presto.operator.Driver.lambda$processFor$9(Driver.java:301)
at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:722)
at com.facebook.presto.operator.Driver.processFor(Driver.java:294)
at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1077)
at com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:162)
at com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:545)
at com.facebook.presto.$gen.Presto_0_248_1_$_git_commit_id_abbrev_____20220218_185330_1.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: alluxio.exception.status.DeadlineExceededException: Timeout waiting for response after 300000ms. clientClosed: false clientCancelled: false serverClosed: false (LocalFileDataReader{request=block_id:<PHONE_NUMBER>9888
promote: false
, address=WorkerNetAddress{host=datanode106, containerHost=, rpcPort=29999, dataPort=29999, webPort=30000, domainSocketPath=, tieredIdentity=TieredIdentity(node=datanode106, rack=null)}})
at alluxio.client.block.stream.GrpcBlockingStream.receive(GrpcBlockingStream.java:187)
at alluxio.client.block.stream.LocalFileDataReader$Factory.(LocalFileDataReader.java:149)
at alluxio.client.block.stream.BlockInStream.createLocalBlockInStream(BlockInStream.java:158)
at alluxio.client.block.stream.BlockInStream.create(BlockInStream.java:127)
at alluxio.client.block.AlluxioBlockStore.getInStream(AlluxioBlockStore.java:217)
at alluxio.client.file.AlluxioFileInStream.updateStream(AlluxioFileInStream.java:363)
at alluxio.client.file.AlluxioFileInStream.read(AlluxioFileInStream.java:188)
at alluxio.hadoop.HdfsFileInputStream.read(HdfsFileInputStream.java:126)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.fillBuffer(UncompressedSplitLineReader.java:62)
at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:238)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:193)
at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.readLine(UncompressedSplitLineReader.java:94)
at org.apache.hadoop.mapred.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:208)
at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:246)
at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:48)
at com.facebook.presto.hive.GenericHiveRecordCursor.advanceNextPosition(GenericHiveRecordCursor.java:213)
... 15 more

Thank you @xinip ! this is very helpful!
two possibilities

the alluxio worker's jvm is paused (probably full gc or some other system pause).  Have you enabled the GC log of alluxio workers? what's the jvm setting of your alluxio workers?
The disk is paused somehow, we have seen this on both solid or mechanical hard drives.  This PR might help https://github.com/Alluxio/alluxio/pull/15166. do you mind to give it a shot?  thanks!

Thank you @xinip ! this is very helpful!
two possibilities

the alluxio worker's jvm is paused (probably full gc or some other system pause).  Have you enabled the GC log of alluxio workers? what's the jvm setting of your alluxio workers?
The disk is paused somehow, we have seen this on both solid or mechanical hard drives.  This PR might help Add timeout logic to local block reading/writing #15166. do you mind to give it a shot?  thanks!

@xinip any update? have you checked the GC logs?

Hello @beinan
1.jvm gc is normal,memory is enough
2.maybe i should try to update to the next alluxio latest version, i can try

@xinip any update? have you checked the GC logs?

jvm gc is normal,memory is enough

Thank you @xinip for the updates! Just let us know if the latest version of alluxio works or not.  I think this commit would be included in 2.7.4, it's already on maven repo but I didn't got a chance to send out the release note.


No nullish coalescing assignment operator

using the following code:
let x;
x ??= 1;

results in a Parsing error: Unexpected token =
Nullish Coalescing Assignment operator was added in ECMAScript 2020 and can be found here
Currently using "eslint": "^8.36.0" with
"parserOptions": {
  "ecmaVersion": 2020
}

@joakim-ronning  does the validation happen correctly when you use eslint in the terminal?

@dbaeumer Yes:  150:7  error  Parsing error: Unexpected token =
ran npx eslint ./

Then this is a more general problem with ESLint. I would suggest to report the problem there since there is nothing I can do about it in the extension.

Thanks Dirk, I'll do that

@joakim-ronning A workaround is to use the extention @typescript-eslint/eslint-plugin.
.eslintrc:
{
    extends: [
        "plugin:@typescript-eslint/recommended",
      ],
}

This is working with eslint 8.X.X and @typescript-eslint/eslint-plugin  5.X.X.


PI-week: Refactor Transpile

[ ] Tests added for any new code
[x] Ran make fmt-fix (or had formatting run automatically on all files edited)
[ ] Documentation added for any new functionality
[ ] Entries added to ./unreleased/ for any new functionality

This PR attempts to change the way the internal transpilation is performed, and, to a lesser extent, the SMT term representation.
The main issue is, that SMT separates two concepts: terms, and declarations/definitions. In an effort to make the transpilation "side-effect free", FunDef was introduced as a Term, which represented a function, to be defined in the SMT header, but used (by name) wherever FunDef appeared in the syntax tree. Later, a Collector would traverse the tree again, and extract the function definitions, along with any uninterpreted literals, which also needed to be declared.
This is clunky, and does not scale with extensions to Terms. Instead, the new approach introduces rules as State computations over a class which holds SMT declarations. This way, rules conceptually still translate TLA to SMT terms directly, but the internal declarations class keeps track of any declarations or definitions, which need to be introduced along the way.
The PR also includes minor QoL changes, such as replacing case classes without parameters with case objects, and standardizing the use of Seq (instead of a mix of Seq and List).

Codecov Report

Merging #2720 (462b034) into main (72e0965) will increase coverage by 0.32%.
The diff coverage is 53.96%.

@@            Coverage Diff             @@
##             main    #2720      +/-   ##
==========================================
+ Coverage   78.55%   78.87%   +0.32%
==========================================
  Files         463      463
  Lines       15904    15870      -34
  Branches     2563     2572       +9
==========================================
+ Hits        12493    12518      +25
+ Misses       3411     3352      -59

Files Changed
Coverage Δ

...apalache/tla/bmcmt/rules/vmt/TermToVMTWriter.scala
0.00% <0.00%> (ø)

...palache/tla/bmcmt/rules/vmt/TlaExToVMTWriter.scala
0.00% <0.00%> (ø)

...lache/tla/bmcmt/rules/vmt/ToTermRewriterImpl.scala
90.00% <ø> (ø)

...forsyte/apalache/tla/bmcmt/rules/vmt/VMTExpr.scala
0.00% <0.00%> (ø)

...la/at/forsyte/apalache/tla/lir/formulas/Base.scala
0.00% <ø> (ø)

...t/forsyte/apalache/tla/lir/formulas/Integers.scala
0.00% <0.00%> (-100.00%)
:arrow_down:

...forsyte/apalache/tla/bmcmt/rules/vmt/EUFRule.scala
53.12% <47.05%> (+19.79%)
:arrow_up:

...orsyte/apalache/tla/bmcmt/rules/vmt/BoolRule.scala
69.23% <63.63%> (-8.55%)
:arrow_down:

...rsyte/apalache/tla/bmcmt/rules/vmt/ValueRule.scala
77.27% <75.00%> (+1.08%)
:arrow_up:

...ala/at/forsyte/apalache/tla/lir/formulas/EUF.scala
89.47% <83.33%> (-3.86%)
:arrow_down:

... and 5 more

... and 4 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more
:loudspeaker: Have feedback on the report? Share it here.

Thanks for the changes @Kukovec!


ci refdoc test errors

troubleshooting https://github.com/Azure/azure-cli-extensions/pull/3026

reproduces issue with 3.8


Moved control plane ready status to planner

Issue:  #39139

Problem

This PR fixes node auto replace by changing how the control plane's ready status field is set.
Solution

Node Auto Replace

Mark the control plane as initialized and ready according to the upstream CAPI spec:

Field
Type
Description
Implementation in Kubeadm Control Plane Controller

initialized
Boolean
a boolean field that is true when the target cluster has completed initialization such that at least once, the target's control plane has been contactable.
Transitions to initialized when the controller detects that kubeadm has uploaded a kubeadm-config configmap, which occurs at the end of kubeadm provisioning.

ready
Boolean
Ready denotes that the target API Server is ready to receive requests.

Testing

Engineering Testing
Manual Testing

I tested node auto replace manually, and it functioned as expected.
Automated Testing

QA Testing Considerations

I recommend running through the entire gambit of tests related to v2prov.
Regressions Considerations

It is currently unknown what effects this may have, hence the recommendation to test as much as possible.

Closing in favor of https://github.com/rancher/rancher/pull/40083


Provide option for Github & other possible social login options through Kavach settings

[x] Level 1: Remove the option for Github login temporarily
[x] Level 2: Capture the details on Kavach studio/server, based on which the relevant login buttons show up as an intermediary solution.
[ ] Level 3: Provide an option in Kavach studio on what social logins are possible and capture the relevant details based on the provider selected. This would be the ideal solution, where all the options and relevant data for the logins to work as expected are passed through the UI. But, right now the options are dependent on YAML files on Kratos for the supported providers on the backend, we will have to wait on this until this data can be passed as REST API: https://www.ory.sh/kratos/docs/concepts/credentials/openid-connect-oidc-oauth2/

@shreeharsha-factly @vsumit89 Is this possible to do with the current APIs exposed by Kratos?


correct ordering Coin dropdown

correct ordering Coin dropdown

Can you please clarify why this is the correct order? The list is alphabetical so as far as I can see BCH should come before BTC.

because I was looking at name and not call symbol for some reason :)


replacing sassc with dartsass-sprockets

sassc is deprecated. Replacing sassc with dart-sass is the simplest and most straightforward methodology to keep using sass. The gem dartsass-sprockets provides a drop-in replacement implementing almost everything that sassc-rails offered utilizing the same namespace and method signatures.
There are several deprecation errors that needs to be addressed with the newer version of sass, but it would be good to upgrade to the latest version of the library.
These errors can be addressed in a future PR.
Deprecation errors:
    /usr/local/bundle/bundler/gems/rails_email_preview-0babc0ff21e4/app/assets/stylesheets/rails_email_preview/_default.sass 60:17   @import
    /usr/local/bundle/bundler/gems/rails_email_preview-0babc0ff21e4/app/assets/stylesheets/rails_email_preview/application.scss 1:9  root stylesheet
Deprecation Warning: Using / for division outside of calc() is deprecated and will be removed in Dart Sass 2.0.0.

Recommendation: math.div($rep-grid-gutter, -2) or calc($rep-grid-gutter / -2)

More info and automated migrator: https://sass-lang.com/d/slash-div

   ╷
61 │   margin-right: ($rep-grid-gutter / -2)
   │                  ^^^^^^^^^^^^^^^^^^^^^
   ╵
    /usr/local/bundle/bundler/gems/rails_email_preview-0babc0ff21e4/app/assets/stylesheets/rails_email_preview/_default.sass 61:18   @import
    /usr/local/bundle/bundler/gems/rails_email_preview-0babc0ff21e4/app/assets/stylesheets/rails_email_preview/application.scss 1:9  root stylesheet
Deprecation Warning: Using / for division outside of calc() is deprecated and will be removed in Dart Sass 2.0.0.

Recommendation: math.div($rep-grid-gutter, 2) or calc($rep-grid-gutter / 2)

More info and automated migrator: https://sass-lang.com/d/slash-div

   ╷
67 │   padding-right: ($rep-grid-gutter / 2)
   │                   ^^^^^^^^^^^^^^^^^^^^
   ╵
    /usr/local/bundle/bundler/gems/rails_email_preview-0babc0ff21e4/app/assets/stylesheets/rails_email_preview/_default.sass 67:19   @import
    /usr/local/bundle/bundler/gems/rails_email_preview-0babc0ff21e4/app/assets/stylesheets/rails_email_preview/application.scss 1:9  root stylesheet
Deprecation Warning: Using / for division outside of calc() is deprecated and will be removed in Dart Sass 2.0.0.

Recommendation: math.div($rep-grid-gutter, 2) or calc($rep-grid-gutter / 2)

More info and automated migrator: https://sass-lang.com/d/slash-div

   ╷
68 │   padding-left: ($rep-grid-gutter / 2)
   │                  ^^^^^^^^^^^^^^^^^^^^
   ╵
    /usr/local/bundle/bundler/gems/rails_email_preview-0babc0ff21e4/app/assets/stylesheets/rails_email_preview/_default.sass 68:18   @import
    /usr/local/bundle/bundler/gems/rails_email_preview-0babc0ff21e4/app/assets/stylesheets/rails_email_preview/application.scss 1:9  root stylesheet
Deprecation Warning: Using / for division outside of calc() is deprecated and will be removed in Dart Sass 2.0.0.

Recommendation: math.div($col, 12) or calc($col / 12)

More info and automated migrator: https://sass-lang.com/d/slash-div

   ╷
75 │       width: percentage($col / 12)
   │                         ^^^^^^^^^
   ╵

There appear to be quite a few Sass engines out there. Perhaps we can take a similar approach to the bootstrap ruby gem and support more of them?
https://github.com/twbs/bootstrap-rubygem/blob/main/lib/bootstrap/engine.rb

@glebm I'd be happy to update the PR with the library loading script. I'll add similar verbiage to the README as the bootstrap gem.

Ruby 3.0 and below are end of life and no longer supported. Should legacy tests be removed from the testing matrix? Only Rails 7.1, 7.0, and 6.1 is receiving support from the official rails team. Should Rails 5.0 and 4.2 be removed from the testing matrix?

Yep, please remove them

@glebm there's a dependency issue with rails_7_1 and ruby_version 3.1, otherwise all the tests pass. Here is the failed run:
https://github.com/wayhomeservices/rails_email_preview/actions/runs/9921493773/job/27409522524
I commented it out in the PR. I'm not 100% sure it is worth tracking down or not.

Thank you for your effort and patience, Jonathan! Merged!


Add GPT-Voice-Conversation-Chatbot to Open-source

This project allows you to have an engaging and safely emotive spoken / CLI conversation with the AI GPT-4 while giving you the option to let it remember things discussed.
Several things have been done to improve the conversational nature of GPT-4, such as using a combination of user base prompts and system prompts tailored for encouraging positive conversation, using multiple instances of GPT-4 within the bot, giving GPT-4 the ability to be aware of the current date and time, and providing GPT-4 with a customizable name, preset, and memory. The bot has also been designed to allow for conversations that go on well past the token limits of GPT-4 through the use of token recycling (basically stepping through the conversation and creating summaries and summaries of summaries until the gist of what was discussed can be preserved through a single summary) so that users can converse without the fear of running out of tokens.

Sorry about that! Just made a new commit that should fix that: moved it to the bottom and shortened the description a bit.
Thanks for letting me know!


clearer, more specific wording about sponsorship and contributing

make it clear that everyone's contributions are welcome, and make it clear that Ruby Together funds contributors, but does not maintain the project directly

@segiddins r?

@segiddins any feedback now?
@wycats can you weigh in here as well?

I think this is much better

@indirect This is a pretty big improvement.
My understanding is that the Bundler project would be happy to allow any company sponsoring significant amount of development time a similar kind of blurb and call to action. I think it's reasonable for Bundler to have a policy allowing sponsors to describe themselves in their own words, as long as it's a general policy that would apply to any sponsor and not specially crafted for RubyTogether.
@indirect I believe you're cool with that?

@wycats that's correct, and there will be a public policy document to that effect as soon as it's drafted and approved by the core team. 👍

@indirect ping on Colby's comments?

@segiddins you can pry my oxford comma from my cold dead hands. 😆 That said, less commas is probably better, so I updated the wording a bit.

@bundlerbot r+


Adds "secoff --> security officer" and "secoffs --> security officers" to chat sanitizer

About the PR

Adds secoff --> security officer
Adds secoffs -> security officers
Why / Balance

secoff is annoying to see in chat.
Technical details

I don't think there are any.
Media

[X] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Breaking changes

none
Changelog

:cl:

add: secoff and secoffs are properly sanitized in chat.

Is this valid?

Is this valid?

I would guess it is but it's encroaching on the territory of iconic slang

Honestly, this is so entrenched it feels like sanitizing CentCom -> Central Command.

Secoff is something that people could reasonably say irl. This feels like saying “cop > police officer” to me.

Yeah I agree I like sec/secoff as station slang, it's something I can see someone saying in real life, similar to 'FYI'.

I think it's fine that secoff exists.


SOMA ingestion integration

SOMA ingest implementation according to VCF standard.
Also includes some minor refactoring

Thank you John!


Address Cargo build out of memory issues

#317 introduces quite a few AWS-related packages to make S3-like storage possible. That appears to have tipped cargo [build|test|...] to run out of memory during ld. I got around this temporarily with this line but want a better long-term solution (less dependencies? other settings?).

Adding the line doesn't seem to have helped once merged into main.
Next experiment is to add a 2GB swap (there's 2GB of memory on this VPS): https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-20-04

That worked! Next: Will try to remove the Cargo.toml setting to see if it is necessary.

Now rustc is failing with
thread 'coordinator' panicked at /rustc/7cf61ebde7b22796c69757901dd346d0fe70bd97/compiler/rustc_codegen_ssa/src/back/write.rs:1762:6:
failed to spawn work thread: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }

To address this, this thread suggests echo 100000 | sudo tee /proc/sys/kernel/threads-max, which worked for a single pass. I'm going to try it for a few more runs on the next PR to see how it holds up.

OK. It's been working, but I have a suspicion that the build is ~5-6m instead of ~2m if I remove the debug flag. Will experiment more during my next session. (In other news, S3 put/list for snapshot backup seems to be working on localstack. Yay!)

Observation: in using localstack for integration testing S3-compatible storage, it's creating a bunch of python3 zombie processes, one every few seconds. I think a lot of what I was correlating as the build slowing down can be attributed to that. Will fix the localstack setup and see if this issue is largely resolved. (Adding swap space to help with the build is still important, so it's more of figuring out if we actually need to remove the debug symbols.)

OK! Despite no longer being an OOM issue, I've kept this code in Cargo.toml:
[profile.dev]
# debug = 2 takes ~50% longer to build for tests on marcua's tiny dev
# server. Comment this out if you need more debugging symbols.
# See https://github.com/marcua/ayb/issues/342.
debug = 1

With debug = 1, building for tests takes ~2 minutes, whereas if this section is commented out, it takes ~3 minutes.


Update cluster and linked services API

Latest improvements:
MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Contribution checklist:

[ ] I have reviewed the documentation for the workflow.
[ ] Validation tools were run on swagger spec(s) and have all been fixed in this PR.
[ ] The OpenAPI Hub was used for checking validation status and next steps.

ARM API Review Checklist

[ ] Service team MUST add the "WaitForARMFeedback" label if the management plane API changes fall into one of the below categories.
adding/removing APIs.
adding/removing properties.
adding/removing API-version.
adding a new service in Azure.

Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs.

[ ] If you are blocked on ARM review and want to get the PR merged urgently, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.
Please follow the link to find more details on API review process.

/azp run automation - sdk

Can one of the admins verify this patch?

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

Currently the BreakingChange Test is failing on the cluster API’s. Those APIS are in private preview and are only used by two customers. Both those customers are aware of the changes in the API and are currently using it as shown in my updates.

/azp run

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

@pilor can you review this? Thanks.

See Or's comment: https://github.com/Azure/azure-rest-api-specs/pull/8674#issuecomment-599556943

/azp run automation - sdk

@jianyexi Why we report error in breakingchange in preview API? Could you please help to have a look?

@jianyexi Why we report error in breakingchange in preview API? Could you please help to have a look?

currently , breaking-change does not  distinguish between preview API and stable API

@jianyexi But breaking change in preview api is allowed and should not block PR from merging. @akning-ms We need to make change and unblock this right?

/azp run automation - sdk

@akning-ms Please help to merge this PR. All the breaking change happens in preview API.

@akning-ms please merge this PR. Thanks.

/azp run automation - sdk

/azp run automation - sdk

@akning-ms Please help to merge this PR


Allow to extend plugins by array or function

I don't always just want to add a plugin to the end.

Thanks for the PR.
Could you also update the documentation please.

Ok, there wasn't a lot to begin with. I added a line.

Any objection with incorporating this?

I'm not sure to undestand why it's necessary. Do you have a concrete example ?

I'm not sure to undestand why it's necessary. Do you have a concrete example ?

Yes, it's right there in my first comment. "I don't always just want to add a plugin to the end."
And in the examples I added to the documentation.
My change allows you to prepend a plugin when plugin order matters (or even to insert).

Thanks, released in<EMAIL_ADDRESS>

@techhead can you add the link to doc?


is it possible to use luminous in vue.js?

I am trying to create photo website with vue.js and Laravel, and use luminous to magnify the photo when it is clicked.
My photos are stored in AWS S3 bucket.
My vue.js structure is like below.
1.PhotoComponent.vue

this component is literally rendering each photo.

<template>

            <img class="photo_image" :src="item.url" :alt="`photo taken by ~~`">

</template>

2.PhotoListComponent.vue

this component is making a list of PhotoComponent.vue.

<template>

            <PhotoComponent
                class="grid_item"
                v-for="photo in photos"
                :key="photo.id"
                :item="photo"
            ></PhotoComponent>

        <PaginateComponent :current-page="currentPage" :last-page="lastPage"></PaginateComponent>

</template>

There are three things that I have tried so far.

put new Luminous() in PhotoComponent, and put the function in mounted section.

<template>

            <img class="photo_image" :src="item.url" :alt="`photo taken by ~~`">

</template>

In this case, first picture works, but "" opens as many as PhotoComponent.vue.
For example, if I make a list of 9 pictures in PhotoListComponent and click the first photo, "" opens 9 times.
However, other pictures won't work at all, but the new tab opens.

put new LuminousGallery() in PhotoListComponent, and put the function in mounted section.

<template>

            <PhotoComponent
                class="grid_item"
                v-for="photo in photos"
                :key="photo.id"
                :item="photo"
            ></PhotoComponent>

        <PaginateComponent :current-page="currentPage" :last-page="lastPage"></PaginateComponent>

</template>

In this case, a new tab opens whenever the photo is clicked, which means that luminous does not work.

not mounted, but add click event in those component.

For example...
<template>

            <img class="photo_image" :src="item.url" :alt="`photo taken by ~~`">

</template>

In this case, the result is almost the same as pattern 1, but the first picture works perfectly as I expected. However, the other picutures won't open anything even a new tab.
Sorry for ranting.
I am stuck with this problem for a week. I really need someone's help.
Thank you for your cooperation.

Konnichiwa @wadakatu,
You should be able to make all of your solutions work.
For 1/ I think you need to be able to target only the image that exists within that component, so something like this could work:
<template>

      <img class="photo_image" :src="item.url" :alt="`photo taken by ~~`" />

</template>

For 2/ I think there might be an issue where the code isn't actually finding the image elements and thus is providing nothing to the LuminousGallery call. Maybe you could try debug that to find your issue.
I'm closing this for now, but I hope you find a solution to your issue!


determine cancelled for some events resets player yaw

on player right clicks entity:
    - determine cancelled

Resets pitch/yaw of the player, sometimes making them face a complete opposite direction..
Current work-around is to save the player's location, then teleport them back to it after the determine cancelled..
Video: https://youtu.be/c1rKEIY0dwQ

What the... this has got to be a Spigot issue. Right, @Morphan1 ?

I'm going to go ahead and say this is a spigot issue, not Denizen.


Trying to Delete entities that are in use silently fails in the UI

This is good, we don't want entities that are in use to be able to be deleted, but the failure shouldn't be silent. This should trigger a notification.

@dcalvom have you started on this? If not I am going to move to 0.16 since I have the rest of the notification stuff there.

This one was tackled by https://github.com/samtecspg/articulate/commit/72c0c62597c7390b80a269d75aab0c97bedc9865, but it's still under consideration to know if the notification on the chat panel is the way to go.

I'll open a new issue if we think we should move it some place else.


merkle tree improvements

Added code to check if a transaction belongs to a merkle root, provided the intermediate merkle tree neighboring values.

too old


AuthorsController CRUDify

I'm splitting up the giant CRUD controllers PR into chunks, starting with the smallest.
They'll still be big, just not as big. Hopefully easier to review.
This one's dainty, though. Truly manageable!

Coverage decreased (-0.01%) to 92.711% when pulling c72c085b1b69ddbf47e9d013e022be765644c7dd on nimmo-CRUD-authors-controller into bd70968eb61e30db734a78836325774e31fccd4a on master.

This is ready to go, will merge today if no objections.
Next PR is larger, requires this to merge first, and needs more scrutiny.

I'm sorry, Nimmo, but I find reviewing PRs where files move around nearly impossible.  I cannot follow all this CRUD stuff, moving actions into sub-controllers and all.  It makes my brain explode.  I'm so glad you've (apparently?) got it down.  I didn't notice anything questionable, FWIW.  If it passes the tests...

Thanks for checking it out. Yep, it's hard to follow.
Joe mentioned this too, and I was examining a method yesterday for maintaining git line blames while splitting up files, but it doesn't help when methods move from one file to another.
Still, i hope it helps breaking it up this way. The next PR is ready, and much bigger, but at least it's not the whole giant one.
Can't think of what else to do. They all kind of build on each other, and breaking these up is consequently very labor-intensive.

I appreciate the effort!  It would be awesome to connect those files.  I
assume you are doing "git mv" when you rename files, right?  (As opposed to
"git rm" the old one, then "git add" the new one.)  I see to remember
trying that one time as a test and it did nothing for the output of "git
diff".
Still, in all, it is probably "better" to have the smaller PRs, from the
perspective of making it easier to track down which PR is to blame in case
something goes wrong.
Someday, eventually, you will be able to work on "normal" PRs that don't
completely rewrite or move large portions of code!   :)
On Wed, Dec 7, 2022 at 2:47 PM andrew nimmo @.***>
wrote:

Thanks for checking it out. Yep, it's hard to follow.
Joe mentioned this too, and I was examining a method yesterday for
maintaining git line blames while splitting up files, but it doesn't help
when methods move from one file to another.
Still, i hope it helps breaking it up this way. The next PR is ready, and
much bigger, but at least it's not the whole giant one.
Can't think of what else to do. They all kind of build on each other, and
breaking these up is consequently very labor-intensive.
—
Reply to this email directly, view it on GitHub
https://github.com/MushroomObserver/mushroom-observer/pull/1213#issuecomment-1341503517,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAYTNNIRQSRHWM6JDFLGR3DWMDSU7ANCNFSM6AAAAAASWNVFYE
.
You are receiving this because your review was requested.Message ID:
@.***>


what does "mean return" mean in the tech report?

Hi,
Does it mean the reward of every episode? Besides, is it normal that ddpg algorithm can only successfully learn how to stand in the humanoid run environment given different seeds? Thanks.

There is some explanation on page 13 of the tech report:

In the state feature case we ran 15 different seeds for each task with A3C and DDPG; for results with D4PG, which was generally found to be more stable, we ran 5 seeds. In the raw-pixel case we also ran 5 different seeds for each task. The seed sets the network weight initialisation for the associated run. In all cases, initial network weights were sampled using standard TensorFlow initialisers. In the figures showing performance on individual tasks (Figures 4-7), the lines denote the median performance and the shaded regions denote the 5th and 95th percentiles across seeds. In the tables showing performance for individual tasks (Tables 1 & 2) we report means and standard errors across seeds.

Here, "performance" refers to the total reward per episode (the reward is between 0 and 1 per timestep, and there are always 1000 steps per episode). I believe "mean" refers to the fact that the learning curves were smoothed by computing the average reward per episode within a sliding window. Most DDPG seeds are able to learn humanoid:stand to some extent from features, although as mentioned in the text the final performance is not as good as D4PG, and there is a lot more variance across seeds.


Admin Login Error

Can't login admin with default email password getting unauthorized.

Assuming this is a fresh install - maybe try the password from the seed data
{
	"users": [
		{
			"email"<EMAIL_ADDRESS>			"password": "supersecret",
			"is_admin": true,
			"status": "active"
		}
	]
}

does not work<EMAIL_ADDRESS>/ supersecretshop.
even if you put values in table psql.

I just tried the same and had the same error

and me
please fix it

I modified the default credentials and then ran "yarn seed". Its working now!


chore: add a make target in order to run integration tests easily

this commit as been merged into https://github.com/sfeir-open-source/sonar-clover/pull/34


[Illuminate\Database\QueryException]   SQLSTATE[42S02]: Base table or view not found: 1146 Table 'blog.roles' doesn't exist (SQL: select * from roles where name = admin l   imit 1)

and also:
  [PDOException]
  SQLSTATE[42S02]: Base table or view not found: 1146 Table 'blog.roles' doesn't exist

did you migrated db correctly?

Truncate all the tables from the database and re-run the php artisan migrate once again to create all the migrations

My apologies I was called out of town and wasn't able to respond. I will be
back at my work environment today and will check your suggestions and will
let you know thanks!
On Wed, Feb 13, 2019, 12:33 PM Mariusz Rajczakowski <
<EMAIL_ADDRESS>wrote:

Closed #244 https://github.com/mariocoski/laravel-blogger/issues/244.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/mariocoski/laravel-blogger/issues/244#event-2137864437,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AV6jIhKxxruxqQBb6AYkp6mPULVi1i0fks5vNHahgaJpZM4azhjY
.


Change eslint warnings to errors

@alexlande I went ahead and changed all warnings to errors. We only had a small instance where warnings were given off, so it looks like all of those warning styles are usually already followed. Addresses #734.

Coverage remained the same at 96.26% when pulling 31f280b580e9e73026c37cc66154a5334819ea68 on jmcriffey:chore-lintWarnings2Errors into 8edd5d2a45599e569aa5034202ec0bbaed1f258d on FormidableLabs:master.

Great! Thanks @jmcriffey.


[BUG] PVC Mount timeout

Describe the bug (🐛 if you encounter this issue)
PVC mount error which caused one of Kafak replicas not to mount. Kafka got stuck on containerCreating. Please see bundle support.
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Perform '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Log or Support bundle
If applicable, add the Longhorn managers' log or support bundle when the issue happens.
You can generate a Support Bundle using the link at the footer of the Longhorn UI.
longhorn-support-bundle_641e11b3-d2f5-417c-8892-58b017d37635_2023-06-25T08-10-40Z.zip
Environment

Longhorn version: 1.3.2
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version:

Number of management node in the cluster: 1
Number of worker node in the cluster: 1

Node config

OS type and version: Ubuntu 20.04
CPU per node:
Memory per node:
Disk type(e.g. SSD/NVMe): HDD
Network bandwidth between the nodes:

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal):
Number of Longhorn volumes in the cluster: 9

Additional context
Add any other context about the problem here.

longhorn-support-bundle_641e11b3-d2f5-417c-8892-58b017d37635_2023-06-25T08-10-40Z.zip

The error message in longhorn-csi-plugin:
2023-06-25T01:51:40.287566874Z Mounting command: mount
2023-06-25T01:51:40.287571954Z Mounting arguments: -t xfs -o defaults /dev/longhorn/pvc-8a3f2df4-a015-4db5-8904-c9c564d3db4f /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8a3f2df4-a015-4db5-8904-c9c564d3db4f/globalmount
2023-06-25T01:51:40.287576332Z Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8a3f2df4-a015-4db5-8904-c9c564d3db4f/globalmount: /dev/longhorn/pvc-8a3f2df4-a015-4db5-8904-c9c564d3db4f already mounted or mount point busy.
2023-06-25T01:51:40.287580460Z
2023-06-25T01:51:40.287585018Z time="2023-06-25T01:51:40Z" level=error msg="NodeStageVolume: err: rpc error: code = Internal desc = mount failed: exit status 32\nMounting command: mount\nMounting arguments: -t xfs -o defaults /dev/longhorn/pvc-8a3f2df4-a015-4db5-8904-c9c564d3db4f /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8a3f2df4-a015-4db5-8904-c9c564d3db4f/globalmount\nOutput: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8a3f2df4-a015-4db5-8904-c9c564d3db4f/globalmount: /dev/longhorn/pvc-8a3f2df4-a015-4db5-8904-c9c564d3db4f already mounted or mount point busy.\n"

Would you check if this is a multipathd issue?

result of troubleshooting commands
ubuntu@node1:~$ ls -l /dev/longhorn
total 0
brw-rw---- 1 root root 8,  96 Jul  6 06:17 pvc-3adeeb3a-ad02-46dc-9e1f-e888f617bfaf
brw-rw---- 1 root root 8, 144 Jul  6 06:17 pvc-610da5b3-96ef-4871-9211-e2c64453b3a8
brw-rw---- 1 root root 8, 176 Jul  6 06:19 pvc-898202d5-cfae-4efb-b596-8ebbc7ff9767
brw-rw---- 1 root root 8, 240 Jul  6 06:17 pvc-8fbefb5d-6a15-4b8b-be4f-c93d69b08ce3
brw-rw---- 1 root root 8, 208 Jul  6 06:19 pvc-998e01db-0aab-41b2-93d0-05361e1416b0
brw-rw---- 1 root root 8, 112 Jul  6 06:17 pvc-a862ee86-6512-4cd4-9ef3-6e7c94cd949a
brw-rw---- 1 root root 8, 192 Jul  6 06:17 pvc-be634e55-5a4d-42f9-bccc-1ebf1346d06a
brw-rw---- 1 root root 8, 224 Jul  6 06:17 pvc-bf961deb-6c57-4b37-a7d8-835b3dd0ba4b
brw-rw---- 1 root root 8, 128 Jul  6 06:17 pvc-c207781e-6d13-46d8-92ea-56e0f3624e04
brw-rw---- 1 root root 8, 160 Jul  6 06:17 pvc-e9b4c5e1-1c5d-4428-bd0e-a02167136cbf
ubuntu@node1:~$ ls -l /dev | grep sdc
brw-rw----  1 root             disk      8,  32 Jul  6 06:14 sdc
ubuntu@node1:~$ lsof | grep sdc
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000

result of troubleshooting commands
ubuntu@node1:$ ls -l /dev/longhorn
total 0
brw-rw---- 1 root root 8,  96 Jul  6 06:17 pvc-3adeeb3a-ad02-46dc-9e1f-e888f617bfaf
brw-rw---- 1 root root 8, 144 Jul  6 06:17 pvc-610da5b3-96ef-4871-9211-e2c64453b3a8
brw-rw---- 1 root root 8, 176 Jul  6 06:19 pvc-898202d5-cfae-4efb-b596-8ebbc7ff9767
brw-rw---- 1 root root 8, 240 Jul  6 06:17 pvc-8fbefb5d-6a15-4b8b-be4f-c93d69b08ce3
brw-rw---- 1 root root 8, 208 Jul  6 06:19 pvc-998e01db-0aab-41b2-93d0-05361e1416b0
brw-rw---- 1 root root 8, 112 Jul  6 06:17 pvc-a862ee86-6512-4cd4-9ef3-6e7c94cd949a
brw-rw---- 1 root root 8, 192 Jul  6 06:17 pvc-be634e55-5a4d-42f9-bccc-1ebf1346d06a
brw-rw---- 1 root root 8, 224 Jul  6 06:17 pvc-bf961deb-6c57-4b37-a7d8-835b3dd0ba4b
brw-rw---- 1 root root 8, 128 Jul  6 06:17 pvc-c207781e-6d13-46d8-92ea-56e0f3624e04
brw-rw---- 1 root root 8, 160 Jul  6 06:17 pvc-e9b4c5e1-1c5d-4428-bd0e-a02167136cbf
ubuntu@node1:$ ls -l /dev | grep sdc
brw-rw----  1 root             disk      8,  32 Jul  6 06:14 sdc
ubuntu@node1:$ lsof | grep sdc
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000
lsof: no pwd entry for UID 2000

@Ahmed20230
Have you followed the suggestions in the multipathd doc https://longhorn.io/kb/troubleshooting-volume-with-multipath/?
lease let me know if you need any help. Thank you.

I have followed the suggestions provided in the multipathd documentation, and I am currently waiting to observe whether the issue will recur or not.

@andyjeffries @Ahmed20230
OK, thanks. BTW, we just got some messages from @innobead. It seems you have some concerns about LH. If possible, can you elaborate more on it, and then we can see if there is anything we can further improve it?

Hi @derekbit
Thanks for getting back to us. We have been having so many issues with our on-prem solution and LH. First, here's the the spec of the machine:

The setup is a single node. and we have the following services along with LH - Minio, Kafka, SFTPGo, and an API service. all installed in a cluster.
Now issues we have faced, is sometimes out of the blue, LH will fail and all the services will go down as a result. We try to get them to work and they try to mount, but they sometimes don't and we get stuck in a loop. The only solution is to restart the server.
The issue with LH that it's not stable. These unexpected failures drop the whole system. Any recommendation to make it more stable would be great.
Also, we are buying another server, so if you have any recommendation, please let us know.
Thank you!
Tareq

@derekbit Thank you for your help. I was telling @innobead. The system is not stable at all. Maybe we have done something wrong. Do you have any steps to audit the system/configuration? What are the things we need to do to make sure that we have the right logs to track the root cause?

@Ahmed20230 @andyjeffries
I see. If possible, can you provide us a support bundle and mail to<EMAIL_ADDRESS>when the system is in an unstable state? Then, we can do a further analysis.

@derekbit Will do, but check out the one you have in this bug. And here's another one:
longhorn-support-bundle_641e11b3-d2f5-417c-8892-58b017d37635_2023-01-29T22-59-34Z.zip
Here's the issue: https://github.com/longhorn/longhorn/issues/5323

@derekbit Will do, but check out the one you have in this bug. And here's another one: longhorn-support-bundle_641e11b3-d2f5-417c-8892-58b017d37635_2023-01-29T22-59-34Z.zip
Here's the issue: #5323

Thank you. Will look into the issues.

@derekbit any comments on the hardware? Is there any issue with the spec?

cc @innobead @shuo-wu @PhanLe1010
Do you have any suggestion for the hardware spec?

From my side, I think storage and network need to be

Storage: SSD disks
Network bandwidth: 10Gb+

@derekbit @innobead Biggest SSD is 8TB. We process a lot of images. Almost 1.2TB per month or more. This won't work.

@aljabertareq @Ahmed20230
https://github.com/longhorn/longhorn/issues/5323
There are 10 volumes stored on the HDD that can easily experience IO starvation when running IO-intensive applications.
Currently, the default timeout between a volume's engine and replica is set to 8 seconds. If the timeout is reached, the volume crashes. Thus, we can see many similar error logs
2023-02-23T03:31:01.384461459Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: Feb 23 03:30:55.719777: -><IP_ADDRESS>:10121 P[  0kB]      173us"
2023-02-23T03:31:01.384473162Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: Feb 23 03:30:57.719837: -><IP_ADDRESS>:10121 P[  0kB]      233us"
2023-02-23T03:31:01.384480106Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: Feb 23 03:30:49.881481: -><IP_ADDRESS>:10121 R[  4kB]  8800334us failed"
2023-02-23T03:31:01.384490015Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: time=\"2023-02-23T03:30:58Z\" level=error msg=\"Replicator.ReadAt:0 r/w timeout\""
2023-02-23T03:31:01.384494564Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: time=\"2023-02-23T03:30:58Z\" level=error msg=\"Setting replica tcp://<IP_ADDRESS>:10120 to ERR due to: r/w timeout\""
2023-02-23T03:31:01.384531416Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: time=\"2023-02-23T03:30:58Z\" level=info msg=\"Set replica tcp://<IP_ADDRESS>:10120 to mode ERR\""
2023-02-23T03:31:01.384541565Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: time=\"2023-02-23T03:30:58Z\" level=error msg=\"I/O error: tcp://<IP_ADDRESS>:10120: r/w timeout\""
2023-02-23T03:31:01.384551335Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: time=\"2023-02-23T03:30:58Z\" level=info msg=\"Monitoring stopped tcp://<IP_ADDRESS>:10120\""

To mitigate the issue, you can do

Replace the SDD with HDD.

But it doesn't work in your https://github.com/longhorn/longhorn/issues/6188#issuecomment-1631804645

We have introduced an engine-replica-timeout since v1.4. You probably can increase the 8 seconds to a big value such as 15 seconds. https://longhorn.io/docs/1.4.2/references/settings/#engine-to-replica-timeout

BTW, could you provide the dmesg of the node? It would be helpful to see if the IO hang occurs frequently.

Biggest SSD is 8TB. We process a lot of images. Almost 1.2TB per month or more. This won't work.

Haven't gotten the point... Why does processing 1.2TB+ images per month would lead to 8TB SSD unfitable while 16TB HDD is fine?

Biggest SSD is 8TB. We process a lot of images. Almost 1.2TB per month or more. This won't work.

Haven't gotten the point... Why does processing 1.2TB+ images per month would lead to 8TB SSD unfitable while 16TB HDD is fine?

Same question.
Why not utilize a RAID such as RAID5 configuration comprising multiple SSD disks?

@shuo-wu If I am not mistaken, we have 5x 16TB HDD. I believe we utilize RAID to comprise 5 HDDs.

@aljabertareq @Ahmed20230
#5323 and this issue are same.
There are 10 volumes stored on the HDD that can easily experience IO starvation when running IO-intensive applications. Currently, the default timeout between a volume's engine and replica is set to 8 seconds. If the timeout is reached, the volume crashes. Thus, we can see many similar error logs
2023-02-23T03:31:01.384461459Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: Feb 23 03:30:55.719777: -><IP_ADDRESS>:10121 P[  0kB]      173us"
2023-02-23T03:31:01.384473162Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: Feb 23 03:30:57.719837: -><IP_ADDRESS>:10121 P[  0kB]      233us"
2023-02-23T03:31:01.384480106Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: Feb 23 03:30:49.881481: -><IP_ADDRESS>:10121 R[  4kB]  8800334us failed"
2023-02-23T03:31:01.384490015Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: time=\"2023-02-23T03:30:58Z\" level=error msg=\"Replicator.ReadAt:0 r/w timeout\""
2023-02-23T03:31:01.384494564Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: time=\"2023-02-23T03:30:58Z\" level=error msg=\"Setting replica tcp://<IP_ADDRESS>:10120 to ERR due to: r/w timeout\""
2023-02-23T03:31:01.384531416Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: time=\"2023-02-23T03:30:58Z\" level=info msg=\"Set replica tcp://<IP_ADDRESS>:10120 to mode ERR\""
2023-02-23T03:31:01.384541565Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: time=\"2023-02-23T03:30:58Z\" level=error msg=\"I/O error: tcp://<IP_ADDRESS>:10120: r/w timeout\""
2023-02-23T03:31:01.384551335Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: time=\"2023-02-23T03:30:58Z\" level=info msg=\"Monitoring stopped tcp://<IP_ADDRESS>:10120\""

To mitigate the issue, you can do

Replace the SDD with HDD.

But it doesn't work in your env [BUG] PVC Mount timeout  #6188 (comment)
For disk size issue, utilizing a RAID such as RAID5 configuration comprising multiple SSD disks is a solution

We have introduced an engine-replica-timeout since v1.4. You probably can increase the 8 seconds to a big value such as 15 seconds. https://longhorn.io/docs/1.4.2/references/settings/#engine-to-replica-timeout

BTW, could you provide the dmesg of the node? It would be helpful to see if the IO hang occurs frequently.

@aljabertareq @Ahmed20230
#5323 and this issue are same.
There are 10 volumes stored on the HDD that can easily experience IO starvation when running IO-intensive applications. Currently, the default timeout between a volume's engine and replica is set to 8 seconds. If the timeout is reached, the volume crashes. Thus, we can see many similar error logs
2023-02-23T03:31:01.384461459Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: Feb 23 03:30:55.719777: -><IP_ADDRESS>:10121 P[  0kB]      173us"
2023-02-23T03:31:01.384473162Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: Feb 23 03:30:57.719837: -><IP_ADDRESS>:10121 P[  0kB]      233us"
2023-02-23T03:31:01.384480106Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: Feb 23 03:30:49.881481: -><IP_ADDRESS>:10121 R[  4kB]  8800334us failed"
2023-02-23T03:31:01.384490015Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: time=\"2023-02-23T03:30:58Z\" level=error msg=\"Replicator.ReadAt:0 r/w timeout\""
2023-02-23T03:31:01.384494564Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: time=\"2023-02-23T03:30:58Z\" level=error msg=\"Setting replica tcp://<IP_ADDRESS>:10120 to ERR due to: r/w timeout\""
2023-02-23T03:31:01.384531416Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: time=\"2023-02-23T03:30:58Z\" level=info msg=\"Set replica tcp://<IP_ADDRESS>:10120 to mode ERR\""
2023-02-23T03:31:01.384541565Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: time=\"2023-02-23T03:30:58Z\" level=error msg=\"I/O error: tcp://<IP_ADDRESS>:10120: r/w timeout\""
2023-02-23T03:31:01.384551335Z time="2023-02-23T03:31:01Z" level=warning msg="pvc-0dd87029-010b-459a-bd00-43cf02e18dbf-e-be680aa1: time=\"2023-02-23T03:30:58Z\" level=info msg=\"Monitoring stopped tcp://<IP_ADDRESS>:10120\""

To mitigate the issue, you can do

Replace the SDD with HDD.

But it doesn't work in your env [BUG] PVC Mount timeout  #6188 (comment)
For disk size issue, utilizing a RAID such as RAID5 configuration comprising multiple SSD disks is a solution

We have introduced an engine-replica-timeout since v1.4. You probably can increase the 8 seconds to a big value such as 15 seconds. https://longhorn.io/docs/1.4.2/references/settings/#engine-to-replica-timeout

BTW, could you provide the dmesg of the node? It would be helpful to see if the IO hang occurs frequently.

@derekbit so now we are on 1.32. To set the timeout value to 15, we will need to upgrade to 1.4 and then do that, right?

@aljabertareq
Yes, exactly! v1.4.2 or the upcoming v1.4.3 is recommanded.

@derekbit, are you sure this is the root cause of the issue? Any data loss risk if we upgrade?
By the way, it happens once a month :). It's as if LH gets tired at the end of the month and decides to crash. I'm not sure if that's a coincidence.

are you sure this is the root cause of the issue

From the logs, I think yes.

Any data loss risk if we upgrade?

For v1.4.2, we didn't receive upgrade issues but still recommend backing up data first.
@shuo-wu any concern?

It's as if LH gets tired at the end of the month and decides to crash

The crash here means some volumes don't work or crash, right?

@derekbit Yes, some volumes don't work and cause a cascading effect on all connected services. Then, when you try to restart some of these services (Kafka, Minio, etc..), some work just fine while others don't get restarted due to mounting issue.

I see. Then, a dmesg is appreciated. Then, we can help check if what the timeout is recommanded. Thanks.

dmesg-output.txt

Agree that:
The prefer approach

We should swap out the HDD and replace by SSD. Then use RAID5 to combine the SSDs into a big directory. Having mix of SSD and HDD will cause problem down the line.
Set a Guaranteed Engine Manager CPU and Guaranteed Engine Manager CPU https://longhorn.io/docs/1.4.2/references/settings/#guaranteed-engine-manager-cpu. I don't see they are set now.

Less prefer approach
If replacing the HDD by SSD is impossible at the moment, we can reduce the pain by:

Detach all volumes by scaling down the workload
Upgrade Longhorn to v1.4.2
Set a Guaranteed Engine Manager CPU and Guaranteed Engine Manager CPU https://longhorn.io/docs/1.4.2/references/settings/#guaranteed-engine-manager-cpu
Change engine to replica timeout to 30s https://longhorn.io/docs/1.4.2/references/settings/#engine-to-replica-timeout

Btw, is it a good idea to build a customized longhorn-engine:v1.3.2 image with bigger timeout to test the theory? @derekbit

Btw, is it a good idea to build a customized longhorn-engine:v1.3.2 image with bigger timeout to test the theory? @derekbit

Good idea. I can try it.

@PhanLe1010, we will definitely go with the latter option. Replacing HDD with SSD is not feasible, and it is also not possible with the new server. The primary concern here is cost, as the price difference for us amounts to nearly 8k. You and your team may want to take this into consideration and explore designing a solution that optimizes performance with HDDs.
@derekbit Testing the scenario before us doing this would be greatly appreciated.

@PhanLe1010 @derekbit @innobead Longhorn just crashed again and customer is complaining:
think its longhorn issue this was the error was “Warning  FailedAttachVolume  9s (x6 over 27s)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-8fbefb5d-6a15-4b8b-be4f-c93d69b08ce3" : rpc error: code = DeadlineExceeded desc = volume pvc-8fbefb5d-6a15-4b8b-be4f-c93d69b08ce3 failed to attach to node node1”
Sigh.....

@aljabertareq Can you provide a support bundle?

Let's first relaunch the system. Give me time. LH is a true nightmare.

Here you go:
longhorn-support-bundle_641e11b3-d2f5-417c-8892-58b017d37635_2023-07-13T04-44-20Z.zip

@aljabertareq
The dmesg didn't show the IO hung for 30 seconds warning, so I think it should be caused by the IO starvation as I mentioned before rather than a bug in the data path.
The support bundle doesn't provide useful information for the crashed volume. The log messages are gone after relaunching the system. I suppose it is still from the same IO timeout issue.

The test was conducted in v1.4.2 (change engine-replica-timeout) as well as v1.3.2 (change a hardcoded timeout value).
Reproduce IO timeout

Prepare a disk that has a dm-delay layer on the top of a block device

fallocate -l 50G blockfile && sudo losetup --direct-io=on /dev/loop0 blockfile
echo "0 $(sudo blockdev --getsize /dev/loop0) delay /dev/loop0 0 0" | dmsetup create delayed-loop0
mkfs.ext4 /dev/dm-0
mkdir -p /mnt/disk && mount /dev/dm-0 /mnt/disk

Create a volume with a single replica residing on the diskapiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-volv-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn-test
  resources:
    requests:
      storage: 2Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: volume-test
  namespace: default
spec:
  restartPolicy: Always
  containers:
  - name: volume-test
    image: nixery.dev/shell/fio
    args:
      - sleep
      - "1000000"
    volumeMounts:
    - name: volv
      mountPath: /data
    ports:
    - containerPort: 80
  volumes:
  - name: volv
    persistentVolumeClaim:
      claimName: longhorn-volv-pvc

Run fio in the volume-test podfio --name=write --ioengine=libaio --iodepth=1 --rw=write --bs=4k --direct=1 --size=10G --numjobs=1 --runtime=3000 --filename=/data/file

Change the delay value on the attached node
: N millisecondsecho "0 $(sudo blockdev --getsize /dev/loop0)"   "delay /dev/loop0 0 <N>"   | dmsetup reload delayed-loop0 && dmsetup resume delayed-loop0

Then, you can observe the R/W IO timeout in the instance-manager-e pod

Mitigation
Increase the setting engine-replica-timeout to a bigger value, e.g. 20. The R/W IO timeout is not seen.
From the test, adjusting engine-replica-timeout is a feasible solution.
cc @PhanLe1010 @shuo-wu @innobead

BTW, I not sure if you can add more nodes or disks to Longhorn system for IO load balancing.
Based on the support bundle info, there are 12 replicas in the only Longhorn disk. Even if this disk is actually a RAID comprised by 5 HDDs, I don't think the preformance is enough when there are intensive IO requests. Adding more RAID disks may be relief the load and reduce the chance of IO starvation. Of course, this would increase the cost. But rely on a single (performance-limited HDD RAID) disk to satisfy all workloads in a cluster is not a good idea anyway.

Yes, a large number of input/output intensive applications can make IO starvation more likely. Therefore, it is essential to increase the timeouts for block devices, iSCSI protocol-related timeouts, and engine-replica timeouts in a demanding environment.

@aljabertareq
BTW, for a single node case, why not use, for example, local-path provisioner as a solution?

@derekbit to change the engine-replica-timeout to a bigger value we will need to upgrade LH to 1.42, right?
If that's the case, the client won't approve any downtime until early August. Anything we can do in the mean time that might help but does not require system restart? cc:
@Ahmed20230

@aljabertareq
BTW, for a single node case, why not use, for example, local-path provisioner as a solution?

Because we thought the customer will have another node, and they might.

@aljabertareq

Upgrade to v1.4.2 to avoid unavailability. Upgrading to a new version will not cause downtime.

After upgrade, you can update engine-replica-timeout to e.g. 20 seconds.
The existing attached volume is still using the old timeout value, i.e. 8 seconds. If an IO timeout happens, the new timeout value is applied to the re-attached volumes.

One possible solution: Is it better that using single disk rather than a RAID for each disk in Longhorn for storage resource isolation to relieve the IO starvation caused by many IO intensive applications? Not sure if this is helpful, but we can check it later.

Thank you, @derekbit , for the suggestions. I have a few questions:
Can you provide a guarantee that implementing this solution will effectively resolve the issue?
You mentioned that upgrading LH won't disrupt the service. Could you explain how that is possible? Wouldn't we need to detach all the volumes, perform the upgrade, adjust timeout settings, and then reattach all the volumes, resulting in downtime for our services like Kafka and others?
Suggestion:
Considering that LH has compatibility issues with HDD, it would be helpful if your specifications clearly state this limitation.

@aljabertareq have u checked @derekbit test result https://github.com/longhorn/longhorn/issues/6188#issuecomment-1634246818? see if having any questions.
The setting for the timeout between engine and replica mentioned above is a way to fine tune your setup which has been discussed here.
Have a question: have u upgrade longhorn before? Do u know about our live upgrade? Or had any challenges for the upgrade. Probably also can check the doc as well.
For other questions, @derekbit will help with that.
P.S. strongly suggest to help verify the suggestions from your staging env and cooperate with us if having any further challenges.

Have a question: have u upgrade longhorn before? Do u know about our live upgrade? Or had any challenges for the upgrade. Probably also can check the doc as well.
@innobead innobead , Yes, we have performed an upgrade to Longhorn v1.3.2, as I mentioned earlier. The process involved detaching all the volumes, bringing down the system, performing the upgrade, and then relaunching it.

Regarding the live upgrade feature, I'm not familiar with it. Could you provide a document or any information about it? Additionally, if there are any timeout settings that need to be changed, would the live upgrade automatically handle it or would it still require restarting the system with the older version?

@innobead @derekbit

@aljabertareq

Can you provide a guarantee that implementing this solution will effectively resolve the issue?

No, as @innobead and I mentioned before. There is still a timeout and what you can do is change the engine-replica-timeout to reduce the probability of running into the issue. In the light of the HDD environment and multiple IO intensive applications, I cannot say the issue must be resolved after increasing the value.

Additionally, if there are any timeout settings that need to be changed, would the live upgrade automatically handle it or would it still require restarting the system with the older version?

Yes, you need to update the setting engine-replica-timeout to a bigger value, e.g. 20, after upgrade.

Could you provide a document or any information about it?

Please refer to the upgrade doc.
Also, the doc Troubleshooting: Some old instance manager pods are still running after upgrade  is worth reading. Then, you can understand the behaviors of instance-manager-r/e after upgrade.

P.S. strongly suggest to help verify the suggestions in your staging env first and then cooperate with us if having any further challenges. This is very important and helpful for open source adoption/contribution.

100% Agree. Suggest verifying the upgrade in your staging env before upgrading your production environment.

Thanks @derekbit!
Please see the attached screenshot. Upgrade engine is disabled.

Thanks @derekbit!
Please see the attached screenshot. Upgrade engine is disabled. Any idea why?

That the system is not upgraded, and the engine image longhornio/longhorn-engine:v1.4.2 is not deployed yet.

@derekbit @innobead Upgrade is done to 1.4.2 and timeout is set to 30 sec. Let's see.

@innobead @derekbit Last night, we had another crash. Kafka was not able to write logs, and I believe Minio went down because of Longhorn. Here's the support bundle: https://drive.google.com/file/d/1Ft8Owy2914f-6_sW0IobNue1Iw0ODz0_/view?usp=sharing
Can you please check and let us know?

@aljabertareq
I don't see any IO timeout or error in the support bundle. Longhorn looks well from 07/15-07/22. Could you tell us the name of the problematic volume and the time it happened?

@aljabertareq
BTW, you can check Minio's log and see if there is any clue.

The issue happened on July 23 at 2:20am PST.  avs-Kafka volume. pvc-3adeeb3a-ad02-46dc-9e1f-e888f617bfaf
Minio went down because that Kafka went down.

The issue happened on July 23 at 2:20am PST. avs-Kafka volume. pvc-3adeeb3a-ad02-46dc-9e1f-e888f617bfaf
Minio went down because that Kafka went down.

Can you provide dmesg -T of the node?

@derekbit sorry for the delayed response. We just had another crash. Here you go:
dmesg_logs.log.txt

I saw many IO hang events in the dmesg. It looks the 30 seconds timeout is still not enough.
[Sat Jul 22 08:03:56 2023] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Can you provide the results sysctl -a | grep dirty?
@shuo-wu @PhanLe1010 Any suggestion for this case?
cc @innobead

@derekbit are we able to verify that the 30s timeout got applied to the volume engine?

@derekbit are we able to verify that the 30s timeout got applied to the volume engine?

Need a support bundle.
@aljabertareq Can you provide a latest support bundle?

Just checked the dmesg result. It seems that the trace always points to xfs stack even if hung_task_timeout errors are triggered by some tasks like tgtd or dockerd.
Then I google this kind of error, some articles mention that they are often encountered in machines with powerful CPU or large RAM and it may be a result of the mismatching between disk throughput and vm cache settings. And they suggest to increase the related config like vm.dirty_ratio or dirty_background_ratio.
Considering that increasing vm cache will lead to a higher possibility of enocuntering data loss or fs corruption after an unexpected node down, decrease the cache value may be a workaround. I am not sure if it's possible.

@derekbit Here's the link to the bundle https://drive.google.com/file/d/1KBVbqJQWJsYch330Eqwj1L5roD-NQbjo/view?usp=sharing
I doubt it's a LH IO issue, but it might be related. Not sure. It's Minio IO error. Last time was Kafka.

Just checked the dmesg result. It seems that the trace always points to xfs stack even if hung_task_timeout errors are triggered by some tasks like tgtd or dockerd. Then I google this kind of error, some articles mention that they are often encountered in machines with powerful CPU or large RAM and it may be a result of the mismatching between disk throughput and vm cache settings. And they suggest to increase the related config like vm.dirty_ratio or dirty_background_ratio. Considering that increasing vm cache will lead to a higher possibility of enocuntering data loss or fs corruption after an unexpected node down, decrease the cache value may be a workaround. I am not sure if it's possible.

Thanks for the suggestion but it's not clear to me whether you want me to increase the cache or decrease it. Could you please test it locally?

@derekbit How do you check if the timeout change has been applied?

I mean, decreasing it may be one workaround.

@PhanLe1010 @shuo-wu
Weird, I don't see IO timeout in the engine-replica path, but I see the io error in the CSI plugin log.
2023-07-26T02:06:04.430619437Z time="2023-07-26T02:06:04Z" level=error msg="NodeGetVolumeStats: req: {\"volume_id\":\"pvc-e9b4c5e1-1c5d-4428-bd0e-a02167136cbf\",\"volume_path\":\"/var/lib/kubelet/pods/97fcd5dd-229a-4543-b088-4d6a37108050/volumes/kubernetes.io~csi/pvc-e9b4c5e1-1c5d-4428-bd0e-a02167136cbf/mount\"} err: rpc error: code = Internal desc = failed to check volume mode for volume path /var/lib/kubelet/pods/97fcd5dd-229a-4543-b088-4d6a37108050/volumes/kubernetes.io~csi/pvc-e9b4c5e1-1c5d-4428-bd0e-a02167136cbf/mount: input/output error"

Got it. Can you please add links to these articles as a reference?
Our system has 1 CPU (64-core).
Thank you!

@shuo-wu dirty_ratio is 20
dirty_background_ratio is 10

@derekbit are we able to verify that the 30s timeout got applied to the volume engine?

@derekbit were you able to verify that the 30s got applied?  Thanks!

@derekbit are we able to verify that the 30s timeout got applied to the volume engine?

@derekbit were you able to verify that the 30s got applied? Thanks!

Yes, it is applied. As I mentioned, there is no engine-replica timeout error, so it is not due to the previous issue.
What I'm concerned is the IO error in https://github.com/longhorn/longhorn/issues/6188#issuecomment-1651007162. Not sure if it is an issue in xfs filesystem.

Here is the article:
https://www.blackmoreops.com/2014/09/22/linux-kernel-panic-issue-fix-hung_task_timeout_secs-blocked-120-seconds-problem/

Yes, it is applied. As I mentioned, there is no engine-replica timeout error, so it is not due to the previous issue.

What I'm concerned is the IO error in https://github.com/longhorn/longhorn/issues/6188#issuecomment-1651007162. Not sure if it is an issue in xfs filesystem.

Weird... Does it mean that the xfs IO error is triggered by the filesystem inside the volume rather than the data path replicas are using?

Weird... Does it mean that the xfs IO error is triggered by the filesystem inside the volume rather than the data path replicas are using?

I think yes.
@aljabertareq
You can apply @shuo-wu's suggestion that changes vm.dirty settings and see if all things can go well.
If the xfs has still IO error, you probably can try ext4 instead.

@derekbit, thanks for that. I’ll try that early next month as it’s a production crunch right now. I’ll keep you posted.

Just fyi and hope it helps someone…
Similar issue experienced when restarting container and attaching large (~4tb used on pvc with millions of files)  30tb pv from 32tb spinning disk lvm. 10gb network. No issue until it just hit a wall..
Got past it by changing rke2 systemd unit file :
Stop rke2 server, change
ExecStart=/usr/bin/rke2 server
To
ExecStart=/usr/bin/rke2 server --kubelet-arg "runtime-request-timeout=90m”
then of course the daemon reload. Start rke2 server.
Wait.. for attach. (Actually took roughly 65 minutes) as far as I can tell, too much timeout is better than too little.. it is working now.
your mileage may vary.
Yes, I realize pushing it.
Though I do have a question:
Instead of one large pv/pvc, if changed to using multiple smaller pv, will those attach in parallel or serially? Even though this works, for future containers, can the total container start up time be reduced by having the pv.s attach in parallel? Is that possible to configure now and I am just missing where to configure it?


FilePond cannot properly upload large file (> 2mb)

When I upload a file that is larger than around 2mb, I found that the data I got from the server side is not complete.  (May be because the file is too large for url to take from client to server ? ) Since I realize the problem, I started to use file-upload middleware, such as Multer or Express-fileupload to upload a large file through FilePond. However, It does not work. Therefore, I want to know is there a way or any configurations allow me to upload a large file successfully?

Closed because doesn't follow issue template.
Please note that uploads above 2MB can be done without issue, this is most likely a server configuration issue.


Production failed Sheet component 'className' does not exist on type 'DialogPortalProps'.

I'm currently builting an e-commerce site withnext js and shadcn. for sidebar I've used Sheet component. Everything works fine in the dev mode. when pushed to production, production failed
Error

Code

I tried readding the sheet component it's not worked

Repository For Reference
https://github.com/Magesh-sam/next-store/
Sidebar component
https://github.com/Magesh-sam/next-store/blob/main/components/Sidebar.tsx
deployed on vercel
https://nextstore-stripe.vercel.app/
log from vercel
[18:50:43.584] Running build in Washington, D.C., USA (East) – iad1 (Hive)
[18:50:43.686] Cloning github.com/Magesh-sam/next-store (Branch: main, Commit: 94897d0)
[18:50:44.141] Cloning completed: 454.702ms
[18:50:48.414] Restored build cache
[18:50:48.465] Running "vercel build"
[18:50:48.906] Vercel CLI 32.4.1
[18:50:49.228] Installing dependencies...
[18:50:49.279] bun install v1.0.3 (25e69c71)
[18:50:49.290]  Resolving dependencies
[18:50:49.450]  Resolved, downloaded and extracted [2]
[18:50:49.454]  Saved lockfile
[18:50:49.506]  +<EMAIL_ADDRESS>[18:50:49.507]  +<EMAIL_ADDRESS>[18:50:49.507]
[18:50:49.507]  2 packages installed [251.00ms]
[18:50:49.511] Detected Next.js version: 13.5.3
[18:50:49.514] Running "bun run build"
[18:50:49.522] $ next build
[18:50:50.099]    Creating an optimized production build ...
[18:51:00.686]  ✓ Compiled successfully
[18:51:00.686]    Linting and checking validity of types ...
[18:51:04.108] Failed to compile.
[18:51:04.108]
[18:51:04.108] ./components/ui/sheet.tsx:17:3
[18:51:04.108] Type error: Property 'className' does not exist on type 'DialogPortalProps'.
[18:51:04.108]
[18:51:04.108]   15 |
[18:51:04.108]   16 | const SheetPortal = ({
[18:51:04.109] > 17 |   className,
[18:51:04.109]      |   ^
[18:51:04.109]   18 |   ...props
[18:51:04.109]   19 | }: SheetPrimitive.DialogPortalProps) => (
[18:51:04.109]   20 |   <SheetPrimitive.Portal className={cn(className)} {...props} />
[18:51:04.166] error: script "build" exited with code 1 (SIGHUP)
[18:51:04.170] Error: Command "bun run build" exited with 1

Hey Magesh,
You're not the only one with this problem. I hope this gets fixed soon. I tried some things but no positive results.

This bug is caused by upstream radix-ui changing their API. https://github.com/radix-ui/primitives/issues/2423
Both @radix-ui/react-dialog (^1.0.5) and @radix-ui/react-alert-dialog (^1.0.5) were affected.
You need upgrade your shadcn/ui code to fix it.
By CLI:
pnpm dlx shadcn-ui@latest add --overwrite alert-dialog

Or just copy new code on website.

Found solution

Created SheetPortal similar to the AlerDialog component.

const SheetPortal = SheetPrimitive.Portal;

Replace the existing component with the new Component or else comment out the existing component.

works fine now in the production
Deployed in vercel
https://nextstore-stripe.vercel.app/
check in small screen to view sheet component