with gps fator,  the map optimization node crashed  when encountering a big loop

Thanks for your hard work!
I run LIO-SAM with my own dataset,  the sensors are velodyne 16, steam 300 and F9P GPS.
when the scene is very small(small loop) or the trajectory distance is very short, the LIO-SAM system runs well with GPS.
But when encountering a big closed loop, the mapOptimization node died and the lio_sam_udi_imuPreintegration process restart.how can i solve this problem?
I was confused by this problem for several weeks and it made me really frustrated. I would be grateful if anyone could help me
I recorded a vedio : vedio

ERROR:
Thrown when a linear system is ill-posed.  The most common cause for this
error is having underconstrained variables.  Mathematically, the system is
underdetermined.  See the GTSAM Doxygen documentation at
http://borg.cc.gatech.edu/ on gtsam::IndeterminantLinearSystemException for
more information.
[lio_sam_udi_imuPreintegration-1] process has died [pid 4210, exit code -6, cmd /home/xchu/udicreator_ws/devel/lib/lio_sam_udi/lio_sam_udi_imuPreintegration __name:=lio_sam_udi_imuPreintegration __log:=/home/xchu/.ros/log/b1e2ffcc-63d9-11ec-8731-1065304bd5b6/lio_sam_udi_imuPreintegration-1.log].
log file: /home/xchu/.ros/log/b1e2ffcc-63d9-11ec-8731-1065304bd5b6/lio_sam_udi_imuPreintegration-1*.log
[lio_sam_udi_imuPreintegration-1] restarting process
process[lio_sam_udi_imuPreintegration-1]: started with pid [4240]
[ INFO] [1640275367.369310875]: ----> IMU Preintegration Started.
system initialization.
currlidar pose: 17.2032814025879, -30.5312538146973, -48.5863075256348
currGPS position: 13.8068151473999, 7.92838048934937, 5.14798021316528
currGPS cov: 11.0224, 11.0224, 176.3584
terminate called after throwing an instance of 'gtsam::IndeterminantLinearSystemException'
  what():
Indeterminant linear system detected while working near variable
7061644215716937728 (Symbol: b0).

Thrown when a linear system is ill-posed.  The most common cause for this
error is having underconstrained variables.  Mathematically, the system is
underdetermined.  See the GTSAM Doxygen documentation at
http://borg.cc.gatech.edu/ on gtsam::IndeterminantLinearSystemException for
more information.
[lio_sam_udi_imuPreintegration-1] process has died [pid 4240, exit code -6, cmd /home/xchu/udicreator_ws/devel/lib/lio_sam_udi/lio_sam_udi_imuPreintegration __name:=lio_sam_udi_imuPreintegration __log:=/home/xchu/.ros/log/b1e2ffcc-63d9-11ec-8731-1065304bd5b6/lio_sam_udi_imuPreintegration-1.log].
log file: /home/xchu/.ros/log/b1e2ffcc-63d9-11ec-8731-1065304bd5b6/lio_sam_udi_imuPreintegration-1*.log
[lio_sam_udi_imuPreintegration-1] restarting process
process[lio_sam_udi_imuPreintegration-1]: started with pid [4268]
[ INFO] [1640275367.905634176]: ----> IMU Preintegration Started.
currlidar pose: 16.9986457824707, -29.6455020904541, -48.5315475463867
currGPS position: 13.9584197998047, 8.71054935455322, 4.71397876739502
currGPS cov: 11.0224, 11.0224, 176.3584
system initialization.
terminate called after throwing an instance of 'gtsam::IndeterminantLinearSystemException'
  what():
Indeterminant linear system detected while working near variable
7061644215716937728 (Symbol: b0).

Thrown when a linear system is ill-posed.  The most common cause for this
error is having underconstrained variables.  Mathematically, the system is
underdetermined.  See the GTSAM Doxygen documentation at
http://borg.cc.gatech.edu/ on gtsam::IndeterminantLinearSystemException for
more information.
[lio_sam_udi_imuPreintegration-1] process has died [pid 4268, exit code -6, cmd /home/xchu/udicreator_ws/devel/lib/lio_sam_udi/lio_sam_udi_imuPreintegration __name:=lio_sam_udi_imuPreintegration __log:=/home/xchu/.ros/log/b1e2ffcc-63d9-11ec-8731-1065304bd5b6/lio_sam_udi_imuPreintegration-1.log].
log file: /home/xchu/.ros/log/b1e2ffcc-63d9-11ec-8731-1065304bd5b6/lio_sam_udi_imuPreintegration-1*.log
[lio_sam_udi_imuPreintegration-1] restarting process
process[lio_sam_udi_imuPreintegration-1]: started with pid [4310]
[ INFO] [1640275368.605006835]: ----> IMU Preintegration Started.
system initialization.
currlidar pose: 16.4861240386963, -28.6083831787109, -48.5112152099609
currGPS position: 13.9197864532471, 9.7651309967041, 4.2459774017334
currGPS cov: 11.0224, 11.0224, 176.3584
terminate called after throwing an instance of 'gtsam::IndeterminantLinearSystemException'
  what():
Indeterminant linear system detected while working near variable
7061644215716937728 (Symbol: b0).

Thrown when a linear system is ill-posed.  The most common cause for this
error is having underconstrained variables.  Mathematically, the system is
underdetermined.  See the GTSAM Doxygen documentation at
http://borg.cc.gatech.edu/ on gtsam::IndeterminantLinearSystemException for
more information.
[lio_sam_udi_imuPreintegration-1] process has died [pid 4310, exit code -6, cmd /home/xchu/udicreator_ws/devel/lib/lio_sam_udi/lio_sam_udi_imuPreintegration __name:=lio_sam_udi_imuPreintegration __log:=/home/xchu/.ros/log/b1e2ffcc-63d9-11ec-8731-1065304bd5b6/lio_sam_udi_imuPreintegration-1.log].
log file: /home/xchu/.ros/log/b1e2ffcc-63d9-11ec-8731-1065304bd5b6/lio_sam_udi_imuPreintegration-1*.log

data  path:
path:        hkust_20201105full_correct2.bag
version:     2.0
duration:    28:19s (1699s)
start:       Nov 05 2020 14:50:04.53<PHONE_NUMBER>.53)
end:         Nov 05 2020 15:18:24.35<PHONE_NUMBER>.35)
size:        13.2 GB
messages:    229287
compression: none [14808/14808 chunks]
types:       geometry_msgs/TwistStamped  [98d34b0043a2093cf9d9345ab6eef12e]
             sensor_msgs/CompressedImage [8f7a12909da2c9d3332d540a0977563f]
             sensor_msgs/Imu             [6a62c6daae103f4ff57a132d6f95cec2]
             sensor_msgs/NavSatFix       [2d3a8cd499b9b4a0249fb98fd05cfa48]
             sensor_msgs/PointCloud2     [1158d486dd51d683ce2f1be655c3c181]
             sensor_msgs/TimeReference   [fded64a0265108ba86c3d38fb11c0c16]
topics:      /fix                                    8494 msgs    : sensor_msgs/NavSatFix
             /imu/data                             169972 msgs    : sensor_msgs/Imu
             /stereo/left/image_color/compressed    16998 msgs    : sensor_msgs/CompressedImage
             /time_reference                         8494 msgs    : sensor_msgs/TimeReference
             /vel                                    8357 msgs    : geometry_msgs/TwistStamped
             /velodyne_points                       16972 msgs    : sensor_msgs/PointCloud2

I recorded a vedio : vedio

my yaml file:
lio_sam_udi:

  # Topics
  pointCloudTopic: "/velodyne_points"                  #"points_raw"               # Point cloud data
  imuTopic: "/imu/data"                  #"imu_raw"                         # IMU data
  odomTopic: "odometry/imu"                   # IMU pre-preintegration odometry, same frequency as IMU
  gpsTopic: "gps_odom"

  use_localmap: 0  # 1 true 使用点云地图原点, 否则使用车辆运动的起点作为地图原点

  # Frames
  lidarFrame: "base_link"
  baselinkFrame: "base_link"
  odometryFrame: "odom"
  mapFrame: "map"

  # GPS Settings
  useGPS: true
  useImuHeadingInitialization: false          # if using GPS data, set to "true"
  useGpsElevation: true                      # if GPS elevation is bad, set to "false"
  gpsCovThreshold: 12.98                        # m^2, threshold for using GPS data
  poseCovThreshold: 12.98                      # m^2, threshold for using GPS data

  # Export settings
  savePCD: true                              # https://github.com/TixiaoShan/LIO-SAM/issues/3
  savePCDDirectory: "/home/xchu/xchujwu_slam/liosam_data/"        # in your home folder, starts and ends with "/". Warning: the code deletes "LOAM" folder then recreates it. See "mapOptimization" for implementation

  # Sensor Settings
  sensor: "velodyne"                      # lidar sensor type, either 'velodyne' or 'ouster' or 'hesai'
  N_SCAN: 16                                  # number of lidar channel (i.e., 16, 32, 64, 128)
  Horizon_SCAN: 1800                          # lidar horizontal resolution (Velodyne:1800, Ouster:512,1024,2048)
  downsampleRate: 1                           # default: 1. Downsample your data if too many points. i.e., 16 = 64 / 4, 16 = 16 / 1
  lidarMinRange: 1.5                          # default: 1.0, minimum lidar range to be used
  lidarMaxRange: 80.0                       # default: 1000.0, maximum lidar range to be used

  # IMU Settings
  imuFrequence: 100
  imuAccNoise: 3.9939570888238808e-03
  imuGyrNoise: 1.5636343949698187e-03
  imuAccBiasN: 6.4356659353532566e-05
  imuGyrBiasN: 3.5640318696367613e-05
  imuGravity: 9.80511
  imuRPYWeight: 0.01

  # Extrinsics (lidar -> IMU)
  extrinsicTrans: [ 0.0, 0.0, -0.06 ]
  extrinsicRot: [ 1, 0, 0,
                  0, 1, 0,
                  0, 0, 1 ]
  extrinsicRPY: [ 1, 0, 0,
                  0, 1, 0,
                  0, 0, 1 ]

  # LOAM feature threshold
  edgeThreshold: 1.0
  surfThreshold: 0.1
  edgeFeatureMinValidNum: 10
  surfFeatureMinValidNum: 100

  # voxel filter paprams
  odometrySurfLeafSize: 0.4                     # default: 0.4 - outdoor, 0.2 - indoor
  mappingCornerLeafSize: 0.2                    # default: 0.2 - outdoor, 0.1 - indoor
  mappingSurfLeafSize: 0.4                      # default: 0.4 - outdoor, 0.2 - indoor

  # robot motion constraint (in case you are using a 2D robot)
  z_tollerance: 1000                            # meters
  rotation_tollerance: 1000                     # radians

  # CPU Params
  numberOfCores: 12                              # number of cores for mapping optimization
  mappingProcessInterval: 0.10                  # seconds, regulate mapping frequency

  # Surrounding map
  surroundingkeyframeAddingDistThreshold: 2.0   # meters, regulate keyframe adding threshold
  surroundingkeyframeAddingAngleThreshold: 0.2  # radians, regulate keyframe adding threshold
  surroundingKeyframeDensity: 2.0               # meters, downsample surrounding keyframe poses
  surroundingKeyframeSearchRadius: 30.0         # meters, within n meters scan-to-map optimization (when loop closure disabled)

  # Loop closure
  loopClosureEnableFlag: true
  loopClosureFrequency: 1.0                     # Hz, regulate loop closure constraint add frequency
  surroundingKeyframeSize: 30                   # submap size (when loop closure enabled)
  historyKeyframeSearchRadius: 30.0             # meters, key frame that is within n meters from current pose will be considerd for loop closure
  historyKeyframeSearchTimeDiff: 30.0           # seconds, key frame that is n seconds older will be considered for loop closure
  historyKeyframeSearchNum: 20                  # number of hostory key frames will be fused into a submap for loop closure
  historyKeyframeFitnessScore: 0.8             # icp threshold, the smaller the better alignment
  loop_method: 1
  registeration_type: 0                        # 0 icp 1 ndt_omp

  scDistThres: 0.4
  scMaximumRadius: 60

  # Visualization
  globalMapVisualizationSearchRadius: 1500.0    # meters, global map visualization radius
  globalMapVisualizationPoseDensity: 2.0       # meters, global map visualization keyframe density
  globalMapVisualizationLeafSize: 1.0           # meters, global map visualization cloud density

  # #Navsat (convert GPS coordinates to Cartesian)
  # navsat:
  #   frequency: 50
  #   wait_for_datum: false
  #   delay: 0.0
  #   magnetic_declination_radians: 0
  #   yaw_offset: 0
  #   zero_altitude: true
  #   broadcast_utm_transform: false
  #   broadcast_utm_transform_as_parent_frame: false
  #   publish_filtered_gps: false

  # #EKF for Navsat
  # ekf_gps:
  #   publish_tf: false
  #   map_frame: map
  #   odom_frame: odom
  #   base_link_frame: base_link
  #   world_frame: odom

  #   frequency: 50
  #   two_d_mode: false
  #   sensor_timeout: 0.01
  # -------------------------------------
  # External IMU:
  #-------------------------------------
  # imu0: imu_correct
  # # make sure the input is aligned with ROS REP105. "imu_correct" is manually transformed by myself. EKF can also transform the data using tf between your imu and base_link
  # imu0_config: [false, false, false,
  #               true,  true,  true,
  #               false, false, false,
  #               false, false, true,
  #               true,  true,  true]
  # imu0_differential: false
  # imu0_queue_size: 50
  # imu0_remove_gravitational_acceleration: true
  # # -------------------------------------
  # # Odometry (From Navsat):
  # # -------------------------------------
  # odom0: odometry/gps
  # odom0_config: [true,  true,  true,
  #                false, false, false,
  #                false, false, false,
  #                false, false, false,
  #                false, false, false]
  # odom0_differential: false
  # odom0_queue_size: 10

  # #                            x     y     z     r     p     y   x_dot  y_dot  z_dot  r_dot p_dot y_dot x_ddot y_ddot z_ddot
  # process_noise_covariance: [  1.0,  0,    0,    0,    0,    0,    0,     0,     0,     0,    0,    0,    0,    0,      0,
  #                              0,    1.0,  0,    0,    0,    0,    0,     0,     0,     0,    0,    0,    0,    0,      0,
  #                              0,    0,    10.0, 0,    0,    0,    0,     0,     0,     0,    0,    0,    0,    0,      0,
  #                              0,    0,    0,    0.03, 0,    0,    0,     0,     0,     0,    0,    0,    0,    0,      0,
  #                              0,    0,    0,    0,    0.03, 0,    0,     0,     0,     0,    0,    0,    0,    0,      0,
  #                              0,    0,    0,    0,    0,    0.1,  0,     0,     0,     0,    0,    0,    0,    0,      0,
  #                              0,    0,    0,    0,    0,    0,    0.25,  0,     0,     0,    0,    0,    0,    0,      0,
  #                              0,    0,    0,    0,    0,    0,    0,     0.25,  0,     0,    0,    0,    0,    0,      0,
  #                              0,    0,    0,    0,    0,    0,    0,     0,     0.04,  0,    0,    0,    0,    0,      0,
  #                              0,    0,    0,    0,    0,    0,    0,     0,     0,     0.01, 0,    0,    0,    0,      0,
  #                              0,    0,    0,    0,    0,    0,    0,     0,     0,     0,    0.01, 0,    0,    0,      0,
  #                              0,    0,    0,    0,    0,    0,    0,     0,     0,     0,    0,    0.5,  0,    0,      0,
  #                              0,    0,    0,    0,    0,    0,    0,     0,     0,     0,    0,    0,    0.01, 0,      0,
  #                              0,    0,    0,    0,    0,    0,    0,     0,     0,     0,    0,    0,    0,    0.01,   0,
  #                              0,    0,    0,    0,    0,    0,    0,     0,     0,     0,    0,    0,    0,    0,      0.015]

I wrote my own gps odometry node and use it in the updateInitialGuess() to align the GPS Coordinate with the LiO-system.
I add some codes in the first key frame.
  if (cloudKeyPoses3D->points.empty()) {
      std::cout << "[mapOptmization3]: Ignore first key frame: " << cloudInfo.imuAvailable << ", "
                << cloudInfo.odomAvailable << std::endl;

      if (useGPS) {
        gpsAvialble = false;
        while (!gpsQueue.empty()) {
          mutex_lock.lock();
          double off_time = timeLaserInfoCur - gpsQueue.front().header.stamp.toSec();
          if (off_time > 0.5) {
            gpsQueue.pop_front();
            mutex_lock.unlock();
          } else if (off_time < -0.5) {
            // ROS_ERROR("failed to algned gps and lidar time, %f", off_time);
            mutex_lock.unlock();
            break;
          } else {
            nav_msgs::Odometry msg = gpsQueue.front();
            gpsQueue.pop_front();
            mutex_lock.unlock();
            ROS_WARN(" gps lidar off time: %f ", off_time);

            double roll, pitch, yaw;
            tf::Matrix3x3(tf::Quaternion(msg.pose.pose.orientation.x,
                                         msg.pose.pose.orientation.y,
                                         msg.pose.pose.orientation.z,
                                         msg.pose.pose.orientation.w)).getRPY(roll, pitch, yaw);
            std::cout << "rpy: " << roll << "," << pitch << ", " << yaw << std::endl;

            if (!std::isnan(yaw)) {
              transGPS.setIdentity();
              transGPS = pcl::getTransformation(msg.pose.pose.position.x,
                                                msg.pose.pose.position.y,
                                                msg.pose.pose.position.z,
                                                roll,
                                                pitch,
                                                yaw);

             // save origin GPS LLA data
              transLLA.setIdentity();
              transLLA = Eigen::Vector3d(msg.pose.covariance[1], msg.pose.covariance[2], msg.pose.covariance[3]);
              gpsAvialble = true;

              Eigen::AngleAxisd rollAngle1(roll, Eigen::Vector3d::UnitX());
              Eigen::AngleAxisd pitchAngle1(pitch, Eigen::Vector3d::UnitY());
              Eigen::AngleAxisd yawAngle1(yaw, Eigen::Vector3d::UnitZ());
              Eigen::Quaterniond q_gps = yawAngle1 * pitchAngle1 * rollAngle1;

              Eigen::AngleAxisd rollAngle(cloudInfo.imuRollInit, Eigen::Vector3d::UnitX());
              Eigen::AngleAxisd pitchAngle(cloudInfo.imuPitchInit, Eigen::Vector3d::UnitY());
              Eigen::AngleAxisd yawAngle(cloudInfo.imuYawInit, Eigen::Vector3d::UnitZ());
              Eigen::Quaterniond q_imu = yawAngle * pitchAngle * rollAngle;

              Eigen::Matrix3d rotation_matrix_imu = q_imu.toRotationMatrix();
              Eigen::Matrix3d rotation_matrix_gps = q_gps.toRotationMatrix();

              std::cout << "gps: " << transGPS.matrix() << std::endl;
              std::cout << "gps LLA: " << std::setprecision(15) << transLLA.matrix() << std::endl;
              // std::cout << "befroe: " << rotation_matrix_imu << ", " << std::endl;

              // Eigen::Affine3f transIMURaw =
              //  pcl::getTransformation(0, 0, 0, cloudInfo.imuRollInit, cloudInfo.imuPitchInit, cloudInfo.imuYawInit);
              //Eigen::Affine3f transIMUAft = transGPS.block<3, 3>(0, 0) * transIMURaw.inverse();
              // Eigen::Matrix3d transIMUAft = rotation_matrix_gps * rotation_matrix_imu.inverse();
              Eigen::Matrix3d transIMUAft = rotation_matrix_gps.inverse() * rotation_matrix_imu;
              Eigen::Vector3d eulerAngle2 = transIMUAft.eulerAngles(0, 1, 2); // roll,pitch,yaw
              transformTobeMapped[0] = eulerAngle2[0];
              transformTobeMapped[1] = eulerAngle2[1];
              transformTobeMapped[2] = eulerAngle2[2];
//              transformTobeMapped[0] = 0;
//              transformTobeMapped[1] = 0;
//              transformTobeMapped[2] = yaw;

              std::cout << "after: " << eulerAngle2[2] << ", " << std::endl;

              if (!useImuHeadingInitialization)
                transformTobeMapped[2] = 0;

              lastImuTransformation = pcl::getTransformation(0,
                                                             0,
                                                             0,
                                                             eulerAngle2[0],
                                                             eulerAngle2[1],
                                                             eulerAngle2[2]); // save imu before return;
//                                                             0,
//                                                             0,
//                                                             yaw); // save imu before return;

              system_init = true;
              ROS_WARN("GPS init success");
            }
            else{
              ROS_ERROR("GPS NAN, waiting for a better yaw");
            }
            break;
          }
        }
      } else {
        transformTobeMapped[0] = cloudInfo.imuRollInit;
        transformTobeMapped[1] = cloudInfo.imuPitchInit;
        transformTobeMapped[2] = cloudInfo.imuYawInit;

        if (!useImuHeadingInitialization)
          transformTobeMapped[2] = 0;

        lastImuTransformation = pcl::getTransformation(0,
                                                       0,
                                                       0,
                                                       cloudInfo.imuRollInit,
                                                       cloudInfo.imuPitchInit,
                                                       cloudInfo.imuYawInit); // save imu before return;

        system_init = true;
      }

      return;
    }

    if (!system_init) {
      ROS_WARN("sysyem need to be inited");
      return;
    }

besides,  i use the gps factor,
void addGPSFactor() {
    if (gpsQueue.empty())
      return;

    // wait for system initialized and settles down
    if (cloudKeyPoses3D->points.empty())
      return;
    else {
      if (pointDistance(cloudKeyPoses3D->front(), cloudKeyPoses3D->back()) < 5.0)
        return;
    }

    // pose covariance small, no need to correct
//    if (poseCovariance(3, 3) < poseCovThreshold && poseCovariance(4, 4) < poseCovThreshold)
//      return;

    // last gps position
    static PointType lastGPSPoint;

    while (!gpsQueue.empty()) {
      mutex_lock.lock();
      if (gpsQueue.front().header.stamp.toSec() < timeLaserInfoCur - 0.1) {
        // message too old
        gpsQueue.pop_front();
        mutex_lock.unlock();
      } else if (gpsQueue.front().header.stamp.toSec() > timeLaserInfoCur + 0.1) {
        // message too new
        mutex_lock.unlock();
        break;
      } else {
        nav_msgs::Odometry thisGPS = gpsQueue.front();
        gpsQueue.pop_front();
        mutex_lock.unlock();

        // GPS too noisy, skip
        float noise_x = thisGPS.pose.covariance[0];
        float noise_y = thisGPS.pose.covariance[7];
        float noise_z = thisGPS.pose.covariance[14];
        //if (noise_x > gpsCovThreshold || noise_y > gpsCovThreshold)
        if (noise_x > gpsCovThreshold || noise_y > gpsCovThreshold)
          continue;

        float gps_x = thisGPS.pose.pose.position.x;
        float gps_y = thisGPS.pose.pose.position.y;
        float gps_z = thisGPS.pose.pose.position.z;
        //if (!useGpsElevation) {
        //  gps_z = transformTobeMapped[5];
        //  noise_z = 0.01;
        // }

        //ROS_WARN(" gps lidar off time: %f ", off_time);
        std::cout << "currlidar pose: " << transformTobeMapped[3] << ", " << transformTobeMapped[4] << ", "
                  << transformTobeMapped[5] << std::endl;
        std::cout << "currGPS position: " << gps_x << ", "<< gps_y << ", "<< gps_z << std::endl;
        std::cout << "currGPS cov: " << thisGPS.pose.covariance[0] << ", " << thisGPS.pose.covariance[7]
                  << ", " << thisGPS.pose.covariance[14] << std::endl;

        // GPS not properly initialized (0,0,0)
        if (abs(gps_x) < 1e-6 && abs(gps_y) < 1e-6)
          continue;

        // Add GPS every a few meters
        PointType curGPSPoint;
        curGPSPoint.x = gps_x;
        curGPSPoint.y = gps_y;
        curGPSPoint.z = gps_z;
        if (pointDistance(curGPSPoint, lastGPSPoint) < 5.0)
          continue;
        else
          lastGPSPoint = curGPSPoint;

        gtsam::Vector Vector3(3);
        Vector3 << max(noise_x, 1.0f), max(noise_y, 1.0f), max(noise_z, 1.0f);
        noiseModel::Diagonal::shared_ptr gps_noise = noiseModel::Diagonal::Variances(Vector3);
        gtsam::GPSFactor gps_factor(cloudKeyPoses3D->size(), gtsam::Point3(gps_x, gps_y, gps_z), gps_noise);
        gtSAMgraph.add(gps_factor);

         aLoopIsClosed = true;
        break;
      }
    }
  }

when encounting small loops, the map optimization node runs well and lidar poses are also well constarined by gps data.

As you have pointed out, the GPS covariance should be reviewed against the smaller set that worked. As the GPS part of the code becomes robust, these problems can handled.  What I have concluded is that for us, we need to start the system using magnetic direction, since the project rarely provides good GPS coverage and time delay for fixed position.  Furthermore, I will correct the vertical on the trajectory with survey ground points.

Review covariance  z
For online SLAM,  tougher problem, would require prior knowledge of starting xy, and z. Maybe closure can be done with post processing and substitute and average z value.  3. Modifying the code to assist with the decision would be necessary to contrain the z.
Also,  filter the GPS and provide a moving average of fixed positions like ekf to achieve a closure. Then, post process as necessary to coverage to a better solution.

I am working on the items now. As for additional information, the author provided the email to the person that assisted the that portion of the code.
Keep posting on how it resolves on shorter drives.  Maybe, fuse vertical checks with imagery for short correction correlation.

As you have pointed out, the GPS covariance should be reviewed against the smaller set that worked. As the GPS part of the code becomes robust, these problems can handled. What I have concluded is that for us, we need to start the system using magnetic direction, since the project rarely provides good GPS coverage and time delay for fixed position. Furthermore, I will correct the vertical on the trajectory with survey ground points.

Review covariance  z
For online SLAM,  tougher problem, would require prior knowledge of starting xy, and z. Maybe closure can be done with post processing and substitute and average z value.  3. Modifying the code to assist with the decision would be necessary to contrain the z.
Also,  filter the GPS and provide a moving average of fixed positions like ekf to achieve a closure. Then, post process as necessary to coverage to a better solution.

I am working on the items now. As for additional information, the author provided the email to the person that assisted the that portion of the code.
Keep posting on how it resolves on shorter drives. Maybe, fuse vertical checks with imagery for short correction correlation.

Thanks for your reply!
when we do not use gps, the origin version of LIO-SAM can not close loop well and the points map appears to be layered, but the gtsam runs well with the dataset i provided above.

To solve this problem,  i try to add some other loop detection methods to enhance the radius search methods, such as ScanContext, the final big loop can be successfully detetced, but gtsam crashed again at almost the same location as the previous problem caused by gps.
Could it be that the loop constraint is too strong, causing the entire factor graph to collapse?Maybe it's the  problem caused by gtsam？

gtsam::IndeterminantLinearSystemException
If you expanded the radius for loop closure, then your covariance settings in LIO-SAM are excluding the elevations; therefore, the indeterminate system error. That is my guess. Try to change the z covariance to an unrealistic number, then decrease until the best result is obtained. The up and down jitter definitely can be the gps value held over the computed or estimated z.
If you are having system resource latency, then downsample the cloud in LIO SAM and change the ICP value. All these items can be done with the bag file. Keep changing the params until the result is satisfactory.
Also, try enough test runs with the live system over clear skies and short runs to confirm the settings. Maybe slow speed on a few runs will help confirm the system resources and gps epochs to give a better result. Tall buildings (city) will experience multipath confusing most gps systems.

Did you get your problem solved? I have a short drive that is uploaded with GPS without RTK corrections at this point. #312

Did you get your problem solved? I have a short drive that is uploaded with GPS without RTK corrections at this point. #312

I did not solve the problem of GPS fusion on this data, but I re-collected the data of the same scene, and made sure that the GPS data in the scene was relatively reliable, and there were some small closed loops in the scene, and finally got a new map successfully.

This is my experiment result, ouster128+6axis-stim300 imu+normal GPS

I evaluated the performance and accuracy of various LIO algorithms on this scene, including fastlio2, lio-mapping, and fasterlio, and the datasets include handheld devices and car collections. It is found that LIO SAM is very stable when the overall data scene is highly balanced, but when the downhill speed is fast, there is a great probability of failure directly, which should be caused by the pre-integration node, causing the entire system to hang up. A very impressive scene is that when our car collects data, we need to go down the next step to the road. This step will lead to a large change in height and eventually cause the system to hang.

如果您遇到系统资源延迟，请在 LIO SAM 中对云进行下采样并更改 ICP 值。所有这些项目都可以通过包文件完成。不断更改参数，直到结果令人满意。

I evaluated the performance and accuracy of various LIO algorithms on this scene, including fastlio2, lio-mapping, and fasterlio, and the datasets include handheld devices and car collections. It is found that LIO SAM is very stable when the overall data scene is highly balanced, but when the downhill speed is fast, there is a great probability of failure directly, which should be caused by the pre-integration node, causing the entire system to hang up. A very impressive scene is that when our car collects data, we need to go down the next step to the road. This step will lead to a large change in height and eventually cause the system to hang.

gtsam::IndeterminantLinearSystemException
If you expanded the radius for loop closure, then your covariance settings in LIO-SAM are excluding the elevations; therefore, the indeterminate system error. That is my guess. Try to change the z covariance to an unrealistic number, then decrease until the best result is obtained. The up and down jitter definitely can be the gps value held over the computed or estimated z. If you are having system resource latency, then downsample the cloud in LIO SAM and change the ICP value. All these items can be done with the bag file. Keep changing the params until the result is satisfactory.
Also, try enough test runs with the live system over clear skies and short runs to confirm the settings. Maybe slow speed on a few runs will help confirm the system resources and gps epochs to give a better result. Tall buildings (city) will experience multipath confusing most gps systems.

gtsam::IndeterminantLinearSystemException
If you expanded the radius for loop closure, then your covariance settings in LIO-SAM are excluding the elevations; therefore, the indeterminate system error. That is my guess. Try to change the z covariance to an unrealistic number, then decrease until the best result is obtained. The up and down jitter definitely can be the gps value held over the computed or estimated z. If you are having system resource latency, then downsample the cloud in LIO SAM and change the ICP value. All these items can be done with the bag file. Keep changing the params until the result is satisfactory.
Also, try enough test runs with the live system over clear skies and short runs to confirm the settings. Maybe slow speed on a few runs will help confirm the system resources and gps epochs to give a better result. Tall buildings (city) will experience multipath confusing most gps systems.

I checked the error output log of the terminal again, and found that the pre-integration node should be the first to die when the system crashes, without any specific error type prompt. After that, the pre-integration node has been restarted, causing the back-end optimization node to raise gtsam::IndeterminantLinearSystemException

gtsam::IndeterminantLinearSystemException
If you expanded the radius for loop closure, then your covariance settings in LIO-SAM are excluding the elevations; therefore, the indeterminate system error. That is my guess. Try to change the z covariance to an unrealistic number, then decrease until the best result is obtained. The up and down jitter definitely can be the gps value held over the computed or estimated z. If you are having system resource latency, then downsample the cloud in LIO SAM and change the ICP value. All these items can be done with the bag file. Keep changing the params until the result is satisfactory.
Also, try enough test runs with the live system over clear skies and short runs to confirm the settings. Maybe slow speed on a few runs will help confirm the system resources and gps epochs to give a better result. Tall buildings (city) will experience multipath confusing most gps systems.

I checked the error output log of the terminal again, and found that the pre-integration node should be the first to die when the system crashes, without any specific error type prompt. After that, the pre-integration node has been restarted, causing the back-end optimization node to raise gtsam::IndeterminantLinearSystemException

have you solved this problem already？I also met this problem，could u tell me how to deal with. thanks for your kindness.

gtsam::IndeterminantLinearSystemException
If you expanded the radius for loop closure, then your covariance settings in LIO-SAM are excluding the elevations; therefore, the indeterminate system error. That is my guess. Try to change the z covariance to an unrealistic number, then decrease until the best result is obtained. The up and down jitter definitely can be the gps value held over the computed or estimated z. If you are having system resource latency, then downsample the cloud in LIO SAM and change the ICP value. All these items can be done with the bag file. Keep changing the params until the result is satisfactory.
Also, try enough test runs with the live system over clear skies and short runs to confirm the settings. Maybe slow speed on a few runs will help confirm the system resources and gps epochs to give a better result. Tall buildings (city) will experience multipath confusing most gps systems.

I checked the error output log of the terminal again, and found that the pre-integration node should be the first to die when the system crashes, without any specific error type prompt. After that, the pre-integration node has been restarted, causing the back-end optimization node to raise gtsam::IndeterminantLinearSystemException

have you solved this problem already？I also met this problem，could u tell me how to deal with. thanks for your kindness.

I have solved this problem, I found that the covariance setting of GPS Factor requires some experience tuning, such as its weight level with odom factor.

gtsam::IndeterminantLinearSystemException
If you expanded the radius for loop closure, then your covariance settings in LIO-SAM are excluding the elevations; therefore, the indeterminate system error. That is my guess. Try to change the z covariance to an unrealistic number, then decrease until the best result is obtained. The up and down jitter definitely can be the gps value held over the computed or estimated z. If you are having system resource latency, then downsample the cloud in LIO SAM and change the ICP value. All these items can be done with the bag file. Keep changing the params until the result is satisfactory.
Also, try enough test runs with the live system over clear skies and short runs to confirm the settings. Maybe slow speed on a few runs will help confirm the system resources and gps epochs to give a better result. Tall buildings (city) will experience multipath confusing most gps systems.

I checked the error output log of the terminal again, and found that the pre-integration node should be the first to die when the system crashes, without any specific error type prompt. After that, the pre-integration node has been restarted, causing the back-end optimization node to raise gtsam::IndeterminantLinearSystemException

have you solved this problem already？I also met this problem，could u tell me how to deal with. thanks for your kindness.

I have solved this problem, I found that the covariance setting of GPS Factor requires some experience tuning, such as its weight level with odom factor.

Thank you very much for your reply. I know the cause of the problem. Can you describe the optimization steps of GPS Factor  in detail

gtsam::IndeterminantLinearSystemException
If you expanded the radius for loop closure, then your covariance settings in LIO-SAM are excluding the elevations; therefore, the indeterminate system error. That is my guess. Try to change the z covariance to an unrealistic number, then decrease until the best result is obtained. The up and down jitter definitely can be the gps value held over the computed or estimated z. If you are having system resource latency, then downsample the cloud in LIO SAM and change the ICP value. All these items can be done with the bag file. Keep changing the params until the result is satisfactory.
Also, try enough test runs with the live system over clear skies and short runs to confirm the settings. Maybe slow speed on a few runs will help confirm the system resources and gps epochs to give a better result. Tall buildings (city) will experience multipath confusing most gps systems.

I checked the error output log of the terminal again, and found that the pre-integration node should be the first to die when the system crashes, without any specific error type prompt. After that, the pre-integration node has been restarted, causing the back-end optimization node to raise gtsam::IndeterminantLinearSystemException

have you solved this problem already？I also met this problem，could u tell me how to deal with. thanks for your kindness.

I have solved this problem, I found that the covariance setting of GPS Factor requires some experience tuning, such as its weight level with odom factor.

Hi. I am having same issues with my GPS, can you tell more about what you did to solve it?


Buttons sometimes don't have touch feedback

Buttons sometimes don't have touch feedback when the background has been set to a color different than "initial".
Android KitKat:
☒ No touch feedback.
Android Lollipop:
☑ Touch feedback as expected.
iOS:
☒ Touch feedback only on the button text.
var page = tabris.create("Page", {
  title: "ProgressBar",
  topLevel: true
});

tabris.create("Button", {
  background: "red",
  text: "Just a button",
  layoutData: {centerX: 0, centerY: 0}
}).appendTo(page);

page.open();

This has been fixed.


Add slash commands support

This PR adds support for Discord's slash commands and Button interactions

This PR was not merged due to the author's inactivity


Refactored sleep delays to speed execution of parallelization tests. …

Refactored sleep delays to cut running time for the parallelization test classes.
Updated per comments from code review on my last pull request
Added support for tracking data provider information for a model execution in the state tracker
Added a sample class using a data provider
Updated some source comments on the base parallelization test class and some of the individual test classes to correspond to the refactored behavior from my last pull request
Refactored the simultaneous method verification code in the base parallelization test class to accurately reflect the intent of the  method.
Added a test class for the following scenario for test case PTP-TC-3 (Parallel by methods mode with sequential test suites using a non-parallel data provider but no dependencies and no factories)

Single suite with a single test consisting of a single test class with five methods with a data provider specifying 3 sets of data
    * Thread count and parallel mode are specified at the suite level
    * The thread count is equal to the number of test methods times 3, the number of times each method will be invoked with a data set from the data provider. Expectation is that only 5 threads will be spawned, one for each of the methods and that in this thread, the method will be invoked 3 times, once for each set of data from the data provider.
    * There are NO configuration methods
    * All test methods pass
    * NO ordering is specified
    * group-by-instances is NOT set
    * There are no method exclusions

Added the new test class to testng.xml

:+1:


Fix parsing of label names which are also keywords

The current separation between lexer and parser is a bit fuzzy when it
comes to operators, aggregators and other keywords. The lexer already
tries to determine the type of a token, even though that type might
change depending on the context.
This led to the problematic behavior that no tokens known to the lexer
could be used as label names, including operators (and, by, ...),
aggregators (count, quantile, ...) or other keywords (for, offset, ...).
This change validates a token value again whether it can be used as a
label. We might want to check whether the specific item identification
should be moved from the lexer to the parser.
@fabxc @beorn7 @brian-brazil

Let me know if you want to release this as 1.1.x and I'll change the target branch.

We fixed this a long time ago for label names in vector selectors (within curly braces). There we went for the solution of having a separate lexer state within curly braces that lexes tokens differently: https://github.com/prometheus/prometheus/blob/master/promql/lex.go#L562
The other general approach is to do less token differentiation in the lexer and do the higher-level interpretation in the parser.
Your approach is a mix which I find a bit unclean: do some differentiation in the lexer, but throw the differentiated token type away if it's in a context where it could be a label name.
Generally the most consistent approach would be to have that extra lexer state similar to what we have for curly braces. If that gets too complicated (you'd have to treat keywords by, without, on, ignoring, group_left, group_right as special in the lexer and enter a new lexer state within their parentheses - basically what is happening right now in the parser instead), then keeping the token totally opaque in the lexer and doing all the higher-level interpretation in the parser is probably the other good alternative. I find it kind of nice to have an understanding of keywords at the lexer level though... so I have a slight preference for the former approach I guess.
Let's see what @fabxc says :)

We fixed this a long time ago for label names in vector selectors (within curly braces). There we went for the solution of having a separate lexer state within curly braces that lexes tokens differently: https://github.com/prometheus/prometheus/blob/master/promql/lex.go#L562

I saw that, but that only works well because curly braces appear only in one context in our grammar and can't be nested.

The other general approach is to do less token differentiation in the lexer and do the higher-level interpretation in the parser.

That was my expectation and would clean things up.

Your approach is a mix which I find a bit unclean: do some differentiation in the lexer, but throw the differentiated token type away if it's in a context where it could be a label name.

I don't find this solution elegant either, but it's bug which started to hurt a lot with the introduction of the quantile aggregator (breaking any aggregation of summaries on the reserved quantile label). I'm not really sure the problem here is my approach or the artifact of the current implementation.

Generally the most consistent approach would be to have that extra lexer state similar to what we have for curly braces. If that gets too complicated (you'd have to treat keywords by, without, on, ignoring, group_left, group_right as special in the lexer and enter a new lexer state within their parentheses - basically what is happening right now in the parser instead), then keeping the token totally opaque in the lexer and doing all the higher-level interpretation in the parser is probably the other good alternative. I find it kind of nice to have an understanding of keywords at the lexer level though... so I have a slight preference for the former approach I guess.

This doesn't match my definition of a lexer, which should operate on a regular grammar without state. I don't find this a consistent approach as it heavily blurs the separation of lexer and parser. We can throw that away of course, but I'd rather move up the classification of PromQL specific items (e.g. operator, aggregator, keyword) into the parser which has the state to accurately identify them.

I'm not sure if the nested nature of parentheses would be problematic, as it isn't the parens that trigger the new lexer state, but certain keywords that happen to also be followed by parens (by (...), without (...), etc.). One would only have to track the non-nested parens after those keywords.
States in lexers are normal. Check out for example https://golang.org/src/text/template/parse/lex.go, which makes liberal use of states. The only question is how "deep" the state analysis goes. In our case here it's kind of at the border where it could be in the lexer or in the parser. Exactly for this case (returning individual keywords vs. just telling the parser that it's a keyword and having the parser figure it out), there are both variants in the wild for various programming languages.

Ehm sorry, not exactly our case (the thing about keywords in lexer vs. parsers), but anyways.
I think both approaches could be ok here. Let's see what @fabxc says.

Yeah, @fabxc is Mr. Parser and should enlighten us here.
And yes, this definitely has to go into release-1.1. If we don't consider this a bug fix, then the 1.1 release was in fact a breaking release and had to be renamed 2.0.

I'm running this on a SC production server right now.
Seems to be the right fix, at lest short-term.

Mh, I see the general problem. If we just had a generic itemIdentifier and would interpret it in the parser, this would certainly work just as well.
I'm wondering where this would stop though. We lex operators into their own items, e.g. itemLTE for <=. But other binary operators like and and unless would fall into the category of identifiers.
So now the parser has to check item types for certain operators and interpret plain strings for others. This seems rather ugly. To be consistent again, we could also just interpret <= etc. as identifiers – and now our lexer can almost be replaced with strings.Split(q, " ").
Now the parser not just has to ask for a list of valid things that follow. It has to look at the next thing, determine what it is, and then check whether it's in the list of valid things to follow.
Of course this could be hidden within further itemType.isX() methods.
Having it as a lexer state would require some amount of look-back, which we don't really want in there.
This seems to be a very comprehensive fix for an easy to understand case. I don't see any further problems occurring from the same problem in the future given that PromQL won't fundamentally evolve anymore.
👍 for merging this.
We have more pressuring issues to address w.r.t parsing, see #1595


fix bug with error display

Closes #92

PR-Codex overview
This PR changes the order of conditions in a ternary operator in LoginForm.tsx to fix an issue where an error message was not displayed correctly.
Detailed summary

Changed order of conditions in ternary operator in LoginForm.tsx.
Error message now displays correctly when loading is "failed" or isTokenValid is false.

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

Codecov Report
Patch coverage: 63.63% and project coverage change: -0.14 :warning:

Comparison is base (e9d8687) 71.18% compared to head (e6f8efd) 71.04%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
Additional details and impacted files
@@             Coverage Diff              @@
##             master      #93      +/-   ##
============================================
- Coverage     71.18%   71.04%   -0.14%
  Complexity      172      172
============================================
  Files            78       78
  Lines          1211     1219       +8
  Branches        103      103
============================================
+ Hits            862      866       +4
- Misses          254      258       +4
  Partials         95       95

Impacted Files
Coverage Δ

...in/src/main/client/src/store/reducers/authSlice.ts
60.31% <55.55%> (-2.19%)
:arrow_down:

qa-admin/src/main/client/src/http/api.ts
64.70% <100.00%> (+2.20%)
:arrow_up:

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Authenticate NSURLRequest

Summary
Authenticate NSURLRequest as specified by OAuth 2.0 guidelines for token type Bearer.
The Provider exposes a convenience method that, in case the available token is expired, refreshes the token and then uses it to authenticate any NSURLRequest.
Why
This feature give to the client an easy way to integrate a valid access token, retrieved via OAuth 2.0, in any NSURLRequest that the client might have created beforehand.
Note
To improve test cases accuracy I needed to add a dependency for the test target: OHTTPStubs.
I found the best way to do it via Carthage. To get started run the shell script in the bin folder.

[WIP] to make Travis CI work.
First time I'm dealing with this.

Thanks @fabiomassimo !
That's interesting but it makes too many assumptions for now... But that's definitely something I'll reconsider later. Sorry :/

Ok, when you'll have time I'd be happy to discuss with you these assumptions that you are concerned about such that maybe I can improve my PR.
Thank you.

SwiftyOAuth focuses on authorization and obtaining token.
There are other OAuth libs that provide authenticated requests (something along provider.get(myURL, callback: myCallback)) but I don't think it's a good design.
Moreover each provider has its own peculiarity: some ask you to pass the token in the query string, other in the header Bearer abc123. I once stumbled upon a provider that only recognized bearer abc123 (lowercase bearer)...
For these reasons I prefer to leave this part to the SwiftyOAuth user.

Thank you for the explanation. It makes totally sense.


Mixtral 8x7B smoothquant failed

System Info
p4de (4 80GB A100 GPUs)
Who can help?
@Tracin @byshiue
Information

[X] The official example scripts
[ ] My own modified scripts

Tasks

[X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction

Install 0.9.0 release
2.python ./llama/convert_checkpoint.py --model_dir ${MODEL_DIR} --output_dir ./ckpt --dtype float16 --tp_size 8 --workers 8 --smoothquant 0.5

Expected behavior
Checkpoint created
actual behavior
Traceback (most recent call last):

in convert_layer
qkv_weight = qkv_para[prefix + 'self_attn.qkv_proj']
KeyError: 'model.layers.0.self_attn.qkv_proj'

additional notes
n/a

@vnkc1 Mixtral MoE Smoothquant is not currently supported as noted in the precision support matrix

Thanks, is there a plan to support MoE Smoothquant?

It is not in our roadmap now. If you are interested, you could create a ticket to request it. Close this issue.


Use noarch: python

As this is become a proper Python package, convert it to noarch: python instead of noarch: generic.

If we are enabling patching: https://github.com/rapidsai/rapids-dask-dependency/pull/27
Though if you would prefer to add this change to your PR, that would be ok with me

In any event this already depends on Python as Dask does

The noarch: python change is already in #27: https://github.com/rapidsai/rapids-dask-dependency/pull/27/files#diff-de85bf9229578be6d0184f527af4dad8160b9189c03ec2f7e31c6d260629c797R14

The noarch: python change is already in https://github.com/rapidsai/rapids-dask-dependency/pull/27:

Right that's why I was confused, #27 is what adds Python code and it already makes this change.

In any event this already depends on Python as Dask does

That's fair, but that's not really how you would typically handle this, right? A package should specify based on its own contents (which as of this PR is still generic and not Python-specific). If its dependencies are more constrained then that will be handled by the package manager's dependency resolver. Of course, in this case the discussion is largely academic since the only purpose of this package is to specify Python package dependencies, but in terms of best practices I would still leave it to those dependencies to specify.

Going to close this as redundant to #27


Assessing Performance

TODOs left in draft

[ ] Animations

[ ] New: Animation showing conditional distributions
[ ] Training error curve
[ ] True error curve
[ ] Combined animation of train/true/test
[ ] Overfitting animation. Change w's to p's
[ ] Average predictor
[ ] Bias
[ ] Variance
[ ] Bias-variance tradeoff
[ ] Animating 2 points vs 100 points (doesn't actually have to be 100)

Closing in favor of #54


[ebs] section ignored in cluster creation

Hello I'm trying to create a cluster with a volume starting by a snapshot, but it looks like the whole [ebs ] section is ignored, the cluster is created succesfully but a new volume not based on the snapshot is created instead.
My configuration is the following

AWS ParallelCluster version: <IP_ADDRESS>
Full cluster configuration:

[aws]
aws_region_name = eu-west-1
[aliases]
ssh = ssh {CFN_USER}@{MASTER_IP} {ARGS}
[global]
cluster_template = default
update_check = true
sanity_check = true
[vpc default]
vpc_id = vpc-xxxxxx
master_subnet_id = subnet-xxxxxx
compute_subnet_id = subnet-xxxxxx
[cluster default]
key_name = aws-pk
scheduler = slurm
master_instance_type = r5.large
base_os = ubuntu2004
vpc_settings = default
queue_settings = compute
[ebs mesmartvol]
shared_dir = /mesmart
ebs_snapshot_id = snap-016bc356a7f298304
[queue compute]
enable_efa = false
enable_efa_gdr = false
compute_resource_settings = default
[compute_resource default]
instance_type = r5.2xlarge
NOTE: the source volume of snapshot has been deleted. I don't think this matters but may be an usefull info.

Found out the cause of the issue, which should be probably specified in the docs: under [cluster] section, ebs_settings must be specified when using ebs volumes, so the final conf file should be something like:
[cluster default]
...
ebs_settings = ebs_name
[ebs ebs_name]
ebs_snapshot_id = snap-xxxxxxxx


cp: use sparse copy on --sparse=always

Correct the behavior of cp --reflink=never --sparse=always so that it performs a sparse copy (Edit: on Linux). Before this commit, it was incorrectly performing a dense copy.
This should cause the GNU test suite file tests/cp/sparse.sh to pass.

side note, i love the bot. Many thanks @tertsdiepraam :)

@jfinkels would it be possible to have a test or it is too hard?
thanks

I have added a test

congrat :)


[Bug]: Tiling does not work with ReverseDistillation

Describe the bug
The attribute 'model' does not exist in the pl_module. Therefore information about the tiler are missing.
With other networks such as Padim the tiling works.
Dataset
Other (please specify in the text field below)
Model
Reverse Distillation
Steps to reproduce the behavior
tiler_config_callback = TilerConfigurationCallback(enable=True, tile_size=[256,256], stride=256)
folder_datamodule = Folder(
name="Name",
normal_dir="C:/Users/healthy",
abnormal_dir="C:/Users/unhealthy,
task=TaskType.CLASSIFICATION,
image_size = (512,512),
num_workers=0,
train_batch_size= 2,
eval_batch_size=2,
seed=40
)
model = ReverseDistillation()
engine = Engine(task=TaskType.CLASSIFICATION,
callbacks = [tiler_config_callback],
max_epochs = 200,
check_val_every_n_epoch= 1,
logger=logger)
output = engine.fit(model=model, datamodule=folder_datamodule)
OS information
OS information:

OS: Windows11
Python version: 3.10
Anomalib version: 1.0.1
GPU models and configuration: MX240
I'm using a custom dataset

Expected behavior
Tiling should work as it is working with other listed networks.
Screenshots
AttributeError: 'ReverseDistillation' object has no attribute 'model'
Pip/GitHub
GitHub
What version/branch did you use?
No response
Configuration YAML
n

Logs
File "Experiment1_TiledImages.py", line 50, in <module>
    output = engine.fit(model=model, datamodule=folder_datamodule)
  File "engine.py", line 533, in fit
    self._setup_trainer(model)
  File "engine.py", line 331, in _setup_trainer
    self._trainer = Trainer(**self._cache.args)
  File "argparse.py", line 70, in insert_env_defaults
    return fn(self, **kwargs)
  File "trainer.py", line 431, in __init__
    self._callback_connector.on_trainer_init(
  File "callback_connector.py", line 79, in on_trainer_init
    _validate_callbacks_list(self.trainer.callbacks)
  File "callback_connector.py", line 227, in _validate_callbacks_list
    stateful_callbacks = [cb for cb in callbacks if is_overridden("state_dict", instance=cb)]
  File "callback_connector.py", line 227, in <listcomp>
    stateful_callbacks = [cb for cb in callbacks if is_overridden("state_dict", instance=cb)]
  File "model_helpers.py", line 42, in is_overridden
    raise ValueError("Expected a parent")
ValueError: Expected a parent
PS> & python.exe Experiment1_TiledImages.py
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
No implementation of `configure_transforms` was provided in the Lightning model. Using default transforms from the base class. This may not be suitable for your use case. Please override `configure_transforms` in your model.
F1Score class exists for backwards compatibility. It will be removed in v1.1. Please use BinaryF1Score from torchmetrics instead
Traceback (most recent call last):
  File "Experiment1_TiledImages.py", line 50, in <module>
    output = engine.fit(model=model, datamodule=folder_datamodule)
  File "engine.py", line 540, in fit
    self.trainer.fit(model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
  File "trainer.py", line 544, in fit
    call._call_and_handle_interrupt(
  File "call.py", line 44, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "trainer.py", line 580, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "trainer.py", line 949, in _run
    call._call_setup_hook(self)  # allow user to set up LightningModule in accelerator environment
  File "call.py", line 93, in _call_setup_hook
    _call_callback_hooks(trainer, "setup", stage=fn)
  File "call.py", line 208, in _call_callback_hooks
    fn(trainer, trainer.lightning_module, *args, **kwargs)
  File "tiler_configuration.py", line 65, in setup
    if isinstance(pl_module, AnomalyModule) and hasattr(pl_module.model, "tiler"):
  File "module.py", line 1709, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'ReverseDistillation' object has no attribute 'model'

Code of Conduct

[X] I agree to follow this project's Code of Conduct

same issues


ISPN-1816 Re-adding an entry doesn't reset creation time when using life...

...spans
https://issues.jboss.org/browse/ISPN-1816
Also merge t_1816_51x for 5.1.x

Handling this...


Add option to increase parallel IKEv1 Phase 2 rekeys

In certain situations remote firewalls might issue a rekey for
all CHILD_SAs in parallel. At least Watchguard firewall clusters
have been seen flooding rekeys after they failover.
strongSwan usually only allows for 3 concurrent rekeys. Nevertheless
it already has the option max_ikev1_exchanges to increase that
value. This patch adds a setting to the advanced VPN dialogue
to control the behaviour of the parameter
More details at https://redmine.pfsense.org/issues/9331

[ ] Redmine Issue: https://redmine.pfsense.org/issues/NNNN
[ ] Ready for review

Fix added. Phase has been written in lower case to match the "Edit Phase 2" dialogue.

Fix added.


Database connection pooling for MySQL & PostgreSQL external DBs

Is your feature request related to a problem? Please describe.
Yes, currently seeing "Error rpc error: code = Unknown desc = Error 1040: Too many connections" in the UI:

Appears this is due to hitting the maximum number of database connections. Currently using an AWS RDS Aurora database (type db.t3.small) with max_connections = 100. Have 2 k3s servers, 0 agents in the cluster. Error frequently occurs when navigating to the local -> System project. Running Rancher v2.3.5.
Describe the solution you'd like
Would like to have a connection limit or database connection pooling implemented in k3s so that each cluster will have a predictable maximum number of DB connections open on the database server. May run multiple k3s clusters against a single RDS cluster and need to know the expected number of connections. If it makes sense, this max or pool size would be a configurable parameter with a reasonable limit.
Describe alternatives you've considered
Arbitrarily raising database max_connections on the server to avoid the error.
gz#8702

My suspicion is this is really a issue with rancher. Is rancher running on the DB backed k3s?
We can put in a conn limit but that does just mean things will be slower. It's probably better to fix Rancher. 2.3 is known to be overly chatty and has already been addressed in the dev versions of Rancher.

Yes, Rancher is running on the k3s being backed by RDS. Is it worth putting in an issue in rancher/rancher to have the overly chatty fixes backported to v2.3? Will likely move to v2.4 once it GAs.

I am having the same problem with RDS PostgreSQL backed k3s cluster. Removing Rancher did not help.
Also, this probably falls under Kine implementation now?

I am having the same problem with mysql backed k3s cluster.

I am also having this same problem, in a k3s cluster backed by mysql (v1.17.2+k3s1). From my experience in my cluster, it gets into a state when 2 of 3 masters are behaving fine, but the 3rd master starts going haywire, attempting to open thousands of connections (which looks to me that there is no upper bound for the connection pool).
The MySQL/MariaDB section on https://rancher.com/docs/k3s/latest/en/installation/datastore/ links to the docs for the go-sql-driver for mysql for advanced options: https://github.com/go-sql-driver/mysql#dsn-data-source-name
This in turn points to the the golang database/sql documentation for the connection pool settings, specifically, these settings:
https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime
https://golang.org/pkg/database/sql/#DB.SetMaxIdleConns
https://golang.org/pkg/database/sql/#DB.SetMaxOpenConns
It looks like the default behavior for database/sql is that MaxOpenConns is set to 0 (unlimited).

Also, this probably falls under Kine implementation now?

That was my assumption as well, as it likes like kine is the wrapper no matter what your cluster datastore backend is. And looking over the source code for kine, it doesn't look like there is anything that currently sets the parameters for the database/sql library.  So, essentially, the k3s service has no upper limit on how many connections it will try and open with the backend, and can lead to the database being overloaded if circumstances are just right. This lines up with the behavior I've seen my cluster exhibiting.
Not only would kine need to be adjusted to set these parameters, we would also need some way to influence the values from k3s as well.

I am seeing the same issues with using mysql.  In my case I have just 2 masters, and the number of connections opened result in mysql logging:
Aborted connection 2888824 to db: 'k3s_search' user: 'k8s' host: 'leader02' (Got an error reading communication packets)
MySQL version: mysql  Ver 14.14 Distrib 5.7.29, for Linux (x86_64)
k3s Cluster: v1.17.4+k3s1

I created a PR for kine here https://github.com/rancher/kine/pull/37 to tackle the issue.
When testing rancher installed on a k3s AWS instance I also stumbled upon this error messages - I used a fairly small AWS db.t2.micro instance for SQL which could not always handle displaying e.g. the workloads in the System project.
The PR is a proposal of how to make connection pooling actually configurable.
Locally I also re-compiled k3s with the proposed changes (had to add configurable cli flags in https://github.com/rancher/k3s/blob/master/pkg/cli/cmds/server.go to test this locally) and it worked! I limited the maximum connections and the error vanished 🎉
For this repo, when the PR in the kine project would be accepted, it should also expose the mentioned cli flags on the k3s server, I imagine something like this:

Flag
Default
Description

--datastore-max-idle-connections
2
Maximum number of idle connections used by datastore. If num <= 0, then no connections are retained

--datastore-max-open-connections
0(unlimited)
Maximum number of idle connections used by datastore. If num <= 0, then there is no limit

--datastore-connection-max-lifetime
1s
Maximum duration a connection is held alive. Desfined as a parsable string, e.g., 1s, 2500ms, and 1h30m are all accepted values

Maybe even make them configurable via environment variables?
I would create a PR for this repo as well with my changes for the server cmd, but since its depending on the kine project, I think the PR should be merged there first before I would create a PR for this project.
Cheers!

Experiencing the same. k3s 1.18 with AWS RDS MySQL as a datastore.

We are seeing this as well. k3s. mysql backend using a aws t2.micro rds.

Hello
I have the same issue using MySQL on another server (Debian 10 MariaDB for the two server).
Is there a workaround for this, because right now, i just can't use rancher. It's up for a few second, then down, and so on ...
And of course i have a lot of :
2020-06-15 14:32:47 21311 [Warning] Aborted connection 21311 to db: 'k3sbase' user: 'k3s-base' host: 'xxx.xxx.xxx.xxx' (Got an error reading communication packets) 2020-06-15 14:32:49 21312 [Warning] Aborted connection 21312 to db: 'k3sbase' user: 'k3s-base' host: 'xxx.xxx.xxx.xxx' (Got an error reading communication packets) 2020-06-15 14:32:57 21256 [Warning] Aborted connection 21256 to db: 'k3sbase' user: 'k3s-base' host: 'xxx.xxx.xxx.xxx' (Got an error reading communication packets) 2020-06-15 14:32:59 21310 [Warning] Aborted connection 21310 to db: 'k3sbase' user: 'k3s-base' host: 'xxx.xxx.xxx.xxx' (Got an error reading communication packets) 2020-06-15 14:33:01 21316 [Warning] Aborted connection 21316 to db: 'k3sbase' user: 'k3s-base' host: 'xxx.xxx.xxx.xxx' (Got an error reading communication packets)

Possibly related: https://github.com/rancher/k3s/issues/1912#issuecomment-645844326
I would like to see connection pooling made available though, and perhaps enabled by default if it turns out to improve performance.

@fabinou do you have error logs from the client side? I saw the same messages on the DB server when k3s crashed due to queries taking way too long to complete. If the lease updates don't complete in time the controller manager goroutine will exit its loop causing k3s to terminate, and you get messages like that on the DB side as it closes all the open connections uncleanly.

@fabinou

Hello
I have the same issue using MySQL on another server (Debian 10 MariaDB for the two server).

Your maximum number of connections seems high enough, so I assume your posted error messages have another cause and is not related to this issue.

I think it is caused by 'SELECT COUNT(*) FROM kine;' when k3s trying to start. The query stuck the whole MySQL server, especially when your DB is a micro/small instance.

I don't believe the size has anything to do with it.  We were using it a full-blown VM instance on-premises with 6 cores and 16GB of RAM.

Based on testing, the connection count seems to increase when there is a slow disk. Further, we have observed that as we fall behind (slow queries) this will "snowball" and get worse if the issue continues.
Additionally, if we exceed the max_connections we seem to fail hard.
We could add a tunable for connection pooling, which could help mitigate the issue. @brandond any other remarks?

hit the same on k3s version v1.19.10+k3s1 (72e8196c) (fresh install) with google cloud sql instance postgresql 13 (also fresh) / db-f1-micro
Bellow are the repartition of the error "pq: remaining connection slots are reserved for non-replication superuser connections" when k3s start. Rancher isn't able to start cleanly.

According to this doc https://cloud.google.com/sql/docs/postgres/quotas the max conn is 25

it seems that there is one request that impact a lot the connection usage time :
SELECT
  (
  SELECT
    MAX(rkv.id) AS id
  FROM
    kine AS rkv),
  (
  SELECT
    MAX(crkv.prev_revision) AS prev_revision
  FROM
    kine AS crkv
  WHERE
    crkv.name = $3),
  kv.id AS theid,
  kv.name,
  kv.created,
  kv.deleted,
  kv.create_revision,
  kv.prev_revision,
  kv.lease,
  kv.value,
  kv.old_value
FROM
  kine AS kv
JOIN (
  SELECT
    MAX(mkv.id) AS id
  FROM
    kine AS mkv
  WHERE
    mkv.name LIKE $1
  GROUP BY
    mkv.name) maxkv
ON
  maxkv.id = kv.id
WHERE
  (kv.deleted = $4
    OR $2)
ORDER BY
  kv.id ASC
LIMIT
  $5

It's fixed if i use a larger db instance.
So i think it worth a note in the docs to mention required db if not present (i didn't notice it) and a switch to limit connections would be nice.

I created a pull request to kine a while ago which addresses exactly that: database connection pooling. See especially this struct https://github.com/k3s-io/kine/blob/master/pkg/drivers/generic/generic.go#L70. The only thing left is another pull request on this repository, to create command line flags/environment vars for k3s, which control parameters of the aforementioned struct.

hi @david-becher
Yes we indeed saw it and event tried the flags, not really getting if this was still a proposal or a merged change in k3s.
We came to the understanding that code was merged and the only thing missing were the flags as you confirmed it.
So let's keep this open and active to have the flags available very soon then ;-)
For the records we have to go for a postgress instance to get around this point just because the flags are not supported yet. This is a bit sad since the guys at the company don't know much PG compared to MySQL/Mariadb.

After compiling K3s with the exposed database arguments, testing has determine that modifying the max connections and idle connections has not produced a meaningful improvement in DB performance. A slow DB with still cause large numbers of queries and connections to exist due to slow response time from the DB on those queries. As such, we will not be exposing these additional toggles to K3s as it simply obfuscates a larger issue.

@dereknola I configured Postgres based on hardware requirements specified in the documentation and this problem occurs very fast, This means that sizing in docs is wrong. Currently, load of Postgres is 50 on the 2 vCPU and 8gb machine for only 24 nodes (2 masters)


[Dev] Adding realistic output names for stubs

Hey, I've fixes the stubs outputs aiming to fix #13 with very simple changes.
This pr only contains new outputs names for the stubs developed before.
I'll e happy on hearing you opinion, feel free to discuss it here!
I'm at your disposal, Hope we have a merge!
Warmly, Davi

Thank you Abhinav, i forgot to ask a new review :eyes:


Add graphql sample backend

Purpose
This PR adds a sample Star Wars backend server which can be used to demonstrate the GraphQL scenarios in APIM.
Related issue: https://github.com/wso2/docs-apim/issues/1909

@RakhithaRR Could you please add graphQL schema to here?

@RakhithaRR Could you please add graphQL schema to here?

Schema is included in the swapiSchema.js file.

Schema is included in the swapiSchema.js file.

@RakhithaRR better to the schema file to this as a separate file with a .graphql extension.

@RakhithaRR better to the schema file to this as a separate file with a .graphql extension.

@HiranyaKavishani @fazlan-nazeem added the schema with ecf7dc3

@HiranyaKavishani @fazlan-nazeem added the schema with ecf7dc3


Move to-iso-string from dev dependency to dependency

to-iso-string is used in lib/utils.js since #2231
Fixes #2272

I can confirm this fixes the errors I'm seeing when trying to run mocha on my project.

thanks, just running tests atm.  I committed bd9450b before I saw the PRs...

No problem, @boneskull , when will new release make?

I'll release 2.5.1 as soon as travis gives the green light

released

Yay!
Incidentally, I've made the exact same mistake. My tests can't test if my app works without the tests installed. :-p


[SPARK-32219][SQL] Add SHOW CACHED TABLES Command

What changes were proposed in this pull request?

Modify SHOW TABLES to support SHOW CACHED TABLES.
After this pr, we can call
SHOW CACHED TABLES;

Why are the changes needed?

Once we cache table in sql mode, we have no way to get it back, so that we don't know which table has cached.
If we want to reduce some cache, we have to do clear cache that remove all cached tables.
SHOW CACHED TABLES can give a way to find which table is cached, then we can uncache the one we want.
Does this PR introduce any user-facing change?

Yes, a new command.
How was this patch tested?

New test.

Test build #125308 has started for PR 29034 at commit 5d362f6.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29919/
Test PASSed.

Test build #125308 has finished for PR 29034 at commit 5d362f6.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125308/
Test FAILed.

Looks useful cc: @HyukjinKwon @viirya

retest this please

Test build #125335 has started for PR 29034 at commit 5d362f6.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29946/
Test PASSed.

Test build #125345 has started for PR 29034 at commit 564725a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29957/
Test PASSed.

Test build #125345 has finished for PR 29034 at commit 564725a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125345/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29984/
Test PASSed.

Test build #125372 has started for PR 29034 at commit 564725a.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125335/
Test FAILed.

Test build #125372 has finished for PR 29034 at commit 564725a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125372/
Test FAILed.

@maropu @viirya Also, i was thinking, if we want to make this information available from one of the SHOW commands, can we not extend show tables to show another attribute isCached ? Do we need a special syntax for it ? I guess, the one advantage of this command is that it acts as a filter and only shows the cached tables.
And we could apply filter to only see individual table types.
spark.sql("show tables").where("isCached= true").show
spark.sql("show tables").where("isTemporary= true").show

What do you guys think ?

@viirya we can, but in sql mode e.g. ThriftServer, we have no way to get it.

Yea, I think this fix is for SQL interfaces.

But having this in one command gives a complete view of all the tables, regular, temporary and cached tables.

hm, adding a new column isCache looks fine, too, for that purpose. WDYT? @ulysses-you
I feel that might be better than adding a new parser token CACHED.

It's ok to add an attribute isCached but it is not enough.  2 reasons:

A database can contains hundreds tables, spark.sql("show tables").where("isCached= true").show can not used  in sql mode. Then it's hard to find all cached tables in all tables.
SHOW TABLES in SparkSQLDriver only show the table name without other attribute which is for compatible with Hive. Then the new attribute is useless in this scene.

@ulysses-you

A database can contains hundreds tables, spark.sql("show tables").where("isCached= true").show can not used in sql mode. Then it's hard to find all cached tables in all tables.

Ok. Sounds good !!

SHOW TABLES in SparkSQLDriver only show the table name without other attribute which is for compatible with Hive. Then the new attribute is useless in this scene.

I was not aware of this !! Thanks !!

retest this please

Looks okay as it is.

Test build #125534 has started for PR 29034 at commit 564725a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30150/
Test PASSed.

Test build #125534 has finished for PR 29034 at commit 564725a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125534/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30171/
Test PASSed.

Test build #125556 has started for PR 29034 at commit 5138cdd.

Test build #125556 has finished for PR 29034 at commit 5138cdd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125556/
Test FAILed.

retest this please

Test build #125584 has started for PR 29034 at commit 5138cdd.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30199/
Test PASSed.

Test build #125584 has finished for PR 29034 at commit 5138cdd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125584/
Test FAILed.

retest this please

@ulysses-you Ur, could you resolve the conflict?

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30259/
Test PASSed.

Test build #125646 has started for PR 29034 at commit 5138cdd.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30268/
Test PASSed.

Test build #125655 has started for PR 29034 at commit 14ead4e.

Test build #125646 has finished for PR 29034 at commit 5138cdd.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125646/
Test FAILed.

Build finished. Test FAILed.

Test build #125655 has finished for PR 29034 at commit 14ead4e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class IsExecutorAlive(executorId: String) extends CoarseGrainedClusterMessage
 class DisableHints(conf: SQLConf) extends RemoveAllHints(conf: SQLConf)

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125655/
Test FAILed.

Test build #125664 has started for PR 29034 at commit 856026b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30276/
Test PASSed.

Test build #125664 has finished for PR 29034 at commit 856026b.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125664/
Test FAILed.

cc @maropu @cloud-fan  thanks for review.

retest this please

Test build #126170 has started for PR 29034 at commit 856026b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30783/
Test PASSed.

Test build #126170 has finished for PR 29034 at commit 856026b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126170/
Test PASSed.

retest this please

Test build #126987 has started for PR 29034 at commit 856026b.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31611/
Test PASSed.

Merged build finished. Test PASSed.

Test build #126987 has finished for PR 29034 at commit 856026b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126987/
Test PASSed.

I'm okay to support it and no more comment. Anyone could check this? also cc: @cloud-fan

Test build #127195 has started for PR 29034 at commit 3a0b049.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31816/
Test PASSed.

Test build #127196 has started for PR 29034 at commit 13e1c35.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31817/
Test PASSed.

Test build #127196 has finished for PR 29034 at commit 13e1c35.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127196/
Test FAILed.

Test build #127195 has finished for PR 29034 at commit 3a0b049.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127195/
Test FAILed.

Test build #127206 has started for PR 29034 at commit 8262fe2.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31827/
Test PASSed.

Test build #127206 has finished for PR 29034 at commit 8262fe2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127206/
Test FAILed.

Test build #127238 has started for PR 29034 at commit 929c00d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31858/
Test PASSed.

Test build #127238 has finished for PR 29034 at commit 929c00d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127238/
Test PASSed.

Test build #130925 has started for PR 29034 at commit a9b88ca.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35530/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35530/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35530/
Test FAILed.

Test build #130925 has finished for PR 29034 at commit a9b88ca.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130925/
Test PASSed.

Test build #133162 has started for PR 29034 at commit 63163ed.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37761/

Test build #133162 has finished for PR 29034 at commit 63163ed.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133162/

Test build #133312 has started for PR 29034 at commit 635d3ab.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37904/

Test build #133312 has finished for PR 29034 at commit 635d3ab.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133312/


Don't copy-paste '.ts.lint' tests for '.js.lint' tests

Many tests have a test.js.lint file that is identical to, or an outdated version of, the test.ts.lint file.
I'm not convinced these tests really need to exist, since rules will run the exact same code on the exact same AST regardless of file extension.
If the concern is that people will forget to set typescriptOnly on a rule that only deals with a TypeScript construct, these tests won't help since TypeScript will let you put types in a .js file, then issue a diagnostic which we ignore.

I agree, we can deduplicate most of these


first commit :D.

working install and uninstall SP. You need to have BepInex already installed for the loader to work, and because of some of the strange cases that can happen because they share some files, it's impossible to smoothly installed SpaceWarp ontop of BepInEx. Shouldn't be a problem though.

you can also delete the branch in the main repo.


HRV Coordinate Data variables are mostly NaN

Describe the bug
For the HRV Zarrs, we save out each timestep's x_geostationary and y_geostationary coordinates to their own data variables, as the HRV image gets moved throughout the day. In at least some of the zarrs on GCP, those data variables are mostly all NaNs. The coordinates are there, based off the first timestep of the year, but the coordinate data variables are  mostly NaNs.
Not sure why, as the only work done is copying the values to new data arrays and adding them to the dataset, and they've been kept in float64 the whole time and not compressed either. Luckily its not a lot of data, although if it does need to be recomputed, that would still take awhile to download the datasets, open them, and save out the coordinates. The actual image data seems fine for all of them, so should only really come into effect for ones on the edges of the images.
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Additional context
Add any other context about the problem here.

We could just download a day's worth of data for a few random days throughout the year, but should be able to calculate the updated coordinates for each of them. The times and dates that HRV changes is fairly well documented, so we could calculate what the coordinates are for each timestep and add those.


Disable signing validation temporarily

problem should be specific to our 'master' branch

/bump @aspnet/build


Thoughts on removing border-bottom from .title-bar

Looks much cleaner, in my opinion, when tabs blend into the custom titlebar.
Before:

After:

Thanks!

I'm in agreement @bensk! I'll get the change made later today. Thanks for the suggestion.

Thank you! Far and away my favorite UI for Atom.

@bensk How about something like this? I've removed the border and added some breathing room between the title bar and the tabs, while also rounding the corners of the active tab to let it stand on its own.

Love it!

@bensk the change is out in the wild! Thanks again for discussing this 😄


Missing block receiptsRoot

Because of typo not crawled block's receiptsRoot, noticed at:

Ethereum block: 14829804
Polygon block: 28672180

Also required migration receipt_root -> receipts_root


[Feature Request] Downloading Status

When deploying a new application (e.g.: Container application to Service Fabric), even small container images take 20+ minutes to deploy, but there's no indication of the progression other than "downloading". It would be great if there was additionally a sort of indicator to show what amount out of the total is completed.

@DolpheusLabs I will talk with the hosting team to see if we can give information on the state of downloading containers.