Converting retrained ssd_resnet50_v1 object detection model to TFLite for use on Coral USB Accelerator

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
TensorFlow installed from (source or binary): tf-nightly installed through pip
TensorFlow version (or github SHA if from source): 2.2.0.dev20200401
Nvidia GeForce GTX 1050 Ti Driver Version: 440.64.00
Cuda compilation tools, release 9.0, V9.0.176
CUDNN_MAJOR 7 CUDNN_MINOR 0 CUDNN_PATCHLEVEL 5
Edge TPU Compiler version 2.1.302470888

Provide the text output from tflite_convert
TensorFlow Lite currently doesn't support control flow ops: Merge, Switch.
We are working on supporting control flow ops,
please see github issue at https://github.com/tensorflow/tensorflow/issues/28485.
Some of the operators in the model are not supported by the
standard TensorFlow Lite runtime and are not recognized by TensorFlow.
If you have a custom implementation
for them you can disable this error with --allow_custom_ops, or by setting
allow_custom_ops=True when calling tf.lite.TFLiteConverter().
Here is a list of builtin operators you
are using: ADD, CAST, CONCATENATION, CONV_2D, EQUAL, EXP, EXPAND_DIMS, FILL,
GATHER, GREATER, GREATER_EQUAL,
LESS, LOGICAL_OR, LOGISTIC, MAXIMUM, MAX_POOL_2D, MINIMUM, MUL, PACK, PAD,
RANGE, RESHAPE, RESIZE_BILINEAR, SELECT,
SHAPE, SLICE, SPLIT, SQUEEZE, STRIDED_SLICE, SUB, SUM, TOPK_V2, TRANSPOSE, UNPACK,
WHERE.Here is a list of operators for which you will need custom implementations:
DecodeGif, DecodeJpeg, DecodePng, DecodeRaw, NonMaxSuppressionV5, Substr.

Standalone code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/Jupyter/any notebook.
import tensorflow as tf

BATCH_SIZE = 32
IMG_HEIGHT = 320
IMG_WIDTH = 320
physical_devices = tf.config.list_physical_devices('GPU')
tf.config.set_logical_device_configuration(physical_devices[0],[tf.config.LogicalDeviceConfiguration(memory_limit=500)])
logical_devices = tf.config.list_logical_devices('GPU')

list_ds_Dianthus = tf.data.Dataset.list_files(str('/home/afayed/coral/detection_ag/ag/Dianthus''*/*/*'))
list_ds_DustyMiller = tf.data.Dataset.list_files(str('/home/afayed/coral/detection_ag/ag/DustyMiller''*/*/*'))
list_ds_Pansy = tf.data.Dataset.list_files(str('/home/afayed/coral/detection_ag/ag/Pansy''*/*/*'))
list_all = list_ds_Dianthus.concatenate(list_ds_DustyMiller).concatenate(list_ds_Pansy)
#len(list(list_all))

def get_label(file_path):
  parts = tf.strings.split(file_path, '/')
  class_name = tf.strings.split(parts[-3], '1')
  class_name = tf.strings.split(class_name, '2')
  class_name = tf.strings.split(class_name, '3')
  class_name = tf.strings.split(class_name, '4')
  return class_name

def decode_img(img):
	img = tf.image.decode_jpeg(img, channels=3) #color images
	img = tf.image.convert_image_dtype(img, tf.float32)
	#convert unit8 tensor to floats in the [0,1]range
	return tf.image.resize(img, [IMG_WIDTH, IMG_HEIGHT])

def process_path(file_path):
  label = get_label(file_path)
  img = tf.io.read_file(file_path)
  img = decode_img(img)
  return img, label

num_calibration_steps= 1000

def representative_dataset_gen():
	for _ in range(num_calibration_steps):
	# Get sample input data as a numpy array in a method of your choosing.
		image = list_all.take(1)
		yield [decode_img(image)]

saved_model_obj = tf.saved_model.load(export_dir='/home/afayed/coral/detection_ag/train/export/Servo/1586490086/')
concrete_func = saved_model_obj.signatures['serving_default']
concrete_func.inputs[0].set_shape([])
converter =  tf.lite.TFLiteConverter.from_concrete_functions([concrete_func])
converter.experimental_new_converter = False
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS,
                                       tf.lite.OpsSet.SELECT_TF_OPS]
converter.optimizations =  [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset_gen
tflite_quant_model = converter.convert()

open("/home/afayed/Desktop/output_tflite_graph.tflite", "wb").write(tflite_quant_model)

Also, please include a link to a GraphDef or the model if possible.
Model ckpt 5 used here as a .tar.
Frozen graph generated here
saved_model.pb used here
Any other info / logs
Detailed explanation
I am trying to use custom training data to retrain 'ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync_2018_07_03' (model obtained from model zoo here ) following Google's Retrain an object detection model tutorial
I retrained it on custom training data using TensorFlow with GPU devel version 1.12.0-rc2 on a docker container built with this Dockerfile.
I also tried retraining with TensorFlow with GPU version 1.15.2. The pipeline config file I am using is available here.
I have made sure I used a fixed_shape_resizer and included "max_number_of_boxes: 200" in train_input_reader and eval_input_reader and all of that.
My image dimensions as found in pipeline_v2.config are 320,320.
I found the new Experimental Converter here
So, my process is as such:
I start the docker container and start retraining from ckpt 0 that is given by downloading the model from the model zoo:
sudo docker run --gpus all --name edgetpu-detect \
--rm -it --privileged -p 6006:6006 \
--mount type=bind,src=${DETECT_DIR},dst=/tensorflow/models/research/learn_pet \
detect-tutorial

For the sake of testing lets only retrain 5 steps (I've tried up to 20,000 giving the same results):
./retrain_detection_model.sh --num_training_steps 5 --num_eval_steps 100

retrain_detection_model.sh found here
I then use 'saved_model_cli.py' to get any info I need about the model. Text file generated [here](https://drive.google.com/open?id=1nDNSpEyjExg-_L_wpaDJaNlbVrRvfbp8):
python /tensorflow/tensorflow/python/tools/saved_model_cli.py show
--dir /tensorflow/models/research/learn_pet/train/export/Servo/1586336368/
--all > /tensorflow/models/research/learn_pet/train/saved_model_cli_output.txt

I then export a frozen graph using export_tflite_ssd_graph.py (unedited) both through 1.12.0-rc2 and 1.15.2 code shown here is using TF 1.15.2:
python object_detection/export_tflite_ssd_graph.py
--pipeline_config_path="/tensorflow/models/research/learn_pet/ckpt/pipeline_v2.config"
--trained_checkpoint_prefix="/tensorflow/models/research/learn_pet/train/model.ckpt-5"
--output_directory="/tensorflow/models/research/learn_pet/models"

And then I use the converter invocation (stated up above + its output) that causes this issue.
I limit memory growth using
physical_devices = tf.config.list_physical_devices('GPU') tf.config.set_logical_device_configuration(physical_devices[0],[tf.config.LogicalDeviceConfiguration(memory_limit=500)]) logical_devices = tf.config.list_logical_devices('GPU')
Please help, I am also trying to do this with faster_rcnn_inception_resnet_v2_atrous_coco_2018_01_28 and that is where I found about the Experimental Converter working for RCNNs. My goal is to run either of these models on the Coral USB Accelerator after retraining with my own custom training data. I did not edit any scripts generating either of these models. Thanks in advanced

SSD models don't convert directly to TFLite. You need an intermediate step to export a graph that uses our custom NMS op to speed up post-processing. Please take a look at these instructions.

@srjoglekar246 Apologies formatting typo in my original comment caused a step of my process to not show. I have updated it, but I am restating it here for your convenience (following executed in TF 1.15.2 GPU).
I do run:
python object_detection/export_tflite_ssd_graph.py
--pipeline_config_path="/tensorflow/models/research/learn_pet/ckpt/pipeline_v2.config"
--trained_checkpoint_prefix="/tensorflow/models/research/learn_pet/train/model.ckpt-5"
--output_directory="/tensorflow/models/research/learn_pet/models"

And then I use the generated tflite_graph.pb as an input graph_def_file to TFLite converter following the Coral retrain object detection tutorial:
tflite_convert  --output_file="/tensorflow/models/research/learn_pet/models/output_tflite_graph.tflite"
--graph_def_file="/tensorflow/models/research/learn_pet/models/tflite_graph.pb"
--inference_type=QUANTIZED_UINT8  --input_arrays='normalized_input_image_tensor'
--output_arrays="TFLite_Detection_PostProcess,TFLite_Detection_PostProcess:1,TFLite_Detection_PostProcess:2,TFLite_Detection_PostProcess:3" --mean_values=128 --std_dev_values=128
--input_shapes=1,320,320,3  --change_concat_input_ranges=false
--allow_nudging_weights_to_use_fast_gemm_kernel=true --allow_custom_ops

At which point I get error:
F tensorflow/lite/toco/tooling_util.cc:1728] Array FeatureExtractor/resnet_v1_50/fpn/top_down/nearest_neighbor_upsampling/stack,
which is an input to the Pack operator producing the output array
FeatureExtractor/resnet_v1_50/fpn/top_down/nearest_neighbor_upsampling/stack_1,
is lacking min/max data, which is necessary for quantization. If accuracy matters,
either target a non-quantized output format, or run quantized training with your model
from a floating point checkpoint to change the input graph to contain min/max information.
If you don't care about accuracy, you can pass --default_ranges_min= and
--default_ranges_max= for easy experimentation.
Fatal Python error: Aborted

If I add the following to the above tflite_convert command:
--default_ranges_min=0 --default_ranges_max=6

It works but then I get this error when using the Edgetpu Compiler for the final step to run inference on the Coral:
Edge TPU Compiler version 2.1.302470888
ERROR: :79 input->params.zero_point != output->params.zero_point (110 != 0)
ERROR: Node number 77 (PACK) failed to prepare.

Internal compiler error. Aborting!

And that is the point where I wrote the above script (stated as standalone code to reproduce issue in original comment)

By the way, while following the link you shared, running this command (in TF 1.12.0rc2 GPU docker image):
root@d97ce87cb12f:/tensorflow/models/research# bazel run -c opt //tensorflow/tensorflow/contrib/lite/toco:toco -- --input_file=$OUTPUT_DIR/tflite_graph.pb --output_file=$OUTPUT_DIR/output_tflite_graph.tflite --input_shapes=1,320,320,3 --input_arrays=normalized_input_image_tensor --output_arrays='TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1','TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3' --inference_type=QUANTIZED_UINT8 --mean_values=128 --std_values=128 --change_concat_input_ranges=false --allow_custom_ops

Weirdly gives me this error:
ERROR: Skipping '//tensorflow/tensorflow/contrib/lite/toco:toco': no such package
'tensorflow/tensorflow/contrib/lite/toco': BUILD file not found on package path

Which is strange since tflite_converter is working on the same docker image. I believe tflite_convert uses toco in the backend doesn't it?

@adfayed Are you using 2.2.0.dev20200401 or TF1.12?. There is no contrib in TF2.x. mixing TF1.x and TF2.x will throw errors as there is a compatibility issue. Thanks!

@jvishnuvardhan No no, I am using TF 1.15.2 GPU for all of this (retraining, generating frozen graph, and attempting conversion to TFLite). I tried the latest TF nightly for the new experimental TF Lite converter added functionality for RCNNs but with no luck.
Can you please reply to my previous comment before that? That is the important one :)
Running tflite_convert as per the detect tutorial on TF1.15.2 on an SSD network (albeit ResNet instead of Mobilenet) gives me that error. I do not have min/max data even though my config file (linked above) includes the graph_rewrite options. After you answer that question please, my second question would be, can I use Faster RCNN with the new experimental TFlite converter? If so how?

It doesn't seem to me that the model is quantized. This might be why the convertor is complaining about min/max data etc.
 If I run tflite_convert with the above command (i.e inference_type FLOAT instead of QUANTIZED_INT8) it will convert, but then the EdgeTPU Compiler errs as such:
Edge TPU Compiler version 2.1.302470888
Invalid model: /home/afayed/coral/detection_ag/models/output_tflite_graph.tflite
Model not quantized

I am trying to make a quantized model to run on the Coral. Isn't quantized re-training done by adding this snippet to the pipeline.config file?
graph_rewriter {
  quantization {
    delay: 0
    weight_bits: 8
    activation_bits: 8
  }
}

My pipeline.config looks like this:
# SSD Resnet 50
# Configured for MSCOCO Dataset.
# Users should configure the fine_tune_checkpoint field in the train config as
# well as the label_map_path and input_path fields in the train_input_reader and
# eval_input_reader. Search for "PATH_TO_BE_CONFIGURED" to find the fields that
# should be configured.

model {
  ssd {
    num_classes: 3
    image_resizer {
      fixed_shape_resizer {
        height: 320
        width: 320
      }
    }
    feature_extractor {
      type: "ssd_resnet50_v1_fpn"
      depth_multiplier: 1.0
      min_depth: 16
      conv_hyperparams {
        regularizer {
          l2_regularizer {
            weight: 0.000399999989895
          }
        }
        initializer {
          truncated_normal_initializer {
            mean: 0.0
            stddev: 0.0299999993294
          }
        }
        activation: RELU_6
        batch_norm {
          decay: 0.996999979019
          scale: true
          epsilon: 0.0010000000475
        }
      }
      override_base_feature_extractor_hyperparams: true
    }
    box_coder {
      faster_rcnn_box_coder {
        y_scale: 10.0
        x_scale: 10.0
        height_scale: 5.0
        width_scale: 5.0
      }
    }
    matcher {
      argmax_matcher {
        matched_threshold: 0.5
        unmatched_threshold: 0.5
        ignore_thresholds: false
        negatives_lower_than_unmatched: true
        force_match_for_each_row: true
        use_matmul_gather: true
      }
    }
    similarity_calculator {
      iou_similarity {
      }
    }
    box_predictor {
      weight_shared_convolutional_box_predictor {
        conv_hyperparams {
          regularizer {
            l2_regularizer {
              weight: 0.000399999989895
            }
          }
          initializer {
            random_normal_initializer {
              mean: 0.0
              stddev: 0.00999999977648
            }
          }
          activation: RELU_6
          batch_norm {
            decay: 0.996999979019
            scale: true
            epsilon: 0.0010000000475
          }
        }
        depth: 256
        num_layers_before_predictor: 4
        kernel_size: 3
        class_prediction_bias_init: -4.59999990463
      }
    }
    anchor_generator {
      multiscale_anchor_generator {
        min_level: 3
        max_level: 7
        anchor_scale: 4.0
        aspect_ratios: 1.0
        aspect_ratios: 2.0
        aspect_ratios: 0.5
        scales_per_octave: 2
      }
    }
    post_processing {
      batch_non_max_suppression {
        score_threshold: 0.300000011921
        iou_threshold: 0.600000023842
        max_detections_per_class: 100
        max_total_detections: 100
      }
      score_converter: SIGMOID
    }
    normalize_loss_by_num_matches: true
    loss {
      localization_loss {
        weighted_smooth_l1 {
        }
      }
      classification_loss {
        weighted_sigmoid_focal {
          gamma: 2.0
          alpha: 0.25
        }
      }
      classification_weight: 1.0
      localization_weight: 1.0
    }
    encode_background_as_zeros: true
    normalize_loc_loss_by_codesize: true
    inplace_batchnorm_update: true
    freeze_batchnorm: false
  }
}
train_config {
  batch_size: 1
  data_augmentation_options {
    random_horizontal_flip {
    }
    random_pixel_value_scale {
    }
    random_vertical_flip {
    }
    random_patch_gaussian {
    }
    random_jitter_boxes {
    }
    random_adjust_brightness {
    }
    random_adjust_contrast {
    }
    random_adjust_hue {
    }
    random_adjust_saturation {
    }
  }
  sync_replicas: true
  optimizer {
    momentum_optimizer: {
      learning_rate: {
        manual_step_learning_rate {
          initial_learning_rate: 0.003
          schedule {
            step: 900000
            learning_rate: 0.00003
          }
          schedule {
            step: 1200000
            learning_rate: 0.000003
          }
        }
      }
      momentum_optimizer_value: 0.9
    }
    use_moving_average: false
  }
  gradient_clipping_by_norm: 10.0
  fine_tune_checkpoint: "/tensorflow/models/research/learn_pet/ckpt/model.ckpt"

  # Note: The below line limits the training process to 200K steps, which we
  # empirically found to be sufficient enough to train the COCO dataset. This
  # effectively bypasses the learning rate schedule (the learning rate will
  # never decay). Remove the below line to train indefinitely.

  num_steps: 25000
  from_detection_checkpoint: true
  load_all_detection_checkpoint_vars: true
  startup_delay_steps: 0.0
  replicas_to_aggregate: 8
  max_number_of_boxes: 100
  unpad_groundtruth_tensors: false
}
train_input_reader {
  label_map_path: "/tensorflow/models/research/learn_pet/ag/ag_label_map.pbtxt"
  tf_record_input_reader {
    input_path: "/tensorflow/models/research/learn_pet/ag/train_dataset.record"
  }
  max_number_of_boxes: 200
}
eval_config {
  metrics_set: "coco_detection_metrics"
  num_examples: 400
}
eval_input_reader {
  label_map_path: "/tensorflow/models/research/learn_pet/ag/ag_label_map.pbtxt"
  shuffle: true
  num_readers: 1
  tf_record_input_reader {
    input_path: "/tensorflow/models/research/learn_pet/ag/validate_dataset.record"
  }
  max_number_of_boxes: 200
}
graph_rewriter {
  quantization {
    delay: 0
    weight_bits: 8
    activation_bits: 8
  }
}

@srjoglekar246 Hello? Can you please reply? I believe my model is quantized.

Does the floating point model run on CPU, maybe with our benchmark_model tool? (EdgeTPU may not support floating-point inference, but atleast CPU should give a sanity check)
TBH, I am not very familiar with the object detection config APIs. But I am not sure that you can re-train with quantization only for fine-tuning.
Could you try quantized re-training with an already-quantized model from the detection zoo, such as ssd_mobilenet_v2_quantized_coco?

Any update about this issue? Same problem here.

Also, looking for some help on this and retraining object detection for the Coral Dev Board.

I believe this problems occurs when training with SIGMOID as a score converter for classes > 2. Im facing a similar issue, but it can be circumvented by using SOFTMAX.
@srjoglekar246

Could you try quantized re-training with an already-quantized model from the detection zoo, such as ssd_mobilenet_v2_quantized_coco?

I did that (Or i used the ssd_mobilenet_edgetpu_coco) and thats when i faced this error when i tried to run inference witht the .tflite model. I was training with SIGMOID as the score converter for classes > 2.

RuntimeError: tensorflow/lite/kernels/kernel_util.cc:129 std::abs(input_product_scale - bias_scale) <= 1e-6 * std::min(input_product_scale, bias_scale) was not true.Node number 108 (CONV_2D) failed to prepare.

@liufengdb I vaguely remember us encountering the issue @NicholaiStaalung mentions above. Do you know what the issue is?

I was facing the same issue:
Array FeatureExtractor/MobilenetV1/fpn/top_down/nearest_neighbor_upsampling/stack, which is an input to the Pack operator producing the output array
FeatureExtractor/MobilenetV1/fpn/top_down/nearest_neighbor_upsampling/stack_1, is lacking min/max data, which is necessary for quantization. If accuracy matters, either target a non-quantized output format, or run quantize
d training with your model from a floating point checkpoint to change the input graph to contain min/max information. If you don't care about accuracy, you can pass --default_ranges_min= and --default_ranges_max= for easy
experimentation.
Aborted (core dumped)

In my case the edge device didn't support nearest neighbor upsampling so I changed to bilinear upsampling and it worked fine. Made the change here:
https://github.com/tensorflow/models/blob/master/research/object_detection/models/feature_map_generators.py#L716
changed tf.image.resize_nearest_neighbor to tf.image.resize_bilinear and it worked just fine.

@srjoglekar246 Hello? Can you please reply? I believe my model is quantized.

have you solved your issue? I am facing the same :/

@srjoglekar246 Hello? Can you please reply? I believe my model is quantized.

have you solved your issue? I am facing the same :/


Python 3.x support

I see clusterpy is not getting updated anymore, but I find its functionality is still useful. Are there any plans of supporting Python 3.x? or maybe development has moved somewhere else and I've missed it?
Anyway, I'm willing to give it a try and port this to Python 3.6, any pointers on what the roadblocks might be?
Thanks!

@plablo09 Unfortunately there is no ongoing development right now and no concrete plans to move to Python 3.x (there are no roadblocks I can think of right now, but it's been a long time since I worked on this code).
Please contact/email @JuancaDuque (research director behind clusterpy) for questions about the future of the tool.
Sorry for the delayed response.

Don't worry, it was a long weekend here so I wasn't doing any work anyway!
I use clusterpy mainly for teaching, and I love it so I'll give it a shot at porting this to Python 3.
Thanks!

Dear Pablo,
Thanks for your interest on ClusterPy!
As Sergio told you, we have no plans to release a Python 3 version of
ClusterPy, not because we are not interested, but because we have not
time... a pity. We know the importance of this migration and it would be
great if you are interested in contributing to this task. Of course, we
guarantee you the acknowledgment of your contribution.
On our side, Sergio and me would be happy to answer any question that you
have regarding the current code.
Sergio: what would this imply in terms of GitHub? should we create a new
project? is so, can you do it so that Pablo can use it?
Cheers
Juan

*Juan Carlos Duque, PhD*

Executive Director, PhD in Economics <http://www.eafit.edu.co/phdeconomia>

Director, Research in Spatial Economics (RiSE <http://www.rise-group.org>)
School of Economics and Finance

Office: 26-217
Phone (57) (4) 261 9500, ext 9993
<EMAIL_ADDRESS>
Linkedin <https://www.linkedin.com/in/juan-carlos-duque-a01a1579>

Google Scholar <http://scholar.google.it/citations?user=JkHBB-wAAAAJ&hl=en>
------------------------------

Carrera 49  N 7 sur – 50, Medellín - Colombia
Línea de atención al usuario: (+574) 4489500  www.eafit.edu.co

On Tue, Mar 21, 2017 at 5:58 PM, pablo lopez<EMAIL_ADDRESS>wrote:

Don't worry, it was a long weekend here so I wasn't doing any work anyway!
I use clusterpy mainly for teaching, and I love it so I'll give it a shot
at porting this to Python 3.
Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/clusterpy/clusterpy/issues/47#issuecomment-288145553,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AI3F2JRjMyYXZcazwDOMfCA4QrdJToiJks5roAGugaJpZM4MetGo
.

Hi, we finally have a Python 3 port of clusterpy. The updated version is here, all the tests are passing.
We also read shapefiles using Fiona instead of the custom reader that was implemented. Should we make a pull request?
Cheers

awesome contribution!
Sergio and Camilo: how can we proceed? It would be nice to have this
version, as well as the one for Python 2 as two projects in our Git Hub
Repository (https://github.com/Rise-group). If you agree, can you
coordinate to make it happen? We have to make sure to give the
corresponding credits.
Cheers!
Juan

*Juan Carlos Duque, PhD*

Director, Research in Spatial Economics (RiSE <http://www.rise-group.org>)

School of Sciences

Department of Mathematical Sciences

Phone (57) (4) 261 9500, ext 9993
<EMAIL_ADDRESS>
Linkedin <https://www.linkedin.com/in/juan-carlos-duque-a01a1579>

Google Scholar <http://scholar.google.it/citations?user=JkHBB-wAAAAJ&hl=en>
------------------------------

Carrera 49  N 7 sur – 50, Medellín - Colombia
Línea de atención al usuario: (+574) 4489500  www.eafit.edu.co

On Wed, Aug 14, 2019 at 3:31 PM yoshiorta<EMAIL_ADDRESS>wrote:

Hi, we finally have a Python 3 port of clusterpy. The updated version is
here https://github.com/CentroGeo/clusterpy/tree/python3_upgrade, all
the tests are passing.
We also read shapefiles using Fiona instead of the custom reader that was
implemented. Should we make a pull request?
Cheers
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/clusterpy/clusterpy/issues/47?email_source=notifications&email_token=ACG4LWGUYC2Q3BUPXOOL3YDQERTTJA5CNFSM4DD22GUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4KAUII#issuecomment-521407009,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACG4LWATVLKLA2EJT3AVUQDQERTTJANCNFSM4DD22GUA
.

Hi! Having both versions sounds like a good plan. I think moving forward on the python 3 version should include full integration with Geopandas and the whole geospatial stack. I  believe the functionality provided by clusterpy has a place within the current ecosystem.
Lets coordinate the way forward
Cheers

Hi everyone!
Really good news, I'm excited to be able to use clusterpy in Python 3 and in an integration with geopandas (if possible).
I appreciate all the work everybody have done.
I am interested in creating some tutorials or user guide with this library, so that they are more user friendly thinking in the end user, who may not know the complexity of the algorithms, but that presents a tangible need to use a tool like this for specific cases in that your data is spatially distributed.
I can contribute with the documentation of the tutorials on how to use these algorithms, perhaps including the visualization with a GIS like QGIS that is a typical end user software and that its users have very different backgrounds.

Dear Eduardo,
Thank you very much for your interest in contributing.
Take a look to this link:
http://www.rise.abetan16.webfactional.com/risem/clusterpy/index.html
We should build from there.
Cheers
Juan

*Juan Carlos Duque, PhD*

Director, Research in Spatial Economics (RiSE <http://www.rise-group.org>)

School of Sciences

Department of Mathematical Sciences

Phone (57) (4) 261 9500, ext 9993
<EMAIL_ADDRESS>
Linkedin <https://www.linkedin.com/in/juan-carlos-duque-a01a1579>

Google Scholar <http://scholar.google.it/citations?user=JkHBB-wAAAAJ&hl=en>
------------------------------

Carrera 49  N 7 sur – 50, Medellín - Colombia
Línea de atención al usuario: (+574) 4489500  www.eafit.edu.co

On Wed, Aug 21, 2019 at 3:31 PM Eduardo Rojas Rodríguez <
<EMAIL_ADDRESS>wrote:

Hi everyone!
Really good news, I'm excited to be able to use clusterpy in Python 3 and
in an integration with geopandas (if possible).
I appreciate all the work everybody have done.
I am interested in creating some tutorials or user guide with this
library, so that they are more user friendly thinking in the end user, who
may not know the complexity of the algorithms, but that presents a tangible
need to use a tool like this for specific cases in that your data is
spatially distributed.
I can contribute with the documentation of the tutorials on how to use
these algorithms, perhaps including the visualization with a GIS like QGIS
that is a typical end user software and that its users have very different
backgrounds.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/clusterpy/clusterpy/issues/47?email_source=notifications&email_token=ACG4LWDOMYYEVZSZKBTEEQLQFWQ33A5CNFSM4DD22GUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD43BBDY#issuecomment-523636879,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACG4LWCMSBHINB74334C2E3QFWQ33ANCNFSM4DD22GUA
.

Dear Pablo,
I am coordinating with Sergio and Camilo to move things around so we can
have both versions of Clusterpy, one beside the other. Once we have the
structure (anytime during this week) we will let you know.
cheers

*Juan Carlos Duque, PhD*

Director, Research in Spatial Economics (RiSE <http://www.rise-group.org>)

School of Sciences

Department of Mathematical Sciences

Phone (57) (4) 261 9500, ext 9993
<EMAIL_ADDRESS>
Linkedin <https://www.linkedin.com/in/juan-carlos-duque-a01a1579>

Google Scholar <http://scholar.google.it/citations?user=JkHBB-wAAAAJ&hl=en>
------------------------------

Carrera 49  N 7 sur – 50, Medellín - Colombia
Línea de atención al usuario: (+574) 4489500  www.eafit.edu.co

On Thu, Aug 15, 2019 at 12:29 PM pablo lopez<EMAIL_ADDRESS>wrote:

Hi! Having both versions sounds like a good plan. I think moving forward
on the python 3 version should include full integration with Geopandas and
the whole geospatial stack. I believe the functionality provided by
clusterpy has a place within the current ecosystem.
Lets coordinate the way forward
Cheers
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/clusterpy/clusterpy/issues/47?email_source=notifications&email_token=ACG4LWGN7DQGQV5CZRUTCB3QEWG6NA5CNFSM4DD22GUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4MOSLI#issuecomment-521726253,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACG4LWDD6ICJXVUQ2VWMVOLQEWG6NANCNFSM4DD22GUA
.

Hi Pablo,
Yes, I know what are you talking about because I work wit Serge since 2005.
This is my thought about this: Instead of moving our codes to PySAL, we
should make clusterPy 100% compatible with PYSAL so that the users in PYSAL
can do an "import clusterpy as cp". In that way, we can keep clusterPy as
THE library for spatial aggregation and, from there, collaborate with
PYSAL, ArcGIS, QGIS, etc. Does it make sense to you?.
I recognize that clusterPy need a big refactoring to achieve the following
goals:

make it easier for other people to contribute.
make it 100% importable from other key partners like the ones I
mentioned above
make it faster

For that refactoring, I am planning to move little by little: first
p-regions, second max-p-regions (which are the two most important models),
etc.
Cheers

*Juan Carlos Duque, PhD*

Director, Research in Spatial Economics (RiSE <http://www.rise-group.org>)

School of Sciences

Department of Mathematical Sciences

Phone (57) (4) 261 9500, ext 9993
<EMAIL_ADDRESS>
Linkedin <https://www.linkedin.com/in/juan-carlos-duque-a01a1579>

Google Scholar <http://scholar.google.it/citations?user=JkHBB-wAAAAJ&hl=en>
------------------------------

Carrera 49  N 7 sur – 50, Medellín - Colombia
Línea de atención al usuario: (+574) 4489500  www.eafit.edu.co

On Mon, Aug 26, 2019 at 10:59 AM pablo lopez<EMAIL_ADDRESS>wrote:

That sounds great!
On a side note, I've noticed there is ongoing work to develop
regionalization algorithms in PySAL
https://github.com/pysal/spopt/tree/master/spopt/region, it seems
they're moving from region https://github.com/pysal/region. As far as I
can see there are only a couple algorithms implemented in region, perhaps
it contributing there would make sense? I already made some tweakings to
clusterpy to adapt it to the way PySAL is structured and it seems totally
feasible.
Cheers
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/clusterpy/clusterpy/issues/47?email_source=notifications&email_token=ACG4LWHZ7FILDOBGNHMVYVTQGP4VTA5CNFSM4DD22GUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5EZ6KI#issuecomment-524918569,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACG4LWCD5GSXUJJM7FMJMODQGP4VTANCNFSM4DD22GUA
.

Awesome,
We will go back to you soon.

*Juan Carlos Duque, PhD*

Director, Research in Spatial Economics (RiSE <http://www.rise-group.org>)

School of Sciences

Department of Mathematical Sciences

Phone (57) (4) 261 9500, ext 9993
<EMAIL_ADDRESS>
Linkedin <https://www.linkedin.com/in/juan-carlos-duque-a01a1579>

Google Scholar <http://scholar.google.it/citations?user=JkHBB-wAAAAJ&hl=en>
------------------------------

Carrera 49  N 7 sur – 50, Medellín - Colombia
Línea de atención al usuario: (+574) 4489500  www.eafit.edu.co

On Mon, Aug 26, 2019 at 1:02 PM pablo lopez<EMAIL_ADDRESS>wrote:

Juan Carlos,
That makes perfect sense to me. Let me know the way forward and count us
in for all the work ahead
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/clusterpy/clusterpy/issues/47?email_source=notifications&email_token=ACG4LWBO4BIEBVFCZ4JYQ7TQGQLDDA5CNFSM4DD22GUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FFAJQ#issuecomment-524963878,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACG4LWCSUFPGUSXCHUY4HYTQGQLDDANCNFSM4DD22GUA
.

That is great news!!
I am wondering whether or not that affects performance. Is there any
statistic or report about that in other contexts?

*Juan Carlos Duque, PhD*

Director, Research in Spatial Economics (RiSE <http://www.rise-group.org>)

School of Sciences

Department of Mathematical Sciences

Phone (57) (4) 261 9500, ext 9993
<EMAIL_ADDRESS>
Linkedin <https://www.linkedin.com/in/juan-carlos-duque-a01a1579>

Google Scholar <http://scholar.google.it/citations?user=JkHBB-wAAAAJ&hl=en>
------------------------------

Carrera 49  N 7 sur – 50, Medellín - Colombia
Línea de atención al usuario: (+574) 4489500  www.eafit.edu.co

On Mon, Aug 26, 2019 at 1:21 PM yoshiorta<EMAIL_ADDRESS>wrote:

Good day!
I'd like to add the following: the first step we took towards making
'ClusterPy' compatible with Python 3 was making all the available scripts
in the library go through the Future http://python-future.org/ library;
specifically, its 'futurize' function
http://python-future.org/quickstart.html#to-convert-existing-python-2-code
.
Not only does this make small changes to the code, but it also adds a
dependency to the 'future' library to the scripts, which makes it that the
code not only becomes compatible with Python 3.X, but also with Python 2.7.
As such, the version present in the fork, in the python3_upgrade brach
https://github.com/CentroGeo/clusterpy/tree/python3_upgrade is
compatible with both Python 3.X and Python 2.7. This makes 'ClusterPy' now,
as a library, available for both language versions.
Thank you for everything so far, and count with us for everything needed
in the future.
Excellent day.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/clusterpy/clusterpy/issues/47?email_source=notifications&email_token=ACG4LWH26HRJ3QHWGRO7G7TQGQNKNA5CNFSM4DD22GUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FGXQI#issuecomment-524970945,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACG4LWHKXI7EJ7R7HJTVFSDQGQNKNANCNFSM4DD22GUA
.

Hi Eduardo,
Yes, Sergio created that plugin several years ago. It is not a
straightforward task. We can go for that once we have a refactored version
of clusterPy.

*Juan Carlos Duque, PhD*

Director, Research in Spatial Economics (RiSE <http://www.rise-group.org>)

School of Sciences

Department of Mathematical Sciences

Phone (57) (4) 261 9500, ext 9993
<EMAIL_ADDRESS>
Linkedin <https://www.linkedin.com/in/juan-carlos-duque-a01a1579>

Google Scholar <http://scholar.google.it/citations?user=JkHBB-wAAAAJ&hl=en>
------------------------------

Carrera 49  N 7 sur – 50, Medellín - Colombia
Línea de atención al usuario: (+574) 4489500  www.eafit.edu.co

On Mon, Aug 26, 2019 at 4:39 PM Eduardo Rojas Rodríguez <
<EMAIL_ADDRESS>wrote:

Hello everyone!
probably I'm getting ahead of, but I think there is a clusterpy plugin for
QGIS. Maybe can be updated and now be compatible with Python 3 and QGIS 3.4

I suppose it would be good to get the utility to a large number of users
with many types of backgrounds and diferentes skills

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/clusterpy/clusterpy/issues/47?email_source=notifications&email_token=ACG4LWGAMAFF4HXL6OELTZLQGRESRA5CNFSM4DD22GUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FYSNI#issuecomment-525044021,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACG4LWGBLHAQ3YXCV27XLETQGRESRANCNFSM4DD22GUA
.

Hello all,
ClusterPy is a really useful package, however I have only Python 3.x so I tried to use what you coded because I'm very interested in using the function geoSOM of the ClusterPy package.
I did 'pip install git+https://github.com/CentroGeo/clusterpy@python3_upgrade ' but when I call a function I have an error such as ": module 'clusterpy' has no attribute 'importShape'".
Do you know how to make it work?
I am using Jupyter.
Thanks in advance,
Kind Regards
Clara Wolf

Hello Clara,
It's kind of a work in progress, most things work in CentroGeo's fork but there are certainly some glitches and we haven't even began making the package git ready.
For the moment, the best way to use it with Python 3 is cloning the repository. Most of the things work and are documented on this notebook (some comments are in spanish, though).
Hope this helps


Added example found in intro post

Added the second example link found in introduction post.

By my reading of the intro post, this is an example of something the author has previously generated with a different typesetting flow rather than an example of something the imp prototype can handle.


Build for Scala 2.13 as well

Cross-build for 2.12 and 2.13
Unfortunately, the tests only run using 2.12, as contextual (test dependency) is not available for 2.13.

Unfortunately, the tests only run using 2.12, as contextual (test dependency) is not available for 2.13.

We should remove it

2.0 has support for 2.13
https://mvnrepository.com/artifact/com.propensive/contextual-examples

#308

LGTM 👍

v1.0.0-M2 is out 🎉


Expand the binary to more complete CLI

Current bin/w3cjs accepts 'validate <FILE_PATH>' as args and default validates as HTML regardless of file type.
It would be a useful tool if expanded to accept and validate CSS as well + provide the validate function options via the command line.

It's an html validator (face slap)


How to write valid query for DynamoDB

System information:

Operating system (distribution) and version: Mac OSX
DBeaver version: DBeaver Enterprise	<IP_ADDRESS>906122139

Connection specification:

Database name and version: Amazon DynamoDB

Describe your question:
I can right click on a table and do "edit data" to see the tabular view of the data, but I cannot figure out how to scan/query the tables. The Wiki also does not mention Dynamo on the right hand side of the landing under "Databases".
How can I query the tables besides just dumping samples of the data?

There are two options.
You can use simple SQL queries like
select * from TableName where KeyName='Value'

But there are a lot of limitations in DynamoDB queries. E.g. you must specify primary key or partition key.
Also you can use JSON-like API queries: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html

🤦♂ I didn't try single ticks around my right-hand value. Also didn't realize I could do the JSON apis 👍  Thank you!


Enhancement: wrap text on tooltips of "Recheck-Cond" and "Filter" values in Explain plan

After a previous request by me, the "Explain execution plan" started wrapping the "Condition" statement text of the tooltip popup that appears when hovering over the "Condition". This allowed long statements to be shown in their entirety, instead of being cut off:

However, the similar "Recheck-Cond" and "Filter" values in the explain plan window, do not yet wrap text. It would be nice to see these also wrapped:

Actually we going to significantly improve execution plan visualization in one of near versions.
Anyhow, marking as enhancement.


Bitcoin address for own profile displays on apps but not on web

When viewing my own profile on a browser after setting a Bitcoin address, it continues to prompt me to add an address. When viewing my profile on the Windows or iOS app, it correctly displays my Bitcoin address. See below screenshot:

I added this Bitcoin address several days ago (October 5th, 2017 to be exact).

we'll take a look, thanks

@johnnyRose Fixed!
The website was hiding it because it was signed by a key was since revoked. But sigs like that are still valid.

@mlsteele nice! Thanks for following up!


Time to best solution not reported in Excel results

Excel table gene
The algorithm calculates the time taken to find the best solution, TTB,  (which is different from the total execution time) and is not reported in the Excel tables.

Fixed in c8cdfebf5d3e1f23d37e344390f5c3702edfc025


PrivateKey example is misleading.

I initially had an issue with privateKey because I naively thought I could pass a string path to a privkey, but this would throw Error: Cannot parse privateKey: Unsupported key format from inside ssh2. I ultimately got it to work by using fs and loading the file that way. It might be worth mentioning this in the examples.
(I'd happily submit a PR using this technique in conjunction with a new field like privateKeyPath or something but that would add a dependency on fs.)
Thanks for this package!

Hi @prurph,
thanks for the information i'll update the example.
privateKey - mixed - Buffer or string that contains the host private key (OpenSSH format). (Required) Default: (none)

Done.


Simplify maven

Try to simplify maven usage and make dependencies explicit for each module (thank you mvn dependency:analyze).
This pull request is created against solution branch in order for you to check that everything is ok with final implementation.

Thanks!


CI: Delete unused .travis.yml configuration file

This PR removes an unused file related to an earlier CI solution.
The CI now connected is CircleCI.

Thank you!


Is this dataset still expected to be released?

Wondering if this is scheduled to be released or abandoned?

Yeah, we ran into some initial hosting issues, but have been working with HuggingFace to resolve them. I just started my initial attempt to upload the full dataset.

I've now nearly uploaded 2/6 splits subsets. HF going down this past week definitely didn't help lol.

I've now nearly uploaded 2/6 splits subsets. HF going down this past week definitely didn't help lol.

do you have the link to the dataset? or where the dataset will be?

My guess would be on his or mosaicml's HF page based upon his last statement:

https://huggingface.co/Skylion007
https://huggingface.co/mosaicml

I'm trying to learn more about Lora's and Embeddings beforehand in order to hopefully merge and train ethical add-ons and checkpoints with it that'll catch on on Civitai, cause then we'd finally have an ethical model base in a readily available source, if the terms and architecture allow it.

I've now nearly uploaded 2/6 splits subsets. HF going down this past week definitely didn't help lol.

Can we expect an update?

I've now nearly uploaded 2/6 splits subsets. HF going down this past week definitely didn't help lol.

is there any update on this @Skylion007

I'm trying to learn more about Lora's and Embeddings beforehand in order to hopefully merge and train ethical add-ons and checkpoints with it that'll catch on on Civitai, cause then we'd finally have an ethical model base in a readily available source, if the terms and architecture allow it.

hope we can finetune it with this high quality dataset: https://www.kaggle.com/datasets/innominate817/pexels-110k-768p-min-jpg/data


Implement Chinese news i18n as per guidelines

https://docs.ghost.org/api/handlebars-themes/helpers/translate/

Is this project still available to work on?


hmac 0.10.1 depends on a deprecated crate

Hi! Please consider updating the hmac dependency. hmac 0.10.1 depends on crypto-mac which is considered deprecated and has been replaced by digest.
This issue prevents me from compiling pwhash with recent versions of reqwest (0.12+).
I see that this has already been fixed in 339faec. Can you please push the updated version to crates.io?
Thank you.

FYI: I'll be mostly offline for the next ten days or so, and I won't be able to handle any issues.


SourceMaps references not properly generated.

Laravel Mix Version: 2.0.0 (npm list --depth=0)
Node Version (node -v): 8.9.0
NPM Version (npm -v): 5.5.1
OS: 17.10

Description:
It seems that since the new update that for the sourcemaps to work properly this also needs to be added:
mix.options({
    uglify: false
});

Without this proper references to individual files do not seem to be generated properly. I would love if somebody could confirm this so I could investigate further and see what is going wrong. .
Optionally it this setting does seem to be right for this case it should be added automatically when sourceMaps for development are being generated(obviously we always wan to uglify for production).
Steps To Reproduce:
Update to 2.0.0.
Try out the development sourceMaps and see if they if references are generated properly  for seperate includes with something like scss.

The same happens to me, whats more disabling uglify does not fix that.

Any fixes for this problem?
The sass sourcemap is being generated but most of the references point to wrong files in the browser inspector.


Hist bins and explicit zero customization

Add explicit hist bins customization in form of i.e. dict in mapping section:
column_mapping['numerical_features'] = {'feature1: {hist_bins: 5}, feature2: (hist_bins:10)}
Also it is common than some features have preavailing 0 values (i.e. product counts within clients purchases). Currently auto binning combines theese zeros with another counts to make bins like 0-4, 5-9 etc. But theese zeros need to be shown separately.
My suggestion is to include this config in mapping section:
column_mapping['numerical_features'] = {'feature1: {hist_bins: 5, explicit_zero: True}, feature2: {hist_bins:10, explicit_zero: False}}

Hi @eccodolf
Thanks for sharing this feature request. I feel your pain with nulls (been there!), and bin customization is something I also wish to add!
We definitely plan to do that, just slightly later on.
The reasoning is the following:

existing column mapping configuration is easy to use, and we do not want to compromise on this usability since many users shared they like the plug-n-play

there are some more parameters we want to make configurable in addition to bins (like p-value boundaries for statistical tests etc.)
we think the best way to combine ease of setup and ability to customize the reports is to add a second (optional) config.  It will include all these more sophisticated parameters that some users might want to tweak.

We will add this more complex config in future releases and address the explicit zero issue. And thanks a lot for proposing the implementation.
Happy to hear your thoughts!

Add explicit hist bins customization in form of i.e. dict in mapping section:
column_mapping['numerical_features'] = {'feature1: {hist_bins: 5}, feature2: (hist_bins:10)}
Also it is common than some features have preavailing 0 values (i.e. product counts within clients purchases). Currently auto binning combines theese zeros with another counts to make bins like 0-4, 5-9 etc. But theese zeros need to be shown separately.
My suggestion is to include this config in mapping section:
column_mapping['numerical_features'] = {'feature1: {hist_bins: 5, explicit_zero: True}, feature2: {hist_bins:10, explicit_zero: False}}

I was also trying to find a way to change the same. Automated binning becomes a problem when you have explicit values such as -9999 which should be a category/bar in itself.

Hello, Could you please provide a way to pass in bin-edges as well for the Histogram in Data Drift Widget. I want to get the bin edges from "reference" data and use the same bin edges for "current" data. I tried exploring graph_objs code but I don't see a way to pass in these explicitly. Will the code work if I swap graph_objs with plotly_express? Or will there be any issues downstream if I modify this code?

Hi @phaniparsa, @eccodolf, @aakash-dusane , we just added something very similar to what you requested: the possibility to set custom bins for the data drift report using column_mapping. Here are the docs: https://docs.evidentlyai.com/reports/data-drift#histogram-plots-customization and here is an example: https://github.com/evidentlyai/evidently/blob/main/evidently/examples/boston_dataset_drift_with_customized_bins.ipynb
Would be great if you could test (please download the latest package version 0.1.23.dev0) and tell us if it solves your feature request!

Thank you @emeli-dral. I'll test this from my end and let you know soon.


Need help?

Hi, looks like you're building out this library with the intent leveraging it in both the web and node clients? Just wondering if there's any timeline or road map and whether there's anything you need help with. We're examining using either jaeger but would prefer to build out with open census and then export to jaeger. Neither have a browser client library so we're looking at trying to help one or the other.

Hi @mnovinger! We are just about getting started, but e haven't progressed enough to have a roadmap or actionable things to do. We would :heart: to have contributions, though! Give us some time to figure things out, and check back in a bit?

Hi! Any progress on this project?

@ofrobots is there any code released from this repo? I am now maintaining the existing OC JS, just want to know if this repo needs to maintenance or is "dead"

I haven't paid much attention for a while in this direction, but it seems to be "dead" as far as I can tell. @kjin as FYI too.


Adds support for updating documents that have been loaded using the s…

After a question/comment from @kureus, I've added a change to support updating documents that are loaded using the session as well as documents that have been added to the session. I've also added an additional test and removed a line that was causing the existing test to pass falsely.

I think the travis build failed because of environment variables, all the tests passed locally for me.

Yeah travis tests are unreliable atm the moment. WIP to get stable.


MIMIC documentation

As much as I love and appreciate MIMIC as an excellent (AND FREE!) resource, I have to say the way the resources are made available is extremely confusing, at least to me.
Why is there more than one MIMIC repository that discusses code? There is MIMIC-IV, but there is also MIMIC-code. Both discuss building mimic, but one is updated, the other one doesn't?
To add to the confusion, this website contains details of what the MIMIC database is: https://mimic.physionet.org/  but is not up to date, has serious documentation issues and links it has don't direct users to appropriate locations (most up to date ones, at least). This is very confusing, as this is the first website we get to after getting access to the database!
Then, there is this website that includes documentation and seems up to date: https://mimic-iv.mit.edu/  but it also has documentation issues and the only way I found to access it, is through a link in a comment in this issue: https://github.com/MIT-LCP/mimic-code/issues/856#issuecomment-800744217_
Not to mention it's quite difficult to get an option to download the latest MIMIC version, because the documentation from this website (https://mimic.physionet.org/) gives you this link https://physionet.org/content/mimiciii/  , which actually is a link to version https://physionet.org/content/mimiciii/1.4/ (1.4 (??)), which is not the most up to date one!
Could we please have either these things updated or at least an overview of the resources available and to which MIMIC version they refer to? Or am I making some very silly mistakes and this is not the way to approach this?
Thank you!

@Erinaceida thanks for your feedback, and we agree that things have become a little scattered. As mentioned by @briangow at https://github.com/MIT-LCP/mimic-code/issues/899#issuecomment-812472496, we are in the process of (1) consolidating web content into a single website at https://mimic.physionet.org/ and (2) consolidating the code repositories. If you have other suggestions for improvement in terms of the way documentation and code are managed, please post them here!

Thanks for your reply!
I have absolutely no expertise in this area, but I would suggest that, at least for the time being, create a repository/add a topic within each repository on GitHub laying out the different resources and to which sections they correspond to.
I noticed people post issues regarding database building which I think are due to them using old SQL code on new releases. Making this overview of resources in relation to their versions would likely decrease the number of new issues too!

create a repository/add a topic within each repository on GitHub laying out the different resources and to which sections they correspond to.

Thanks, this is what we had in mind too :)

I noticed people post issues regarding database building which I think are due to them using old SQL code on new releases.

Yes, consolidating code repos should help. SQL poses its own special challenges because syntax varies widely across different database systems (postgres, bigquery, mysql, etc). In the past we worked almost entirely in postgres, but now a lot of our work is in bigquery.


Training on Scannet: data prepare  rRaise an exception

Hi Jonas! This is a fantastic job.
When I train on the scannet dataset, I met a problem about the dataset prepare module.  In semseg.py line 194, when  the code runs to "self._labels = self._select_correct_labels(labels, num_labels)",  there exists an exception. So I'd like to ask whether the Scannet data need some other processing.
The detail error information is below:
"/Desktop/program/Mask3D/trainer/trainer.py", line 872, in prepare_data
self.train_dataset = hydra.utils.instantiate(self.config.data.train_dataset)
raise HydraException(f"Error calling '{cls}' : {e}") from e
hydra.errors.HydraException: Error calling 'datasets.semseg.SemanticSegmentationDataset' : not available number labels, select from:
200, 200

Hi! :)
Thanks for your kind words!
It looks like you created the ScanNet200 dataset (with 200 labels) but want to train a model for the original ScanNet dataset with 20 labels) on it.
Can you verify in your preprocessing scripts if you enabled scannet200?

Yes, I understand it. During process the Scannet data. In the script parameters, I used the is_scannet200=false (False), this leads to create the scannet200 dataset. Thank you very much.

I faced the same problem, and I do not know how to fix it. Could you please give me more details? Thanks a lot!

@RongkunYang @JonasSchult @francisengelmann @eltociear


Types that are missing or could be improved

Looking at the official API documentation, there are some types that are missing or could be improved to be more explicit(especially with type-fest now included in the repository).
I don't know the overall big picture of the project so it would be nice if someone on the team could take a look and see if I am misunderstanding or missing something.
Also, some of these might be a breaking change.

AuthTokenResponse - The response for the third step of the authentication process returns an accessToken and a tokenType, but right now the expected fields includes a state and is missing tokenType.

getSharedLabels - There is an optional parameter omitPersonal that looks like is missing.

DueDate - I'm getting a lang field in the response that is missing. But the documentation also doesn't include it, so I'm not sure about this one.

Project - viewStyle could be an union of 'list' and 'board' like it is in the ProjectViewStyle type used in requests.ts.

AddTaskArgs and UpdateTaskArgs - Use RequireOneOrNone for due fields(expect due_lang).

AddCommentArgs - Use RequireExactlyOne for taskId and projectId.

@pawelgrimm

@pawelgrimm, what are your thoughts on these? If everything seems reasonable I will open a PR with the changes.

Great! I'm glad to hear that, thank you.


[T4A3][W13-B3] Mok Wei Xiong, Edmund

…classes

Accepting this PR without a review because another of your PRs for this tutorial was reviewed in detail.


Problems of the code

I check your Enet-SAD-model.py,and I found that the lane prediction branch in your code is different from the paper,can you tell me if there is any reason for this?Thanks in advance.

@zhangtingyu11 Hi, I also have this question. Have you had any ideas? Thanks!

Hi @zhangtingyu11 and @jyang68sh
Thanks for your interest in my implementation. Following the original paper(Section 3.3: Architecture Design), I adopt the same network block introduced in SCNN.
I'm now working on rearranging my implementation more similarly to the original code. If you find something different to the paper, please feel free to open another issue for other questions.
I hope this answers your questions. I'm closing this issue for now.


Katebranch

I added a little

Please edit the content with a correct formatting

and please create another pull request with new changes.


deploy gets stuck and doesn't print anything

Steps to reproduce
cap production deploy
and that's it, just gets stuck forever until I abort it
Expected behavior

Capistrano must to print a TimeOut error log or something else when it doesn't connect to server

Actual behavior
When I run: cap production deploy it shows nothing, just keeps waiting
But when I run: cap production deploy --trace it shows this:
** Invoke production (first_time)
** Execute production
** Invoke load:defaults (first_time)
** Execute load:defaults
** Invoke rbenv:validate (first_time)
** Execute rbenv:validate

System configuration
https://gist.github.com/anyelopetit/7320fc8141554ea4b75ec88bb02f8132

I would suggest you reopen this on capistrano-rbenv that's the last plugin that is doing something when this is locking up, perhaps there's an exhaustive debug mode you can run.
I'll leave this issue open for now, but I don't think we can assist really.

Yes @anyelopetit please follow up with us when you track down what was causing the freeze. There may be something capistrano can do better in terms of error messaging.
My assumption is that rbenv:validate is the very first task that actually requires an SSH connection to the remote host, and that SSH connection is failing to open. You could try changing the connection timeout to see if that changes anything.
set :connection_timeout, 5

We probably can't help you fix the reason for the timeout (that is likely due to a firewall between you and the remote host, or an incorrect port number). But we may be able to improve capistrano to provide better error messaging, or maybe a more reasonable default timeout so capistrano doesn't appear to hang indefinitely.

Hey guys! @leehambley @mattbrictson Thank you for answering so quickly!
I just tried changing the connection timeout with:
set :connection_timeout, 5

but it doesn't change anything.
Do you have any idea about what is causing that error? How can I find out the reason why it gets stuck?
That is preventing me from deploying

Sorry I don't know what would cause this. My first test would be to try using ssh directly (i.e. take capistrano out of the picture) and verify that you can connect to the server that you specified in deploy/production.rb.
Perhaps the hang is because ssh is waiting for something on stdin? Like asking for a password? Make sure you have the necessary public/private key and ssh-agent setup so that you aren't challenged for a passphrase when authenticating. There is information about that here: https://capistranorb.com/documentation/getting-started/authentication-and-authorisation/

Hi @mattbrictson! I just verified that I can access to the server specified in deploy/production.rb running ssh deploy@[server_ip] and it's working! I also verified that my ssh key is in the ~/.ssh/authorized_keys file and in my Bitbucket account.
There is not any prompt or something else waiting when I access to the server. Do you have any other idea for this weird bug?

You can try to look at https://capistranorb.com/documentation/faq/why-does-something-work-in-my-ssh-session-but-not-in-capistrano/ and try to simulate more and more of what the rvm plug-in is doing by tweaking the way you use ssh.

To be clear, are you using https://github.com/rvm/rvm1-capistrano3 or https://github.com/capistrano/rvm/issues/83 or something else?

@leehambley I'm using capistrano-rbenv, this is my capistrano gems set in my Gemfile:
group :development do
  gem 'capistrano'
  gem 'capistrano-bundler'
  gem 'capistrano-passenger'
  gem 'capistrano-rails'
  gem 'capistrano-rbenv', github: 'capistrano/rbenv'
  gem 'capistrano-sidekiq'
end

Hi guys! @mattbrictson @leehambley I was trying what is in https://capistranorb.com/documentation/faq/why-does-something-work-in-my-ssh-session-but-not-in-capistrano/ and I could notice this:
In staging, my shell is Interactive and Not Login, but in production my shell is Interactive and Login.
So, I tried running this commands in staging:
me@localhost $ ssh user@staging_ip "[[ $- == *i* ]] && echo 'Interactive' || echo 'Not interactive'"
Interactive
me@localhost $ ssh user@staging_ip "shopt -q login_shell && echo 'Login shell' || echo 'Not login shell'"
Not login shell

It works!
But I tried in production:
me@localhost $ ssh user@staging_ip "[[ $- == *i* ]] && echo 'Interactive' || echo 'Not interactive'"
# NO ANSWER, it keeps waiting for something
me@localhost $ ssh user@staging_ip "shopt -q login_shell && echo 'Login shell' || echo 'Not login shell'"
# SAME HERE, NO ANSWER

How can I change my shell to a Not Login shell?

Great bit of research, I'm not sure how that can change the behaviour of rbenv much, but at least we have something to work on.
I'd suggest to check with the people who configured the servers, there's a bunch of settings on the SSH daemon, and the default shell that would change this behaviour.
What do you know about the configurations of both environment's servers?

How can I change my shell to a Not Login shell?

A "log-in" shell, by the way isn't necessarily the problem, it just means that the server has been configured to treat connections like a log-in shell, meaning loading your "dot files" if any.. it's not impossible that one of them is acting weird when you connect without a tty (without a terminal)

I have the same issue. But I can catch it only using one internet provider. In another, all works perfectly. I'm not sure if this info can help you

I have the same problem. Deploy get stuck on macOS. In my case, these are 2 different fixes:

Using another provider fixes an issue on macOS.
Deploy from Linux works well on both internet providers.

I suspect something is blocking waiting for input that you are not able to provide. If some part of the deploy is .. say, requesting a password, maybe on Linux it works "by accident" for you because of some combination of shared environment vars, something in your ruby env.
Can you clarify please @mamantoha what you mean "another provider", like internet provider?!

@leehambley yes, internet provider.

We are facing a similar issue. Deploy runs fine on Linux on all server. However, on MacOS with docker, the deploy runs fine on our production server but not on the staging server.
The deployment is stuck at:
0:28 deploy:assets:precompile
      01 $HOME/.rbenv/bin/rbenv exec bundle exec rake assets:precompile

I can see on the server that the pre-compilation is running but Capistrano is stuck.
However, when it is run on a Linux machine or a CI server, it works fine.
00:09 deploy:assets:precompile
      01 $HOME/.rbenv/bin/rbenv exec bundle exec rake assets:precompile
      01 yarn install v1.22.17
      01 [1/4] Resolving packages...
      01 [2/4] Fetching packages...
      01 [3/4] Linking dependencies...
      01 warning " ><EMAIL_ADDRESS>has incorrect peer dependency<EMAIL_ADDRESS>      01 [4/4] Building fresh packages...
      01 Done in 22.10s.
      01 yarn run v1.22.17
      01 $ RAILS_ENV=${RAILS_ENV:-production} NODE_ENV=${NODE_ENV:-production} && webpack --config webpack.prod.js --env NODE_ENV=${NODE_ENV} --env RAILS_ENV=${RAILS_ENV} --progress
      01 <s> [webpack.Progress] 0%
      01
      01 <s> [webpack.Progress] 1% setup before run
      01 <s> [webpack.Progress] 1% setup before run NodeEnvironmentPlugin
      01 <s> [webpack.Progress] 1% setup before run

Both deployments are running via Docker.
The problem is on only MacOS and for only one server. Exact same Capistrano config and setup work for production server.
I'm not sure on how to continue debugging this further and I would be very happy if someone could help and point me into the right direction.
Additionally, if I run Capistrano only for deploy:assets:precompile job, the job runs without any issues and print the output.

There is a chance this could be an SSH or network timeout issue. See the keep-alive solutions discussed in #2118.

Don't know if this is still an issue for people but it is for me. Findings:

Shared linked solution doesn't work
Using watch -n 1 'ps aux | grep ssh | grep -v grep' to monitor the server's ssh connections I can see the ssh connection stays alive and well, so it doesn't seem to be a server doing anything wrong
To me it does it at the end of the deployment, after the deploy:log_revision task
It would seem like Capistrano just doesn't close the connection
To give more information, my capistrano gems used are:

  gem "capistrano", "~> 3.6.1", :require => false
  gem "capistrano-rails", "~> 1.1", :require => false
  gem "capistrano-bundler", "~> 1.1", :require => false
  gem "capistrano-faster-assets", "~> 1.0", :require => false

@stoicAlchemist sorry to hear you are running into this. Have you tried tweaking the SSHKit connection pool? I would try disabling connection pooling entirely and seeing if that makes a difference.
SSHKit::Backend::Netssh.pool.idle_timeout = 0 # disabled

https://github.com/capistrano/sshkit?tab=readme-ov-file#connection-pooling
Also, you could check your log/capistrano.log file and see what the last lines look like prior to the hang. Maybe there are some clues there as to what capinstrano is waiting on.

@mattbrictson thanks for the fast response. Using the pool config you mentioned got the problem worst 😅 , executing the first task it got just "frozen" and didn't go through. I'm trying to deploy to our QA environment and it just hangs:
** Invoke qa (first_time)
** Execute qa
** Invoke load:defaults (first_time)
** Execute load:defaults
** Invoke bundler:map_bins (first_time)
** Execute bundler:map_bins
** Invoke deploy:set_rails_env (first_time)
** Execute deploy:set_rails_env
** Invoke deploy:set_linked_dirs (first_time)
** Execute deploy:set_linked_dirs
** Invoke deploy (first_time)
** Execute deploy
** Invoke deploy:starting (first_time)
** Execute deploy:starting
** Invoke deploy:check (first_time)
** Execute deploy:check
** Invoke git:check (first_time)
** Invoke git:wrapper (first_time)
** Execute git:wrapper
00:00 git:wrapper
      01 mkdir -p /tmp

This is the output using the --trace flag. Log is stuck there too and the remote host did create the ssh connection and just hangs there.
Without the config, the log file ends on:
 DEBUG [333093e5] Command: cd /data/*********/releases && ( export PATH="/opt/rubies/2.2.3/bin:$PATH" ; /usr/bin/env echo "Branch qa (at b7fa7386bd87238a8bb8ac7ca0abebd656351eb1) deployed as release<PHONE_NUMBER>4107 by *******" >> /data/*********/revisions.log )
  INFO [9e29ffab] Finished in 0.306 seconds with exit status 0 (successful).
  INFO [8d9834e3] Finished in 0.308 seconds with exit status 0 (successful).
  INFO [333093e5] Finished in 0.358 seconds with exit status 0 (successful).
  INFO [aaf2a6f8] Finished in 0.389 seconds with exit status 0 (successful).

** I had to anonymize user and project name, other than that, all is current output
It seems that it is hanging after the deployment:log_revision  is successfully completed.

That sounds perfect, I'll start debugging that part on the server, might take some time as I may need a hand from my OPS team. Will update here with results.

@stoicAlchemist Did you ever find a root cause for this? I'm experiencing it now and can't find the cause.

For me, we were using a bastion host so this was our ssh_options
ssh_options[:proxy] = Net::SSH::Proxy::Command.new(
	"ssh -i #{ssh_options[:keys].first} -l #{fetch(:user)} #{jump_server_ip} nc -w 7200 %h %p"
)

Removing the nc -w 7200 and reworking our ssh_options to something like the following worked
set :ssh_options, {
	user: fetch(:user),
	keys: KEY_HERE,
	keepalive: true,
	keepalive_interval: 60,
	timeout: 1800
}
fetch(:ssh_options)[:proxy] = Net::SSH::Proxy::Command.new("ssh -i #{fetch(:ssh_options)[:keys].first} -l #{fetch(:user)} #{fetch(:jump_server_ip)} -W %h:%p")

Best I could tell, it was preventing the connection from exiting after whatever command was complete.

This makes a lot of sense. I'm using EC2 connect to avoid a bastion host, but there's still an underlying hop. I'll look into this. Thank you so much, @JonTheStone.

This makes a lot of sense. I'm using EC2 connect to avoid a bastion host, but there's still an underlying hop. I'll look into this. Thank you so much, @JonTheStone.

Good luck!

The proxy command for an EC2 connection is aws ec2-instance-connect open-tunnel --instance-id <INSTANCEID> but unfortunately I think it's the AWS CLI itself that's preventing the pool from closing in this case, so it's not as simple as I thought. I will continue debugging and post any solution I find here.

Quickfix to address this for EC2 Instance Connect Endpoint
For those who are using (or considering) the newer EC2 Instance Connect Endpoints instead of bastion hosts, I wanted to provide details on an issue relating to this as well as a workaround.
Background
EC2 Instance Connect allows you to connect to a private EC2 instance using a local AWS CLI command
instead of hopping over a bastion host. Once implemented, you can ssh using a command such as:
ssh ec2-user@[INSTANCE] \
    -i [SSH-KEY] \
    -o ProxyCommand='aws ec2-instance-connect open-tunnel \
    --instance-id %h'

Where [INSTANCE] is the instance-id of our EC2 server.
Our Setup
Our keys and users were already managed for Capistrano, so we simplified this to adding the following
to our [REPO]/.ssh/config file:
Host test-server
  HostName [INSTANCE]
  ProxyCommand aws ec2-instance-connect open-tunnel --instance-id %h

Allowing us to have a [REPO]/config/deploy/test.rb have the following server entry:
server "test-server", user: "httpuser", roles: %w{web}

The Problem
This worked, but I immediately saw the behavior in this ticket. The deployment would hang following the
deploy:log_revision task. This is the final task in the Capistrano Flow
As noted in this comment, setting SSHKit::Backend::Netssh.pool.idle_timeout = 0 caused a hang after the first task. So the behavior for EC2 Instance Connect Endpoints seemed similar to that for bastion hosts and other scenarios.
After some extensive debugging, I figured out that the aws command was remaining open, which made it impossible
for socket.close to finish here, causing the hang.
What's notable about this is that ssh -l USER test-server would work as expected, because somehow the aws
command can see that the SSH connection has been broken and close when using a TTY, but not when using a socket connection via SSHKit. This strikes me as a problem with SSHKit, but I can't confirm that right now.
Less a solution than a hack
I found that if I manually kill the aws process while the deploy is stuck, the connection pool closes and the deploy
finishes as I'd expect. So I created the following task:
…
  task :kill_aws do
   sh 'killall aws'
  end
…
after 'deploy:finished', 'deploy:kill_aws'

Note, this is a pretty rough hack, and that task can be fleshed out quite a bit to make it more robust, but this gives
you the idea. Basically, we need a final task that shuts down anything that may be artificially holding the connection open.


Update to the native layer documentation

Hey,
I was just trying to get going with snow and noticed in the process that the native layer build documentation is missing a few things. In fact, I couldn't get it working with the provided instructions at all.
I realise there's no need to build these, however I'm sure anyone wanting to contribute to snow will appreciate the additional details :)
Stefan

Just added an extra link to the builds and fixed a couple typos.

The build process is no longer part of the snow workflow!


fix: Drag to select widgets within container like widgets ends up selecting Parent widget

Description
This PR fixes, drag to select widget feature inside container like widgets. The changes include,

Add condition to stop triggering select action when, drag to select is still active
Delay stopping drag to select to the end of the execution stack, to prevent triggering selection action
Change name of isDragging to isMouseDown to avoid confusion with the other isDragging in the same file
Trigger start dragging to select action after mousedown and mousemove instead on every mousedown

Fixes #20804
Media
Type of change

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Manual
Before

https://user-images.githubusercontent.com/71900764/220725344-a4a50770-1335-405f-ac32-2ec63d3c9e6f.mp4
After
https://user-images.githubusercontent.com/71900764/220725390-9d94cd31-28d2-4b21-ae62-dbb98c2678ea.mp4
Test Plan

Add Testsmith test cases links that relate to this PR

Issues raised during DP testing

Link issues raised during DP testing for better visiblity and tracking (copy link from comments dropped on this PR)

Checklist:
Dev activity

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] PR is being merged under a feature flag

QA activity:

[ ] Test plan has been approved by relevant developers
[ ] Test plan has been peer reviewed by QA
[ ] Cypress test cases have been added and approved by either SDET or manual QA
[ ] Organized project review call with relevant stakeholders after Round 1/2 of QA
[ ] Added Test Plan Approved label after reveiwing all Cypress test

@rahulramesha Nice PR description, it has most things a reviewer will need. The fix looks good to me.

/ok-to-test sha=b50b75c

@rahulramesha Jest tests are failing in this PR.

/build-deploy-preview

/ok-to-test sha=10bc06b

/ok-to-test sha=862f183

/build-deploy-preview

@kamakshibhat-appsmith @Sripriya93 This Bug fix is ready for QA, please test the Drag to select widget, click to selecting a single widget and selecting multiple widgets by clicking on them scenarios.

@rahulramesha please check this issue https://www.loom.com/share/3a22363a65884edbaf5d02473cf87e80

@kamakshibhat-appsmith this issue seems to be previously existing. Fixing it in a wholistic manner would require more changes because if the scroll is not there, the options will be cut off by the edge of the container. Please create a new issue for the same.

Created issue here https://github.com/appsmithorg/appsmith/issues/21172
Rest LGTM!


Reproducing Figure 1

Impressive work! :  )
I tried to reproduce the Cross-Entropy results of Figure 1, yet was unsuccessful. Specifically, the fraction of examples memorized at the end are far below than the plotted.
I used the same configurations specified in Appendix G.3. on CIFAR-10 with 40% of the labels flipped at random. For those 40% noisy data, at the ending epochs (around the 120th epoch), the fraction of memorized examples (i.e., those whose predicted label = the fake label) is only around 17%, while that of the generalized examples (i.e., predicted label = the true label) is around 67%. Also, the red line appeared to be always below the green line for the upper right plot in Figure 1.
Would you please share more details on your implementation of this part? Thank you.

I removed data augmentation (originally with random crop and padding etc.), and was able to produce similar figures now.


Increase and make the B2 concurrent connection limit configurable

Let's wait for https://github.com/erikw/restic-systemd-automatic-backup/issues/12#issuecomment-433683763

@erikw Isn't this done as I see ba732cb2 did this?


chore: Add git and build-essential to the Meltano Docker image

Context: https://meltano.slack.com/archives/CKHP6G5V4/p1660060697122949?thread_ts=1659552634.170039&cid=CKHP6G5V4
Successful build & publish to the GitHub registry: https://github.com/meltano/meltano/actions/runs/2827386004
We can pursue more complex/time-consuming solutions to reduce our dependencies and image size later. For now we should ensure the image is useful by adding these packages.

@WillDaSilva is the plan to re-publish the 2.4.0 tags?

Yes.


Anyway to save the session as an object but not a file?

Hi,
I know that the tensorflow can save all variables into a file as a checkpoint
Is there anyway to save the tensor flow session into an object? like
a=Queue()
a.put( sess.status({%variables%}) )  #or maybe a.put( tf.train.status({%variables%}) )
So when I want to restore a session status. I just do a.get()
Is that possible?
Thank you.

I'm not sure what semantics you're suggesting for sess.status(...) or tf.train.status(...). If you're within the same Python process, you can treat a tf.Session like any other object.
A tf.Session is not serializable, but you can use the tf.MetaGraphDef protocol buffer, along with a checkpoint file, to represent an entire model.

I understand that tf.Session is as any other object. Maybe I did not explain my question clearly.
session_at_step1=tf.session().????
tf.session().run(...some feeds....)
session_at_step2=tf.session().????
In this case, the object will be changed. I mean, is there any way (a function at ????) to let session_at_step1 be different with session_at_step2?

I think the answer is "probably no," but I'm still unclear on what you want the behavior of ???? to be. If you describe a high-level use case, we would consider a feature request.

I've updated the question and made it clear. I mean, instead of having a handle(pointer) to the session, if there is anyway to copy the session.

Ah, ok. Right now there is no way to copy an entire session, and the best way to do what you're asking about is to use the tf.train.Saver to write out a MetaGraphDef proto, and import that into another session using tf.train.import_meta_graph(). This is still quite a new feature, but you can look at the relevant tests to see an example of how it's used. It does give you the ability to restore an entire graph and resume training etc.
If there is any improved tooling you'd like to see around this format, please let us know in a feature request!


cant overide anchor version

I tried to install the anchor to my computer but it keeps outputting this error
Ignored package anchor-cli v0.27.0 (https://github.com/coral-xyz/anchor?tag=v0.27.0#3b451447) is already installed, use --force to override
Error: The system cannot find the file specified. (os error 2)

Can you specify the OS pls ?

this is the full error report
C:\Users\steve>avm install 0.27.0
Updating git repository https://github.com/coral-xyz/anchor
warning: skipping duplicate package swap found at C:\Users\steve\.cargo\git\checkouts\anchor-50c4b9c8b5e0501f\3b45144\tests\swap\programs\swap
warning: skipping duplicate package shared found at C:\Users\steve\.cargo\git\checkouts\anchor-50c4b9c8b5e0501f\3b45144\tests\typescript\programs\shared
warning: skipping duplicate package shared found at C:\Users\steve\.cargo\git\checkouts\anchor-50c4b9c8b5e0501f\3b45144\tests\zero-copy\programs\shared
Ignored package anchor-cli v0.27.0 (https://github.com/coral-xyz/anchor?tag=v0.27.0#3b451447) is already installed, use --force to override
Error: The system cannot find the file specified. (os error 2)


Terminal Split Action Icon is using css to rotate, should update codicon

https://github.com/microsoft/vscode/pull/93440#discussion_r399230906

Also this only worked previously when the terminal was panel on the side. To verify test that the split terminal icon is correct in the following cases:

Terminal in bottom panel
Terminal in left panel
Terminal in right panel
Terminal in viewlet


lost frame when sliding to next picture in wechat browser

thanks to provide the nice component.
I got a problem when using react-slick in the wechat browser.
here is my code: https://codesandbox.io/s/react-slick-playground-e51q7
I run it nice in chrome or safari, but have something wrong in the browser of wechat.
here is a screenshot:
could you do me a favor and how to solve this problem.

要不你试试我自己写的一个Carousel, 看看有没这个问题？https://github.com/YIZHUANG/react-multi-carousel

@YIZHUANG thanks, it seems that useTransform will make the mistake. I set it to false and everything work fine.


Fix cache_marc_files script

Description
This PR fixes the broken cache_marc_files script.  The script does have any direct references to the core/s3.py library and as a result the MirrorUploader.IMPLEMENTATION_REGISTRY is not being loaded with the S3 implementations.  This PR ensures that registration occurs when loading the MARC Exporter.
I added a test_marc2.py test to ensure that nothing from core/s3.py was being called directly.  The problem is that tests in both tests/api and tests/core rely on test classes (MockS3Client and MockS3Uploader) and thus if I put my unit test in core/test_marc.py, it passes with or without my fix.
I contemplated doing a larger refactor to get the test classes out of core/s3.py so that the test could live in tests/core/test_marc.py but that would involve a larger refactor.   I'm happy to do that but wanted to check in before proceeding.
Motivation and Context

https://www.notion.so/lyrasis/MARC-record-generator-not-running-8677e5dd7502412fa50900b404853eb2#7b1afef923f0419dab2b4e2abb9d1f19
How Has This Been Tested?
I added a unit test to reproduce the problem and confirmed that the code change fixes it.
I also manually tested the script to verify that the issue has been resolved.

Checklist

[ ] I have updated the documentation accordingly. (N/A)
[x] All new and existing tests passed.

Okay @tdilauro : thanks for the review. It should be good to go now.

@dbernstein Hmm. Still some linter errors. Have you installed the pre-commit hook as discussed at the top of the Code Style section of the README? It should run the linters and prevent commits that don't satisfy them.

Will do.
On Tue, Apr 26, 2022 at 8:26 AM Tim DiLauro @.***>
wrote:

@dbernstein https://github.com/dbernstein Hmm. Still some linter
errors. Have you installed the pre-commit hook as discussed at the top of
the Code Style section of the README
https://github.com/ThePalaceProject/circulation#code-style? It should
run the linters and prevent commits that don't satisfy them.
—
Reply to this email directly, view it on GitHub
https://github.com/ThePalaceProject/circulation/pull/292#issuecomment-1109937595,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACUG2YOZSGJC4OCNIAKARTVHADLNANCNFSM5T4XNI2Q
.
You are receiving this because you were mentioned.Message ID:
@.***>


Web view variables ordering

My issue is that I want my variables in the web view to keep the same ordering as in my CSV, however, they are currently being sorted.
I was going to simply workaround this issue by numbering my vars, but I ran into nunjucks errors when trying to do so.
Any space or special character in a variable name (For example: {{1 firstVar}} or {{1.firstVar}} throws the following error:
Template render error - (unknown path) [Line 1, Column 1268]
expected variable end

Not at all familiar with nunjucks, so this may be expected behaviour. I just wanted to point it out as a potential additional thing to fix.
The main ask here is to maintain ordering of variables from the dataset.

This is expected behavior from nunjucks!  Variables can't contain spaces and . is a special operator.  You could probably name them 1_firstVar, 2_secondVar, etc.
Var outputs are sorted because aside from the spreadsheets, there's no guarantee that the var will appear in every testcase.  This means for most implementations (which do not use spreadsheet), the ordering is not well defined.  The way I'd implement this request is probably by adding a varOrder: [...] field, or by making the columns reorderable on the frontend.


Feature/issue 29/account login

Related Feature: #29

Implement login page UI
Add tests for login page
TODO: integration with backend

Login page UI:

Before merging this we might want to complete these first:

use the logo file and not a screenshot for better quality
adjust tailwind classes to better render the UI dynamically between different sizes of screens (Might need to meet with me Youssef so that register and login pages are coherent)

This PR has been moved to another branch and has been resolved there due to a conflict with the current remote branch. Therefore, this PR had been closed without a merge.
Please click here to view the revised version of this PR.


about the dataset "df_interv_merged.pkl" ,"all_hourly_data.h5" ,"data_statics_.pkl" and "pid_list.pkl"

I didn't find the above datasets in the code, and there are no files in the MIMIC3 repository, so how are these datasets handled？

I did not find the df_interv_merged.pkl file either. However the all_hourly_data.h5 is gathered from mimic-extract and the others are generated in the static py file.

Did you find the df_interv_merged.pkl file?

Same problem, did you find the df_interv_merged.pkl file? @mirzafarhan7

Nope. I emailed the authors twice but got no response.

@jli0117 Hi. Would you be able to share the df_interv_merged.pkl  file as without it, the code cannot be run.

@jli0117 Hi. Would you be able to share the df_interv_merged.pkl file as without it, the code cannot be run.

Hi, have you ever got this file? Can you share with me if you know about the file? Thanks!

Nope. I emailed the authors twice but got no response.

Hello, did you manage to locate the file 'df_interv_merged.pkl'? Please let me know. Thankyou!

If you download the h5 file from https://github.com/MLforHealth/MIMIC_Extract called 'all_hourly_data.h5', you can extract the intervention dataset as follows:
# read intervention dataset interv = pd.read_hdf(DATA_FILEPATH, 'interventions') interv = interv.reset_index() interv = interv.drop(['subject_id', 'hadm_id'], axis=1) interv = interv.set_index(['icustay_id', 'hours_in'])
This should be equivalent to the 'df_interv_merged.pkl'.

Ha encontrado los dos conjuntos de datos de "all_hourly_data.h5" y "df_interv_merged.pkl", ¿puede enviármelos directamente, el tiempo reciente es relativamente ajustado.h5 conjunto de datos no tiene tiempo de ir a otro proyecto para obtener, gracias! @WoodPecker1111 @kaigegao @mahmoudibrahim98 @arra131 @mirzafarhan7

If you download the h5 file from https://github.com/MLforHealth/MIMIC_Extract called 'all_hourly_data.h5' and store it in DATA_FILEPATH, you can extract the intervention dataset as follows: # read intervention dataset interv = pd.read_hdf(DATA_FILEPATH, 'interventions') interv = interv.reset_index() interv = interv.drop(['subject_id', 'hadm_id'], axis=1) interv = interv.set_index(['icustay_id', 'hours_in'])
This should be equivalent to the 'df_interv_merged.pkl'.

Hi, Could you please share with me the input data of this model? Thanks!


Replace lesser with less than in error messages

Lesser shouldn't be used together with than, the preferred form is "less than". Also, it is a bit more consistent with other error messages

Hmm, the github check has bugged out or I configured something incorrectly. Same with SonarCloud, I guess it worked only partially :D Nevertheless, thanks for finding and fixing it


Problem creating another 3D Text Font Material

Basically I have a font file, Franklin Gothic, I have imported it to Unity and followed all of the steps mentioned in the 3DText ReadMe file. I'm not sure if this is a Unity bug or not but, I have tried importing two or three other fonts and it hasn't worked for them either.
I believe the problem lies with either the font file or how Unity handles font file importing. As you can see in the picture that looking at the selawk (example font) the Font Material shows up which I believe is the reason that it works correctly. However, after importing two or three fonts and none of them showed up like this one I feel as this is either a bug or maybe I just am getting something wrong or maybe even missing a step.
I have created the material and attached to a text prefab. I was able to successfully replicate the selawk in 3D but, none of the others. If anyone could give me a hand with this it would be very grateful.
Thank you!

@dtb49 I tried with Franklin Gothic font but I was not able to repo the issue. (screen shot below shows proper font & occlusion)

Imported TTF font file
Duplicated 3DTextSelawik material, renamed it to 3DTextFranklinGothic
Duplicated 3DTextPrefab, renamded it to 3DTextPrefabFranklinGothic
Assigned TTF font file's texture to 3DTextFranklinGothic material
Assigned 3DTextFranklinGothic material to 3DTextPrefabFraklinGothic

That is quite odd, after you did it I went back and duplicated the material and then changed out the texture over it, works fine now. I must have missed a step or something when creating the material before.

@dtb49 Thank you for your confirmation!


Breaking error when using Anchor or Text components - Uncaught TypeError: defaultValidatorFn is not a function

I have a brand new react/Vite project and when I add a simple Anchor or Text component the site crashes with the below error message.
  "dependencies": {
    "framer-motion": "^10.16.0",
    "grommet": "^2.33.1",
    "grommet-icons": "^4.11.0",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "styled-components": "^6.0.7"
  },
  "devDependencies": {
    "@types/react": "^18.0.15",
    "@types/react-dom": "^18.0.6",
    "@typescript-eslint/eslint-plugin": "^5.33.0",
    "@typescript-eslint/parser": "^5.33.0",
    "@vitejs/plugin-react": "^3.0.0",
    "autoprefixer": "^10.4.8",
    "eslint": "^8.21.0",
    "eslint-config-standard": "^17.0.0",
    "eslint-plugin-import": "^2.26.0",
    "eslint-plugin-n": "^15.2.4",
    "eslint-plugin-promise": "^6.0.0",
    "eslint-plugin-react": "^7.30.1",
    "postcss": "^8.4.16",
    "tailwindcss": "^3.1.8",
    "typescript": "^4.6.4",
    "vite": "^4.0.1",
    "vite-plugin-eslint": "^1.8.1"
  }

Grommet doesn't currently support styled-components v6. There is work in progress to support v6.

So, I need to revert to v5?

I just tried with version 5.3.10 and the same failure persists. Which version does Grommet work with?

Be sure to clear out your node_modules. It should work fine with most v5 versions. For example I'm using 5.3.11 in one situation as we speak.

Clearing it out again worked. Cheers mate! Thanks for the help!


fix: running on circuitpython

This adds changes that were need for circuitpython because of these differences:

no time.time() support. The majority of boards don't support the RTC which is required for time.time(). Time is only used to compute timeouts, so using ticks_ms() is also a suitable alternative, with appropriate "wrap-around" protection.

implementation name is circuitpython

I2C bus locking is required

The changes are implemented conditionally by platform to preserve the behavior on other platforms.
Some open questions:

what fixtures are needed to test on micropython and regular python?

Pushed an update to fix some lint errors when I ran locally, but otherwise this looks good.


The stack fails at last stage due to custom key store failing to connect with HSM cluster

When you choose to create custom key store, the stack gets stuck for more than 1.5 hours and then rolls back. The custom key store gets created but fails  to connect  with HSM cluster with error- "KMS cannot connect the custom key store to its CloudHSM cluster. Error code: USER_NOT_FOUND". I assume 'kmsuser' is not getting configured correctly.

@awspankj thanks for the report. We're in the process of preparing a heavily refactored version of this automation for publishing here in aws-samples. In the meantime, did you get a chance to inspect the cfn-init.log data to potentially better understand the issue? As mentioned in Troubleshooting stack creation.
I've also provided you separately with a pointer to the heavily refactored fork in case you'd like to try that version.

@awspankj it appears that the kmsuser was created, but at the point that the CloudHSM key store was being connected, the kmuser is in an inconsistent state. i.e. the user is not present on each of the two HSMs in the cluster. The connect operation fails due to the user not being present in all HSMs of the cluster.
I'm investigating why, under some circumstances, the user gets into that state. This failure appears to be a result of enhancing the code to use the cloudhsm-cli package vs the cloudhsm-client package.

Hi @Christopher @.***> Thanks for the update. This is interesting  and something new I learned.
After the stack was failing in the end, I learned to do it manually with latest cloudhsm CLI commands + console – attached are the steps.
Hope to see the github version working again!  Meanwhile I will test the fork version.
Warm Regards,
Pankaj Patil
Partner Solutions Architect WWPS
Mob- +919967936244
From: Christopher Kampmeier @.>
Reply to: aws-samples/aws-cloudhsm-cloudformation-template @.>
Date: Saturday, 13 May 2023 at 3:56 AM
To: aws-samples/aws-cloudhsm-cloudformation-template @.>
Cc: "Patil, Pankaj" @.>, Mention @.***>
Subject: Re: [aws-samples/aws-cloudhsm-cloudformation-template] The stack fails at last stage due to custom key store failing to connect with HSM cluster (Issue #12)
@awspankjhttps://github.com/awspankj it appears that the kmsuser was created, but at the point that the CloudHSM key store was being connected, the kmuser is in an inconsistent state. i.e. the user is not present on each of the two HSMs in the cluster. The connect operation fails due to the user not being present in all HSMs of the cluster.
I'm investigating why, under some circumstances, the user gets into that state. This failure appears to be a result of enhancing the code to use the cloudhsm-cli package vs the cloudhsm-client package.
—
Reply to this email directly, view it on GitHubhttps://github.com/aws-samples/aws-cloudhsm-cloudformation-template/issues/12#issuecomment-1546375847, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AZQPQMO7RRGATCUO75GF4OTXF22H7ANCNFSM6AAAAAAX4WPLZM.
You are receiving this because you were mentioned.Message ID: @.***>

@awspankj I reverted this repository to the changes prior to introducing use of the cloudhsm-cli package in place of the cloudhsm-client package so that the kmsuser creation is stable. I'll send a note to you once the newly refactored form of the overall automation is published to this repository. In the meantime, you can use the internal fork I referenced separately.

@Christopher @.***> I gave it a try just now:

The stack completed  (with parameter ‘custom key store’)
Cluster is in active state. However, Custom key store is not created.

@.***

The Kmsuser is not created

@.***

Creation of kmsuser succeeds when done manually:

@.***

Now using the kmsuser and anchor certificate stored  in secrets manager, custom key store is created from aws console.

@.***
Warm Regards,
Pankaj Patil
Partner Solutions Architect WWPS
Mob- +919967936244
From: Christopher Kampmeier @.>
Reply to: aws-samples/aws-cloudhsm-cloudformation-template @.>
Date: Sunday, 14 May 2023 at 9:43 PM
To: aws-samples/aws-cloudhsm-cloudformation-template @.>
Cc: "Patil, Pankaj" @.>, Mention @.***>
Subject: Re: [aws-samples/aws-cloudhsm-cloudformation-template] The stack fails at last stage due to custom key store failing to connect with HSM cluster (Issue #12)
@awspankjhttps://github.com/awspankj I reverted this repository to the changes prior to introducing use of the cloudhsm-cli package in place of the cloudhsm-client package so that the kmsuser creation is stable. I'll send a note to you once the newly refactored form of the overall automation is published to this repository. In the meantime, you can use the internal fork I referenced separately.
—
Reply to this email directly, view it on GitHubhttps://github.com/aws-samples/aws-cloudhsm-cloudformation-template/issues/12#issuecomment-1546935013, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AZQPQMKAUTPRM43US6RW4P3XGEABTANCNFSM6AAAAAAX4WPLZM.
You are receiving this because you were mentioned.Message ID: @.***>


!feat: migrate v1beta1 manifest

This moves module manager to lifecycle manager
TODOs:

[ ] Change all Docs
[x] Migrate to v1beta1 Manifest Generation in Lifecycle Manager
[x] Change Manifest Installs to have cardinality 1 instead of N
[x] Fix Tests
[x] CLI Change Request to only deploy Lifecycle Manager instead of Module Manager and not to verify module manager deployment: https://github.com/kyma-project/cli/pull/1513/files

closing in favour of https://github.com/kyma-project/lifecycle-manager/pull/390


Re-use graph display between client-display and dashboard

Currently each project implements its own version of graph display, they are extremely similar and should be re-used

obe


Remove trailing whitespace

This PR removes all the trailing whitespace, previously I had missed the test/ directory, this should be everything.

Coverage remained the same when pulling 3604e8597b0451fb9c0641c7aeb09648b555e94a on bryangarza:master into 4dce9d99a009fdec8a6b83a8cbc99d119a73c561 on jaredhanson:master.

Coverage remained the same when pulling 3604e8597b0451fb9c0641c7aeb09648b555e94a on bryangarza:master into 4dce9d99a009fdec8a6b83a8cbc99d119a73c561 on jaredhanson:master.

Closing.  Whitespace is not significant, and merging this only increases the chance that merging to other branches will cause conflicts.


2022-07-25-Rollup

This PR is a Rollup of multiple PRs

https://github.com/cagov/covid-static-data/pull/2427

Errant PR


Cleaned Up UI, Implemented Audio, Set TrackObject Size

Set size based on duration
Removed unnecessary components
Implemented Audio
Cleaned up UI

Made some updates (including audio!)

LGTM 🤩