i got SQLSTATE[HY000] [2002] Connection refused

i've running apache2 and mysql containers and i follow your tutorial but i got SQLSTATE[HY000] [2002] Connection refused

i work on macOS Sierra

ok Solved by adding MYSQL_HOST=mysql but there is no way to configure with <IP_ADDRESS> ? and same for redis ?

@Lopez-i Glab you got it to work. Thing is, these services are all runing in different containers, that's why you can't access other services through <IP_ADDRESS>.
Do you have a particular reason for wanting them accessible through <IP_ADDRESS>?

no i've not and i notice that laradock running very slowly

@Lopez-i I'm closing this since your issue has been fixed.
About LaraDock running slowly, please have a look at #353.


Replay a message in queue-triage-core via an HTTP endpoint

Given a Failed Message exists
When the Failed Message is marked for replay
Then the message is sent to queue on the original broker

Starting work

I've added a new field statusHistory sub-document to the failedMessage document to hold the status history (the field name might have been a bit of a giveaway).  The document when persisted will look like this (with the latest status update at position [0] in the array:
{
        "_id" : "ea0ec359-233a-4d27-b275-5686241fecc2",
        "destination" : {
                "brokerName" : "broker",
                "name" : "queue.name"
        },
        "sentDateTime" : ISODate("2017-07-26T22:40:38.937Z"),
        "failedDateTime" : ISODate("2017-07-26T22:40:38.937Z"),
        "content" : "Hello",
        "properties" : "{\"@class\":\"java.util.HashMap\"}",
        "statusUpdates" : [
                {
                        "status" : "RESENT",
                        "updatedDateTime" : ISODate("2017-07-26T22:40:43.356Z")
                },
                {
                        "status" : "RESEND",
                        "updatedDateTime" : ISODate("2017-07-26T22:40:41.569Z")
                },
                {
                        "status" : "FAILED",
                        "updatedDateTime" : ISODate("2017-07-26T22:40:38.937Z")
                }
        ]
}

👍

PR #50 submitted:


Full days (24h) vs date day difference

Actual behavior:
When using onlyBusinessDays=true, it doesn't take into account full days (i.e. 24h -> a PR opened at 23:59 on 2024-07-23 will be marked stale at 0:01 2024-07-24, even if only 2 minutes have passed).
When using onlyBusinessDays=false, it does take into account full days (i.e. a PR opened at 23:59 on 2024-07-23 will not be marked stale before 23:59 2024-07-24 because 24h haven't passed).
Expected behavior: time difference is taken into effect in both cases, not just date day difference.
Cause:
The use of differenceInBusinessDays and differenceInDays from date-fsn cause this

differenceInBusinessDays counts based on date-day
differenceInDays counts based on date-time

Hi, thanks for your issue. Would you please consider create a PR to fix this bug?


07-11 11:38:30.743 24857 24981 E AndroidRuntime:        at libmediapipe_jni.tflite::gpu::cl::(anonymous namespace)::InferenceEnvironmentImpl::NewInferenceBuilder(tflite::gpu::cl::InferenceOptions const&, tflite::gpu::GraphFloat32, std::__ndk1::unique_ptr<tflite::gpu::InferenceBuilder, std::__ndk1::default_deletetflite::gpu::InferenceBuilder >)(:900)
07-11 11:38:30.743 24857 24981 E AndroidRuntime:        at libmediapipe_jni.tflite::gpu::TFLiteGPURunner::InitializeOpenCL(std::__ndk1::unique_ptr<tflite::gpu::InferenceBuilder, std::__ndk1::default_deletetflite::gpu::InferenceBuilder >)(InitializeOpenCL:764)
07-11 11:38:30.743 24857 24981 E AndroidRuntime:        at libmediapipe_jni.tflite::gpu::TFLiteGPURunner::Build()(Build:348)
07-11 11:38:30.743 24857 24981 E AndroidRuntime:        at libmediapipe_jni.mediapipe::api2::InferenceCalculatorGlImpl::InitTFLiteGPURunner(mediapipe::CalculatorContext)(InitTFLiteGPURunner:2280)
07-11 11:38:30.743 24857 24981 E AndroidRuntime:        at libmediapipe_jni.std::__ndk1::__function::__func<mediapipe::GlContext::Run(std::__ndk1::function<absl::lts_20210324::Status ()>, int, mediapipe::Timestamp)::$_8, std::__ndk1::allocator<mediapipe::GlContext::Run(std::__ndk1::function<absl::lts_20210324::Status ()>, int, mediapipe::Timestamp)::$_8>, absl::lts_20210324::Status ()>::operator()()(Run:48)
07-11 11:38:30.743 24857 24981 E AndroidRuntime:        at libmediapipe_jni.mediapipe::GlContext::DedicatedThread::Run(std::__ndk1::function<absl::lts_20210324::Status ()>)(Run:92)
07-11 11:38:30.743 24857 24981 E AndroidRuntime:        at libmediapipe_jni.mediapipe::GlContext::Run(std::__ndk1::function<absl::lts_20210324::Status ()>, int, mediapipe::Timestamp)(Run:752)
07-11 11:38:30.743 24857 24981 E AndroidRuntime:        at libmediapipe_jni.mediapipe::GlCalculatorHelperImpl::RunInGlContext(std::__ndk1::function<absl::lts_20210324::Status ()>, mediapipe::CalculatorContext*)(RunInGlContext:208)
07-11 11:38:30.743 24857 24981 E AndroidRuntime:        at libmediapipe_jni.mediapipe::GlCalculatorHelper::RunInGlContext(std::__ndk1::function<absl::lts_20210324::Status ()>)(RunInGlContext:208)
07-11 11:38:30.743 24857 24981 E AndroidRuntime:        at libmediapipe_jni.mediapipe::api2::InferenceCalculatorGlImpl::Open(mediapipe::CalculatorContext*)(Open:1288)
07-11 11:38:30.743 24857 24981 E AndroidRuntime:        at libmediapipe_jni.mediapipe::CalculatorNode::OpenNode()(OpenNode:840)
07-11 11:38:30.743 24857 24981 E AndroidRuntime:        at libmediapipe_jni.mediapipe::internal::SchedulerQueue::OpenCalculatorNode(mediapipe::CalculatorNode*)(OpenCalculatorNode:276)
07-11 11:38:30.743 24857 24981 E AndroidRuntime:        at libmediapipe_jni.mediapipe::internal::SchedulerQueue::RunNextTask()(RunNextTask:220)
07-11 11:38:30.743 24857 24981 E AndroidRuntime:        at libmediapipe_jni.std::__ndk1::__function::__func<mediapipe::GlContext::RunWithoutWaiting(std::__ndk1::function<void ()>)::$_9, std::__ndk1::allocator<mediapipe::GlContext::RunWithoutWaiting(std::__ndk1::function<void ()>)::$_9>, void ()>::operator()()(RunWithoutWaiting:52)
07-11 11:38:30.743 24857 24981 E AndroidRuntime:        at libmediapipe_jni.mediapipe::GlContext::DedicatedThread::ThreadBody()(ThreadBody:276)
07-11 11:38:30.743 24857 24981 E AndroidRuntime:        at libmediapipe_jni.mediapipe::GlContext::DedicatedThread::ThreadBody(void*)(ThreadBody:8)
07-11 11:38:30.743 24857 24981 E AndroidRuntime:        at libc.__pthread_start(void*)(__pthread_start:264)
07-11 11:38:30.743 24857 24981 E AndroidRuntime:        at libc.__start_thread(__start_thread:64)
07-11 11:38:30.822 24857 24857 I Unity   : onPause

It depends on your implementation.
If you don't want to flip the image on the screen, try setting AnnotationController#isMirrored to true.
https://github.com/homuler/MediaPipeUnityPlugin/blob/6efa5e1b75b7c6aa487254ade15fe7cd49fc6842/Assets/MediaPipeUnity/Samples/Common/Scripts/Solution.cs#L65

facemesh from oficial solution run on same M62... Still trying to understand...

If you succeeded to resolve the error, may I ask how you did it?
I can't find it right away, but I think I've had the same issue reported before.

facemesh from oficial solution run on same M62... Still trying to understand...

If you succeeded to resolve the error, may I ask how you did it? I can't find it right away, but I think I've had the same issue reported before.

I'm sorry for the late reply.
I saw that the same apk is running on older smartphones with 2Gb, 4Gb ram and my M62 has 8Gb ram. Since the update to Android 12 of the M62 in Brazil, a feature called Ram Plus has been enabled in the memory settings. On the Samsung website, they explain that this feature allows you to add from 2Gb to 8Gb (virtual) in RAM, which use storage space. I have 128Gb of internal storage space, with an additional 64Gb MicroSD card installed in this device. Samsung says it uses internal memory (I suspect not when you have MicroSD installed). The cause of the problem, as I understand it, is that the plugin is requesting frames (as the errors were in Movenext() ) faster than memory can store with this Ram Plus enabled. Yesterday, I set the use of this Ram Plus to a minimum (it was at 8GB, I changed it to 2GB because there is no way to disable this use of Ram Plus). From this moment, the applications started to work. If I activate 4Gb from Ram Plus, the application no longer works. It only runs with minimum enabled. I believe that the System must occupy these 2Gb first so that it can use the Ram later (since this feature, in my opinion, allows you to leave the RAM free for applications, although, according to Samsung, they will be the last to be used). I don't know if this is correct, but it worked for me.

@portari Thanks!

@homuler It is I who am grateful for your immeasurable help.

Everything is on place.

If this code is useful to anyone, it is available at
https://github.com/portari/MediaPipeUnityPlugin_PoseSampleGPU_Android.git

@portari Thanks!

Update:
It's not correct. After working for 10 days in a row, with several updates, today it didn't want to open anymore. I set several RAM Plus settings (with 8Gb, 4Gb, 2Gb) and nothing worked. Finally, I set 6Gb and it worked. I still don't understand why this occurs, but I still think it's related to this memory.


[server][common] Fixed bug in AA/WC parallel processing support

Summary, imperative, start upper case, don't end with a period
This PR fixed the following issues:

AASIT should pass non-null KeyLevelLocksManager to IngestionBatchProcessor, otherwise, race condition will happen.
Fixed the locking order in IngestionBatchProcessor to avoid deadlock.
Updated SparseConcurrentList#computeIfAbsent to skip adjust list size if the computed result is null.

How was this PR tested?
CI

Does this PR introduce any user-facing changes?

[x] No. You can skip the rest of this section.
[ ] Yes. Make sure to explain your proposed changes and call out the behavior change.

Sorry, I am doing mostly white line commenting in this round of review. I should have reviewed the first round of implementation of this functionality, but didn't get a chance, so I'm coming to the party a bit late. I just want to provide the following feedback:
The part of produceToStoreBufferServiceOrKafkaInBatch which splits the records coming from Kafka into "mini batches" seems not ideal to me, and I feel like we should get rid of it. To be clear, I don't think the current code cannot work, but just that it is more complex than it should be. I will list below some issues/concerns that I see with it, and all of these can be fixed individually, but I still think getting rid of that part of the algorithm could be overall better:

The batch size comes from serverConfig.getAAWCWorkloadParallelProcessingThreadPoolSize(), which does not seem to make sense. The default value of that config is 8, which is likely too small. I think the intent was probably to have a separate config for the batch size.
If the Kafka batch is smaller than the configured batch size, then we still instantiate an ongoingBatch of the configured size, which is wasteful in terms of memory allocation (easy to fix, but is it needed?).
I expect that the batch size coming from Kafka should not be too large to handle, and so there is no point in splitting it. If anything, I would think there may be cases where the batch from Kafka is too small and that the optimal thing may be to accumulate records for some time and then create a mega batch (the opposite of a mini batch!).
One of the benefits we hope to get from this functionality is to process together the updates to the same key, so that we do fewer RocksDB reads and writes in those cases. This effect is reduced by the mini batches.

So my recommendation would be to simplify this code by deleting the mini batching altogether, and just pass the Kafka batch directly into the next step. No need to implement the mega batching yet, we can make that call later.

The biggest reason of mini-batching is to make the key-level lock usage deterministic...
If we always pass what we received to IngestionBatchProcessor, the KeyLevelLockManager will need to consider the max possible batch size, which is 100 IIRC.
4) is not implemented yet, so the smaller batch size in theory shouldn't affect performance much.
When working on 4), we need to use a separate config to control the batch size, so that we can tune the memory usage/performance benefits gained by de-duping.
For 2), today's interface is Iterable, so we can't get the size without iterating it and to reduce the unnecessary copy, we need to change the interface to use something like List.


Fix items being dropped by the player not appearing in the DespawnedItems loot pool.

Quick mixin to make items dropped by the player on death retain their ownership.

Same idea but slightly better mixin here
https://github.com/TreeOfSelf/PandaArchaeology/commit/1dc704be6dedd10aa105f5bf8efdeb0e1995c79f
thanks


Unable to start consumer with remote Kafka Broker, keep getting Kafka::InvalidReplicationFactor error

If this is a bug report, please fill out the following:

Version of Ruby: 2.6.3
Version of Kafka: 2.6.0
Version of ruby-kafka: 1.2.0

Please verify that the problem you're seeing hasn't been fixed by the current master of ruby-kafka.
Steps to reproduce
kafka = Kafka.new(...)
consumer = kafka.consumer(group_id: 'il-consumer')
consumer.subscribe(ENV['PROD_TOPIC'], start_from_beginning: false)

consumer.each_message do |message|
        puts message.value
end

Expected outcome
The consumer would start, and listening to messages coming from a remote Kafka Broker.
Actual outcome

Traceback (most recent call last):
        19: from InstrumentationLayerConsumer.rb:65:in `<main>'
        18: from InstrumentationLayerConsumer.rb:54:in `main'
        17: from /Library/Ruby/Gems/2.6.0/gems/ruby-kafka-1.2.0/lib/kafka/consumer.rb:222:in `each_message'
        16: from /Library/Ruby/Gems/2.6.0/gems/ruby-kafka-1.2.0/lib/kafka/consumer.rb:417:in `consumer_loop'
        15: from /Library/Ruby/Gems/2.6.0/gems/ruby-kafka-1.2.0/lib/kafka/instrumenter.rb:35:in `instrument'
        14: from /Library/Ruby/Gems/2.6.0/gems/ruby-kafka-1.2.0/lib/kafka/instrumenter.rb:23:in `instrument'
        13: from /Library/Ruby/Gems/2.6.0/gems/ruby-kafka-1.2.0/lib/kafka/consumer.rb:419:in `block in consumer_loop'
        12: from /Library/Ruby/Gems/2.6.0/gems/ruby-kafka-1.2.0/lib/kafka/consumer.rb:223:in `block in each_message'
        11: from /Library/Ruby/Gems/2.6.0/gems/ruby-kafka-1.2.0/lib/kafka/consumer.rb:546:in `fetch_batches'
        10: from /Library/Ruby/Gems/2.6.0/gems/ruby-kafka-1.2.0/lib/kafka/consumer.rb:475:in `join_group'
         9: from /Library/Ruby/Gems/2.6.0/gems/ruby-kafka-1.2.0/lib/kafka/consumer_group.rb:49:in `join'
         8: from /Library/Ruby/Gems/2.6.0/gems/ruby-kafka-1.2.0/lib/kafka/consumer_group.rb:183:in `synchronize'
         7: from /Library/Ruby/Gems/2.6.0/gems/ruby-kafka-1.2.0/lib/kafka/round_robin_assignment_strategy.rb:27:in `assign'
         6: from /Library/Ruby/Gems/2.6.0/gems/ruby-kafka-1.2.0/lib/kafka/round_robin_assignment_strategy.rb:27:in `flat_map'
         5: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/set.rb:338:in `each'
         4: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/set.rb:338:in `each_key'
         3: from /Library/Ruby/Gems/2.6.0/gems/ruby-kafka-1.2.0/lib/kafka/round_robin_assignment_strategy.rb:29:in `block in assign'
         2: from /Library/Ruby/Gems/2.6.0/gems/ruby-kafka-1.2.0/lib/kafka/cluster.rb:183:in `partitions_for'
         1: from /Library/Ruby/Gems/2.6.0/gems/ruby-kafka-1.2.0/lib/kafka/protocol/metadata_response.rb:134:in `partitions_for'
/Library/Ruby/Gems/2.6.0/gems/ruby-kafka-1.2.0/lib/kafka/protocol.rb:160:in `handle_error': Kafka::InvalidReplicationFactor (Kafka::InvalidReplicationFactor)

I used telnet and confirmed that the Kafka Broker is running. Describing the topic gives me this:

devops@kafka:~/kafka_2.12-2.5.0$ bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic rsil
Topic: rsil     PartitionCount: 1       ReplicationFactor: 1    Configs:
        Topic: rsil     Partition: 0    Leader: none    Replicas: 0     Isr:

I seem to have fixed the issue, using the following steps.

Stop the server.
Delete the meta.properties file in the /tmp/kafka_logs directory.
Restart the server.

Still not sure what the problem was, but seems like it's fixed.


Model can't be checkpointed with Keras+MultiworkerMirroredStrategy

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template
System information
Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Darwin-18.0.0-x86_64-i386-64bit
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 2.0.0-beta1
Python version:3.6.8
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:
You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with: 1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" 2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"
Describe the current behavior
This is a followup for https://github.com/tensorflow/tensorflow/issues/31070
I tried following 2 solutions
1.I applied 6345ad5
to my tensorflow installed code
2. I install latest nightly dev build
both gave me following error, seems though previous commit change data type to int64, somewhere else still expects int32
2019-08-01 22:41:51.971726: W tensorflow/core/framework/op_kernel.cc:1546] OP_REQUIRES failed at collective_ops.cc:354 : Internal: RecvBufResponse returned 8 bytes where to_tensor expected 4
Traceback (most recent call last):
  File "example_tf2.py", line 124, in <module>
    steps_per_epoch = parallel_steps)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/training.py", line 643, in fit
    use_multiprocessing=use_multiprocessing)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/training_distributed.py", line 776, in wrapper
    mode=dc.CoordinatorMode.INDEPENDENT_WORKER)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/distribute/distribute_coordinator.py", line 853, in run_distribute_coordinator
    task_id, session_config, rpc_layer)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/distribute/distribute_coordinator.py", line 360, in _run_single_worker
    return worker_fn(strategy)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/training_distributed.py", line 771, in _worker_fn
    return fn(instance, model, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/training_distributed.py", line 681, in fit
    steps_name='steps_per_epoch')
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/training_arrays.py", line 294, in model_iteration
    batch_outs = f(actual_inputs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/distribute/distributed_training_utils.py", line 813, in execution_function
    return [out.numpy() for out in distributed_function(input_fn)]
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/eager/def_function.py", line 416, in __call__
    self._initialize(args, kwds, add_initializers_to=initializer_map)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/eager/def_function.py", line 359, in _initialize
    *args, **kwds))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/eager/function.py", line 1360, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/eager/function.py", line 1648, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/eager/function.py", line 1541, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/func_graph.py", line 716, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/eager/def_function.py", line 309, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/func_graph.py", line 706, in wrapper
    raise e.ag_error_metadata.to_exception(type(e))
tensorflow.python.autograph.impl.api.StagingError: in converted code:

    /usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/distribute/distributed_training_utils.py:804 distributed_function  *
        outputs = strategy.experimental_run_v2(
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/distribute/distribute_lib.py:708 experimental_run_v2
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/distribute/distribute_lib.py:1710 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/distribute/mirrored_strategy.py:708 _call_for_each_replica
        fn, args, kwargs)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/distribute/mirrored_strategy.py:195 _call_for_each_replica
        coord.join(threads)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/coordinator.py:389 join
        six.reraise(*self._exc_info_to_raise)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/coordinator.py:297 stop_on_exception
        yield
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/distribute/mirrored_strategy.py:926 run
        self.main_result = self.main_fn(*self.main_args, **self.main_kwargs)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/training.py:908 train_on_batch
        output_loss_metrics=self._output_loss_metrics)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/training_eager.py:307 train_on_batch
        output_loss_metrics=output_loss_metrics))
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/training_eager.py:260 _process_single_batch
        model.trainable_weights))
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:434 apply_gradients
        self._create_hypers()
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:608 _create_hypers
        aggregation=tf_variables.VariableAggregation.ONLY_FIRST_REPLICA)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:770 add_weight
        aggregation=aggregation)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/tracking/base.py:713 _add_variable_with_custom_getter
        **kwargs_for_getter)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/base_layer_utils.py:154 make_variable
        shape=variable_shape if variable_shape else None)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py:260 __call__
        return cls._variable_v1_call(*args, **kwargs)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py:221 _variable_v1_call
        shape=shape)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py:60 getter
        return captured_getter(captured_previous, **kwargs)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/distribute/shared_variable_creator.py:69 create_new_variable
        v = next_creator(*args, **kwargs)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py:60 getter
        return captured_getter(captured_previous, **kwargs)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/distribute/distribute_lib.py:1250 creator_with_resource_vars
        return self._create_variable(*args, **kwargs)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py:368 _create_variable
        _real_mirrored_creator, *args, **kwargs)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/distribute/mirrored_strategy.py:251 _create_mirrored_variable
        value_list = real_mirrored_creator(devices, *args, **kwargs)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py:355 _real_mirrored_creator
        v = next_creator(*args, **kwargs)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py:60 getter
        return captured_getter(captured_previous, **kwargs)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/eager/def_function.py:347 variable_capturing_scope
        lifted_initializer_graph=lifted_initializer_graph, **kwds)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py:264 __call__
        return super(VariableMetaclass, cls).__call__(*args, **kwargs)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/eager/def_function.py:139 __init__
        initial_value() if init_from_fn else initial_value,
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py:330 _overridden_initial_value_fn
        group_key, collective_instance_key)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/collective_ops.py:161 broadcast_recv
        instance_key=instance_key)
    /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_collective_ops.py:66 collective_bcast_recv
        _six.raise_from(_core._status_to_exception(e.code, message), None)
    /root/.local/lib/python2.7/site-packages/six.py:737 raise_from
        raise value

    InternalError: RecvBufResponse returned 8 bytes where to_tensor expected 4 [Op:CollectiveBcastRecv]

Describe the expected behavior
Keras model could be checkpoint-ed under multi worker training
Code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.
from __future__ import absolute_import, division, print_function, unicode_literals
import datetime
import json
import os
import tensorflow_datasets as tfds
import tensorflow as tf
import subprocess
import shlex
import sys

tfds.disable_progress_bar()

BUFFER_SIZE = 60000
BATCH_SIZE = 64

NUM_WORKERS = 2
GLOBAL_BATCH_SIZE = NUM_WORKERS * BATCH_SIZE

if __name__ == "__main__":
  worker_addrs = ['localhost:9999', 'localhost:9998']
  os.environ['TF_CONFIG'] = json.dumps({
      'cluster': {
          'worker': worker_addrs,
      },
      'task': {'type': 'worker', 'index': int(sys.argv[1])}
  })

  strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()

  def scale(image, label):
    image = tf.cast(image, tf.float32)
    image /= 255
    return image, label

  def build_and_compile_cnn_model():
    model = tf.keras.Sequential([
        tf.keras.layers.Conv2D(32, 3, activation='relu', input_shape=(28, 28, 1)),
        tf.keras.layers.MaxPooling2D(),
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(64, activation='relu'),
        tf.keras.layers.Dense(10, activation='softmax')
    ])
    model.compile(
        loss=tf.keras.losses.sparse_categorical_crossentropy,
        optimizer=tf.keras.optimizers.SGD(learning_rate=0.001),
        metrics=['accuracy'])
    return model

  datasets, info = tfds.load(name='mnist',
                             with_info=True,
                             as_supervised=True)

  train_datasets_unbatched = datasets['train'].map(scale).shuffle(BUFFER_SIZE)

  train_datasets = train_datasets_unbatched.batch(GLOBAL_BATCH_SIZE)

  with strategy.scope():
    multi_worker_model = build_and_compile_cnn_model()

  checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(
        filepath='/tmp/chk.hdf5',
        monitor='val_loss',
        save_best_only=True,
        load_weights_on_restart=True)

  multi_worker_model.fit(x=train_datasets, epochs=100, callbacks = [checkpoint_callback])

Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

@rchao

@jtang7, I don't seem to be able to repro this. Can you try tf nightly and see if it fixes the problem?

Guess this is a known issue of tf 2.0?
At the time of writing, fault tolerance is supported when eager execution is disabled (see following code snippet for how this can be done). TensorFlow team is actively investing in making fault tolerance available in eager execution as well.
https://www.tensorflow.org/beta/tutorials/distribute/multi_worker_with_keras#fault_tolerance

@PigApple eager execution used to only affect worker failure-recovery, not regular training with model.fit(), plus now this is resolved (tutorial is now being updated). Would like to learn if this still happens with tf-nightly.

@rchao Thanks! Will double check with nightly build and get back here

@rchao, thanks for the reply. Besides tf-nightly, will there be a tf 2.0 new beta version soon or we will directly jump to GA?

We're making progress for 2.0 release and feel free to join TensorFlow announcement for the latest updates. Thanks!

Using the MultiworkerMirrored example I
TF2.1.0
print("Found Cluster spec ", cluster)
print("Found TFConfig", tf_config)
os.environ['TF_CONFIG'] = json.dumps(tf_config)
BUFFER_SIZE = 1000
BATCH_SIZE = 8
def make_datasets_unbatched():
#Scaling MNIST data from (0, 255] to (0., 1.]
def scale(image, label):
image = tf.cast(image, tf.float16)
image /= 255
return image, label
datasets, info = tfds.load(name='mnist', with_info=True, as_supervised=True)
return datasets['train'].map(scale, num_parallel_calls=tf.data.experimental.AUTOTUNE).cache().repeat()
strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()
print("Created strategy " , strategy)
NUM_WORKERS = strategy.num_replicas_in_sync
print("Number of workers " ,NUM_WORKERS )
GLOBAL_BATCH_SIZE = 1 * NUM_WORKERS
print("Creating datasets inside scope...")
train_datasets = make_datasets_unbatched().batch(GLOBAL_BATCH_SIZE)
print("Created datasets " , train_datasets)
options = tf.data.Options()
options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.DATA  # AutoShardPolicy.OFF can work too.
train_datasets_no_auto_shard = train_datasets.with_options(options)
def build_and_compile_cnn_model():
model = tf.keras.Sequential([
tf.keras.layers.Flatten(input_shape=(28,28,1)),
tf.keras.layers.Dense(256, activation='relu'),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer=tf.keras.optimizers.Adam(),
metrics=['accuracy'])
return model
if my_task_index == 0:
callbacks = [tf.keras.callbacks.TensorBoard(log_dir='logs'), tf.keras.callbacks.ModelCheckpoint(filepath="models", save_best_only=False)]
else:
callbacks = None
with strategy.scope():
print("Creating model inside scope...")
model = build_and_compile_cnn_model()
print(model.summary())
print("Starting to fit model....")
verbose = 1 if my_task_index == 0 else 0
model.fit(x=train_datasets_no_auto_shard, epochs=100, steps_per_epoch=150, callbacks = callbacks , verbose=verbose)
print("Model fit completed")
Yields .....which goes away if your disable Modelcheckpoint.
Starting to fit model....
Traceback (most recent call last):
File "main_2.py", line 72, in
model.fit(x=train_datasets_no_auto_shard, epochs=100, steps_per_epoch=150, callbacks = callbacks , verbose=verbose)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training.py", line 819, in fit
use_multiprocessing=use_multiprocessing)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_distributed.py", line 790, in fit
*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_distributed.py", line 777, in wrapper
mode=dc.CoordinatorMode.INDEPENDENT_WORKER)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/distribute_coordinator.py", line 853, in run_distribute_coordinator
task_id, session_config, rpc_layer)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/distribute_coordinator.py", line 360, in _run_single_worker
return worker_fn(strategy)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_distributed.py", line 772, in _worker_fn
return method(model, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 342, in fit
total_epochs=epochs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 128, in run_one_epoch
batch_outs = execution_function(iterator)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 98, in execution_function
distributed_function(input_fn))
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/def_function.py", line 568, in call
result = self._call(*args, **kwds)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/def_function.py", line 615, in _call
self._initialize(args, kwds, add_initializers_to=initializers)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/def_function.py", line 497, in _initialize
*args, **kwds))
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/function.py", line 2389, in _get_concrete_function_internal_garbage_collected
graph_function, _, _ = self._maybe_define_function(args, kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/function.py", line 2703, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/function.py", line 2593, in _create_graph_function
capture_by_value=self._capture_by_value),
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/func_graph.py", line 978, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/def_function.py", line 439, in wrapped_fn
return weak_wrapped_fn().wrapped(*args, **kwds)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 85, in distributed_function
per_replica_function, args=args)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/distribute_lib.py", line 763, in experimental_run_v2
return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/distribute_lib.py", line 1819, in call_for_each_replica
return self._call_for_each_replica(fn, args, kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 694, in _call_for_each_replica
fn, args, kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 201, in _call_for_each_replica
coord.join(threads)
Traceback (most recent call last):
File "main_2.py", line 72, in
model.fit(x=train_datasets_no_auto_shard, epochs=100, steps_per_epoch=150, callbacks = callbacks , verbose=verbose)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training.py", line 819, in fit
use_multiprocessing=use_multiprocessing)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_distributed.py", line 790, in fit
*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_distributed.py", line 777, in wrapper
mode=dc.CoordinatorMode.INDEPENDENT_WORKER)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/distribute_coordinator.py", line 853, in run_distribute_coordinator
task_id, session_config, rpc_layer)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/distribute_coordinator.py", line 360, in _run_single_worker
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/home/pi/.local/lib/python3.7/site-packages/six.py", line 703, in reraise
raise value
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/coordinator.py", line 297, in stop_on_exception
yield
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 917, in run
self.main_result = self.main_fn(*self.main_args, **self.main_kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/autograph/impl/api.py", line 292, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 433, in train_on_batch
output_loss_metrics=model._output_loss_metrics)
return worker_fn(strategy)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_distributed.py", line 772, in _worker_fn
return method(model, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 342, in fit
total_epochs=epochs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 128, in run_one_epoch
batch_outs = execution_function(iterator)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 98, in execution_function
distributed_function(input_fn))
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/def_function.py", line 568, in call
result = self._call(*args, **kwds)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/def_function.py", line 615, in _call
self._initialize(args, kwds, add_initializers_to=initializers)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_eager.py", line 312, in train_on_batch
output_loss_metrics=output_loss_metrics))
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_eager.py", line 273, in _process_single_batch
model.optimizer.apply_gradients(zip(grads, trainable_weights))
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py", line 433, in apply_gradients
self._create_hypers()
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py", line 655, in _create_hypers
aggregation=tf_variables.VariableAggregation.ONLY_FIRST_REPLICA)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py", line 817, in add_weight
aggregation=aggregation)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/def_function.py", line 497, in _initialize
*args, **kwds))
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/function.py", line 2389, in _get_concrete_function_internal_garbage_collected
graph_function, _, _ = self._maybe_define_function(args, kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/function.py", line 2703, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/function.py", line 2593, in _create_graph_function
capture_by_value=self._capture_by_value),
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/func_graph.py", line 978, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/tracking/base.py", line 744, in _add_variable_with_custom_getter
**kwargs_for_getter)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py", line 142, in make_variable
shape=variable_shape if variable_shape else None)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/variables.py", line 258, in call
return cls._variable_v1_call(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/variables.py", line 219, in _variable_v1_call
shape=shape)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/variables.py", line 65, in getter
return captured_getter(captured_previous, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/shared_variable_creator.py", line 69, in create_new_variable
v = next_creator(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/def_function.py", line 439, in wrapped_fn
return weak_wrapped_fn().wrapped(*args, **kwds)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 85, in distributed_function
per_replica_function, args=args)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/distribute_lib.py", line 763, in experimental_run_v2
return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/distribute_lib.py", line 1819, in call_for_each_replica
return self._call_for_each_replica(fn, args, kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 694, in _call_for_each_replica
fn, args, kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/variables.py", line 65, in getter
return captured_getter(captured_previous, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/distribute_lib.py", line 1330, in creator_with_resource_vars
return self._create_variable(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 548, in _create_variable
values.SyncOnReadVariable, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/values.py", line 1034, in create_mirrored_variable
value_list = real_mirrored_creator(devices, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 540, in _real_mirrored_creator
v = next_creator(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/variables.py", line 65, in getter
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 201, in _call_for_each_replica
coord.join(threads)
return captured_getter(captured_previous, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/def_function.py", line 485, in variable_capturing_scope
lifted_initializer_graph=lifted_initializer_graph, **kwds)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/home/pi/.local/lib/python3.7/site-packages/six.py", line 703, in reraise
raise value
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/coordinator.py", line 297, in stop_on_exception
yield
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 917, in run
self.main_result = self.main_fn(*self.main_args, **self.main_kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/autograph/impl/api.py", line 292, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 433, in train_on_batch
output_loss_metrics=model._output_loss_metrics)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/variables.py", line 262, in call
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_eager.py", line 312, in train_on_batch
output_loss_metrics=output_loss_metrics))
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_eager.py", line 273, in _process_single_batch
model.optimizer.apply_gradients(zip(grads, trainable_weights))
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py", line 433, in apply_gradients
self._create_hypers()
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py", line 655, in _create_hypers
aggregation=tf_variables.VariableAggregation.ONLY_FIRST_REPLICA)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py", line 817, in add_weight
aggregation=aggregation)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/tracking/base.py", line 744, in _add_variable_with_custom_getter
**kwargs_for_getter)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py", line 142, in make_variable
shape=variable_shape if variable_shape else None)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/variables.py", line 258, in call
return cls._variable_v1_call(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/variables.py", line 219, in _variable_v1_call
shape=shape)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/variables.py", line 65, in getter
return captured_getter(captured_previous, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/shared_variable_creator.py", line 69, in create_new_variable
v = next_creator(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/variables.py", line 65, in getter
return captured_getter(captured_previous, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/distribute_lib.py", line 1330, in creator_with_resource_vars
return self._create_variable(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 548, in _create_variable
values.SyncOnReadVariable, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/values.py", line 1034, in create_mirrored_variable
value_list = real_mirrored_creator(devices, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 540, in _real_mirrored_creator
v = next_creator(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/variables.py", line 65, in getter
return captured_getter(captured_previous, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/def_function.py", line 485, in variable_capturing_scope
lifted_initializer_graph=lifted_initializer_graph, **kwds)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/variables.py", line 262, in call
return super(VariableMetaclass, cls).call(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/def_function.py", line 178, in init
initial_value() if init_from_fn else initial_value,
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/collective_all_reduce_strategy.py", line 383, in initial_value_fn
collective_instance_key)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/collective_ops.py", line 176, in broadcast_recv
communication_hint=communication_hint.lower())
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/gen_collective_ops.py", line 57, in collective_bcast_recv
_ops.raise_from_not_ok_status(e, name)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 6606, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.InternalError: RecvBufResponse returned 8 bytes where to_tensor expected 4 [Op:CollectiveBcastRecv]
return super(VariableMetaclass, cls).call(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/def_function.py", line 178, in init
initial_value() if init_from_fn else initial_value,
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/distribute/collective_all_reduce_strategy.py", line 383, in initial_value_fn
collective_instance_key)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/collective_ops.py", line 176, in broadcast_recv
communication_hint=communication_hint.lower())
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/gen_collective_ops.py", line 57, in collective_bcast_recv
_ops.raise_from_not_ok_status(e, name)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 6606, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.InternalError: RecvBufResponse returned 8 bytes where to_tensor expected 4 [Op:CollectiveBcastRecv]

@ianferreira is it possible for you to test this again on TF nightly? (or 2.2rc) We think there was a fix that was made after 2.1 that might have fixed this issue.

This is still an issue in 2.2.0-rc4. I ran into the same error:
2020-05-16 21:41:22.789209: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at collective_ops.cc:401 : Internal: RecvBufResponse returned 8 bytes where to_tensor expected 4
INFO:tensorflow:Error reported to Coordinator: RecvBufResponse returned 8 bytes where to_tensor expected 4 [Op:CollectiveBcastRecv]

Tried to enable checkpoints through tf.keras.callbacks.ModelCheckpoint while running with MultiWorkerMirroredStrategy. This fails deterministically at the end of the first epoch. The error went away when I remove the callback.

@guptapriya could you point me to the commit that you think might have fixed this? I want to see if I can cherry-pick it.

Hi @andrewor14, if you are still facing this issue can you please provide reproducible code + stack trace so we can try to debug? Thanks.

from datetime import datetime
from packaging import version
import os
import tensorflow as tf
import numpy as np
import json
# Create a TensorBoard callback
logs = "logs/" + datetime.now().strftime("%Y%m%d-%H%M%S")
tboard_callback = tf.keras.callbacks.TensorBoard(log_dir = logs,
                                                 histogram_freq = 1,
                                                 profile_batch = '2048')
os.environ['TF_CONFIG'] = json.dumps({
    'cluster': {
        'worker': ["node67:12345", "node68:23456"]
    },
    'task': {'type': 'worker', 'index': 1}
})
def mnist_dataset(batch_size):
    (x_train, y_train), _ = tf.keras.datasets.mnist.load_data()
    # The `x` arrays are in uint8 and have values in the range [0, 255].
    # We need to convert them to float32 with values in the range [0, 1]
    x_train = x_train / np.float32(255)
    y_train = y_train.astype(np.int64)
    train_dataset = tf.data.Dataset.from_tensor_slices(
      (x_train, y_train)).shuffle(60000).repeat().batch(batch_size).prefetch(100)
    return train_dataset

def build_and_compile_cnn_model():
    model = tf.keras.Sequential([
      tf.keras.Input(shape=(28, 28)),
      tf.keras.layers.Reshape(target_shape=(28, 28, 1)),
      tf.keras.layers.Conv2D(256, 2, activation='relu'),
      tf.keras.layers.Conv2D(128, 2, activation='relu'),
      tf.keras.layers.Conv2D(32, 1, activation='relu'),
      tf.keras.layers.Conv2D(32, 2, activation='relu'),
      tf.keras.layers.Flatten(),
      tf.keras.layers.Dense(2048, activation='relu'),
      tf.keras.layers.Dense(1024, activation='relu'),
      tf.keras.layers.Dense(128, activation='relu'),
      tf.keras.layers.Dense(10)
    ])
    model.compile(
      loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
      optimizer=tf.keras.optimizers.SGD(learning_rate=0.001),
      metrics=['accuracy'])
    return model
strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()
num_workers = 2
per_worker_batch_size = 2048
# Here the batch size scales up by number of workers since
# `tf.data.Dataset.batch` expects the global batch size. Previously we used 64,
# and now this becomes 128.
global_batch_size = per_worker_batch_size * num_workers
multi_worker_dataset = mnist_dataset(global_batch_size)

with strategy.scope():
  multi_worker_model = build_and_compile_cnn_model()
multi_worker_model.fit(multi_worker_dataset, epochs=103, steps_per_epoch=70,callbacks = [tboard_callback])

@ @---------------------------------------------------------------------------
InternalError                             Traceback (most recent call last)
<ipython-input-7-402c6ad248bd> in <module>
     14 # number of steps per epoch. Note that the numbers here are for demonstration
     15 # purposes only and may not sufficiently produce a model with good quality.
---> 16 multi_worker_model.fit(multi_worker_dataset, epochs=103, steps_per_epoch=70,callbacks = [tboard_callback])

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py in _method_wrapper(self, *args, **kwargs)
     73         lambda _: method(self, *args, **kwargs),
     74         self.distribute_strategy,
---> 75         mode=dc.CoordinatorMode.INDEPENDENT_WORKER)
     76
     77   return tf_decorator.make_decorator(

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/distribute/distribute_coordinator.py in run_distribute_coordinator(worker_fn, strategy, eval_fn, eval_strategy, mode, cluster_spec, task_type, task_id, session_config, rpc_layer)
    851         # All jobs run `worker_fn` if between-graph.
    852         return _run_single_worker(worker_fn, strategy, cluster_spec, task_type,
--> 853                                   task_id, session_config, rpc_layer)
    854       else:
    855         # Only one node runs `worker_fn` if in-graph.

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/distribute/distribute_coordinator.py in _run_single_worker(worker_fn, strategy, cluster_spec, task_type, task_id, session_config, rpc_layer, worker_barrier, coord)
    358         return worker_fn(strategy)
    359     else:
--> 360       return worker_fn(strategy)
    361
    362

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py in <lambda>(_)
     71
     72     return dc.run_distribute_coordinator(
---> 73         lambda _: method(self, *args, **kwargs),
     74         self.distribute_strategy,
     75         mode=dc.CoordinatorMode.INDEPENDENT_WORKER)

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
    828       self.stop_training = False
    829       train_function = self.make_train_function()
--> 830       callbacks.on_train_begin()
    831       # Handle fault-tolerance for multi-worker.
    832       # TODO(omalleyt): Fix the ordering issues that mean this has to

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/keras/callbacks.py in on_train_begin(self, logs)
    445     logs = self._process_logs(logs)
    446     for callback in self.callbacks:
--> 447       callback.on_train_begin(logs)
    448
    449   def on_train_end(self, logs=None):

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/keras/callbacks.py in on_train_begin(self, logs)
   1948
   1949   def on_train_begin(self, logs=None):
-> 1950     self._init_batch_steps()
   1951     if self._start_batch == 1:
   1952       self._enable_trace()

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/keras/callbacks.py in _init_batch_steps(self)
   1893       self._total_batches_seen = {
   1894           self._train_run_name: variables.Variable(0, dtype='int64'),
-> 1895           self._validation_run_name: variables.Variable(0, dtype='int64')
   1896       }
   1897     else:

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/ops/variables.py in __call__(cls, *args, **kwargs)
    259       return cls._variable_v1_call(*args, **kwargs)
    260     elif cls is Variable:
--> 261       return cls._variable_v2_call(*args, **kwargs)
    262     else:
    263       return super(VariableMetaclass, cls).__call__(*args, **kwargs)

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/ops/variables.py in _variable_v2_call(cls, initial_value, trainable, validate_shape, caching_device, name, variable_def, dtype, import_scope, constraint, synchronization, aggregation, shape)
    253         synchronization=synchronization,
    254         aggregation=aggregation,
--> 255         shape=shape)
    256
    257   def __call__(cls, *args, **kwargs):

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/ops/variables.py in getter(**kwargs)
     64
     65   def getter(**kwargs):
---> 66     return captured_getter(captured_previous, **kwargs)
     67
     68   return getter

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/distribute/distribute_lib.py in creator_with_resource_vars(next_creator, **kwargs)
   1765         kwargs["initial_value"] = kwargs["initial_value"].wrapped_value
   1766
-> 1767       return self._create_variable(next_creator, **kwargs)
   1768
   1769     def distributed_getter(getter, *args, **kwargs):

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/distribute/mirrored_strategy.py in _create_variable(self, next_creator, **kwargs)
    608                                            _real_mirrored_creator,
    609                                            values.MirroredVariable,
--> 610                                            values.SyncOnReadVariable, **kwargs)
    611
    612   def _validate_colocate_with_variable(self, colocate_with_variable):

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/distribute/values.py in create_mirrored_variable(strategy, real_mirrored_creator, mirrored_cls, sync_on_read_cls, **kwargs)
    692   # here.
    693   with tape.stop_recording():
--> 694     value_list = real_mirrored_creator(**kwargs)
    695     var_cls = sync_on_read_cls if is_sync_on_read else mirrored_cls
    696     result = var_cls(strategy, value_list, aggregation)

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/distribute/mirrored_strategy.py in _real_mirrored_creator(**kwargs)
    600             # variable creation.
    601             with tape.stop_recording():
--> 602               v = next_creator(**kwargs)
    603           assert not isinstance(v, values.DistributedVariable)
    604           value_list.append(v)

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/ops/variables.py in <lambda>(**kws)
    234                         shape=None):
    235     """Call on Variable class. Useful to force the signature."""
--> 236     previous_getter = lambda **kws: default_variable_creator_v2(None, **kws)
    237     for _, getter in ops.get_default_graph()._variable_creator_stack:  # pylint: disable=protected-access
    238       previous_getter = _make_getter(getter, previous_getter)

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py in default_variable_creator_v2(next_creator, **kwargs)
   2645       synchronization=synchronization,
   2646       aggregation=aggregation,
-> 2647       shape=shape)
   2648
   2649

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/ops/variables.py in __call__(cls, *args, **kwargs)
    261       return cls._variable_v2_call(*args, **kwargs)
    262     else:
--> 263       return super(VariableMetaclass, cls).__call__(*args, **kwargs)
    264
    265

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py in __init__(self, initial_value, trainable, collections, validate_shape, caching_device, name, dtype, variable_def, import_scope, constraint, distribute_strategy, synchronization, aggregation, shape)
   1432           aggregation=aggregation,
   1433           shape=shape,
-> 1434           distribute_strategy=distribute_strategy)
   1435
   1436   def _init_from_args(self,

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py in _init_from_args(self, initial_value, trainable, collections, caching_device, name, dtype, constraint, synchronization, aggregation, distribute_strategy, shape)
   1565           with ops.name_scope("Initializer"), device_context_manager(None):
   1566             initial_value = ops.convert_to_tensor(
-> 1567                 initial_value() if init_from_fn else initial_value,
   1568                 name="initial_value", dtype=dtype)
   1569           if shape is not None:

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py in initial_value_fn()
    384                                                    initial_value.dtype,
    385                                                    group_size, group_key,
--> 386                                                    collective_instance_key)
    387           return initial_value
    388

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/ops/collective_ops.py in broadcast_recv(shape, dtype, group_size, group_key, instance_key, communication_hint)
    174       group_key=group_key,
    175       instance_key=instance_key,
--> 176       communication_hint=communication_hint.lower())

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/ops/gen_collective_ops.py in collective_bcast_recv(T, group_size, group_key, instance_key, shape, communication_hint, name)
     55         pass  # Add nodes to the TensorFlow graph.
     56     except _core._NotOkStatusException as e:
---> 57       _ops.raise_from_not_ok_status(e, name)
     58   # Add nodes to the TensorFlow graph.
     59   T = _execute.make_type(T, "T")

~/miniconda2/envs/tf2/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in raise_from_not_ok_status(e, name)
   6651   message = e.message + (" name: " + name if name is not None else "")
   6652   # pylint: disable=protected-access
-> 6653   six.raise_from(core._status_to_exception(e.code, message), None)
   6654   # pylint: enable=protected-access
   6655

~/.local/lib/python3.6/site-packages/six.py in raise_from(value, from_value)

InternalError: RecvBufResponse returned 4 bytes where to_tensor expected 8 [Op:CollectiveBcastRecv]

Thanks for providing this example @nimaaghli. I'm running this script in 2.2 but it seems to be training without problems, though I haven't made it to the last epoch yet. Does this fail right away for you, or after a certain number of epochs? What TF version are you using?

I had the Tensorboard configured only on one worker and thought I can show results from only one worker. After replicating  Tensorboard callback code on other workers as well, the error goes away.

Great! I'll close this issue now since checkpointing with Keras and MWMS is now working. Feel free to reopen if anything feels unresolved.

@nikitamaia could you point me to the commit(s) that may have fixed this? I'd like to see if I can cherry-pick them into my branch if necessary.

I am facing the same issue with 2.3 tf

For anyone who is struggling with RecvBufResponse returned 4 bytes where to_tensor expected 8.
Try disable all your callbacks first.
(If it works for you, pls add a +1 reaction ;)

For those who are still stuck on this error, I was able to get rid of it by making sure every worker has the same set of callbacks. In my case, I only added the ModelCheckpoint callback in the first worker, e.g.
if is_master:
  callbacks.add(tf.keras.callbacks.ModelCheckpoint(...))

This caused the RecvBufResponse error in TF 2.2.0, and cause hanging in TF 2.4.0. Removing the is_master check solved the problem for me. IMO this should be better documented, or at least the error message should be more descriptive.


Suggestion: Change chown command when copying apps

When following the steps in the copying apps exercise: https://colorado.rstudio.com/rsc/pro-admin-training/using-connect-exercise/#section-task-copy-the-example-apps
The command given in the exercise gives an error when I run it:
admin-user@ip-10-8-8-93:/home/jen/pro_admin_training_assets$ sudo chown -R jen:jen /home/jen
chown: invalid group: ‘jen:jen’

In order to get the output shown in the exercise (owned by jen in group root), I had to run this command:
sudo chown -R jen /home/jen
If I dropped jen from the group, and ran this command:
sudo chown -R jen: /home/jen
Then the ownership changed to jen in group rstudio-connect.
For example:
drwxr-xr-x 2 jen rstudio-connect     4096 Jan  2 12:53 00_app

I tried the command from this folder too:
admin-user@ip-10-8-8-93:/usr/share/class/pro_admin_training_assets$ sudo chown -R jen:jen /home/jen
chown: invalid group: ‘jen:jen’

AFAIK, I have already updated the instructions.  It should now say:
sudo chown jen: .....

And there should be a drop-down that explains why this works.
(The command chown jen:jen will definitely not work, since jen isn't a valid group.)
https://colorado.rstudio.com/rsc/pro-admin-training/security-rsp-exercise/#section-task-create-user-accounts

Closing this issue

You did make the change above, but it is the instructions on the copying apps exercise that need to be updated.  The screenshot below is from this page: https://colorado.rstudio.com/rsc/pro-admin-training/using-connect-exercise/#section-task-copy-the-example-apps


[Feature][Daemon] Support daemon on windows

Type
Feature
Description
In this PR, I implemented the daemon functionality on Windows as requested in issue #32.
Implementation
Since os.fork() and the python-daemon library cannot be used on Windows, I used the win32service and win32serviceutil modules to achieve similar functionality.
We registered a service named NEETBOX on Windows and installed and started it in daemon/__init__.py.
Notes

⚠ Because installing the service requires Windows administrator privileges, please test it with administrator privileges.

@visualDust cc. @therainisme

? 意想不到的人来搅局了 (x)

? 意想不到的人来搅局了 (x)


Plot trajectory batch

Annotate where last move was played (green).
Bottom axis shows channel stats. Title displays step number.

Usage should plot only a batch of 2


got error when starting the webui

[W 220404 15:02:18 run:413] phantomjs not found, continue running without it.
[I 220404 15:02:20 result_worker:49] result_worker starting...
[I 220404 15:02:20 processor:211] processor starting...
[I 220404 15:02:20 tornado_fetcher:638] fetcher starting...
[I 220404 15:02:20 scheduler:647] scheduler starting...
[I 220404 15:02:20 scheduler:782] scheduler.xmlrpc listening on <IP_ADDRESS>:23333
[I 220404 15:02:20 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0
[I 220404 15:02:20 app:84] webui exiting...
Traceback (most recent call last):
File "/usr/local/Caskroom/miniconda/base/envs/web/bin/pyspider", line 8, in
sys.exit(main())
File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/pyspider/run.py", line 754, in main
cli()
File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 1637, in invoke
super().invoke(ctx)
File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/pyspider/run.py", line 165, in cli
ctx.invoke(all)
File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/pyspider/run.py", line 497, in all
ctx.invoke(webui, **webui_config)
File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/pyspider/run.py", line 384, in webui
app.run(host=host, port=port)
File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/pyspider/webui/app.py", line 59, in run
from .webdav import dav_app
File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/pyspider/webui/webdav.py", line 207, in
'/': ScriptProvider(app)
TypeError: Can't instantiate abstract class ScriptProvider with abstract methods get_resource_inst

我遇到了类似的问题，启动webui提示找不到字典

I have the same problem as well. My environment is:

Ubuntu 20.04
Python 3.6 installed from ppa:deadsnakes (alongside with default Python3.8 package)
Python development libraries are installed
Running in a virtual environment

And the application console log is:
$ pyspider
[W 220426 09:19:52 run:413] phantomjs not found, continue running without it.
[I 220426 09:19:54 result_worker:49] result_worker starting...
[I 220426 09:19:54 processor:211] processor starting...
[I 220426 09:19:54 tornado_fetcher:638] fetcher starting...
[I 220426 09:19:54 scheduler:647] scheduler starting...
[I 220426 09:19:54 scheduler:782] scheduler.xmlrpc listening on <IP_ADDRESS>:23333
[I 220426 09:19:54 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0
[I 220426 09:19:55 app:84] webui exiting...
Traceback (most recent call last):
  File "/home/amenmd/myfs/temp/intergen/omim/env36/bin/pyspider", line 8, in <module>
    sys.exit(main())
  File "/home/amenmd/myfs/temp/intergen/omim/env36/lib/python3.6/site-packages/pyspider/run.py", line 754, in main
    cli()
  File "/home/amenmd/myfs/temp/intergen/omim/env36/lib/python3.6/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/amenmd/myfs/temp/intergen/omim/env36/lib/python3.6/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/amenmd/myfs/temp/intergen/omim/env36/lib/python3.6/site-packages/click/core.py", line 1637, in invoke
    super().invoke(ctx)
  File "/home/amenmd/myfs/temp/intergen/omim/env36/lib/python3.6/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/amenmd/myfs/temp/intergen/omim/env36/lib/python3.6/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/amenmd/myfs/temp/intergen/omim/env36/lib/python3.6/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/amenmd/myfs/temp/intergen/omim/env36/lib/python3.6/site-packages/pyspider/run.py", line 165, in cli
    ctx.invoke(all)
  File "/home/amenmd/myfs/temp/intergen/omim/env36/lib/python3.6/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/amenmd/myfs/temp/intergen/omim/env36/lib/python3.6/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/amenmd/myfs/temp/intergen/omim/env36/lib/python3.6/site-packages/pyspider/run.py", line 497, in all
    ctx.invoke(webui, **webui_config)
  File "/home/amenmd/myfs/temp/intergen/omim/env36/lib/python3.6/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/amenmd/myfs/temp/intergen/omim/env36/lib/python3.6/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/amenmd/myfs/temp/intergen/omim/env36/lib/python3.6/site-packages/pyspider/run.py", line 384, in webui
    app.run(host=host, port=port)
  File "/home/amenmd/myfs/temp/intergen/omim/env36/lib/python3.6/site-packages/pyspider/webui/app.py", line 59, in run
    from .webdav import dav_app
  File "/home/amenmd/myfs/temp/intergen/omim/env36/lib/python3.6/site-packages/pyspider/webui/webdav.py", line 207, in <module>
    '/': ScriptProvider(app)
TypeError: Can't instantiate abstract class ScriptProvider with abstract methods get_resource_inst

同问？如何解决呢？

[W 220404 15:02:18 run:413] phantomjs not found, continue running without it. [I 220404 15:02:20 result_worker:49] result_worker starting... [I 220404 15:02:20 processor:211] processor starting... [I 220404 15:02:20 tornado_fetcher:638] fetcher starting... [I 220404 15:02:20 scheduler:647] scheduler starting... [I 220404 15:02:20 scheduler:782] scheduler.xmlrpc listening on <IP_ADDRESS>:23333 [I 220404 15:02:20 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0 [I 220404 15:02:20 app:84] webui exiting... Traceback (most recent call last): File "/usr/local/Caskroom/miniconda/base/envs/web/bin/pyspider", line 8, in sys.exit(main()) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/pyspider/run.py", line 754, in main cli() File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 1128, in call return self.main(*args, **kwargs) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 1637, in invoke super().invoke(ctx) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/pyspider/run.py", line 165, in cli ctx.invoke(all) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/pyspider/run.py", line 497, in all ctx.invoke(webui, **webui_config) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/pyspider/run.py", line 384, in webui app.run(host=host, port=port) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/pyspider/webui/app.py", line 59, in run from .webdav import dav_app File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/pyspider/webui/webdav.py", line 207, in '/': ScriptProvider(app) TypeError: Can't instantiate abstract class ScriptProvider with abstract methods get_resource_inst

看了源码，抽象类定义的是get_resource_inst，而webdav里定义的是getResourceInst，所以抽象类找不到对应的构建函数，就报错了

应该是webdav的问题，建议不要修改源码，而是给webdav降版本

同时还有问题的应该还有werkzeug这个包，也是版本更新后有些模块路径变了

pip install wsgidav==2.4.1
pip uninstall flask
pip install werkzeug==0.16.0 flask


thread ID reported incorrectly in procfs

On rs_preview build 14905:
The per-thread "stat" files under /proc seem to report the "pid" differently from Linux.  For example, if I have a process with pid 57003, containing thread 57004, on Linux I see this:

cat /proc/57003/task/57004/stat
57004 (dotnet-run) S 56995 56995 56807 34827 56995<PHONE_NUMBER> 3 41412 0 23 0 0 84 8 20 0 15 0 328515728<PHONE_NUMBER> 7893<PHONE_NUMBER>3709551615 4194304 4243012<PHONE_NUMBER>11440<PHONE_NUMBER>22640<PHONE_NUMBER>83341 0<PHONE_NUMBER> 4096 1246<PHONE_NUMBER>1586489339 0 0 -1 1 0 0 0 0 0 6344088 6345208 27213824<PHONE_NUMBER>12824<PHONE_NUMBER>12857<PHONE_NUMBER>12857<PHONE_NUMBER>14871 0

Note that the first field in the "stat" file is "57004", which is the thread ID, not the process ID.
On Windows, the per-thread "stat" file seems to report the process ID in this field, when it should be the thread ID.
FWIW, this is causing some .NET Core tests to fail, due to System.Diagnostics.Process making use of the contents of these files.

Thanks, @ericeil for reporting this. I have opened a bug at our end to track this, which should get auto-triaged.

I have checked in a fix for this.

@misenesi How/when should we expect to see the effects of your fix?

@CIPop - I would say ~2 weeks if I had to guess.  Since you're also a Microsoft employee I'm sure you can relate to how hard it can be to track when things reach Insider builds. Shoot me a mail and I can tell you which internal branch the fix is in.

This should be fixed on build 14986 out today.
14986 Release Notes


Add the UI toggle for changing theme

Fixes

Fixes #4309 by @zackkrida
Fixes #4155 by @fcoveram
Description

This PR

adds icon assets and i18n strings for theme switching
updates the VSelectField component to hide the active choice
creates the UI component for switching between color modes

wires the component to functionality implemented in #4810
adds the UI component to the footer based on whether the "dark_mode_ui_toggle" feature flag is enabled

deletes the "force_dark_mode" feature flag as it is redundant

deletes usage and tests for the feature flag

Testing Instructions

Check out the PR and run the dev server.
Enable the "dark_mode_ui_toggle" feature flag. You should see the theme selector in the footer.
Switch between light and dark theme. The site should update to show the your chosen color theme (regardless of the system theme). The icon in the footer should also update.
Switch between the OS light and dark theme. The text for the "System" choice in the theme selector should update.
Switch to the "System" theme.
Switch between the OS light and dark theme. The text for the "System" choice in the theme selector should update. The actual dark/light theme should also update. The icon in the footer should also update.

Checklist

[x] My pull request has a descriptive title (not a vague title likeUpdate index.md).
[x] My pull request targets the default branch of the repository (main) or a parent feature branch.
[x] My commit messages follow best practices.
[x] My code follows the established code style of the repository.
[x] I added or updated tests for the changes I made (if applicable).
[x] I added or updated documentation (if applicable).
[x] I tried running the project locally and verified that there are no visible errors.
[ ] I ran the DAG documentation generator (ov just catalog/generate-docs for catalog
PRs) or the media properties generator (ov just catalog/generate-docs media-props
for the catalog or ov just api/generate-docs for the API) where applicable.

Developer Certificate of Origin

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

This PR is drafted till #4810 is merged and labelled as "⛔ status: blocked" (to prevent accidentally merging it). Once that is merged, it will be rebased and undrafted. Feel free to review in the meantime, I do not anticipate UI/functionality changes.

@zackkrida that is so weird, I'm not seeing the select field sizing issue on any major browser! This is Safari, Arc (Chromium) and Firefox:

@dhruvkb it looks like removing overflow-hidden from the VSelectField wrapper div fixes the collapsing issue. The problem makes sense to me: a flex child with overflow-hidden and two absolutely-positioned children would collapse, as I understand.
I'm seeing it in Chrome and Firefox on Fedora. In any case, I think we can safely remove the overflow hiding.

Will implement as soon as possible.
I cannot see the white outline.

Firefox:

Chrome:


NextJs - user sign in on /api/auth/user succeeds in local development. Fails Production

Bug report
user sign in on /api/auth/user succeeds in local development. Fails Production
Describe the bug
For some weird reason, withPageAuth and withApiAuth or [...supabase].ts fail to authenticate in production when I deploy to Vercel.
However, in development, everything works fine. Specifically, I get a 404 error when the client tries to call the [...supabase].ts
To Reproduce
Try deploy the nextjs example to vercel.
Expected behavior
Authentication Fails and you are sent to the redirect path. Call to /api/auth/[...supabase].ts returns 404
System information

OS: Works on Windows. Fails on Vercel
Browser Edge, Safari, Firefox
Version of supabase-js: 1.35.3
Version of Node.js: 16

I believe This is not the first time this has happened, and there was a fix for it but I am afraid something similar has returned again

I am sorry. I found the mistake. I had an API folder in the root directory where all my external API functions were implemented, but apparently, vercel looks at a folder called that in that directory. Renaming that folder to something other than API makes everything work again.


Write unit tests for CodeInterpreter

[ ] Write unit tests for DataStructureExtractor.py
[ ] Write unit tests for TableExtractor.py
[ ] Write unit tests for TableToDataStructure.py
[ ] Write unit tests for Card.py
[ ] Write unit tests for CardData.py
[ ] Write unit tests for CombatManuever.py
[ ] Write unit tests for Feat.py
[ ] Write unit tests for MagicItem.py
[ ] Write unit tests for Spell.py
[ ] Write unit tests for FileHandler.py
[ ] Write unit tests for ObjectSaver.py
[ ] Write unit tests for a5eScraper.py
[ ] Write unit tests for TableScraper.py


[OCU-90] Make Dracon-dev into one Helm Package

Implements #150

Please split away the RFC part into a separate PR, we first review and merge that, then we can proceed to reviewing this, but only after it's working and the commit history has been cleaned up

thanks for the review, done


Add missing pt br relative time translation

Summary of changes
Msgid "now" is empty in the relative time translation file, returning english version
I'm having to do the translation inside my application instead of using the translation of the timex
# before
Timex.Translator.translate("pt_BR", "relative_time", "now")
=> "now"

# now
Timex.Translator.translate("pt_BR", "relative_time", "now")
"agora"

Checklist

[ ] New functions have typespecs, changed functions were updated
[ ] Same for documentation, including moduledocs
[ ] Tests were added or updated to cover changes
[x] Commits were squashed into a single coherent commit

Coverage increased (+0.04%) to 71.239% when pulling 4997abe8ec3f7629c3596340589fd0bc9104b6ff on joaovitoras:missing-pt-br-translation into 6ff35a5ae1ccaae632729444ca0aa993528f40c5 on bitwalker:master.

Thank you for the PR!

Closed without merge?

Closed without merge? @ckhrysze


Remove hardcoded IPs

Follow-up from discussion in https://github.com/servo/saltfs/issues/281 .
We have a bunch of IPs hardcoded into common/map.jinja. This was useful when we had to manage DNS through Mozilla support tickets, but we control everything directly through CloudFlare now.
Todo:

[ ] Remove the IPs from map.jinja
[ ] Remove the states which add the IPs to the hosts file
[ ] Test everything in Vagrant
[ ] Deploy changes to one buildslave, see if it still connects and runs builds successfully

Duplicate of https://github.com/servo/saltfs/issues/287

This is clearer and has better instructions than the other issue I opened, so closing it in favor of reopening this one.

Hi! I'm looking to contribute to Servo, and this was mentioned on Servo Starters, so I'm going to try it. Thanks!

@setupminimal It's yours, let us know if you have any questions!

Hey I'm a little confused, is this still an issue? Is this still assigned?

This actually got taken care of in https://github.com/servo/saltfs/pull/577/commits/95ab185a7be4cc5f245ea2c925ecf7185c0cf84a already, thanks for checking!


docs: update README.md for local setup instructions

added checking logs

Thanks for the update! Changes approved and merged.


super ball game

Description 📜
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context.
Fixes #issue_number_here

Type of change 📝

[ ] Au#dio (Should be in mp3 format Includes speech clarity, Concise ,Low distortion)
[ ] Vi#deo (Animations, screen-recordings, presentations and regular explanatory films are all possibilties etc)
[x] Doc#umentation (Content Creation in the form of codes or tutorials)
[ ] Other (If you choose other, Please mention changes below)

Domain of Contribution 📊

[ ] Android Dev #(Flutter)
[ ] Android Dev #(Java)
[ ] Android Dev #(Kotlin)
[ ] Backend Dev #(Java)
[ ] Backend Dev #(.NET)
[ ] Backend Dev #(PHP)
[ ] Backend Dev #(Python)
[x] C/#CPP
[ ] Competitive #Programming
[ ] Cyber #Security
[ ] DS#A
[ ] Data#base
[ ] Datascience with #Python
[ ] Datascience with #R
[ ] Frontend Dev #HTML/CSS/JS
[ ] Frontend Dev #React/Angular/Vue
[ ] Go#lang
[ ] Interview #Prep
[ ] Java_#Domain
[ ] JavaScript_#Domain
[ ] ME#RN
[ ] Machine #Learning
[ ] Open #Source
[ ] Python_#Domain
[ ] Ru#st
[ ] Statis#tics
[ ] UI/#UX

Checklist ✅

[x] I follow Contributing Guidelines & Code of conduct of this project.
[x] I have performed a self-review of my own code or work.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] My changes generates no new warnings.
[x] I'm GWOC'21 contributor

Screenshots / Gif (Optional) 📸

kindly merge this!


uninitialized constant StringIO (NameError)

Version: Chef Gem 12.4.1
Environment: CentOS 7
Scenario:
require 'chef/cookbook/metadata' in a ruby script
Steps to Reproduce:
Create a ruby script and in the require section add
require 'chef/cookbook/metadata'
Expected Result:
I am able to use the Metadata class to parse metadata.rb files
Actual Result:
Run time exception:
/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require': uninitialized constant    StringIO (NameError)
from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/local/share/gems/gems/ffi-yajl-2.2.1/lib/ffi_yajl/ext.rb:42:in `<class:Encoder>'
from /usr/local/share/gems/gems/ffi-yajl-2.2.1/lib/ffi_yajl/ext.rb:41:in `<module:FFI_Yajl>'
from /usr/local/share/gems/gems/ffi-yajl-2.2.1/lib/ffi_yajl/ext.rb:30:in `<top (required)>'
from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/local/share/gems/gems/ffi-yajl-2.2.1/lib/ffi_yajl.rb:45:in `<top (required)>'
from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/local/share/gems/gems/chef-12.4.1/lib/chef/json_compat.rb:20:in `<top (required)>'
from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/local/share/gems/gems/chef-12.4.1/lib/chef/cookbook/metadata.rb:28:in `<top (required)>'
from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:135:in `require'
from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:135:in `rescue in require'
from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:144:in `require'

Looks like the actual issue is in ffi-yajl? /cc @lamont-granquist

Yep, looks like I'll be pushing out 2.2.2 later this afternoon...

Upstream issue is fixed, closing this out.


Modal accesibility

Closes #498

@sabakhilji your build is failing. Do you need help investigating?

Yes, @ecurrencyhodler!! I don't know the reason for the build fail, this PR addressed a very small change though.

tACK! Great work getting this displayed! I think we need another code review b4 merging.

Sorry, this was not quite ready. The onClick function still has to be moved to the button, as I pointed out multiple times. It cannot be on the image.
When I use VoiceOver right now, I can highlight the different avatars. From a technical perspective, the button element is highlighted (not the image). But pressing does not do anything. That's because the onClick is not on the button.
key={i} can also be removed from the Avatar, since the mapping happens on the button element.

Hey Christoph. I created another issue based on your comment. Apologies for merging prematurely. Saba you can try to address it again if you want while you wait for #509 to be unblocked. If not, we can backlog this to the next cycle or wait for someone else to pick it up.

That's my bad, I thought it was resolved. I can pick it up.


chore: jsonfmt compute

Ran `jsonfmt -w "specification/compute/**/*.json"

Can one of the admins verify this patch?

Automation for azure-sdk-for-js

Automation for azure-sdk-for-python

Automation for azure-sdk-for-ruby

Automation for azure-sdk-for-go

Automation for azure-sdk-for-java


Integrate with SonarCloud and various code refactoring

Codecov Report

Merging #94 (640cf03) into main (e0d69f5) will decrease coverage by 0.15%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##               main      #94      +/-   ##
============================================
- Coverage     27.27%   27.11%   -0.16%
  Complexity       39       39
============================================
  Files            31       32       +1
  Lines           341      343       +2
  Branches         14       14
============================================
  Hits             93       93
- Misses          247      249       +2
  Partials          1        1

Flag
Coverage Δ

unittests
27.11% <0.00%> (-0.16%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

.../es/lnsd/videotape/core/backend/BackendLoader.java
0.00% <0.00%> (ø)

.../videotape/core/exception/MissingBackendError.java
0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e0d69f5...640cf03. Read the comment docs.


Discard unknown command/operation

When a message arrives with an unknown Header.command or Header.operation it must be safely discarded (and error-logged) — don't panic.
This could happen due to a bad client version.

Some more discussion in https://github.com/tigerbeetle/tigerbeetle/pull/1850


[api] support processDefineProps to process runtime props

Clear and concise description of the problem
It's a todo in the api/src/vue/analyze.ts to support runtime props.
Suggested solution

It may provide the following methods like handleObjectPropsDefinition and handleReferencePropsDefinition in theapi/src/vue/props.ts.
Or just one single method like handleRuntimePropsDefinition.

Alternative
No response
Additional context
I'd like to do this.
But I'm not sure about the type definitions of the methods and their return.
Should they be like TSProps?
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guide.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

Hmmm, the internal API can be one of them, which is up to the contributor. If you want to contribute to the feature, I'd like to accept it.


Add SimdUtilBenchmark

Differential Revision: D51555528

@Yuhta, can we put all the benchmarks under velox/benchmarks since it is easier to find them? Thanks.
CC: @pedroerp, @mbasmanova

can we put all the benchmarks under velox/benchmarks since it is easier to find them?

@majetideepak This may not scale well. Benchmarks are similar to tests. There are lots of them and they "belong" to different parts of the code. Perhaps, we could adopt a convention of having xxx/tests and xxx/benchmarks directories for each "production" xxx directory.

Perhaps, we could adopt a convention of having xxx/tests and xxx/benchmarks directories for each "production" xxx directory.

@mbasmanova I did not understand this statement. Can you please explain what you mean by "production"?

Benchmarks are similar to tests

Velox users would benefit from benchmarks on testing various platforms. Tests not so much. For example, the ReadBenchmark is helpful to analyze the throughput from a storage but is hidden somewhere in the Velox code. To make it harder, there are three different versions of this benchmark for each filesystem duplicating a lot of code.
I opened a PR here to combine them here: https://github.com/facebookincubator/velox/pull/8813

I agree with Masha to put micro benchmark under the directory with its own production code.  Only a few benchmark belonging to integration level should be moved to the common benchmark directory.  And with those benchmarks, we need to be careful about the dependencies and not making the common benchmark depending on the whole world.  Maybe we will need some subdirectory under the common benchmark directory at some point.
BTW this PR is already merged.


Login errors

People are getting random errors when trying to log in.  Some are able to refresh and get in so it's not a password issue.  Need to check logs.

this doesn't seem to be happening anymore.