TF 2.0 regression: cloudpickle cannot serialize tf.keras.Sequential.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes (code included below in the issue)
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS 10.14.3
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
TensorFlow installed from (source or binary): pip
TensorFlow version (use command below): v2.0.0-beta1-5101-gc75bb66a99 2.0.0-rc0
Python version: Python 3.6.7 :: Anaconda, Inc.
Bazel version (if compiling from source): N/A
GCC/Compiler version (if compiling from source): N/A
CUDA/cuDNN version: N/A
GPU model and memory: N/A

Using cloudpickle to serialize a Python function that uses tf.keras.Sequential fails with a recursion error.
Note that this works with tensorflow==1.14.0.
I imagine it also fails with other things, not just tf.keras.Sequential.
import cloudpickle  # cloudpickle.__version__ == '1.2.1'
import tensorflow as tf  # tf.__version__ == '2.0.0-rc0'

def f():
    tf.keras.Sequential

cloudpickle.loads(cloudpickle.dumps(f))  # This fails.

The last line fails with
---------------------------------------------------------------------------
RecursionError                            Traceback (most recent call last)
<ipython-input-23-25cc307e6227> in <module>
----> 1 cloudpickle.loads(cloudpickle.dumps(f))

~/anaconda3/lib/python3.6/site-packages/tensorflow/__init__.py in __getattr__(self, item)
     48
     49   def __getattr__(self, item):
---> 50     module = self._load()
     51     return getattr(module, item)
     52

~/anaconda3/lib/python3.6/site-packages/tensorflow/__init__.py in _load(self)
     42   def _load(self):
     43     """Import the target module and insert it into the parent's namespace."""
---> 44     module = _importlib.import_module(self.__name__)
     45     self._parent_module_globals[self._local_name] = module
     46     self.__dict__.update(module.__dict__)

... last 2 frames repeated, from the frame below ...

~/anaconda3/lib/python3.6/site-packages/tensorflow/__init__.py in __getattr__(self, item)
     48
     49   def __getattr__(self, item):
---> 50     module = self._load()
     51     return getattr(module, item)
     52

RecursionError: maximum recursion depth exceeded while calling a Python object

See https://stackoverflow.com/questions/57750920/ray-tensorflow-gpu-2-0-recursionerror/57761034#57761034

I have tried on colab with TF 1.14 and able to execute the code.However i am able to reproduce the issue with TF 2.0.0-rc0 and 2.0 nightly versions.Please, find the gist here.Thanks!

What is the goal of serialization here? We have several saving methods that allow you to save and revive a keras model that might be more appropriate here.

@karmel Note that this issue is not actually about serializing a keras model, but rather about serializing a function that creates a keras model. The APIs you refer to don't help in this case, I think.
Cloudpickle is the standard when it comes to general purpose serialization of arbitrary Python objects (including functions and classes). This is used by many distributed computing frameworks (like Ray, PySpark, Dask, IPython Parallel) that serialize arbitrary user-defined functions and ship them to remote worker processes to be executed.
As long as TensorFlow plays nicely with cloudpickle, then cloudpickle will be able to serialize arbitrary functions/classes that use TensorFlow. Serializing arbitrary functions/classes is most likely out of scope for TensorFlow, and so it makes sense to have cloudpickle handle that.

@yifeif Can you take a look at the cloud pickle issue? It looks like it's getting caught up in an infinite recursion loop in LazyLoader

Looks like we might need to handle getstate, setstate for the LazyLoader at virtual pip level? cc @mihaimaruseac

Seems that that is the case, __getstate__, __setstate__, __getinitargs__ and __getnewargs__.
I will send a fix later today/tomorrow.

Update: the issue comes from the unpickling part, as shown from the script below:
_p = print
import cloudpickle  # cloudpickle.__version__ == '1.2.1'
import tensorflow as tf  # tf.__version__ == '2.0.0-rc0'

def f():
  _p("f() called")
  tf.keras.Sequential
  _p("f() ending")

_p("Dumping...")
s = cloudpickle.dumps(f)
_p("dumped, loading...")
cloudpickle.loads(s)
_p("done")

This outputs:
Dumping...
dumped, loading...
Traceback (most recent call last):
  File "test.py", line 13, in <module>
    cloudpickle.loads(s)
  File "/tmp/gh/1/lib/python3.6/site-packages/tensorflow/__init__.py", line 51, in __getattr__
    _p("{}.__getattr__({})".format(self._local_name, item))
  File "/tmp/gh/1/lib/python3.6/site-packages/tensorflow/__init__.py", line 51, in __getattr__
    _p("{}.__getattr__({})".format(self._local_name, item))
  File "/tmp/gh/1/lib/python3.6/site-packages/tensorflow/__init__.py", line 51, in __getattr__
    _p("{}.__getattr__({})".format(self._local_name, item))
  [Previous line repeated 330 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object

Further investigation reveals that during unpickling __setattr__ needs to be called (equivalently, __setstate__ could be called but it needs at least one __setattr__ to store the new state on the module's __dict__). However, the lazy loading approach we're using assumes read only modules, we cannot add new attributes. Even defining an emtpy __setattr__ results in infinite recursion at the import tensorflow line.

Another update:
tf.keras.Sequential, tf.keras and tf.estimator all result in the infinite recursion errors.
tf.math.sin doesn't.

We cannot fix this in time for TF 2.0 final release. In fact, we cannot really fix this unless we give up Python 2 support, so we're looking at a fix that should come up by start of next year or so.
Sorry for the delay, but as we didn't support serialization via pickling we never tested if this functionality would get broken by our changes. We'll fix this in the future

Thanks for the update @mihaimaruseac. Fixing this in the future would be great. Out of curiosity, why does fixing it mean giving up support for Python 2?

We are using a custom lazy loader object to mimic functionality that is present only in Python3.5 and later to create some modules on the fly.

@mihaimaruseac I saw https://github.com/tensorflow/tensorflow/commit/4675891bd3c9e9ee7a57552486ec5bdc40379787 . Is it relevant to this issue?

I'll have to check this, as it is on a different path.

Is there any type of workaround for this? Running Ray and TF 2.0 and now facing this issue. Would be great to see a fix for this any time soon, rather than next year.

If you are just blocked by some framework that ships the serialized function you could bypass the tensorflow serialization by using importlib.import_module and the during deserialization making sure the module you use is shipped/available in the PYTHONPATH.
Something like:
mymodule.py
def tf_fn():
   tf.keras.Sequential

def f():
    module = importlib.import_module("mymodule")
    return module.tf_fn()

In our use case to run distributed TensorFlow on Hadoop we provide a safe_experiment function  function and then we upload the TensorFlow functions inside a module to the cluster. This works as a workaround for the moment with tf2.

@jharaldson the easiest workaround might be the one described in https://github.com/ray-project/ray/issues/5614#issuecomment-527292289.
Another workaround is described in https://stackoverflow.com/a/57761034/7858504

Coming back to the example at https://github.com/tensorflow/tensorflow/issues/32159#issuecomment-528016376
In python2 all works
(py2) mihaimaruseac@ankh:/tmp/pickle/py2$ python test.py
Dumping...
dumped, loading...
done

In python3.5 the error is from PyCapsule objects:
(py35) mihaimaruseac@ankh:/tmp/pickle/py35$ python test.py
Dumping...
Traceback (most recent call last):
  File "test.py", line 11, in <module>
    s = cloudpickle.dumps(f)
  File "/tmp/pickle/py35/lib/python3.5/site-packages/cloudpickle/cloudpickle.py", line 1125, in dumps
    cp.dump(obj)
  File "/tmp/pickle/py35/lib/python3.5/site-packages/cloudpickle/cloudpickle.py", line 482, in dump
    return Pickler.dump(self, obj)
  File "/usr/lib/python3.5/pickle.py", line 408, in dump
    self.save(obj)
  File "/usr/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/tmp/pickle/py35/lib/python3.5/site-packages/cloudpickle/cloudpickle.py", line 556, in save_function
    return self.save_function_tuple(obj)
  File "/tmp/pickle/py35/lib/python3.5/site-packages/cloudpickle/cloudpickle.py", line 758, in save_function_tuple
    save(state)
  File "/usr/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.5/pickle.py", line 814, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.5/pickle.py", line 840, in _batch_setitems
    save(v)
  File "/usr/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.5/pickle.py", line 774, in save_list
    self._batch_appends(obj)
  File "/usr/lib/python3.5/pickle.py", line 801, in _batch_appends
    save(tmp[0])
  File "/usr/lib/python3.5/pickle.py", line 520, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python3.5/pickle.py", line 627, in save_reduce
    save(state)
  File "/usr/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.5/pickle.py", line 814, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.5/pickle.py", line 840, in _batch_setitems
    save(v)
  File "/usr/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.5/pickle.py", line 814, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.5/pickle.py", line 840, in _batch_setitems
    save(v)
  File "/usr/lib/python3.5/pickle.py", line 495, in save
    rv = reduce(self.proto)
TypeError: can't pickle PyCapsule objects

In Python3.7 the error is from _LazyLoader
(py37) mihaimaruseac@ankh:/tmp/pickle/py37$ python test.py
Dumping...
Traceback (most recent call last):
  File "test.py", line 11, in <module>
    s = cloudpickle.dumps(f)
  File "/tmp/pickle/py37/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 1125, in dumps
    cp.dump(obj)
  File "/tmp/pickle/py37/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 482, in dump
    return Pickler.dump(self, obj)
  File "/usr/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/tmp/pickle/py37/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 556, in save_function
    return self.save_function_tuple(obj)
  File "/tmp/pickle/py37/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 758, in save_function_tuple
    save(state)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.7/pickle.py", line 816, in save_list
    self._batch_appends(obj)
  File "/usr/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(tmp[0])
  File "/usr/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
TypeError: can't pickle _LazyLoader objects

Can't test python3.6 anymore due to an issue in my system.
Manually applying the fix from 4675891bd3c9e9ee7a57552486ec5bdc40379787 to the other codepath makes that test pass in all 3 instances.
I'm running the change through more tests and will submit a fix.

Let's wait for a new tf-nightly and test this again. Or, you can build from source, with 353b8a1

Seems the fix works with Ray. However if we use custom layers with functions decorated with @tf.function there are still pickling issues. As a workaround for that I figured one could save the model as a "savedmodel" on a distributed storage and then have the ray worker load the model from the distributed storage, but this throws an error.
LookupError: No gradient defined for operation 'while' (op type: While)

Code to reproduce
import tensorflow as tf
import ray
import numpy as np

ray.init()

def build_save_model():
    lstm_in = tf.keras.Input(shape=(24,1))
    lstm_out = tf.keras.layers.LSTM(6)(lstm_in)
    dense_out = tf.keras.layers.Dense(24)(lstm_out)
    model = tf.keras.Model([lstm_in], dense_out)
    model.save('/path/in/common/storage/lstm_model')

@ray.remote
class Worker():
    def __init__(self):
        self.model = tf.keras.models.load_model('/path/in/common/storage/lstm_model')
        self.model.compile(optimizer=tf.keras.optimizers.Adam(1e-1), loss=tf.keras.losses.mse)
        self.data = np.arange(24).reshape(1,24,1)
        self.label = np.arange(24).reshape(1,24)

    def train(self):
        history = self.model.fit(self.data, self.label, epochs=10)
        return history.history

build_save_model()
lstm_worker = Worker.remote()
w = ray.get(lstm_worker.train.remote())

Error
---------------------------------------------------------------------------
RayTaskError                              Traceback (most recent call last)
<ipython-input-3-a18941ca631a> in <module>
     22 build_save_model()
     23 lstm_worker = Worker.remote()
---> 24 w = ray.get(lstm_worker.train.remote())

/opt/conda/lib/python3.6/site-packages/ray/worker.py in get(object_ids)
   2245             if isinstance(value, RayError):
   2246                 last_task_error_raise_time = time.time()
-> 2247                 raise value
   2248
   2249         # Run post processors.

RayTaskError: ray_worker (pid=1397, host=thesis-clustering-7dfb7867df-pk5fc)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 2326, in get_attr
    c_api.TF_OperationGetAttrValueProto(self._c_op, name, buf)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Operation 'StatefulPartitionedCall' has no attr named '_XlaCompile'.

During handling of the above exception, another exception occurred:

ray_worker (pid=1397, host=thesis-clustering-7dfb7867df-pk5fc)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/ops/gradients_util.py", line 331, in _MaybeCompile
    xla_compile = op.get_attr("_XlaCompile")
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 2330, in get_attr
    raise ValueError(str(e))
ValueError: Operation 'StatefulPartitionedCall' has no attr named '_XlaCompile'.

During handling of the above exception, another exception occurred:

ray_worker (pid=1397, host=thesis-clustering-7dfb7867df-pk5fc)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 2326, in get_attr
    c_api.TF_OperationGetAttrValueProto(self._c_op, name, buf)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Operation 'StatefulPartitionedCall' has no attr named '_XlaCompile'.

During handling of the above exception, another exception occurred:

ray_worker (pid=1397, host=thesis-clustering-7dfb7867df-pk5fc)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/ops/gradients_util.py", line 331, in _MaybeCompile
    xla_compile = op.get_attr("_XlaCompile")
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 2330, in get_attr
    raise ValueError(str(e))
ValueError: Operation 'StatefulPartitionedCall' has no attr named '_XlaCompile'.

During handling of the above exception, another exception occurred:

ray_worker (pid=1397, host=thesis-clustering-7dfb7867df-pk5fc)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/ops/gradients_util.py", line 607, in _GradientsHelper
    grad_fn = ops.get_gradient_function(op)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 2495, in get_gradient_function
    return _gradient_registry.lookup(op_type)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/framework/registry.py", line 97, in lookup
    "%s registry has no entry for: %s" % (self._name, name))
LookupError: gradient registry has no entry for: While

During handling of the above exception, another exception occurred:

ray_worker (pid=1397, host=thesis-clustering-7dfb7867df-pk5fc)
  File "<ipython-input-3-a18941ca631a>", line 19, in train
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 785, in fit
    use_multiprocessing=use_multiprocessing)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 337, in fit
    total_epochs=epochs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 127, in run_one_epoch
    batch_outs = execution_function(iterator)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 86, in execution_function
    distributed_function(input_fn))
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 568, in __call__
    result = self._call(*args, **kwds)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 615, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 497, in _initialize
    *args, **kwds))
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 2366, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 2675, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 2565, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/framework/func_graph.py", line 974, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 439, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 73, in distributed_function
    per_replica_function, args=(x, y, sample_weights))
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/distribute/distribute_lib.py", line 763, in experimental_run_v2
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/distribute/distribute_lib.py", line 1819, in call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/distribute/distribute_lib.py", line 2164, in _call_for_each_replica
    return fn(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/autograph/impl/api.py", line 292, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 264, in train_on_batch
    output_loss_metrics=model._output_loss_metrics)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_eager.py", line 312, in train_on_batch
    output_loss_metrics=output_loss_metrics))
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_eager.py", line 269, in _process_single_batch
    grads = tape.gradient(scaled_total_loss, trainable_weights)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/backprop.py", line 1029, in gradient
    unconnected_gradients=unconnected_gradients)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/imperative_grad.py", line 77, in imperative_grad
    compat.as_str(unconnected_gradients.value))
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 766, in _backward_function
    return self._rewrite_forward_and_call_backward(call_op, *args)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 685, in _rewrite_forward_and_call_backward
    forward_function, backwards_function = self.forward_backward(len(doutputs))
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 594, in forward_backward
    forward, backward = self._construct_forward_backward(num_doutputs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 642, in _construct_forward_backward
    func_graph=backwards_graph)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/framework/func_graph.py", line 974, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 632, in _backprop_function
    src_graph=self._func_graph)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/ops/gradients_util.py", line 669, in _GradientsHelper
    lambda: grad_fn(op, *out_grads))
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/ops/gradients_util.py", line 336, in _MaybeCompile
    return grad_fn()  # Exit early
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/ops/gradients_util.py", line 669, in <lambda>
    lambda: grad_fn(op, *out_grads))
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 685, in _rewrite_forward_and_call_backward
    forward_function, backwards_function = self.forward_backward(len(doutputs))
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 594, in forward_backward
    forward, backward = self._construct_forward_backward(num_doutputs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 642, in _construct_forward_backward
    func_graph=backwards_graph)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/framework/func_graph.py", line 974, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 632, in _backprop_function
    src_graph=self._func_graph)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/ops/gradients_util.py", line 669, in _GradientsHelper
    lambda: grad_fn(op, *out_grads))
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/ops/gradients_util.py", line 336, in _MaybeCompile
    return grad_fn()  # Exit early
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/ops/gradients_util.py", line 669, in <lambda>
    lambda: grad_fn(op, *out_grads))
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 685, in _rewrite_forward_and_call_backward
    forward_function, backwards_function = self.forward_backward(len(doutputs))
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 594, in forward_backward
    forward, backward = self._construct_forward_backward(num_doutputs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 642, in _construct_forward_backward
    func_graph=backwards_graph)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/framework/func_graph.py", line 974, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 632, in _backprop_function
    src_graph=self._func_graph)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/ops/gradients_util.py", line 623, in _GradientsHelper
    (op.name, op.type))
LookupError: No gradient defined for operation 'while' (op type: While)

Can you run a model with while but without pickling/unpickling? Afaik, while doesn't have gradients (maybe with gradient tape but then I don't know if those get pickled anyway)

Two examples that runs without errors:

the above example commenting out the @ray.remote decorator and call the train function without the remote call.

the above example adding a return statement to the build_save_model() to return the built model. We swap out tf.keras.models.load_model() in the Worker to self.model = build_save_model() and call train()

One example that runs with error:

we build and save the model in the Worker (as part of remote call) and tries to load the saved model in the main python session (not remote)

Any updates on this?

Are there any updates regarding this issue?

No, but if you want to make a cherry-pick we can merge it if and when we do a new patch release on 1.15

@mihaimaruseac I opened https://github.com/tensorflow/tensorflow/pull/39034 for this.

I think this can be closed now as it has been solved and backported to 1.15 too.


boldigger3 not working in apscale environment

Hi TillMacher!
As you requested, I open an issue here where I cannot get boldigger3 to work in the apscale env. If I do vsearch --help or the same with blastn, they work fine. But if I try with boldigger3 --help I get this message:
File "C:\ProgramData\Anaconda3\envs\apscale\lib\runpy.py", line 196, in _run_module_as_main                                                                     return run_code(code, main_globals, None,                                                                                File "C:\ProgramData\Anaconda3\envs\apscale\lib\runpy.py", line 86, in run_code                                                                                exec(code, run_globals)                                                                                File "C:\ProgramData\Anaconda3\envs\apscale\Scripts\boldigger3.exe_main.py", line 7, in                                                           File "C:\ProgramData\Anaconda3\envs\apscale\lib\site-packages\boldigger3_main.py", line 68, in main                                                         latest_version = obtainer("boldigger3")                                                                                File "C:\ProgramData\Anaconda3\envs\apscale\lib\site-packages\get_pypi_latest_version\main.py", line 40, in call                                            raise e                                                                                File "C:\ProgramData\Anaconda3\envs\apscale\lib\site-packages\get_pypi_latest_version\main.py", line 38, in call                                            all_versions_web = self.get_by_spider_web(package_name)                                                                                File "C:\ProgramData\Anaconda3\envs\apscale\lib\site-packages\get_pypi_latest_version\main.py", line 83, in get_by_spider_web                                   raise ValueError(f'No version information of {package_name}. '                                                                                ValueError: No version information of boldigger3. Please check if the package name correct.
please let me know if you know what could be the issue. I do by the way have a file named boldigger3.exe in the same env..

Solved in the boldigger3 thread Need to downgrade numpy to version 1.26.4, and get-pypi-latest-version to version 0.1.0 using commands:
pip install --force-reinstall -v "numpy==1.26.4"
pip install --force-reinstall -v "get-pypi-latest-version==0.1.0"


Fix unhandled side for redstone connection

Fixed crash caused by redstone connection from bottom of block. Resolves crash discussed in Discord

Build error is
/home/runner/work/OpenComputers/OpenComputers/src/main/scala/li/cil/oc/integration/gregtech/RecipeHandler.scala:97:46: value addCNCRecipe is not a member of gregtech.api.interfaces.internal.IGT_RecipeAdder

yea addCNCRecipe was deprecated for a while and recently removed in gregtech

Do we support OC outside of the pack?

its on 'passive' in the support list

I did check mention to CNC recipemap but didn't about CNC RA :exploding_head:
But did it ever used for anything in the era of GregoriusT?

Actually GT compat module is really ancient and doesn't include compat with crucial machines like EBF.
https://github.com/GTNewHorizons/OpenComputers/blob/master/src/main/scala/li/cil/oc/integration/gregtech/RecipeHandler.scala
Also MT script is available for adding recipes.
So I highly doubt that anyone is using it, especially CNC one.

most of the OC/GT compat is in Computronics, OC integration module is minimal

I updated existing recipe handlers to RA2. If there is request to implement other handlers - ping me.
This PR should be checked with current config of OC in GTNH (I didn't tested all recipes myself).

we edit recipes via core mod so i guess not much original recipes are needed


Syntax error with python 3.6

Hi, I'm trying to use image watermark with sorl-thumbnail. But when I add it in app settings it complaints for the following syntax error-
    from sorl_watermarker.engines.base import WatermarkEngineBase
  File "/home/raj/.local/lib/python3.6/site-packages/sorl_watermarker/engines/base.py", line 80
    except TypeError, e:
                    ^
SyntaxError: invalid syntax

My settings
THUMBNAIL_ENGINE = 'sorl_watermarker.engines.pil_engine.Engine'
THUMBNAIL_WATERMARK = 'display/avatar/anyone.png' (This is in static dir of my application)

Please suggest. Thank you.

Hey @patelrajnath .. as it seems we have fixed this syntax issue already. Could you try simply using the latest git master?


PartitionDurationFormatPattern appends the result of FormatNumericUnits twice

In PartitionDurationFormatPattern, we have:
e. If style is "numeric" or "2-digit", then
    i. Append FormatNumericUnits(durationFormat, duration, unit, signDisplayed) to result.
    ii. Let numericPartsList be FormatNumericUnits(durationFormat, duration, unit, signDisplayed).
    iii. If numericPartsList is not empty, append numericPartsList to result.
    iv. Set numericUnitFound to true.

This results in appending the same formatted list twice - was this a mistake in 0b2e3eda5706d99ad7e8104b082ee4385f9551fa?

Yeah, looks like a minor bug introduced as part of the fix to #223.


Feature/create component dialogs better button [OSF-6443]

Purpose
Fix the weird Add Component button.
Changes

add unwrap to mithrilHelpers
add disabled placeholder button

Side effects
Disabled button shows until mithril renders the actual button.
Ticket
https://openscience.atlassian.net/browse/OSF-6443

Turns out we can use the nodeCategories that are already in contextVars. Made this change in https://github.com/CenterForOpenScience/osf.io/commit/71e7467057a80348159e29dd53c626dd41132ab6 . 🐙


BUG - A scenario submition can execute multiple times the same task

if a scenario has some overlapping pipelines, the tasks that belong to multiple pipelines are submitted once per pipeline when submitting a scenario.
Example:
Scenario:
Task A -> Task B -> Task C
Pipeline 1:
Task A -> Task B
Pipeline 2:
Task B -> Task C

@jrobinAV
I think this really depends on the use case, for the example if I have datanode D as input dn of task T, task T here is used for printing out the result of datanode D. If some task T1 in pipeline P1 with task T1 make changes to D, T can be used to print the new result, then pipeline P2 with task T2 is used to make a different change to D, should T still run to print the new value of D? Task T here can also have some conditional calculation that might gives different result based on D, if task T has output datanode D_out, it might be changed based on the input D and the condition in T. In that case I think T should be rerun as different pipeline might produce different value of D which can result in different result of T.
Though this does raise a question for me, if T is cachable, we run it in P1 and now it is cached, no changes is made to input dn D in pipeline P2, should T still rerun? If not, could there be a problem with it as this is the same task but in 2 different pipeline. I think the answer is No for both question but what do you think?

That s a specific use case where two tasks write the same output. I believe this is strange, but we can do it in Taipy.

So I think from the previous discussion, we can note down some main points:

There are some use cases where it makes sense to have a submitted task run twice (either in the same pipeline or different pipeline)
However, it can create some undesired side effect like where the result of several pipelines depend on which tasks finished first (because they will write to the same datanode but the input can be affect in different pipelines)
Another strong point is that if a user desire said behavior, they should know and be awared of what they are doing. At most, we only need to log a warning for potential unwanted behavior and don't have to change what we already have.


WaitForSingle/MultipleObject improvements for process handles

Process handles don't have a real good support in WaitForSingle/MultipleObject:

WaitForSingleObject doesn't support a timeout which might cause the calling process to block
WaitForMultipleObject doesn't support process handles at all

Some (not complete) implementation ideas:

use eventfd, with SIGCHLD unmasked, and use select
use WNOHANG for waitpid and sleep until the timeout (in this case we should check multiple times until the set wait timeout to minimize delays)

This has been implemented months ago.


[Fix] Add tenant to the context used for the stellar.toml file

What
Add tenant to the context used for the stellar.toml file
Also, split the tenant middleware into two: InjectTenantMiddleware, and EnsureTenantMiffleware, for increase flexibility.
Why
The stellar.toml was not loading differently per tenant.
Checklist
PR Structure

[x] This PR has a reasonably narrow scope (if not, break it down into smaller PRs).
[x] This PR title and description are clear enough for anyone to review it.
[x] This PR does not mix refactoring changes with feature changes (split into two PRs otherwise).

Thoroughness

[x] This PR adds tests for the new functionality or fixes.
[x] This PR contains the link to the Jira ticket it addresses.

Configs and Secrets

[x] No new CONFIG variables are required -OR- the new required ones were added to the helmchart's values.yaml file.
[x] No new CONFIG variables are required -OR- the new required ones were added to the deployments (pr-preview, dev, demo, prd).
[x] No new SECRETS variables are required -OR- the new required ones were mentioned in the helmchart's values.yaml file.
[x] No new SECRETS variables are required -OR- the new required ones were added to the deployments (pr-preview secrets, dev secrets, demo secrets, prd secrets).

Release

[x] This is not a breaking change.
[x] This is ready for production.. If your PR is not ready for production, please consider opening additional complementary PRs using this one as the base. Only merge this into develop or main after it's ready for production!

Deployment

[x] Does the deployment work after merging?

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr220.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr220.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr220.previews.kube001.services.stellar-ops.com

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr220.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr220.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr220.previews.kube001.services.stellar-ops.com

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr220.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr220.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr220.previews.kube001.services.stellar-ops.com

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr220.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr220.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr220.previews.kube001.services.stellar-ops.com


Lazy load and AOT : Working but splitted chunk ONLY WITH AOT FLAG

Please provide us with the following information:

OS?
Ubuntu 16.10 (x64)
Versions.
angular-cli: 1.0.0-beta.21
Repro steps.
After I read more about lazy load + AOT on this issue, I decided to give a try and it's working really nice for the first time in my app.
So I decided to have some fun and benchmark different results. Here's the ouput :
+-----------------------+-------------+--------------+-----------+-------------+
|                       | Main bundle |   Chunk 0    | Scripting | First paint |
+-----------------------+-------------+--------------+-----------+-------------+
| ng serve              | 4.5 MB      | Not splitted | 6075 ms   | 5500+ ms    |
| ng serve --prod       | 334 KB      | Not splitted | 5587 ms   | 4750+ ms    |
| ng serve --aot        | 3.3 MB      | 326 KB       | 4011 ms   | 4400+ ms    |
| ng serve --prod --aot | 243 KB      | 18.1 Kb      | 3860 ms   | 4250+ ms    |
+-----------------------+-------------+--------------+-----------+-------------+

The --prod --aot build size is 27% smaller than --prod build
The --prod --aot build is 31% faster when scripting than --prod build
AOT is cool !

BUT
As you can see, ng serve and ng serve --prod doesn't have the chunk expected and I'm able to find the code (that's supposed to be in the chunk) inside the main bundle. With aot flag, I always have both the main bundle and my chunk.
Even tho it's working, devs who want to put their app in prod without building with --aot will not enjoy the lazy loading they've setup ;)

It works fine on my end (here: https://github.com/MrCroft/lazy)
Plus I already had a bigger app and updated the cli from beta.20-4 to beta.21, and ng build --prod or ng serve --prod both work as before (separate chunks get created for each lazy module).
Make sure you're using angular 2.2.1 (they've locked the version to that I see, in the latest CLI release).
Ultimately, if the app wasn't initially generated with beta.21, remove node_modules and do an ng init, check the file differences.

Also noticed this. I have a 3.2mb main bundle only now

AOT has never worked for me, due to at least two open issues elsewhere.
I upgraded from Angular 2.1.2 to 2.2.1 and CLI from beta 18 to beta 21. Instantly lost lazy-loading; no longer seeing any chunks but one big main bundle.

That's definitely odd. You should always have the lazy loaded chunks.
Does this still happen with the latest versions? If so, can you get me a repro I can debug?

Not happening for me anymore with the latest stuff (b24)

I've got many errors since beta 24 which make me stay on beta 22.
(for example : No module factory available for dependency type: ContextElementDependency).
When I manage to make my project works with beta 24 I'll let you know about the lazy loading problem. :)

Dupe of #2496 (I think), fixed in #3079.

I'm running angular cli beta 26. Testing on an empty project (ng new).

built with aot: got 0.bundle.js.
built without aot: got no chunk file.

I know this is an empty project, which doesn't have any lazy module. But is this issue really fixed?
And just curious. What special thing does AOT do to make Webpack code splitting? I checked the tsconfig.json. Typescript modules are compiled into es6 modules (import and export keywords), and Webpack recognizes es6 modules, right? Isn't it enough for Webpack to do code splitting? What else is needed?


[fix] Bitcoin altcoins were removed from the new app, using the legac…

…y repo for those

Codecov Report
Base: 77.72% // Head: 77.72% // No change to project coverage :thumbsup:

Coverage data is based on head (3da8e1c) compared to base (8e016af).
Patch has no changes to coverable lines.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop      #44   +/-   ##
========================================
  Coverage    77.72%   77.72%
========================================
  Files           12       12
  Lines         2352     2352
========================================
  Hits          1828     1828
  Misses         524      524

Flag
Coverage Δ

unittests
77.72% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Error Running Sample Code

ScheduleAPI-AV-0001:a668ff30-2872-ec11-8943-000d3a34a1d1: Input to the column msdyn_progress in the table msdyn_projecttask during create operation is not allowed.'
This is the method
  private OperationSetResponse CallPssCreateAction(Entity entity, string operationSetId)
    {
        OrganizationRequest operationSetRequest = new OrganizationRequest("msdyn_PssCreateV1");
        operationSetRequest["Entity"] = entity;
        operationSetRequest["OperationSetId"] = operationSetId;
        return GetOperationSetResponseFromOrgResponse(organizationService.Execute(operationSetRequest));
    }

[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f269741b-6823-92ab-066d-90e68ac3e418
Version Independent ID: e39d8b04-27e4-9345-5dc5-d752268d04a2
Content: Use Project schedule APIs to perform operations with Scheduling entities
Content Source: articles/project-management/schedule-api-preview.md
Service: dynamics-365-project-operations
GitHub Login: @sigitac
Microsoft Alias: sigitac

@Couchiman - Thank you for the feedback.
@sigitac - Can you please take a look and update the topic as necessary? Thanks!

The topic has been updated and published.


Add MC packages as a filter and move the System filter and the MC Package filter to the left hand side of the filters

Expected outcome

Add MC packages as a filter and move the System filter and the MC Package filter to the left hand side of the filters - discussed with Aman on Teams

Business Value / Developer Experience

[ ] Add MechanicalCompletionPackage as a column in the table

[ ] Add MechanicalCompletionpackage as a filter (hidden filter)


Audio call not working anymore

Product and Version [VS/VSCode]: VSCode
OS Version [macOS/Windows]: Linux (Fedora 31)
Live Share Extension Version: 0.3.4
Target Platform or Language [e.g. Node.js]: N/R
Steps to Reproduce / Scenario:

Start a session
Click on 'Start audio call'
Nothing happens

( When I dblclick on "Start audio call..." I can see a "starting audio call" appearing, then nothing )
From the logs, I see this:
[2019-12-02 10:27:52.471 LiveShareAudio V] Current state: LiveShareWithoutAudio, Action: AttemptConnectAudio
[2019-12-02 10:27:52.776 ExternalAppCallingService E] [JsonRPC] Dispose start
[2019-12-02 10:27:52.776 ExternalAppCallingService E] [JsonRPC] Dispose end
[2019-12-02 10:27:52.776 ExternalAppCallingService I] Getting SkypeToken
[2019-12-02 10:27:52.777 LiveShareAudio V] Current state: AudioConnecting, Action: AudioDisconnected
[2019-12-02 10:27:52.778 ExternalAppCallingService E] Electron closed unexpectedly: code:9 signal:null
[2019-12-02 10:27:53.026 ExternalAppCallingService I] Initializing SlimCore
[2019-12-02 10:27:53.027 ExternalAppCallingService E] Failed to initialize electron app: Call initialization failed Error: Call initialization failed
        at ExternalAppCallingService.<anonymous> (/home/mbitard/.vscode/extensions/ms-vsliveshare.vsliveshare-audio-0.1.80/out/calling/externalAppCallingService.js:200:27)
        at Generator.next (<anonymous>)
        at fulfilled (/home/mbitard/.vscode/extensions/ms-vsliveshare.vsliveshare-audio-0.1.80/out/calling/externalAppCallingService.js:4:58)
        at processTicksAndRejections (internal/process/task_queues.js:89:5)
[2019-12-02 10:27:53.027 LiveShareAudio W] startOrJoinAudio: cancellation on connect to audio: Call initialization failed
[2019-12-02 10:27:53.027 LiveShareAudio V] Current state: LiveShareWithoutAudio, Action: AudioConnectionFailed
[2019-12-02 10:27:53.027 LiveShareAudio W] No transition exists for action: AudioConnectionFailed

I attached the full logs here: logs.zip
It worked on this same machine a couple weeks ago, not sure what changed though.

@MichaelBitard: thanks for reporting this issue! I suspect it is specific to Fedora 31 as I've not been able to repro it on Ubuntu. I can see from the logs it is failing immediately after our Audio Application is launched in Electron. A few things to confirm:

Do you have the latest Live Share Audio extension installed? (1.1.80)
Can you confirm that the required dependencies are installed to the global storage location. You should see the following folders that contain assets:
electron-4.2.7
slimcore-4.0.2<IP_ADDRESS>

at this location:
~/.config/Code/User/globalStorage/ms-vsliveshare.vsliveshare-audio

Can you try to launch electron on its own to see if you're able to start that successfully.
Open this folder in a terminal:
/.config/Code/User/globalStorage/ms-vsliveshare.vsliveshare-audio/electron-4.2.7/dist
then ./electron
Are you seeing any errors?

Well spotted
I have the latest version (0.1.80 and not 1.1.80 right?)
The dependencies are correctly installed.
When I launch electron, I get this error message:
electron: --max-http-header-size= is not allowed in NODE_OPTIONS

Removing this option fixes the issue.
I remember now that some times ago when I launched vscode it did not start at all with this error message, but now it can start, that's weird.
Thanks a lot!


last version from flutter SDK have a problem with google map by using flutter_google_map plugin

Steps to Reproduce

...
...
...

Target Platform:
Target OS version/browser:
Devices:
Logs

duplicate https://github.com/flutter/flutter/issues/42326


Add support for deleting from Data Index by Data ID Range

This can be expressed through the API but currently the datastore backends don't have logic to support it.

Fixed through query by data ID range followed by delete.  Technically could be more efficient with direct backend logic but not a big deal.


Interpolation doesn't work (AreaSeries/AreaOnlySeries) in the hybrid mode.

Hybrid:

SVG:

<AreaSeries
  yAccessor={askAccessor}
  interpolation={curveStepAfter}
  stroke="#0F0"
  fill="#0F02"
/>
<AreaSeries
  yAccessor={bidAccessor}
  interpolation={curveStepAfter}
  stroke="#F00"
  fill="#F002"
/>

could you post back how it looks if you change AreaSeries to LineSeries please

I am guessing svg is the correct chart you want to see. is that right?
Also I would guess bid has to be stepBefore while ask is stepAfter do you agree?

I am guessing svg is the correct chart you want to see. is that right?

Yes SVG is a view I want to see in the hybrid mode.

Also I would guess bid has to be stepBefore while ask is stepAfter do you agree?

You right.

could you post back how it looks if you change AreaSeries to LineSeries please

LineSeries SVG:

LineSeries Hybrid:

something is not right, in this line series Hybrid looks correct, also svg is showing the area between the start and end points.
I looked at the code and I am not sure what is wrong. are you available now for a chat on gitter?

https://codesandbox.io/s/q9jqp20lyq
the only problem I see is that interpolation is not implemented for canvas at all. it works for svg.
However I dont see the other svg vs canvas problem you face for interpolation.

the only problem I see is that interpolation is not implemented for canvas at all. it works for svg.

It is exactly the issue I reported for :)

:) I understood it as that stepAfter works differently between svg and canvas.

fixed and published 0.7.0 (not beta, the final 0.7.0) install latest as npm install react-stockcharts

@rrag  Wow! Thank you very much!


Maintenance status of this app

Hi, it would be great to know the maintenance status of this application.
Was it abandoned? Are contributions still welcome? A short status sentence on the README or in the GitHub description would be awesome.

@claudep It was thrown to the wayside for a while, but it now has an active maintainer. Would you be interested in joining, as well?

Hi @claudep,
The project had been abandoned for a while, but I was recently made a maintainer and have been working on getting the package up to date for modern versions of both Python and Django. There is a branch that I'm hoping to deploy within the next couple weeks to PyPI that will, unfortunately, change the installation package name.
Once that is pushed out, I will absolutely welcome contributions again.


Sussy Scan páginas não está carregando

Source information
Sussy Scan 1.4.39
Source language
Português (Brasil)
Steps to reproduce
As páginas das obras não está carregando na extensão, No Webview está carregando normalmente.
https://github.com/user-attachments/assets/97c9dee0-efe8-4179-a62d-02bc906b37d1
Expected behavior
A extensão estar funcionando normalmente
Actual behavior
Não estar carregando as páginas
Mihon/Tachiyomi version
TachiyomiSY 1.10.5
Android version
Android 11
Other details
No response
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a short but informative title.
[X] I have updated the app to version 0.15.3.
[X] I have updated all installed extensions.
[X] I have tried the troubleshooting guide.
[X] If this is an issue with the app itself, I should be opening an issue in the app repository.
[X] I will fill out all of the requested information in this form.

Seems to load up fine on my end, try going to SY Settings > Data and storage, and "Clear cache"
If that doesn't work, try migrating the title back into Sussy Scan

Tá do mesmo jeito ainda 😓

Tá do mesmo jeito ainda 😓

Do you have Data Saver enabled in SY Settings > Advanced? I know that could cause issues too. Try turning it off, clearing chapter cache, and restart the app.

Muito obrigado pela ajuda


Add pager number

For onboarding of NOMS generally and HMPS specifically it is useful to have a pager number field
as this is a common form of communication for the users.

This looks good to me 👍


Add 6 more AES-CBC algorithms

Latest improvements:
MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Contribution checklist:

[x] I have reviewed the documentation for the workflow.
[x] Validation tools were run on swagger spec(s) and have all been fixed in this PR.
[ ] The OpenAPI Hub was used for checking validation status and next steps.

ARM API Review Checklist

[ ] Service team MUST add the "WaitForARMFeedback" label if the management plane API changes fall into one of the below categories.
adding/removing APIs.
adding/removing properties.
adding/removing API-version.
adding a new service in Azure.

Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs.

[ ] If you are blocked on ARM review and want to get the PR merged urgently, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.
Please follow the link to find more details on API review process.

/azp run automation - sdk

Can one of the admins verify this patch?

/azp run automation - sdk

@qinxgit thanks for fixing the spell-check. are we good to merge or more changes are coming in?

@anuchandy I fixed the required failing checks. Let me know what else is needed.

@qinxgit thanks for fixing the spell-check. are we good to merge or more changes are coming in?

I don't have more to check in. Please merge.


Question: Scrolling in sap.ui.table.Table

Hey!
I'm currently trying to scroll inside a sap.ui.table.Table. Most of the tutorials out there tend to use the scrollIntoView() method, which unfortunately doesn't work for these SAP tables (API Ref: "When the user scrolls, only the row contexts are changed, but the rendered controls remain the same"). I've tried other things like scroll() and scrollBy() described in the mdn web docs but haven't been able to get them to work. Is there something I'm missing?
Thanks in advance :)

OK, so I've found a solution, but I don't think it's very nice:
it("should scroll with arrowDown", async () => {     let firstCell = await browser.asControl(firstRowFirstCell)     await firstCell .press()     for (let i = 0; i < 80; i++) {       browser.keys("ArrowDown")     }     expect(await firstCell .getText()).toEqual("templateName 43")   })
If there's a cleaner solution, I'd like to hear it :)

Since this is very specific to this table control, maybe the OPA5 team knows a cleaner solution.

OK, so I've found a solution, but I don't think it's very nice:
it("should scroll with arrowDown", async () => {  let firstCell = await browser.asControl(firstRowFirstCell)  await firstCell .press()  for (let i = 0; i < 80; i++) {  browser.keys("ArrowDown")  }  expect(await firstCell .getText()).toEqual("templateName 43")  })
If there's a cleaner solution, I'd like to hear it :)

na man, i think you're good here 😄 - we never had the use case before.
Looking at your code, the only remark I have is that you want to make sure your browser always starts with the same width x height ratio, so that pressing ⬇ 80 times reliably gets you to the templateName 43 point.


Doco - added Tasks info. First stab. Please feedback

Added UI to list of projects
Added detail to each, including PORTs involved.

Ta


Implement Lenght typehint annotation

Annotation already existing in: PropertyTypeHintAnnotation.kt (already there, needs to be uncommented.
Add typehint and typelimiting support in PropertyHintTypeMapper.kt

Closing in favor of #155


Doesn't break for input

Hi Kyle!
When running a piece of Stata code which includes a prompt, such as display "Question" _request(answer), Stata won't break as it usually does to allow the user to input the "answer." I can't seem to find a way around it—other than not using such prompts...
Is that something that can be fixed, or should I just come to terms with this absolutely tragic flaw of your otherwise awesome package? ;)
Thanks,
Jesse

I didn't even know Stata had the possibility of doing that.
This package is designed for interactive work, so it sends each command individually instead of as a do file. There really isn't much I can do to work around that. I'd say just don't run those lines interactively

Thanks for the heads-up. All the best!


Development of a Circom Gadget standard library for Lurk

Description

When our users will want to implement some of their projects in Lurk they will need some basic building tools to be integrated within our library (at least until they are able to easily integrate some by themselves).
As the Circom community has already produced quite a lot of really useful circuits, I am all for creating a standard library of pre-integrated Circom Gadgets that can be used by our users.
Ideally, we should also properly document the process of integrating a new gadget to such a library so that any of our user will be able to do it.
Motivation

This  idea comes from discussions with @huitseeker and @winston-h-zhang.
To implement our idea for Light Client we will need to be able to do operations around the different hash algorithms used by the chains of interest to us (e.g., SHA3-256 or Keccak-256).
As such circuit does not exist in any of the bell* repository, the quickest route to implement them would be to leverage our CircomCoprocessor and some existing circom circuits.
Current behavior
This section aims to bring everyone to the same level of knowledge around the current DevX around the CircomCoprocessor.
If I am a user, here are the current setup steps to get a Circom circuit to work with Lurk:
Pre-setup

Procure themselves a valid Circom circuit.

CLI

Import the Circom circuit within Lurk context

Implementation

Create a structure that will serve as a CircomGadget foundation.
Implement both into_circom_input and evaluate_simple for the given CircomGadget/circuit.
Create an enumeration with the Coproc trait to use at the instantiation of Lang.

For a concrete example please refer to the Sha-256 example.
Observed pitfalls
Two elements in the current implementation can create pain points to our users:

Import restricted to CLI: By restricting Circom circuit import to CLI we are saying that we expect our users to find and get their own circom circuit by themselves, which can clearly be a pain point in the developer experience.
Implement their own gadget logic: Closely related to the previous pain point. As the users are the one importing their circuits, they also need to implement the plumbing for the gadgets. This provides flexibility but comes at the cost of simplicity.

Along with a reported bug that needs to be tested & fixed if needed:

Arguments for Circom  circuits: See #628 and #789

Solution exploration
The core concept of a solution would be to directly integrate static files related to the circuits directly in some crates / repository / cdn to make them available in Lurk.
Static files
When it comes to the static files related to a given circuit we only need to have the related *.wasm and *.r1cs. A pair of those files can be around 30 MB. Here are the solutions I thought of:

Include files directly in the crate: Easiest implementation but will bloat the library in the long run.
Load it at runtime from an external source
a. Github repositories: We could create one or multiple Github repositories with our Circom sources, and have their compiled files available in their different releases.
b. A CDN/dCDN: We could store the static files on a content distribution network. But as this involves increasing costs for us, this does not seem like a great idea.

Recommendation: We should aim for standardized repository structure and behavior from which we could download static files at runtime.
Rust glue code
For now, I believe that we should restrict the glue code implementation to our library? However, it should be possible to create a generic interface between some Rust crates and our library to allow for the Gadgets' repositories to not only contain Circom code, but also Rust crates containing the glue code along with any other bit of logic needed.
Requirements

The requirements that will follow will be for a proof-of-concept over a Keccak-256 implementation. If this PoC is convincing enough, then we could switch to a more generic implementation.

Define a basic Gadget repository structure
Implement it over a Keccak-256 implementation in Circom.
Implement the Gadget code over lurk-rs
Add arguments specification over CircomCoprocessor

Open questions

Circom binary
I have a few open question about the Circom binary that we use in Lurk.
Currently I can see that we have a hard-coded version in our CLI files.

Is there a reason for it to be hard-coded or do we mean to use the latest?
Should we allow for Circom gadgets to use different versions of Circom in the future?

Are you planning to do it yourself in a pull request?

Yes

Maige please label issues that are mentioned as for our "light client" with one among "C-1 Merkle Proofs", "C-2 Block header validation" or ""C-3 LC Infrastructure". Further, if they mention SHA-3 or Keccak, you should use "C-1 Merkle Proofs".
Please apply this to the present issue.

While I'm here:

I have a few open question about the Circom binary that we use in Lurk.
Currently I can see that we have a hard-coded version in our CLI files.
Is there a reason for it to be hard-coded or do we mean to use the latest?

We're just trying to install a release of Circom which can be overridden by a user setting. The idea is that Circom's license makes it hard to distribute in-project.

Should we allow for Circom gadgets to use different versions of Circom in the future?

Sure, though that compatibility is not an imminent priority.


bug of the number of connection

        if (!this.isClosed) {
            final Channel channelFromIdlePool = ApnsChannelPool.this.idleChannels.poll();

            if (channelFromIdlePool != null) {
                if (channelFromIdlePool.isActive()) {
                    acquirePromise.trySuccess(channelFromIdlePool);
                } else {
                    // The channel from the idle pool isn't usable; discard it and create a new one instead
                    this.discardChannel(channelFromIdlePool);
                    this.acquireWithinEventExecutor(acquirePromise);
                }
            } else {
             // We don't have any connections ready to go; create a new one if possible.
            }
        }

I don't think it is a good idea. When I use it to push, I find some time channel.writeAndFlush use little time, so ApnsClient use only one channel to push. But the time of  apple server response is at 200 pieces per second and I have to set the semaphore to prevent the OOM. Therefore, ApnsClient just sends 200 pieces per second. So In my opinion, we should create max number of  channels in the initialization of ApnsChannelPool.

I wouldn't call this a bug, per se, but think it's an interesting idea all the same. We'll look into it.

Actually, I take it back. I think this is a bug. There's no fundamental guarantee that we'll ever create the number of connections callers are expecting, and are relying on one thread probably being faster than another, which is bogus design.


TIFF format does not show up through 'LOAD' function, and the image can not be classified through "LOAD" function.

Describe the bug
A clear and concise description of what the bug is.
TIFF format does not show up through 'LOAD' function, and the image can not be classified through "LOAD" function.
To Reproduce
Steps to reproduce the behavior:

Go to 'https://ai-cfia.github.io/nachet-frontend/'
Click on 'CAPTURE' -> "LOAD" ->

-> "CLASSIFY" ->

For a TIFF image, check
1.zip

See error "Error fetching inference data"

Expected behavior
A clear and concise description of what you expected to happen.
The seeds in the photo should be segmented and classified.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: Windows (version, 10, 11, etc.) : Windows
Browser [e.g. chrome, safari, edge, firefox] : Chrome

Additional context
Add any other context about the problem here.

The front end version at that link is 0.2.0 , the current version on the main branch of the front end is 0.8.0 . We'll have to check the deployment process of when main should be deployed. I saw in another end point that the front end version there is 0.4.0 . So not sure why everything is not on par with the current main branch.

Humm.. Do you have any idea on which front end version did Taran used to conduct the end-user testing?

Humm.. Do you have any idea on which front end version did Taran used to conduct the end-user testing?

0.4.0

Do you know the link?

Looks like the problem is the current workflow run doesn't have a deploy step for the github pages endpoint

Do you know the link?

Probably best to DM Taran

https://nachet.inspection.alpha.canada.ca/
This is the link Taran is using. However, I cannot access this link and the below message showed up:
403 Forbidden
For support, please contact your network administrator or whomever configured Nginx to use Vouch Proxy.
For help with Vouch Proxy or to file a bug report, please visit https://github.com/vouch/vouch-proxy
Do you have any idea? Taran should have went through a github verification, but i was not directed into any verification page :)

I get the same thing. I assume this portal is only available if you have the VPN connected.

https://nachet.inspection.alpha.canada.ca/ This is the link Taran is using. However, I cannot access this link and the below message showed up: 403 Forbidden For support, please contact your network administrator or whomever configured Nginx to use Vouch Proxy. For help with Vouch Proxy or to file a bug report, please visit https://github.com/vouch/vouch-proxy
Do you have any idea? Taran should have went through a github verification, but i was not directed into any verification page :)

The DevOps team merged a fix to update the application on https://nachet.inspection.alpha.canada.ca/  can you confirm it is now > 0.8.0

From my end, the problem looks persists, the exactly same message pop out.

Fixed in 0.9.2


"Benesat" , "Aluniș" , "Biușa" are listed in the drop-down menu

Description:
When user writes "Benesat" , "Aluniș" , "Biușa" the locations appear in the drop-down menu next to the comune and the county which is a part according to the law.
Precondition:
The website is up an running.

Step 1
Write in the search bar "Benesat"

Expected results
The location appears in the drop-down menu as "Sat Benesat, SĂLAJ (Benesat)".

Step 2
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 3
Write in the search bar "Aluniș"

Expected results
The location appears in the drop-down menu as "Sat Aluniș, SĂLAJ (Benesat)".

Step 4
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 5
Write in the search bar "Biușa"

Expected results
The location appears in the drop-down menu as "Sat Biușa, SĂLAJ (Benesat)".


Release version 1.3.3

I don't know when new versions are released and what criteria is used to release them, but I'd certainly like to see a 1.3.3 and a new swagger2markup maven plugin release as well.

Hi, the criteria is just my time :(
I try to release a new version as soon as possible.

v1.3.3 of Swagger2Markup has been released.
I release the maven plugin soon.
But you can already use the new version with the old plugin if required.

Maven Plugin v1.3.6 has been released.


Add proper support for TextBox.InnerLeftContent with watermarks and labels

Discussed in https://github.com/AvaloniaCommunity/Material.Avalonia/discussions/279

Originally posted by CastelloBrancoTecnologia August 14, 2023

inestead of this

in
<TextBox  ..... >
<TextBox.InnerLeftContent>
<icons:MaterialIcon
Width="24"
Height="24"
Margin="4"
HorizontalAlignment="Left"
VerticalAlignment="Bottom"
Foreground="{DynamicResource PrimaryHueMidBrush}"
Kind="KeyVariant" />
</TextBox.InnerLeftContent>

I think this is not an enhancement, but a BUG a floting label/whatermark colide with Left content part


fix(core): supports TS 4.8.3 #3514

closes #3514

v14.2.1 has been released and contains a fix for the issue. Feel free to reopen the issue or to submit a new one if you meet any problems.


Updating dependencies, making BatchingSettings/FlowControlSettings st…

…able

Damn, the newer version of auto-value requires > Java 1.7, so we can't use it yet...

Codecov Report

Merging #542 into master will increase coverage by 0.04%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master     #542      +/-   ##
============================================
+ Coverage     73.27%   73.32%   +0.04%
+ Complexity      880      874       -6
============================================
  Files           172      172
  Lines          3989     3951      -38
  Branches        315      304      -11
============================================
- Hits           2923     2897      -26
+ Misses          917      905      -12
  Partials        149      149

Impacted Files
Coverage Δ
Complexity Δ

...le/api/gax/core/InstantiatingExecutorProvider.java
57.14% <ø> (+3.8%)
3 <0> (ø)
:arrow_down:

...google/api/gax/core/GoogleCredentialsProvider.java
0% <ø> (ø)
0 <0> (ø)
:arrow_down:

.../google/api/gax/retrying/TimedAttemptSettings.java
100% <ø> (+20%)
2 <0> (ø)
:arrow_down:

...ain/java/com/google/api/gax/rpc/ClientContext.java
81.66% <ø> (-0.88%)
8 <0> (-1)

...ava/com/google/api/gax/retrying/RetrySettings.java
29.78% <ø> (-2.87%)
2 <0> (-1)

.../com/google/api/gax/batching/BatchingSettings.java
80% <ø> (+2.72%)
2 <0> (ø)
:arrow_down:

...m/google/api/gax/batching/FlowControlSettings.java
76.92% <ø> (+3.58%)
2 <0> (ø)
:arrow_down:

...m/google/api/gax/httpjson/HttpJsonCallOptions.java
33.33% <0%> (-16.67%)
1% <0%> (ø)

...n/java/com/google/api/gax/grpc/GrpcStatusCode.java
90.9% <0%> (-4.55%)
20% <0%> (ø)

... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d4907e6...1a9de99. Read the comment docs.


docs: correcting route ref, system design

Confused by the documented cardinality of Gateway components and their Proxy representations, I deployed the gateway with two HTTPRoutes and did a config dump of the gateway. The docs Route refers to the proxy RouteConfiguration, but it should indeed refer to Route.
Signed-off-by: Fredrik Geijer Haeggström<EMAIL_ADDRESS>

Codecov Report

Merging #570 (0cebb39) into main (61f0999) will decrease coverage by 0.12%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #570      +/-   ##
==========================================
- Coverage   60.33%   60.20%   -0.13%
==========================================
  Files          45       45
  Lines        5564     5564
==========================================
- Hits         3357     3350       -7
- Misses       2001     2006       +5
- Partials      206      208       +2

Impacted Files
Coverage Δ

internal/provider/kubernetes/helpers.go
73.91% <0.00%> (-6.53%)
:arrow_down:

internal/provider/kubernetes/gatewayclass.go
70.28% <0.00%> (-2.90%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Iguazio - Please add our column

Hello,
I would like to add our column to the spreadsheet.
This is the link to my copy. If you need to have it uploaded please let me know,
https://docs.google.com/spreadsheets/d/1PqUVgup6C1tRckdg2vDt6NYHiZlJQgiSaDhEEKD4fVA/edit?usp=sharing
Thanks,
Jason Novich
Technical Writer - Iguazio.com

Could you add some links to the product documentation for some of those features? I see links in the 'product focus' and 'pricing' rows but not in the rows highlighting features

I have added links to the categories in the features sections.
Let me know if there is anything else that I should add.
Thanks,
Jason Novich
Technical Writer - Iguazio.com

I've made some minor alterations and added the column to the public version now. Let us know if there's anything you'd like to be changed.


Authentication with PostgreSQL - password is in the clear, can it be encrypted

HI. My company's security policy won't allow passwords to be stored in the clear, so the yaml file would need to be encrypted. Or ldap2pg would need to read the password from an encrypted source. Integration with Vault might be an option. Is there a way to do this in the current version or are there plans to include this functionality please as without it we cannot use the product and it looks very useful. Thanks!

Hi @love2freefly ,
Can you use environment variables ?
Note that ldap2pg refuses to read passwords from a world readable file.

Closing stale issue.


Bump hadoop-client dependency to 3.1.4

Moves the hadoop-client dependency of the tools project from 3.1.1 to 3.1.4.

build


Shift-enter doesn't create a new cell

This is tested on the deploy for https://github.com/ToposInstitute/CatColab/commit/2de1f45991343a59f676721e0c0fac55cd37f2ef in firefox. Maybe this was introduced by #50? Serves us right deploying on a Friday afternoon!

It seems to work locally, so this is a tricky bug!

Seems to be working for me in the deployment! Is it completely broken for you or just under certain conditions?


Draft loses attachment

Please search to check for an existing issue (including closed issues, for which the fix may not have yet been released) before opening a new issue: https://github.com/k9mail/k-9/issues?q=is%3Aissue
Expected behavior
Saving a mail with attachment from KMail, syncing it with K9:
On the overview in the draft-folder the attachment is shown (paper clip symbol), when opening the mail no attachment is in, and when sending the mail there is no attachment either.
Actual behavior
see the attachment in the mail, download the full attachment, and send draft with attachment
Steps to reproduce

write draft
attach file
save as draft

Environment
K-9 Mail version: 5.206
Android version:
7 on Galaxy S7 Edge
Account type (IMAP, POP3, WebDAV/Exchange): IMAP

Hello ,
I believe that the "expected behavior" and "actual behavior" should be switched.
I encountered the same problem here: almost lost important files that I saved on draft.
Only had to shut down the internet connexion from my phone before sync so I can get them directly from my computer.
People should be aware to not open a draft with attachment on their phones, whether it is to complete it or to send it.
I would gladly help to provide any information to help solving this issue.

You are right, actual and expected are swapped...sorry

I just had the same problem in 5.208
Please fix, this is a really annoying issue.

same to me in <IP_ADDRESS> with the speciality, that I wrote the e-mail with the attachment online, then saved as draft. In the mail overview (draft drawer) I see the e-mail and showing that it contains an attachment. When opened in k9 editor the attachment seems to be vanished, because it's no more viewable. Saving the draft as draft again finally looses it.

Is this a duplicate of #954 and #1566?
I would also say it is related to #2485, which is a duplicate of #3110.
So it seems like this is affecting a number of people. Are there plans to fix it? What can I/we do to help?

Duplicate of #954


Staging branch

Merging featured Carousel into Production.
This includes the new carousel as well as firebase integration.

Added the environment variables to the NIghtly-hub environment on Vercel. The check should pass now


Gets stuck connecting via ssh

Describe the bug
I'm having trouble getting an example from the docs work.
The command I'm running is as follows:
venv/bin/python -m pyinfra -vvv --debug inventory.py exec -- echo "hello world"
--> Loading config...
--> Loading inventory...
    [pyinfra_cli.inventory] Creating fake inventory...
    [pyinfra_cli.inventory] Checking possible group_data at: ...

--> Connecting to hosts...
    [pyinfra.connectors.ssh] Connecting to: <IP_ADDRESS> ({'allow_agent': False, 'look_for_keys': False, '_pyinfra_ssh_forward_agent': False, '_pyinfra_ssh_config_file': None, '_pyinfra_ssh_known_hosts_file': '...', '_pyinfra_ssh_strict_host_key_checking': 'accept-new', '_pyinfra_ssh_paramiko_connect_kwargs': None, 'username': '...', 'port': 22, 'timeout': 10, 'pkey': ...})
    [pyinfra.connectors.sshuserclient.client] Loading SSH config: None

And it gets stuck here, until I press Ctrl+C.
Then it suddenly connects (and indeed I see it in journalctl on the server):
^CExiting upon user request!
    [<IP_ADDRESS>] Connected

Then again nothing happens, until I Ctrl+C again and it terminates.
Firstly, not sure why it says Loading SSH config: None when I gave it path to the config, but more importanly why does it get stuck?
Any way to debug this further?
My inventory is
servers = [
    ('<IP_ADDRESS>',{
        'ssh_port': 22,
        'ssh_user': '...',
        'ssh_key': '...',
        'ssh_key_password': '',
        'ssh_known_hosts_file': '...'
    }),
]

Meta
    System: Linux
      Platform: Linux-6.6.54-2-MANJARO-x86_64-with-glibc2.40
      Release: 6.6.54-2-MANJARO
      Machine: x86_64
    pyinfra: v3.1.1
      click: v8.1.7
      configparser: v7.1.0
      distro: v1.9.0
      gevent: v24.10.3
      jinja2: v3.1.4
      packaging: v24.1
      paramiko: v3.5.0
      python-dateutil: v2.9.0.post0
      pywinrm: v0.5.0
      setuptools: v75.2.0
      typeguard: v4.3.0
      typing-extensions: v4.12.2
    Executable: /home/roman/dev/homelab/venv/lib/python3.12/site-packages/pyinfra/__main__.py
    Python: 3.12.6 (CPython, GCC 14.2.1 20240805)

Debug log
  File "/home/roman/dev/homelab/venv/lib/python3.12/site-packages/pyinfra_cli/main.py", line 222, in cli
    _main(*args, **kwargs)
  File "/home/roman/dev/homelab/venv/lib/python3.12/site-packages/pyinfra_cli/main.py", line 348, in _main
    connect_all(state)
  File "/home/roman/dev/homelab/venv/lib/python3.12/site-packages/pyinfra/api/connect.py", line 27, in connect_all
    with progress_spinner(greenlet_to_host.values()) as progress:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/home/roman/dev/homelab/venv/lib/python3.12/site-packages/pyinfra/progress.py", line 146, in progress_spinner
    spinner_thread.join()
  File "/usr/lib/python3.12/threading.py", line 1149, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.12/threading.py", line 1169, in _wait_for_tstate_lock
    if lock.acquire(block, timeout):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/roman/dev/homelab/venv/lib/python3.12/site-packages/gevent/thread.py", line 276, in acquire
    acquired = BoundedSemaphore.acquire(self, blocking, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/gevent/_semaphore.py", line 184, in gevent._gevent_c_semaphore.Semaphore.acquire
  File "src/gevent/_semaphore.py", line 263, in gevent._gevent_c_semaphore.Semaphore.acquire
  File "src/gevent/_semaphore.py", line 253, in gevent._gevent_c_semaphore.Semaphore.acquire
  File "src/gevent/_abstract_linkable.py", line 521, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait
  File "src/gevent/_abstract_linkable.py", line 487, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait_core
  File "src/gevent/_abstract_linkable.py", line 490, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait_core
  File "src/gevent/_abstract_linkable.py", line 442, in gevent._gevent_c_abstract_linkable.AbstractLinkable._AbstractLinkable__wait_to_be_notified
  File "src/gevent/_abstract_linkable.py", line 451, in gevent._gevent_c_abstract_linkable.AbstractLinkable._switch_to_hub
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
gevent.exceptions.LoopExit: This operation would block forever
        Hub: <Hub '' at 0x7f69e6171580 epoll default pending=0 ref=0 fileno=3 resolver=<gevent.resolver.thread.Resolver at 0x7f69e80567b0 pool=<ThreadPool at 0x7f69e5f509e0 tasks=0 size=0 maxsize=10 hub=<Hub at 0x7f69e6171580 thread_ident=0x7f69e9850b80>>> threadpool=<ThreadPool at 0x7f69e5f509e0 tasks=0 size=0 maxsize=10 hub=<Hub at 0x7f69e6171580 thread_ident=0x7f69e9850b80>> thread_ident=0x7f69e9850b80>
        Handles:
[]

Looks like it's paramico that is the culprit here. Will check its repo.

I'm having the same problem.  The pyinfra commands work eventually but it takes a long time between the first connection and the rest.  I have 5 items in my inventory,
Connecting to hosts...
[pyinfra.connectors.ssh] Connecting to: <IP_ADDRESS> ({'allow_agent': True, 'look_for_keys': True, '_pyinfra_ssh_forward_agent': False, '_pyinfra_ssh_config_file': None, '_pyinfra_ssh_known_hosts_file': None, '_pyinfra_ssh_strict_host_key_checking': 'accept-new', '_pyinfra_ssh_paramiko_connect_kwargs': None, 'username': 'redacted', 'timeout': 10, 'password': 'redacted'})
[pyinfra.connectors.sshuserclient.client] Loading SSH config: None
I think the settings allow_agent and look_for_keys may be the cause.   Where is this configured globally or for specific hosts.

Oh, I forgot to update this issue but I've managed to track it down to gethostbyname where it apparently gets stuck. So in my case looks like it's got nothing to do with pyinfra.


Add DiskInfo to HeartBeat

This logs disk info on heat beat:
"PhysicalDiskInfo":{"DriveLetter":"C","VolumeDriveType":"Fixed","VolumeFileSystem":"NTFS","VolumeFileSystemLabel":"OSDisk","VolumeSize":"510928089088","VolumeSizeRemaining":"103830548480","DiskNumber":"0","DiskModel":"MTFDDAK512MBF1A","DiskIsSystem":"True","DiskIsBoot":"True","DiskSerialNumber":"XXXXXX","PhysicalMediaType":"SSD","PhysicalBusType":"SATA","PhysicalSpindleSpeed":"0"}
This should come in very handy when investigating why users run out of disk space.
@sanoursa  I'm using the existing GetPhysicalDiskInfo method, which  brings back more info that we need.  Let me know if you think it's worth seperating queryVolumeString into it's own method and just calling that here.   I'll update the failing unit test once we make a decision there.

Two questions:

Is there a perf hit from querying all the extra stats that we don't care about?
Are we bloating the logs / telemetry with redundant info, since most of those stats will be redundant on every heartbeat message?

Keep in mind that "queryVolumeString" is totally Windows-specific. What you might consider doing is add a flag to GetPhysicalDiskInfo so that you can ask it for all drive stats, or just size-related stats, and let each platform make its own decision about what that means.

@sanoursa that makes sense
To answer both questions.
Yes there is a perf hitting in getting all the disk info (an extra query).  Yes it also bloats the logs.
I'll update GetPhysicalDiskInfo to have a sizeStatsOnly flag.  That should solve both problems nicely.  Thanks!

@sanoursa any chance you may be able to give this a final review.  If your too busy let me know and I'll ask someone else.  Thank You!


Preconstruct DelegateX before calling connect/disconnect

This means fewer overloaded methods will be needed on the SignalSource object. It also more tightly associates this pointers with member function pointers by composing them into an obvious compound object.
One reason to want to reduce connect overloads, is to add debugging capabilities to the connect method. This is easier when only a single overload needs to be updated.
Reference: Issue #1269

I should probably mention, this was accomplished with a global search/replace regex:
Search: connect\(([^{].*)\)
Replace: connect({$1})
Note that the above updates both connect and disconnect calls.


Discrepancy in optimal threshold calculation between sklearn and torchmetrics ROC implementations

Bug description
There's a noticeable difference in the calculated optimal thresholds when comparing the ROC curve implementations between sklearn.metrics.roc_curve and torchmetrics.functional.roc. Specifically, using the same input data for similarity scores and labels, sklearn produces a significantly lower optimal threshold value compared to torchmetrics.
What version are you seeing the problem on?
v2.2
How to reproduce the bug
import numpy as np
import torch
from sklearn.metrics import roc_curve
import torchmetrics.functional as tm

# Given values
similarities = torch.tensor([0.0938, 0.0041, -0.1011, 0.0182, 0.0932, -0.0269, -0.0266, -0.0298,
                             -0.0200, 0.0816, -0.0122, -0.0026, 0.1237, -0.0149, 0.0840, -0.0192,
                             -0.0488, 0.0114, -0.0076, -0.0583])
is_kin_labels = torch.tensor([1, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0])

# Ensure data is on CPU for sklearn compatibility
similarities_ = similarities.cpu().numpy()
is_kin_labels_ = is_kin_labels.cpu().numpy()

# Sklearn calculation
fpr_, tpr_, thresholds_ = roc_curve(is_kin_labels_, similarities_)
maxindex_ = (tpr_ - fpr_).argmax()
best_threshold_sklearn = thresholds_[maxindex_]

# Torchmetrics calculation (assuming similarities and is_kin_labels are already on CPU or CUDA compatible)
fpr, tpr, thresholds = tm.roc(similarities, is_kin_labels, task='binary')
maxindex = (tpr - fpr).argmax()
best_threshold_torchmetrics = thresholds[maxindex].item()

# Output comparison
print(f"Best threshold sklearn: {best_threshold_sklearn:.6f} @ {maxindex_} index of {len(thresholds_)} (fpr={fpr_[maxindex_]:.6f}, tpr={tpr_[maxindex_]:.6f})")
print(f"Best threshold torchmetrics: {best_threshold_torchmetrics:.6f} @ {maxindex} index of {len(thresholds)} (fpr={fpr[maxindex]:.6f}, tpr={tpr[maxindex]:.6f})")

# Best threshold sklearn: 0.093200 @ 2 index of 10 (fpr=0.000000, tpr=0.428571)
# Best threshold torchmetrics: 0.523283 @ 3 index of 21 (fpr=0.000000, tpr=0.428571)

Error messages and logs
No response
Environment

Current environment

CUDA:

GPU:

NVIDIA GeForce RTX 3070 Laptop GPU

available:         True
version:           12.1

Lightning:

lightning:         2.2.1
lightning-utilities: 0.10.1
pytorch-lightning: 2.2.1
torch:             2.2.1
torchmetrics:      1.3.1
torchvision:       0.17.1

Packages:

absl-py:           2.1.0
aiohttp:           3.9.3
aiosignal:         1.3.1
asttokens:         2.4.1
attrs:             23.2.0
beautifulsoup4:    4.12.3
certifi:           2024.2.2
cfgv:              3.4.0
chardet:           5.2.0
charset-normalizer: 3.3.2
click:             8.1.7
contourpy:         1.2.0
cycler:            0.12.1
daemonize:         2.5.0
debugpy:           1.8.1
decorator:         5.1.1
distlib:           0.3.8
docstring-parser:  0.16
executing:         2.0.1
filelock:          3.13.1
fonttools:         4.50.0
frozenlist:        1.4.1
fsspec:            2023.12.2
gdown:             5.1.0
grpcio:            1.62.1
guildai:           0.9.0
identify:          2.5.35
idna:              3.6
importlib-resources: 6.3.2
ipython:           8.20.0
jedi:              0.19.1
jinja2:            3.1.3
joblib:            1.3.2
jsonargparse:      4.27.6
kiwisolver:        1.4.5
lightning:         2.2.1
lightning-utilities: 0.10.1
markdown:          3.6
markupsafe:        2.1.3
matplotlib:        3.8.3
matplotlib-inline: 0.1.6
mpmath:            1.3.0
multidict:         6.0.5
natsort:           8.4.0
networkx:          3.2.1
nodeenv:           1.8.0
numpy:             1.26.4
nvidia-cublas-cu12: <IP_ADDRESS>
nvidia-cuda-cupti-cu12: 12.1.105
nvidia-cuda-nvrtc-cu12: 12.1.105
nvidia-cuda-runtime-cu12: 12.1.105
nvidia-cudnn-cu12: <IP_ADDRESS>
nvidia-cufft-cu12: <IP_ADDRESS>
nvidia-curand-cu12: <IP_ADDRESS>
nvidia-cusolver-cu12: <IP_ADDRESS>
nvidia-cusparse-cu12: <IP_ADDRESS>
nvidia-nccl-cu12:  2.19.3
nvidia-nvjitlink-cu12: 12.3.101
nvidia-nvtx-cu12:  12.1.105
opencv-python:     <IP_ADDRESS>
packaging:         24.0
parso:             0.8.3
pexpect:           4.9.0
pillow:            10.2.0
pip:               24.0
pkginfo:           1.10.0
platformdirs:      4.2.0
pre-commit:        3.6.2
prompt-toolkit:    3.0.43
protobuf:          4.25.3
psutil:            5.9.8
ptyprocess:        0.7.0
pure-eval:         0.2.2
pygments:          2.17.2
pyparsing:         3.1.2
pysocks:           1.7.1
python-dateutil:   2.9.0.post0
pytorch-lightning: 2.2.1
pyyaml:            6.0.1
requests:          2.31.0
scikit-learn:      1.4.1.post1
scipy:             1.12.0
setuptools:        69.0.3
six:               1.16.0
soupsieve:         2.5
stack-data:        0.6.3
sympy:             1.12
tabview:           1.4.4
tensorboard:       2.16.2
tensorboard-data-server: 0.7.2
threadpoolctl:     3.3.0
torch:             2.2.1
torchmetrics:      1.3.1
torchvision:       0.17.1
tqdm:              4.66.2
traitlets:         5.14.1
triton:            2.2.0
typeshed-client:   2.5.1
typing-extensions: 4.9.0
urllib3:           2.2.1
virtualenv:        20.25.1
wcwidth:           0.2.13
werkzeug:          3.0.1
wheel:             0.42.0
yarl:              1.9.4

System:

OS:                Linux
architecture:

64bit
ELF

processor:
python:            3.11.8
release:           6.7.9-arch1-1
version:           Lightning-AI/pytorch-lightning#1 SMP PREEMPT_DYNAMIC Fri, 08 Mar 2024 01:59:01 +0000

More info
The output from thresholds_ (using sklearn) and thresholds (using torchmetrics) reveals a significant difference in the threshold values range and granularity:
[ins] In [6]: thresholds_
Out[6]:
array([    inf,  0.1237,  0.0932,  0.0114, -0.0026, -0.0149, -0.0192,
       -0.02  , -0.0266, -0.1011], dtype=float32)

[ins] In [7]: thresholds
Out[7]:
tensor([1.0000, 0.5309, 0.5234, 0.5233, 0.5210, 0.5204, 0.5045, 0.5028, 0.5010,
        0.4993, 0.4981, 0.4970, 0.4963, 0.4952, 0.4950, 0.4934, 0.4933, 0.4926,
        0.4878, 0.4854, 0.4747])

I think I found the problem. The returned thresholds are probabilities, because

preds (float tensor): (N, ...). Preds should be a tensor containing probabilities or logits for each observation. If preds has values outside [0,1] range we consider the input to be logits and will auto apply sigmoid per element.

So it makes sense. My fault. However, I didn't fit very clearly at first.

thresholds: an 1d tensor of size (n_thresholds, ) with decreasing threshold values

So it makes sense. My fault... However, I didn't find it very clear at first.

Could you pls suggest how to clarify it in docs or examples?


Few mp4 video not playing getting this error

logs
E/libc    ( 9028): Access denied finding property "vendor.display-size"
E/libc    ( 9028): Access denied finding property "vendor.display-size"
I/ExoPlayerImpl( 9028): Init 5b44a88 [ExoPlayerLib/2.15.1] [generic_x86, AOSP TV on x86, Google, 30]
I/flutter ( 9028): https://mydomain.com/api/subtitles/by/episode/11/4F5A9C3D9A86FA54EACEDDD635185/0214e686-443a-4924-8c6a-95d1de438a25/
E/ExoPlayerImplInternal( 9028): Playback error
E/ExoPlayerImplInternal( 9028):   com.google.android.exoplayer2.ExoPlaybackException: Source error
E/ExoPlayerImplInternal( 9028):       at com.google.android.exoplayer2.ExoPlayerImplInternal.handleIoException(ExoPlayerImplInternal.java:624)
E/ExoPlayerImplInternal( 9028):       at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:596)
E/ExoPlayerImplInternal( 9028):       at android.os.Handler.dispatchMessage(Handler.java:102)
E/ExoPlayerImplInternal( 9028):       at android.os.Looper.loop(Looper.java:223)
E/ExoPlayerImplInternal( 9028):       at android.os.HandlerThread.run(HandlerThread.java:67)
E/ExoPlayerImplInternal( 9028):   Caused by: com.google.android.exoplayer2.upstream.HttpDataSource$InvalidResponseCodeException: Response code: 403
E/ExoPlayerImplInternal( 9028):       at com.google.android.exoplayer2.upstream.DefaultHttpDataSource.open(DefaultHttpDataSource.java:389)
E/ExoPlayerImplInternal( 9028):       at com.google.android.exoplayer2.upstream.StatsDataSource.open(StatsDataSource.java:84)
E/ExoPlayerImplInternal( 9028):       at com.google.android.exoplayer2.source.ProgressiveMediaPeriod$ExtractingLoadable.load(ProgressiveMediaPeriod.java:1014)
E/ExoPlayerImplInternal( 9028):       at com.google.android.exoplayer2.upstream.Loader$LoadTask.run(Loader.java:409)
E/ExoPlayerImplInternal( 9028):       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
E/ExoPlayerImplInternal( 9028):       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
E/ExoPlayerImplInternal( 9028):       at java.lang.Thread.run(Thread.java:923)
E/flutter ( 9028): [ERROR:flutter/lib/ui/ui_dart_state.cc(209)] Unhandled Exception: PlatformException(VideoError, Video player had error com.google.android.exoplayer2.ExoPlaybackException: Source error, , null)

I/VideoCapabilities( 9028): Unsupported profile 4 for video/mp4v-es
I/OMXClient( 9028): IOmx service obtained
D/SurfaceUtils( 9028): connecting to surface 0xeb847e08, reason connectToSurface
I/MediaCodec( 9028): [OMX.google.h264.decoder] setting surface generation to 9244673
D/SurfaceUtils( 9028): disconnecting from surface 0xeb847e08, reason connectToSurface(reconnect)
D/SurfaceUtils( 9028): connecting to surface 0xeb847e08, reason connectToSurface(reconnect)
E/ACodec  ( 9028): [OMX.google.h264.decoder] setPortMode on output to DynamicANWBuffer failed w/ err -1010
I/ACodec  ( 9028): codec does not support config priority (err -1010)
I/OMXClient( 9028): IOmx service obtained
I/ACodec  ( 9028): codec does not support config priority (err -2147483648)
I/ACodec  ( 9028): codec does not support config operating rate (err -2147483648)
I/yal.alenwan_ot( 9028): Background young concurrent copying GC freed 23169(1064KB) AllocSpace objects, 3(124KB) LOS objects, 12% free, 7397KB/8413KB, paused 1.997ms total 101.811ms
D/MediaCodec( 9028): [OMX.google.h264.decoder] setting dataspace on output surface to #104
W/Gralloc4( 9028): allocator 3.x is not supported
D/SoftwareRenderer( 9028): setting dataspace on output surface to #104
D/AudioTrack( 9028): getTimestamp_l(34): device stall time corrected using current time<PHONE_NUMBER>815

Hi @iamwasimaftab, thanks for filing the issue.
Please provide the output of flutter doctor -v as well as the version of video_player you're using.
Please also provide a sample URL/video that you are experiencing this issue with as well as a complete minimal reproducible code sample.
Thank you

[✓] Flutter (Channel stable, 2.5.3, on macOS 12.0.1 21A559 darwin-x64, locale en-AE)
• Flutter version 2.5.3 at /Users/Cylinder/flutter
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision 18116933e7 (4 months ago), 2021-10-15 10:46:35 -0700
• Engine revision d3ea636dc5
• Dart version 2.14.4
[✓] Android toolchain - develop for Android devices (Android SDK version 32.0.0)
• Android SDK at /Users/Cylinder/Library/Android/sdk
• Platform android-32, build-tools 32.0.0
• Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
• Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7281165)
• All Android licenses accepted.
[!] Xcode - develop for iOS and macOS
✗ Xcode installation is incomplete; a full installation is necessary for iOS development.
Download at: https://developer.apple.com/xcode/download/
Or install Xcode via the App Store.
Once installed, run:
sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer
sudo xcodebuild -runFirstLaunch
• CocoaPods version 1.11.2
[✓] Chrome - develop for the web
• Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
[✓] Android Studio (version 2020.3)
• Android Studio at /Applications/Android Studio.app/Contents
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart
• Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7281165)
[✓] Connected device (2 available)
• AOSP TV on x86 (mobile) • emulator-5554 • android-x86    • Android 11 (API 30) (emulator)
• Chrome (web)            • chrome        • web-javascript • Google Chrome 98.0.4758.80
! Doctor found issues in 1 category.

To update it, use flutter pub upgrade.
7  dependencies are constrained to versions that are older than a resolvable version.
To update these dependencies, edit pubspec.yaml, or run flutter pub upgrade --major-versions.

Please provide a sample url that we can use to verify this behavior.
Thank you