MovingAverage does not work with MirroredStrategy

System information

OS Platform and Distribution: Ubuntu 18.04
TensorFlow version and how it was installed (source or binary): 2.0 from PyPi
TensorFlow-Addons version and how it was installed (source or binary): 0.5.2 from PyPi
Python version: 3.6

Describe the bug
If I compile a Keras model with a MovingAverage optimizer and a LearningRateScheduler, I get an error "Optimizer must have a "lr" attribute." at tensorflow_core/python/keras/callbacks.py:1342. I can fix that by the following code:
@keras_utils.register_keras_custom_object
class LRMovingAverage(tfa.optimizers.MovingAverage):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    @property
    def lr(self):
        return self._optimizer.lr

However, my model is compiled under tf.distribute.MirroredStrategy().scope() and I crash in fit():
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 681, in on_epoch
    yield epoch_logs
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 324, in fit
    total_epochs=epochs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 123, in run_one_epoch
    batch_outs = execution_function(iterator)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 86, in execution_function
    distributed_function(input_fn))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 457, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 503, in _call
    self._initialize(args, kwds, add_initializers_to=initializer_map)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 408, in _initialize
    *args, **kwds))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 1848, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2150, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2041, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 358, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 73, in distributed_function
    per_replica_function, args=(model, x, y, sample_weights))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/distribute/distribute_lib.py", line 760, in experimental_run_v2
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/distribute/distribute_lib.py", line 1787, in call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 661, in _call_for_each_replica
    fn, args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 196, in _call_for_each_replica
    coord.join(threads)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/coordinator.py", line 389, in join
    six.reraise(*self._exc_info_to_raise)
  File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/coordinator.py", line 297, in stop_on_exception
    yield
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 190, in _call_for_each_replica
    **merge_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py", line 446, in_distributed_apply
    ds_reduce_util.ReduceOp.SUM, grads_and_vars)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/distribute/distribute_lib.py", line 1481, in batch_reduce_to
    return self._batch_reduce_to(reduce_op, value_destination_pairs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 707, in _batch_reduce_to
    value_destination_pairs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/distribute/cross_device_ops.py", line 317, in batch_reduce
    value_destination_pairs[0][0].values) == 1:
IndexError: list index out of range

Code to reproduce the issue
TODO

@PhilJd Hi, Phil, could you take a look? Thanks

@vmarkovtsev Hi, Vadim, can you provide a minimal reproducible example? Thank you

cc @dubey @guptapriya - probably this is an upstream problem

@facaiy: I think @Squadrick  might be more familiar with it as he implemented it?

@vmarkovtsev I get the error of TFA 0.5.2 but building and running TFA doesn't give me an error. Could you test it with the latest TFA version?

I tried 0.6.0 and it did not work. Then I tried pip install git+https and the build failed with stub.cc: No such file or directory. I cannot install tfa-nightly because it requires tf-nightly which is 2.1.0 and it breaks my world. So nope, I cannot test that myself, sorry.

@Squadrick I will be happy to install and test a wheel for Python 3.6 if you are able to build it and attach here.

I tried 0.6.0 and it did not work. Then I tried pip install git+https and the build failed with stub.cc: No such file or directory. I cannot install tfa-nightly because it requires tf-nightly which is 2.1.0 and it breaks my world. So nope, I cannot test that myself, sorry.

Hi @VladimirStarostenkov could you try installing pip install tfa-nightly --no-deps so there is no requirement for tf-nightly?

@seanpmorgan not sure if that helps, but I was able to reproduce it.
Python 3.6.8
vladimir@vladmsi:~/tf-additions$ python3 -m venv ./env
vladimir@vladmsi:~/tf-additions$ source env/bin/activate
(env) vladimir@vladmsi:~/tf-additions$ pip install --upgrade pip
...
Successfully installed pip-19.3.1

(env) vladimir@vladmsi:~/tf-additions$ pip install tfa-nightly --no-deps
Collecting tfa-nightly
  Downloading https://files.pythonhosted.org/packages/69/1d/782a3dcc8690b76f15f6c3abd7928986848b1d7dcbcf46887209b57f044b/tfa_nightly-0.7.0.dev20191103-cp36-cp36m-manylinux2010_x86_64.whl (1.9MB)
     |████████████████████████████████| 1.9MB 1.1MB/s
Installing collected packages: tfa-nightly
Successfully installed tfa-nightly-0.7.0.dev20191103

(env) vladimir@vladmsi:~/tf-additions$ pip install tensorflow
Collecting tensorflow
  Downloading https://files.pythonhosted.org/packages/46/0f/7bd55361168bb32796b360ad15a25de6966c9c1beb58a8e30c01c8279862/tensorflow-2.0.0-cp36-cp36m-manylinux2010_x86_64.whl (86.3MB)
     |████████████████████████████████| 86.3MB 2.7MB/s
...
ERROR: tfa-nightly 0.7.0.dev20191103 requires tf-nightly, which is not installed.
...
Successfully installed absl-py-0.8.1 astor-0.8.0 cachetools-3.1.1 certifi-2019.9.11 chardet-3.0.4 gast-0.2.2 google-auth-1.6.3 google-auth-oauthlib-0.4.1 google-pasta-0.1.7 grpcio-1.24.3 h5py-2.10.0 idna-2.8 keras-applications-1.0.8 keras-preprocessing-1.1.0 markdown-3.1.1 numpy-1.17.3 oauthlib-3.1.0 opt-einsum-3.1.0 protobuf-3.10.0 pyasn1-0.4.7 pyasn1-modules-0.2.7 requests-2.22.0 requests-oauthlib-1.2.0 rsa-4.0 setuptools-41.6.0 six-1.12.0 tensorboard-2.0.1 tensorflow-2.0.0 tensorflow-estimator-2.0.1 termcolor-1.1.0 urllib3-1.25.6 werkzeug-0.16.0 wheel-0.33.6 wrapt-1.11.2

(env) vladimir@vladmsi:~/tf-additions$ python moving_average.py
2019-11-04 09:54:57.279453: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-11-04 09:54:57.306281: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-11-04 09:54:57.307197: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4383400 executing computations on platform Host. Devices:
2019-11-04 09:54:57.307212: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
WARNING:tensorflow:Entity <function main.<locals>.<lambda> at 0x7f2dffc44c80> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: expected exactly one node node, found []
WARNING:tensorflow:There is non-GPU devices in `tf.distribute.Strategy`, not using nccl allreduce.
Model: "densenet121"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
...
==================================================================================================
Total params: 7,047,754
Trainable params: 6,964,106
Non-trainable params: 83,648
__________________________________________________________________________________________________
Train for 100 steps, validate for 10 steps
  1/100 [..............................] - ETA: 4:42Traceback (most recent call last):
  File "moving_average.py", line 37, in <module>
    sys.exit(main())
  File "moving_average.py", line 33, in main
    model.fit(ds_train, validation_data=ds_val, epochs=1, steps_per_epoch=100)
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 728, in fit
    use_multiprocessing=use_multiprocessing)
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 324, in fit
    total_epochs=epochs)
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 123, in run_one_epoch
    batch_outs = execution_function(iterator)
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 86, in execution_function
    distributed_function(input_fn))
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 457, in __call__
    result = self._call(*args, **kwds)
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 503, in _call
    self._initialize(args, kwds, add_initializers_to=initializer_map)
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 408, in _initialize
    *args, **kwds))
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1848, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 2150, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 2041, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 358, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 73, in distributed_function
    per_replica_function, args=(model, x, y, sample_weights))
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/distribute/distribute_lib.py", line 760, in experimental_run_v2
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/distribute/distribute_lib.py", line 1787, in call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 661, in _call_for_each_replica
    fn, args, kwargs)
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 196, in _call_for_each_replica
    coord.join(threads)
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/training/coordinator.py", line 389, in join
    six.reraise(*self._exc_info_to_raise)
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/training/coordinator.py", line 297, in stop_on_exception
    yield
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 190, in _call_for_each_replica
    **merge_kwargs)
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py", line 446, in _distributed_apply
    ds_reduce_util.ReduceOp.SUM, grads_and_vars)
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/distribute/distribute_lib.py", line 1481, in batch_reduce_to
    return self._batch_reduce_to(reduce_op, value_destination_pairs)
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 707, in _batch_reduce_to
    value_destination_pairs)
  File "/home/vladimir/tf-additions/env/lib/python3.6/site-packages/tensorflow_core/python/distribute/cross_device_ops.py", line 317, in batch_reduce
    value_destination_pairs[0][0].values) == 1:
IndexError: list index out of range

The code runs fine on my local machine. Pulled the latest master and built it from scratch on Google Colab and it runs without errors as well.
Link: https://colab.research.google.com/drive/17dYDWJJo7vJOAoO6JCSR-BBwlPWH1fKM

@Squadrick Colab works because this requires multiple devices.

@vmarkovtsev I was able to recreate the error on Colab with the same hardware as before (no accelerators) and no multiple devices.
I used tensorflow_addons==0.6.0 and tensorflow==2.0.0 instead of the tfa_nightly and tf_nightly.
Link to recreated error: https://colab.research.google.com/drive/1VFzf57e5v6awNi_Y4edFeH4t3GzPFL96

Great, so since @VladimirStarostenkov reproduced it with tfa_nightly and tensorflow==2.0, I can conclude that upgrading tensorflow to the future 2.1 should fix the problem.

@vmarkovtsev Closing this issue, feel free to reopen it if you run into any more problems.

I will ping here if I have problems because issue authors cannot reopen their issues on GitHub if they were closed by maintainers.

tf 2.4.4, tfa 0.14.0
With mirrored strategy, tfa.ema and tfa.swa will raise InvalidArgError. But without mirrored strategy, model.fit runs fine
  File "/workspace/trainer.py", line 647, in train_step
    self.model.fit(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1095, in fit
    tmp_logs = self.train_function(iterator)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 828, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 955, in _call
    return function_lib.defun(fn_with_cond)(canon_args, canon_kwds,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 2942, in __call__
    return graph_function._call_flat(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1918, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 555, in call
    outputs = execute.execute(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: 3 root error(s) found.
  (0) Invalid argument:  Trying to assign variable with wrong dtype. Expected INVALID got int64
         [[{{node cond/else/_1/cond/StatefulPartitionedCall/iter/cond/else/_5113/iter/cond/Assign}}]]
         [[cond/else/_1/cond/StatefulPartitionedCall/ReadVariableOp_2/_1762]]
  (1) Invalid argument:  Trying to assign variable with wrong dtype. Expected INVALID got int64
         [[{{node cond/else/_1/cond/StatefulPartitionedCall/iter/cond/else/_5113/iter/cond/Assign}}]]
         [[ConstantFoldingCtrl/cond/else/_1/cond/StatefulPartitionedCall/SWA/SWA/update_19/update_1/StatefulPartitionedCall/cond/switch_pred/_10666_0/_3993]]
  (2) Invalid argument:  Trying to assign variable with wrong dtype. Expected INVALID got int64
         [[{{node cond/else/_1/cond/StatefulPartitionedCall/iter/cond/else/_5113/iter/cond/Assign}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference_fn_with_cond_41727]

Function call stack:
fn_with_cond -> fn_with_cond -> fn_with_cond

I'm facing a similar InvalidArgError, with MovingAverage on tf 2.6.2 and tfa 0.14.0, if I enable XLA with MirroredStrategy. If I disable XLA, it works:
Traceback (most recent call last):
  File "tf_imagenet_rc_v7.py", line 711, in <module>
    main(args, logger)
  File "tf_imagenet_rc_v7.py", line 420, in main
    model.fit(pipeline_train, validation_data=pipeline_val, epochs=args.epochs_n, callbacks=callbacks,
  File "/opt/conda/envs/tf262/lib/python3.8/site-packages/keras/engine/training.py", line 1184, in fit
    tmp_logs = self.train_function(iterator)
  File "/opt/conda/envs/tf262/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 885, in __call__
    result = self._call(*args, **kwds)
  File "/opt/conda/envs/tf262/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 1017, in _call
    return function_lib.defun(fn_with_cond)(canon_args, canon_kwds,
  File "/opt/conda/envs/tf262/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 3039, in __call__
    return graph_function._call_flat(
  File "/opt/conda/envs/tf262/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1963, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "/opt/conda/envs/tf262/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 591, in call
    outputs = execute.execute(
  File "/opt/conda/envs/tf262/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  Cannot update variable with shape [0] using a Tensor with shape [], shapes must be equal.
         [[{{node cond/else/_1/cond/StatefulPartitionedCall/cond_5/then/_19229/cond_5/MovingAverage/MovingAverage/update_1_1/AssignAddVariableOp}}]]
  (1) Invalid argument:  Cannot update variable with shape [0] using a Tensor with shape [], shapes must be equal.
         [[{{node cond/else/_1/cond/StatefulPartitionedCall/cond_5/then/_19229/cond_5/MovingAverage/MovingAverage/update_1_1/AssignAddVariableOp}}]]
         [[GroupCrossDeviceControlEdges_0/cond/else/_1/cond/StatefulPartitionedCall/cond_5/branch_executed/_21374/_3935]]
0 successful operations.
0 derived errors ignored. [Op:__inference_fn_with_cond_113358]


Added Thanos sort in java

Snap! Its October! Hacktober! and Thanos is in play as he sorts an array in his distinct style.

What a great code 😄 !
Thank you for contributing! 🚀


Scaffolder: Template action to create partial templates

Feature Suggestion
Right now, fetch comes with plain and template actions, and working on a template with GH actions and Django templates, I really felt the urge to have a mix between the two.
Possible Implementation
My idea (and WIP patch) is to apply templating to files with a dedicated extension (.njk is suggested by nunjucks) to trigger content processing, and to do path processing at all times.
Context
The idea is to have a single template with

proper syntax and syntax highlighting for unmodified files
proper syntax and syntax highlighting for modified files
consistently adjusted paths for files and directories
all in one template

I think templates are they are are heavily coupled between the template contents and the template declaration (yaml file). With that I think it could be reasonable to add ignore/include options for which files end up being templated? I can see that GitHub workflows are a bit of a pain to template.
Another workaround you can do right now is to split it into two separate directories and bring the workflows in with a plain fetch and then template the rest, but it's not entirely optimal.
Tbh I also don't think it's that big of a deal to have to escape a few things, so we shouldn't go out of our way to fix this imo 😅

It's been easy enough to just hack up an action. I have a patch and tests locally, I'll create a PR tomorrow.
One thing that's nice is that I didn't have to bother about cookiecutter compat, and it's problems around templating file names. Which I think is a cookiecutter bug to some extent, which I filed as https://github.com/cookiecutter/cookiecutter/issues/1588.


fix TYPE_DYNAMIC_ATTRIBUTE support

fixes #1382

Thanks for making a quick PR and helping me research that one. Will get this merged.


feat: add nai-v3 samplers

V3 Sampler Options
Samplers

k-euler
k_euler_ancestral
k_dpmpp_2s_ancestral
k_dpmpp_2m
k_dpmpp_sde
ddim_v3

Notes

The "noise_schedule" is automatically set to "exponential" for k_dpmpp_2m. For other samplers, it defaults to "noise_schedule": "native".

Limitations under ddim_v3 Sampler

The following features are not supported:

"noise_schedule": "native" (This option should be removed)
"sm": false
"sm_dyn": false (These two options should be set to false)

New Options

"sm": false (Needs to be enabled for the next option)
"sm_dyn": false (Can be used only when "sm" is true)
"dynamic_thresholding": bool

A Valid Reques
  {
  "input": "1boy, 1girl, blurry background, {red eyes}, medium hair, crown braid, purple hair, gradient hair, blonde hair, blunt ends, hair intakes, flat chest, female service cap, hat ribbon, armor, ok sign, hood, orange eyes, bags under eyes, big hair, asymmetrical hair, brown hair, skinny, multicolored argyle sweater, flip-flops, arm grab, pendant, best quality, amazing quality, very aesthetic, absurdres",
  "model": "nai-diffusion-3",
  "action": "generate",
  "parameters":
    {
      "params_version": 1,
      "width": 1216,
      "height": 2560,
      "scale": 5,
      "sampler": "k_euler_ancestral",
      "steps": 8,
      "n_samples": 1,
      "ucPreset": 0,
      "qualityToggle": true,
      "sm": true,
      "sm_dyn": true,
      "dynamic_thresholding": true,
      "controlnet_strength": 1,
      "legacy": false,
      "add_original_image": false,
      "uncond_scale": 1,
      "cfg_rescale": 0,
      "noise_schedule": "native",
      "legacy_v3_extend": false,
      "seed":<PHONE_NUMBER>,
      "negative_prompt": "nsfw, lowres, {bad}, error, fewer, extra, missing, worst quality, jpeg artifacts, bad quality, watermark, unfinished, displeasing, chromatic aberration, signature, extra digits, artistic error, username, scan, [abstract]",
    },
}

@MaikoTan done, take a look?

fix #218

fix #215

fix #212


Mobile gestures support

Feature Description
I want to be able to open and close a sidenav via swipe on mobile devices.
Some components have no touch support for mobile gestures.
Use Case
No response

Duplicate of https://github.com/angular/components/issues/5712.


You aren't supposed to have a readme.html

Try renaming the file index.html
Then put the site up to github pages.


Calibnet reset causing network_version error!

Currently network_version is determined by block height. However, as calibnet is reset, it will always get NetworkVersion::V13.
Related code
https://github.com/ChainSafe/forest/blob/775e9e86a23ef429e14f2d27ff51d467ef9ab5b5/networks/src/lib.rs#L202

Fixed in main. A new release will be out in a bit.


WIP #321 tests are green after migration to play 2.6

Hello @yanns, it's me again. :)
I finally managed to find some time to dig into akka-streams and after that I was able to make tests green.
woohoo! (I guess :smile: )
I had to use play.api.libs.streams.Execution.Implicits.trampoline in order to not get into concurrency issues with MimeHeaders, which is not thread-safe : https://bz.apache.org/bugzilla/show_bug.cgi?id=47655#c9
@jroper regarding blocking IO in your comment https://github.com/play2war/play2-war-plugin/pull/323#discussion_r199371075 - I tried to use https://github.com/jroper/reactive-streams-servlet but unfortunately Sink.fromSubscriber(...) returnes Sink[T, NotUsed] without materialized future value. I tried using .watchTermination() and other similar solutions, but I still was getting concurrency issues above.
If you guys have any time to check the code, and maybe suggest something, I am open for discussion.
Next steps will be to clean logging, usage of deprecated API, and comments, once you think that we are ready to go. :fearful:
PS.
I am going to proceed with upgrading my app that uses this plugin and see if it works with our app.


Fix link to phantom page in the documentation.

The README has a link to the Feature page for Redux which actually doesn't exist. So, removing it to prevent confusion.

Merging. Thanks!


Problem when loading a saved Scion - Ascendant build

When I load my Scion - Ascendant build, the center node for the Ascendant ascendancy class gets unallocated, even if I reallocate it, save again, and try to load.  This happens any time I double click the build or restart the program.
The consequences of this are strange.  Allocating more passives works fine, but because the build takes Path of the Shadow, if I forget to reallocate that Ascendant node before unallocating any node, the program also unallocates all of the Ascendency points in the Ascendant section, including Path of the Shadow, which in turn unallocates everything that begins from the shadow's starting point.
So every time I load my build, I can't make any significant changes to it without first remembering to hit the center node in the Ascendant class again, or else I have to redo half of my tree (or reload the save.)

This has been fixed but not released yet, sorry.

Ah sorry, I didn't think to check closed issues.  My mistake.


Add appveyor for windows testing.

I've never used appveyor, but my understanding is that it would automatically answer questions like https://github.com/scipy/scipy/pull/4374#issuecomment-69768585.

I don't think it has a Fortran compiler.
http://www.appveyor.com/docs/installed-software

https://github.com/numpy/numpy/pull/5328

NumPy does not require Fortran and can be built with Microsoft's VisualStudio tools. Those are on appveyor. SciPy is typically built with gcc and gfortran or icc and ifort. We do not even have a free toolchain able to produce binary 64-bit installers.

Fixed by gh-7616, closing.

Sample appveyor (and other CI systems) configurations for using intel C++, Fortran, DPC++
https://github.com/oneapi-src/oneapi-ci


Add SMD chip resistors

IPC naming for chip resistors
Height in the name is the max height (I pulled the most common resistor heights from Digikey)
The smaller variants don't have silkscreen anymore
Silkscreen lines are offset slightly, so that they should still be visible when the part has been mounted

3216 resistor:

0603 resistor:

5025 resistor:

Generated with https://github.com/LibrePCB/librepcb-parts-generator/pull/29.

Some of these packages already exist in this library -> please reuse UUIDs to overwrite them.
Pad 1 is at the right, but should be at the left.
Footprints "reflow" and "hand soldering" are exactly the same -> would be good to replace them by IPC density levels A/B/C anyway :)

Some of these packages already exist in this library -> please reuse UUIDs to overwrite them.

I'll try to do that, although it requires manually patching the UUID cache file.

Pad 1 is at the right, but should be at the left.

Easy fix, thanks for spotting.

Footprints "reflow" and "hand soldering" are exactly the same -> would be good to replace them by IPC density levels A/B/C anyway :)

Oops, I missed that one.
Regarding density levels, I found heel/toe/side guidelines in IPC and started implementing them. However, I don't think we can implement density level C in generic parts. The level C has very little tolerances, so it needs to match the part perfectly. (Remember, IPC specifies protrusion, not size.)
The thing is that we don't know the size of the coated leads of the chip resistors. The sizes are standardized, but not the length of the leads. So if we can't know the length of the lead, we can't really do well-matched footprints.
What about implementing footprint variants "Density Level A (Hand Soldering)" and "Density Level B (Reflow Soldering)" and leaving Density Level C to manufacturer specific libraries?

Courtyard was added (according to IPC):
2012 footprint (level B):

2012 footprint (level A):

The thing is that we don't know the size of the coated leads of the chip resistors. The sizes are standardized, but not the length of the leads. So if we can't know the length of the lead, we can't really do well-matched footprints.

But then it might make sense to specify the lead length in our resistor packages (choosing the most commonly used lead length). Then we can add a density level C footprint, and these packages can be used for real devices if the lead length matches. In future we could also add more packages with different lead lengths.

What about implementing footprint variants "Density Level A (Hand Soldering)" and "Density Level B (Reflow Soldering)" and leaving Density Level C to manufacturer specific libraries?

We can't move only one footprint to a manufacturer specific library, we would have to create a completely new package with density levels A and B specified again.
I noticed that you added packages with heights in 0.05mm interval. I think this is a far too small interval, it would lead to a huge amount of resistor packages. I would rather use an interval of at least 0.2mm. Then probably we only need one height per size to cover 90% of the available real resistors ;)
Now I just found this sentence in the German Wikipedia:

Die Höhe der Bauteile ist nicht standardisiert, bei Widerständen ist sie jedoch meist gleich, während sie bei Kondensatoren je nach Hersteller, Kapazität und Spannungsfestigkeit stark schwanken kann.

So maybe it would make sense to even ignore the different heights completely (adding only one package per size, without specifying a height)? This would simplify things a lot, and for the 3D model a height error of e.g. 0.2mm doesn't matter IMHO...

Some parts are not currently generated that were previously available as R-XXXX packages. I believe some of them are actually duplicates of others, but with metric name instead of imperial:

Oh f*ck, I didn't realize that (I imported them from a Eagle library) :sob:
I would suggest to add the R-1218 and R-4527 as metrical variants 3246 resp. 11470 (or 11570?). But I'm not sure about the duplicate packages. I tend to remove them, although normally we shouldn't remove packages... We can just hope that nobody use them already ;)

We can't move only one footprint to a manufacturer specific library, we would have to create a completely new package with density levels A and B specified again.

Yes, but the different variants could then exactly match the manufacturer recommendations.

I noticed that you added packages with heights in 0.05mm interval. I think this is a far too small interval, it would lead to a huge amount of resistor packages. I would rather use an interval of at least 0.2mm.

I used Digikey to find common heights. I only added them for variants which actually exist for thousands of different resistors, I didn't generate all combinations. (For example, the 1005 resistor only has 1 height variant.) Do you think the current amount of resistor packages is too much?
Generating 3D models later for different heights is trivial, but I would prefer it if the max height would actually match the part (for when you do very thin, space-constrained designs). And when considering mechanical constraints, then the height is important (much more so than the color).

I would suggest to add the R-1218 and R-4527 as metrical variants

I can do that, won't hurt much.

We can just hope that nobody uses them already ;)

Since the parts are copied to the project, it won't break anything, right? And they can be trivially replaced.

Ok, branch updated.
I added 3246 (1218) footprints:

I did not add the 11470 (4527) footprint. These are packages that look different from other chip resistors, they have J-style leads and are quite rare. Here's an example:
https://www.newark.com/vishay/wsc45271k000fea/res-wirewound-1k-1-2w-4527/dp/65K4016
I don't think these should be generated with the same generator (I'd have to add quite a few conditionals because the leads are less wide than the package). I'd actually vote for removal of the package, I think the current one has pads that are too wide anyways.

@ubruhin ping :slightly_smiling_face:

OK, another idea: Probably most users (especially hobby users) will use "generic" resistor devices anyway (called e.g. "R-0805") because they don't care about which exact resistor will be assembled, and because it would take a lot of time to add all of the (thousands?) available (manufacturer-specific) resistors to our libraries. Thus maybe the best option is to make the corresponding packages for these devices also that generic that there is only one package per size, i.e. using only one specific lead length and one specific height (or even not specifying the height at all), without density level C footprint. So basically what you did now, but removing the packages with different heights.
In addition to these generic resistor packages and devices, specific packages and devices can be added step by step, with lead lengths and heights according the datasheets and so on. These are completely independent of the generic packages/devices, and thus out of scope of this PR.
So, a hobby user typically just uses the generic "R-0805" device with its generic "RESC2012 (0805)" package and assembles the random 0805 resistor he found in his resistor set. He doesn't care at all about lead length and package height, as long as he can hand-solder it and the 3D view looks nice.
On the other hand, a professional which might want to do a design with density level C and needs an accurate 3D export chooses the specific device of the exact resistor he evaluated, which has the corresponding package assigned, with exact lead length and package height, so it comes with density level C footprint and exact 3D model.
I think this would be a good compromise and suits both workflows very well. What do you think @dbrgn?
Btw:

Since the parts are copied to the project, it won't break anything, right? And they can be trivially replaced.

Removing packages wouldn't break existing projects, but it would break existing devices. Meaning that you get an error when placing such a device on a board because the package can't be found.

Yes, I think that sounds sensible!
It would be good to have another opinion from @rnestler or @HBadertscher, if one of you has time :slightly_smiling_face:

The nightly build shows clearance warnings between pads and placement layer

That's a float rounding issue. The silkscreen ends at X 2.15 and the pad starts at X 2.3. The difference (2.3 - 2.15) is 0.14999999999, so your tool complains.
I'll file a bug report (maybe I can also fix it).

IMHO we should clearly distinguish between generic and specific packages

You're right, I changed it. The max height was primarily meant for the 3D model, where we need to choose some height. I left in the parameters in the source file, but won't render it.
I also added a note to the description that the packages are generic.

You also need to remove the corresponding devices as they are no longer valid ;)

Done.

Looks very good now :+1:
The only thing which would be nice, is to see the actual lead size in the documentation layer, so the pad protrusion gets visually visible (at least the outer dimension, as the lead length of the generic packages aren't specified). Actually on the master branch this is already the case, but your change removes that "feature" :grin:

Would it be possible that add this again? (the glue rect in the center might also be nice, but let's ignore that for the moment)
I noticed that even the density level A pads are much smaller that the pads imported from Eagle, I wonder if they are not too small now for hand soldering :thinking:

Updated doc layer.


Bug Report: Song crashing when hovered over in freeplay

I tried adding a song to the Freeplay system and it keeps crashing when hovered over. I even tried it on a new downloaded kade engine folder without any modification of the code to see if it's anything related to the code I changed. still crashes. I've put "Breaking Point:spirit:7" in the freeplaysonglist.txt. The data folder, chart JSON, and song folder are named "breaking-point". I'm not sure if I'm probably missing any steps or it's the engine bug. I also have the "Inst" and "Voices" files named correctly in the song folder with both versions of mp3 and Ogg files inside.

same that also happened to me ;3;

also happening to me :/

did you add the easy normal and hard diffs (if you don't have them copy and paste the original diff and rename them.)
Ke requires all of the diffs to be there. This is more or less not a bug, and more user error. Though I can't blame you since it doesn't tell you this, which has been clarified in the latest commits.


[front] - fix: radix console logs first batch

Description
This PR aims at fixing the radix-related console logs, namely some hydration errors.
Risk
Low
Deploy Plan
Deploy front

Note: we still have some of them in the assistant builder coming from the following components:

Sharing
AdvancedSettings

Those require to pass a ref to the Button which is currently not possible. The Button is to be refactored soon, will tackle this after the refactoring.

There's also one in the EmojiSelector in front/components/assistant/conversation/messages/MessageActions.tsx, right before the EmojiPicker


[loc][Query][vscode-azure-sphere-tools] Localizability of string: 0; "./src/resources.errorConfigUnavailable"

Hi,
please help localization team to answer question from translator.
In case translator is not sure about context/meaning/localizability of string we raise a question to Core team.
Could you help please? Thanks!**
Source:
Error: Required azure sphere settings not found. Please configure in settings.json or run command 'Azure Sphere: Configure settings' to apply default settings.
File:
Src\vscode-extensions\vscode-azure-sphere-tools.xlf
Query:
Please, determine whether this command should be translated

Please check https://marketplace.visualstudio.com/ for the extension this belongs to.


redirect to latest version

I released a new version of my code but docland didn't pick it up yet when I visit the URL without a version in it. It'd be nice to have it auto-redirect to the latest version whenever someone navigates to the URL without a version in it.
e.g. https://doc.deno.land/https://deno.land/x/xnd/_index.ts →<EMAIL_ADDRESS>(currently)

The doc website, per instance, caches the documentation in memory, so it takes a while for it to purge.
That being said, when the doc website gets directed on the URL being supplied, it bubble the redirect up to the doc URL, which in this case would do what is suggested here.  It is a little complex, because currently we just hand the URL over to deno_doc and don't fetch it initially.  Certainly worth considering.


output raw plantuml

related to #501 and #474.
we should add a way to output the raw plantuml for debugging (proposed by @anzdaddy)

whoops nevermind, to output raw uml you can set the output filename to have .uml extension. The generated raw plantuml is still wrong, will close this issue and investigate further.

Hehe. I'd forgotten about that.


JSDoc Recommendations for properties, attributes, and events, etc.

Should there be standards for documenting properties, probably using jsdoc / typedoc? If so it could describe:

properties, attributes
events fired

yeah for that we don't have a clear recommendation yet 🙈
typedoc seems good for all the js for sure; for attributes it will probably be something in line of https://github.com/w3c/webcomponents/issues/776
for properties, events there is nothing really out there right now... maybe someone can join the discussion and/or implement something so we can piggyback on the attributes solution?

Jsdoc actually has @fires for events. I really like what the vscode team has done!

There's also JSON Schema which has Mozilla support

On this, something i really liked about polymer based components was the documentation, you have all of your events/properties documented along side the demos.
Is there any way with story book to inject your jsdoc comments to some form of api docs page? (i've never used storybook so might be a dumb question)

For someone who had never used storybook i found it all very daunting in terms of demoing. One of things I loved about Polymer was how simple it was to document and demo your element. It would just parse your jsdoc comments and you provide some basic demos in a demo folder. I generated a component using the vanilla yeoman template on your site and I was a little overwhelmed and didn't know where to start. Just some feedback for someone familar with web component, and really wants a nice documentation/demo experience.

yes that is currently "lacking" we are moving to npm init open-wc so the generators are getting a little out of sync (we have a PR open to fix that... still needs some finishing touches)
with storybook 5 we should be able to show documentation and even provide a nice playground with your full properties API... I think that can be a nice place to bring all the info together next to the live demos :)
as this is pretty new we don't yet have a separate demo up and running somewhere but you can look at https://open-wc.org/demoing/ where you will see a gif showing it 👍  let us know what you think 🤗

@georgegriff I second on the idea to follow Polymer's method of documenting. I liked that a lot

So I think really two things are being discussed here:

Demoing(/docs) with storybook
A jsdoc kind of documentation

I think, like george said, not everyone will want to set up a storybook for their project, so should we have recommendations for that?

There is a new analyzer on the block 🤗
looks really promising 💪
https://github.com/runem/web-component-analyzer

This looks great!

@thepassle would it be a lot to make the web-component analyzer tool above part of the standard?

Iirc this is from the same author as the vscode lit-plugin, which personally im a big fan of.
I think we could look into this, and I'd have to try it out more, but im also wondering where we should put this. We do have a Storybook section and a Types section.
Also curious to hear what the other guys think about this.

I think it would benefit the entire ecosystem if we could agree on, and recommend, a common way of documenting web components. This will enable much better IDE-support, linting and docgen. I think a really good start would be to agree on some common jsdoc tags :-)
I like the comment by Justin here because I think it sums up the situation very well. In my comment I will specifically be focusing on jsdoc tags.
I proposed some jsdoc tags in web-component-analyzer. I took the freedom to choose some myself, but I'm very open for changing them if you have any concerns/requests. Below I will describe each of these jsdoc tags in details and why I chose them.
Compared to Justins comment my proposal covers:

Tag name: @element
Slots/Permitted content: @slot
Events fired: @fires
Properties: @property
Attributes: @attribute
CSS variables: @cssprop

It doesn't cover the following, but I think we can always add more later on.

DOM interface: methods, superclass, implemented interfaces
CSS Shadow Parts/Themes exposed
Permitted parents
Default / Permitted ARIA roles

Proposed jsdoc tags
🏷 Tag name
This jsdoc tag documents that the declaration is tied to a custom element with a certain tag name. This jsdoc tag is especially useful if your customElements.define is called dynamically eg. using a custom function.
Syntax
@element [<tagName>]

Synonym: @customelement

Note: I can't find any standard camel cased jsdoc tags. Therefore I suggest that @element would be the recommended jsdoc tag because I think it's easier to read than @customelement.
Example
/**
 * @element my-element
 */
class MyElement extends HTMLElement { ... }

Thoughts
Polymer recommends adding @customElement to custom elements in their section about documenting custom elements. I couldn't find documentation about specifying a tag name alongside the jsdoc tag (maybe I'm not looking the right place?), but I found the approach in this file in a test for the polymer analyzer.
In addition it seems like sometimes Polymer uses @element instead, but I couldn't find any documentation for this jsdoc tag.
📬 Slots
This jsdoc tag documents which slots names are available within your custom element.
Syntax
@slot [{<permittedTagNames>}] [<slotName>]

Example
/**
 * @slot - This is a comment. Using the "slot" jsdoc without a name indicates an unnamed slot.
 * @slot left - Content placed to the left
 * @slot right
 */
class MyElement extends HTMLElement { ... }

I suggest it should be possible to further indicate which tag names are permitted inside a slot by specifying string literals in the jsdoc type like this:
/**
 * @slot {"my-item"} - You must only use "my-item" tag names for unnamed slots.
 * @slot {"div"|"span"} footer - You must only use "div" and "span" for slots named "footer".
 */
class MyList extends HTMLElement { ... }

Thoughts
This jsdoc tag seems to be used by the Ionic team.
⚡ Events
This jsdoc tag documents which custom events your custom element dispatches.
Syntax
Jsdoc already has a tag for specifying which events a can be fired: @fires. I think this can be used for documenting events.
@fires [{<detailType>}] <eventName>

Synonym: @emits

Note I actually suggested using @event instead of @fires in web-component-analyzer, but I actually think @fires would be the best approach because it's standard jsdoc (see my thoughts below).
Example
/**
 * @fires change - Dispatched when the content of this element changes.
 * @fires my-event
 */
class MyElement extends HTMLElement { ... }

I suggest it should be possible to further indicate the type of the detail property on the CustomEvent like this:
/**
 * @fires {string} change - A change event where detail is the new text of this element.
 */
class MyElement extends HTMLElement { ... }

Thoughts
I'm in doubt if/how it should be possible to also document that an event bubbles, is cancelable or is composed, but I think it's out of scope for now.
Polymer suggest using the standard jsdoc tag @event like this:
/**
 * Fired when `element` changes its awesomeness level.
 *
 * @event awesome-change
 * @param {number} newAwesome New level of awesomeness.
 */

It's not documented anywhere but I assume that this jsdoc block must be somewhere within the class declaration. Here's an example with 'iron-input'.
The challenge I've had with this approach is, that if compiling with Typescript this entire jsdoc block with @event disappears from the declaration file because it's not attached to any emitted declaration. I like using the standard @fires better because I think it's lightweight and more in line with the other jsdoc tags proposed.
In addition @fires documents that the class fires a certain event and @event can be used to document an event itself. Therefore I think @fires better communicates the purpose which is to tell that a particular custom element can dispatch a certain event.
It's also important to note that these two tags do not exclude each other. In fact I still think it should be allowed to use them together (like the following example) because it's part of standard jsdoc:
/**
 * @fires awesome-change
 */
class MyElement extends HTMLElement { ... }

/**
 * Fired when `element` changes its awesomeness level.
 *
 * @event awesome-change
 * @param {number} newAwesome New level of awesomeness.
 */

Therefore I would suggest using @fires only with an @event declaration if you want to document more complex event types. And I wouldn't suggest using an @event jsdoc block without a corresponding @fires.
👪 Properties
This jsdoc tag documents which properties are available on your custom element.
Syntax
Jsdoc already has a tag for specifying properties: @property. I think this can be used for documenting properties on custom elements.
@property [{<jsdocType>}] <propertyName>

Synonym: @prop

Example
/**
 * @property {"red"|"green"} color - The color of this element
 * @property person - A person object that this element must show
 */
class MyElement extends HTMLElement { ... }

It also has syntax for optional and default values.
/**
 * @property [person] - Optional property
 * @property {"red"|"green"} [color="red"] - Optional property with default value
 */
class MyElement extends HTMLElement { ... }

👪 Attributes
This jsdoc tag documents which attributes are available on your custom element.
Syntax
I think we can have exactly the same syntax/rules as the standard @property jsdoc tag, but with the name @attribute instead.
@attribute [{<jsdocType>}] <attributeName>

Synonym: @attr

Regarding <jsdocType>: It's worth noting that an attribute in reality of course is always a string type. However I think it's important to be able to indicate if the attribute expects a string, string-union, number or if it's a boolean attribute. Therefore <jsdocType> would be the type that this attribute coerce to.
I also think it should be possible to use this tag in jsdoc on properties in order to indicate that the attribute has a relationship to the property (see example below). I think this is important because often properties and attributes are in sync with each other.
Example
/**
 * @attribute name
 * @attribute {"red"|"green"} color - The color of this element
 * @attribute {"small"|"large"} [size="large"]
 */
class MyElement extends HTMLElement {
  /**
   * This property can be set using either an attribute or the property itself.
   * @attr my-attribute
   */
  myProperty = "hello";

  /**
   * When omitting <attributeName> on the @attr jsdoc tag we document that there exists a corresponding attribute with the same name as the property.
   * @attr
   */
  text = "hello";
}

💅 CSS Custom Properties
This jsdoc tag documents which CSS custom properties are available on your custom element.
Syntax
@cssproperty <cssPropertyName>

Synonyms: @cssprop, @cssvariable, @cssvar

Writing <cssPropertyName> with or without the -- prefix is optional.
Example
/**
 * @cssprop --element-font-size - A CSS custom property that sets the font size.
 * @cssprop --element-border-radius
 * @cssprop [--element-color="red"]
 */
class MyElement extends HTMLElement { ... }

Thoughts
Maybe it would be relevant to also document the type of the custom property like @cssprop {number} --font-size, but here I'm in doubt how to indicate types like "color" or "px". Therefore I think it's out of scope for now.
Conclusion
Here I proposed some jsdoc tags that could be recommended for documenting web components. I'm very open for suggestions and concerns.
If we agree on some jsdoc tags, I would gladly write a more official guide/recommendation :-)

@runem I really like this! I'd say let's all go for web-component-analyzer and we can always iterate and improve where needed. But by far the most important thing of all is the fact that everyone follows the same standards.
How well does this interop with typescript?

Im a big pro for this. We could have a new page under Developing -> Documentation(/documenting) maybe? And maybe refer to it from the types page

What can we do to document class mixin functions a la polymer cli's @appliesMixin?

@bennypowers my understanding is that much of the mixin capacity of polymer will be deprecated in polymer 4 and they were previously deprecated in react. I'm not sure if we should recommend people even use mixins.

@goatandsheep thanks for the notes. Let's take this opportunity to dispel a misconception. The React mixins which were considered "harmful" aren't the same thing as js class mixins, which are the only way to share code or implement "multiple inheritance", unless you use a library like hybrids which abstracts over element classes and cusyomElements.define. Class mixins are still very much the recommended way to do that for vanilla and vanilla-like libs.
Side note, there will be no Polymer 4 library. The polymer team now recommends LitElement for new projects.
Please forgive typos - on Mobile

Properties & Attributes
imho we should stay as barebone as possible
therefore for attributes and properties, I would say stick to what works by default and that would be "auto" definitions in the constructor?
e.g. we should make it a simple as possible to get fully documented elements without needing to add arbitrarily double information. (e.g. if you write good custom elements in most cases you only document additional information not declare types/properties that are already clear)
class Foo extends LitElement {
  // these types are just a hint for the updatingElement
  static get properties() {
    return { color: { type: String } };
  }

  constructor() {
    super();
    // this is the "real" type automatically determined (by typescript and js) by setting it to a string
    this.color= '';

    // if you want additional information like that it is an attribute we could just add it there?
    /**
     * @attribute {"red"|"green"} color - The color of this element
     */
    this.color = '';

    // only if the default value needs to be undefined/null you are forced to set a specific type yourself
    /**
     * @type {Function}
     */
    this.formatter = null;
  }
}

so every property that does not start with an _ (for until will have protected properties) should be considered a public property. So usually there is no need to annotate it extra with a @property. Only if there is no default type.
another thought - jsDoc should be used to enhance the documentation by writing valuable information besides types - an example (Note: there is no @property as the type can be auto determined - the text written is purely to provide better documentation and intellisense)
/**
 * The title to display inside the title bar
 * - should be less then 100 characters
 * - should not contain HTML
 * - should be between 2-5 words
 *
 * @example
 * // DO:
 * el.title = 'Welcome to the jungle';
 *
 * // DON'T:
 * el.title = 'Info';
 * el.title = 'Welcome to the jungle';
 * el.title = 'We like to talk about more then just what sees the eye';
 */
this.title = 'You are awesome';

I did some exploration in this direction already https://dev.to/dakmor/type-safe-web-components-with-jsdoc-4icf#how-does-it-work
element
The class usually does NOT dictate the custom element name.
So it's probably better to define it where it actually happens.
/**
 * @element
 */
customElements.define('foo-bar', SomeThing);

can the tag name foo-bar and the Class SomeThing be auto read in this case or do we need to provide it in the jsDoc?
Slot & CSS Custom Properties
using @slot && @cssproperty on the class level looks good to me 🤗
Events
I am not sure why would you need to declare them separately on the class "again"?
if I have this code
someFuntion() {
  this.dispatchEvent(new CustomEvent('foo', { ... }));
}

it should be good enough to know that there is an event fired right?
e.g. rule could be to add

every this.dispatchEvent
every new CustomEvent('foo', { compose: true }) as it will go outside as well

If I want additional documentation something like this should be good enough?
someFuntion() {
  /**
   * This is my awesome event will allow you to do...
   */
  this.dispatchEvent(new CustomEvent('foo', { ... }));
}

what do you think?

I agree that we should stay as barebone as possible, and that as many features as possible, like properties and events, should be analyzed when possible, to prevent adding double information to our components 😊
I actually support it with web-component-analyzer. Here I analyze features like class fields, new CustomEvent("foo") and static get observedAttributes() { return ["my-attr"]; }, so it's unnecessary to specify them in JSDoc (I also work on analyzing property assignments in the constructor at the moment).
Web components come in many flavors (eg. LitElement and Stencil), so I built WCA with an architecture that makes it easy to add support for library specific syntax like LitElements static get properties() { return { color: { type: String } }; }. In WCA "Vanilla custom elements", JSDoc and LitElement are just flavors, and different flavours can co-existing in the same web component. For example I could choose to make a LitElement, implement static get observedAttributes and add a @slot JSDoc tag.
While I agree with you, I still think it's relevant that it's possible to specify all information using JSDoc, because it's needed under some circumstances. In the rest of my comment I will try to outline why/when I would recommend using JSDoc tags sometimes :-)
Custom helper functions
Where I think JSDoc really shines is when it's not possible to statically analyze the code. Sometimes we need to analyze a Web Component built with custom helper functions.
For example, I could have built my own function that generates a tag name and defines the element:
class MyElement extends HTMLElement { ... }
defineElement(MyElement);

Now it would make sense to use the @element JSDoc because customElements.define is not present in the code:
/*
 * @element my-element
 **/
class MyElement extends HTMLElement { ... }
defineElement(MyElement);

Properties & Attributes
I like your example, and I would also recommend doing it that way (and that's a very interesting blog post!). To add to your example, the only situation where I would use the @attribute JSDoc tag on the class declaration is if I'm going to use the attribute ONLY from CSS (so it's not present in the Javascript at all):
/**
 * @attribute {boolean} disabled - Disables this element
 */
class Foo extends LitElement { ... }

The same principle applies to the @property JSDoc, or if a custom helper function is used to setup the component.
Element
In your example the @element is actually not needed, because an analyzer would simply analyze the customElement.define function call. However as mentioned before I would use @element if it's not statically analyzable like this:
customElement.define(generateTagName(), MyElement);

I would like @element to be used on the class declaration because the aim of this JSDoc tag is to tie together a tagName and a declaration (just like a customElements.define statement would do). Maybe the <tagName> should be required and not optional alongside this JSDoc (so it's not possible to just use @element alone):
/*
 * @element my-element
 **/
class MyElement extends HTMLElement {
}

customElement.define(generateTagName(), MyElement);

Events
Again, I agree, and I think the @fires JSDoc tag should only be needed when the event dispatching is not statically analyzable (if you use a custom helper function to dispatch your events). In fact, right now WCA is be able to properly analyze your example :-)
Consuming Web Component Libraries
Here's a point to make about Web Components and editor support.
I built vscode-lit-plugin as a Typescript language service. Now I will give you an overview of what challenge I encountered when consuming web components from a library.
In a TS language service you can analyze all project-specific code (so everything is good here), but you only can only analyze declaration files (d.ts) from library code. Therefore an element (where it's easy to analyze the corresponding javascript), would just look like this to me:
export class MyElement extends HTMLElement {
  myProp: string;
  static readonly observedAttributes: string[];
  value: string;
}

The analyzer wouldn't be able to tell anything about this custom element other than its properties. I see different solutions to this challenge:

Find, parse and analyze the corresponding js-file (it would require a lot of extra work for the machine, so I don't like this approach)
Ship JSDoc with the custom element like @attribute my-attr and @element my-element (this would be supported out of the box if we agree on a JSDoc format).
Extend the global TS interface HTMLElementTagNameMap with the element (This would only partially help, because now we only know tag-name + properties).
Ship JSON metadata alongside the Web Component.

While I think both 2, 3 & 4 should be possible (2 & 3 is already possible with vscode-lit-plugin), I think the best solution in this case is to define a common JSON format for Web Components that can be shipped with the library. It would be really interesting to also look into recommending a standard JSON format (after agreeing on JSDoc tags). These JSON files can be built by analyzing the code and JSDoc. This discussion has already started in this issue. Right now WCA can output markdown and vscode custom html data (this format only specifies tags and attributes), but I would really like to implement a more general Web Component JSON output format in the future.
Conclusion
Sorry, now I wrote a wall of text again 🙈
So to conclude, I think your points are great and I agree with you: it should be as simply as possible to get fully documented elements without needing to add the information twice. It should be possible (not required) to specify all the same information using JSDoc because it's needed sometimes (see my examples), but the code itself should be analyzed when possible. So we should recommend only reaching out for JSDoc tags when it's not possible to statically analyze the code you write.
In addition I think next step after discussing a JSDoc format would be to discuss a JSON format that can be shipped with a Web Component library in order to consume web components. I think JSDoc is great when needing to analyze your own code, but I think a metadata JSON file would be more suitable when shipping a library :blush:

@runem so basically it comes down to this: JSDoc is in most cases option but can be used to enhance or provide otherwise missing information - sounds good to me 🤗
Consuming Web Component Libraries
I am wondering how heavy 1 would be e.g. if you provide types via jsDoc and "include" the code to your type checking like we do for open-wc - https://open-wc.org/developing/types.html#setup-for-javascript
We do it like that as to the best of my knowledge there is no nice and simple JSDoc to type definitions generator available (although webpack has their own custom solution... )
what ultimately would be nice is to be able to write "just" js with JSDoc and automatically generate typescript definition files (*.d.ts) and web component definition files (how about my-tag.d.json ?).
Side note
as I was just watching https://youtu.be/ViyTYEv9dM8?t=741 ... apparently, Custom Events are sort of discouraged...

Update Regarding Class Mixins
I've had some success with the following JSDoc tags:
The Mixin Function:
/**
 * Provides `isSlotted` method to check if a slot has content.
 *
 * @param {*} superclass the class to mix in to
 * @return The mixed class
 * @mixin IsSlottedMixin
 */
export const IsSlottedMixin = superclass =>
  /** @extends { HTMLElement } */
  class extends superclass {
    /**
     * Checks whether a slot has assigned nodes.
     * @param {String=} slotName Slot name to check.
     * @return {Boolean}
     * @memberof IsSlottedMixin#
     */
    isSlotted(slotName) { /* ... */ }
  };

The mixing class:
/**
 * <can-has-slot>
 * @customElement
 * @mixes IsSlottedMixin
 */
class CanHasSlots extends IsSlottedMixin(HTMLElement) { /* ... */ }

This is working well in simple cases, I have autocomplete and tooltips in Atom (typescript lang server) but is failing if I also apply @extends LitElement to the mixing class.

Without @extends, I don't have tooltips for LitElement methods like requestUpdate, but I do have them for isSlotted,
but with @extends I have tooltips for requestUpdate but not isSlotted

stale has no chill

Hey Stale, we all want stuffs that's in this thread :p

We are discussing this with different people within the web component community, we will standardize on a common format.

@LarsDenBakker What are the latest news on this?

Would like to get @runem 's opinion on this, specifically, how would you see static analysis tools that generate custom-elements.json consuming jsdoc for properties ,attributes, methods , etc.

heel, Stale! Down, boy!

Should we break this up into smaller issues, maybe?

I'm going to close this issue, we are still pushing hard for standardizing custom-elements.json and discussing with different parties. https://github.com/webcomponents/custom-elements-json is likely where most of the documentation will go, watch that repository for updates!
In the meantime, https://www.npmjs.com/package/web-component-analyzer is your best bet and an excellent choice along with https://github.com/web-padawan/api-viewer-element for generating API docs.

I don't understand. How would the web component analyzer create a standard way of documenting components using jsdoc?

We've been pushing to standardize a custom-elements.json format, which you can find at the link that Lars mentioned. We think the custom-elements.json will open a lot of useful usecases.
Getting that format standardized is the first priority, and it has proven to be a fairly time consuming process so far. Once thats formalized, we can use tools like web-component-analyzer to automatically scan our code, based on jsdoc, and generate a custom-elements.json. Its important that we can get all of the web component community behind the format as well.

Awesome! I must have missed that update previously. Looking forward to it! ❤


[Android][Firebase] Implement Firebase Authentication

We will authenticate all users by the simplest means possible(Google + GitHub integration)
All users can host their own events,
All users can attend an event via an event UUID / Barcode
Users need to take selfies / be present in the photos to get attendance

adding firebase authentication. Please send me your email to be added as collaborators to the firebase project


Design .urbit claimer relay server architecture

See #11 and #12. Users of the NEAR app on Urbit will need to be able to claim a .near account that matches their Urbit ID. Because most Urbit planets are on the L2 roll-up, this can't be a permissionless on-chain operation, and will instead need to go through a server run by either the Urbit Foundation or NEAR Foundation. We'll likely have the NEAR Foundation run this server, so to lighten the maintenance load on them, we're considering not running this server on Urbit.
A development benefit here is that it saves us from having to do RPC calls from within a Gall agent. A marketing benefit is that we get to illustrate the use of Auth Server.
However, Auth Server still requires a running Urbit ship:

A user tries to login to the site example.com by entering their ship ~sampel-palnet.
example.com sends an authorization request to their ship ~master running Auth Server.
Auth Server on ~master sends the request to the Auth Client app on ~sampel-palnet.
Auth on ~sampel-palnet gets the request and makes an HTTP request for http://example.com/.well-known/appspecific/org.urbit.auth.json and retrieves an attestation that ~master is an agent of example.com.
Auth Client verifies the signature in the attestation using the pubkey of ~master it got from Azimuth, then displays an authorization request for the user.
The user of ~sampel-palnet clicks "Approve" in Auth Client.
Auth Client on ~sampel-palnet sends an update to Auth Server on ~master saying the request was approved.
Auth Server notifies example.com that the request was authorized.
example.com logs the user in.

Authentication of an Urbit ID likely requires Urbit somewhere in the loop to make use of Ames. At that point, does it then make most sense to have only the one Gall agent acting as a relay server on NEAR's ship, as opposed to an Auth Server Gall agent working in conjunction with a standard web server? Is there something we can do here with lib-ames?

@SuperCoolYun My uncertainty here means that #12 isn't very well specified yet. Would be good to keep these design questions in mind as you investigate how NEAR smart contract calls are done, and see if there are any general principles we can keep in mind for them.
For instance: what if we end up running a Python web server instead of a Node.js one? Are there any good Python libraries for interacting with the NEAR blockchain?
Lots of research to do here about the best way of architecting this.

Two more ideas here:

We could essentially build a libames into the smart contract (#16) that verifies the user's Urbit ID in the same way that Ames does. (I don't exactly know how this works, but I think it makes sense; even L2 ships must have their own private/public keypair for networking, right?)
We could write this as a Gall agent and deploy this Gall agent on multiple ships, so that there isn't a single NF or UF point of failure. (Potentially using Vaporware's soon-to-be open-sourced %make library for chain operations.)

An important note here is that using a web server here gives us much better security than just keeping the private keys for the issuing account in an Urbit ship.

So far, we've generated four options:

Permission the .near account issuer smart contract to only respond to one set of keys. Store these keys in a Gall agent. Run this Gall agent on the NF galaxy, and have it act as a relay server that users request their .near account through.
Pros: Gives us Urbit ID authentication for free via Ames, simple architecture
Cons: Requires us to store valuable keys on an unsecure Urbit ship, requires us to make smart contract calls within Hoon code (which is difficult, though maybe easier once Vaporware's code is open-sourced)

Permission the .near account issuer smart contract to only respond to one set of keys. Store these keys on a web server. Run a Gall agent on the NF galaxy that uses this web server as a sidecar, and have this pair act as a relay server that users request their .near account through.
Pros: Gives us Urbit ID authentication for free via Ames, lets us use pre-existing libraries for smart contract calls
Cons: More complex architecture that requires the NF to maintain both an Urbit ship and a web server (as well as the connection between them)

Permission the .near account issuer smart contract to only respond to one set of keys. Store these keys on a web server. Have the NF run this web server and have it act as a relay that users request their .near account through.
Pros: Very simple architecture
Cons: Requires us to authenticate Urbit ships on the web server itself (maybe lib-ames solves this?)

Have the smart contract itself authenticate Urbit ID.
Pros: Simplest architecture, totally decentralized
Cons: Most difficult option to write

Option 1 is probably off the table for security reasons. Option 2 makes me nervous due to the number of moving parts.
If Option 4 is possible, then it's the ideal due to its simplicity, but it's likely hard to pull off and there are lots of unknowns. Option 3 is essentially an easier version of Option 4.

As a note on Option 2: this is potentially difficult if we used %lick or bespoke http requests, but auth-server would likely reduce the number of moving parts. We'd still have to maintain both an Urbit node and a web server, but that isn't too bad as long as we can rely on auth-server.


Show more Metadata for uploaded Docker Images

Show created, size, hash, architecture.

This is now implemented. All images that are not published for the amd64 architecture are not possible to select.


num_threads

Hi. Thanks for developing useful program.
I have some issue about lib.num_threads on gpu4pyscf.
No matter what calculation I do (DFT, HF, CCSD...) no matter what value of num_threads I enter, it only actually uses one.
Is there anything you can tell me about this issue?

@sehun1618 It sounds expected. For better performance, the CPU workload in GPU4PySCF is significantly minimized. Most OpenMP tasks (usually heavy) in PySCF have been offloaded to GPU. Although OpenMP is used in some tasks, the execution time is probably unnoticeable.

Thanks.
I have another question.
If i got more than 2 GPU units, how can i allocate specific GPU unit for computation?

@sehun1618 In a multi-GPU environment, the safest way is to run it with docker.
https://docs.docker.com/config/containers/resource_constraints/#gpu
Alternatively, you can try to specify the device with CuPy api. We haven't tested it globally. If you find any issues, please let us know.
https://docs.cupy.dev/en/stable/user_guide/basic.html#current-device


rules print should have the option to disable color

Many tools do not handle shell colors very well. In my case, I am trying to parse the YAML output using https://github.com/kislyuk/yq, but it fails due to the colorized input. As a workaround, I have to pipe the output through sed -e $'s/\x1b\[[0-9;]*m//g', which frankly I barely understand.

#25 allows the option to disable color on rule and alert commands


Website Down or something

I get this error whenever I try and access the website. Does anybody else get this error or is it just me?

the same

The message is all good and well, but completely breaking the documentation for a library for political reasons is just evil. A banner would've sufficed.

I'm getting a similar error.
Looking into the page source, it's actually a Cloudflare access denied error page:

Access denied

You do not have access to www.pygame.org.

The site owner may have set restrictions that prevent you from accessing the site.

I can visit the site just fine from other IP addresses, but not from my home IP - it appears that I've been banned somehow, which is weird. Who can I contact about getting this lifted?

I see this issue is closed as completed. This is what I currently get, trying to visit https://www.pygame.org/docs/ref/pygame.html:

   <meta charset="UTF-8">

The pygame website is going down for a little nap.

NO LOUD NOISES... please :)

    ::ALWAYS_ONLINE_NO_COPY_BOX::

    ::CLOUDFLARE_ERROR_500S_BOX::

    ::CLOUDFLARE_ERROR_1000S_BOX::

I’ve been getting exactly the same “No loud noises” error as @DavidBuchanan314 did last year for the last 24 hours or so whenever I try and access the front page of the website (pygame.org/news). This time though it seems like it’s happening across all my devices, on my home wifi and on cellular, both with and without my VPN active, which is a bit odd! The other odd thing is that I can still access the documentation pages and dashboard on all devices/networks just fine - it’s just the rest of the website that’s down!

Getting the same error as @DavidBuchanan314 What's the point of braking the docs page of an open source project? I'd assume @illume can answer this question.

By the way, I don't have the issue anymore, on my end at least


Added support for runtime expressions embedded in string literals

Addresses issue #287

Can we add one test using composite expression in https://github.com/Microsoft/OpenAPI.NET/blob/master/test/Microsoft.OpenApi.Readers.Tests/V3Tests/OpenApiCallbackTests.cs ?

@PerthCharern I will make these changes.  How do feel about releasing 1.1, rather than me back porting these changes to 1.0?

Sounds fine to me.
Perth Charernwattanagul
Sent from my mobile device
On Jul 9, 2018, at 12:13, Darrel <notifications<EMAIL_ADDRESS>wrote:
@PerthCharernhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FPerthCharern&data=02|01|Perth.Charernwattanagul%40microsoft.com|c0812b22644e44f21f1d08d5e5d00599|72f988bf86f141af91ab2d7cd011db47|1|0|636667603971401302&sdata=gQ3s45EjxE2kr5qShLE6%2BVu3L%2Fb1OXVF6p8t4PPft7w%3D&reserved=0 I will make these changes. How do feel about releasing 1.1, rather than me back porting these changes to 1.0?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoft%2FOpenAPI.NET%2Fpull%2F290%23issuecomment-403589300&data=02|01|Perth.Charernwattanagul%40microsoft.com|c0812b22644e44f21f1d08d5e5d00599|72f988bf86f141af91ab2d7cd011db47|1|0|636667603971401302&sdata=YXjcjtQVVoJKLn92%2BqhAkGLOiwWIE7DZmQ08EStO%2F68%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAZMdvsJgeOdANz-eWBEw-32R1N7JcifOks5uE6tKgaJpZM4VGx-3&data=02|01|Perth.Charernwattanagul%40microsoft.com|c0812b22644e44f21f1d08d5e5d00599|72f988bf86f141af91ab2d7cd011db47|1|0|636667603971411307&sdata=%2FjxOjR8KrccehxLbmEQDRZ9L1yzCKq8IywmJFcDY3V0%3D&reserved=0.


Batch up posting of metrics to Librato

When posting to Librato publish the metrics every X (10) seconds and in batches of Y (300), wont post anything if no metrics.

This PR was successfully deployed to production by CircleCI build 139 at 2017-01-12 11:33 GMT.


Network error: request error at connector

Hi,
I have multiple schema that are based in a table and everything works fine.
But when a simple one page schema is created (like DEMO: PERSONAL PROFILE), when trying to save, an error is thrown in deployed env (not in local): Network error: request error at connector
The code of this schema is:
/* eslint-disable react/prop-types */
import CannerScript from 'canner-script';
import * as React from 'react';
const Configuration = () => (
    <object keyName="configuration" title="Settings" >
        <string keyName="copy" title="Copyright" />
    </object>
)
export default Configuration

Other sections works fine, without connectors.
What's wrong on the code?

Hi, AlexGach, thanks for the report, what is the version of the CLI (enter canner --version)? And is there any information from the console and network panel?

Current version: 2.7.0
Do you think uploading to 3.1.5 will solve it? I tried but there was an error on installation, I'll try again


[Bug][gitextractor] Deadlocks when collecting commits

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
I am running into deadlocks when trying to execute two parallel pipelines for distinct repositories.
This is the error I meet:

What do you expect to happen
There should be no deadlocks, or the system should be resilient and retry transactions a few times (perhaps a configurable amount?)
I have found the following issue on this same behavior: https://github.com/apache/incubator-devlake/issues/5819
Is it expected that we disable the commit file extraction to avoid this error? I just want to confirm we would not be missing any relevant metrics with this setting.
How to reproduce
Trigger simultaneous pipelines on repositories with over 20k commits. (we were able to trigger this with 2 pipelines on a db.t3.micro and 3 pipelines on a db.m5.xlarge)
Anything else
No response
Version
v0.18.0, v0.19.0, v0.20.0
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

UPDATE: I tried using SKIP_COMMIT_FILES and although it seems faster, I still observe deadlocks 🤔

this is the latest locks information on innodb:
------------------------
LATEST DETECTED DEADLOCK
------------------------
2024-03-06 09:28:06<PHONE_NUMBER>3120
*** (1) TRANSACTION:
TRANSACTION 9964411, ACTIVE 1 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1128, 2 row lock(s), undo log entries 5
MySQL thread id 354, OS thread handle<PHONE_NUMBER>0768, query id 995814 <IP_ADDRESS> devlake update
INSERT INTO `commit_parents` (`created_at`,`updated_at`,`_raw_data_params`,`_raw_data_table`,`_raw_data_id`,`_raw_data_remark`,`commit_sha`,`parent_commit_sha`) VALUES ('2024-03-06 09:28:05.308','2024-03-06 09:28:05.308','github:GithubRepo:1:189630055','gitextractor',0,'','7b2bc5000669d9b3fd5d231a581cc52271e36f4b','a4b384bf1c0f675739d075a146607518c1ce1457'),('2024-03-06 09:28:05.308','2024-03-06 09:28:05.308','github:GithubRepo:1:189630055','gitextractor',0,'','a4b384bf1c0f675739d075a146607518c1ce1457','8eadbeed5aee0e7b7c0ee0dba4dab06059950416'),('2024-03-06 09:28:05.308','2024-03-06 09:28:05.308','github:GithubRepo:1:189630055','gitextractor',0,'','8eadbeed5aee0e7b7c0ee0dba4dab06059950416','4ed72106c022a15b53adfa259510f0dcbc78ac99'),('2024-03-06 09:28:05.308','2024-03-06 09:28:05.308','github:GithubRepo:1:189630055','gitextractor',0,'','8eadbeed5aee0e7b7c0ee0dba4dab06059950416','6bf4ac71efaaf5407a9a9c8f7f27200

*** (1) HOLDS THE LOCK(S):
RECORD LOCKS space id 2340 page no 783 n bits 160 index PRIMARY of table `lake`.`commit_parents` trx id 9964411 lock_mode X locks rec but not gap
Record lock, heap no 83 PHYSICAL RECORD: n_fields 10; compact format; info bits 64
 0: len 30; hex<PHONE_NUMBER>63373165666161663534303761396139633866376632373230; asc 6bf4ac71efaaf5407a9a9c8f7f2720; (total 40 bytes);
 1: len 30; hex<PHONE_NUMBER>31303663303232613135623533616466613235393531306630; asc 4ed72106c022a15b53adfa259510f0; (total 40 bytes);
 2: len 6; hex 000000980b7b; asc      {;;
 3: len 7; hex 81000036b50278; asc    6  x;;
 4: len 7; hex 99b2cc97050c08; asc        ;;
 5: len 7; hex 99b2cc97050c08; asc        ;;
 6: len 29; hex<PHONE_NUMBER>623a4769746875625265706f3a313a313839363330303535; asc github:GithubRepo:1:189630055;;
 7: len 12; hex<PHONE_NUMBER>74726163746f72; asc gitextractor;;
 8: len 8; hex<PHONE_NUMBER>000000; asc         ;;
 9: len 0; hex ; asc ;;

*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 2340 page no 1944 n bits 168 index PRIMARY of table `lake`.`commit_parents` trx id 9964411 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 14 PHYSICAL RECORD: n_fields 10; compact format; info bits 96
 0: len 30; hex<PHONE_NUMBER>63313161326163326630383431383039323835633161363065; asc 4ed83c11a2ac2f0841809285c1a60e; (total 40 bytes);
 1: len 30; hex<PHONE_NUMBER>31376263313931353639313063346336323335373636656162; asc d4d2917bc19156910c4c6235766eab; (total 40 bytes);
 2: len 6; hex 000000980b69; asc      i;;
 3: len 7; hex 01000027fc2b94; asc    ' + ;;
 4: len 7; hex 99b2ca77d6000a; asc    w   ;;
 5: len 7; hex 99b2ca77d6000a; asc    w   ;;
 6: len 28; hex<PHONE_NUMBER>623a4769746875625265706f3a313a3936353934363639; asc github:GithubRepo:1:96594669;;
 7: len 12; hex<PHONE_NUMBER>74726163746f72; asc gitextractor;;
 8: len 8; hex<PHONE_NUMBER>000000; asc         ;;
 9: len 0; hex ; asc ;;

*** (2) TRANSACTION:
TRANSACTION 9964393, ACTIVE 1 sec fetching rows
mysql tables in use 1, locked 1
LOCK WAIT 333 lock struct(s), heap size 41080, 19875 row lock(s), undo log entries 15662
MySQL thread id 342, OS thread handle<PHONE_NUMBER>9376, query id 995725 <IP_ADDRESS> devlake updating
DELETE FROM `commit_parents` WHERE _raw_data_table = 'gitextractor' AND _raw_data_params = 'github:GithubRepo:1:96594669'

*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 2340 page no 1944 n bits 168 index PRIMARY of table `lake`.`commit_parents` trx id 9964393 lock_mode X locks gap before rec
Record lock, heap no 14 PHYSICAL RECORD: n_fields 10; compact format; info bits 96
 0: len 30; hex<PHONE_NUMBER>63313161326163326630383431383039323835633161363065; asc 4ed83c11a2ac2f0841809285c1a60e; (total 40 bytes);
 1: len 30; hex<PHONE_NUMBER>31376263313931353639313063346336323335373636656162; asc d4d2917bc19156910c4c6235766eab; (total 40 bytes);
 2: len 6; hex 000000980b69; asc      i;;
 3: len 7; hex 01000027fc2b94; asc    ' + ;;
 4: len 7; hex 99b2ca77d6000a; asc    w   ;;
 5: len 7; hex 99b2ca77d6000a; asc    w   ;;
 6: len 28; hex<PHONE_NUMBER>623a4769746875625265706f3a313a3936353934363639; asc github:GithubRepo:1:96594669;;
 7: len 12; hex<PHONE_NUMBER>74726163746f72; asc gitextractor;;
 8: len 8; hex<PHONE_NUMBER>000000; asc         ;;
 9: len 0; hex ; asc ;;

*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 2340 page no 783 n bits 160 index PRIMARY of table `lake`.`commit_parents` trx id 9964393 lock_mode X waiting
Record lock, heap no 83 PHYSICAL RECORD: n_fields 10; compact format; info bits 64
 0: len 30; hex<PHONE_NUMBER>63373165666161663534303761396139633866376632373230; asc 6bf4ac71efaaf5407a9a9c8f7f2720; (total 40 bytes);
 1: len 30; hex<PHONE_NUMBER>31303663303232613135623533616466613235393531306630; asc 4ed72106c022a15b53adfa259510f0; (total 40 bytes);
 2: len 6; hex 000000980b7b; asc      {;;
 3: len 7; hex 81000036b50278; asc    6  x;;
 4: len 7; hex 99b2cc97050c08; asc        ;;
 5: len 7; hex 99b2cc97050c08; asc        ;;
 6: len 29; hex<PHONE_NUMBER>623a4769746875625265706f3a313a313839363330303535; asc github:GithubRepo:1:189630055;;
 7: len 12; hex<PHONE_NUMBER>74726163746f72; asc gitextractor;;
 8: len 8; hex<PHONE_NUMBER>000000; asc         ;;
 9: len 0; hex ; asc ;;

*** WE ROLL BACK TRANSACTION (1)

Perhaps the dataaccess layer code should be hardened to attempt a few retries in the event of deadlocks. @klesh

@keon94  Maybe, or we could try other tricks to avoid massive locks because there is performance penalty.
Batch insertion/updation maybe?

hey guys, thanks for getting back to me!
I think it's not unusual to go for the retry attempts, but indeed the more we could avoid them, the better.. it sound like a combination of both would be ideal if that's not too much to ask:

maybe configurable deadlock retries (ex exponential retry, number of attepmts, delay)
maybe batch insertion, but maybe that also triggers deadlocks and maybe even more? because you would batch a series of records in distinct pipelines, and while a pipeline is committing its data, the other one with another big batch would probably need a lot of locks for its records (just brainstorming here)

i am trying to take some time to understand a bit better on the code where the deadlock lies.. if you have any pointers that would be very welcome!

I just checked the code and there is already batch insertion there, but I still don't understand how the deadlock happened, it doesn't make sense to me. It should be  OK for different threads to insert records into the same table.
I feel like we should figure out why it happened before we could fix it correctly.

@pimguilherme That is very kind of you, thanks for taking the tough part of all. Looking forward to hearing from you. 💪

my pleasure @klesh! spoiler -- I was apparently able to trigger it using 3 apache repositories.. I will try to capture the steps properly and share them here
one thing I wanted to clear.. we are using MySQL, but do you support PostGres?
thanks!

@pimguilherme Good to know.
You may use pg to store the data without any problem.
However, all Grafana dashboards are designed for MySQL, which may use some dialect syntax.

Cool, thanks for confirming @klesh!
I have put together a small script to trigger the deadlock on an empty local devlake instance running 0.18.0 (it would probably work on newer instances with probably some small adjustments to the script)
Here are the steps:

copy env.example to .env and set PIPELINE_MAX_PARALLEL=5 and ENCRYPTION_SECRET=secret
online devlake with docker-compose -f docker-compose-dev.yml up
once devlake is up, run the attached bash script: deadlock-setup.txt
bash deadlock-setup.txt (NOTE: you need jq, curl, gunzip and grep installed to run this script)
( it will just do a series of calls to the API to set up 3 apache repositories/projects, druid, arrow-datafusion and incubator-devlake, set a smaller sync window and trigger the pipelines continuously until deadlock is met.. I have included a test user github token for public repos)
wait until deadlock is hit (it took about 7 minutes in my tests, according to attached logs)
deadlock-example.txt
deadlock-example2.txt

please let me know how it goes
PS I also met this error somewhat consistently in the first pipelines, but i think we can focus on the deadlocks:

cheers!

Found a article might be helpful https://medium.com/assoconnect/dealing-with-deadlocks-on-insert-requests-59eb93ef8dce

oh, that's pretty interesting @klesh, I will try that out as well, beucase it would be easier than having to migrate infra-structure to Postgres
thanks for the pointer!@

@pimguilherme Good hunting 🦆

thanks @klesh! we have been running smooth on the deadlock side after changing the transaction isolation :)
I think it would be the most ideal scenario for the code to be able to handle things even without this setting, but just behaving much slower, or at least with a configurable amount of deadlock attempts. But that may be overengineering for now considering it's a simple and sensible change to set transaction_isolation to READ-COMMITTED (which is the default for Postgres apparently, so that's why it runs smooth there too probably)
it would probably be a good idea to mention this in the docs, and maybe set this as a default configuration for devlake on MySQL.. what do you think?
thanks!!

@pimguilherme I think we could do both 😄

sounds good @klesh, thanks! :smile:
we have been able to run 3 parallel pipelines very stable in this fashion.. we did a few load tests and I think the summary for being able to cope with parallelism in devlake is that it is ideal to have 1 CPU + 4 GB memory on a MySQL database per parallel pipeline
so, for 3 pipelines in parallel, at least 3 CPUs and 12 GB memory.. at least in our setup, devlake is quite CPU intensive on the database during some pipeline steps (specially the dora calculation on bigger repos I think, but without further investigation yet)
as for the devlake process itself, we have been running it with 3Gi memory and it doesnt seem to be very CPU intensive
we basically use the Github connector (without issue tracking) for all our repos, and DORA and Azure Devops for some
we haven't hit any rate limits on github yet :raised_hands:
thanks for the support!

@pimguilherme Thanks for sharing, this is very informative.
I wonder if we could set the isolation level on the devlake end or should it be done by setting up the mysql server?
Either way, would you like to contribute the code or docs for the matter? Thanks in advanced.

okay @klesh, thanks for the suggestion! I will try to allocate some time to do this :)


Fix Azure Stack auth

(message created by the CI based on PR content)

Codecov Report

Merging #4903 into master will increase coverage by <.01%.
The diff coverage is 77.77%.

@@            Coverage Diff             @@
##           master    #4903      +/-   ##
==========================================
+ Coverage   48.74%   48.74%   +<.01%
==========================================
  Files        6603     6603
  Lines      184902   184909       +7
==========================================
+ Hits        90124    90129       +5
- Misses      94778    94780       +2

Impacted Files
Coverage Δ

azure-common/azure/common/_version.py
100% <ø> (ø)
:arrow_up:

azure-common/azure/common/client_factory.py
92% <77.77%> (-2.12%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a06b0ec...5ccb61e. Read the comment docs.


Always uses the quantum limit, even when the minimum quantum is sufficient. Introduces uneccesary latency.

By default, if I run firefox, its quantum in pipewire will be the absolute upper limit set in pipewire's config. I can use the environment variable PULSE_LATENCY_MSEC when starting firefox, to set the quantum to the minimum setting in pipewire, and there are no xruns or dropouts whatsoever. Normally, the minimum should be used by default, and increased only to the maximum, to avoid xruns if necessary. In the case of firefox pulse, this is not what happens.
The result of this is an order of magnitude higher latency to and from firefox.

I made an interesting discovery regarding this:
I previously set the environment variable PULSE_LATENCY_MSEC=3, in order to avoid this bug and force the buffer/quantum in pipewire which I desired. This got me the 128 quantum I needed.
In an unrelated(? maybe spotify uses cubeb as its backend?) issue, I suggested using this environment variable to avoid this same behaviour in Spotify. Since I was suggesting latencies which were less demanding and more likely to work for the other users I was assisting, I suggested PULSE_LATENCY_MSEC=20 and PULSE_LATENCY_MSEC=10
I tested firefox with PULSE_LATENCY_MSEC=10, just to make sure I was giving correct syntax to the others in that issue, and saw that firefox was following the settings I had made in pipewire's configuration, and did NOT use a 10ms latency at all, rather, it used the 128 sample buffer which was specified in my pipewire configuration.
Just to be sure, I changed the pipewire config to instruct pulseaudio clients to request a 256 sample buffer, set the environment variable to PULSE_LATENCY_MSEC=10 (globally, it's set that way in my environment.d) and sure enough, firefox started and used the correct quantum/buffer size as per my pipewire config.
Just out of pure curiosity, I wondered if any other time for PULSE_LATENCY_MSEC would have the same effect, so I tried with
PULSE_LATENCY_MSEC=8 firefox
And sure enough, it used the 256 sample buffer as instructed by pipewire.
OK So What if I go higher? I tried 50, and it didn't obey pipewire's settings. 40, doesn't obey. 30, doesn't obey. 20, DOES obey. 25, doesn't, 23, doesn't, 21 DOES, 22 DOES.
So the TL;DR here is that if you set PULSE_LATENCY_MSEC=22 or anything less than 22, then the pipewire settings are adhered to,
If you set anything from PULSE_LATENCY_MSEC=23 upwards, or not having it set at all, causes this bug.
I have to make an "illuminti confirmed" joke here
But it is completely bizarre that setting PULSE_LATENCY_MSEC to anything less than 23ms, works as it should and follows pipewire's configuration. Anything 23 or above, or not setting that environment variable at all, ignores pipewire's configuration and pushes the buffer all the way to the quantum limit.