train_on_batch fails with MirroredStrategy

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): RHEL 7.5
TensorFlow installed from (source or binary): source
TensorFlow version (use command below): 2.1.0
Python version: 3.7.4
Bazel version (if compiling from source): 0.29.1
GCC/Compiler version (if compiling from source): 8.3.0
CUDA/cuDNN version: 10.1
GPU model and memory: GTX1080Ti

Describe the current behavior
When running train_on_batch on a model created under a distribution strategy such as MirroredStrategy the error ValueError: handleis not available outside the replica context or atf.distribute.Strategy.update() call. is thrown
Putting the train_ob_batch into the strategy scope does not change that error. However when doing so with a model containing a batch normalization layer (e.g. Resnet50) it throws a different error: RuntimeError: add_update was called in a cross-replica context. This is not expected. If you require this feature, please file an issue. (the other error location is not reached in that case as it bails out with the above)
Describe the expected behavior
train_on_batch should work
Standalone code to reproduce the issue
import tensorflow as tf

strategy = tf.distribute.MirroredStrategy()

with strategy.scope():
    (x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
    x_train, x_test = x_train / 255.0, x_test / 255.0
    train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train)).batch(32)

    model = tf.keras.models.Sequential([
        tf.keras.layers.Flatten(input_shape=(28, 28)),
        tf.keras.layers.Dense(128, activation='relu'),
        tf.keras.layers.Dense(10),
    ])

    model.compile(
        optimizer=tf.keras.optimizers.SGD(),
        loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
        metrics=['accuracy'])

for x, y in train_dataset:
    model.train_on_batch(x, y)

Other info / logs Include any logs or source code that would be helpful to
Full traceback:
Traceback (most recent call last):
  File "tf_issue_train_on_batch.py", line 24, in <module>
    model.train_on_batch(x, y)
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 1078, in train_on_batch
    standalone=True)
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 433, in train_on_batch
    output_loss_metrics=model._output_loss_metrics)
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 568, in __call__
    result = self._call(*args, **kwds)
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 615, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 497, in _initialize
    *args, **kwds))
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2389, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2703, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2593, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 978, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 439, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 968, in wrapper
    raise e.ag_error_metadata.to_exception(e)
ValueError: in converted code:

    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_eager.py:305 train_on_batch  *
        outs, total_loss, output_losses, masks = (
    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_eager.py:273 _process_single_batch
        model.optimizer.apply_gradients(zip(grads, trainable_weights))
    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py:444 apply_gradients
        kwargs={"name": name})
    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/distribute/distribute_lib.py:1949 merge_call
        return self._merge_call(merge_fn, args, kwargs)
    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/distribute/distribute_lib.py:1956 _merge_call
        return merge_fn(self._strategy, *args, **kwargs)
    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py:485 _distributed_apply
        scope_name), distribution.extended.colocate_vars_with(var):
    /sw/installed/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/contextlib.py:112 __enter__
        return next(self.gen)
    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:4112 _colocate_with_for_gradient
        with self.colocate_with(op, ignore_existing):
    /sw/installed/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/contextlib.py:112 __enter__
        return next(self.gen)
    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:4161 colocate_with
        op = _op_to_colocate_with(op, self)
    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:6548 _op_to_colocate_with
        if hasattr(v, "handle") and isinstance(v.handlTraceback (most recent call last):
  File "tf_issue_train_on_batch.py", line 24, in <module>
    model.train_on_batch(x, y)
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 1078, in train_on_batch
    standalone=True)
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 433, in train_on_batch
    output_loss_metrics=model._output_loss_metrics)
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 568, in __call__
    result = self._call(*args, **kwds)
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 615, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 497, in _initialize
    *args, **kwds))
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2389, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2703, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2593, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 978, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 439, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 968, in wrapper
    raise e.ag_error_metadata.to_exception(e)
ValueError: in converted code:

    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_eager.py:305 train_on_batch  *
        outs, total_loss, output_losses, masks = (
    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_eager.py:273 _process_single_batch
        model.optimizer.apply_gradients(zip(grads, trainable_weights))
    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py:444 apply_gradients
        kwargs={"name": name})
    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/distribute/distribute_lib.py:1949 merge_call
        return self._merge_call(merge_fn, args, kwargs)
    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/distribute/distribute_lib.py:1956 _merge_call
        return merge_fn(self._strategy, *args, **kwargs)
    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py:485 _distributed_apply
        scope_name), distribution.extended.colocate_vars_with(var):
    /sw/installed/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/contextlib.py:112 __enter__
        return next(self.gen)
    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:4112 _colocate_with_for_gradient
        with self.colocate_with(op, ignore_existing):
    /sw/installed/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/contextlib.py:112 __enter__
        return next(self.gen)
    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:4161 colocate_with
        op = _op_to_colocate_with(op, self)
    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:6548 _op_to_colocate_with
        if hasattr(v, "handle") and isinstance(v.handle, Tensor):
    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/distribute/values.py:720 handle
        raise ValueError("`handle` is not available outside the replica context"

    ValueError: `handle` is not available outside the replica context or a `tf.distribute.Strategy.update()` call.

```e, Tensor):
    /sw/installed/TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow_core/python/distribute/values.py:720 handle
        raise ValueError("`handle` is not available outside the replica context"

    ValueError: `handle` is not available outside the replica context or a `tf.distribute.Strategy.update()` call.

@Flamefire,
I was able to reproduce the error with TF v2.1. However, the issue seems to be fixed with TF v2.2.0-rc4. Please find the attached gist. Thanks!

@amahendrakar I can confirm this. However I noticed a change in TF 2.2 to 2.1: Using train_on_batch in a tf.function is no longer possible as it errors out with AttributeError: 'Tensor' object has no attribute 'numpy'. Is train_on_batch supposed to be called outside a tf.function? My understanding was to put as much as possible into tf.functions.
This used to work in TF 2.1
See gist: https://colab.research.google.com/drive/1e3UwhKhQUFyYh5GsIO3TWOmvMynAPJUg?usp=sharing

Was able to run the code with TF v2.1.
However, running the code with TF v2.2 throws an error stating     AttributeError: 'Tensor' object has no attribute 'numpy'. Please find the attached gist. Thanks!

As a note on that issue: It seems that it converts the Tensor (which is likely on GPU, isn't it?) to a numpy type (on CPU, AFAIK numpy types can't be on GPU), which involves a Device->Host copy and a synchronization. The value converted is the metric and loss values.
So doing that on every step seems wasteful to me as it may not be used at all. So it might be a good idea not to convert that eagerly into a numpy type which would resolve the AttributeError and maybe improve performance.

@Flamefire the original issue of train_on_batch failed with MirroredStrategy is fixed in 2.2.0.
The second issue of Attribute Error when using train_on_batch with tf.function
model.train_on_batch is disallowed inside a tf.function , since it is a high-level end point that manages its own tf.function since it creates an Iterator, converts results to NumPy, etc. The best thing to use is Model.train_step, which is just the logic of the training step and is safe to tf.function.
Please let us know if that answers your questions and it is ok to close this issue.

Just for my understanding: In another issue I was told that tf.function is supposed to be the replacement for the TF 1 "graph mode" to recover (some of) the lost performance due to eager execution.
However it now seems to be impossible to create a whole training loop or at least a single epoch iteration in "graph mode" (tf.function) as e.g. calling train_on_batch inside a tf.function is (now?) disallowed. So this still looses quite some performance especially because of the (potentially unneccessary) conversion to numpy types after each batch. To my understanding feeding the GPU new batches as fast as possible without first waiting on the result of previous computations is vital in leveraging the full performance of the GPU.
My point in using train_on_batch over model.train_step (besides the latter didn't exist in 2.1) was to remove as much overhead as possible as I figure model.fit does quite some work so it works with any model and any kind of inputs and outputs.
So bottom line: Shouldn't also "high level functions" like rain_on_batch be able to be put in tf.function so at least parts of the overhead are removed? I suppose quite some checks can be effectively removed when that functions is compiled for a given model and dataset type, and not going through all those Python abstractions but executing (compiled) binary code instead would be a win.

@Flamefire I executed the code in TF 2.5 and didnt face any errors. Could you please confirm and let us know if we can close the issue.

Thanks for the update. Well there is at least an open question:

Shouldn't also "high level functions" like train_on_batch be able to be put in tf.function so at least parts of the overhead are removed? I suppose quite some checks can be effectively removed when that functions is compiled for a given model and dataset type, and not going through all those Python abstractions but executing (compiled) binary code instead would be a win.

See my previous message for details.


failed to build on win10

Bevy version
0.3.0
Operating system & version
Windows 10
Got the following link error when build bevy-snake
glslang.glsltospirv.lib(intermOut.obj) : error LNK2019: unresolved external symbol __imp__fpclass referenced in function "void __cdecl glslang::OutputDouble(class TInfoSink &,double,enum glslang::TOutputTraverser::EExtraOutput)" (?OutputDouble@glslang@@YAXAEAVTInfoSink@@NW4EExtraOutput@TOutputTraverser@1@@Z)
glslang.glsltospirv.lib(ParseHelper.obj) : error LNK2019: unresolved external symbol __imp_log2 referenced in function "double __cdecl log2<int,0>(int)" (??$log2@H$0A@@@YANH@Z)
fatal error LNK1120: 2 unresolved externals

I'm guessing this is another case of #788. Can you try installing Visual Studio 2019 build tools?
Lets leave this issue open until we've updated the Bevy Setup part of The Book to include a note that VS2019 build tools are a dependency on Windows.

I had used vs2019 build tools.
I had searched that issue, and know vs2019 is required.
So I open this issue.

I ran into the same issue while following the bevy book getting started guide on windows 10. This happened right after adding DefaultPlugins
Should we add the vs 2019 build tools to the book somewhere?

closed by https://github.com/bevyengine/bevy-website/pull/79

closed by https://github.com/bevyengine/bevy-website/pull/79


#157: added integration testing

Lets see how travis handles IDE GUIs popping up ;=)

Wow! This is incredible. So devon-ide is now certified to run on travis-ci! Very cool!


Child account should update routing table

When an account connects as a child to the Java connector the routing table should get updated automatically. Currently this does not happen and you have to create a static route.
Common use case: when connecting an SPSP server to a Java connector as a child

Fixed via https://github.com/interledger4j/ilpv4-connector/pull/575


API Documentation

Describe the changes you have made in this PR -

Adds API documentaion

Note: Please check Allow edits from maintainers. if you would like us to assist in the PR.

Coverage remained the same at 85.81% when pulling 01d48c5656dd03e4af61ca22ca1260b547ccc406 on Nitish145:api_docs into e8bd8cf2795c86581c46b5961fb2b230d312692b on CircuitVerse:master.

@tachyons next steps here?

@tachyons is there anyting i can do here ?


Add loader_class, parser_class as topen parameters?

As an author of libraries using tabulator I've understood that it will be better for topen to have all parameters to tweak table initiating without the usage of https://github.com/okfn/tabulator-py#advanced-interface. Because sometimes library in the middle just proxying topen parameters from end-user.
As a continuation of loader_options and parser_options addition.

DONE


fixed use of deleted function in network.h

line 470,
bool write(std::ofstream ofs, ...
calls the copy constructor of std::ofstream, which is "deleted".
Passing it by reference works, if you also change the write function @ line 513 to create a temporary std::ofstream.

Thanks!  I wasn't looking at pull requests because I thought no one was using this. I've fixed the issue and will commit soon.


Cannot merge LORA layers when the model is loaded in 8-bit mode

When I load the model as following, throw the error: Cannot merge LORA layers when the model is loaded in 8-bit mode
How can I load model with 4bit when inferencing?
model_path = 'decapoda-research/llama-30b-hf' adapter_path = 'timdettmers/guanaco-33b' quantization_config = BitsAndBytesConfig(     load_in_4bit=True,     bnb_4bit_compute_dtype=torch.bfloat16,     bnb_4bit_use_double_quant=True,     bnb_4bit_quant_type='nf4' ), model = AutoModelForCausalLM.from_pretrained(     model_path,     low_cpu_mem_usage=True,     load_in_4bit=True,     quantization_config=quantization_config,     torch_dtype=torch.float16,     device_map='auto' ) model = PeftModel.from_pretrained(model, adapter_path) model = model.merge_and_unload()

model_path = 'decapoda-research/llama-30b-hf'
adapter_path = 'timdettmers/guanaco-33b'
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type='nf4'
),
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    low_cpu_mem_usage=True,
    load_in_4bit=True,
    quantization_config=quantization_config,
    torch_dtype=torch.float16,
    device_map='auto'
)
model = PeftModel.from_pretrained(model, adapter_path)
model = model.merge_and_unload()

Just fyi decapoda-research is extremely out of date. Please use huggyllama instead.

did you solve this? and its the same result with huggyllama

@bodaay what is the sie of the adapter.bin you are getting? Mine is like in bytes.
Btw, I just commented out the model = model.merge_and_unload() line and it works. Merging is not necessary

@bodaay what is the sie of the adapter.bin you are getting? Mine is like in bytes.
Btw, I just commented out the model = model.merge_and_unload() line and it works. Merging is not necessary
remove "model = model.merge_and_unload()", and it works.

remove "model = model.merge_and_unload()", and it works.

@yangjianxin1, Based on the code snippet you provided, it seems that you are loading the model using AutoModelForCausalLM with 4-bit quantization enabled. However, when attempting to merge the LORA layers using merge_and_unload(), the mentioned error occurs.
To address this issue, we recommend the following approach:

Remove the line model = model.merge_and_unload() from your code. The merging step is not necessary for inference and can be omitted.

By removing the merge_and_unload() line, you should be able to successfully load and use the model for inference without encountering the "Cannot merge LORA layers when the model is loaded in 8-bit mode" error.

@yangjianxin1, Based on the code snippet you provided, it seems that you are loading the model using AutoModelForCausalLM with 4-bit quantization enabled. However, when attempting to merge the LORA layers using merge_and_unload(), the mentioned error occurs.
To address this issue, we recommend the following approach:

Remove the line model = model.merge_and_unload() from your code. The merging step is not necessary for inference and can be omitted.

By removing the merge_and_unload() line, you should be able to successfully load and use the model for inference without encountering the "Cannot merge LORA layers when the model is loaded in 8-bit mode" error.

How can i optain one model so i can use it in llama.cpp if i cant merge them?
Do you have any idea how to make the fine-tuned model applicable inside llama.cpp ?
Any help is highly appreciated

Removing merge_and_unload() is not the solution!!.
Of course, the inference works fine, there is no attempt to merge your LORA weights into the base model. What is the real solution here?

You can see a workaround here: https://github.com/substratusai/model-falcon-7b-instruct/blob/430cf5dfda02c0359122d4ef7f9b6d0c01bb3b39/src/train.ipynb
Effectively I reload the base model in 16 bit to work around the issue. It works fine for my use case.

Anyone found a way to solve this without loading the model in 16bit? My GPU cannot load whole Falcon-40B, using device_map=auto and offload_folder causes the python code to be Killed. Loading the base model in 4 bit mode and merging with LORA adapters still fail with "Cannot merge LORA layers in 8-bit mode"

I hope in the future this code could work... its more natural.
model = peft_model.merge_and_unload()
model.save_pretrained("/model/trained")

I hope in the future this code could work... its more natural.
model = peft_model.merge_and_unload()
model.save_pretrained("/model/trained")

Thanks,
but this saves adapters only. I need the model WITH the LORA weights inside and the corresponding config.json,  so that I can use it for "Huggingface Inference Endpoint"

Any updates on this?

You can see a workaround here: https://github.com/substratusai/model-falcon-7b-instruct/blob/430cf5dfda02c0359122d4ef7f9b6d0c01bb3b39/src/train.ipynb
Effectively I reload the base model in 16 bit to work around the issue. It works fine for my use case.

The link is broken

You can see a workaround here: https://github.com/substratusai/model-falcon-7b-instruct/blob/430cf5dfda02c0359122d4ef7f9b6d0c01bb3b39/src/train.ipynb
Effectively I reload the base model in 16 bit to work around the issue. It works fine for my use case.

Same here, the link is broken. can pls re-share the link?

New link: https://github.com/substratusai/images/blob/main/model-trainer-huggingface/src/train.ipynb

New link: https://github.com/substratusai/images/blob/main/model-trainer-huggingface/src/train.ipynb

Thank you so much!

New link: https://github.com/substratusai/images/blob/main/model-trainer-huggingface/src/train.ipynb

May I ask another baby question: this is training code and saving model code. After model is saved, did you test to load the saved model and do inference and check whether generated results are good? If so, do you have this separated inference/generating script? Thanks a lot in advance.

Yes in substratus.ai we separate model loading, finetuning and serving in separate images. I did check whether the finetuned model provided different results and it did.
In the notebook that I linked, the following paths are used
# original base model e.g. falcon-7b
model_path = "/content/saved-model/"
# path of final finetuned model (merged model)
trained_model_path = "/content/model"

Yes in substratus.ai we separate model loading, finetuning and serving in separate images. I did check whether the finetuned model provided different results and it did.
In the notebook that I linked, the following paths are used
# original base model e.g. falcon-7b
model_path = "/content/saved-model/"
# path of final finetuned model (merged model)
trained_model_path = "/content/model"

Thank you so much, I will try on my side and let you know.

I am worried about whether this quick fix would be harmful to the model's ability? Is there any other way to fix this problem?

I am worried about whether this quick fix would be harmful to the model's ability? Is there any other way to fix this problem?

@Tim_Detters suggests this is the case

Just to let you know, we noticed that you get better merged models if you (1) quantize the base model, (2) merge, (3) convert to bf16/fp16/fp32, (4) do any further conversion (like GPTQ). We have not run benchmarks yet for this, but the generation quality is quite telling.
https://twitter.com/Tim_Dettmers/status/1694654191325573456

Would be great to see quantized merging!

Anyone found a way to solve this without loading the model in 16bit? My GPU cannot load whole Falcon-40B (in 16-bit), using device_map=auto and offload_folder causes the python code to be Killed. Loading the base model in 4 bit mode and merging with LORA adapters still fail with "Cannot merge LORA layers in 8-bit mode"

I found a way that works in my case I hope it works, the problems working with Llama2 in terms of training time, inference time while we just have one (not big memory) GPU can be splited into 3 different parts which I will go through each of them.

Training: It is not possible to perform pure 8bit or 4bit training so by leveraging parameter efficient fine tuning methods (PEFT) and train for example adapters on top of them we can fine tune Llama2 model. prepare_model_for_kbit_training plays an important role for preparing model for training.
Merging: on the other hand for merging we can merge our LoRa Layers to Llama2 even on CPU. here is sample template:

llama_model = AutoModel.from_pretrained(model_name,
                        device_map={"": "cpu"},
                        torch_dtype=torch.float16)

model = PeftModel.from_pretrained(llama_model,
                        peft_model_path,
                        torch_dtype=torch.float16,
                        device_map={"": "cpu"})

model = model.merge_and_unload()

model.save_pretrained("merged-model")

Inference Time:  for loading now you can just need to read model from merged-model path, you can load model in 8bit or whatever needs. here is sample template:

model_for_infer = AutoModel.from_pretrained("merged-model",
                        device_map="auto",
                        load_in_8bit=True)

While not the most elegant solution, @hamidahmadian solution works for me.

I have an issue with this approach when I add a special token

On my laptop with 16GB RAM + 16GB VRAM, @hamidahmadian allows me to load the models, but still gives me an OOM error when doing merge_and_unload(). However, the following works for me:
llama_model = AutoModelForCausalLM.from_pretrained(model_name, device_map={"": "cpu"}, torch_dtype=torch.float16)
model = PeftModel.from_pretrained(llama_model, peft_model_path)
model = model.merge_and_unload()

This should be the correct way to fix the issue:
peft_model = model

# When you execute the commonly used `model = model.merge_and_unload()`, the error `Cannot merge LORA layers when the model is loaded in 8-bit mode` occurs. The reason is that the base model was loaded as 4 bit. Therefore, the base model must be reloaded as 16 bit.
model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    torch_dtype=torch.hh,
    load_in_8bit=False,
    device_map="auto",
    trust_remote_code=True,
)
from peft import PeftModel

peft_model = PeftModel.from_pretrained(model, NEW_MODEL_PATH)
merged_model = peft_model.merge_and_unload()

Solved, Thank @praharshbhatt


Removing reference to a bugfix done in 2017

Category

[X] Content fix
[ ] New article

Related issues:
N/A
What's in this Pull Request?
Removing reference to a bugfix done in 2017

thx @KoenZomers, appreciated.


Drive contents being wiped in creative mode

Issue description:
While in creative mode, moving an rs drive from one inventory slot to another clears the contents on all drives currently held in player inventory.
Before moving drive in inventory:

After moving drive in inventory:

This is only an issue in creative, drives are not cleared in survival.
What happens:
The "Items" NBT tag is cleared while the "Stored" tag is left intact.
What you expected to happen:
The "Stored" and "Item" NBT tags should remain intact.
Steps to reproduce:

Get any number of storage drives, the capacity does not matter.
Put items into the drives, either using a full RS system or a portable grid.
Move the drives back into your inventory.  At this point the drives should all still be intact.
Move one drive to a different slot in your inventory.  At this point the "Items" tag will have been cleared on all drives in your inventory.

Version (make sure you are on the latest version before reporting):

Minecraft: 1.12.2
Forge: <IP_ADDRESS>15-universal
Refined Storage: 1.5.34
Modpack: All The Mods 3 v5.9.8d

Does this issue occur on a server? [yes/no]
Yes

(make sure you are on the latest version before reporting)


Food Container Size Increase

About the PR
Increased the container size of all foods (that I could find, anyways) which had their reagents go over the container size.
Each container is at the nearest multiple of 5, with the exception of:

Nettle soup, increased to 30.5 as it has .5 Histamine in it, this means that you wont end up with having .5 of the reagent you had spare. It's purely so it looks pretty.
Super bite burger, it's a big burger, so it can hold a lot of flavouring reagents.
Tofu, made it consistent with it's slices.
Laughin' pea pod and Lemoon, made it exactly the amount because that's a thing that produce has. Presumably for mutagen or something, idk.

Why / Balance
Some foods would have less food then they should have, and things like soups and salads which take bowls would actually  spill when made.
Technical details
Changed the max container size of a bunch of food items. Not sure if this will cause the universe to explode.
Media
Do you want a screenshot of bungo curry? get it yourself
Requirements

[X] I have read and am following the Pull Request and Changelog Guidelines.
[X] I have added media to this PR or it does not require an ingame showcase.

can you make chicken sandwiches hold more? when I tried putting a certain addictive chemical in it was already full so I just had to serve the sandwiches and glasses of meth as a side

What I did was only increase the foods that were over capacity, not right on, but yeah, I'll probably do that

i don't know how git works, but the PR is done. enjoy!


Fix Bezier Editor throwing error when adding key with CTRL+click

fixes #87861

I don't really know how cherry-picking works yet, but note that this code relies on the make_default_bezier_key method I added in the copy-paste PR #87250

Thanks for the headsup. That other PR likely won't be cherry-picked, so it might be worth making a dedicated backport of this PR for the 4.2 branch to fix the bug there, keeping the manually defined array.

Thanks!

Cherry-picked for 4.2.2.


botsing-preprocessing doc error and missing download

Characteristics

Issue Type: [documentation]
Severity: [minor]
Reporter: [Pierre-Yves Gibello<EMAIL_ADDRESS>
Description
In the botsing-preprocessing doc (README.md), at least 2 issues:

The link to botsing release points to a place where there is no botsing-preprocessing jar.
The command line options are wrong: it is -l instead of -i, and there is no "=" sign after options (but a space, as usual).

Please refer to this doc

It's what I did. No botsing-preprocessing jar in the releases, and wrong option for the command line.

I made a release with the latest version of the preprocessing tool (see https://github.com/STAMP-project/botsing/releases/tag/botsing-1.0.4). @pasquy73 can you fix the documentation and move it to the Command line interface section. Right now it is in the Contributing section, which is wrong.

Fixed in #65


[NEW ALGORITHM] Kruskal’s Algorithm - Minimum Spanning Tree

Name:
[NEW ALGORITHM] Kruskal’s Algorithm - Minimum Spanning Tree
About:
Kruskal’s Algorithm is a greedy algorithm used to find the Minimum Spanning Tree (MST) of a connected, undirected graph. It works by sorting all edges by their weights and adding them one by one to the MST, ensuring no cycles are formed. This algorithm is efficient for sparse graphs and widely used in network design, where connecting nodes with minimum cost is crucial.

Labels:
new algorithm, gssoc-ext, hacktoberfest, level1

Assignees:

[X] Contributor in GSSoC-ext
[ ] Want to work on it

Issue will be closed if:

You mention more than one algorithm. You can create a separate issue for each algorithm once the current one is completed.
You propose an algorithm that is already present or has been mentioned in a previous issue.
You create a new issue without completing your previous issue.

Note: These actions will be taken seriously. Failure to follow the guidelines may result in the immediate closure of your issue.

i would like to work this issue.

i would like to work this issue.

Hi @surbhisharma004 I have raised this issue, if you want you can later raise an issue to improve my algorithm,

@pankaj-bind there's no issue for this


Plz update V hud for GTA SA

Why you stopped working on this mod,
Plz update it,
See my review about your mod in request page

This is not even my mod, and this is not a repository for that. I don't know why you are asking me about that.


WIP use typesafe config

Fixed #9

Only ssp and stun module left to migrate to typesafe config. Thing is, now the config will no longer be reloadable, so i need to add an expiring map in a wrapper class that makes sure that the typesafe config is reloaded and flushed every so often. But do the users of freeacs rely on hot reloading the properties file?
In the future, freeacs will move towards container less self running jar files. Possibly inside docker files. If not inside docker, and init script in linux will suffice, and you will start each module with java -jar -Ddb.xaps.url=... -Dmonitor.url=.... freeacs-web.jar etc.

Nah. This stinks. It's so much hassle working with legacy apps that i should rewrite them all to spring boot. It's future proof. And it serves the same purpose in regards to injecting system properties. And i get ezecutable jar for free.


Module not found: Error: Can't resolve 'fs' in 'C:\Dev\client-apps\react-query-builder\node_modules\sql-parser-mistic\lib'

I use it with react-app and I've got an error:
Module not found: Error: Can't resolve 'fs' in '\node_modules\sql-parser-mistic\lib' .
How can I use it within browser application?

Try to configure your build chain to use sql-parser-mistic/browser/sql-parser.js

By the way I don't see any usage of fs in the sources.


Error when trying to train: RuntimeError: cuda runtime error (59) : device-side assert triggered

❓ Questions and Help
I'm trying to train a custom data set with train.py as seen in the tools folder. I followed the instructions to build a consistent dataloader. (cross checking with the coco dataloader it seems all the types/dim are in order)
Using the CUDA_LAUNCH_BLOCKING=1 before python3 train.py (to get more info in the output)
I get the following error:
2018-12-03 17:06:30,676 maskrcnn_benchmark.trainer INFO: Start training
/ibm/gpfs-homes/ial/.local/tmp_compilation/pytorch-master-at10.0/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [31,0,0] Assertion `t >= 0 && t < n_classes` failed.
THCudaCheck FAIL file=/ibm/gpfs-homes/ial/.local/tmp_compilation/pytorch-master-at10.0/pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu line=111 error=59 : device-side assert triggered
Traceback (most recent call last):
  File "relational_rxn_graphs/detector/train.py", line 228, in <module>
    main()
  File "relational_rxn_graphs/detector/train.py", line 221, in main
    model = train(cfg, data_cfg, args.local_rank, args.distributed)
  File "relational_rxn_graphs/detector/train.py", line 71, in train
    arguments,
  File "/ibm/gpfs-homes/ial/github/maskrcnn-benchmark/maskrcnn_benchmark/engine/trainer.py", line 66, in do_train
    loss_dict = model(images, targets)
  File "/u/ial/.local/deeplearning/pytorch-master/lib64/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/ibm/gpfs-homes/ial/github/maskrcnn-benchmark/maskrcnn_benchmark/modeling/detector/generalized_rcnn.py", line 52, in forward
    x, result, detector_losses = self.roi_heads(features, proposals, targets)
  File "/u/ial/.local/deeplearning/pytorch-master/lib64/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/ibm/gpfs-homes/ial/github/maskrcnn-benchmark/maskrcnn_benchmark/modeling/roi_heads/roi_heads.py", line 23, in forward
    x, detections, loss_box = self.box(features, proposals, targets)
  File "/u/ial/.local/deeplearning/pytorch-master/lib64/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/ibm/gpfs-homes/ial/github/maskrcnn-benchmark/maskrcnn_benchmark/modeling/roi_heads/box_head/box_head.py", line 55, in forward
    [class_logits], [box_regression]
  File "/ibm/gpfs-homes/ial/github/maskrcnn-benchmark/maskrcnn_benchmark/modeling/roi_heads/box_head/loss.py", line 139, in __call__
    classification_loss = F.cross_entropy(class_logits, labels)
  File "/u/ial/.local/deeplearning/pytorch-master/lib64/python3.5/site-packages/torch/nn/functional.py", line 1928, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/u/ial/.local/deeplearning/pytorch-master/lib64/python3.5/site-packages/torch/nn/functional.py", line 1771, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: cuda runtime error (59) : device-side assert triggered at /ibm/gpfs-homes/ial/.local/tmp_compilation/pytorch-master-at10.0/pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:111

I have the following configuration
PyTorch version: 1.0.0a0+5c89190
Is debug build: No
CUDA used to build PyTorch: 9.2.148

OS: Red Hat Enterprise Linux Server release 7.5 (Maipo)
GCC version: (GCC) 6.4.1 20170720 (Advance-Toolchain-at10.0) IBM AT 10 branch, based on subversion id 250395.
CMake version: version <IP_ADDRESS>

Python version: 3.5
Is CUDA available: Yes
CUDA runtime version: 9.2.148
GPU models and configuration:
GPU 0: Tesla P100-SXM2-16GB
GPU 1: Tesla P100-SXM2-16GB
GPU 2: Tesla P100-SXM2-16GB
GPU 3: Tesla P100-SXM2-16GB

Nvidia driver version: 396.37
cuDNN version: Probably one of the following:
/usr/local/cudnn-8.0-v5.1/lib64/libcudnn.so.5.1.10
/usr/local/cudnn-8.0-v5.1/lib64/libcudnn_static.a
/usr/local/cudnn-8.0-v6.0/lib64/libcudnn.so.6.0.20
/usr/local/cudnn-8.0-v6.0/lib64/libcudnn_static.a
/usr/local/cudnn-9.0-v7.0/lib64/libcudnn.so.7.0.5
/usr/local/cudnn-9.0-v7.0/lib64/libcudnn_static.a
/usr/local/cudnn-9.1-v7.1.2/lib64/libcudnn.so.7.1.2
/usr/local/cudnn-9.1-v7.1.2/lib64/libcudnn_static.a
/usr/local/cudnn-9.2-v7.1.3/lib64/libcudnn.so.7.1.3
/usr/local/cudnn-9.2-v7.1.3/lib64/libcudnn_static.a
/usr/local/cudnn-9.2-v7.2.1/lib64/libcudnn.so.7.2.1
/usr/local/cudnn-9.2-v7.2.1/lib64/libcudnn_static.a

This is a continuation of  #230, it seems that the initial bug was fixed by reinstalling the library + torch.

This probably means that your class labels are larger than the number of outputs from the model.
Could you check that?

It's working now.
Yeah that is what I thought, I was using a dataloader which had a function to return the nb_classes, I did not realize you had to add it to the config. I thought that this was the standard way to get the nb_classes... big mistake.
Thanks a lot!

@Nacho114 @fmassa running into this error as well.. In my config, I have ROI_BOX_HEAD.NUM_CLASSES set to 3. I'm using the R-50.pkl pretrained weights, so I figured I do not need to follow the instructions in #15. What else is missing? Do you need to set the number of classes within the dataset class? Any help would be appreciated

Hi @jbitton ,
I've replied to your other issue in https://github.com/facebookresearch/maskrcnn-benchmark/issues/273. Let's continue the discussion there


Logical calculation

hi.In my project how many parameters are deciding by user And it will use Logical calculation
just like "a+b>0 and c<0 or d>5 and not e".
In example i found that
Entity expr = "a and x > 3";
var func = expr.Compile<bool, double, bool>("a", "x");
WriteLine(func(true, 6));
but  if i have  more than 8 parameters ，what should I do？

But the number of parameters are design by users.So the method of  expr.Compile<Func>(.....)   is hard to use in my case.....is any other way else?

I see.
1. Reflectoin (easy way)
You (theoreticall) can construct type Func on the fly, using typeof(Func<>) and method MakeGeneric, and then invoking method Compile using reflection too
2. Rewriting the compiler (hard way)
There's also old compiler, which is as simple as
myExpr.Compile("a", "b", "c")

It returns FastExpression type instead of a delegate. It isn't as fast as delegate, but its main problem is that it's far less feature-rich (here is the interpreter, here is the compiler).
And becuse it's not as feature rich, to make it work you will have to fork AngouriMath and

Add bool operators to the compiler and interpreter (it's not too hard, changes are expected here, here, and here)
Make FastExpression.Substitute return a bool (or make SubstituteBool or something)

thanks a lot. I will try it later。


integration tests: timeout flake

I've been investigating occasional integration test failures where the stream-sum chunker doesn't receive an expected sum within its SLA (30 seconds).
When it happens, the fault is consistently between TestDeleteAllBrokers and TestDeleteAllSummers (consumers). What's happening is that:

TestCrashOneSummer runs (and passes), but causes summers to re-balance shards among themselves going into TestDeleteAllBrokers.
TestDeleteAllBrokers runs (and passes), such that all-new brokers start up, and old ones persist fragments and exit.
However, a summer consumer may be completing recovery around this time, is beginning to read from its respective source journal at offset X, but offset X is in a fragment that the (new) broker isn't aware of yet (refresh_interval is 1m). It will necessarily block until an index refresh occurs, which violates expected test SLA.

Another compounding issue is that gazette K8S service endpoints are emptied by TestDeleteAllBrokers until new brokers pass readiness checks (and they're using the default 10s periodSeconds. On top of that, gRPC uses exponential backoff in dialing the service. This causes ~10s of delay, but not quite enough to itself trigger an SLA failure.
To fix this properly, the integration runner needs to do a deeper cluster inspection to verify that all components are stable before starting the next crash test (all pods are ready / not terminating, all shards have recovered or are tailing, are consuming at journal write-heads, etc). Dropping periodSeconds wouldn't hurt, either.

Integration tests are passing consistently (up through 10 rounds). Almost all of the work was in identifying and tightening various backoffs to ensure failure was handled within the test's SLA expectation. Some tweaks of tests and fixtures was also required.
Calling this good for now.


Switch current directory in Jenkins tests

pfnCI, test this please.

Jenkins CI test (for commit 34b00f2f467406a7ffdf8b54ff1f35969276d25a, target branch master) failed with status FAILURE.

Please point the basis of this fix.

Jenkins, test this please.

As the commit message says, it'll fix an error in setting up conda in docker.
17:42:22 + docker build -t image1 --build-arg WORKSPACE_DIR=/workspace --build-arg WORK_DIR=/workspace/work --build-arg CONDA_DIR=/workspace/conda /tmp/tmp.sMfEaynh7a

...
17:42:23 Step 20/21 : RUN bash "$TEMP_DIR"/setup-conda.sh "$USER_TEMP_DIR"/conda "$CONDA_DIR"

...
17:42:42 Preparing transaction: ...working... done
17:42:43 Executing transaction: ...working... WARNING conda.core.envs_manager:register_env(46): Unable to register environment. Path not writable or missing.
17:42:44   environment location: /workspace/conda
17:42:44   registry file: /home/testuser/.conda/environments.txt
17:42:44 done
17:42:44 installation finished.
17:42:44 + bin_dir=/workspace/conda/bin
17:42:44 + /workspace/conda/bin/conda config --set changeps1 no
17:42:44 + /workspace/conda/bin/conda update -y -q conda
17:42:44 Collecting package metadata (current_repodata.json): ...working... done
17:42:46 Solving environment: ...working... done
17:42:47
17:42:47 ## Package Plan ##
17:42:47
17:42:47   environment location: /workspace/conda
17:42:47
17:42:47   added / updated specs:
17:42:47     - conda
17:42:47
17:42:47
17:42:47 The following packages will be downloaded:
17:42:47
17:42:47     package                    |            build
17:42:47     ---------------------------|-----------------
17:42:47     certifi-2019.6.16          |           py37_1         156 KB
17:42:47     chardet-3.0.4              |        py37_1003         173 KB
17:42:47     ------------------------------------------------------------
17:42:47                                            Total:         329 KB
17:42:47
17:42:47 The following packages will be UPDATED:
17:42:47
17:42:47   certifi                                  2019.6.16-py37_0 --> 2019.6.16-py37_1
17:42:47   chardet                                      3.0.4-py37_1 --> 3.0.4-py37_1003
17:42:47
17:42:47
17:42:47 Preparing transaction: ...working... done
17:42:47 Verifying transaction: ...working... done
17:42:47 Executing transaction: ...working... done
17:42:47 + /workspace/conda/bin/conda create -y -q --name testenv python=3.6 pip
17:42:47 Collecting package metadata (current_repodata.json): ...working... done
17:42:47 Solving environment: ...working... done
17:42:53
17:42:53 [Errno 13] Permission denied: '/.cph_tmpvcwkmshg'
17:42:53 [Errno 13] Permission denied: '/.cph_tmpqm03zzdk'
17:42:53 [Errno 13] Permission denied: '/.cph_tmpuqjhnma5'

(taken from https://jenkins.preferred.jp/job/chainer/job/chainer_pr/TEST=CHAINERX_chainerx-c,label=mn1-p100/1705/console)

Jenkins CI test (for commit 56fe8127bafb2a611dbb9162daf343cfcb4a501d, target branch master) failed with status FAILURE.

The Jenkins failures
FAIL ../../../repo/tests/chainerx_tests/unit_tests/routines_tests/test_creation.py::test_TrilTriu_param_5_{k=1, shape=(2, 3, 4)}[float16-cuda:0]

and #7741 are unrelated to the PR.


GitAtomic added to the list of git GUI clients

Hi guys,
Could you please review my pull request and add new git GUI client: GitAtomic to the list of GUI clients?
Thank you

seems good @smallB007 :) thanks for the contribution!

(the change should be live soon )

Thanks guys!


Fix 'GC.SuppressFinalize within the finally block' #776

Fixes #776
DisposablesShouldCallSuppressFinalizeAnalyzer did not find GC.SuppressFinalize(this) in code blocks. Now it checks for GC.SuppressFinalize(this) in any block - finally/if etc.
I added proper tests to check this issue.

Coverage remained the same at 91.622% when pulling 3908445df97d385e091c6f500066038294d0cf02 on vs-krasnov:v1.0.x into f7d2893e8ea03ace30451a553a00f80615553987 on code-cracker:v1.0.x.


How did example claim get set to expired during claimtrie sync

example https://chainquery.lbry.io/api/sql?query=SELECT * FROM claim WHERE claim_id ="9377895520f71f82702b2dc56cc45d2f046ada8a"
There was an issue where the lbrycrd node was not updated until well after the hardfork. This may be the cause, but we should investigate to make sure there is not a bug somewhere.

deployed with v0.5.2


Add scaling functions

Implement a couple of scaling methods in a fashion similar to how they're implemented in Sass.
I need these functions for use with elm-css.  I figured I'd try and get them merged into an established library before creating a separate package!

Hi Brian,
thanks for the pull request. Could you please improve the documentation for people that are not familiar with the SASS functionality. I had to switch to the SASS docs to understand why its implemented like this. Maybe you can summarise these section from the SASS docs in one or two sentences:

That means that lightening an already-light color with scale-color won’t change the lightness much, but lightening a dark color by the same amount will change it more dramatically. This has the benefit of making scale-color($color, ...) have a similar effect regardless of what $color is.
For example, the lightness of a color can be anywhere between 0% and 100%. If scale-color($color, $lightness: 40%) is called, the resulting color’s lightness will be 40% of the way between its original lightness and 100. If scale-color($color, $lightness: -40%) is called instead, the lightness will be 40% of the way between the original and 0.

Done 👍

Thanks again, its merged and the package is published.
Btw you missed to add the 2 new function to the @doc at the top of the file. Next time run the elm-package diff to check if the package is ready to get published.

Ahh, thanks for the heads up!

I just noticed that the way I wrote the docs are causing them to render incorrectly here.  I will submit another PR tonight to fix.  Lesson learned: elm-package diff and verify the docs render correctly!


DBZ-3647 initial commit

https://issues.redhat.com/browse/DBZ-3647
Supersedes PR #1
The following are TODOs that are outstanding:

[x] Test sink connector built and deployed inside Kafka Connect, particularly offset management & failures
[ ] Add tests & verify support for remaining geometry and text-based PostgreSQL data types (i.e. Ltree, CiText, etc)
[ ] Add Db2 source support to the test template matrix
[ ] Add performance benchmark tests
[ ] Update documentation
[ ] Verify GitHub CI works with the workflows (appears to require merging first)
[x] Assembly step takes an exceptionally long time, why?

Current Known Issues

Open source drivers are not being packaged with the plugin - Fixed locally.
The quote.identifiers property has a type conflict, should be set to ConfigDef.Type.BOOLEAN - Fixed locally.

@Naros One question about the naming of the connector/project and repository too. Other connectors use the convention debezium-connector-<name>. IMHO to be consistent we should either use debezium-connector-jdbc-sink or debezium-sink-jdbc. WDYT?

@Naros Added commit with connector rename for testing release.

@Naros Applied, great job! Let's see if CI is executed as expected.
Could you please create follow-up Jira(s) for the remaining tasks?

@jpechane Jiras are created and they're referenced in the main comment on this PR.

@Naros Thanks!


News Item on DFG presentation in Bonn

Trying to add a news item ...

Haven't seen any further checks, hope that it works.


Create and Configure Forms

Tutorials: https://developers.sap.com/tutorials/spa-create-forms.html
Hi,
There's a few spots in the Tutorial where markdown tables are not rendering properly.  Here's an example:
Design the Approval Form in the form builder by simply dragging-and-dropping fields onto the form editor and configuring respective field settings.
| Form Fields | Field Settings with Label | :————- | :————- | Headline 1 | Approve Sales Order | Paragraph | A new order has been received. Please review and confirm whether the requirements can be met or not. | Paragraph | Sales Order Details:

Tutorial formats are fixed now


updates(home slider): slider button navigation

Description
config buttons to navigate to a specific location
Why
NA
Issue
NA
Checklist
Please delete options that are not relevant.

[x] I have followed the contributing guidelines
[x] I have performed a self-review of my own code
[x] I have successfully tested my changes locally

Please include changelog

No log info needed for this task


possibility to add few POI defined by us

For example possibility to add :

amenity=recycling, recycling_type=container, recycling:clothes=yes
amenity=recycling, recycling_type=container, recycling:glass=yes, recycling:glass_bottles=yes
amenity=recycling*

From Guillaume AMAT on 2015-08-06 00:43:52+00:00
Do you mean add some new tags directly in the contribution column?

From Vincent Bergeot on 2015-08-06 09:55:51+00:00
no, to create new nodes perhaps only main tag, for example only amenity=recycling. In fact, i need to create a lot of new POI in ID and i characterize them in ID. But it isn't possible to use ID on smartphone so I suggest possibility to add "a simple POI" in Rudomap :)

From Guillaume AMAT on 2015-08-06 10:30:25+00:00
So this issue is a duplicate of the issue #62?

From Guillaume AMAT on 2015-08-06 10:31:05+00:00
And yeah, I miss this functionnality a lot too ;)

From Vincent Bergeot on 2015-08-06 11:04:44+00:00
Duplicate of #62.


[DO NOT MERGE] Add dataflow.properties file to modules

This is a placeholder PR for when https://github.com/spring-cloud/spring-cloud-dataflow/pull/432 is in.
At the moment, only the cassandra-sink module is concerned, to ease review of the aforementioned PR.

Is this really the only way to support filtering of the properties for completion?
I am strongly opposed to anything that requires a spring-cloud-stream app to be aware of Data Flow. cc @mbogoevici

Closing this for now, as we slightly change approach


[music3] four-split and maximized actions menu cleaned

From https://github.com/Inria-Asclepios/medInria-public/pull/259

Four-split button becomes a menu action in "Tools"
Maximized menu action used "toggle" to display if the view is maximized or not
clean the interactions with/without data

:m:

This PR solves the crash opening four-split view with several layers


Mini-issue: Refactor timeline pages to reuse code, add queries to reflect endpoints

As a user, I want to have pages on all endpoints which display cheeps, so that I am not overwhelmed.
As a developer, I want to have all timelines reuse common code for cheep pagification and display, so that I only have to make changes to the timeline in one place.
Acceptance Criteria

Refactoring cheep displaying pages to use pagination and query code from common records or a superclass
Implement author querying on timeline endpoints
Rewrite to ensure that page 0 is considered as page 1 in queries

Definition of Success

All cheep-displaying endpoints display cheeps by page and have queries for both author and page.

We created this issue before he noted it in the lecture, so we are actually the best group


The error message of "az network traffic-manager profile create" is not friendly

Describe the bug
$ az network traffic-manager profile create --name TrafficManagerProfile --resource-group ${RESOURCE_GROUP_NAME} --routing-method Performance --unique-dns-name <unique-dns-name>--monitor-protocol TCP --monitor-port 443 --monitor-path null
Operation failed with status: 'Bad Request'. Details: The Monitor Path must be null when using TCP as a monitoring protocol.

The Monitor Path must be null is not friendly to users.
Expected behavior
The error message should be more clear. For example:
Operation failed with status: 'Bad Request'. Details: The Monitor Path must be empty string when using TCP as a monitoring protocol.

What you are seeing is the service error, which does expect a null value. However, for the CLI, that simply means don't send that.
We can look at the internal command validation to see if it can be improved.

If I don't send --monitor-path, the default value of --monitor-path is /.  CLI gives me exactly same error message Operation failed with status: 'Bad Request'. Details: The Monitor Path must be null when using TCP as a monitoring protocol..
My point is that CLI should ask for an empty string instead of null for the monitor path.

Based on this, I would argue that --monitor-path should not have any default. This may have been a service-change. Thanks for brining it up.

This will be fixed in the next release.


Add a default attribute to HED XML

Unit classes have a default attribute.   We need to be able to add a default to a # for certain cases that over ride the default unit class.
Example:  Direction has unit class angle but its default is degrees not radians.

Maybe we can add another attribute to HED node called "defaultUnit"? With that if a node has multiple unit classes it must also have the same number of corresponding default units to make it clear for supporting tools?

No, just look at time of unitClass.  No need to specify default for all unitClasses specified.

That was just look at the type of unit that is specified in the default. It is already implemented.  We currently call this default, but we should rename this attribute defaultUnit.

Default units were added with 8.0.0.


How to configure the parameter for a long short strategy

What are you trying to do?
Hi I have been using PyPortfolioOpt and Vectorbt for backtesting with portfolio optimization. However, I am confused in configuring the parameter for a long short strategy. For example, I would like to long the top 5 sharpe ratio and short the lowest 5.
After I set the weight_bounds=(-1,1), and I set the short ratio in the DiscreteAllocation(clean_weights, latest_prices, total_portfolio_value=25000, short_ratio = 0.5), now I am unsure how to deal with the constraint, the min_weight and max_weight.
def pyopt_find_weights(sc, price, num_tests):
    price = pd.DataFrame(price, columns=symbols)
    avg_returns = expected_returns.mean_historical_return(price)
    cov_mat = risk_models.CovarianceShrinkage(price).ledoit_wolf()

    ef = EfficientFrontier(avg_returns, cov_mat, weight_bounds=(-1,1))

    min_weight, max_weight = -1, 1
    constraints=[
            # {"type": "eq", "fun": lambda w: np.sum(w) - 1},  # sum to 1
            {"type": "ineq", "fun": lambda w: w - min_weight},  # greater than min_weight
            {"type": "ineq", "fun": lambda w: max_weight - w},  # less than max_weight
        ]

    weights = ef.nonconvex_objective(
        objective_functions.sharpe_ratio,
        objective_args=(avg_returns, cov_mat),
        weights_sum_to_one=True,
        constraints = constraints
    )

    clean_weights = ef.clean_weights()
    weights = np.array([clean_weights[symbol] for symbol in symbols])
    best_sharpe_ratio = base_optimizer.portfolio_performance(weights, avg_returns, cov_mat)[2]
    latest_prices = get_latest_prices(price)
    da = DiscreteAllocation(clean_weights, latest_prices, total_portfolio_value=25000, short_ratio = 0.5)
    allocation, leftover = da.lp_portfolio(reinvest=True)

    w = pd.DataFrame(clean_weights, columns=clean_weights.keys(), index = [0])
    w.to_excel('w.xlsx')
    return best_sharpe_ratio, weights

Here is the full version of my code:
import os
import numpy as np
import pandas as pd
import yfinance as yf
from datetime import datetime
import pytz
from numba import njit
import matplotlib.pyplot as plt
import seaborn as sns

from pypfopt.efficient_frontier import EfficientFrontier
from pypfopt import risk_models
from pypfopt import expected_returns
from pypfopt import base_optimizer
from pypfopt import objective_functions
from pypfopt.discrete_allocation import DiscreteAllocation, get_latest_prices
from pypfopt import EfficientSemivariance
from pypfopt.expected_returns import mean_historical_return
from pypfopt.expected_returns import returns_from_prices

import vectorbt as vbt
from vectorbt.generic.nb import nanmean_nb
from vectorbt.portfolio.nb import order_nb, sort_call_seq_nb
from vectorbt.portfolio.enums import SizeType, Direction
from IPython.display import set_matplotlib_formats
%matplotlib inline
%config InlineBackend.figure_format='retina'

symbols = ['NVDA','AMD','TSLA','NET','JPM','AAPL']
start_date = datetime(2012, 1, 1, tzinfo=pytz.utc)
end_date = datetime.today().strftime('%Y-%m-%d')

num_tests = 1000000

vbt.settings.array_wrapper['freq'] = 'days'
vbt.settings.returns['year_freq'] = '252 days'
vbt.settings.portfolio['seed'] = 42
vbt.settings.portfolio.stats['incl_unrealized'] = True

yfdata = vbt.YFData.download(symbols, start=start_date, end=end_date)
ohlcv = yfdata.concat()
price = ohlcv['Close'].fillna(method='ffill')

srb_sharpe = np.full(price.shape[0], np.nan)

@njit
def pre_sim_func_nb(c, every_nth):
    # Define rebalancing days
    c.segment_mask[:, :] = False
    c.segment_mask[every_nth::every_nth, :] = True
    return ()

@njit
def find_weights_nb(c, price, num_tests):
    # Find optimal weights based on best Sharpe ratio
    returns = (price[1:] - price[:-1]) / price[:-1]
    returns = returns[1:, :]  # cannot compute np.cov with NaN
    mean = nanmean_nb(returns)
    cov = np.cov(returns, rowvar=False)  # masked arrays not supported by Numba (yet)
    best_sharpe_ratio = -np.inf
    #best_sharpe_ratio = -np.inf
    weights = np.full(c.group_len, np.nan, dtype=np.float_)

    for i in range(num_tests):
        # Generate weights
        w = np.random.random_sample(c.group_len)
        w = w / np.sum(w)

        # Compute annualized mean, covariance, and Sharpe ratio
        p_return = np.sum(mean * w) * ann_factor
        p_std = np.sqrt(np.dot(w.T, np.dot(cov, w))) * np.sqrt(ann_factor)
        sharpe_ratio = p_return / p_std
        if sharpe_ratio > best_sharpe_ratio:
            best_sharpe_ratio = sharpe_ratio
            weights = w

    return best_sharpe_ratio, weights

@njit
def pre_segment_func_nb(c, find_weights_nb, history_len, ann_factor, num_tests, srb_sharpe):
    if history_len == -1:
        # Look back at the entire time period
        close = c.close[:c.i, c.from_col:c.to_col]
    else:
        # Look back at a fixed time period
        if c.i - history_len <= 0:
            return (np.full(c.group_len, np.nan),)  # insufficient data
        close = c.close[c.i - history_len:c.i, c.from_col:c.to_col]

    # Find optimal weights
    best_sharpe_ratio, weights = find_weights_nb(c, close, num_tests)
    srb_sharpe[c.i] = best_sharpe_ratio

    # Update valuation price and reorder orders
    size_type = SizeType.TargetPercent
    direction = Direction.LongOnly
    order_value_out = np.empty(c.group_len, dtype=np.float_)
    for k in range(c.group_len):
        col = c.from_col + k
        c.last_val_price[col] = c.close[c.i, col]
    sort_call_seq_nb(c, weights, size_type, direction, order_value_out)

    return (weights,)

@njit
def order_func_nb(c, weights):
    col_i = c.call_seq_now[c.call_idx]
    return order_nb(
        weights[col_i],
        c.close[c.i, c.col],
        size_type=SizeType.TargetPercent
    )

ann_factor = returns.vbt.returns.ann_factor

def pyopt_find_weights(sc, price, num_tests):  # no @njit decorator = it's a pure Python function
    price = pd.DataFrame(price, columns=symbols)
    avg_returns = expected_returns.mean_historical_return(price)
    cov_mat = risk_models.CovarianceShrinkage(price).ledoit_wolf()

    ef = EfficientFrontier(avg_returns, cov_mat, weight_bounds=(0,1))

    min_weight, max_weight = 0.05, 0.35
    constraints=[
            # {"type": "eq", "fun": lambda w: np.sum(w) - 1},  # sum to 1
            {"type": "ineq", "fun": lambda w: w - min_weight},  # greater than min_weight
            {"type": "ineq", "fun": lambda w: max_weight - w},  # less than max_weight
        ]

    weights = ef.nonconvex_objective(
        objective_functions.sharpe_ratio,
        objective_args=(avg_returns, cov_mat),
        weights_sum_to_one=True,
        constraints = constraints
    )

    clean_weights = ef.clean_weights()
    weights = np.array([clean_weights[symbol] for symbol in symbols])
    best_sharpe_ratio = base_optimizer.portfolio_performance(weights, avg_returns, cov_mat)[2]
    latest_prices = get_latest_prices(price)
    da = DiscreteAllocation(clean_weights, latest_prices, total_portfolio_value=25000)
    allocation, leftover = da.lp_portfolio(reinvest=True)

    w = pd.DataFrame(clean_weights, columns=clean_weights.keys(), index = [0])
    w.to_excel('w.xlsx')
    return best_sharpe_ratio, weights

pyopt_srb_sharpe = np.full(price.shape[0], np.nan)

pyopt_srb_pf = vbt.Portfolio.from_order_func(
    price,
    order_func_nb,
    pre_sim_func_nb=pre_sim_func_nb,
    pre_sim_args=(63,), #63 #84
    pre_segment_func_nb=pre_segment_func_nb.py_func,  # run pre_segment_func_nb as pure Python function
    pre_segment_args=(pyopt_find_weights, -1, ann_factor, num_tests, pyopt_srb_sharpe),
    cash_sharing=True,
    group_by=True,
    use_numba=False  # run simulate_nb as pure Python function
)

I have tried to look for information or resources on long short strategy but no luck on the internet.
Greatly appreciate any help or expert view on my code.
What data are you using?
I am using vbt.YFData.download(symbols, start=start_date, end=end_date) for the data.
yfdata = vbt.YFData.download(symbols, start=start_date, end=end_date)
ohlcv = yfdata.concat()
price = ohlcv['Close'].fillna(method='ffill')

I'm not too sure about the vectorbt side, but your constraints seem to be linear so you should just be able to:
price = pd.DataFrame(price, columns=symbols)
avg_returns = expected_returns.mean_historical_return(price)
cov_mat = risk_models.CovarianceShrinkage(price).ledoit_wolf()

ef = EfficientFrontier(avg_returns, cov_mat, weight_bounds=(-1,1))
weights = ef.max_sharpe()

to get the weights in each loop.

Closing due to inactivity

For clarity sake, why is yf called but not used, and vbt is used instead?

Not sure if it answers your question. As its just part of my codes, and it's more on the vectorbt side, I am just using the feature vbt.YFData.fetch(['TICKER']).get('Close') here.


Mixing Side-Effects and Assertion Checks

The following generates a precondition violation for reasons unknown:
type Reader is method()->int[]

public function apply(int[] data) -> int[]:
    return data

method main(Reader read):
    int[] input = apply(read())

It looks to be some kind of problem with lambda's?

This is to do with the ordering of side-effects versus assertion checks.  Specifically in this case we have the following:
   assert Test_Valid_1$apply$faIaI#pre(m);
   call HEAP, m := apply$mVaI(read,HEAP);

The problem is that m is not assigned until the second statement.

This is to do with the ordering of side-effects versus assertion checks.  Specifically in this case we have the following:
   assert Test_Valid_1$apply$faIaI#pre(m);
   call HEAP, m := apply$mVaI(read,HEAP);

The problem is that m is not assigned until the second statement.

This problem is larger than I first thought, as illustrated by the following:
method m(int x) -> bool
requires x >= 0:
    return false

method main(int v):
    if v >= 0 && m(v):
        skip

The challenge is that we need to retain window inference whilst extracting side-effects.  This is actually quite challenging.

This problem is larger than I first thought, as illustrated by the following:
method m(int x) -> bool
requires x >= 0:
    return false

method main(int v):
    if v >= 0 && m(v):
        skip

The challenge is that we need to retain window inference whilst extracting side-effects.  This is actually quite challenging.

Options:

Get rid of side-effects altogether.  Is that even possible?

Options:

Get rid of side-effects altogether.  Is that even possible?

Other suggestions:

(Post-Process) Revert to HEAP as global variable. Then allow method calls in expressions. Then post process to eliminate them before final printing.

(Intermediate Language) Have an intermediate language which has expression “blocks”. Like { x = y; x+1 }. The if is a unit block can translate directly. Otherwise need something else.

(Three-Address Code). Convert everything into three address code. Yuk!

Other suggestions:

(Post-Process) Revert to HEAP as global variable. Then allow method calls in expressions. Then post process to eliminate them before final printing.

(Intermediate Language) Have an intermediate language which has expression “blocks”. Like { x = y; x+1 }. The if is a unit block can translate directly. Otherwise need something else.

(Three-Address Code). Convert everything into three address code. Yuk!

Yet another option:

Replace both function and method invocations with procedures, and use CALL.  This offers some serious benefits if it can work (e.g. #21)

Yet another option:

Replace both function and method invocations with procedures, and use CALL.  This offers some serious benefits if it can work (e.g. #21)

Here is one plan

Translate Expressions as Blocks.  That is a sequence of statements followed by an expression.
Translate Loops as Unstructured Control-Flow.  That is, using if and goto statements.

This kind of works, though I don't know how to address the use of function calls within specifications.

Here is one plan

Translate Expressions as Blocks.  That is a sequence of statements followed by an expression.
Translate Loops as Unstructured Control-Flow.  That is, using if and goto statements.

This kind of works, though I don't know how to address the use of function calls within specifications.

The easiest solution would be to ban side-effects (hence method calls) within expressions.
But this might mean that a few existing programs (1%?  2%?) need to be restructured to avoid this.
Otherwise, I think the preprocessing approach looks promising.  It is easy to pull out method calls to just before the expression and assign the result to a temporary variable -- except in the case where that method call appears after a short-cut logical operator, such as the example above:  if v >= 0 && m(v): ...
I think this can be addressed by either:

expanding such short-cut operators into control flow logic, using rules like:
if A || B: T else: F  ==>  if A: T else { if B: T else: F }
if A && B: T else: F ==> if A: { if B: T else: F } else: F
so that all method calls in any resulting expression must be called, so can be called before that expression.
or
moving the method call before the expression, but making its call conditional on the window inference context?
E.g. translate:  if v >= 0 && m(v): ...
to:
bool tmp = false;  // any default value
if v >= 0:
tmp = m(v);
if v >= 0 && tmp: ...

The easiest solution would be to ban side-effects (hence method calls) within expressions.
But this might mean that a few existing programs (1%?  2%?) need to be restructured to avoid this.
Otherwise, I think the preprocessing approach looks promising.  It is easy to pull out method calls to just before the expression and assign the result to a temporary variable -- except in the case where that method call appears after a short-cut logical operator, such as the example above:  if v >= 0 && m(v): ...
I think this can be addressed by either:

expanding such short-cut operators into control flow logic, using rules like:
if A || B: T else: F  ==>  if A: T else { if B: T else: F }
if A && B: T else: F ==> if A: { if B: T else: F } else: F
so that all method calls in any resulting expression must be called, so can be called before that expression.
or
moving the method call before the expression, but making its call conditional on the window inference context?
E.g. translate:  if v >= 0 && m(v): ...
to:
bool tmp = false;  // any default value
if v >= 0:
tmp = m(v);
if v >= 0 && tmp: ...

Hey Mark,
It's good to discuss all this, since everything is up in the air at the moment.  For me to fully solve this issue, and a number of other issues (e.g. #21) I think some big changes are required.  It's frustrating since I need to change up the code more.
Let's start with this:

The easiest solution would be to ban side-effects (hence method calls) within expressions.

Yes, that would be easiest solution by a landslide and that is what Dafny does.  But, in my opinion, it is absolutely the wrong solution (and the more I think on this, the more certain of it I become).  The whole point of a language like Whiley is to provide an easy-to-use interface for verification.  We want the language to be driven by users, not by the underlying tools (e.g. Boogie in this case).
Fundamentally, there is no reason for disallowing method calls within expressions.  We know this, since there is a (relatively clear) translation in Boogie.  Furthermore, everyday programmers call methods from expressions all the time, and they will expect this.  See this stack overflow question as just one example.

Hey Mark,
It's good to discuss all this, since everything is up in the air at the moment.  For me to fully solve this issue, and a number of other issues (e.g. #21) I think some big changes are required.  It's frustrating since I need to change up the code more.
Let's start with this:

The easiest solution would be to ban side-effects (hence method calls) within expressions.

Yes, that would be easiest solution by a landslide and that is what Dafny does.  But, in my opinion, it is absolutely the wrong solution (and the more I think on this, the more certain of it I become).  The whole point of a language like Whiley is to provide an easy-to-use interface for verification.  We want the language to be driven by users, not by the underlying tools (e.g. Boogie in this case).
Fundamentally, there is no reason for disallowing method calls within expressions.  We know this, since there is a (relatively clear) translation in Boogie.  Furthermore, everyday programmers call methods from expressions all the time, and they will expect this.  See this stack overflow question as just one example.

Right, second:

Otherwise, I think the preprocessing approach looks promising

Yeah, I just cannot decide what is the best approach.  Your option (2) is one I am seriously considering.  But, in the end, I am wondering whether there is actually a solution where we don't pull method calls out at all.  We can treat them as uninterpreted functions as we do for Whiley functions.  The problem is that they modify the HEAP and we would need to thread this through.  I'm trying to think if I can make that work.  Hmmmmm, maybe but I don't know if it is worth it.

Right, second:

Otherwise, I think the preprocessing approach looks promising

Yeah, I just cannot decide what is the best approach.  Your option (2) is one I am seriously considering.  But, in the end, I am wondering whether there is actually a solution where we don't pull method calls out at all.  We can treat them as uninterpreted functions as we do for Whiley functions.  The problem is that they modify the HEAP and we would need to thread this through.  I'm trying to think if I can make that work.  Hmmmmm, maybe but I don't know if it is worth it.

I should add that option (2) mandates that loops must now be translated using goto statements rather than Boogie's While statement.  This allows for method calls within loop conditions to be properly handled.  Potentially, you could use While in cases where there are no method calls in the condition.

I should add that option (2) mandates that loops must now be translated using goto statements rather than Boogie's While statement.  This allows for method calls within loop conditions to be properly handled.  Potentially, you could use While in cases where there are no method calls in the condition.

Let me play devil's advocate here for a moment.
I think that concept of 'easy-to-use' is a bit ambiguous.
Which do you mean:
1. easy to verify?   (and sometimes a little harder to write)
2. easy to write?   (and sometimes a little harder to verify)
I think the answer to that Dafny stack overflow question was quite nice - just add one line of code to avoid the problems of side-effects, and perhaps make the code more readable.  That one line of code seems a small price to pay for making the verification rules (and the explanation of those rules) easy to understand.
Another reason I really like the idea of banning side-effects in expressions is readability - it means that everyone who reads Whiley code can automatically know that all expressions are pure, so never needs to think about the effects of the method calls in the expressions.
Part of the problem here is that in Whiley like in Java/C etc., there is no syntactic difference between function calls and method calls, so when reading some new code, one has to treat every call as suspicious = possibly have side effects.  One could insist on having a warning keyword before any possibly-side-effecting call within expressions, like:
if v >= 0 && call m(v): ...
This would warn readers about the (rare) calls that might have side-effects, and they would know that all other calls are safe.
But I think that always moving such side-effecting calls out into a separate statement makes the control-flow more explicit, and means that we get the global readability advantage of knowing that all expressions are pure.  This might allow more optimisation opportunities too.

Let me play devil's advocate here for a moment.
I think that concept of 'easy-to-use' is a bit ambiguous.
Which do you mean:
1. easy to verify?   (and sometimes a little harder to write)
2. easy to write?   (and sometimes a little harder to verify)
I think the answer to that Dafny stack overflow question was quite nice - just add one line of code to avoid the problems of side-effects, and perhaps make the code more readable.  That one line of code seems a small price to pay for making the verification rules (and the explanation of those rules) easy to understand.
Another reason I really like the idea of banning side-effects in expressions is readability - it means that everyone who reads Whiley code can automatically know that all expressions are pure, so never needs to think about the effects of the method calls in the expressions.
Part of the problem here is that in Whiley like in Java/C etc., there is no syntactic difference between function calls and method calls, so when reading some new code, one has to treat every call as suspicious = possibly have side effects.  One could insist on having a warning keyword before any possibly-side-effecting call within expressions, like:
if v >= 0 && call m(v): ...
This would warn readers about the (rare) calls that might have side-effects, and they would know that all other calls are safe.
But I think that always moving such side-effecting calls out into a separate statement makes the control-flow more explicit, and means that we get the global readability advantage of knowing that all expressions are pure.  This might allow more optimisation opportunities too.

Hey Mark,

Which do you mean:

easy to verify? (and sometimes a little harder to write)
easy to write? (and sometimes a little harder to verify)

I think we should go with: what will be familiar to everyday programmers.  That is my goal: "verification for the masses"

I think the answer to that Dafny stack overflow question was quite nice

Right, but I thought it was horrific.  Think about it more.  What do we do when calling a method from a loop condition?  Also, how to we chain method calls together, like x.m1().m2().m3().  Remember, we've going to be interoperating with objects from other languages (like Java) where there are a lot of methods.  Already I'm encountering this when representing the JavaScript DOM.  Somethings I can treat as pure, but other things I cannot.  The example above which started this thread arose from me writing code for a web application.  See my current DOM model to witness all the methods!

it means that everyone who reads Whiley code can automatically know that all expressions are pure

Well, I get it and I'm more supportive of this.  For example, we could have a syntax like m!() for method calls (Rust uses this syntax for calling macros).  However, at the same time, I don't think it's necessary.  When calling a function or a method, to understand what it does you need to consult its spec.  It's just a given.  At least in Whiley, if you do this you'll see it's declared a method which immediately tells you it can have side effects.  Ideally, we have syntax which allows us to severely constrain those side-effects as well.

Hey Mark,

Which do you mean:

easy to verify? (and sometimes a little harder to write)
easy to write? (and sometimes a little harder to verify)

I think we should go with: what will be familiar to everyday programmers.  That is my goal: "verification for the masses"

I think the answer to that Dafny stack overflow question was quite nice

Right, but I thought it was horrific.  Think about it more.  What do we do when calling a method from a loop condition?  Also, how to we chain method calls together, like x.m1().m2().m3().  Remember, we've going to be interoperating with objects from other languages (like Java) where there are a lot of methods.  Already I'm encountering this when representing the JavaScript DOM.  Somethings I can treat as pure, but other things I cannot.  The example above which started this thread arose from me writing code for a web application.  See my current DOM model to witness all the methods!

it means that everyone who reads Whiley code can automatically know that all expressions are pure

Well, I get it and I'm more supportive of this.  For example, we could have a syntax like m!() for method calls (Rust uses this syntax for calling macros).  However, at the same time, I don't think it's necessary.  When calling a function or a method, to understand what it does you need to consult its spec.  It's just a given.  At least in Whiley, if you do this you'll see it's declared a method which immediately tells you it can have side effects.  Ideally, we have syntax which allows us to severely constrain those side-effects as well.

Closing this as subsumed by #74


/nllg counts weirdly

I'm in 18 groups:
/nllg or /nllg 0 gives me 'page 1 of 0', showing 10 groups
/nllg 1 gives me 'page 2 of 1', showing 9 groups (including a duplicate of the last group)
/nllg 2 gives me 'page 3 of 2', though it doesn't list any groups
Any higher values give me 'page 3 of 2' and no groups
Confirmed with a friend who has less groups and it's the same for him, though his stops at '2 of 1', rather than '3 of 2'. It also lists our last group twice, for some reason, though the group in question is different for both of us.
Seemingly just a cosmetic issue though, so no hurry.

Was the duplicate group ever merged?

In my friend's case, his last group is 'nixsn', which was merged (specifically with chansnitch, with chansnitch being the group to be gone).
In my case, I'm not sure about the status of TreeKuns, as I was added to it by a friend who I can't get in touch with right now.
As a test, I created the group 'xx' and it showed up at the end of my list with TreeKuns still being duplicated. I then merged 'containment' into 'xx' and checked my groups, but it only had the one entry, with TreeKuns still having two.

I guess I'll see if it changes after the next server restart.

The Page 2 of 1 is just an index issue. I noted it during my last review but forgot to go back and fix it.
The duplicate just means I'll have to add dedup to the output; it's because of a merge and a bungled backend fix for an unrelated issue; I can look into it some more if it bugs you but it's safe to ignore.

Nah, was just making sure it was noted somewhere.

Perfect, thank you!

Fixed as part of https://github.com/Civcraft/NameLayer/pull/191


Initalize default Page Output Cache Provider

Closes #2577

Summary
Initializes default value for host setting 'Page Output Cache Provider' during install

No need to implement.
The host setting has no use.


Show excalidraw image

Closing in favor of https://github.com/dvargas92495/roamjs-presentation/issues/6


pull request for e2e test at 02/01/2017 09:45:12

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


beforeRouteEnter does not working

Version
3.0.1
Reproduction link
https://codepen.io/ROVER_K/project/editor/ZrMzOx
Node and OS info
Node v10.9.0 /  macOS 10.13.6
Steps to reproduce

install @vue/cli v3.0.1

2.vue create pro  with  vue-router, ts,scss,babel
3.just add  these simple code in default About.vue file
<script lang="ts">
import { Component, Vue } from 'vue-property-decorator';

@Component
export default class About extends Vue {
  // here I try to fire this function automatic when come in this component
  beforeRouteEnter (to, from, next) {
    console.log('tssssss223')
    next();
  }
}
</script>

What is expected?
I create a new project with vue-router by vue-cli v3 . And I 'm try beforeRouteEnter function and I found it does not fired. I just add simple codes on default project... So I am sure it's not other reason.. So is that any wrong with my code or my way to use it?
What is actually happening?
The beforeRouteEnter  does not fired,either beforeRouteEnter.

Am I use this function wrong?
I just follow the  guide and I have no idea about
My first aim is pass the params between the components.But I found this problem,so I create a new default project with VUE-CLI 3, and reproduce this problem again .
Thanks for your reading,wating for your reply~~
Best regards

I'm not sure it's related to vue-cli. Also, GitHub issues are only for bug reports and feature requests. If you need help, please ask on the forums or the discord.


Support interface for state

Would be great to be able to define state using Class interface instead of having value like Map<String, dynamic>.

I'm not sure how using a Class interface would work given every user of the package would have their own class implementation and there are those that may not exactly want their data to be in a class.
It would be better for you, if you want to use a class, to have a fromJson() and toJson() implementation within your class.
If you have figured out how a class interface could be incorporated, kindly raise a PR and we'll review. In the meantime, I'm gonna shelf this.

Similar to #276


youtube-dl

Can not down load YouTube video. When trying to update youtube-dl it shows " Can't find the current version."

#30839 !


Nordic theme for Beeper

This theme applies the Nord color palette (https://www.nordtheme.com/docs/colors-and-palettes) to the default Beeper theme.
It is heavily inspired by the Nordic GTK theme from @EliverLara (https://github.com/EliverLara/Nordic).

I LOVE v1 so far.  I'd love to be able to theme based on the channel. Maybe I'll try to add some code to apply the CSS based on a chat network variable.

Sorry for the delay, I had my notifications disabled for this repo!


Error: ambiguous identifier: 'Time'

zip-0.2.1/zip/libzip.nim(71, 13) Error: ambiguous identifier: 'Time' -- use one of the following:
  posix.Time: Time
  times.Time: Time

Code imports posix and times.
https://github.com/nim-lang/zip/blob/master/zip/libzip.nim#L71

Workaround
import posix except Time  # On the code that uses zip

:slightly_smiling_face:

PR?

This seems like a breaking Nim change though. @Araq?

@dom96 Looks to me the compiler previously accepted wrong code and the compiler got fixed.

I dunno, need to investigate.


Moving api reference to api folder

I've noticed that we have an empty api folder, and the api reference is inside reference folder. We can merge both into the api folder, if you see it right @dcfidalgo

hm, i don't see the api folder: https://github.com/recognai/rubrix/tree/master/docs
But yes, let's put all future content of the Reference section simply in the reference folder.

It seems that empty folders are not uploaded into git. I will just erase it in my local environment, and once y make a merge, it will be gone forever. I'm closing this, thank you!


Migrate types to alloy

Is there any plan to do this?

Partially closed by #15, but still waiting to remove reth_primitives in favor of alloy when they are ready.

Actually, since we rely on types such as PooledTransactionsElement, we'll need reth_primitives as these won't be migrated over to Alloy. As such, the issue can be closed.


Installing CLI 2.0 Preview onto macOS 10.12 fails, with OpenSSL-related C compilation error

Attempting to install the CLI 2.0 Preview onto a macOS 10.12 (aka, "macOS Sierra") machine fails.  Specifically, it fails when attempting to build a file, "build/temp.macosx-10.12-intel-2.7/_openssl.c".
The command run to attempt installation was as follows:
pip install --user azure-cli
The error generated was as follows:
cc -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch x86_64 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c build/temp.macosx-10.12-intel-2.7/_openssl.c -o build/temp.macosx-10.12-intel-2.7/build/temp.macosx-10.12-intel-2.7/_openssl.o

build/temp.macosx-10.12-intel-2.7/_openssl.c:434:10: fatal error: 'openssl/opensslv.h' file not found

#include <openssl/opensslv.h>

         ^

1 error generated.

error: command 'cc' failed with exit status 1

----------------------------------------
Command /usr/bin/python -c "import setuptools;__file__='/private/var/folders/x0/nmqpprms1ss69gt8yppnxgl40000gn/T/pip-build-davidl/cryptography/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/x0/nmqpprms1ss69gt8yppnxgl40000gn/T/pip-tMqwwB-record/install-record.txt --single-version-externally-managed --user failed with error code 1 in /private/var/folders/x0/nmqpprms1ss69gt8yppnxgl40000gn/T/pip-build-davidl/cryptography
Storing complete log in /Users/davidl/Library/Logs/pip.log

A full copy of the pip log is attached.
Xcode 7.3 is installed on the machine, along with its command-line tools, and is set (via 'xcode-select') to be the primary Xcode.
If I remember correctly, macOS 10.12 changed if or how it installed OpenSSL, in some manner, but I might be mistaken.  I do recall other, open-source libraries running into trouble with macOS 10.12, and OpenSSL.
If need be, I could try installing OpenSSL via brew, or MacPorts.
pip.log.zip

If it helps, I could try installation via a different version of Xcode.  I have both 7.3.1 and 8.1 installed on my machine, and could get other version(s), if needed.

@DavidLudwig Did upgrading Xcode or installing an update to openssl resolve your issue?

Hmmm, switching my system from MacPorts, to Brew, seemed to fix the issue.
Installing the CLI without pip (via the InstallAzureCli shell script) also seemed to work.

Following up with a solution:
xcode-select --install

http://stackoverflow.com/a/19604913


feat: add .devcontainer files

This adds universal devcontainer files for this project, containing deno-only config, so others can quickly get started contributing.

Thanks, but it's too much to maintain.


[Feature Request] Save denoiser settings in rtx.conf

Some games require custom denoiser settings to fix visual bugs or otherwise improve visual quality. An example: FNV, BeamNG.drive, and Skyrim all benefit from using ReBLUR instead of ReLAX. Saving these settings in rtx.conf would be a great help.

This was a conscious decision. The denoiser settings depend on intrinsic characteristics of the renderer and are not meant to be tweaked per game. We often have to modify these settings together with changes to the renderer itself, so we consider them part of the denoiser integration, instead of graphical settings that can be tuned by a mod.
That said, the implication here is that the denoisers (+ default settings) should largely work independent of content. If the denoisers are not working correctly on specific games, that sounds like a bug.
Can we get tickets filed for the IQ issues in those titles?

Currently we have been filing tickets per game, but there hasn't been much progress on them. So I figured a more generalized solution might be better. Your explanation makes sense though.
https://github.com/NVIDIAGameWorks/rtx-remix/issues/277
Here's one example. I believe there were some others in the past, although integrated into other issues.

Thanks for your feedback @Kim2091! It's always helpful to raise issues like this, even if we ultimately decide to handle them differently. We'll focus on the tickets filed for individual games for now, but please keep sending feedback when you have it.