Add support for converting mixed_float16 models to TF_lite

System information

Windows 10
tensorflow==2.4.0 installed via pip

Project details:

notebook
github repository

Command used to run the converter or code if you’re using the Python API
# Convert the model
converter = tf.lite.TFLiteConverter.from_saved_model(tf_models_dir) # path to the SavedModel directory
tflite_model = converter.convert()

# Save the model.
with open('model.tflite', 'wb') as f:
  f.write(tflite_model)

The output from the converter invocation
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
E:\projects\DaKanjiRecognizer\.venv_dev\lib\site-packages\tensorflow\lite\python\convert.py in toco_convert_protos(model_flags_str, toco_flags_str, input_data_str, debug_info_str, enable_mlir_converter)
    209     try:
--> 210       model_str = wrap_toco.wrapped_toco_convert(model_flags_str,
    211                                                  toco_flags_str, input_data_str,

E:\projects\DaKanjiRecognizer\.venv_dev\lib\site-packages\tensorflow\lite\python\wrap_toco.py in wrapped_toco_convert(model_flags_str, toco_flags_str, input_data_str, debug_info_str, enable_mlir_converter)
     31   """Wraps TocoConvert with lazy loader."""
---> 32   return _pywrap_toco_api.TocoConvert(
     33       model_flags_str,

Exception: <unknown>:0: error: loc(callsite(callsite("DaKanjiRecognizer/conv2D_1_input/Conv2D@__inference__wrapped_model_1048891" at "StatefulPartitionedCall@__inference_signature_wrapper_1049439") at "StatefulPartitionedCall")): 'tfl.conv_2d' op operand #0 must be tensor of 32-bit float or QI8 type or QUI8 type or QI16 type values, but got 'tensor<?x64x64x1xf16>'
<unknown>:0: note: loc("StatefulPartitionedCall"): called from
<unknown>:0: error: loc(callsite(callsite("DaKanjiRecognizer/conv2D_2/Conv2D@__inference__wrapped_model_1048891" at "StatefulPartitionedCall@__inference_signature_wrapper_1049439") at "StatefulPartitionedCall")): 'tfl.conv_2d' op operand #0 must be tensor of 32-bit float or QI8 type or QUI8 type or QI16 type values, but got 'tensor<?x31x31x32xf16>'
<unknown>:0: note: loc("StatefulPartitionedCall"): called from
<unknown>:0: error: loc(callsite(callsite("DaKanjiRecognizer/conv2D_3/Conv2D@__inference__wrapped_model_1048891" at "StatefulPartitionedCall@__inference_signature_wrapper_1049439") at "StatefulPartitionedCall")): 'tfl.conv_2d' op operand #0 must be tensor of 32-bit float or QI8 type or QUI8 type or QI16 type values, but got 'tensor<?x14x14x32xf16>'
<unknown>:0: note: loc("StatefulPartitionedCall"): called from
<unknown>:0: error: loc(callsite(callsite("DaKanjiRecognizer/conv2D_4/Conv2D@__inference__wrapped_model_1048891" at "StatefulPartitionedCall@__inference_signature_wrapper_1049439") at "StatefulPartitionedCall")): 'tfl.conv_2d' op operand #0 must be tensor of 32-bit float or QI8 type or QUI8 type or QI16 type values, but got 'tensor<?x6x6x64xf16>'
<unknown>:0: note: loc("StatefulPartitionedCall"): called from

During handling of the above exception, another exception occurred:

ConverterError                            Traceback (most recent call last)
<ipython-input-15-758d3ba4bcca> in <module>
      7 # Convert the model
      8 converter = tf.lite.TFLiteConverter.from_saved_model(tf_models_dir)
----> 9 tflite_model = converter.convert()
     10
     11 # Save the model.

E:\projects\DaKanjiRecognizer\.venv_dev\lib\site-packages\tensorflow\lite\python\lite.py in convert(self)
    737     converter_kwargs.update(quant_mode.converter_flags())
    738
--> 739     result = _convert_saved_model(**converter_kwargs)
    740     calibrate_and_quantize, flags = quant_mode.quantizer_flags()
    741     if calibrate_and_quantize:

E:\projects\DaKanjiRecognizer\.venv_dev\lib\site-packages\tensorflow\lite\python\convert.py in convert_saved_model(saved_model_dir, saved_model_version, saved_model_tags, saved_model_exported_names, **kwargs)
    630     model_flags.saved_model_exported_names.extend(saved_model_exported_names)
    631   toco_flags = build_toco_flags(**kwargs)
--> 632   data = toco_convert_protos(
    633       model_flags.SerializeToString(),
    634       toco_flags.SerializeToString(),

E:\projects\DaKanjiRecognizer\.venv_dev\lib\site-packages\tensorflow\lite\python\convert.py in toco_convert_protos(model_flags_str, toco_flags_str, input_data_str, debug_info_str, enable_mlir_converter)
    214       return model_str
    215     except Exception as e:
--> 216       raise ConverterError(str(e))
    217
    218   if distutils.spawn.find_executable(_toco_from_proto_bin) is None:

ConverterError: <unknown>:0: error: loc(callsite(callsite("DaKanjiRecognizer/conv2D_1_input/Conv2D@__inference__wrapped_model_1048891" at "StatefulPartitionedCall@__inference_signature_wrapper_1049439") at "StatefulPartitionedCall")): 'tfl.conv_2d' op operand #0 must be tensor of 32-bit float or QI8 type or QUI8 type or QI16 type values, but got 'tensor<?x64x64x1xf16>'
<unknown>:0: note: loc("StatefulPartitionedCall"): called from
<unknown>:0: error: loc(callsite(callsite("DaKanjiRecognizer/conv2D_2/Conv2D@__inference__wrapped_model_1048891" at "StatefulPartitionedCall@__inference_signature_wrapper_1049439") at "StatefulPartitionedCall")): 'tfl.conv_2d' op operand #0 must be tensor of 32-bit float or QI8 type or QUI8 type or QI16 type values, but got 'tensor<?x31x31x32xf16>'
<unknown>:0: note: loc("StatefulPartitionedCall"): called from
<unknown>:0: error: loc(callsite(callsite("DaKanjiRecognizer/conv2D_3/Conv2D@__inference__wrapped_model_1048891" at "StatefulPartitionedCall@__inference_signature_wrapper_1049439") at "StatefulPartitionedCall")): 'tfl.conv_2d' op operand #0 must be tensor of 32-bit float or QI8 type or QUI8 type or QI16 type values, but got 'tensor<?x14x14x32xf16>'
<unknown>:0: note: loc("StatefulPartitionedCall"): called from
<unknown>:0: error: loc(callsite(callsite("DaKanjiRecognizer/conv2D_4/Conv2D@__inference__wrapped_model_1048891" at "StatefulPartitionedCall@__inference_signature_wrapper_1049439") at "StatefulPartitionedCall")): 'tfl.conv_2d' op operand #0 must be tensor of 32-bit float or QI8 type or QUI8 type or QI16 type values, but got 'tensor<?x6x6x64xf16>'
<unknown>:0: note: loc("StatefulPartitionedCall"): called from

Also, please include a link to the saved model or GraphDef
There is no TF lite mixed precision model but here is a model trained with default precision.
Any other info / logs
When I train the model using mixed_float16 set with:
physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)

from tensorflow.keras.mixed_precision import experimental as mixed_precision
policy = mixed_precision.Policy('float32')
#policy = mixed_precision.Policy('mixed_float16')      # <-- uncomment this
mixed_precision.set_policy(policy)

the converter crashes. As stated here (if I understand correctly) there is currently no support for this scenario.
If this is not planned to be implemented, are there any workarounds to still use float16 on supported GPUs. So one can still speed up training and is able to train larger models.

Sorry for encountering this. Currently, TFLite do not support mixed float16 models in TF yet.

Sorry for encountering this. Currently, TFLite do not support mixed float16 models in TF yet.

Is there a way to convert tha savedModel to float32 and than convert it to a tf_lite model?

Is there a way to convert tha savedModel to float32 and than convert it to a tf_lite model?

We are running into this issue as well since we've switched from the graph rewrite to the Keras mixed precision API.
@reedwm What is the recommended way to convert a model with dtype_policy="mixed_float16 back to float32 in order to  support conversion to TFLite? I cannot seem to find any official docs on what the recommended workflow is for this case.

Checkout the following minimal example which results in a segfault:
import tensorflow as tf

tf.keras.mixed_precision.set_global_policy("mixed_float16")

img = tf.keras.layers.Input(shape=(224, 224, 3))
x = tf.keras.layers.Conv2D(16, 3)(img)
x = tf.keras.layers.BatchNormalization(momentum=0.7)(x)
model = tf.keras.Model(img, x)

converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()

I never found a solution and I am still interested in it.

I never found a solution and I am still interested in it.

So one workaround that seems to work for us is to clone the model and overwrite the dtype:
model = tf.keras.models.clone_model(
    model,
    clone_function=lambda layer: layer.__class__.from_config(
        {**layer.get_config(), "dtype": "float32"}
    ),
)

However this doesn't play nicely with custom objects and seems like a quite brittle solution.

Thank you!
I will give this a try because using float16 on RTX GPU's is such a big improvement for me.
However I really hope that somebody from the TF team will look at this again.

Is there a way to convert tha savedModel to float32 and than convert it to a TF_lite model?

@lgeiger solution works, but as he said, it is brittle (doesn't work with subclassed models, there is no guarantee a layer's config has a dtype attribute, etc). It will work in many cases though.
Alternatively, the model can be rebuilt as a float32 model. This will work in any case as long as the model can be rebuilt and supports float32. In either case, the weights of the mixed precision model must then be copied to the weights of the float32 model. For example
import tensorflow as tf
import numpy as np

# Build a toy model
def get_model():
  img = tf.keras.layers.Input(shape=(224, 224, 3))
  x = tf.keras.layers.Conv2D(16, 3)(img)
  x = tf.keras.layers.BatchNormalization(momentum=0.7)(x)
  model = tf.keras.Model(img, x)
  return model

# Create a mixed_float16 model
tf.keras.mixed_precision.set_global_policy("mixed_float16")
model = get_model()

# Compile and train model
model.compile('sgd', 'mse')
x = np.random.normal(size=(64, 224, 224, 3))
y = np.random.normal(size=(64, 222,222, 16))
model.fit(x, y)

# Create a float32 model with the same weights as the mixed_float16 model, so
# that it loads into TF Lite
tf.keras.mixed_precision.set_global_policy("float32")
f32_model = get_model()
f32_model.set_weights(model.get_weights())

# Load model into TF lite
converter = tf.lite.TFLiteConverter.from_keras_model(f32_model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()

This strategy works for any case where a model is trained in mixed precision but a float32 model is needed for inference.

And is it planned to support mixed_float16 for TF_lite models?

@abattery can you answer this? Not supporting mixed_float16 for TF lite is a significant usability issue. I think this should be done in TF lite (and any other inference backend), but an alternative would be to have a tool to convert a Keras model or SavedModel to float32.

Checkout the following minimal example which results in a segfault:

This raises an exception for me (probably was fixed recently). Better than a segfault, but we should still support this use case.

@reedwm Thanks for reminding us of the feature request for mixed_float16 for TFLite.
CCing @talumbau

@reedwm thank you very much for the code snippet!!
Now I am able to train with float 16 precision but still save a tflite model.

@talumbau @abattery Has there been any progress on this issue?
Since the deprecation of the grappler mixed precision training workflow people using the new recommended mixed precision training with Keras are not able to convert models to TFLite anymore without additional code. This is very problematic if the workflow relies on storing saved models which are expected to be reloadable without any code but unfortunately cannot be easily converted to float32 to workaround this issue.

@abattery After a training a model with mixed precision why is not possible to save it with float16 quantization flags enabled?
# Convert the model
converter = tf.lite.TFLiteConverter.from_keras_model(eff)

# set quantization, additional ops
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_types = [tf.float16]

What is the difference between a float16 quantized model and with mixed_precision trained model?

The issue still persists with TF 2.11 and TF Nightly 2.12.0-dev20230106 as well. Please find the gist here. Thank you.

Hi All, @CaptainDario, this should now work (with fallback) using mixed-precision quantization: gist

@pkgoogle thanks for the info. I have currently no setup to test that.

Hi @CaptainDario, the gist should be self-contained, let me know if you don't believe it is a resolution to your issue.

Yes the gist looks like what I wanted. Thanks !

@CaptainDario, great! Can you close as completed if you have no more open items :). Thanks!

I tried all the above-mentioned solutions, but none of them is working properly.
The attempt to convert the precision to float32 by changing the policy and setting weights did not work, and the weight remained in the same policy.
tf.keras.mixed_precision.set_global_policy("float32") f32_model = ghost_model f32_model.set_weights(ghost_model.get_weights()) for layer in f32_model.layers:     print(layer.name, layer.trainable, layer.dtype, layer.dtype_policy)
but did not work.
batch_normalization_75 True float32 <Policy "mixed_float16">
activation_43 True float32 <Policy "mixed_float16">
depthwise_conv2d_39 True float32 <Policy "mixed_float16">
batch_normalization_76 True float32 <Policy "mixed_float16">
activation_44 True float32 <Policy "mixed_float16">
concatenate_30 True float32 <Policy "mixed_float16">
global_average_pooling2d_6 True float32 <Policy "mixed_float16">


Change .gitmodules to use https instead of git

When I try cloning this repo, the submodules will fail due to the git SSH settings.
 git clone --recurse-submodules https://github.com/shuai132/asio_net.git
Cloning into 'asio_net'...
remote: Enumerating objects: 1004, done.
remote: Counting objects: 100% (83/83), done.
remote: Compressing objects: 100% (42/42), done.
Receiving objects:  96% (964/1004)sed 58 (delta 41), pack-reused 921
Receiving objects: 100% (1004/1004), 139.83 KiB | 11.65 MiB/s, done.
Resolving deltas: 100% (689/689), done.
Submodule 'include/asio_net/rpc_core' (git@github.com:shuai132/rpc_core.git) registered for path 'include/asio_net/rpc_core'
Cloning into 'C:/Users/me/Downloads/asio_net/include/asio_net/rpc_core'...
<EMAIL_ADDRESS>Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:shuai132/rpc_core.git' into submodule path 'C:/Users/me/Downloads/asio_net/include/asio_net/rpc_core' failed
Failed to clone 'include/asio_net/rpc_core'. Retry scheduled
Cloning into 'C:/Users/me/Downloads/asio_net/include/asio_net/rpc_core'...
<EMAIL_ADDRESS>Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:shuai132/rpc_core.git' into submodule path 'C:/Users/me/Downloads/asio_net/include/asio_net/rpc_core' failed
Failed to clone 'include/asio_net/rpc_core' a second time, aborting

this is due to the fact that you added the submodule using the git protocol.
[submodule "include/asio_net/rpc_core"]
	path = include/asio_net/rpc_core
	url = git@github.com:shuai132/rpc_core.git

In order to fix this I forked the repo (https://github.com/piperoc/asio_net) and changed the .gitmodules file to use the https url instead:
[submodule "include/asio_net/rpc_core"]
	path = include/asio_net/rpc_core
	url = https://github.com/shuai132/rpc_core.git

This way everything works.
I needed to do this because in my project I manage all the dependencies with the CMake's FetchContent method.
I would rather use your repo than keep syncing my fork.
Hopefully this will help others running into this issue.
Thanks

You should config ssh settings in your github account.
Then, you can download success I guess.
BTW:
submodule should use ssh, becasue https will be complained by github (github will not support https in submodule soon).
I use https first, and change to ssh after got the notice log.

Thank you , that makes sense.
The issue I had was that I was using FetchContent_Declare to connect to asio_net, like so:
...
FetchContent_Declare(
        asio_net
        GIT_REPOSITORY https://github.com/shuai132/asio_net.git
        GIT_TAG v2.0.1
    )
...

which would start downloading the repo, but eventually got stuck without saying why. I then figured there was a submodule and noticed the git-ssh url. CMake did not like to mix it.
So, I changed that section to use git-ssh:
Therefore, the solution is to use git-ssh all the way (like @shuai132 said)
...
FetchContent_Declare(
        asio_net
        GIT_REPOSITORY git@github.com:shuai132/asio_net.git
        GIT_TAG v2.0.1
    )
...

This will not work until you have the SSH fingerprint recorded in your computer. To do that, I cloned the repo once using the command line
git clone --recurse-submodules git@github.com:shuai132/asio_net.git
Cloning into 'asio_net'...
The authenticity of host 'github.com (<IP_ADDRESS>)' can't be established.
ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'github.com,<IP_ADDRESS>' (ECDSA) to the list of known hosts.

After that, all the FetchContent stuff will work.
Not as easy as https, but that's how it's going to be. I figured I described this in case someone else runs into this issue.
Thank you again for replying.


Fixes Three Elements in Row

Describe the changes you have made in this PR -
This PR Fixes three in a row elements.
Screenshots of the changes (If any) -
I will Update this soon
Note: Please check Allow edits from maintainers. if you would like us to assist in the PR.

Coverage remained the same at 84.279% when pulling 32a27c5d7c18ab8db2a3c22e6b992f3f043614fa on Shivansh2407:patch-38 into 7fac3cd55a6d8e22914aea3ec9a00be59e323987 on CircuitVerse:master.

Please review and merge @satu0king 👍


Clicking on file links in calltree of uncaught exceptions popover does not work

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version: 1.72
OS Version: Windows 10

Steps to Reproduce:

turn on "break onuncaught exceptions" and cause one
clicking on links to source code link on the call tree does not work, even tough it says "ctrl click to follow"

Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.72.0. Please try upgrading to the latest version and checking whether this issue remains.
Happy Coding!

I'll consider this part of https://github.com/microsoft/vscode/issues/34026


WIP: Docs: Cover database APIs

STILL A WORK IN PROGRESS
What was wrong?
Database APIs aren't covered in docs yet
How was it fixed?
Adds docs for database APIs
Cute Animal Picture

cc @carver
@cburgdorf this API is likely to change a lot very soon... so maybe punt on this for a bit to remove the overhead of rewriting this.

@pipermerriam good point. I actually wanted to add and tune the docstrings in the source files but I'll postpone. That said, we could take the PR as is then to just have some sort of database coverage and since we are just pulling in the docs from the source files it will automatically pull in any updates at least for the types that keep their place and name.
I'll just add the cutest animal to the PR and have you decide to either close or merge.

@pipermerriam what's your current thinking here? I actually would like to quickly bring up all API docs that already exist in the code. The effort seems pretty manageable since it's mostly just pointers using autoclass directives. So, even if things get dropped it's not that we waste a lot of time.
Once the API docs are online in some way we can start to improve the actual docstrings in the code which is automatically reflected in the online docs.


ExactPartial

I have this code
        drinks.push(await naf.drink(await naf.order({ kiosk })));

drink has this signature
  async drink(part: Partial<DrinkEntity> = {}): Promise<DrinkEntity> {

problem is, I was able to pass an entity completely unrelated to DrinkEntity
I want it so that it only has keys that are in the Partial, or no keys at all, if it has any keys that aren't in the partial it shouldn't compile.

@xenoterracide can you show a complete minimal example of what you're describing? Partial<T> should already accept only keys from T.

https://stackoverflow.com/q/62207479/206466

Partial<DrinkEntity> was excepting an OrderEntity and that does not have a 100% matching signature.
Opaque, might be a decent answer, but I had the same problem with the Opaque provided in ts-essentials as I described here.

had a problem with mymethod, accessing the properties inside of the method were resulting in a compiletime failure.

the Opaque from ts-fest seems to work, and doesn't require me to also pass a string, but the silly as is still required...

Ok, you're right, Partial<T> forbids non-T keys only for literal objects -- makes sense with the explanation you linked: in other cases you can pass e.g. a subclass instance that has more properties.
interface Foo { a: string; b: string }
// Correctly doesn't compile
const test1: Partial<Foo> = { a: 'a', d: 'd' };
const bar = { a: 'a', d: 'd' };
// Passes
const test2: Partial<Foo> = bar;

On a first look though, it doesn't look like this is possible to be solved with just a type, only with a generic method, like mentioned in the UPDATE section of the answer to your SO question.

yeah... I'm going to add my own answer to use Opaque... but like I said, Opaque on here... has ... issues... I can open a new bug for that, if you want?

Opaque types, at least how they can be done in TS, are intended to be used with explicit casts here and there. But yeah, if you have some problems with them, you can post a new issue (and tag @krzkaczor since he's the author of those)

I think we cannot do anything here
I will close the issue, but if you find a bug, please create a related issue


Migrate Siddhi to use JSR 107 Cache API

Reporter : @suhothayan
Jirra url : https://wso2.org/jira/browse/SIDDHI-9
Description :
to enable siddhi to have a stranded cache API

This is not requested by anyone hence not implementing. Please reopen this if it's useful for any use case.


HDDS-10077. Add hsync metadata to hsync'ed keys in OpenKeyTable as well

What changes were proposed in this pull request?
Currently, only those keys in KeyTable would have metadata HSYNC_CLIENT_ID when those keys have been hsync'ed (and not closed/committed yet). One problem with this is that it makes getExpiredOpenKeys() and listOpenKeys() (HDDS-8830) inefficient by forcing them to look up KeyTable while they could have just used OpenKeyTable solely to determine whether an open key is hsync'ed or not.
Changes

During key hsync(), persist metadata HSYNC_CLIENT_ID to OmKeyInfo in OpenKeyTable as well, in addition to KeyTable. Only write when the client ID changes so it doesn't lead to excessive writes. Ideally, only the first hsync() would lead to a OpenKeyTable write of that key.
During key close(), remove HSYNC_CLIENT_ID from OpenKeyTable if any, so that HSYNC_CLIENT_ID isn't unnecessarily written to the final key (which is useless when the key is committed/finalized.
Adopt the more efficient approach in getExpiredOpenKeys() and listOpenKeys().

What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10077
How was this patch tested?

[x] Existing tests that cover the changes shall pass.
[x] Add new test testKeyMetadata to check the intended behavior.

When a key is newly created, neither OpenKeyTable nor KeyTable should have hsync metadata.
When the key is hsync'ed, both OpenKeyTable and KeyTable should have hsync metadata.
When the key is hsync'ed again, both OpenKeyTable and KeyTable should still have hsync metadata.
When the key is closed, KeyTable entry should no longer have hsync metadata. Key should not exist in OpenKeyTable anymore.
All three types of buckets (LEGACY/OBS/FSO) should behave the same.

@smengcl Thanks for working over this, having few query,

adding metadata while commit will not resolve missing hsync commit for first time till fsync not called by client. Do we have plan to resolve by adding hsync flag while open keyfile ?

Which issue are you talking about? If it is another existing jira then it is not in the scope of this PR.

Related to performance, this interface will not be called in normal path. additionally checking key existence for open key is very fast. Do this provide any measurable performance improvement?

It's not merely checking key existence in KeyTable, it is extra key look up + extra DB value deserialization + metadata check. Deserialization the large value byte array that stores OmKeyInfo is probably more expensive than DB get(seek).

Regarding the test failure with the last commit:
Error:    TestLeaseRecovery.testGetCommittedBlockLengthTimeout:330->closeIgnoringKeyNotFound:98 expected: <KEY_NOT_FOUND> but was: <KEY_UNDER_LEASE_RECOVERY>

After a few hours of debugging, I found the new test testGetCommittedBlockLengthTimeout (added in HDDS-10044) does NOT throw KEY_NOT_FOUND when forceRecovery == false during stream.close(), because the key still exists in OpenKeyTable when the stream is being closed even without this PR's change. (Note when forceRecovery is true, the open key is indeed gone. Thus will throw KEY_NOT_FOUND.)
This specific test wasn't previously failing because HSYNC_CLIENT_ID was NEVER put to OpenKeyTable before this patch, as I have detailed above already. And now with this PR, KEY_UNDER_LEASE_RECOVERY is thrown here:
https://github.com/apache/ozone/blob/a15c9ae1b30267db5000bfa2a307e9b57e601f6d/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java#L256-L262
So far it looks like a test issue rather than the code issue and I have adjusted the test to make it pass. Pls check if this is the desired behavior @ChenSammi .

LGTM

So far it looks like a test issue rather than the code issue and I have adjusted the test to make it pass. Pls check if this is the desired behavior @ChenSammi .

@smengcl , thanks for the finding. The fix is good.

Thanks @sumitagrawl @ChenSammi @jojochuang for reviewing this.


Updates README.rst with svg badges

Coverage remained the same at 96.931% when pulling 99b68b29145778bdff469f7b73d837c3087d6841 on sobolevn:patch-1 into 53398f8727aa153ad957b36aaf7da68be97fc854 on Tivix:master.


Dump & Restore project in scripts

Description
Added commands in manage scripts to dump and restore project from database.

BTW: that part where the insert statement is searched for - if I understand it correctly, it only greps projectname without single quotes.
If there are projects project1 and project10, it would dump both project records from public.project if you try to backup just project1.
The solution i use in the makefile is just quoting the string, so it searches for 'project1'. Maybe even better would be to look for VALUES ('project1',  since the project name is always the first value in the insert statement. just quoting does not handle cases, when there's something like {"description": "this project is exactly like 'project1'. hooray"} in project10's  attributes.

BTW: that part where the insert statement is searched for - if I understand it correctly, it only greps projectname without single quotes.

Fixed

Is this available in the web UI and does it still need to be rolled into the docker build?

These are just command line options. which don't affect backend or frontend in any way. We'll examine options to do this using the UI, but at this point, these script just dump project related data to a single SQL file (and restore this file back to the database).
On linux, you run make dump projectname=<yourprojectname> to create dump.<projectname>.sql file in the ayon-docker directory and then make restore projectname=<yourprojectname> takes the file and pushes it back (original project in the database will be overwritten.
On windows i suspect the syntax is slightly different ./manage.ps1 dump <yourprojectname> and ./manage.ps1 restore <youprojectname> respectively.


Update saliency methods to use a framework-agnostic "core" subpackage, and moves all TF1 functionality to an optional "tf1" subpackage.

Changes in this PR:

Creates two subpackages, "core" and "tf1"
"core" is a framework-agnostic package that re-implements all existing methods (with the exception of GuidedBackprop). Each method calls a model through a user-supplied "call_model_function", which receives the inputs and any auxiliary model arguments, and returns relevant outputs (e.g. gradients for a single output neuron with respect to the input) that are used to calculate feature attributions.
"tf1" is functionally equivalent to the prior version, so any code that uses saliency can simply use saliency.tf1 instead.
To avoid unnecessary dependencies, tensorflow is no longer required as part of install, but is instead listed as an extra in saliency[full] and saliency[tf1].
Adds notebooks Examples_core.ipynb and Examples_tf1.ipynb, demonstrating how to use these subpackages.
Gradient path methods (IG, BlurIG, XRAI) now have a batch_size parameter, which calls the model with multiple steps in a single batch. Default batch_size is 1, which mirrors previous behavior (i.e. each step is a separate model call).
Adds new tests for methods that previously had none (Occlusion, Gradients), as well as additional tests for error handling.

Bugfixes:

Fixed an off-by-one error in Occlusion
Added "scikit-image" as a package dependency

Tests for this change:

Running all tests on a local virtual environment
Running example notebooks and verifying the resulting masks match previous notebook outputs.

Looks good, could you update the PR description and title please?

This also resolves #16. We may want to think about adding more notebook examples for other frameworks in future.


More than one file was found with OS independent path 'lib/armeabi-v7a/libijkplayer.so'

问题描述：
More than one file was found with OS independent path 'lib/armeabi-v7a/libijkplayer.so'
问题机型/系统：
真机 小米note2 Android7.0
GSY依赖版本
使用了以下依赖
implementation('com.shuyu:GSYVideoPlayer:6.0.3') {
    exclude group: "com.android.support"
}
implementation('com.shuyu:gsyVideoPlayer-armv7a:6.0.3') {
    exclude group: "com.android.support"
}
implementation('com.shuyu:gsyVideoPlayer-ex_so:6.0.3') {
    exclude group: "com.android.support"
}
implementation('com.shuyu:GSYVideoPlayer-exo2:6.0.3') {
    exclude group: "com.android.support"
}

Demo中的复现步骤
编译无报错，run的时候报错。

看问题集锦，ndk过滤了解下

implementation 'com.shuyu:GSYVideoPlayer:7.1.1'
implementation 'com.shuyu:gsyVideoPlayer-java:7.1.1'
implementation 'com.shuyu:gsyVideoPlayer-ex_so:7.1.1'
ndk{
abiFilters('armeabi', 'armeabi-v7a')
}
More than one file was found with OS independent path 'lib/armeabi-v7a/libijkplayer.so'


Why is filtering by X-Azure-FDID recommended over X-Forwarded-Host

X-Forwarded-Host is the "recognizable" hostname of Front Door that requests are made to.
However, in the section How do I lock down the access to my backend to only Azure Front Door? the X-Azure-FDID header is used
What is the reason not to recommend X-Forwarded-Host instead?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b45a0b48-4e80-9b04-d4c6-e42cbf534ef5
Version Independent ID: dc80b25f-374d-8775-16e7-ae7d57b4b94f
Content: Azure Front Door - Frequently asked questions
Content Source: articles/frontdoor/front-door-faq.yml
Service: frontdoor
GitHub Login: @duongau
Microsoft Alias: duau

@hansmbakker Thanks for your comment! We will review and provide an update as appropriate.

@hansmbakker , below is the update from the Product group team on this query:
Both mechanisms are acceptable however the value (GUID) sent in X-Azure-FDID is unique to the customers’ AFD instance and is much more difficult to discover.
We are closing this issue for now. If there are further questions regarding this matter, please reply and we will gladly continue the discussion.


Add files via upload

Added Universidad Hispanoamericana domain (uhispano).

@esaenz7 First stage of verification passed, official domain is uh.ac.cr as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision. Proof of domain ownership: https://sa.uh.ac.cr/

@esaenz7 Thank you for your patience while waiting for a review and processing your request. Pull request merged.


Created a footer for about page

#101 Added footer to the about page , please review my pull request.

Well Done!


Google Tag manager in docs pages

@tobifroe we want to track where users may leave the journey in the getting started guide. Therefore I need our tag manager snippet to also be included in all pages under https://unikube.io/docs/
Can you integrate the following snippet just at the top of the  tag:

And the following snippet directly after the opening  tag:

@tobifroe man, it really sucks to integrate the code snippets here. I put them in a doc:
https://docs.google.com/document/d/1OuesTrHY2v7fQuLMPaipFT8DGxQlxB1k7XVTPNesja0/edit?usp=sharing


Which paper did the hbmp model come from?

Describe the Question
Which paper did the hbmp model come from?

From the paper "Natural Language Inference with Hierarchical BiLSTM Max Pooling Architecture".

Okay, thank you


Schema Error trying to Sync Favorite Performers

It looks like stashapp has updated past a point where this function is working.  Below is the entry I see in my stashapp log when attempting to sync favorites.
msg="[Plugin / Userscript Functions] schema version mismatch. schema version is 43, expected 36"
Is there a simple change or workaround I can test for this?

Looks like there was a stashapp release/update last night.  Now the schema is 45.

Just confirming that I am experiencing the same issue (on both local installations, fresh installs, and remote hosts (in case it was endpoint specific... but no dice).

same here, on qnap docker install

This is an error from https://github.com/7dJx1qP/pystashlib, which needs to be updated for current schemas


Support server passwords

This PR adds support for server passwords. The underlying IRC server_spec data structure already supports them, we just add them after the port.

I amended the commit, brackets should now be properly aligned

Hey @Emantor, sorry I did not manage to get this integrated at the time.
Licensing of contributions was one thing I hadn't figured out. The other was me being unhappy with appending passwords to the IRC server value (I have resolved this eventually by introducing a configuration file with a separate property for the password). And unfortunately, our in-person chat at the congress didn't happen.
My apologies for the uncomfortable experience of a PR getting ignored, and thanks for making what I think was the first PR to any of my projects!


Wrong encoding on windows?

Describe the bug
When using the plugin with 'reformat code', UTF-8 characters wrongly get decoded with windows-1250.
To Reproduce

place UTF8 character in module, e.g. →
Run reformat code (ctrl+alt+L)
Character gets changed to â†’ (i.e. the bytestring b'\xe2\x86\x92' gets decoded with windows-1250 instead of utf-8)

Expected behavior
Document should remain utf-8 encoded
Screenshots

Environments (please complete the following information):

IDE: PyCharm Community 2022.3.3
OS: Windows 10
Ruff Version 0.0.254
Plugin version 0.0.9 & 0.0.10

Additional information
Directly applying ruff check module.py on the document does not change the encoding.
Disabling the "Run ruff when reformat code" option stops this behaviour.

I think this is a highly critical bug that makes the plugin completely unusable on Windows.

Could you please release this to marketplace? I think I bumped into this bug also

@lilaboc
I tried to publish it to the market.
but it failed. I will fix the problem soon.
https://github.com/koxudaxi/ruff-pycharm-plugin/actions/runs/4739238071

@lilaboc @ChrisOKay  @KotlinIsland
I just published it !!
https://plugins.jetbrains.com/plugin/20574-ruff/versions/stable/322790

JetBrains is reviewing this update. Please expect it to be approved within two business days of the upload.

Great

Manually installed and it's working fine now. Thanks guys!


Multiple destinations

When sending mail with this API to multiple destinations, only the first received mail.
mailjet-sendemail displays logs correctly with all mails (see below), but in mailjet console, i see only one recipient.
from=no-reply%40eqwall.com&to=monadresse%40free.fr%2C%monadresse%40facebook.com%2C%monadresse%40gmail.com&cc=&bcc=&subject=mail%20subject&text=mail%20content%20de%20test
STATUS: 200
HEADERS: {"server":"nginx","date":"Fri, 24 Apr 2015 06:39:41 GMT","content-type":"text/html","content-length":"70","connection":"close","vary":"Accept-Encoding"}
BODY: { "Count" : 1, "Data" : [{ "ID" :<PHONE_NUMBER>0762336 }], "Total" : 1 }

Looks like the send method not create correctly for multiple

Resolved in my fork:
https://github.com/StephaneC/Mailjet-sendemail
Code to ugly for now to ask a Pull Request


Add Zkverse grant application

Project Abstract
There are three main tasks for this project:
integrate some zkp libray(like bellman, arkworks and eg...) into substrate pallet
Maximize the generation efficiency and convenience of zk proof(The proof is not generated on the chain, but through some developer friendly libraries(like snarkjs) to adapt the zk lib on substrate chain.)
Publish some tutorial blogs to let more substrate community developers enjoy the convenience of the above zk development
Also, this project mainly focus on privacy function with ZKP technology. Meanwhile, We know that snakjs and circom are excellent and popular zkp development libraries. They are very popular in the Ethereum ecosystem and can automatically generate verification contract codes. We observed this very good feature, so we want to write circuit codes more easily through snakjs, and generate proofs for verification on the substrate-based chain. Because there are many people using snakjs, and circom is developer friendly, Unlike some domain-specific circuit writing methods, our project can attract many developers who are familiar with snarkjs to develop zkp Dapps in substrate/polkadot ecosystem.
Grant level

[ ] Level 1:  Up to $10,000, 2 approvals
[x] Level 2:  Up to $30,000, 3 approvals
[ ] Level 3:  Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

[x] The application template has been copied and aptly renamed (project_name.md).
[x] I have read the application guidelines.
[x] Payment details have been provided (bank details via email or BTC, Ethereum (USDC/DAI) or Polkadot/Kusama (USDT) address in the application).
[x] The software delivered for this grant will be released under an open-source license specified in the application.
[x] The initial PR contains only one commit (squash and force-push if needed).
[x] The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Validator type improvements

Replaces type_match validator class with validate_type, which takes a single arg of the expected type
Removes type from validatable field definition
Reimplements type_match as comparing the types of the rikolti and comparison/solr value
General clean up and improvement of type comparison behavior

@amywieliczka @timfrazee I don’t know exactly what the ask was here, but here are my observations after reading through the changes in this PR:

It looks like the new verify_type() function could replace type_match()? If so, we'd just need to update the sdasm_mapper to use verify_type(). We could then also remove both the type_match() and derive_type() function, or keep both with a note that they are stubs.
The signature for add_validatable_field() has changed. There are some subclassed Validators that need to be updated accordingly: sppl_mapper, sdasm_mapper, flickr_mapper.
add_validatable_fields() is a new function as of this PR, but is already used in the cca_vault_mapper. I guess this means that Lucas's PR must have merged in this PR?

@barbarahui

Yes, verify_type replaces type_match. I refactored type_match to perform a different job - it now compares the type of the Rikolti data against the type of the Solr data. This is not currently used anywhere, and could be removed if it's not useful. But it was easy to implement and seemed like it might be desirable if we were doing a lax content comparison (which casts values to strings) content comparison but still wanted to make sure the types matched.
Yes, but only to remove type, which is no longer part of the validatable field definition, but instead part of the validation callable itself. I've updated the Validator subclasses that were present on the branch as of my last rebase. Before merging, we'd definitely want to rebase on main and update any others that have been introduced since then.
In this branch, I updated cca_vault_mapper. If it's present on another branch, then I'd assume it was based on this one.

@timfrazee Cool, then it looks good to me, pending making sure that subclasses are updated. Also pending @amywieliczka taking a look I think!

I count 3 direct subclasses of Validator, and 2 subclasses of FlickrValidator.

Completed this work & merging it in!


Website: Manual should be more prominent in left sidebar

It took me a while to find the website's API Separation page today. In hindsight, this is understandable since less important things come higher in the sidebar to the left.
It may be difficult to fully sort the sidebar's elements by importance, but I believe it can easily be improved. Specifically, while facilitating contributions is critical, the manual should surely be above the FOR CONTRIBUTORS section.

The team has debated this for quite some time. At least one PMC member wants to rewrite the whole manual from scratch. Personally, I think the technology we are using causes the majority of the problems.

In the meantime, we could actually move the contributors section down.
I also moved Javadoc, Runtime Deps, Support and Thanks to the "Project Information" section, as I don't think this is relevant except for a few.

Fixed by several changes implemented for #2535.

I (still) think that the manual should be sorted before the Support and Articles and Tutorials pages, but I agree that this is mostly solved.
Thanks @vy


add Synchronize operator to toolkit

Good catch. Fixed.

com.ibm.streamsx.plumbing/com.ibm.streamsx.plumbing.synchronize/.namespace appears to be an empty file. Is its commit intentional?


Azure Cosmos Reactor driver not honoring take while using with byPage

Hi,
Am using the latest version of Azure Cosmos reactor driver and while using byPage and take together, take is not honoring the overall limit.
For Example: if there are 4 records and if below is my code
CosmosPagedFlux response  =  asyncContainer.readAllItems(new PartitionKey("someKey"), Object.class);
response.byPage(2).take(3,true).log().subscribeWith(subscriber);
Am still receiving 4 records in output, instead of 3.  On further debugging noticed that take is being applied to each page result instead of the complete result
For ex: if  the above code is changed as below
CosmosPagedFlux response  =  asyncContainer.readAllItems(new PartitionKey("someKey"), Object.class);
response.byPage(2).take(1,true).log().subscribeWith(subscriber);
Am getting 2 records as part of final result

Scenario:
Lets say for a given PK we have around 200 records, and the client has requested 120 records out of it.
If we like to apply pagination logic to fetch the records say around 50 for each fetch and limit the overall limit to 120. Since limit in the above case is applying to individual Page Flux unable to limit the overall result count to 120.
response.byPage(50).take(120,true).log().subscribeWith(subscriber); -- Returns all 200 records

Will consider how this requirement can be met (either with feature or via samples) in next semester planning.

Closing this because of inactivity, @saivikas-dharma if the issue still persists, please re-open this issue.


Unresponsive to SIGTERM.

Run Cage in a terminal
Observe that Ctrl+C has no effect

Cage should clean up and exit on SIGTERM if possible.

This works for me and is implemented: https://github.com/Hjdskes/cage/blob/master/cage.c#L92.
Sometimes there's a delay, but I suppose that is the application within Cage shutting down (or crashing), since the display terminates. We should probably properly shut down the applications within Cage, if this is possible. What application are you noticing this with?

firefox-wayland. I gave it about 10 seconds or so but it seems to ignore my requests.

static int
handle_signal(int signal, void *data)
{
	struct wl_display *display = data;

	switch (signal) {
	case SIGINT:
		/* Fallthrough */
case SIGTERM:
		wl_display_terminate(display);
+		exit(0);
-		return 0;
	default:
		return 1;
	}
}

This fixes it for me, but I suppose the int returned from this function is supposed to be used by Wayland as an exit code anyway, right?

This also makes Firefox close instantly for me:
diff --git a/cage.c b/cage.c
index 7f5b6f4..cf8c71f 100644
--- a/cage.c
+++ b/cage.c
@@ -48,6 +48,9 @@ spawn_primary_client(char *argv[], pid_t *pid_out)
 {
        pid_t pid = fork();
        if (pid == 0) {
+               sigset_t set;
+               sigemptyset(&set);
+               sigprocmask(SIG_SETMASK, &set, NULL);
                execvp(argv[0], argv);
                _exit(1);
        } else if (pid == -1) {

However, Cage does segfault on close now. Still looking into that, but it'd be great if you could test the above.

However, Cage does segfault on close now.

Scratch that, it did that before. If the above works for you, I'll get that in and open a new issue for the segfault.

The patch works :+1:. As you say, there is a segfault on close, but it isn't critical.

Great! I'll push the fix tomorrow.

I've just noticed this is still happening if an application isn't passed. Same unresponsiveness to SIGTERM.

Can you elaborate? What do you mean when you say "an application isn't passed?"
When I launch Cage with Termite and from Termite launch Epiphany, then CTRL-C Cage, everything closes immediately.

master currently isn't building with the latest version of wlroots so I can't retest. I mean just running cage with no parameters.

For what it's worth, personally I keep wlroots at 0.3 and develop against that. It's enough work to develop Cage that I'm not wanting to constantly keep up with every breaking change in wlroots :)

Hm.. you're not supposed to be able to run cage without any application to start. I'll look into that.

I personally prefer sticking to tagged releases of wlroots, but most projects don't seem to do that. I've built master now and it is still an issue.

This is fixed in 848929c88c71ee9bb6c97d0eabc0afe3246113ff.


Support allocating Egress IPs from ExternalIPPools

If users don't specify EgressIP of an Egress, antrea-controller will allocate one IP from the specified ExternalIPPool if it's available.
Based on #2236
For #2128

Codecov Report

Merging #2237 (f12d25c) into main (382cd88) will decrease coverage by 21.96%.
The diff coverage is 0.00%.

:exclamation: Current head f12d25c differs from pull request most recent head ee0754c. Consider uploading reports for the commit ee0754c to get more accurate results

@@             Coverage Diff             @@
##             main    #2237       +/-   ##
===========================================
- Coverage   61.95%   39.98%   -21.97%
===========================================
  Files         276      273        -3
  Lines       21178    20642      -536
===========================================
- Hits        13120     8253     -4867
- Misses       6699    11243     +4544
+ Partials     1359     1146      -213

Flag
Coverage Δ

kind-e2e-tests
39.98% <0.00%> (-13.15%)
:arrow_down:

unit-tests
?

Flags with carried forward coverage won't be shown. Click here to find out more.


[Bug]: Standalone editors are not properly rendering the image. Exceptions are thrown.

Package Version
4.3.3
Flutter Version
3.22.3
Platforms
Android
How to reproduce?
Just create a simple standalone cropeditor and try any of the constructors (i tried memory and asset both are throwing the same exceptions).
Just as I am writting this post I have found the problem!! there is indeed a bug.
if i push the route as:
                    Navigator.of(context).push(
                      MaterialPageRoute(
                          builder: (context) => MyCropEditor()),
                    );

it works perfectly, no issues. However, if i push with go_router package then there is a ton of exceptions
                    context.pushNamed('mycropeditor');

Logs (optional)
I/flutter ( 4008): null
D/EGL_emulation( 4008): app_time_stats: avg=259.38ms min=2.97ms max=3936.92ms count=16

════════ Exception caught by rendering library ═════════════════════════════════
Unsupported operation: Infinity or NaN toInt
The relevant error-causing widget was:
════════════════════════════════════════════════════════════════════════════════

════════ Exception caught by scheduler library ═════════════════════════════════
Unsupported operation: Infinity or NaN toInt
════════════════════════════════════════════════════════════════════════════════

════════ Exception caught by rendering library ═════════════════════════════════
Unsupported operation: Infinity or NaN toInt
The relevant error-causing widget was:
════════════════════════════════════════════════════════════════════════════════

════════ Exception caught by rendering library ═════════════════════════════════
Unsupported operation: Infinity or NaN toInt
The relevant error-causing widget was:
════════════════════════════════════════════════════════════════════════════════

════════ Exception caught by rendering library ═════════════════════════════════
Unsupported operation: Infinity or NaN toInt
The relevant error-causing widget was:
════════════════════════════════════════════════════════════════════════════════

Example code (optional)
class MyCropEditor extends StatefulWidget {
          const MyCropEditor({super.key});

          @override
          State<MyCropEditor> createState() => _MyCropEditorState();
        }

        class _MyCropEditorState extends State<MyCropEditor>
            with ExampleHelperState<MyCropEditor> {
          @override
          Widget build(BuildContext context) {
            return CropRotateEditor.asset(
              'assets/sun.png',
              key: editorKey,
              initConfigs: CropRotateEditorInitConfigs(
                onDone: (transformations, fitToScreenFactor) {
                  print('onDone');
                  print(transformations.toMap());
                  print('done');
                },
                onImageEditingComplete: (bytes) async {
                  print('editing complete');
                },
                onCloseEditor: () {
                  print('closing');
                  context.pop();
                },
                theme: Theme.of(context).copyWith(scaffoldBackgroundColor: Colors.red),
                //convertToUint8List: true,
                configs: ProImageEditorConfigs(
                  designMode: platformDesignMode,
                  cropRotateEditorConfigs: const CropRotateEditorConfigs(
                    //roundCropper: true,
                    canChangeAspectRatio: false,
                    initAspectRatio: 1,
                  ),
                ),
              ),
            );
          }
        }

Device Model (optional)
pixel 3

I just tested it, but everything works fine for me, even when I use the go_router package you mention. However, I think the problem is that you are trying to send data with the flutter package flutter_bloc but your provider is below the GoRoute which can't send the data there. To test it, you can replace the editor with the code below. In case it works, the problem is as I described.
CropRotateEditor.network(
          'https://picsum.photos/id/230/2000',
          /// Other code
)

@hm21 I was wrong earlier. even after simplifying the process by removing all bloc and that sort of stuff to try to narrow down the error i ended up with a very simple widget like:
    class MyCropEditor extends StatefulWidget {
      const MyCropEditor({super.key});

      @override
      State<MyCropEditor> createState() => _MyCropEditorState();
    }

    class _MyCropEditorState extends State<MyCropEditor> {
      final editorKey = GlobalKey<ProImageEditorState>();
      @override
      Widget build(BuildContext context) {
        return CropRotateEditor.asset(
          'assets/nature.jpg',
          key: editorKey,
          initConfigs: CropRotateEditorInitConfigs(
            onDone: (transformations, fitToScreenFactor) {
              print('onDone');
              print(transformations.toMap());
              print('done');
            },
            onImageEditingComplete: (bytes) async {
              print('editing complete');
            },
            onCloseEditor: () {
              print('closing');
              context.pop();
            },
            theme: Theme.of(context).copyWith(scaffoldBackgroundColor: Colors.red),
            //convertToUint8List: true,
            configs: ProImageEditorConfigs(
              designMode: platformDesignMode,
              cropRotateEditorConfigs: const CropRotateEditorConfigs(
                //roundCropper: true,
                canChangeAspectRatio: false,
                initAspectRatio: 1,
              ),
            ),
          ),
        );
      }
    }

I have tried with network constructor as you mentioned in your answer and it throws the exact same exceptions.
What i said earlier is incorrect, pushing the Navigator.of -> materialroutepage also doesnt work and throws the same exception as pushing with go_router.
Can you test using pixel 3 with android 13 ("tiramisu")?
I now tested in pixel 8 ("tiramisu") and it works. I believe there is a bug with pixel 3 tiramisu emulator

@hm21 i just cleaned my Build folder and started again and I am not getting errors. Anyways, just to let you know but then I tested in a real device and there was an overflow issue of 1 pixel at the bottom

@hm21 a little more context on the overflow issue but that was my mothers phone and she uses a bigger sized letter maybe that's the issue

If you want to support multiple font sizes in your application, you can replace the bottombar and appbar as you can see in this example. In this case you can change that there is only an icon when your users have a bigger font size or your bottombar will grow.


client API version: 1.20, server API version: 1.19

Whenever I'm trying to docker build i've got this error:
Error response from daemon: client is newer than server (client API version: 1.20, server API version: 1.19)
It seems something is out of sync and dinghy up/destroy/upgrade don't seems to work

Found a couple of reference to the same kind of bug:
https://github.com/kitematic/kitematic/issues/875
Seems to be related to boot2docker but still unsure how to fix it... :(

This means your version of docker-machine is older than your version of docker. If you're using Docker Toolbox you'll have to upgrade that, but if you are just using homebrew to install them, a brew up && brew upgrade docker-machine && dinghy upgrade should do it.
I've been wanting to add some code to dinghy to detect version problems and suggest how to fix, but haven't gotten a chance yet.

super strange, i'm using only homebrew but by upgrading everything it doesn't work.
what did worked althought, i've discovered in /usr/local/var/dinghy/vagrant/VagrantFile the box version was completely out-of-date:
config.vm.box_version = "1.7.1.rev1"

I changed it to the latest version (1.8.2) and now it work. I'm just wondering why this file was not upgraded by doing dinghy upgrade. Any idea ?

Oh, you are on a pretty old version of dinghy, OK. About 2 months ago dinghy switched to using docker-machine and doesn't even use Vagrant anymore, I was referring to that new setup. That makes sense then.
I'd recommend upgrading when you get a chance, I won't be releasing any versions of the Vagrant image newer than 1.8.2.

Ohhh just found https://github.com/codekitchen/dinghy/blob/latest/UPGRADE_FROM_VAGRANT.md :)
Thanks :+1:


Tags are not working properly for xamarin forms ios when I set enableTestSend to false

unable to send push notifications to specific tag on ios when I set enableTestSend to false in production, but if I send empty tag it works fine and also tags are working fine on android.

@usman0910 Can you please be more specific?   Do you get an error message? Does the app not compile?

Closing due to no response.


Can handleRedirectPromise be documented better with Examples? Otherwise can lead to race conditions

Please follow the issue template below. Failure to do so will result in a delay in answering your question.
Library

[ ]<EMAIL_ADDRESS>or<EMAIL_ADDRESS>[X]<EMAIL_ADDRESS>[ ]<EMAIL_ADDRESS>[ ]<EMAIL_ADDRESS>[ ]<EMAIL_ADDRESS>
Documentation location

[ ] docs.microsoft.com
[ ] MSAL.js Github Wiki
[ ] README file
[ ] Other (please fill in)
[X] Documentation does not exist

Description
Hey guys, first of all, I do like to thank you for all the work you guys are doing on the libraries and I understand auth is a hard battle especially when you have so many libraries. Now to my point, I was upgrading from ADAL-angular to msal yesterday for basically a 3 framework TS/JS application but for these purposes, adal/msal was used in a utility file in vanilla JS i.e framework agnostic
All was going mostly well until I ran into handleRedirectPromise and how its behavior of changed/been upgraded from previous angular models(handleWindowCallback)....our old code mostly had the format of:

When library is imported this code ran, this would cause a redirect (we are using redirects not popups)

  const authContext = new AuthenticationContext(context);

  authContext.handleWindowCallback();
  authContext.getCachedUser();
  ... other function...

We exported 3 functions, the important one was getToken() which looked something like this:

  function getToken() {
    const token = authContext.getCachedToken(CLIENT_ID);
    if (!token && !authContext.loginInProgress()) {
      return renewAuthToken(); // (uses loginRedirect)
    } else {
      return Promise.resolve(token);
    }
  }

We didn't have much protection around when this loads because angular-adal was able to handle it quite well, the important part that almost never caused a race condition was authContext.loginInProgress() e.g if we were already trying to auth then don't try to renew again...this is not the most robust logic but it worked for 2 years without any issues.
When we switched to msal I keep most of the same structure but I could no longer check authContext.loginInProgress() since msal-browser no longer had this function...after hitting my head against a wall for a while and reading the documentation more I finally found out what we had to do which was basically only call renewAuthToken (which will use loginRedirect) inside handleRedirectPromise (if resp was null and no user account was found) again we want to do this on onload. (Our getToken function used acquireTokenSilent if user was already logged in)
Before I gave to this realization, I keep running into race conditions because getToken would be called before handleRedirectPromise was fulfilled which led to this cat and mouse game. Finally coming to my point:
We can provide a better example that shows users how to make sure handleRedirectPromise is resolved before they call any other functions, I know in the docs it says you need to await this but doesn't show any examples. I think this is going to very confusing for beginners especially if they are doing it onPageLoad. I do realize that is mostly an application problem and how it is set up but it wouldn't surprise me many people have this setup.
Thanks
Jason

@jaslloyd Have you seen our onPageLoad sample? The redirect flow is definitely confusing and we're always looking for ways to improve it and our documentation. We appreciate the feedback and will try to make this more clear in the docs!

@tnorling Thanks for the reply, yeah I took a look at it and it is very similar to how we are actually doing it. I think the problem comes from even in that example what stops you using async function signIn before handleRedirectPromise? Imagine it is not a user action, lets say they load the page and it automatically called signIn...if they manage to call signin before handleRedirectPromise they will get an error because sigin "beat" handleRedirectPromise.
I know you guys make it clear you need to await handleRedirectPromise but in that example its not really doing that right?

That sample is intended to demonstrate signing in on page load without user interaction. There's nothing stopping a developer from moving the signIn call out of handleRedirectPromise but they do so at risk of breaking the flow we're trying to demonstrate with that sample. To clarify, are you asking for an example that shows await handleRedirectPromise() instead of handleRedirectPromise().then(handleResponse)?

There's nothing stopping a developer from moving the signIn call out of handleRedirectPromise but they do so at risk of breaking the flow we're trying to demonstrate with that sample.

eah, 100% right and this was mostly my point that it leaves a lot to go wrong or can leave the developer scratching their head trying to figure out the race condition.
Maybe there could be an example of it showing handleRedirectPromise being called first then allowing the use of other commands or maybe add a bit more detailed description. I know this is a rather difficult problem to describe :)

We can certainly add some comments to the sample with a warning and expand the docs with a disclaimer but I'm not sure I understand what exactly the ask is here beyond that. We have a sample demonstrating the flow and docs explaining handleRedirectPromise must be called when using redirect flows. If someone ignores our recommendations to make this work they shouldn't be surprised when it doesn't work. If it would be easier for you to submit a PR with what you're envisioning those changes look like, we are always open and more than happy to take a look!


[7.x] Implement env and production Blade directives

I often find myself doing this:
@if (app()->environment('production'))
    <!-- Some tracking code here -->
@endif

This pr allows for:
@production
    <!-- Some tracking code here -->
@endproduction

And also for more granualar environment checking if needed:
@env('staging')
    <!-- Some tracking code here -->
@endenv

The docs have an example of a custom @env directive. The example should probably be changed if this gets merged.

I keep adding this custom directive to my project so that would be very useful! Plus, custom directive breaks the syntax coloring and open/close pairs 😄

Thank you Dries! 👏

I keep adding this custom directive to my project so that would be very useful! Plus, custom directive breaks the syntax coloring and open/close pairs 😄

Thank you Dries! 👏

No, it does not break the syntax coloring if you are using it correctly. You should close it using @endenv.

Thanks everyone. I've sent in a PR to update the existing docs example: https://github.com/laravel/docs/pull/6043


Enable Jfrog CLI to download the latest Maven snapshot just like curl

Today, when using a simple curl the a Maven repo path with -SNAPSHORT suffix we can download the latest deployed artifact.
example:
curl <artifactory_url>/libs-snapshot-local/com/search/foo/1.0/foo-1.0-SNAPSHOT
Jfrog CLI / SPEC does not support this and we are required to develop special workarounds / wrappers around this...
Would appreciate matching those behaviors so we can use the CLI properly
Thanks

+1
Appears this is still an issue in 2.70.0 for downloading unique snapshots.
This works:
wget <artifactory_url>/artifactory/libs-snapshot/com/foobar/modules/foo-client/1.0.0-SNAPSHOT/foo-client-1.0.0-SNAPSHOT.jar

Yet this fails:
jfrog rt download libs-snapshot/com/foobar/modules/foo-client/1.0.0-SNAPSHOT/foo-client-1.0.0-SNAPSHOT.jar --fail-no-op
{
  "status": "success",
  "totals": {
    "success": 0,
    "failure": 0
  },
  "files": []
}
[🚨Error] No errors, but also no files affected (fail-no-op flag).

Searching returns nothing / fails to resolve the latest snapshot stamp from the maven-metadata:
jfrog rt search libs-snapshot/com/foobar/modules/foo-client/1.0.0-SNAPSHOT/foo-client-1.0.0-SNAPSHOT.jar
[🔵Info] Searching artifacts...
[🔵Info] Found 0 artifacts.
[]

Only workaround I've found is to download the maven-metadata.xml and parse it manually with ugly scripting, or just not use jfrog-cli...


Please support newer base-compat-0.14

There's base-compat 0.14 for GHC 9.10. Could you please release/revise with increased upper bounds? See

https://hackage.haskell.org/package/base-compat
https://github.com/commercialhaskell/stackage/issues/7403

@cocreature Would it be possible to make a revision for this bump?


Modify to work with AWS ECS

In ECS the /proc/self/cgroup contains the prefix /ecs/ not /docker/ (likely the issue with other hosts such as kubernetes)
The "aws" fix here is to use the ECS_CONTAINER_METADATA_FILE env var provided by ecs
Im open to modifying it as well to instead take an override for the sed:
CGROUP_PREFIX="${CGROUP_PREFIX:-docker}"
export CONTAINER_ID=$(cat /proc/self/cgroup | sed -nE "s/^.+$CGROUP_PREFIX[\/-]([a-f0-9]{64}).*/\1/p" | head -n 1)

Hi ! Thanks for that PR. I have two remarks:

could you use a parameter substitution instead of removing set -u and testing for a non null variable ? Something like this:

DOCKER_PROVIDER=${DOCKER_PROVIDER:-}
if [[ "${DOCKER_PROVIDER,,}" == "aws" ]]; then

similarly, could you do grep 'ContainerID' directly from the file  rather than disabling SC2002 ?

btw 1) is currently lacking on this line and that line could directly sed from the file as suggested on 2). If you don't mind adding that to the PR that would be neat 👍

@buchdag done

@buchdag makes sense, changed

I got the same issue as described here and the proposed fix here helped me to get this working again. However i got two problems:

ECS_ENABLE_CONTAINER_METADATA is not enabled by default and the proposed change does not warn you about that.
The check for if [[ -n "${DOCKER_PROVIDER:-}" ]] && [[ "${DOCKER_PROVIDER,,}" == "aws" ]]; then did not seem to work and i had to remove that part to get CONTAINER_ID from ECS_CONTAINER_METADATA_FILE

for 1) youre right, but there's no reason to check for ecs.config to do it. If it's not t here it will fail the check for -n and behave the same as if the  code didnt exist which is good behavior
2)  ill look

@myoung34 is it possible that in some case AWS ECS does not set the DOCKER_PROVIDER environment variable ?
Is this DOCKER_PROVIDER variable a standard among container management services or unique to ECS ?
If it's not widely used and/or if it is sometimes not set by ECS, maybe the fix should rather directly test [[ -n "${ECS_CONTAINER_METADATA_FILE}" ]] instead ?

DOCKER_PROVIDER is custom and meant for here to swap between kubernetes, gcp, aws, etc.
ECS_CONTAINER_METADATA_FILE is standard to AWS in ECS but has to be enabled by ecs.config
I ilke the current way because there's no behavior change existing or across cloud providers, but is "opt in" by setting DOCKER_PROVIDER and enabling the use of "cloud lookups"

@buchdag I pushed up what it would look like to test for AWS ECS.
My only complaint would be without "opting in" by telling it which one to use, if there were 15 providers (15 if()'s) it would be a long time for it to go through the failed checks to hit the original method.

@myoung34 do you think a better approach would be to check in /proc/self/cgroup first, then if it returns nothing either use ECS_CONTAINER_METADATA_FILE or a different regex ?
From what @DurinMusicspear said, if ECS_CONTAINER_METADATA_FILE isn't enabled by default the alternative regex seems to be a bit more reliable.
There is a similar proposition for the same issue in docker-gen: https://github.com/jwilder/docker-gen/issues/263

I think the best way to approach it is with an opt-in flag like DOCKER_PROVIDER so that you can be explicit and not hope the trickle through works honestly. Im not a fan of changing default behavior so either that or having the cat /proc happen first would be my choices

Ok I might have misunderstood something from the beginning, the DOCKER_PROVIDER was supposed to be a new user provided environment variable ?

It is, but it acts like an opt-in. not providing it would be no change in current behavior, manually providing it is a way for the user to set behavior, basically saying "i want this to act under ECS" , so if its ECS, use the ECS_CONTAINER_METADATA_FILE var. Then in the future someone can add a block that does something special to get it to work, say, under Kubernetes by saying DOCKER_PROVIDER==k8s then in that block doing whatever it takes to work under kubernetes, etc. Its faster since its explicit, and its opt in meaning that you have to set it manually to get it to behave differently for you (not changing behavior for anyone else currently using it by default)

Got it. I thought DOCKER_PROVIDER was supposed to be set by ECS itself and I'm under the impression that @DurinMusicspear did not understand that it was supposed to be set by the user either.
If you introduce a new environment variable you'll have to document its use in the README.
With all that in mind i think the sed override might be simpler / more reliable. Something like, let's say:
DOCKER_PROVIDER=${DOCKER_PROVIDER:-docker}

case $DOCKER_PROVIDER in
aws|AWS)
    CGROUP_PREFIX=ecs
    ;;
*)
    CGROUP_PREFIX=docker
    ;;
esac

export CONTAINER_ID=$(sed -nE "s/^.+$CGROUP_PREFIX[\/-]([a-f0-9]{64}).*/\1/p" /proc/self/group | head -n 1)

Unless I've missed something, default behavior is unchanged, it is explicit and it work under ECS even if ECS_CONTAINER_METADATA_FILE hasn't been enabled. We could directly set the CGROUP_PREFIX but I think it's better to abstract away what the container really does to get its own ID under different container management services and just have the user set an environment variable with a self explanatory name and value.
If in the future getting the container ID under Kubernetes or another container management service require something more complex than a regexp change, the export CONTAINER_ID can be moved inside the case.
Your thought ?

/proc/cgroup is not a good choice on ECS. it might work, but it might not as well. The container ID is provided first-hand in the environment variable given via the API (ECS_CONTAINER_METADATA_FILE). I dont think using /proc/cgroup is a good idea. I honestly think my first submission is the best, being opt-in (with DOCKER_PROVIDER) then using the recommended method (the env var ECS_CONTAINER_METADATA_FILE)

Ok so let's go for ECS_CONTAINER_METADATA_FILE. There are still a some changes needed for this to be merged:

document the new DOCKER_PROVIDER environment variable use (and clarify the fact that in ECS,
ECS_CONTAINER_METADATA_FILE has to be manually enabled).
use the value ecs rather than aws (you can run containers on AWS without using ECS at all).
use a case rather than if/else, it's cleaner and will be easier for us to maintain in the long term if the plan is to add other container management services.
ideally check in the ecs case that ECS_CONTAINER_METADATA_FILE is set, and  if it's not exit with a message explaining that it needs to be enabled.
no need to check DOCKER_PROVIDER with -n if it is a user set environment variable. Either it's set by the user or it's not, get assigned a default value and is caught by the default case

proto code:
DOCKER_PROVIDER=${DOCKER_PROVIDER:-default} #or docker or whatever default value seems more relevant to you

case $DOCKER_PROVIDER in
ecs|ECS)
    if [[ -z ${ECS_CONTAINER_METADATA_FILE:-} ]]; then
        echo instructions
        exit 1
    fi
    export CONTAINER_ID=$(ecs method)
    ;;
*)
    export CONTAINER_ID=$(default method)
    ;;
esac

If you're worried about speed, it does not make any difference whether you have two patterns in your case statement or several thousands (tested for real with time and script-generated case).

Hi, I'm still having the "Error: can't get my container ID !"
i'm following your instructions about "Separate containers" on https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion
I'm trying to mount that on AWS ECS Task Definition with those 3 containers.
I'm able to do it manually, but the cool thing it's to do it on the console.
Can anyone tell me if i'm missing some new instruction?
thanks a lot

@sunomie I also started getting the same issue with the latest ECS agent (1.21.0) however previous versions work as per the documentation. I was able to fix it on the latest agent by removing the DOCKER_PROVIDER environment variable completely. It looks like the prefix in /proc/self/cgroup is now /docker/ and not /ecs/ like in the previous versions.

Concur.  Was also able to fix it on the latest agent by removing the
DOCKER_PROVIDER environment variable completely.  Thank you.
On Tue, Oct 23, 2018 at 5:00 AM Stephen Law<EMAIL_ADDRESS>wrote:

@sunomie https://github.com/sunomie I also started getting the same
issue with the latest ECS agent (1.21.0) however previous versions work as
per the documentation. I was able to fix it on the latest agent by removing
the DOCKER_PROVIDER environment variable completely. It looks like the
prefix in /proc/self/cgroup is now /docker/ and not /ecs/ like in the
previous versions.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion/pull/300#issuecomment-432160364,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAaK7_6nYJeKMLlfT5diihg-KkqnWRFpks5untqogaJpZM4Q-5rT
.

--
Greg

This self container ID issue with AWS ECS bugs me.
The sole purpose of getting the LE container ID are a configuration check during container startup and the ability to automatically get the nginx-proxy container ID with --volumes-from.
None of them are actually required to correctly run the container.
I'm a newbie with ECS but I guess the --volumes-from flag of docker run does not even make sense in this context ?


Revise And Review Stage 1 portion of the script

Nothing was changed. Since, Stage 1 is mostly getting player known with the gameplay, there was already some thought put into it. But will edit things in in the future, if mechanics demands it.


Reliable MIME type detection for essential file types

Motivation:
HttpVfs.AbstractEntry relies on URLConnection.guessContentTypeFromName()
to determine the MIME type of an entry.
It seems like OpenJDK as well as other JREs do not return the correct
MIME types for some widely-used file extensions such as .css and .js.
Modern browsers warns about missing/incorrect 'Content-Type' of a file.
Some browsers such as Internet Explorer even refuse to load it.
Modification:

Add MimeTypeUtil to provide more reliable MIME type detection for the
essential file extensions
Set the content type of 'specification.json' to application/json in
DocService
Add more constructors to the HttpVfs.Entry implementations so that a
user can specify the content type explicitly

Result:

DocService works fine on Internet Explorer
Chrome does not complain about missing/incorrect content type for .css

@delegacy PTAL (because you are the front-end expert among us :-))

Left some minor comments. Please take a look. :smile:

@delegacy Thanks! Addressed and replied. :-)

@delegacy Addressed your comments.

:shipit:


refactor stream handling

motivation: consolidate and clarify which output stream is used throught the code
changes:

reanme all local variables called "stdoutStream" to "outputStream" to differenciate with the global stdoutStream from TSC
audit code for use of stdoutStream and replace with a configurable outputStream so that we can centraliy control the output
clean up diagnostics print code

@swift-ci please smoke test

@swift-ci please smoke test

This makes sense, I particularly like the emitting of diagnostics rather than just stream text.


Using with Docker

I can successfully send messages from a container, but AsyncSniffer does not invoke the started callback, even though AsyncSniffer.running is true after starting (https://github.com/MRIIOT/ESPythoNOW/blob/d6749b17dfc575c8b139468b9f6b088a1db3d758/ESPythoNOW.py#L110-L111).  I will update this issue once we figure out how to receive messages.
https://github.com/MRIIOT/ESPythoNOW/blob/main/Dockerfile
https://github.com/MRIIOT/ESPythoNOW/blob/main/docker-compose.yml

libpcap was missing.  Above works sending and receiving.

Here is a ESPNOW to MQTT relay.
https://github.com/SharcTech/espnow-relay

Not at all, works great.


Highlighting does not work correctly with annotations and public/private modifiers

This chunk of code for example:
export class TemplateMappings {
  @HostBinding('attr.class')
  public class = 'w-100';

  constructor(private tmplMapService: TemplateMappingsService) {}
}

public and attr.class are not highlighted at all.

This actually does appear to work in the latest. I'm not sure how atom installed an outdated version.


Missmatch Sample rate Opus files

Hello,
I saw that sample_rate=16000 in GigaSpeech.Json does not match with the one in opus file SR=48000:
ffmpeg -i /workspace/datasets/GigaSpeech_corpus/audio/podcast/P0001/POD0000000001.opus
ffmpeg version 4.3 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 7.3.0 (crosstool-NG <IP_ADDRESS>9-a04d0)
  configuration: --prefix=/opt/conda --cc=/opt/conda/conda-bld/ffmpeg_1597178665428/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc --disable-doc --disable-openssl --enable-avresample --enable-gnutls --enable-hardcoded-tables --enable-libfreetype --enable-libopenh264 --enable-pic --enable-pthreads --enable-shared --disable-static --enable-version3 --enable-zlib --enable-libmp3lame
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
Input #0, ogg, from '/workspace/datasets/GigaSpeech_corpus/audio/podcast/P0001/POD0000000001.opus':
  Duration: 00:10:29.94, start: 0.000000, bitrate: 32 kb/s
    Stream #0:0: Audio: opus, 48000 Hz, mono, fltp
    Metadata:
      encoder         : Lavc58.54.100 libopus
      artist          : Roman Mars
      date            : 2011
      genre           : Podcast
      title           : 99% Invisible-18- Check Cashing Stores
      album           : 99% Invisible
      encoded_by      : iTunes 9.2.1
At least one output file must be specified

is there any problem ?

Hi aheba,
Opus standard defines a set of band-widths but it doesn't enforce sample rates. So an Opus decoder may have 48k sample rate(many implementation choose 48k as default sample rate) where it actually contains narrowband signal(e.g. 8k).  In GigaSpeech repo's readme, search "resampling", there is a section talking about this, and it may help you through the problem.


Feature/k6han/966242934 multi lang for books api

Administrative Info
Monday Board ID: 853642866
Make sure your branch name conforms to: <feature/staging/hotfix/...>/[username]/[Monday Item ID]-[3-4 word description separated by dashes]. Otherwise, please rename your branch and create a new PR.
Changes
What changes did you make?
Refactored the database to include support for multiple languages, and refactored the routing to work with the new database.
Testing
How did you confirm your changes worked?
I wrote tests in book_test.go in controllers, and verified that the new languages were being read.
Confirmation of Change
Run make test in the backend directory. Open internal/controllers/book_test.go to see the tests.

Just noticed the .DS_Store is still there, can you make sure that gets deleted when you fix conflicts? Thanks!

The code itself looks good, but one small problem. Running it on my laptop, i get the error: failed to migrate: ../../migrations directory does not exist. I think fixing the merge conflict should fix it, because it should have the patch that David added. Once that is fixed, the branch should be g2g

@Anshul-Birla could you test again and let me know if it works? I thought I had fixed the merge conflicts already, my bad on taking so long I had a busy week.

The code itself looks good, but one small problem. Running it on my laptop, i get the error: failed to migrate: ../../migrations directory does not exist. I think fixing the merge conflict should fix it, because it should have the patch that David added. Once that is fixed, the branch should be g2g

@Anshul-Birla could you test again and let me know if it works? I thought I had fixed the merge conflicts already, my bad on taking so long I had a busy week.

It still complains with the same error. Its odd because if I run the backend on main, it works perfectly fine.

@k6han Merge conflicts aren't fixed yet, did you make sure to pull main before merging/rebasing?


Stremio torrentio nyaa

Hello
I noticed that the torrentio addon only fetches torrents with "english translated" subtitles (dropdown list on nyaa site).  As a result, the search is reduced and I cannot find my language (vostfr) in quite a few cases.  On nyaa there may be many more torrents than those displayed due to this filtering limitation.
Is there a way to ask him not to filter? An alternative ? I really need help.
Thank,

Hi,
Yeah, it would be helpful. Like french torrents they use (VOSTFR) and it doesn't detect it due to filtering limitations.

Yeah, that would be awesome, as a spanish speaker there are very few in my language too.

Similar to #133


[CARBONDATA-2792][schema restructure] Create external table fails post schema restructure.

…
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

[X] Any interfaces changed?
NA
[X] Any backward compatibility impacted?
NA
[X] Document update required?
NA
[X] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
added test case to validate the create external table post restructure scenario.
[X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA

Build Failed with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/6311/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/7569/

SDV Build Fail , Please check CI http://<IP_ADDRESS>:8080/job/ApacheSDVTests/6029/

SDV Build Fail , Please check CI http://<IP_ADDRESS>:8080/job/ApacheSDVTests/6037/

retest sdv please

retest this please

retest this please

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/7637/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/6363/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/6379/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/7662/

LGTM

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/6393/

Build Failed  with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/7681/

Build Failed  with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/7686/

SDV Build Fail , Please check CI http://<IP_ADDRESS>:8080/job/ApacheSDVTests/6082/

Build Failed with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/6412/

SDV Build Fail , Please check CI http://<IP_ADDRESS>:8080/job/ApacheSDVTests/6084/

Build Failed  with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/7692/

Build Failed with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/6418/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/7721/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/6447/

SDV Build Fail , Please check CI http://<IP_ADDRESS>:8080/job/ApacheSDVTests/6114/

LGTM


Official Cypress 5.2.0 docker images (cypress/base) contains >=1 security vulnerability (Debian)

Current behavior

The official Cypress docker images (cypress/base) contains >=1 security vulnerability (Debian)
As a result, the images created in our CI pipeline (Azure DevOps based), that depend on cypress/base, also contains the same security vulnerabilities (like cypress/browser, or, cypress/include, or both) - since cypress/browser includes the cypress/base image using the FROM command)
Consequently, the custom Dockerfile that we use, that depends on cypress/include, also exhibits the same vulnerability
The security vulnerabilities show up in Azure Security Center. We assume so far that the vulnerabilities are there for all.

Desired behavior

The official Cypress docker images (cypress/base) should not contain several security vulnerabilities, especially since other Dockerfiles depend on the base image, and it is not straight forward to control the contents of the (cypress/base) Dockerfile
Similarly, the Cypress docker images that depend on cypress/base or cypress/browser, also should not contain security vulnerabilities (cypress/include) or images that depend on cypress/include in themselves
No (derived) Cypress docker images should contain security vulnerabilities (you can check with Azure Security Center)
Obviously, we would prefer not to have to combine Dockerfile contents, or maintain our own Dockerfiles, but the vulnerabilities were there for a while. In the end we wish-list that Cypress.io going forward try to avoid security vulnerabilities more swiftly.

Vulnerabilities
372268-GNU Bash Privilege Escalation Vulnerability for Debian (Zero Day)

Description: GNU Bash. Bash is the GNU Project's shell.
Vulnerability: An attacker with command execution in the shell can use "enable -f" for runtime loading of a new builtin, which can be a shared object that calls setuid() and therefore regains privileges.
Affected by following vulnerability: CVE-2020-6509: Use after free in extensions.
Affected Version: This checks for vulnerable version of Bash shell in Debian 9 and 10.
Severity: High
CVEs: CVE-2019-18276
Vendor references: https://security-tracker.debian.org/tracker/CVE-2019-18276
Links: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-18276

177947-Debian Security Update for ffmpeg (DSA 4722-1)

Description: Debian has released security update for ffmpeg to fix the vulnerabilities.
Vulnerability: An attacker with command execution in the shell can use "enable -f" for runtime loading of a new builtin, which can be a shared object that calls setuid() and therefore regains privileges.
Affected by following vulnerability: CVE-2020-6509: Use after free in extensions.
Affected Version: This checks for vulnerable version of Bash shell in Debian 9 and 10.
Severity: High
CVEs: CVE-2019-13390, CVE-2019-17539, CVE-2019-17542, CVE-2020-12284, CVE-2020-13904
Vendor references: https://lists.debian.org/debian-security-announce/2020/msg00127.html
Links:
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-13390
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17539
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17542
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-12284
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-13904

N.B. there might be other vulnerabilities, here I am only including those with the severity "High", for good order's sake.
Related Cypress.io issues:

https://github.com/cypress-io/cypress/issues/8771
https://github.com/cypress-io/cypress/issues/8773

Versions

cypress/browsers:node12.18.0-chrome83-ff77 FROM cypress/browsers:node12.18.0-chrome83-ff77
Any earlier/prior versions of this Cypress.io Docker image (potentially)

This issue refers to the Cypress Docker image cypress/browsers:node12.18.0-chrome83-ff77 and vulnerabilities related to Debian components.
The listed vulnerabilities have all been fixed:
https://security-tracker.debian.org/tracker/CVE-2019-18276
https://security-tracker.debian.org/tracker/CVE-2019-13390
https://security-tracker.debian.org/tracker/CVE-2019-17539
https://security-tracker.debian.org/tracker/CVE-2019-17542
https://security-tracker.debian.org/tracker/CVE-2020-12284
https://security-tracker.debian.org/tracker/CVE-2020-13904
Cypress Docker images are now based on Debian 12.5 (bookworm).
As described in the README > Usage section, Cypress Docker images are immutable once released, so there will be no correction published for cypress/browsers:node12.18.0-chrome83-ff77.
The underlying Node.js 12.18.0 entered end-of-life on Apr 30, 2022 and by implication the above Docker image is no longer supported.
cypress/browsers lists replacement Cypress Docker images using current supported components.
Closing as resolved.


python3Packages.ipydatawidgets: unbreak

ZHF: #122042
Package version has been bumped by: https://github.com/NixOS/nixpkgs/pull/117452
New version fails during setup requiring npm. It didn't seem trivial to get rid of the npm requirement.
Nothing depends on this package and last version worked fine, so I just downgraded the package. Is this allowed?

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 122345 at 18cb82a3 run on x86_64-linux 1

2 packages built successfully:

python38Packages.ipydatawidgets
python39Packages.ipydatawidgets


"Illegal instructions (core dumped)" whenever loading a model with Auto-GPTQ [BUG]

Describe the bug
Version 0.5.1 will not load models and results in an "Illegal instructions (core dumped)" error.  Models (other than Mistral) load in 0.4.2 but nothing loads with 0.5.1.
Hardware details
Linux container running in ProxMox, 12 cores from 2x Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz passed in, 2x Nvidia Quadro P6000 GPUs passed in, 256GB DDR4 passed in
Software version
Ubuntu 20.04, CUDA 12.1, python 3.11, auto-gptq 0.5.1 pytorch 2.1.1, transformers 4.36, accelerate 0.25.0
To Reproduce
Steps to reproduce the behavior:

Update auto-gptq to 0.5.1
Try to load a quantized model, either with Ooba-text-gen-webui or a Python script
Watch as the "Illegal instruction (core dumped)" error appears

Expected behavior
I expect the model to load without error
Screenshots

Additional context
I thought it was me (realistically, I'm the problem it's me), but I removed every Nvidia package, removed Conda, and re-started and I'm still getting this error.  The CPUs support AVX2, but I still tried a build auto-gptq from source without AVX2 and still could not load models with auto-gptq 0.5.1.  Any help would be appreciated, even on where to look next.

Can you do the following:
pip3 uninstall -y auto-gptq
git clone https://github.com/PanQiWei/AutoGPTQ
cd AutoGPTQ
DISABLE_QIGEN=1 pip3 install -v .

Then test with a Python script and show the output with that, and show the Python script you're using

The install fails when I try with DISABLE_QIGEN=1
Output below:
(textgen) gptj6b@huggingface:~/AutoGPTQ$ git checkout v0.5.1
HEAD is now at f4c9681 fix version
(textgen) gptj6b@huggingface:~/AutoGPTQ$ DISABLE_QIGEN=1 pip3 install -v .
Using pip 23.3.1 from /home/gptj6b/miniconda3/envs/textgen/lib/python3.11/site-packages/pip (python 3.11)
Processing /home/gptj6b/AutoGPTQ
Running command python setup.py egg_info
Traceback (most recent call last):
File "/home/gptj6b/AutoGPTQ/./autogptq_extension/qigen/generate.py", line 1475, in
gen_module_search(args.r, args.p, [2,3,4])
File "/home/gptj6b/AutoGPTQ/./autogptq_extension/qigen/generate.py", line 1418, in gen_module_search
gen_and_compile(n,m,t,n,mb,tb,1,mu,tu,p,u,bits=bits, gs=gs, module=True)
File "/home/gptj6b/AutoGPTQ/./autogptq_extension/qigen/generate.py", line 800, in gen_and_compile
subprocess.check_output(["./autogptq_extension/qigen/mmm", f"{n}", f"{m}", f"{t}", f"{bits}", f"{gs}"])
File "/home/gptj6b/miniconda3/envs/textgen/lib/python3.11/subprocess.py", line 466, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/gptj6b/miniconda3/envs/textgen/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./autogptq_extension/qigen/mmm', '1', '4096', '4096', '4', '-1']' died with <Signals.SIGILL: 4>.
Traceback (most recent call last):
File "/home/gptj6b/AutoGPTQ/setup.py", line 103, in
subprocess.check_output(["python", "./autogptq_extension/qigen/generate.py", "--module", "--search", "--p", str(p)])
File "/home/gptj6b/miniconda3/envs/textgen/lib/python3.11/subprocess.py", line 466, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/gptj6b/miniconda3/envs/textgen/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', './autogptq_extension/qigen/generate.py', '--module', '--search', '--p', '6']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/home/gptj6b/AutoGPTQ/setup.py", line 105, in
raise Exception(f"Generating QiGen kernels failed with the error shown above.")
Exception: Generating QiGen kernels failed with the error shown above.
Generating qigen kernels...
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /home/gptj6b/miniconda3/envs/textgen/bin/python -c '
exec(compile('"'"''"'"''"'"'
This is  -- a caller that pip uses to run setup.py

- It imports setuptools before invoking setup.py, to enable projects that directly
import from distutils.core to work with newer packaging standards.
- It provides a clear error message when setuptools is not installed.
- It sets sys.argv[0] to the underlying setup.py, when invoking setup.py so
setuptools doesn'"'"'t think the script is -c. This avoids the following warning:
manifest_maker: standard file '"'"'-c'"'"' not found".
- It generates a shim setup.py, for handling setup.cfg-only projects.
import os, sys, tokenize
try:
import setuptools
except ImportError as error:
print(
"ERROR: Can not execute setup.py since setuptools is not available in "
"the build environment.",
file=sys.stderr,
)
sys.exit(1)
file = %r
sys.argv[0] = file
if os.path.exists(file):
filename = file
with tokenize.open(file) as f:
setup_py_code = f.read()
else:
filename = ""
setup_py_code = "from setuptools import setup; setup()"
exec(compile(setup_py_code, filename, "exec"))
'"'"''"'"''"'"' % ('"'"'/home/gptj6b/AutoGPTQ/setup.py'"'"',), "", "exec"))' egg_info --egg-base /tmp/pip-pip-egg-info-3nj1qdja
cwd: /home/gptj6b/AutoGPTQ/
Preparing metadata (setup.py) ... error
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

@TheBloke the DISABLE_QIGEN env var is not on the v0.5.1 tag but only on main. @The1Bill can you try on main?

Ahh sorry I didn't realise
Suggestion: make DISABLE_QIGEN=1 the default for 0.6, as it's a constant pain for people trying to build and when I last tried to test QiGen inference, it didn't even work

I will

That sorted it. Thank you - both of you.
I noticed that main is now version 0.6.0 - that was fast!

main has been v0.6.0 for a while, but release is v0.5.1. main will always be the next version(except fixes), v0.6.0 should release quite soon and will be available in pypi with built wheels.


Add sleep to test_job_logs

Summary
This testing is failing on cloud - I believe the db takes a second to update because the logs should always be returned and this change seems to fix it
Details and comments
Fixes #

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 74.079%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
2432

Relevant Lines:
3283

💛  - Coveralls


[Bug]:

What happened?
A bug happened!

URL
FxPXgufiSkEmFqNxF9TUNN2U9JGbWm8LEXOluXUqpYI
What browsers are you seeing the problem on?
Safari
Do you see the issue on Mobile or Desktop?
Desktop
Relevant log output
Fix

cannot debug with the information given. please provide your wallet address if you want additional help


Confusing name

I feel like the library name extendr sounds confusing and unclear. Is it possible to just call it as something similar to RustR?

The intention is quite different from RCpp and R-rust and a few other abandoned RCpp-alikes already exist.
RCpp and R-rust require in-depth knowledge of the R-internals API and we wish to make life
a lot easier than that. Rust is a very flexible language and can embed other languages within
itself (like R) using procedural macros.
The lower case name ending in 'r' is a convention used by dplyr and tidyr.