[BUG] ONNX optimization fails when optimizing AlbertXXL despite the weights being under 2GB

System Info
Optimum 1.2.3[onnxruntime-gpu],
PyTorch 1.12.0a0+bd13bc6,
CUDA 11.6, Ubuntu 18.04,
Transformers 4.19.0,
Onnxruntime nightly build (ort-nightly-gpu 1.12.0.dev20220616003) because otherwise there's an error: Traceback (most recent call last):   File "run_qa.py", line 524, in <module>     main()   File "run_qa.py", line 311, in main     optimizer.export(   File "/opt/conda/lib/python3.8/site-packages/optimum/onnxruntime/optimization.py", line 131, in export     optimizer = optimize_model(   File "/opt/conda/lib/python3.8/site-packages/onnxruntime/transformers/optimizer.py", line 215, in optimize_model     temp_model_path = optimize_by_onnxruntime(input,   File "/opt/conda/lib/python3.8/site-packages/onnxruntime/transformers/optimizer.py", line 96, in optimize_by_onnxruntime     session = onnxruntime.InferenceSession(onnx_model_path, sess_options, **kwargs)   File "/opt/conda/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 335, in __init__     self._create_inference_session(providers, provider_options, disabled_optimizers)   File "/opt/conda/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 363, in _create_inference_session     raise ValueError("This ORT build has {} enabled. ".format(available_providers) + ValueError: This ORT build has ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'] enabled. Since ORT 1.9, you are required to explicitly set the providers parameter when instantiating InferenceSession. For example, onnxruntime.InferenceSession(..., providers=['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'], ...)
The Onnxruntime needs a newer version than the most recently released tag because even the newest release (1.11.1) doesn't yet specify an explicit execution provider, hence we use the nightly build instead.
Who can help?
@JingyaHuang @lewtun @mich
Information

[x] The official example scripts
[ ] My own modified scripts

Tasks

[X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction
The provided run_qa.py script in the examples/onnxruntime/optimization/question-answering doesn't work as expected with ahotrod/albert_xxlargev1_squad2_512.
To reproduce:
python run_qa.py \     --model_name_or_path ahotrod/albert_xxlargev1_squad2_512 \     --dataset_name squad_v2 \     --optimization_level 99 \     --do_eval \     --output_dir /home/ubuntu/albert_xxlargev1_squad2_512_onnx_optimized \     --execution_provider CUDAExecutionProvider \     --optimize_for_gpu
The resulting error:
Traceback (most recent call last):   File "run_qa.py", line 524, in <module>     main()   File "run_qa.py", line 311, in main     optimizer.export(   File "/opt/conda/lib/python3.8/site-packages/optimum/onnxruntime/optimization.py", line 142, in export     optimizer.save_model_to_file(onnx_optimized_model_output_path, use_external_data_format)   File "/opt/conda/lib/python3.8/site-packages/onnxruntime/transformers/models/gpt2/../../onnx_model.py", line 934, in save_model_to_file     save_model(self.model, output_path)   File "/opt/conda/lib/python3.8/site-packages/onnx/__init__.py", line 202, in save_model     s = _serialize(proto)   File "/opt/conda/lib/python3.8/site-packages/onnx/__init__.py", line 71, in _serialize     result = proto.SerializeToString() ValueError: Message onnx.ModelProto exceeds maximum protobuf size of 2GB:<PHONE_NUMBER>
This error triggers the ModelProto > 2gb error despite the model weights being less than 2gb (~870MB, rather).
Expected behavior
The optimized ONNX model is successfully saved.

Hi @michaelroyzen,
Although the size of PyTorch model is smaller than 2GB, it is still possible that the exported ONNX model exceeds 2GB. (Link to a related issue).
Ideally, by setting the argument use_external_data_format=True in optimizer shall allow you to export onnx model larger than 2GB with external files. So for your case, inserting a line use_external_data_format=True in the run_qa example.
However, unfortunately, the write_external_data_tensors in onnx library doesn't seem to work well for the moment (link to a related issue), and this needs to be reported to ONNX side 😞 .

Yes, adding use_external_data_format=True to run_qa didn't work unfortunately @JingyaHuang -- it seems the issue is with ONNX.
This is what I tried in run_qa.py:
optimizer.export(         onnx_model_path=model_path,         onnx_optimized_model_output_path=optimized_model_path,         optimization_config=optimization_config,         **use_external_data_format=True**     )

@JingyaHuang @michaelroyzen  I think this is what we need. https://github.com/huggingface/optimum/pull/302

@JingyaHuang it seems to me the issue is caused by an incorrect import:

/opt/conda/lib/python3.8/site-packages/onnxruntime/transformers/models/gpt2/../../onnx_model.py

when

from onnxruntime.transformers.optimizer import optimize_model

is called, when in fact the file should be this

Hi @michaelroyzen, have you solved the issue of saving the onnx model?

BTW, if you are not using a vision model,  setting optimization_level is generally good enough.

To follow up on the issue, here is the thread in onnx where we continued the discussion.
It turns out that with the hard constraint of protobuf size limit(2GB), ONNX offers some options which export large tensors to external files. Users can tune the parameters to find the best fit.
However, for several extremely large models(the case of AlbertXXL), the structural proto could still exceed 2GB after exporting all tensors to external files. In this case, a workaround would be to either load the fp16 model weights(if the model was also trained with mixed precision) or use ORTQuantizer to proceed with the quantization.
A PR is in progress to improve the compatibility of ORTOptimizer and ORTQuantizer in cases of large ONNX proto.

HI,did you resolved the bug? I still cant manage to convert a Albert xxl-model.

@shaked571 Could you share the command/script you used to export the model please?


Update composer.json

There is conflict in main invoiceninja project, because:

omnipay/omnipay 2.3.2 requires omnipay/stripe ~2.0
this package invoiceninja/omnipay-collection v0.9 requires omnipay/stripe dev-master

So I updated this package to same stripe version as omnipay

Looks good, thanks!

FYI... we plan to update the omnipay collection lib in Invoice Ninja soon to remove Dwolla

great, I also hit problem with Dwolla...


Build tensorflow error with keras api

I build tensorflow follow link

use r2.0
bazel 0.25.0
CUDA Version: 10.0
ubuntu 18.04
Here my error

ERROR: /root/tensorflow/tensorflow/python/keras/api/BUILD:13:1: Executing genrule //tensorflow/python/keras/api:keras_python_api_gen failed (Exit 1)
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_1_keras_python_api_gen.runfiles/org_tensorflow/tensorflow/python/tools/api/generator/create_python_api.py", line 27, in <module>
    from tensorflow.python.tools.api.generator import doc_srcs
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_1_keras_python_api_gen.runfiles/org_tensorflow/tensorflow/python/__init__.py", line 83, in <module>
    from tensorflow.python import keras
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_1_keras_python_api_gen.runfiles/org_tensorflow/tensorflow/python/keras/__init__.py", line 32, in <module>
    from tensorflow.python.keras import datasets
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_1_keras_python_api_gen.runfiles/org_tensorflow/tensorflow/python/keras/datasets/__init__.py", line 25, in <module>
    from tensorflow.python.keras.datasets import imdb
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_1_keras_python_api_gen.runfiles/org_tensorflow/tensorflow/python/keras/datasets/imdb.py", line 25, in <module>
    from tensorflow.python.keras.preprocessing.sequence import _remove_long_seq
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_1_keras_python_api_gen.runfiles/org_tensorflow/tensorflow/python/keras/preprocessing/__init__.py", line 21, in <module>
    import keras_preprocessing
ModuleNotFoundError: No module named 'keras_preprocessing'
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 11062.314s, Critical Path: 746.52s
INFO: 17931 processes: 17931 local.
  That is included in the instructions you linked.

Yes
Installing collected packages: keras-preprocessing
Successfully installed keras-preprocessing-1.1.0

Can you run pip show keras-preprocessing and does the value in Location  match the value in .tf_configure.bazelrc for build --action_env PYTHON_LIB_PATH ?

here is it
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Name: Keras-Preprocessing
Version: 1.1.0
Summary: Easy data preprocessing and data augmentation for deep learning models
Home-page: https://github.com/keras-team/keras-preprocessing
Author: Keras Team
Author-email: None
License: MIT
Location: /root/.local/lib/python3.6/site-packages
Requires: numpy, six
Required-by:

And cat .tf_configure.bazelrc show what?
When you ran ./configure
For Please input the desired Python library path to use. the answer should have been /root/.local/lib/python3.6/site-packages as that is the location where the python package are installed.

Running into the same error
Executing genrule //tensorflow/python/keras/api:keras_python_api_gen failed
Aditionally there is an import error, might be the same issue
ImportError: cannot import name 'gen_checkpoint_ops' from 'tensorflow.python.ops' (/private/var/tmp/_bazel_nikhilchitlurnavakiran/ed132e8d3b760bae902e7d17da8676dc/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_1_keras_python_api_gen.runfiles/org_tensorflow/tensorflow/python/ops/init.py)
going through the above discussion have listed the commands to verify
pip3 install -U --user keras_preprocessing --no-deps
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Requirement already up-to-date: keras_preprocessing in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (1.1.0)
pip3 show keras-preprocessing
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Name: Keras-Preprocessing
Version: 1.1.0
Summary: Easy data preprocessing and data augmentation for deep learning models
Home-page: https://github.com/keras-team/keras-preprocessing
Author: Keras Team
Author-email: None
License: MIT
Location: /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
Requires: six, numpy
Required-by: tensorflow
cat .tf_configure.bazelrc
build --action_env PYTHON_BIN_PATH="/Library/Frameworks/Python.framework/Versions/3.7/bin/python3"
build --action_env PYTHON_LIB_PATH="/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages"
build --python_path="/Library/Frameworks/Python.framework/Versions/3.7/bin/python3"
build:xla --define with_xla_support=true
build --config=xla
build:opt --copt=-march=native
build:opt --copt=-Wno-sign-compare
build:opt --host_copt=-march=native
build:opt --define with_default_optimizations=true
test --flaky_test_attempts=3
test --test_size_filters=small,medium
test --test_tag_filters=-benchmark-test,-no_oss,-oss_serial
test --build_tag_filters=-benchmark-test,-no_oss
test --test_tag_filters=-gpu,-nomac,-no_mac
test --build_tag_filters=-gpu,-nomac,-no_mac
build --action_env TF_CONFIGURE_IOS="0"

@phamkhactu,
Any updates regarding this issue? Thanks!

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

What is it with this bizarre coupling of keras to tensorflow (especially in TF 2.0)? TensorFlow should just be TensorFlow... 2.0 didn't make me use keras anymore than I did before. It just made me start using pytorch.


That plugin run with Google Ad Exchange (AdX)?

I've seen that this implementation (AdMob) is very similar with Google Ad Exchange (AdX), my ask is, this plugin work with Adx or i need change something?

From what I understand, AdX is just a network, so you should be able to use it with the PublisherBanner

@1fabiopereira could you use it with "adx"?

@1fabiopereira could you use it with "adx"?

Yes, you just need use "Adx" as PublisherBanner


Added SwiftLint

Can you add docs or link on how to setup SwiftLint to README.md and if possible run SwiftLint in a GitHub action?


Add showing Messenger Plans

When creating/updating/responding to a plan on FBM nothing gets posted to Discord. The bot console session shows it is receiving these events but doing nothing with them.
In a perfect world:

Creating a plan on FBM creates an embed on Discord (And maybe pins the message..?)
Embed shows plan name, location, date and numbers confirmed as going/not going
Changes made in FBM update the existing embed on Discord.

I'll see if that's possible (if facebook-chat-api has a functionality to listen for event changes).

That's a shame - sounds like the functionality is there once they fix it though.

Update: https://github.com/Schmavery/facebook-chat-api/pull/608#issuecomment-372362626
So... it's possible, but requires changes in facebook-chat-api to fetch plan data.

So with the fix they've merged in now does it just not crash any more but still can't see details?

Yup, that's right. It's impossible to see the details without another request with event ID.

@ajstewart see #275, there's an issue with Facebook API

In case anyone is desperate, inspired by the duct tape over a hole in a boat solution for polls in #139 I've written a similar solution for plan messages.
Like that solution I do not really know json so I'm sure it could be done far better, but it seems to work and does what I want it to.
Follow the instructions in #139 - edit the file:
miscord\node_modules\facebook-chat-api\src\listen.js
Go to line 240 or so where it should say case "delta": and paste the code below - be sure to change your timezone!
But again this is not a 'fix', just a temporary solution!
var eventdeltas = ['lightweight_event_update_time',
                   'lightweight_event_update_title',
                   'lightweight_event_delete',
                   'lightweight_event_create',
                   'lightweight_event_notify',
                   'lightweight_event_rsvp'
                    ];
if (eventdeltas.indexOf(v.delta.type) > -1)
{
    var eventinfo = "-------------------------------------------------------\n_**" +
    v.delta.messageMetadata.adminText + "**" + '\n\n**Plan information:**\n';
    var event = JSON.parse(v.delta.untypedData.guest_state_list);
    var going = 0;
    var declined = 0;
    // created plans do not have event_title
    if (v.delta.type != 'lightweight_event_create')
    {
        var event_title = v.delta.untypedData.event_title;
        if (event_title == "") {event_title = "Plan not named!"}
    }
    else
    {
        var event_title = "New plan!"
    }
    eventinfo = eventinfo + "Name: " + event_title + "\n"
    var event_time = Number(v.delta.untypedData.event_time);
    var dateTime = new Date(event_time* 1000);
    var options = { weekday: "long", day:"2-digit", month: "long", hour:"2-digit", minute:"2-digit",
                     timeZone: 'Australia/Sydney', localeMatcher:"lookup"}
    var formatted = dateTime.toLocaleString("en-GB", options);
    eventinfo = eventinfo + "Time: " + formatted + "\n"
    for (i in event)
    {
        var status = event[i].guest_list_state;
        if (status == "GOING") {going+=1}
        else if (status == "DECLINED") {declined+=1}
    }
    eventinfo = eventinfo + "Going: "+JSON.stringify(going)+"\nDeclined: "+JSON.stringify(declined)+"_";
    eventinfo = eventinfo + "\n-------------------------------------------------------"
    v.delta = {
    attachments:[],
    body: eventinfo,
    irisSeqID: v.delta.irisSeqID,
    messageMetadata: v.delta.messageMetadata,
    ttl: v.delta.ttl,
    class: 'NewMessage'
    }

}

@ajstewart I could try to integrate it into facebook-chat-api by reviewing this code you wrote, maybe slightly improving it and publishing as a PR to facebook-chat-api

or... I can just create my own fork of facebook-chat-api with these changes and add it to miscord until it's added to the main (Schmavery's) repository

@Bjornskjald yeah I was wondering what the aim of the API was in this case. As even if you got the status messages you still couldn't see the plan details such as numbers attending. If the code is a helping start I'm happy for it to propagate to the facebook-chat-api.
It sounds like your own branch might give more freedom of fixing things quicker but then you run the danger of diverging too much I guess. I think I'll leave that up to you!
I would be interested to see an improved version! I struggled with the time string a bit, couldn't get it how I wanted, mostly there in the end. Also the IDs of the facebook users are also there so you could even put a list of actual attendees. I wasn't sure how to get the names so I left it (you can see their response message anyway).

oh, I have an idea, I could make the plan message pinned and edit it with new "going"/"can't go" info

Yeah that could work, that would be pretty sweet. You’d have to listen out for other changes too like title, time etc but that should be easy.
If you want to go really far you could add emoticons to the pinned message and then when a Discord user clicks on that emotion an automatic message is posted saying that Discord user wants to join the plan (as if from them) e.g. “Discord user @{user} responded going to {plan name}”. That’s probably as close as an integration you could get!

On 15 Sep 2018, at 16:03, Bjornskjald<EMAIL_ADDRESS>wrote:
oh, I have an idea, I could make the plan message pinned and edit it with new "going"/"can't go" info
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/miscord/miscord/issues/87#issuecomment-421534069, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJysWIdlht1QLB8RNF_MECn1V9L6_tZks5ubJgdgaJpZM4SmfmK.

Damn, the biggest problem would be timezones. Another field in the config?

I think that would be simple enough to include in the config, though you would have to state clearly that there is a set list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones.


Feature request: lookup / define on text selection

It'd be really useful to have a shortcut in the contextual menu to open a tab on the definition of a word / a search lookup. I'm not sure if there's a default search engine for Gemini yet, but that'd be using it. Cheers!

I suppose geminispace.info/search is pretty much the default, but this feature could just use whatever search query URL the user has configured in Lagrange.
The OS may also be able to provide word definitions, or query dictd if available.


url decode 中文显示乱码

url decode 中文显示乱码

What probably happened

By default, cyberchef shows the output as raw bytes
How to solve this problem

You can click on the encoding to choose a proper encoding for your output

ok, thx


In Readme file added table of contents

fixed #1602

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for your contribution @shassshank!


Can O' Spam: Watch bitcointalk.org

Can O' Spam requests the watch of the watch_keyword bitcointalk\.org. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

bitcointalk\.org has been seen in 14 true positives, 82 false positives, and 13 NAAs.

Rejected by Jeff Schaller in Charcoal HQ. Reason: 'there's a substantial number of FP on-site (over 1,000)'


Make top Nav responsive

The nav is overflowing in x axis causing all pages to break.

@tnmyk go ahead and fix this up !


fix(camelcase): remove camelcase dependency

remove camelcase in favour of simple mapping of http-proxy events.
hopefully fixes #513

Coverage decreased (-0.08%) to 87.38% when pulling 62d0264266e2983a6968c1f788275cdedec79fcc on remove-camelcase into ba3391f22436d111860054930125d23eea24901c on master.


[PDI-19578] Introduce property KETTLE_USE_META_FILE_CACHE (default N) to

control whether trans/job/step metas are cached when being loaded
multiple times over the life of a parent trans/job.  Cache is currently
broken; variable space is not updated even if a parent parameter value
has changed, resulting in undesired behavior.
commit used for backporting - 087ca7d78340d302b4c63a4f4f32b9b801ddf59b

:white_check_mark: Build finished in 20m 53s
Build command:
mvn clean verify -B -e -Daudit -amd -pl core,engine

:exclamation: No tests found!
:information_source:  This is an automatic message


[SPARK-41938][BUILD] Upgrade sbt to 1.8.2

What changes were proposed in this pull request?
This pr aims to upgrade sbt from 1.8.0 to 1.8.2
Why are the changes needed?
Release notes:

https://github.com/sbt/sbt/releases/tag/v1.8.2
https://github.com/sbt/sbt/releases/tag/v1.8.1

Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GA.

Merged to master.

late LGTM


setting inter_op_parallelism_threads automatically?

Given that this is currently hidden, maybe it could be automatically set based on core number? Or then through a one line utility which the user can use in the same way as gpu_utils are now used.
2019-02-20 20:26:09.208696: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-02-20 20:26:09.209009: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 4. Tune using inter_op_parallelism_threads for best performance.

One thing to bare in mind here is that this would be specific to tensorflow.


Remove Assets List from Map File Format

The map file should not have a list of assets. This duplicates the information stored within the drawables and creates a second source of truth.

Outdated?


delete unneeded concepts.md

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/add-ins/concepts.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


[Feature request] Mute Option

its like ban or kicking a guest out, exept u mute the person who joins who they cannot chat in the chat system

Fair enough, seems simple to add too.

yay lol, exept for all of my other feature requests, sorry if i didnt read the documents

It's fine, GitHub also doesn't help newcomers as it hides by default the closed issues.


Failed to resolve: com.google.http-client:google-http-client-parent:1.24.1

I have these lines in my app gradle file.
endpointsServer project(path: ':backend', configuration: 'endpoints')
implementation 'com.google.api-client:google-api-client:1.24.1'
implementation 'com.google.api-client:google-api-client-android:1.22.0'

I keep getting the error: Failed to resolve: com.google.http-client:google-http-client-parent:1.24.1

Looks like com.google.http-client:google-http-client-parent:1.24.1 was not published to Maven Central at the same time the other artifacts were.

Also, in this specific case don't depend on 'com.google.api-client:google-api-client:1.24.1'. It's not ready yet.

Hi - apologies for the problems. As @elharo mentioned, you can work around this for the moment by pinning to 1.23.0. We're yanking 1.24.1 (https://issues.sonatype.org/browse/OSSRH-41618) and will publish 1.24.2 ASAP.

There is  a work around posted on this stackoverflow thread. https://stackoverflow.com/questions/51623523/android-plugin-for-gradle-3-0-0-could-not-find-com-google-http-clientgoogle-h/51631503#51631503
It works. I have tested myself.

We have manually pushed a release of the missing 1.24.1 parent. I've tested locally and it seems to fix the issue. Please double check and let me know if the problem still persists. I'll close the issue for now, but if anyone notes this problem still occurring I am happy to re-open this.

Can confirm this fixed our builds! Thanks!


Fix Makefile for new docs structure

Current coverage is 96.54% (diff: 100%)

Merging #1085 into master will not change coverage

@@             master      #1085   diff @@
==========================================
  Files            49         49
  Lines          2168       2168
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits           2093       2093
  Misses           75         75
  Partials          0          0

Powered by Codecov. Last update 4802f0f...5707bee


pit-ray.win-vind version 5.0.1

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?
[x] This PR only modifies one (1) manifest
[x] Have you validated your manifest locally with winget validate --manifest <path>?

[x] Does your manifest conform to the 1.4 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

2023-04-03 08:59:07.515 [FAIL] Installer failed security check. Url: https://github.com/pit-ray/win-vind/releases/download/v5.0.1/setup_win-vind_5.0.1_32bit.exe Result: 0x80004005

@Trenly
Thanks for your feedback.
I didn't change the installation way from its previous version, so I am confused to do.
Please let me know how to check defects locally.
Thanks.

@wingetbot run

@Trenly @stephengillie
Thanks for your generous support.


[bug] if use forceFallback, cannot get the dragOver elements

Describe the bug
if use forceFallback, cannot get the dragOver elements

To Reproduce
Steps to reproduce the behavior:

forceFallback: true to have the custom mirror(floating) element
addEventListener  like dragover, mouseover.., etc
the listener not works
and if we want to save the evt.related in onMove and use it in onEnd, the variable in onEnd takes null for any reason.
Expected behavior
I just want to both mirror element and getting drag over element works fine.

Information

Versions - Look in your package.json for this information:
sortablejs = ^1.15.0
Additional context
Null
Reproduction
codesandbox: <your_url_here>

Confirmed: when using 1.15.1 with forceFallback: true, events like dragenter and dragleave won't trigger on the target container anymore.

It won't work because it's not native drag and drop, you can manually trigger these events using dispatchEvent

It won't work because it's not native drag and drop, you can manually trigger these events using dispatchEvent https://developer.mozilla.org/en-US/docs/Web/API/EventTarget/dispatchEvent

can you give me example


Add valueFileSecrets example to Redis release

Fix https://github.com/weaveworks/flux/issues/1617

@squaremo can you please review this?
Thanks

If we're using an unexcrypted secret, and just suggesting people use sealed secrets, do we need the sealed-secret chart release?

I've put the sealed-secret chart release in this repo so I can link to it when people ask about secrets. I can remove it or use the ignore annotation, we don't have an example for that anyway.

Your gitops-helm repo has the more complete example (which you can point to) -- I think you can remove it here.

Done


Hitting Undo in the design tool deselects the element

STR

Insert a View
change its layout to 'flow'
CMD+Z

Expected: stays selected
Observed: gets deselected

I've been unable to reproduce this, potentially this has been fixed by #1308 and/or some other fixes.


support SHA-256 signature verification headers from Webhooks

Feature Request
Is your feature request related to a problem? Please describe.
GitHub will soon begin sending SHA-256 signature headers with Webhooks, when configured with a secret. The header will look like:
X-Hub-Signature-256: sha-256=kljsdfjkldasjfkaejkfwekuweur3298r2398r89fdsuf3
Describe the solution you'd like
Probot should use the SHA-256 header instead of the SHA-1 header for verification. Note that both headers will be sent, and there is no deprecation date on SHA-1 support at this time.
Describe alternatives you've considered
N/A
Teachability, Documentation, Adoption, Migration Strategy
N/A

This will need to be solved in https://github.com/octokit/webhooks.js, I'll make a follow up issue there

resolved via https://github.com/probot/probot/pull/1390, published in 10.9.5


nextValidRequestTime is incorrect in failCallback?

Hi @AdamPflug, I have trouble to understand the nextValidRequestTime in failCallback
Let's say I have this config
new ExpressBrute(store, {
    freeRetries: 5,
    minWait: 5000,
    maxWait: 5000
});

The current implementation has delays = [5000], lifetime = 30.
Current behavior:
failCallback returns nextValidRequestTime is lastValidRequestTime + 5 seconds (delay = maxWait)
Expected behavior:
failCallback should return nextValidRequestTime is lastValidRequestTime + 30 seconds.
Because I try to request again 5 seconds after the last request, the request will be stopped. But after 30 seconds after the last request, the request will go through.
Am I missing something here? Thanks.

nextValidRequestTime is determined by the min/max wait and the number of previous tries, not by lifetime. So after your first requests (6 in this case), you can issue a request every 5 seconds. lifetime is basically how long the store will remember each session, as way to prevent the store from filling up with a bunch of old sessions. After lifetime (30 seconds in this case), express-brute basically forgets about the user and they get a new set of free retries (again, 6 more requests) before being rate limited again.
By default it's calculated such that there isn't a benefit for waiting until the lifetime expires to issue new request. In this case if you wait 30 seconds you get 6 requests before the rate limiting kicks in, which is the same number of requests you'd get in the 30 second period if you just tried every 5 seconds (as specified by max/min wait).
You can manually set lifetime if you want, which can be useful if you want to limit the number of requests in a specific timeframe. For example, this limits the user to 10 requests every minute (the minute starts counting from the first request):
var tenRequestsPerMinute = new ExpressBrute(store, {
	freeRetries: 10,
	refreshTimeoutOnRequest: false,
	minWait: 61*1000, // 61 seconds (we will never reach this wait time)
	maxWait: 61*1000, // 61 seconds (we will never reach this wait time)
	lifetime: 60 // 60 seconds (in seconds not milliseconds)
});

Let me know if that answers your question
-Adam

I understood your example.
However, in my case, let's say I set up my endpoint like this with the config I specified above.
router.post('/', expressBruteMiddleware, (req, res, next) => {})
So at the moment if I send 6 requests consecutively:

The first 5 requests will go through.
On the 6th request, expressBruteMiddleware will step in and throw error - which is as expected.
But the value nextValidRequestTime which is returned by fail failCallback doesn't indicate the next time I can send another request and it goes through.
In fact, I can successfully send another request after 30 seconds (lifetime) after 6th request.

Hopefully you understand my question now. Thanks you Adam.

I'm not sure I understand. Are you saying that if you get rate limited, then wait 6 seconds and try again it doesn't work, you need to wait the full 30 for the lifetime to expire? The time that nextValidRequestTime is reporting should be correct, so that wouldn't be the correct behavior. That's unexpected because this is case that should be covered by the unit tests, which store are your using?

I use express-redis-store. I was tinkering around with your unit tests and they work as I expected.
I think I know what is problem. I have one ExpressBrute instance and its expressBruteMiddleware is used in 2 different endpoints GET /foo GET /bar.
Front-end tries to GET /foo and GET /bar one after another from back-end, and here comes the problem.
So my guess is that I probably need to create different instances and different prefixes for those endpoints :thinking:
Thank you for your support. We can close the issue now.


Could not read partition values correctly with special characters in partition column name

Environment
Delta-rs version: 0.4.1
Binding: Python
Environment:

Cloud provider: N/A
OS: Windows/Ubuntu

Bug
What happened:
When reading partitioned tables with special characters in partition column name, the partition column would get NaN values.
What you expected to happen:
Could be able to read these tables and got the same results with native Spark.
How to reproduce it:
Create a partitioned table with special character in the partition column name.
sdf = spark.createDataFrame([[1, 10], [2, 20]], schema=['x%20x', 'y'])
sdf.write.format("delta").partitionBy(["x%20x"]).save("partition_table")

Then try to read the table:
In [1]: from deltalake import DeltaTable

In [2]: dt = DeltaTable('partition_table')

In [3]: dt.to_pandas()
Out[3]:
   x%20x   y
0    NaN  10
1    NaN  20

It seems that we are leveraging pyarrow to handle the partition folders. But Delta generated file paths differently than Hive?
We can see from _delta_log:
{"metaData":{"id":"7e6ec837-c032-4cec-adab-d2299042e622","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"x%20x\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"y\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["x%20x"],"configuration":{},"createdTime":1636641415873}}

The field name is 'x%20x'. But the file path generated by Delta is:
├─x%2520x=1
├─x%2520x=2
└─_delta_log

Could anyone take a look? Thanks.

@zijie0 could you check if dt.pyarrow_schema contains the right partition column name? The path generated by spark looks correct to me. The path needs to be uri encoded, that's why %20 is saved as %2520. Pyarrow's hive partitioning implementation by default does uri decode as well: https://github.com/apache/arrow/blob/caf1e1ed459e2c6ff66255fc915a5a7b16c0f5c5/python/pyarrow/_dataset.pyx#L2283.

@houqp dt.pyarrow_schema contains the correct partition column name:
In [10]: dt.pyarrow_schema()
Out[10]:
x%20x: int64
y: int64

I think we pass the correct file paths and schema to Pyarrow but the partition column values are inferred by Pyarrow:
https://github.com/delta-io/delta-rs/blob/36d56ecfcf1046ba29ff26dd17e207c6419ead4f/python/deltalake/table.py#L282-L288
I'm not sure if we need to do some modification to the partitioning parameter in this situation.

Looking at the upstream arrow code, I think it's only doing url decode for partition value, but not the column name? https://github.com/apache/arrow/blob/e5f3e04b4b80c9b9c53f1f0f71f39d9f8308dced/cpp/src/arrow/dataset/partition.cc#L593-L596.
@zijie0 can you try creating the dataset without providing the pyarrow schema to confirm whether this is the case?
Ideally, we shouldn't depend on the file path for partitioning discovery because the delta spec doesn't care how the file paths are generated. In theory, a writer can just use random value as the file path without any partition value encoding. It's the delta table reader's responsibility to populate the partition column info based off the partition value info from the corresponding add action. This would be more of a long term fix for this problem.

@houqp you are right, arrow doesn't decode the column name:
In [3]: ds = dataset(paths, format="parquet", partitioning=partitioning(flavor="hive"))

In [4]: ds.to_table().to_pandas()
Out[4]:
    y  x%2520x
0  20        2
1  10        1

Thanks @zijie0 for the deep dive :)
Unfortunately there is no quick and easy way to fix this. The two options that come to my mind are:

Fix this in upstream pyarrow, add partition name url decode in their c++ core.
Do it properly in delta-rs by performing the io and arrow record batch conversion entirely in rust, where we will be able to populate partition values using table metadata instead of relying on file path.

At the very minimum, I think we should file an upstream arrow github issue :)

@houqp I just tested the case on arrow version 7.0.0, it is fixed by https://github.com/apache/arrow/pull/11858 . Thanks for following up this issue.

I am having a similar issue with special characters in azure data lake gen2 on databricks.
In the below code pyarrow decodes spaces in the partition paths as %20 when they need to be %250 for delta.
I can resolve my issue by passing the full path to partitions arg with spaces replaced with %250.
However, I would like to be able to resolve the encoding so I don't have to use this work-around.
Note: It appears that using the dbfs filesystem is interpreted as local rather than azure gen2.
Running this:
from deltalake import DeltaTable
dt = DeltaTable('file:///dbfs/<path-to-delta-table>')
pa_df = dt.to_pyarrow_table()

Results in:
PyDeltaTableError: Object at location /dbfs///.parquet not found: No such file or directory (os error 2).
Any help would be appreciated.
delta-rs==0.6.1
pyarrow==9.0.0

After doing a deep dive of the docs and the codebases of deltalake and arrow I found a very simple solution. Just pass a local pyarrow filesystem instance Sharing below in case it helps anyone else.
from pyarrow import fs
from deltalake import DeltaTable

local = fs.LocalFileSystem()

dt = DeltaTable('file:///dbfs/<path-to-delta-table')
df = dt.to_pyarrow_table(partitions=[('key', '=', 'spaced value')], filesystem = local)


Parking Lot

Post any questions you may have here during the training.

delete local repo, what happens. deleting staged files?

Basic git commands

Additional commands:

git branch -a
git status
git checkout <branch-name> or git switch <branch-name> to switch to a different (already-existing) branch

git checkout -b <branch-name> to create a new branch

Assume you have already cloned this repository on your local machine, you clone with git clone and you have a working feature branch on GitHub and on your local machine.

If you need to create a new branch, you can do so with the "Branch: master" button in the Code tab OR create locally with git checkout -b BRANCHNAME

Assuming you're also working in a team and not just by yourself in a repo

Update your local main branch because someone might have make updates while you're away/ sleeping: Make sure you're on the main branch, do git pull on master branch to get your local master branch updated
Do a reverse merge to your local feature branch so your feature branch is updated: we’re going to switch over to that branch first with git checkout FEATURE-BRANCH-NAME and then do a reverse merge git merge main

Make your commits: git add, git commit -m

Push it to Github: git push
Open a pull request, get your approvals and then merge it into master. Delete the branch you just merged.
Clean up your local machine, with

git checkout main
git pull --prune
git branch -d FEATURE-BRANCH-NAME


Sans-Serif Styling

Would you be open to adding another CSS theme that pretty much just overrides the font-family to use system UI and sans-serif fonts?
I've gotten used to San Francisco on my iOS devices and I think having an option to try to keep fonts similar to the OS isn't a bad idea. (As suggested by this CSS Tricks article (no affiliation)).
If you're okay with this, I'd be happy to put a pull request together. I just didn't want to do the work if it's something you'd rather not see added.

Sure, why not.

You PR has been merged.


Create a new file, then add -- doesn't show as staged

If you create a new file, it appears in the working tree tab. If you check it and add it, it doesn't appear as staged; instead, it just vanishes from that tab. It does still appear in the "all files" tab, but with the "unchanged" label.

And then you can't commit, because Elegit insists that "You need to add files before committing."


AWS sqs scaler to support scaleOnDelayed

Proposal
Similar to scaleOnInFlight, it'd be appreciated if we could have scaleOnDelayed to indicate whether messageDelayed metric should be counted in to a trigger check.
Use Case
Our service consumes messages from SQS, queue them internally and deletes them from SQS immediately (we can't change this at the moment).
Hence it's possible that scale down would happen while the service is still processing the messages, because SQS is already empty.
We could set a longer cooldownPeriod (e.g. 30mins), but it's still possible that our service could consume and delete new messages between two polling checks (hence 30mins cool down is not refreshed).
If scaleOnDelayed is available, we could set delaySeconds = 10s when we add messages to the queue, and pollingInterval=5s.
This way 30mins cool down would always be refreshed when there is a new message.
Is this a feature you are interested in implementing yourself?
Maybe
Anything else?
No

Hey @ding1992
I think that this could be a good addition if the SDK supports it.
Are you willing to contribute with the feature?
You would have to add the new parameter and the parsing in this file and covering with tests in this other file (and of course update the docs with the new parameter :) )

#4383 #https://github.com/kedacore/keda/pull/4383](https://github.com/kedacore/keda/pull/4383)

I'll check it tomorrow


update reproducible configuration to match current practice

workarounds useful 6 months ago are not any more, and are more a pain to debug eventual remaining issues nowadays

Hi @hboutemy , thanks for all the PRs submitted.
@aharui, I'll merge this 3 ones but since you switched back to the old zlika plugin, I think better left to you.
I see the PRs completely right.
Thanks

I may not have time to review this PR until Friday evening.  It might be helpful to have an explanation of why the changes are proposed.  I'm unable to understand them just by looking at them.

Looks like .xml, .properties, and .html file have different whitespace.  Can we control which files have which line-endings?


Image gallery nonjs

What is this PR trying to achieve?
Creating a non-js version of the image gallery.
What does it look like?

For interface components

[ ] Browser testing - Have you checked the component in our supported browsers
[x] No javascript - Have you checked the component with javascript disabled
[x] A11y testing - Have you run pa11y against the component and fixed/verified all errors, warnings and notices?

Apologies for the last commits, rebasing is proving like I might screw something up, so I'll wrap these into one when merging.


Upgraded the Default PKCS12 Encryption Algorithms JDK-8153005

Question
Why were the Oracle upgraded default PKCS12 Encryption Algorithms not in the AdoptOpenJDK release  8u302 ?  I see that they are there in the Oracle 8u301 release which I thought was the basis for the AdoptOpenJDK release. .
Context
Java version:
openjdk version "1.8.0_302"
OpenJDK Runtime Environment (Temurin)(build 1.8.0_302-b08)
OpenJDK 64-Bit Server VM (Temurin)(build 25.302-b08, mixed mode)Paste the entire output of java -version.
Your operating system and platform:
Windows 64 bit

https://bugs.openjdk.java.net/browse/JDK-8258401 is the backport for reference. Can you show us some output demonstrating the out of date algos?

No, it's not. This is Oracle JDK 8u (which is private). Temurin doesn't use Oracle JDK as base, but OpenJDK 8u. An OpenJDK backport would have "Fix Version: openjdk8u". The above bug has 8u301. So no, this hasn't been backported to OpenJDK yet. Feel free to contribute a patch upstream!

Aha - I got confused again, thanks, Severin!


Quantization & ONNX export through Brevitas

As per title.
Port & adapt part of the code of https://github.com/Xilinx/brevitas/tree/optimum in optimum-amd.
Left to do:

[x] Fix FX compatibility, handling the KV cache in the dataset directly
[x] Fix tied weight logs during ONNX export weight dedup (e.g. model.decoder.layers.0.self_attn.k_proj.weight_quant.tensor_quant.scaling_impl.parameter_list_stats.first_tracked_param.parameter)
[x] Fix the perplexity computation to avoid grouping different sentences.
[x] Write tests
[ ] Test exported models on Ryzen AI (maybe for a later PR) => dynamic quantization currently can't be consumed
[x] Documentation & cleaning remainings todos

Maybe for another PRs, but I guess the documentation needs to be more extensive. Given so many knobs, it would be good to have description directly in the doc and a support matrix for the parameters.

mostly @Giuseppe5 & @nickfraser's!

^Don't be modest @fxmarty! <3

Only that small typo, otherwise, LGTM @fxmarty!


feat: add derived Clone

Motivation
Easy to use with cheap clone.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Baklava cLabs validators do not have locked CELO quorum to pass governance proposals with 0.9+ agreement)

Expected Behavior
Contract releases can be conducted on baklava via governance proposal
Current Behavior
Proposals cannot be passed by voting from cLabs validator 0 and validator 1

Fixed by adding more funds to the validator-0 in these two tx:
https://baklava-blockscout.celo-testnet.org/tx/0x67a27e53f8a9b6a919f5c1e3708f50d6f7a2542e33120ce0de3838c5344e5dad/token-transfers
https://baklava-blockscout.celo-testnet.org/tx/0x0060865b4d1ef5320f1ca6882cbcf663281afebc0c0b5c57b35eae7849ada046/token-transfers


Adding the ability to translate mails / notifications

See: https://github.com/laravel/internals/issues/513

I think a better approach to this would be to use the message keys using the new JSON translation files. But, regardless, all of this can be exported and translated easily.


[BCL builder] Make postcss prefixer option more flexible

Problem
BCL builder allows passing the prefix option when compiling sass files, but it is not flexible enough as it only passes the prefix and not the entire postcss-prefix-selector config which we would need.
Also, the plugin gets run with an empty prefix when no prefix are specified, which I believe is slowing down the compilation for no reason. And the current transform callback is not necessary and optional.
Context:
We would need such feature/flexibility for ckeditor5 stylesheets which expects all CSS to be prefixed with ".ck-content", but we need to specify the transform callback to deal with ":root".
Also relates to https://github.com/openeuropa/oe_bootstrap_theme/issues/404
To be:
{
  entry: path.resolve(baseDir, "resources/sass/style.scss"),
  dest: path.resolve(baseDir, "assets/css/ckeditor5-style.css"),
  options: {
    ...
    prefixer: {
      prefix: ".ck-content",
      // Replace ":root" with prefix.
      transform: (prefix, selector, prefixedSelector) => {
        if (selector.includes(":root")) {
          return selector.replace(/:root/g, prefix);
        }
        return prefixedSelector;
      }
    }
  }
}

Proposed resolution
Deprecate the prefix option in favor of prefixer which holds the entire prefixer config.
Run the plugin only when one of these options is passed.

ok, i am going to recreate the same pr on bcl directly, the solution seems good on my side

Why? Sorry but this isn't promoting contributions... no cherry picking...
I understand you may need to further refine the code or create a dedicated branch but in this case the code is exactly the same between PRs...

Recreated because of checks - in order to have the tests pass.. sorry for this. I saw you got 500 internal server error..
https://github.com/openeuropa/bootstrap-component-library/pull/610


QBImagePicker/QBAssetsViewController.m:198:31: error: no known class method for selector 'labelColor'

I am getting this error while compiling my react-native app.
I am using react-native-image-crop-picker and that library uses QBImagePicker.
Any help is appreciated!
https://github.com/ivpusic/react-native-image-crop-picker/issues/1157
Versions:
react-native-cli: 2.0.1
react-native: 0.61.1
<EMAIL_ADDRESS>cocoapods: 1.8.3
Platform: iOS 12.2
Error:
QBImagePicker/QBAssetsViewController.m:198:31: error: no known class method for selector 'labelColor'
Podfile:
platform :ios, '9.0'
use_frameworks!
target 'myProject' do
pod 'React', :path => '../node_modules/react-native/'
pod 'React-Core', :path => '../node_modules/react-native/'
pod 'React-Core/DevSupport', :path => '../node_modules/react-native/'
pod 'React-RCTActionSheet', :path => '../node_modules/react-native/Libraries/ActionSheetIOS'
pod 'React-RCTAnimation', :path => '../node_modules/react-native/Libraries/NativeAnimation'
pod 'React-RCTBlob', :path => '../node_modules/react-native/Libraries/Blob'
pod 'React-RCTImage', :path => '../node_modules/react-native/Libraries/Image'
pod 'React-RCTLinking', :path => '../node_modules/react-native/Libraries/LinkingIOS'
pod 'React-RCTNetwork', :path => '../node_modules/react-native/Libraries/Network'
pod 'React-RCTSettings', :path => '../node_modules/react-native/Libraries/Settings'
pod 'React-RCTText', :path => '../node_modules/react-native/Libraries/Text'
pod 'React-RCTVibration', :path => '../node_modules/react-native/Libraries/Vibration'
pod 'React-Core/RCTWebSocket', :path => '../node_modules/react-native/'
pod 'React-cxxreact', :path => '../node_modules/react-native/ReactCommon/cxxreact'
pod 'React-jsi', :path => '../node_modules/react-native/ReactCommon/jsi'
pod 'React-jsiexecutor', :path => '../node_modules/react-native/ReactCommon/jsiexecutor'
pod 'React-jsinspector', :path => '../node_modules/react-native/ReactCommon/jsinspector'
pod 'Yoga', :path => '../node_modules/react-native/ReactCommon/yoga'
pod 'DoubleConversion', :podspec => '../node_modules/react-native/third-party-podspecs/DoubleConversion.podspec'
pod 'glog', :podspec => '../node_modules/react-native/third-party-podspecs/glog.podspec'
pod 'Folly', :podspec => '../node_modules/react-native/third-party-podspecs/Folly.podspec'
pod 'RNCharts', :path => '../node_modules/react-native-charts-wrapper'
pod 'QBImagePickerController', :path => '../node_modules/react-native-image-crop-picker/ios/QBImagePicker/QBImagePickerController.podspec'
pod 'RNImageCropPicker', :path => '../node_modules/react-native-image-crop-picker'
pod 'Firebase/Core', '~> 6.3.0'
pre_install do |installer|
installer.analysis_result.specifications.each do |s|
s.swift_version = '5.0' unless s.swift_version
end
end
end

I see the same issue

+1

+1

+1


pbwatcher returns null

Hi @dmayer
I am running pbwatcher under iOS 9.2 64bit but pbwatcher just seems to return null continually even when I copy/cut and paste stuff. Do you have any idea why? Thanks.
root# ./pbwatcher 1
2016-09-29 09:27:51.175 pbwatcher[5132:232424] general:(null)
2016-09-29 09:27:52.290 pbwatcher[5132:232424] general:(null)
2016-09-29 09:27:53.378 pbwatcher[5132:232424] general:(null)
2016-09-29 09:27:54.453 pbwatcher[5132:232424] general:(null)
2016-09-29 09:27:55.496 pbwatcher[5132:232424] general:(null)
2016-09-29 09:27:56.593 pbwatcher[5132:232424] general:(null)
2016-09-29 09:27:57.697 pbwatcher[5132:232424] general:(null)
2016-09-29 09:27:58.716 pbwatcher[5132:232424] general:(null)
2016-09-29 09:27:59.758 pbwatcher[5132:232424] general:(null)
2016-09-29 09:28:00.797 pbwatcher[5132:232424] general:(null)
2016-09-29 09:28:01.862 pbwatcher[5132:232424] general:(null)
2016-09-29 09:28:02.886 pbwatcher[5132:232424] general:(null)

Having the same issue on iOS 9.3.3!

@tghosth  did you ever manage to fix/address the issue?


Transfer Ceph Best Practice badge

Hi,
Could you please transfer the Ceph badge from current owner @Thingee (https://www.bestpractices.dev/en/users/2906) to me?
This is a due to the job change of current owner.
Thanks!

Transfer complete.


Remove coverage collection from web3 client

Because runtimeverification/firefly#585 is passing, this is ready for review.


Feature request: Enable customized order of the arguments on the usage text

Awesome library! Definitely way handier than Commons CLI. Love the fact that it's always possible to resort to Commons CLI if needed, good job!
The following feature would be very cool. Currently the usage text displays the arguments and flags in alphabetical order, eg:

usage: java -jar .jar -b  -f  [-h] -z
-b,--bar    roflmao
-f,--foo    yadayada
-h,--help        show this message
-z,--zoo    blabla

It would be great to have a way to specify the order of the options, for instance:
    new CmdLineAppBuilder(args)
        .withRequiredArg(
            "bar", "roflmao", 2)
        .withRequiredArg(
            "foo", "yadayada", 3)
        .withRequiredArg(
            "zoo", "blabla", 1)
        .withEntryPoint(
            cmdLine -> {
              if (cmdLine.hasOption("flag")) {
                // do something
              }
              if (cmdLine.hasOption("a")) {
                // do something
              }
            })
        .build().run();

would produce the following usage text:

usage: java -jar .jar -b  -f  [-h] -z
-z,--zoo    blabla
-b,--bar    roflmao
-f,--foo    yadayada
-h,--help        show this message

Thanks.
Yes, it is possible - apache cli allows to set up custom comparator for options
Only one thing.. I believe it is better to have the natural order (without the new parameter that will pollute the interface), so you will need just move an argument declaration in the right place for the right order.
Will implement soon

implemented in the 2.2.0 version


Loaded DeepSpeech model not displaying the targeted transcript despite being exact same as one used in attack

Hello,
I recently stumbled upon a peculiar issue regarding adversarial audio and model prediction. I have PyTorch load the same model twice and create the estimator wrapper for the both. I then generate an example for one model and have both models label the example.
For the model used in the attack, the model outputs the targeted label. However, the second model (loaded the same way from the same path) outputs the original label. I am wondering if this behavior is expected as I'm uncertain if this is a bug or not.
Below is some code to recreate the issue:
# load models
pretrained_model1 = PyTorchDeepSpeech(pretrained_model='librispeech')
pretrained_model2 = PyTorchDeepSpeech(pretrained_model='librispeech')

# perform attack
x_original = [ ... ]
global_max_length = ...
attack_phrase = np.array([ ... ])

att = ImperceptibleASRPyTorch(
        estimator=pretrained_model1,
        initial_eps=.0075,
        max_iter_1st_stage=1000,
        max_iter_2nd_stage=100,
        learning_rate_1st_stage=.00003,
        learning_rate_2nd_stage=.00003,
        optimizer_1st_stage=torch.optim.Adam,
        optimizer_2nd_stage=torch.optim.Adam,
        global_max_length=global_max_length,
        initial_rescale=1.0,
        rescale_factor=0.8,
        num_iter_adjust_rescale=20,
        initial_alpha=0.01,
        increase_factor_alpha=1.2,
        num_iter_increase_alpha=20,
        decrease_factor_alpha=0.8,
        num_iter_decrease_alpha=20,
        batch_size=len(x_original),
        use_amp=True,
        opt_level="O1"
    )

# generate the example
x_adv = att.generate(x_original, attack_phrase)

# get predictions
pretrain1_output = pretrained_model1.predict(np.array([x_adv[0]]), batch_size=1, transcription_output=True)
print(pretrain1_output)

pretrain2_output = pretrained_model2.predict(np.array([x_adv[0]]), batch_size=1, transcription_output=True)
print(pretrain2_output) # should be the same as pretrain1_output?

For the record, I am using version 1.5.2 of ART
Regards

Hi @scfang Does this also happen with ART 1.5.3 or 1.6.1?

Hi @beat-buesser,
I am receiving this error when I attempt to run the attack on 1.5.3
NotImplementedError                       Traceback (most recent call last)
<ipython-input-8-083f3f9e074b> in <module>
     43 # generate the adversaries
     44 start = time.time()
---> 45 x_adv = att.generate(x_original, attack_phrase)
     46 end = time.time()
     47 seconds = end - start

~/anaconda3/envs/amazonei_pytorch_latest_p36/lib/python3.6/site-packages/art/attacks/attack.py in replacement_function(self, *args, **kwargs)
     72                 if len(args) > 0:
     73                     args = tuple(lst)
---> 74                 return fdict[func_name](self, *args, **kwargs)
     75
     76             replacement_function.__doc__ = fdict[func_name].__doc__

~/anaconda3/envs/amazonei_pytorch_latest_p36/lib/python3.6/site-packages/art/attacks/evasion/imperceptible_asr/imperceptible_asr_pytorch.py in generate(self, x, y, **kwargs)
    267
    268             # Then compute the batch
--> 269             adv_x_batch = self._generate_batch(adv_x[batch_index_1:batch_index_2], y[batch_index_1:batch_index_2])
    270
    271             for i in range(len(adv_x_batch)):

~/anaconda3/envs/amazonei_pytorch_latest_p36/lib/python3.6/site-packages/art/attacks/evasion/imperceptible_asr/imperceptible_asr_pytorch.py in _generate_batch(self, x, y)
    323         # Second stage of attack
    324         successful_adv_input_2nd_stage = self._attack_2nd_stage(
--> 325             x=x, y=y, theta_batch=theta_batch, original_max_psd_batch=original_max_psd_batch
    326         )
    327

~/anaconda3/envs/amazonei_pytorch_latest_p36/lib/python3.6/site-packages/art/attacks/evasion/imperceptible_asr/imperceptible_asr_pytorch.py in _attack_2nd_stage(self, x, y, theta_batch, original_max_psd_batch)
    553                 local_delta_rescale=local_delta_rescale,
    554                 theta_batch=theta_batch,
--> 555                 original_max_psd_batch=original_max_psd_batch,
    556             )
    557

~/anaconda3/envs/amazonei_pytorch_latest_p36/lib/python3.6/site-packages/art/attacks/evasion/imperceptible_asr/imperceptible_asr_pytorch.py in _forward_2nd_stage(self, local_delta_rescale, theta_batch, original_max_psd_batch)
    623         for i in range(len(theta_batch)):
    624             psd_transform_delta = self._psd_transform(
--> 625                 delta=local_delta_rescale[i, :], original_max_psd=original_max_psd_batch[i]
    626             )
    627

~/anaconda3/envs/amazonei_pytorch_latest_p36/lib/python3.6/site-packages/art/attacks/evasion/imperceptible_asr/imperceptible_asr_pytorch.py in _psd_transform(self, delta, original_max_psd)
    800             torch.pow(torch.tensor(10.0), torch.tensor(9.6)).to(self.estimator.device)
    801             / torch.reshape(torch.tensor(original_max_psd).to(self.estimator.device), [-1, 1, 1])
--> 802             * psd
    803         )
    804

~/anaconda3/envs/amazonei_pytorch_latest_p36/lib/python3.6/site-packages/apex/amp/wrap.py in wrapper(*args, **kwargs)
     60             raise NotImplementedError('Do not know how to handle ' +
     61                                       'these types to promote: {}'
---> 62                                       .format(types))
     63     return wrapper
     64

NotImplementedError: Do not know how to handle these types to promote: {'FloatTensor', 'DoubleTensor'}

I have tried running this on Torch 1.4.0 and 1.8.1. I am also on CUDA 11.0 if that is relevant

Hi @scfang Thank you for the updates and great that you got it to work with recent versions of ART.
@minhitbk Have you seen this error with amp before?

@scfang , @beat-buesser : I understood that there are 2 issues here. First, we got 2 different outputs for the same model when being used with attack and not, assumed amp is True. This is expected because amp will change the weights of the model used in the attack by casting its type with shorter bit length to speedup the process.
Second, the error described above could be a potential bug that I will follow. @scfang : Could you please if possible share me a bit of your anonymous data that can reproduce the error?


Using entypo characters instead of img for the tooltip

I'm wondering if you could help me with the code to use an entypo character in my blog theme vs. an image.
The code that shows the correct character is:
I'd love to be able to inject that in instead of the default image.  Let me know what you think!  Thanks in advance!
Josh

I've made some substantial changes now in how this is put together, and it uses a combination of an icon font and CSS instead of an image and JS. you should be able to filter it as you see fit.


Added missing EditView.getUrl parameter

Patch for #871 (EditView, not ListView)

Could you remove the built files from your PR? They make it hard to merge. I'll rebuild the files myself once merged.

:rewind:  It's done

I ask my self if this PR is ready ...
For example a route like this (entity with a composite keys id): /#/crew_members/edit/2501,1947
The id will be passed as string ('2501,1947') at EditView.getUrl, i think the id argument must be an array [2501,1947]
I don't know how to get the id as array from $state.params ...

Thanks!


Roll engine 93aa035dd469..0b014c4ac469 (10 commits)

https://github.com/flutter/engine/compare/93aa035dd469...0b014c4ac469
git log 93aa035dd46999353d8f7fe4be61e6599c794b4b..0b014c4ac469344d5166567f174ef2592cc1b099 --no-merges --oneline
0b014c4ac Roll src/third_party/skia 5f5e1d57a4ce..0ec8ec27b81c (6 commits) (flutter/engine#7703)
2f87713d4 Revert "Disable the persistent cache (#6835)" (flutter/engine#7689)
2c25936ec Roll src/third_party/skia 2462ca36c800..5f5e1d57a4ce (7 commits) (flutter/engine#7699)
ac003c483 Roll src/third_party/skia 0783aca7ba31..2462ca36c800 (2 commits) (flutter/engine#7698)
e26eab83a Roll src/third_party/skia bb60677d5244..0783aca7ba31 (1 commits) (flutter/engine#7697)
ad525410d Roll src/third_party/skia 2d35a1c87553..bb60677d5244 (1 commits) (flutter/engine#7696)
cc27cafb8 Implemented Dark Mode for Android (#25525) (flutter/engine#7488)
9c05cbcfb Roll src/third_party/dart b53dceadaa..5823be65af (5 commits) 5823be65af [vm/compiler] Continued graph checker development (reland) 8231cdb7a3 [gardening] Update status for issue 35854 db7f848632 [vm] Remove dead BigInt code. 35ab1755f4 Support more type propagation for code-as-ui features 569ee07f91 [vm] Cleanup class finalization checks
ec5e6f6ef Ensure dart2js and kernel worker snapshots are copied out of gen dir (flutter/engine#7692)
8b5fa65c4 Roll src/third_party/skia 50ea3c06b80f..2d35a1c87553 (6 commits) (flutter/engine#7693)
The AutoRoll server is located here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md
If the roll is causing failures, please contact the current sheriff (chinmaygarde@google.com), and stop
the roller if necessary.

Trybots failed. These were the failed builds: https://cirrus-ci.com/task/6032079042117632 , https://cirrus-ci.com/task/4906179135275008

Commit queue failed; closing this roll.


[Chores] Move images to docs/images

This PR moves images to docs/images to clean up docs folder for later
use.
Self evaluation:
Build test: [ ]Passed [ ]Failed [X]Skipped
Run test: [ ]Passed [ ]Failed [X]Skipped
Signed-off-by: Jihoon Lee<EMAIL_ADDRESS>

:memo: TAOS-CI Version: 1.4.20191203. Thank you for submitting PR #59. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ec2-54-180-96-14.ap-northeast-2.compute.amazonaws.com/.


Error after deleting user

I got error on deployments page after delete old user.
error in this string: <?php echo e($deployment->user->email); ?>	 because user not exists

I have fixed this bug.
Please try the version 0.42.0.


Results grouped by currency

Trace evaluation results should be returned grouped by currency (currently we only get the values for a single currency - Ada)
See #727

Closed by #770.


[BUG] Closing a tab leads to page crash

Describe the bug
Closing a tab leads to page crash.
To Reproduce
Steps to reproduce the behavior:

Go to explorer page
Click on '+Add new' to add more tabs
Click on any tab before the last tab
Close it

Expected behavior
A clear and concise description of what you expected to happen.
Plugins
Please list all plugins currently enabled.
Screenshots
If applicable, add screenshots to help explain your problem.
Host/Environment (please complete the following information):

OS: [e.g. iOS]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Fixed and merged to 1.2


Update Retry-After to follow the coding conventions

Small update to Retry-After to follow the conventions, removing the use of FiniteDuration for Long instead

Do you mean by the existence of sub-seconds on Instant

Yes, exactly.  Just like the existence of subseconds on Duration.

This is affected by #1287

When #1287 gets approved, I'll backport this on top of that.

I'm just folding this into #1287.


config:consider config change in readstats

close https://github.com/tikv/tikv/issues/12014
Signed-off-by: Jayice<EMAIL_ADDRESS>
What is changed and how it works?

Issue Number: Close https://github.com/tikv/tikv/issues/12014
What's Changed:

add a static var `SPLIT_CONFIG`, in order to take into account the config change in the creation of `Readstats`

Tests

Unit test

Release note
take into account the config change in the creation of `Readstats`

/retest

@andylokandy Hi. Could you PTAL if you have time？😃

Oh, is there anyone who can help review?

/merge

Oh, it seems like some conflicts occurred...I will try to fix it.

@AndreMouche Hi, I've sent a new commit to fix conflicts. PTAL again😆

/merge

/merge

/test


add client scripts for T2

As we're starting to use T2 more and more for integration tests, it should offer client scripts to conveniently access T2 (launch and terminate clusters).
These scripts already exist in PIT and should be extracted from there. PIT is about to be removed once the integration tests are running within the GitHub projects of the respective system under test.

When using this script, make sure to check out the correct branch!

Client script URL is https://raw.githubusercontent.com/stackabletech/t2/client-script/client/t2.py


How to add dependencies from integrationTest of another gradle module.

Hello,
I have a case where I have a common module (ex : C) for the dao and two applications modules.
I need to write integration tests for the two different applications modules (A and B), but I want to have the injector used to initialize the database, in the common module.
How can I make the classes in C accessible from A and B ?
Regards.

Hi,
if I understand you correctly you have an integrationTest testset in C which contains some tests in addition to shared test utils that you would like to reuse in other modules.
First, I would strongly suggest to move the shared code to a separate "test library" that contains no test but only utilities. Otherwise, with the default settings of JUnit or TestNG, you would run those tests in every project that imports the library.
This scenario isn't something that the plugin covers directly. In general (without the plugin) you would follow these steps:

In project C:
a) Create a source set for the test library (e.g. testLib)
b) Create a Jar task that packages the test library (e.g. testLibJar)
c) Create a configuration under which you will publish it (e.g. testLib)
d) Declare an artifact that connects the configuration with the JAR
e) optionally, if in project C you also have tests that depend on the testLib, use the testsets plugin and declare a dependency like this
integrationTestImplementation sourceSets.testLib.output

Now, I suppose you could probably use the test sets plugin to achieve part of this because it covers (a) and (b) for you, by making a testSet that doesn't actually contain any tests. However you would still need to do (c) and (d) yourself.

In projects A and B:
a) use the test sets plugin
b) Declare a dependency like
integrationTestImplementation project(path: ':C', configuration: 'testLib')

So something like this (just jotting down, not tested code ;-) )
In project C
// 1a
sourceSets { testLib }

// 1b
task testLibJar(type: Jar) {
    from sourceSets.testLib.output
}

// 1c
configurations { testLib }

// 1d
artifacts {
    testLib testLibJar
}

// 1e
dependencies {
    integrationTestImplementation sourceSets.testLib.output
}

It could definitely be easier, but this is not really the scenario the plugin tries to address. I've been thinking about how to allow something like test libraries in the plugin, but so far there is no direct support.
Hope that helps!

Hi thank you for your answer.
My project C contains only shared test utils no tests.


Avoidables carried over to profiles where they are invalid

When you select an avoid features option in one profile and then switch to another where this avoidable is not available, the system shows an "Invalid parameter" message, but the avoidable doesn't show in the list so I have to go back to the original profile and deselect it before switching profiles. Ideally the system should strip out the avoidable that isn't valid when the profile is changed.
Here's what I did:

Generate a car route with tollways set as avoid features:

Switch to a non-driving profile

Closed via https://github.com/GIScience/ors-map-client/commit/1d562c6f294c1995cb93ec3f576fea5d4f1b660a


Dot support in identifiers for structure

This is my attempt to fix #93 – I've setup a new stencil template named "structured" and improved the tests to cover these cases, too.
Unfortunately with enums it isn't possible to use the L10n.Settings.NavigationBar.Title dot syntax structure in a clean way. If this is required then we need to basically rewrite a lot of code as we would probably need to use Structs for that (that's is how Laurine works). But given we have enums I tried to still get the advantage of scoped autocompletion and succeeded, the structure now looks like this: L10n.Settings(.NavigationBar(.Title)).
The typing behavior is as good as with normal dot syntax:

type "L10n."
hit down key until autocompletion finds the right key
hit enter
press "." and hit down key until found (via autocompletion)
hit enter
press "." and hit down key until found (via autocompletion)
hit enter
...

Note that I have not changed any of the existing features, so releasing this should be easy as it just adds a new feature. Sometime later in a newer version we could of course use the dot syntax by default but as this will mean people using SwiftGen already will need to update their code this is not a good approach as a first step.
Also there will probably be some edge cases we will encounter once this is used in production (I tested in one of my own projects and it worked like a charm – no issues for me). As an example I think using "strings" as part of a key path like "settings.navigation_bar.strings.title" will be a problem as it is a reserved key. But I think these issues can be fixed over time and don't prevent from releasing a new version so long as this feature is opt-in (which it is right now).
Note that I couldn't find a way to do deep looping within the stencil so I needed to do this manually. The downsides of this approach are that there is a maximum level of depth (currently 5 dots) and that the stencil repeats a lot of code 5 times. I'm not sure if this is even possible in a different way when using the Stencil framework.
I also think it would be good to document how to use this feature somewhere – both, how to change up the commands to generate structured localization enums and how to use it in code. I don't have time right now for this though, just wanted to complete the feature and push it to get some feedback - here it is. :)

Cool thanks! 👍
I'll try to take a look soon (even if I have a busy week but will do my best).
I think maybe we could address this dot-support feature in multiple steps.

The first step being this very PR, providing basic initial support with a single template based on enums, like you did.
Propose an alternate template that use struct instead of enum so we can use x.y.z.t syntax instead of enums with nested associated values .x(.y(.z(.t))). It would be up to the use to choose the template they prefer
Try to implement it using recursion:

either using Stencil's include tag so that the part being recursively repeated could be in a separate file, the root template include it at some point, and the included file could include itself recursively then? (just an idea, not sure it can really work by probably worth a try).
Or implement it using a custom tag (maybe the custom set node I added in GenumKit to extend Stencil could help? Basically it's used to store a part of a stencil expression into a variable and re-execute (re-render) it each time — it can be seen a bit like a local function that you then call when you need it except it doesn't take any argument, but it re-interpret its content each time so can re-interpret inner variables so it might be enough. It potentially could help (in a similar way Stencil's include tag would work, except it would be all in the same file), but again not totally sure it covers everything until we try
Or lastly if SetNode doesn't cover what we need, and doesn't allow us to implement it the way we want, we could consider creating a new custom Stencil node that would let us implement it the way we want

So we are at step 1 with this PR and I think I'll merge it soon once reviewed, we can still imagine steps 2 and 3 as future improvements for future versions, but let's keep it one step at a time

PS: Please add an entry in the CHANGELOG.md to explain your changes and credit yourself :wink:

Sounds all reasonable to me, also added an entry to the Changelog.md and wrote an additional test to ensure everything works fine when no keys are specified in root level.

Cool!
Seems like travis hand and failed to report the CI status, and the branch seems to have conflicts anyway (probably just the CHANGELOG.md entries because I merged other requests in the meantime), preventing me to merge. Could you please rebase your branch on top of master so I can merge the PR? Thx!

Rebased.

@Dschee As explained in the CONTRIBUTING.md file you are helping me put together, I just given you push access to SwiftGen!
This means that you can directly use the repository (instead of keeping using your fork and having to keep it up-to-date at all times — now you can remove your fork and work on SwiftGen directly!) and create branches on the SwiftGen repository, add labels on issues and PR, close issues, etc.
Master is still locked to force us ensure that travis builds our changes and pass to green before we add anything on master, but otherwise you can push on any branch you create on SwiftGen.

Thanks. I'm uploading my contributing branch from my fork then directly to this repo. That'll probably close the MR so I'll have to create a new one then, right? Or is there any trick for such things? :)

Sadly GitHub doesn't allow to modify an already open PR to point to a different fork or branch, so I think you'll have to create a new one indeed.

Let's try, maybe it picks it up automagically ... a moment ...

Didn't work, created a new MR: #149.


Consider disabling back button on final screen(s) if email to court sent successfully?

Need to consider the pros and cons. Could be good to prevent people going back and being confused about whether they really e-mailed or not (though maybe can do something about that on review screen? is there a way to create a variable that is not undone when you go back?). But could be bad if there was actually a problem with the email and they have a good reason to go back and resubmit. Or if they submit in one county, that gets dismissed, and they later want to file in another county?

Emily will discuss with subject matter experts.

This has currently been implemented, still want to check with subject matter experts though.


Add python bindings to vcf2parquet

First off, thank you very much for this awesome rust library !
So I thought it would be nice to have python bindings to integrate this module into more user-friendly code bases ;)
Also I must say I'm pretty new to Rust, so this may not be best practices. For instance, this pull request lacks a github action workflow to publish to PyPI (don't know how to do it anyway).
To make it work:
Create virtual env anywhere you like, activate it and:
pip install maturin
Within that python virtual environment:
# Need pyproject.toml and Cargo.toml in $PWD
cd vcf2parquet-py
# Compile and link... (takes some time)
maturin develop
# Run python/Ipython and type in
# >>> from vcf2parquet import convert_vcf, Compression
# >>> convert_vcf("chr7.ann.vcf.gz","chr7.parquet",read_buffer=8192,compression=Compression.Snappy)

Should work, if it doesn't feel free to report any issue!

Could you rebase you branch on my main branch.
git rebase {my main branch remote name in your repo}/main

Could you add some pytest for vcf2parquet-py module.

Codecov Report
Attention: 66 lines in your changes are missing coverage. Please review.

Comparison is base (50d1f76) 80.30% compared to head (d7195f2) 68.15%.

Files
Patch %
Lines

vcf2parquet-py/src/error.rs
0.00%
47 Missing :warning:

vcf2parquet-py/src/lib.rs
0.00%
19 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@             Coverage Diff             @@
##             main       #8       +/-   ##
===========================================
- Coverage   80.30%   68.15%   -12.15%
===========================================
  Files           7        9        +2
  Lines         396      515      +119
===========================================
+ Hits          318      351       +33
- Misses         78      164       +86

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Private Tracker Support?

Is private tracker support planned? Would be nice to implement in a project I maintain.
https://github.com/HDInnovations/UNIT3D-Community-Edition

@HDVinnie which one do you use? udp or HTTP or ws ?

HTTP

aquatic is quite heavily focused on the open tracker use case. It keeps all swarm data in in-process memory, and additionally, locally to the threads accessing it when possible. A lot of the work that has gone into aquatic has been centered on finding out how to do this in a way that maximizes throughput.
A private tracker will want to persist data in a relational database. I assume that it might also want to perform access control for requests based on data in the database. In other words, you're looking at something much more similar to a typical back-end service: spawn a number of threads which receive HTTP requests, do a number of async calls to a DB and send back responses.
In other words, supporting private trackers is not compatible with aquatic_http as it is currently structured. The only part that could be reused is the functionality for parsing and serializing requests and responses in aquatic_http_protocol crate.
Additionally, I doubt that there is a way to implement private tracker support generic enough to support multiple implementations, so in practice, what would be needed is a custom new Rust tracker for UNIT3D that maybe uses some aquatic code.
Unfortunately, for the time being, I don’t have time to develop such an implementation.

@greatest-ape thanks. Are there any examples, spec related to the private tracker, etc..?

@greatest-ape thanks. Are there any examples, spec related to the private tracker, etc..?

Hmm, I don't know, maybe check out HDVinnie's link?

BEP 27: Private Torrents

@HDVinnie Come to think of it, I think there might be a way to do this.
I'm thinking along the lines of having aquatic call a DB stored procedure on every announce request. It would return a boolean indicating if the request is accepted, i.e., the peer can download/upload the torrent, as well as an optional failure reason or warning message.
The actual contents of the stored procedure would be defined by the whole site surrounding the tracker (e.g. UNIT3D), in order to decouple from the implementation.
Stored procedure name
aquatic_announce_v1 (v1 to ease developing the protocol later)
Stored procedure IN parameters

parameter name
mysql type
notes

info_hash
VARCHAR(20) NOT NULL

user_token
VARCHAR(40) NOT NULL
Would be included in announce URL and be unique per user

peer_id
VARCHAR(20) NOT NULL
Could be used to check that only whitelisted clients are used

event
VARCHAR(9) NOT NULL
Useful for bookkeeping and cheating prevention

uploaded
BIGINT NOT NULL
Necessary for bookkeeping and cheating prevention

downloaded
BIGINT NOT NULL
Necessary for bookkeeping and cheating prevention. Could also be used to determine if user is allowed to proceed with download

Stored procedure OUT parameters

parameter name
mysql type
notes

announce_allowed
BOOLEAN NOT NULL

failure_reason
VARCHAR(255)
optional

warning_message
VARCHAR(255)
optional

aquatic would keep its own swarm state, while the rest of the tracker would be responsible for detecting cheating, managing ratios and so on.
What do you think? Is there anything else that would actually be needed from aquatic?

You can look at ocelot's data layer implementation as a reference: https://github.com/WhatCD/Ocelot/blob/master/db.cpp
But since it was based on the extremely dated and known to be inefficient XBT, it might not be the greatest to replicate.
I like your minimal schema, tbh. Seems like it wouldn't run into some of those performance issues XBT did with recording too much information about snatches and swarm metadata.
I can't think of anything else off the top of my head that the tracker itself would absolutely need to be involved with. For example, when a user of the web UI its integrated with creates a new torrent, it could inspect the .torrent file to get the filesize to record in its own torrent metadata table that would also be used to set leech modifiers (freeleech, doubleseed).
As long as you're setting an integer column as the primary key on the aquatic-maintained table, a web frontend integrating with it would be able to reference that for table joins and efficient indexing.
(Full disclosure: while I've quite a bit of experience with data design and DB operations, my experience with tracker development and operations is limited comparatively.)

Looks good....maybe user_agent might be good to have in there?

Yeah, user_agent makes sense. I added it to the proposal.
I think p_downloaded and p_uploaded should be exactly what the peer reports, not a calculated change since last announce. The reason is that the stored procedure will have to check the DB to determine if it is a restart, fresh download etc.
A few more thoughts on my design:

It has to hit the database on each announce, which will limit performance

On the other hand, even if DB accesses are batched like in Ocelot, a query will have to be executed for each announce request anyway, which might not be much faster
Load could maybe be reduced to some extent by also keeping lists of all allowed user_tokens and info_hashes (updated say every 10 seconds) and filtering by them before checking with the DB. This would mostly make a difference for malicious requests, though, and might be considered a premature optimisation at this point.

Business logic has to be written in SQL (unless stored procedure mostly just stores data and processes it later). This might be unwieldy and means higher DB load than doing calculations in aquatic. However, I don't see another way to keep aquatic decoupled.

@HDVinnie do you have an up to date structure-only UNIT3D DB dump that I could use for testing?
@erincerys Thanks for the feedback. To clarify, I'm not thinking of having an aquatic-maintained table. The stored procedure would be responsible to doing what it wants to do with data it receives (storing it etc) according to internal requirements.

Here's schema dump from latest UNIT3D.
https://anonfiles.com/Tc95BcSbxc/mysql-schema_dump

Closing due to inactivity.


Unexpected behavior of the instruction table.init

Build commands
I compile the code with commit id 892a94fd05f2e95fc68535d763bf67ab1b8e6263.
Platform: Ubuntu 20.04
CPU: amd64
Here we take the warm on JIT mode for example. This issue also exists in warm on other modes (classic interpreter mode, fast interpreter mode, fast JIT mode, multi JIT mode).
compile:
export CC=/usr/lib/llvm-16/bin/clang
export CXX=/usr/lib/llvm-16/bin/clang++
cd product-mini/platforms/linux/;rm -rf build
cmake -DWAMR_BUILD_AOT=1 -DWAMR_BUILD_FAST_JIT=1   -DWAMR_BUILD_REF_TYPES=1 -DCMAKE_BUILD_TYPE=Release  -DWASM_ENABLE_BULK_MEMORY=1 -DWAMR_BUILD_LIBC_WASI=0 -DWAMR_BUILD_LIBC_BUILTIN=1 -DWAMR_BUILD_SIMD=1  -Bbuild

iwasm --heap-size=0  --fast-jit -f to_test <tc_name>

Case
fast_jit_wamr_table.init_no_exception.zip
Actual output:
No exception
Expected output:
There is an exception indicating "out of bounds table access".

In the interpreter mode and fast JIT mode, there is a "memory OOB" exception.

@erxiaozhou thanks for reporting the issue, please try #3031.

Close this issue as it was fixed.


reduce test data size

reduce kidney.zip test data size and update tests around it
tar file is now 4.5 MB and below the 5 MB requirement
> devtools::test("~/SpatialOmicsOverlay/")
ℹ Testing SpatialOmicsOverlay
══ Results ═════════════════════════════════════════════════════════════
Duration: 123.2 s

[ FAIL 0 | WARN 0 | SKIP 0 | PASS 422 ]

> devtools::test_active_file(file = "~/SpatialOmicsOverlay/longtests/testthat/test_coordinateGeneration.R")
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 31 ]

Lower retina reduces vignette size but keeps the image looking clean. Might affect Apple users for blurrier images.

I do not see any differences in blurriness when I toggle between my retina display and my external (non-retina) monitor.

Task:
In the terminal, run R CMD build path/to/SpatialOmicsOverlay/, this will create a tar file and see if that file is <5MB in size.
Result:
File was 4.4M.

Task: Run two test commands.
devtools::test("~/data/mgrout/soo_v1.1_testing/pr_52/SpatialOmicsOverlay/")
# ══ Results ═════════════════════════════════════════════════════════════════════════════════════════════════════
# Duration: 126.6 s
#
# [ FAIL 0 | WARN 0 | SKIP 0 | PASS 422 ]

devtools::test_active_file(file = "~/data/mgrout/soo_v1.1_testing/pr_52/SpatialOmicsOverlay/longtests/testthat/test_coordinateGeneration.R")
# |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed=19s
# |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed=20s
# [ FAIL 0 | WARN 0 | SKIP 0 | PASS 31 ]

Result: No problems

Task: evaluate images in the vignette.
Result: I opened the vignette Rmd and knitted it with the RStudio knit button. I opened up the resulting HTML and looked at it side by side with the Bioconductor vignette HTML. The image resolutions looked the same to me.

Task: run R CMD check path/to/SpatialOmicsOverlay.tar.gz this will run all of the package building tests: knitting vignettes, running tests, running examples, checking documentation, etc.
Result: finished with 1 warning, 1 error, and 4 notes. The error and warning pertain to generating PDFs and Maddy has confirmed this is expected output. See 00check.log contents below.
* using log directory ‘/home/rstudio/data/mgrout/soo_v1.1_testing/pr_52/SpatialOmicsOverlay.Rcheck’
* using R version 4.3.1 (2023-06-16)
* using platform: x86_64-pc-linux-gnu (64-bit)
* R was compiled by
    gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0
    GNU Fortran (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0
* running under: Ubuntu 22.04.2 LTS
* using session charset: UTF-8
* checking for file ‘SpatialOmicsOverlay/DESCRIPTION’ ... OK
* this is package ‘SpatialOmicsOverlay’ version ‘1.1.1’
* package encoding: UTF-8
* checking package namespace information ... OK
* checking package dependencies ... OK
* checking if this is a source package ... OK
* checking if there is a namespace ... OK
* checking for executable files ... OK
* checking for hidden files and directories ... OK
* checking for portable file names ... OK
* checking for sufficient/correct file permissions ... OK
* checking whether package ‘SpatialOmicsOverlay’ can be installed ... OK
* checking installed package size ... NOTE
  installed size is  5.5Mb
  sub-directories of 1Mb or more:
    doc       2.0Mb
    extdata   3.0Mb
* checking package directory ... OK
* checking ‘build’ directory ... OK
* checking DESCRIPTION meta-information ... NOTE
License components which are templates and need '+ file LICENSE':
  MIT
* checking top-level files ... NOTE
File
  LICENSE
is not mentioned in the DESCRIPTION file.
* checking for left-over files ... OK
* checking index information ... OK
* checking package subdirectories ... OK
* checking R files for non-ASCII characters ... OK
* checking R files for syntax errors ... OK
* checking whether the package can be loaded ... OK
* checking whether the package can be loaded with stated dependencies ... OK
* checking whether the package can be unloaded cleanly ... OK
* checking whether the namespace can be loaded with stated dependencies ... OK
* checking whether the namespace can be unloaded cleanly ... OK
* checking loading without being on the library search path ... OK
* checking dependencies in R code ... OK
* checking S3 generic/method consistency ... OK
* checking replacement functions ... OK
* checking foreign function calls ... OK
* checking R code for possible problems ... NOTE
plotSpatialOverlay: no visible binding for global variable ‘xcoor’
plotSpatialOverlay: no visible binding for global variable ‘ycoor’
plotSpatialOverlay: no visible binding for global variable ‘sampleID’
plotSpatialOverlay: no visible binding for global variable ‘Target’
scaleBarPrinting: no visible binding for global variable ‘X’
scaleBarPrinting: no visible binding for global variable ‘Y’
Undefined global functions or variables:
  Target X Y sampleID xcoor ycoor
* checking Rd files ... OK
* checking Rd metadata ... OK
* checking Rd cross-references ... OK
* checking for missing documentation entries ... OK
* checking for code/documentation mismatches ... OK
* checking Rd \usage sections ... OK
* checking Rd contents ... OK
* checking for unstated dependencies in examples ... OK
* checking installed files from ‘inst/doc’ ... OK
* checking files in ‘vignettes’ ... OK
* checking examples ... OK
* checking for unstated dependencies in ‘tests’ ... OK
* checking tests ... OK
  Running ‘testthat.R’
* checking for unstated dependencies in vignettes ... OK
* checking package vignettes in ‘inst/doc’ ... OK
* checking running R code from vignettes ... NONE
  ‘SpatialOmicsOverlay.Rmd’ using ‘UTF-8’... OK
* checking re-building of vignette outputs ... OK
* checking PDF version of manual ... WARNING
LaTeX errors when creating PDF version.
This typically indicates Rd problems.
* checking PDF version of manual without index ... ERROR
Re-running with no redirection of stdout/stderr.
* DONE
Status: 1 ERROR, 1 WARNING, 4 NOTEs


feat: Redis7 as an ISB svc

Fixes: #707
Update the hosted version of Redis to Redis 7.
Tested a Redis 7 ISBSVC with a simple pipeline running:

Deleted Leader Pod and observed Sentinel Pods elect new Leader. Simple pipeline was able to recover after new Leader elected.
Deleted both Slave Pods and observed them get recreated and rediscovered. Afterward, deleted Leader Pod. Simple pipeline was able to recover after new Leader elected.

@juliev0 - can we make the redis default settings like below?
      redis:
        # Default Redis settings, could be overridden by InterStepBufferService specs
        settings:
          # Redis config shared by both master and replicas
          redis: |
            min-replicas-to-write 1
            # Disable RDB persistence, AOF persistence already enabled.
            save ""
            # Enable AOF https://redis.io/topics/persistence#append-only-file
            appendonly yes
            auto-aof-rewrite-percentage 100
            auto-aof-rewrite-min-size 64mb
            maxmemory 512mb
            maxmemory-policy allkeys-lru
          # Special config only used by master
          master: ""
          # Special config only used by replicas
          replica: ""
          # Sentinel config
          sentinel: |
            sentinel down-after-milliseconds mymaster 10000
            sentinel failover-timeout mymaster 2000
            sentinel parallel-syncs mymaster 1

@juliev0 - could you resolve the conflict?

@juliev0 - could you resolve the conflict?

resolved and confirmed everything still works


Warning: Add license to package.json

Removes the warning
npm WARN EPACKAGEJSON<EMAIL_ADDRESS>No license field.

during npm install by adding the MIT license that is used in LICENSE to package.json .

Thank you! I've published...

cool, thanks!


Feedback request on potential name change: i686 to w32devkit

I'm considering renaming the "i686" variant to "w32devkit". On my systems I keep both 64-bit and 32-bit toolchains side-by-side and I rename the latter locally in order to tell them apart. For releases I'd rename both the zip and the top-level directory. I'm also dropping the "mini" variants since the difference has become so little. That makes for these four build artifacts:

w32devkit-x.y.z.zip unzips to w32devkit/
w32devkit-fortran-x.y.z.zip unzips to w32devkit/
w64devkit-x.y.z.zip unzips to w64devkit/
w64devkit-fortran-x.y.z.zip unzips to w64devkit/

I like that the names are simpler and more concise, and that I can unzip 32-bit and 64-bit versions side-by-side without intervention. The 64-bit builds are unchanged, so no disruption there.
My concern is that some people, especially newcomers, have trouble navigating (and even finding!) GitHub's "Releases" page. The default and least specialized build is w64devkit-x.y.z.zip, and by sheer luck, it currently sorts to the top, ignoring source.tar. However, with the rename it will be 5th in the list, counting .sigs, which is a less obvious download. Though perhaps I could kill two birds with one stone by putting download links in README.md — or get around to finally building a simple, official website where I control navigation.
I've long wished it was possible to blend releases, by, say, unzipping one atop the other, or putting both in PATH. The buried toolchain would be accessible by its architecture triple (e.g. i686-w32-mingw32-gcc). Renaming the 32-bit toolchain would make merging an installation more difficult. However, even in the relatively simple PATH case, the buried toolchain is confused about which tools to use, and I haven't yet figured out how to fix it.
(Side note: I've also considered not building Fortran releases — while still maintaining the variant patch — but there are a substantial number of users, notably the llama.cpp community, so it's worth continuing.)
Feedback welcome.

Normal mingw-w64 toolchains (at least the arch package is) are prefixed by (x86_64|i686)-w64-mingw32.
At least, I've never seen an i686-w32-mingw32-gcc. Is there a good reason to change this part of the prefix?
There's a bit of discussion in #36 about a multilib w64devkit, I think having a variant like that would be ideal.
Installing two w64devkits also implies duplicating all the tools within, like the command aliases, busybox, cppcheck, etc.
I think really most users are only going to care about the toolchain in this circumstance.

I'd love this change. I've always found i686 a little too cryptic and confusing, as if deliberately not wanting to be helpful, i.e. minimalistic on cognitive load: my usually too tired brain always stops to reprocess to interpret that tag to make sure it's not what it initially predicts by all those 6es in it. :)
Also, I've been manually suffixing the 32-bit dir with -32 anyway. With this change, if the default dir name would also reflect this name, that would be unnecessary.


No Amount field for NFTs in Treasury

ERC-1155 allows multiple amounts for the same token id, so we'll need an amount column here.

Gnosis doesn't support ERC1155 atm. closing for now