Willow edits

NV ensemble spectra methods and example Jupiter Notebook complete.

My apologies, I don't think I pushed the last button the other day.  Let's see if this works.


[FLINK-3492] Configurable interval between Checkpoints

This PR allows the user to set a minimum interval between checkpoints. This interval is used if a checkpoint is triggered, while another one is still in progress. At that point, the next checkpoint trigger is scheduled for timeOfLastCompletedCheckpoint + minInterval.

Good work @zentol :-) I had some inline comments. It could be good to add a test case for the min pause interval.

The changes from the PR should also be reflected in the documentation and there are no tests for it.

Is it in line with the documentation that this only applies if there are queued requests, i.e. is it OK that a checkpoint might be triggered right after a checkpoint has completed? For example a checkpoint needs X ms to complete and the interval is X+1 ms. There will be no queued requests and the min pause will be ignored, right?

you are correct @uce , looking back that's what the ScheduledTrigger was supposed to do; which i reverted a bit too quickly.

i mean, the ScheduledTrigger was supposed to make sure that a min Interval is kept between the last completed Checkpoint and the next one to trigger.

I've revisited this PR:

ScheduledTrigger remains unchanged
StreamEecutionEnvironment remains unchanged
if the minIntervalBetweenCheckpoints is > 0 a different schedule method is used
scheduleAtFixedRate() is used when minInterval == 0. This is the current behaviour, meaning checkpoints can build up if they aren't processed fast enough
schedule() is used when minInterval > 0. New stuff right here. It schedules the next checkpoint relative to the completion of the last. So we just pop the interval as the period in there and are done.

I had a feeling that this solution was too simple to be correct, and in fact it is the case.
on the bright side, i have a pretty good idea on how to solve this properly now, stay tuned.

Third time's the charm:

the time of the last triggered checkpoint is stored
when a checkpoint is triggered we check whether the interval has passed, if so we reschedule the next checkpoint with a delay of minInterval
test added

The code looks good, but the tests are bound to be unstable. Making the assumption that sleeps actually sleep as long as they should is often violated (seems especially on Travis).
I think we need to find a better way to test this...

How about adjusting the test such that it works the following way:

it polls the number of checkpoint calls (from the atomic integer), say every 100 ms
it starts measuring time from when the count goes to 1
it validates that (numCalls - 2) * pause is always smaller than the elapsed time since starting the time measurement

reworked test to be more reliable.

Very nice!
+1 to merge this

Thanks, Chesnay! Can you merge this to master? :-)

merged in 2e63d1afb2358760109ba0c90011ef565eaae0ff


Filter German datasets

Re-opening #1 with new task focus, which is first to filter all of the existing German datasets.
Also depends on "finishing" a working version of Cleaner.
List of obtained datasets:

[x] Klexikon (own version and on Huggingface)
[x] MLSUM (available on Huggingface)
[x] Wiki_lingua (available on Huggingface)
[x] MassiveSumm (got access via the author to the German subset)
[ ] LegalSum (repository available, but access format unsure, because they internally use a database?)
[x] Own source 2 (access available, but no dedicated German subset parsed (yet))

Aside from this, there is the current state of finishing stat processing on each of those.

[ ] Klexikon. Here, my main concern is that I noticed a small number of duplicates, which should be removed first, and adjusted accordingly on the official dataset on Huggingface.
[ ] MLSUM. Already linked to the post by Philip May before, but it is really essentially to remove duplicates. Also unsure how to deal with changes to the test set (especially the fully extractive samples strongly skew results in favor if lead-based methods).
[ ] Wiki_lingua. Still have to look at the data samples, because the paragraph vs full text bothers me, mainly due to the total length / instructional type of samples.
[ ] MassiveSumm. Preliminary experiments in #48 show that there are some issues with duplicate content.
[ ] LegalSum. Have no access yet, so not more information available than author-reported scores.
[ ] Own source 2. Similar to LegalSum, but some more details known.

Also blocker by #52, given that we want to preserve previous splits.

Also #52 done now, so cleaning is generally possible with the preliminary filters.
On top, though, it would be great to compare the stats before and after filtering. Need to include Jiahui's code first for this, see #50.

Incorporated in #63 .


Error with diagramsnet

Hey,
i want to reference to a diagram generated by diagramsnet. So i create a drawio-file with diagrams.net and want to include it with the following snippet (which works).
.Software architecture
diagramsnet::images/architecture.drawio[]

But if i add an id, so i can reference it later, then i get an 400 Internal Server Error from the kroki server. In the logs i can see, that the request now contains ?id=testref at the end.
.Software architecture
[[testref]]
diagramsnet::images/architecture.drawio[]

<<testref>>

This error is reproducable, if i type this:
diagramsnet::images/architecture.drawio[title="Software architecture",id="testref"]

<<testref>>

If i use another service, in my case excalidraw, the exact same setup works like expected and no error is returned.
I hope you guys can follow my explanation and can fix my problem.
Thank you and have a great day.
Heiss

I think we should fix this issue upstream. Even if diagrams.net service/endpoint does not support options it should ignore them (not produce a 400).
Please open an issue at yuzutech/kroki.
Thanks!

Thank you for your quick response.

Fixed upstream!


CodeWhisper: Implement add kittens

fix: 5
Automated PR for issue #5
make all generated prs have kitten ascii art

CodeWhisper commit information:
Summary
This change adds kitten ASCII art to all generated pull requests. It introduces a new utility function to generate random kitten ASCII art and modifies the PR creation process to include this art in the PR description.
Files we looked at:

/Users/gregoryguterman/code/CodeWhisper/src/ai/generate-ai-response.ts
/Users/gregoryguterman/code/CodeWhisper/src/ai/pull-request-workflow.ts
/Users/gregoryguterman/code/CodeWhisper/src/github/github-api.ts
/Users/gregoryguterman/code/CodeWhisper/src/templates/pr-diff-prompt.hbs
/Users/gregoryguterman/code/CodeWhisper/src/utils/template-utils.ts

Potential Issues:

The ASCII art might not render correctly in all GitHub clients or devices.
The added content might make PR descriptions longer, potentially affecting readability.
Some users might find the ASCII art unprofessional or distracting.

You can reply with instructions such as:

"Revert the last commit"
"Fix the typo in line 10"
"Add a new function to the file"


Support for arrow flight SQL

Currently the only protocol starrocks supporting is MySQL. It is great for interacting for row oriented data. If we want to use for large data transfers and zero copy semantics it is good to support arrow as an alternative. Arrow flight server or adbc will help in getting data efficiently and faster from starrocks.

That's an excellent suggestion! This is on our roadmap, but it will take some time to finish.

Would love to have this!

Is here maybe a timeline for this feature?

+1
Just to put it here some additional context:
There is new modern SQL way to retrieve data in a faster way through Arrow Flight SQL driver.
https://arrow.apache.org/docs/java/flight_sql_jdbc_driver.html#
https://arrow.apache.org/blog/2022/02/16/introducing-arrow-flight-sql/
https://www.postgresql.org/about/news/apache-arrow-flight-sql-adapter-for-postgresql-010-2716/
Some explanations and research:
https://www.vldb.org/pvldb/vol10/p1022-muehleisen.pdf
Some server examples:
https://github.com/voltrondata/flight-sql-server-example
https://www.kamu.dev/blog/2023-09-datafusion-flightsql/ https://github.com/kamu-data/kamu-cli  (Flight SQL working with Tableau in Postrges dialect)

any progress?

This would be incredibly useful for efficient data analysis

Our use case is for streaming Apache Arrow data to the web browser to use the DuckDB WASM (used with mosaic) for real-time analytics dashboards. As of now, we can do that querying from either BigQuery or DuckDB, but not StarRocks.

https://cloud.google.com/bigquery/docs/reference/storage#arrow_schema_details
https://duckdb.org/2021/12/03/duck-arrow.html

We could get the data from StarRocks and convert it ourselves in app code on the server before sending it to the client, but it would be awesome if that could just be passthrough.

not stale


Improve/fix querySelector usage

!= null is implied
querySelectorAll always returns an NodeList, never a null
new Image() is the shortest way to create an <img>
querySelector can check the content of an attribute
document.body is shorter than querySelector

Love that cleaner code. Thanks!


Issue on how to get started

Hello on the main pages there are instruction to apply. I put those in a notebook however they don't work.
with pip install tensorflow as you wrote I got
ValueError: Please install Intel® Optimizations for TensorFlow or MKL enabled TensorFlow from source code within version >=1.14.0 and <=2.8.0.
2022-05-20 14:25:56 [ERROR] Specified timeout or max trials is reached! Not found any quantized model which meet accuracy goal. Exit.
Here https://www.intel.com/content/www/us/en/developer/articles/guide/optimization-for-tensorflow-installation-guide.html
it is said to perform
pip install intel-tensorflow==2.8.0
but then another error appear
2022-05-20 14:31:58 [ERROR] Specified timeout or max trials is reached! Not found any quantized model which meet accuracy goal. Exit.
Could you be please precise
default_netcompressor.zip
so that results are reproduceable ?
Thanks

Hello. pls ensure the framework version you are using is the one INC supports.
from the error log "Please install Intel® Optimizations for TensorFlow or MKL enabled TensorFlow from source code within version >=1.14.0 and <=2.8.0", you didn't install official tensorflow with oneDNN enabled or intel tensorflow.
as for the second issue, could you pls paste the whole log after you installed intel-tensorflow 2.8.0?

Guys I using your instructions on the main page https://github.com/intel/neural-compressor and they don't work as you wrote
the 1st issue diasappeared with pip install intel-tensorflow==2.8.0
the second issue the full log is (by the way you have the notebook here so you can reproduce
default_netcompressor.zip
2022-05-20 14:31:52 [INFO] Generating grammar tables from /usr/lib/python3.7/lib2to3/Grammar.txt
2022-05-20 14:31:53 [INFO] Generating grammar tables from /usr/lib/python3.7/lib2to3/PatternGrammar.txt
2022-05-20 14:31:53 [WARNING] Force convert framework model to neural_compressor model.
2022-05-20 14:31:53 [WARNING] Output tensor names should not be empty.
2022-05-20 14:31:53 [WARNING] Input tensor names should not be empty.
2022-05-20 14:31:53 [INFO] Generate a fake evaluation function.
2022-05-20 14:31:53 [WARNING] Found possible input node names: ['input'], output node names: ['MobilenetV1/Predictions/Reshape_1'].
2022-05-20 14:31:54 [INFO] ConvertLayoutOptimizer elapsed time: 10.68 ms
2022-05-20 14:31:54 [INFO] Pass GrapplerOptimizer elapsed time: 304.17 ms
2022-05-20 14:31:54 [INFO] Pass SwitchOptimizer elapsed time: 4.09 ms
2022-05-20 14:31:54 [INFO] Pass RemoveTrainingNodesOptimizer elapsed time: 3.5 ms
2022-05-20 14:31:54 [INFO] Pass SplitSharedInputOptimizer elapsed time: 2.0 ms
2022-05-20 14:31:54 [INFO] Pass GraphFoldConstantOptimizer elapsed time: 1.9 ms
2022-05-20 14:31:54 [INFO] Pass FuseColumnWiseMulOptimizer elapsed time: 3.59 ms
WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/tf_utils/util.py:322: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
2022-05-20 14:31:54 [WARNING] From /usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/tf_utils/util.py:322: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
2022-05-20 14:31:54 [INFO] Pass StripUnusedNodesOptimizer elapsed time: 23.08 ms
2022-05-20 14:31:54 [INFO] Pass GraphCseOptimizer elapsed time: 3.5 ms
2022-05-20 14:31:54 [INFO] Pass FoldBatchNormNodesOptimizer elapsed time: 3.52 ms
2022-05-20 14:31:54 [INFO] Pass UpdateEnterOptimizer elapsed time: 1.58 ms
2022-05-20 14:31:54 [INFO] Pass ConvertLeakyReluOptimizer elapsed time: 3.19 ms
2022-05-20 14:31:54 [INFO] Pass ConvertAddToBiasAddOptimizer elapsed time: 3.43 ms
2022-05-20 14:31:54 [INFO] Pass FuseTransposeReshapeOptimizer elapsed time: 3.77 ms
2022-05-20 14:31:54 [INFO] Pass FuseConvWithMathOptimizer elapsed time: 3.53 ms
2022-05-20 14:31:54 [INFO] Pass ExpandDimsOptimizer elapsed time: 13.64 ms
2022-05-20 14:31:54 [INFO] Pass InjectDummyBiasAddOptimizer elapsed time: 4.6 ms
2022-05-20 14:31:54 [INFO] Pass MoveSqueezeAfterReluOptimizer elapsed time: 2.71 ms
2022-05-20 14:31:55 [INFO] Pass Pre Optimization elapsed time: 959.05 ms
2022-05-20 14:31:55 [INFO] Get FP32 model baseline.
2022-05-20 14:31:55 [INFO] Save tuning history to /content/nc_workspace/2022-05-20_14-31-52/./history.snapshot.
2022-05-20 14:31:55 [INFO] FP32 baseline is: [Accuracy: 1.0000, Duration (seconds): 0.0000]
2022-05-20 14:31:55 [CRITICAL] Please set environment variable TF_ENABLE_ONEDNN_OPTS=1 when Tensorflow 2.6.x installed.
2022-05-20 14:31:55 [WARNING] Found possible input node names: ['input'], output node names: ['MobilenetV1/Predictions/Reshape_1'].
2022-05-20 14:31:55 [WARNING] Found possible input node names: ['input'], output node names: ['MobilenetV1/Predictions/Reshape_1'].
2022-05-20 14:31:56 [INFO] Pass Quantization elapsed time: 563.2 ms
2022-05-20 14:31:57 [ERROR] Fail to quantize graph due to fileno.
2022-05-20 14:31:57 [WARNING] Fail to forward with batch size=1, set to 1 now.
2022-05-20 14:31:57 [CRITICAL] Please set environment variable TF_ENABLE_ONEDNN_OPTS=1 when Tensorflow 2.6.x installed.
2022-05-20 14:31:57 [WARNING] Found possible input node names: ['input'], output node names: ['MobilenetV1/Predictions/Reshape_1'].
2022-05-20 14:31:57 [WARNING] Found possible input node names: ['input'], output node names: ['MobilenetV1/Predictions/Reshape_1'].
2022-05-20 14:31:57 [INFO] Pass Quantization elapsed time: 341.39 ms
2022-05-20 14:31:58 [ERROR] Fail to quantize graph due to fileno.
2022-05-20 14:31:58 [ERROR] Unexpected exception UnsupportedOperation('fileno') happened during tuning.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/tensorflow.py", line 514, in quantize
data_loader=data_loader).convert()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/tf_utils/graph_converter.py", line 240, in convert
model = self.quantize()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/tf_utils/graph_converter.py", line 544, in quantize
with CaptureOutputToFile(tmp_dump_file):
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/utils/utility.py", line 374, in init
self.orig_stream_fileno = stream.fileno()
io.UnsupportedOperation: fileno
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/experimental/quantization.py", line 151, in execute
self.strategy.traverse()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/strategy/strategy.py", line 392, in traverse
tune_cfg, self.model, self.calib_dataloader, self.q_func)
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/utils/utility.py", line 242, in fi
res = func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/tensorflow.py", line 529, in quantize
data_loader=data_loader).convert()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/tf_utils/graph_converter.py", line 240, in convert
model = self.quantize()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/tf_utils/graph_converter.py", line 544, in quantize
with CaptureOutputToFile(tmp_dump_file):
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/utils/utility.py", line 374, in init
self.orig_stream_fileno = stream.fileno()
io.UnsupportedOperation: fileno
2022-05-20 14:31:58 [ERROR] Specified timeout or max trials is reached! Not found any quantized model which meet accuracy goal. Exit.

your instructions https://github.com/intel/neural-compressor are not reproduceable as they are
I am not interested on the accuracy
I just need the onnx output model

If I add 2 more cells
An ONNX Example
!pip install onnx==1.9.0 onnxruntime==1.10.0 onnxruntime-extensions
Prepare fp32 model
!wget https://github.com/onnx/models/blob/main/vision/classification/resnet/model/resnet50-v1-12.onnx
from neural_compressor.experimental import Quantization, common
tf.compat.v1.disable_eager_execution()
quantizer = Quantization()
quantizer.model = './resnet50-v1-12.onnx'
dataset = quantizer.dataset('dummy', shape=(1, 224, 224, 3))
quantizer.calib_dataloader = common.DataLoader(dataset)
quantizer.fit()
I got 2022-05-20 15:01:37 [WARNING] Force convert framework model to neural_compressor model.
AssertionError                            Traceback (most recent call last)
 in ()
2 tf.compat.v1.disable_eager_execution()
3 quantizer = Quantization()
----> 4 quantizer.model = './resnet50-v1-12.onnx'
5 dataset = quantizer.dataset('dummy', shape=(1, 224, 224, 3))
6 quantizer.calib_dataloader = common.DataLoader(dataset)
2 frames
/usr/local/lib/python3.7/dist-packages/neural_compressor/model/model.py in get_model_fwk_name(model)
203         if fwk_name != 'NA':
204             break
--> 205     assert fwk_name != 'NA', 'Framework is not detected correctly from model format.'
206
207     return fwk_name
AssertionError: Framework is not detected correctly from model format.

default_netcompressor.zip
here the colab notebook

INC tensorflow backend doesn't support Jupyter notebook yet. it's because Jupyter captures the system io but INC need to get the output of standard io for calibration.
pls don't forget to set TF_ENABLE_ONEDNN_OPTS=1 if you are using official tensorflow. Intel TensorFlow doesn't need to set this micro.
pls strictly follow examples readme instructions for result reproduce, if you still have problem, pls let us know

as for your onnx model error, it fails when loading this onnx model by onnx runtime.
"onnxruntime.capi.onnxruntime_pybind11_state.InvalidProtobuf: [ONNXRuntimeError] : 7 : INVALID_PROTOBUF : Load model from /home/ftian/resnet50-v1-12.onnx failed:Protobuf parsing failed."
It should be some compatibility issue existing in onnx runtime and this onnx model. it's not INC issue.

It should be some compatibility issue existing in onnx runtime and this onnx model. it's not INC issue.
so why you indicate in your main page if instructions are not accurately reproduceable ?

so why you indicate in your main page if instructions are not accurately reproduceable ?

I am totally confused, could you pls let me know which instruction/example is not reproducible?

You are confused because you did not see the notebook I share 2 times putting there the instructions from https://github.com/intel/neural-compressor
As I said a simple colab notebook will be reproduceable while the instructions are not

what I need is

a simple colab notebook
or
a docker to install and run an fp32 network example preferably from pytorch
It seems to me neither the two are available from the main page

Danilo, I saw your notebook and mentioned the root cause in the thread. This is a known issue and was raised before https://github.com/intel/neural-compressor/issues/35
if you are saying instructions of some examples are wrong or not able to reproduce, pls let me know the exact place.
The main page of INC is just used to tell users how to install INC. That's why I am confused on what you say "main page is not reproducible".
Detailed examples pls refer to examples/ directory. those are all reproducible in linux bash env.
as for jupyter notebook, only tensorflow model has such issue. I would suggest you following the instructions in examples/ directory for model downloading and quantization.  you can run same instructions in Juptyer notebook without issue.

@danilopau Hi, pls try wget https://github.com/onnx/models/raw/main/vision/classification/resnet/model/resnet50-v1-12.onnx to download the onnx model file. Sorry for the link error and we will update it soon.

@mengniwang95 thanks
I tried following sequence
!python --version >> Python 3.7.13
pip install neural-compressor
pip install onnx==1.9.0 onnxruntime==1.10.0 onnxruntime-extensions
wget https://github.com/onnx/models/raw/main/vision/classification/resnet/model/resnet50-v1-12.onnx
from neural_compressor.experimental import Quantization, common
quantizer = Quantization()
quantizer.model = './resnet50-v1-12.onnx'
dataset = quantizer.dataset('dummy', shape=(1, 224, 224, 3))
quantizer.calib_dataloader = common.DataLoader(dataset)
quantizer.fit()
and the issue is
2022-05-21 08:08:35 [INFO] NumExpr defaulting to 2 threads.
2022-05-21 08:08:36 [INFO] Generating grammar tables from /usr/lib/python3.7/lib2to3/Grammar.txt
2022-05-21 08:08:36 [INFO] Generating grammar tables from /usr/lib/python3.7/lib2to3/PatternGrammar.txt
2022-05-21 08:08:36 [WARNING] Force convert framework model to neural_compressor model.
AssertionError                            Traceback (most recent call last)
 in ()
4 quantizer = Quantization()
5 quantizer.model = './resnet50-v1-12.onnx'
----> 6 dataset = quantizer.dataset('dummy', shape=(1, 224, 224, 3))
7 quantizer.calib_dataloader = common.DataLoader(dataset)
8 quantizer.fit()
1 frames
/usr/local/lib/python3.7/dist-packages/neural_compressor/experimental/data/datasets/dataset.py in init(self, framework)
128                              "mxnet", "onnxrt_qlinearops", "onnxrt_integerops",
129                              "pytorch", "pytorch_ipex", "pytorch_fx", "engine"],
--> 130                              "framework support tensorflow pytorch mxnet onnxrt engine"
131         self.datasets = framework_datasetsframework.datasets
132
AssertionError: framework support tensorflow pytorch mxnet onnxrt engine

Hi @danilopau , for onnx model, we support 3 frameworks: onnxrt_qlinearops & onnxrt_qintegerops & onnxrt_qdqops, they will use different int8 ops and INC asks user to choose one of them.
Currently model type detection can only confirm your model is an onnx model and set the framework type to 'onnxruntime' and so sorry we don't add datasets for 'onnxruntime'. Thank you for your feedback and I will fix this bug soon.
For a quick fix, you can add 2 lines of code
from neural_compressor import conf
conf.model.framework = 'onnxrt_qlinearops'
quantizer = Quantization(conf)

@mengniwang95
appreciated. I hope to have understood correctly your advice. See blow
python --version >> Python 3.7.13
pip install neural-compressor
pip install onnx==1.9.0 onnxruntime==1.10.0 onnxruntime-extensions
wget https://github.com/onnx/models/raw/main/vision/classification/resnet/model/resnet50-v1-12.onnx
from neural_compressor import conf
from neural_compressor.experimental import Quantization, common
conf.model.framework = 'onnxrt_qlinearops'
quantizer = Quantization(conf)
quantizer.model = './resnet50-v1-12.onnx'
dataset = quantizer.dataset('dummy', shape=(1, 224, 224, 3))
quantizer.calib_dataloader = common.DataLoader(dataset)
quantizer.fit()
unfortunately I got another error.
Question is there a way to get the quantized onnx model even if max trials is reached nor target accuracy was meet.
so that I can debug my importer regardeless of the accuracy please ?

env: TF_ENABLE_ONEDNN_OPTS=1
2022-05-20 15:22:56 [INFO] Generating grammar tables from /usr/lib/python3.7/lib2to3/Grammar.txt
2022-05-20 15:22:57 [INFO] Generating grammar tables from /usr/lib/python3.7/lib2to3/PatternGrammar.txt
2022-05-20 15:22:57 [WARNING] Force convert framework model to neural_compressor model.
2022-05-20 15:22:57 [WARNING] Output tensor names should not be empty.
2022-05-20 15:22:57 [WARNING] Input tensor names should not be empty.
2022-05-20 15:22:57 [INFO] Generate a fake evaluation function.
2022-05-20 15:22:57 [WARNING] Found possible input node names: ['input'], output node names: ['MobilenetV1/Predictions/Reshape_1'].
2022-05-20 15:22:57 [INFO] ConvertLayoutOptimizer elapsed time: 0.43 ms
2022-05-20 15:22:57 [INFO] Pass GrapplerOptimizer elapsed time: 145.32 ms
2022-05-20 15:22:57 [INFO] Pass SwitchOptimizer elapsed time: 3.39 ms
2022-05-20 15:22:57 [INFO] Pass RemoveTrainingNodesOptimizer elapsed time: 1.88 ms
2022-05-20 15:22:57 [INFO] Pass SplitSharedInputOptimizer elapsed time: 2.03 ms
2022-05-20 15:22:57 [INFO] Pass GraphFoldConstantOptimizer elapsed time: 2.38 ms
2022-05-20 15:22:57 [INFO] Pass FuseColumnWiseMulOptimizer elapsed time: 3.59 ms
WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/tf_utils/util.py:322: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
2022-05-20 15:22:57 [WARNING] From /usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/tf_utils/util.py:322: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.

Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/tensorflow.py", line 514, in quantize
data_loader=data_loader).convert()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/tf_utils/graph_converter.py", line 240, in convert
model = self.quantize()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/tf_utils/graph_converter.py", line 544, in quantize
with CaptureOutputToFile(tmp_dump_file):
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/utils/utility.py", line 374, in init
self.orig_stream_fileno = stream.fileno()
io.UnsupportedOperation: fileno
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/experimental/quantization.py", line 151, in execute
self.strategy.traverse()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/strategy/strategy.py", line 392, in traverse
tune_cfg, self.model, self.calib_dataloader, self.q_func)
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/utils/utility.py", line 242, in fi
res = func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/tensorflow.py", line 529, in quantize
data_loader=data_loader).convert()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/tf_utils/graph_converter.py", line 240, in convert
model = self.quantize()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/tf_utils/graph_converter.py", line 544, in quantize
with CaptureOutputToFile(tmp_dump_file):
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/utils/utility.py", line 374, in init
self.orig_stream_fileno = stream.fileno()
io.UnsupportedOperation: fileno
2022-05-20 15:23:00 [ERROR] Specified timeout or max trials is reached! Not found any quantized model which meet accuracy goal. Exit.

I am a little confused, your log shows you are quantizing a tf model but not an onnx model.
you can add
conf.tuning.exit_policy.performance_only = True

@mengniwang95
I just rerun above lines without your latest advice and I confirm the issue
2022-05-21 10:20:21 [INFO] NumExpr defaulting to 2 threads.
2022-05-21 10:20:22 [INFO] Generating grammar tables from /usr/lib/python3.7/lib2to3/Grammar.txt
2022-05-21 10:20:22 [INFO] Generating grammar tables from /usr/lib/python3.7/lib2to3/PatternGrammar.txt
2022-05-21 10:20:23 [WARNING] Force convert framework model to neural_compressor model.
2022-05-21 10:20:26 [INFO] Generate a fake evaluation function.
2022-05-21 10:20:28 [INFO] Get FP32 model baseline.
2022-05-21 10:20:28 [INFO] Save tuning history to /content/nc_workspace/2022-05-21_10-20-22/./history.snapshot.
2022-05-21 10:20:28 [INFO] FP32 baseline is: [Accuracy: 1.0000, Duration (seconds): 0.0000]
2022-05-21 10:20:31 [WARNING] Fail to forward with batch size=1, set to 1 now.
2022-05-21 10:20:33 [ERROR] Unexpected exception InvalidArgument('[ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got invalid dimensions for input: data for the following indices\n index: 1 Got: 224 Expected: 3\n index: 3 Got: 3 Expected: 224\n Please fix either the inputs or the model.') happened during tuning.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/onnxrt.py", line 124, in quantize
quantize_config, tmp_iterations)
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/onnxrt.py", line 334, in _get_quantize_params
quantize_params = augment.dump_calibration()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/ox_utils/onnxrt_mid.py", line 358, in dump_calibration
node_output_names, output_dicts_list = self.get_intermediate_outputs()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/ox_utils/onnxrt_mid.py", line 234, in get_intermediate_outputs
intermediate_outputs.append(session.run(None, ort_inputs))
File "/usr/local/lib/python3.7/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 192, in run
return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got invalid dimensions for input: data for the following indices
index: 1 Got: 224 Expected: 3
index: 3 Got: 3 Expected: 224
Please fix either the inputs or the model.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/experimental/quantization.py", line 151, in execute
self.strategy.traverse()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/strategy/strategy.py", line 392, in traverse
tune_cfg, self.model, self.calib_dataloader, self.q_func)
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/utils/utility.py", line 242, in fi
res = func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/onnxrt.py", line 131, in quantize
quantize_config, calib_sampling_size)
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/onnxrt.py", line 334, in _get_quantize_params
quantize_params = augment.dump_calibration()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/ox_utils/onnxrt_mid.py", line 358, in dump_calibration
node_output_names, output_dicts_list = self.get_intermediate_outputs()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/ox_utils/onnxrt_mid.py", line 234, in get_intermediate_outputs
intermediate_outputs.append(session.run(None, ort_inputs))
File "/usr/local/lib/python3.7/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 192, in run
return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got invalid dimensions for input: data for the following indices
index: 1 Got: 224 Expected: 3
index: 3 Got: 3 Expected: 224
Please fix either the inputs or the model.
2022-05-21 10:20:33 [ERROR] Specified timeout or max trials is reached! Not found any quantized model which meet accuracy goal. Exit.

and following your advice I run
from neural_compressor import conf
from neural_compressor.experimental import Quantization, common
conf.tuning.exit_policy.performance_only = True
conf.model.framework = 'onnxrt_qlinearops'
quantizer = Quantization(conf)
quantizer.model = './resnet50-v1-12.onnx'
dataset = quantizer.dataset('dummy', shape=(1, 224, 224, 3))
quantizer.calib_dataloader = common.DataLoader(dataset)
quantizer.fit()

2022-05-21 10:24:21 [INFO] NumExpr defaulting to 2 threads.
2022-05-21 10:24:22 [INFO] Generating grammar tables from /usr/lib/python3.7/lib2to3/Grammar.txt
2022-05-21 10:24:22 [INFO] Generating grammar tables from /usr/lib/python3.7/lib2to3/PatternGrammar.txt
2022-05-21 10:24:22 [WARNING] Force convert framework model to neural_compressor model.
2022-05-21 10:24:25 [INFO] Generate a fake evaluation function.
2022-05-21 10:24:27 [INFO] Get FP32 model baseline.
2022-05-21 10:24:27 [INFO] Save tuning history to /content/nc_workspace/2022-05-21_10-24-22/./history.snapshot.
2022-05-21 10:24:27 [INFO] FP32 baseline is: [Accuracy: 1.0000, Duration (seconds): 0.0000]
2022-05-21 10:24:30 [WARNING] Fail to forward with batch size=1, set to 1 now.
2022-05-21 10:24:32 [ERROR] Unexpected exception InvalidArgument('[ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got invalid dimensions for input: data for the following indices\n index: 1 Got: 224 Expected: 3\n index: 3 Got: 3 Expected: 224\n Please fix either the inputs or the model.') happened during tuning.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/onnxrt.py", line 124, in quantize
quantize_config, tmp_iterations)
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/onnxrt.py", line 334, in _get_quantize_params
quantize_params = augment.dump_calibration()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/ox_utils/onnxrt_mid.py", line 358, in dump_calibration
node_output_names, output_dicts_list = self.get_intermediate_outputs()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/ox_utils/onnxrt_mid.py", line 234, in get_intermediate_outputs
intermediate_outputs.append(session.run(None, ort_inputs))
File "/usr/local/lib/python3.7/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 192, in run
return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got invalid dimensions for input: data for the following indices
index: 1 Got: 224 Expected: 3
index: 3 Got: 3 Expected: 224
Please fix either the inputs or the model.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/experimental/quantization.py", line 151, in execute
self.strategy.traverse()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/strategy/strategy.py", line 392, in traverse
tune_cfg, self.model, self.calib_dataloader, self.q_func)
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/utils/utility.py", line 242, in fi
res = func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/onnxrt.py", line 131, in quantize
quantize_config, calib_sampling_size)
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/onnxrt.py", line 334, in _get_quantize_params
quantize_params = augment.dump_calibration()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/ox_utils/onnxrt_mid.py", line 358, in dump_calibration
node_output_names, output_dicts_list = self.get_intermediate_outputs()
File "/usr/local/lib/python3.7/dist-packages/neural_compressor/adaptor/ox_utils/onnxrt_mid.py", line 234, in get_intermediate_outputs
intermediate_outputs.append(session.run(None, ort_inputs))
File "/usr/local/lib/python3.7/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 192, in run
return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got invalid dimensions for input: data for the following indices
index: 1 Got: 224 Expected: 3
index: 3 Got: 3 Expected: 224
Please fix either the inputs or the model.
2022-05-21 10:24:32 [ERROR] Specified timeout or max trials is reached! Not found any quantized model which meet accuracy goal. Exit.

@mengniwang95
With that I generated the onnx. Thanks ! But the onnx model is in fp32 not any QConv instantiated into it
from neural_compressor import conf
from neural_compressor.experimental import Quantization, common
conf.model.framework = 'onnxrt_qlinearops'
conf.tuning.exit_policy.performance_only = True
quantizer = Quantization(conf)
quantizer.model = './resnet50-v1-12.onnx'
dataset = quantizer.dataset('dummy', shape=(1, 224, 224, 3))
quantizer.calib_dataloader = common.DataLoader(dataset)
quantizer.fit()

@mengniwang95
with that sequence under Python 3.7.13 I was able to generate optimized, renamed and augmented onnx model
unfortunately these are not quantized. How to achieve an onnx with quantized standard onnx layers please ?
pip install neural-compressor
An ONNX Example
!pip install onnx==1.9.0 onnxruntime==1.10.0 onnxruntime-extensions
Prepare fp32 model
!wget https://github.com/onnx/models/raw/main/vision/classification/resnet/model/resnet50-v1-12.onnx
from neural_compressor import conf
from neural_compressor.experimental import Quantization, common
conf.model.framework = 'onnxrt_qlinearops'
conf.tuning.exit_policy.performance_only = True
quantizer = Quantization(conf)
quantizer.model = './resnet50-v1-12.onnx'
dataset = quantizer.dataset('dummy', shape=(1, 224, 224, 3))
quantizer.calib_dataloader = common.DataLoader(dataset)
quantizer.fit()

@danilopau hi, this error is caused by your input shape, according to the model it should be (1, 3, 224, 224)

@mengniwang95
I did it still the onnx model (renamed, optimized, augmented are float) where are the onnx quantized model saved please ?
Python 3.7.13
An ONNX Example
!pip install onnx==1.9.0 onnxruntime==1.10.0 onnxruntime-extensions
Prepare fp32 model
!wget https://github.com/onnx/models/raw/main/vision/classification/resnet/model/resnet50-v1-12.onnx
from neural_compressor import conf
from neural_compressor.experimental import Quantization, common
conf.model.framework = 'onnxrt_qlinearops'
#conf.tuning.exit_policy.performance_only = True
quantizer = Quantization("./conf.yaml")
quantizer.model = './resnet50-v1-12.onnx'
dataset = quantizer.dataset('dummy', shape=(1, 3, 224, 224))
quantizer.calib_dataloader = common.DataLoader(dataset)
quantizer.fit()

the conf.yaml is
version: 1.0
model:
name: resnet50-v1-12
framework: onnxrt_qlinearops
evaluation:
accuracy:
metric:
topk: 1
tuning:
accuracy_criterion:
relative: 0.89

Did it raise any error info?
Further you should do save operation
model = quantizer.fit()
model.save(output_path)

@mengniwang95 thanks again for your great support.
I did
from neural_compressor import conf
from neural_compressor.experimental import Quantization, common
conf.model.framework = 'onnxrt_qlinearops'
#conf.tuning.exit_policy.performance_only = True
quantizer = Quantization("./conf.yaml")
quantizer.model = './resnet50-v1-12.onnx'
dataset = quantizer.dataset('dummy', shape=(1, 3, 224, 224))
quantizer.calib_dataloader = common.DataLoader(dataset)
model = quantizer.fit()
model.save("./resnet_quantized.onnx")  >> weights are float unfortunately, there are no QConv there
---- log is ---
2022-05-22 14:10:40 [INFO] NumExpr defaulting to 2 threads.
2022-05-22 14:10:43 [INFO] Generating grammar tables from /usr/lib/python3.7/lib2to3/Grammar.txt
2022-05-22 14:10:43 [INFO] Generating grammar tables from /usr/lib/python3.7/lib2to3/PatternGrammar.txt
2022-05-22 14:10:44 [WARNING] Force convert framework model to neural_compressor model.
2022-05-22 14:10:50 [INFO] Generate a fake evaluation function.
2022-05-22 14:10:53 [INFO] Get FP32 model baseline.
2022-05-22 14:10:53 [INFO] Save tuning history to /content/drive/MyDrive/Colab/neuralcompressor/onnx/nc_workspace/2022-05-22_14-10-43/./history.snapshot.
2022-05-22 14:10:53 [INFO] FP32 baseline is: [Accuracy: 1.0000, Duration (seconds): 0.0000]
2022-05-22 14:11:07 [INFO] |*Mixed Precision Statistics|
2022-05-22 14:11:07 [INFO] +----------------------+--------+-------+
2022-05-22 14:11:07 [INFO] |       Op Type        | Total  |  INT8 |
2022-05-22 14:11:07 [INFO] +----------------------+--------+-------+
2022-05-22 14:11:07 [INFO] |         Conv         |   53   |   53  |
2022-05-22 14:11:07 [INFO] |        MatMul        |   1    |   1   |
2022-05-22 14:11:07 [INFO] |       MaxPool        |   1    |   1   |
2022-05-22 14:11:07 [INFO] |  GlobalAveragePool   |   1    |   1   |
2022-05-22 14:11:07 [INFO] |         Add          |   17   |   17  |
2022-05-22 14:11:07 [INFO] |    QuantizeLinear    |   2    |   2   |
2022-05-22 14:11:07 [INFO] |   DequantizeLinear   |   2    |   2   |
2022-05-22 14:11:07 [INFO] +----------------------+--------+-------+
2022-05-22 14:11:08 [INFO] Pass quantize model elapsed time: 14732.17 ms
2022-05-22 14:11:08 [INFO] Tune 1 result is: [Accuracy (int8|fp32): 1.0000|1.0000, Duration (seconds) (int8|fp32): 0.0000|0.0000], Best tune result is: [Accuracy: 1.0000, Duration (seconds): 0.0000]
2022-05-22 14:11:08 [INFO] |Tune Result Statistics|
2022-05-22 14:11:08 [INFO] +--------------------+----------+---------------+------------------+
2022-05-22 14:11:08 [INFO] |     Info Type      | Baseline | Tune 1 result | Best tune result |
2022-05-22 14:11:08 [INFO] +--------------------+----------+---------------+------------------+
2022-05-22 14:11:08 [INFO] |      Accuracy      | 1.0000   |    1.0000     |     1.0000       |
2022-05-22 14:11:08 [INFO] | Duration (seconds) | 0.0000   |    0.0000     |     0.0000       |
2022-05-22 14:11:08 [INFO] +--------------------+----------+---------------+------------------+
2022-05-22 14:11:08 [INFO] Save tuning history to /content/drive/MyDrive/Colab/neuralcompressor/onnx/nc_workspace/2022-05-22_14-10-43/./history.snapshot.
2022-05-22 14:11:08 [INFO] Specified timeout or max trials is reached! Found a quantized model which meet accuracy goal. Exit.
2022-05-22 14:11:08 [INFO] Save deploy yaml to /content/drive/MyDrive/Colab/neuralcompressor/onnx/nc_workspace/2022-05-22_14-10-43/deploy.yaml

@danilopau  Hi, sorry for my late reply, I tried in my local env with your code + yaml and the saved model has int8 nodes like below:

Actually the process is same as you set in colab. Just install needed py pkgs


[Deque] check deque equality with buffer identity

Background
https://github.com/apple/swift-collections/pull/335
Similar to what we added for OrderedDictionary, we can try and optimize our Deque equality checks for the case where two Deque instances point to the same identity.
Unlike OrderedDictionary, our Deque is not implemented on ContiguousArray. We can try and mimic the approach from ContiguousArray here in Deque by checking against the identity of the underlying ManagedBufferPointer instance.

Changes
From:
public static func ==(left: Self, right: Self) -> Bool {
  left.elementsEqual(right)
}

To:
let lhsCount = left.count
if lhsCount != right.count {
  return false
}

// Test referential equality.
if lhsCount == 0 || left._storage._buffer.buffer === right._storage._buffer.buffer {
  return true
}

return left.elementsEqual(right)

Test Plan
Three new tests are added:

DequeTests.test_equal
DequeTests.test_not_equal
DequeTests.test_equal_elements

Open Questions
Are there any additional edge-casey behavior we should be testing for?

Checklist

[x] I've read the Contribution Guidelines
[x] My contributions are licensed under the Swift license.
[x] I've followed the coding style of the rest of the project.
[x] I've added tests covering all new code paths my change adds to the project (if appropriate).
[x] I've added benchmarks covering new functionality (if appropriate).
[x] I've verified that my change does not break any existing tests or introduce unexplained benchmark regressions.
[ ] I've updated the documentation if necessary.

Here are benchmarks from running locally.

Note to self: letting Deque.SubSequence be the default Slice<Deque> means that it can't conform to Equatable/Hashable, and that is probably a defect in the Standard Library.

@swift-ci test

@lorentey Thanks for reviewing!


Question, need a way to reduce fall damage without disabling it completely

I have a question regarding creating custom powers. I'm trying to create a power that will reduce fall damage and increase safe fall distance, but I couldn't do any of that, the logs say something like:
Identifier "minecraft:fall_damage_multiplier" was not registered in registry "minecraft:attribute"
In general, I'm not sure if I'm writing this name correctly or if the attribute may not be compatible with the player at all. I doubt it's even possible anymore. Has anyone encountered similar problems, and found a solution? Are there any workarounds?
Also i'm on 1.20.1

You can use the origins:modify_damage_taken power type and use the origins:from_falling damage condition type in its damage_condition field to modify the damage taken from fall damage.

You can use the origins:modify_damage_taken power type and use the origins:from_falling damage condition type in its damage_condition field to modify the damage taken from fall damage.

This is brilliant! I have to try it

I also experience problems using origins:set_fall_distance, I want, in addition to decreasing fall damage, to increase the safe fall distance. I don’t understand what to use it to passively change the safe fall distance. I get errors of incorrect or absent power type.

Everything said above worked to partially reduce damage taken, but I decided to use "origin:in_tag" and "minecraft:is_fall" as damage condition.


Nouns as Verbs?

Some words can appear as more than one class (i.e. noun, verb, adjective) within a given line of text.  I am not sure if someone has mentioned this already but it may be wise to create a new function to handle/categorize these kinds of words.  An example can be seen in the PCM Body.hs in the s4_intro paragraph with the line S "instance" +:+ (phrase $ model ^. term) +:+ S "(ODE) that models the solar water heating tank.".  Note that model is used as both a noun and a verb.

And those are different concepts.  Right now, our NamedIdeas are all nouns.  We do not have good representations for modifiers of nouns (aka adjectives), verbs, or verb modifiiers (aka adverbs).  We have not needed them sorely enough.  Looks like we are rapidly getting there!

Close for now. It will come up again when it is necessary/relevant.  Parameterized unitals will likely come before #91 .


[Terra-Clinical-Header] A11y Updates of terra-clinical-header

Summary

Allows users to be able to select a heading level, so that it does not have accessibility assumptions or defaults that create incorrect context for users and my application meets WCAG Success Criterion 1.3.1 (Info and Relationships) plus Success Criterion 1.3.2 (Meaningful Sequence).
Allows users to be able to add id to the header to be used with ‘aria-labelledby’ to associate headings with the corresponding page region
This PR includes changes from #792 , #794

Closes # UXPLATFORM-8173
Deployment Link

https://terra-clinical-deployed-pr-#.herokuapp.com/
Testing

Tested with VO and Jaws.
Additional Details

Thank you for contributing to Terra.
@cerner/terra

Can you rebase with main since this PR has gotten stale?

Can you rebase with main since this PR has gotten stale?

Done !!

Is this PR still needed?

Is this PR still needed?

yes !! we lost track of this PR while fixing WDIO snapshot issue. I have rebased the branch with main and updated the WDIO snapshots as well to fix build issue

@sdadn  I have addressed all the comments. Can you review again !!


KBC-1513 create typed table

Jira: KBC-1513
Connection : https://github.com/keboola/connection/pull/3201
READ FIRST https://keboola.atlassian.net/browse/KBC-1513?focusedCommentId=30088

[x] New client method(s) has tests
[x] Apiary file is updated
[x] You declared if there is a BC break or not (will affect next release of a client)
[ ] Revert composer.json and update lib before merging

👁

@jirkasemmler pridal som ten test na tie eventy. Tak ked to bude cajk za teba tak sem dam approve .

@fejjo za me cajk :)

Ok dokumentaci dam sem https://github.com/keboola/storage-api-php-client/pull/616


In my Modpack, the Furnance, Brewingstand and Water does not work. In Inventory they get displayed 3D but, when i place them its old modell. I installed the mod on server and client.

Please help
https://www.curseforge.com/minecraft/modpacks/deathdusk

https://imgur.com/a/9ODs2so

Restarted server, it works now.


Remove django-allauth-2fa

This PR remove django-allauth-2fa and replaces it with the inline functions of django-allauth. This reduces the depenecdy footprint and sets us up for further improvements that are planned upstream. Those will reduce the dependency footprint even more and close #4002
Closes #6281

Upstream is not supporting enforced MFA right now - postboning to 0.16.0

OK MFA enforcing can be build by us and upstreamed but https://github.com/pennersr/django-allauth/issues/3830 is a new blocking

Site-Note: It seems I always end up ripping out/replacing the features I write

@sur5r this re-implements the MFA stack and now keeps the user completely in PUI. It is not fully done but I would appreciate if you could test this and report back. Please do this on a test-database, we are migrating the factors to a new DB schema so there could be issues.
Registering factors is not working currently but will be implemented before the merge.

Will do, but might take some days before I get around to it.
@matmair could pass by at PoC desk if you have time?

@sur5r which PoC desk?

At 38c3, first floor. https://38c3.c3nav.de/l/c:1:193.66:143.06/


Error Policies

The top level API consists of:

ErrorPolicy - an existencial wrapper for exception handler.

Another PR will add ErrorPolicies to withServerNode, so that we can change peer state on exceptions thrown by a producer.

TODOS:

@karknu I haven't yet removed BeforeConnect callback in #913

closing in favour of #1127


QR Code Undefined

Overview
During the onboarding session, the QR code event ID was undefined for some reason.

The offending event: U2NSuyBGiSZTW5t0b99f
For some reason the id is undefined in the app's qr code on the front end

This bug happens when accessing an event through the "committees" tab


fix: Remove profile api/cache code/module from the pinning server

fix #81.

Looks good, only some linting problem to fix.
Also I don't think we'll merge this yet for a while 🙂

I npm rm the modules that where not used and npm prune to make sure there are no unused modules, not sure if that's enough to get rid of the useless dependencies of dependencies.

Yeah, I think that's fine @lsenta

Codecov Report

Merging #88 into develop will increase coverage by 14.56%.
The diff coverage is 100%.

@@             Coverage Diff             @@
##           develop     #88       +/-   ##
===========================================
+ Coverage    78.53%   93.1%   +14.56%
===========================================
  Files            4       2        -2
  Lines          191     116       -75
  Branches        20      11        -9
===========================================
- Hits           150     108       -42
+ Misses          35       5       -30
+ Partials         6       3        -3

Impacted Files
Coverage Δ

src/pinning.js
98.92% <100%> (-0.03%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d1a38cc...7ee259b. Read the comment docs.

Yeah going to create a medium term branch for all scaling/reliability stories that we can both merge our PRs into. Will keep in separate branch from develop until release dates become more known.


Find a better way to store the password

The FTP password is stored in the script and will be written to the cron.d-file. From a security point of view, the password should be encrypted and stored in a separate file.

[ ] Investigate a solution
[ ] Implement a better way to store the FTP password

There is a way to do that... You can delete the vars from the script file and add in the on_boot.d script the argument --env-file=/mnt/data/backup/conf.env and store in /mnt/data/backup/conf.env those vars.


TIMEOUT parameters

Hi!
Can you make a separate TIMEOUT parameters for query timeout and database connection timeout?
According to connection_string.go source file, TIMEOUT, CONNECT TIMEOUT and CONNECTION TIMEOUT are all the same thingines.

TIMEOUT URL option is a global timeout for the driver this protect the code from hang up.
if you want to use specific timeout use Context version of sql function ConnectContext, QueryContext, ExecContext, PingContect, ...


Error: Unsupported model type: whisper

When uploading an mp4 file I get the error: failed fetch.
The docker log output shows the following:
2024-01-05 09:53:44 -- Working The PisaSales user interface.mp4 -- 2024-01-05 09:53:44 [Conversion Required] .mp4 file detected - converting to .wav 2024-01-05 09:53:45 [Conversion Processing]: 4KB converted 2024-01-05 09:53:46 [Conversion Processing]: 37376KB converted 2024-01-05 09:53:46 [Conversion Processing]: 54596KB converted 2024-01-05 09:53:46 [Conversion Complete]: File converted to .wav! 2024-01-05 09:58:45 fetch failed 2024-01-05 09:49:08 ┌─────────────────────────────────────────────────────────┐ 2024-01-05 09:49:08 │  Update available 5.3.1 -> 5.7.1                        │ 2024-01-05 09:49:08 │  Run the following to update                            │ 2024-01-05 09:49:08 │    npm i --save-dev prisma@latest                       │ 2024-01-05 09:49:08 │    npm i @prisma/client@latest                          │ 2024-01-05 09:49:08 └─────────────────────────────────────────────────────────┘ 2024-01-05 09:49:09 fatal: not a git repository (or any of the parent directories): .git 2024-01-05 09:49:09 getGitVersion Command failed: git rev-parse HEAD 2024-01-05 09:49:09 fatal: not a git repository (or any of the parent directories): .git 2024-01-05 09:49:09  2024-01-05 10:01:27 Failed to load the native whisper model: Error: Unsupported model type: whisper 2024-01-05 10:01:27     at AutoModelForCTC.from_pretrained (file:///app/collector/node_modules/@xenova/transformers/src/models.js:4667:19) 2024-01-05 10:01:27     at async file:///app/collector/node_modules/@xenova/transformers/src/pipelines.js:2677:33 2024-01-05 10:01:27 node:internal/process/promises:288 2024-01-05 10:01:27             triggerUncaughtException(err, true /* fromPromise */); 2024-01-05 10:01:27             ^ 2024-01-05 10:01:27  2024-01-05 10:01:27 Error: Unsupported model type: whisper 2024-01-05 10:01:27     at AutoModelForCTC.from_pretrained (file:///app/collector/node_modules/@xenova/transformers/src/models.js:4667:19) 2024-01-05 10:01:27     at async file:///app/collector/node_modules/@xenova/transformers/src/pipelines.js:2677:33 2024-01-05 10:01:27  2024-01-05 10:01:27 Node.js v18.19.0
I am running anythingllm with my Apple M1 Max running macOS 14.2.1 (23C71)

Related: https://github.com/xenova/transformers.js/issues/314

Can you shell into your container and see if there are any files in /app/server/storage/models/Xenova/whisper-small. The model should only download when needed for the first time and be loaded from the disk on subsequent runs. However, if for whatever reason HuggingFace blocks the download, then the model would fail to download and thus, inference would fail
I do not have an M-series chip so I am unable to replicate this issue currently. We have the "patched" version of transformers running already so that should be resolved.

I get something similar fetch failed in my docker logs as well on an M1 Pro running macOS 12.4
2024-01-05 09:45:41 -- Working jfk.wav -- 2024-01-05 09:45:41 [INFO] The native whisper model has never been run and will be downloaded right now. Subsequent runs will be faster. (~250MB) 2024-01-05 09:45:41  2024-01-05 09:45:41  2024-01-05 09:46:17 Failed to load the native whisper model: TypeError: fetch failed 2024-01-05 09:46:17     at Object.fetch (node:internal/deps/undici/undici:11730:11) 2024-01-05 09:46:17     at async getModelFile (file:///app/collector/node_modules/@xenova/transformers/src/utils/hub.js:471:24) 2024-01-05 09:46:17     at async getModelJSON (file:///app/collector/node_modules/@xenova/transformers/src/utils/hub.js:575:18) 2024-01-05 09:46:17     at async AutoProcessor.from_pretrained (file:///app/collector/node_modules/@xenova/transformers/src/processors.js:1752:44) 2024-01-05 09:46:17     at async Promise.all (index 2) 2024-01-05 09:46:17     at async loadItems (file:///app/collector/node_modules/@xenova/transformers/src/pipelines.js:2694:5) 2024-01-05 09:46:17     at async pipeline (file:///app/collector/node_modules/@xenova/transformers/src/pipelines.js:2640:19) 2024-01-05 09:46:17     at async LocalWhisper.client (/app/collector/utils/WhisperProviders/localWhisper.js:34:14) { 2024-01-05 09:46:17   cause: SocketError: other side closed 2024-01-05 09:46:17       at TLSSocket.onSocketEnd (node:internal/deps/undici/undici:8280:26) 2024-01-05 09:46:17       at TLSSocket.emit (node:events:529:35) 2024-01-05 09:46:17       at endReadableNT (node:internal/streams/readable:1400:12) 2024-01-05 09:46:17       at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { 2024-01-05 09:46:17     code: 'UND_ERR_SOCKET', 2024-01-05 09:46:17     socket: { 2024-01-05 09:46:17       localAddress: '<IP_ADDRESS>', 2024-01-05 09:46:17       localPort: 40518, 2024-01-05 09:46:17       remoteAddress: undefined, 2024-01-05 09:46:17       remotePort: undefined, 2024-01-05 09:46:17       remoteFamily: undefined, 2024-01-05 09:46:17       timeout: undefined, 2024-01-05 09:46:17       bytesWritten: 263, 2024-01-05 09:46:17       bytesRead: 0 2024-01-05 09:46:17     } 2024-01-05 09:46:17   } 2024-01-05 09:46:17 } 2024-01-05 09:46:17 node:internal/process/promises:288 2024-01-05 09:46:17             triggerUncaughtException(err, true /* fromPromise */); 2024-01-05 09:46:17             ^ 2024-01-05 09:46:17  2024-01-05 09:46:17 TypeError: fetch failed 2024-01-05 09:46:17     at Object.fetch (node:internal/deps/undici/undici:11730:11) 2024-01-05 09:46:17     at async getModelFile (file:///app/collector/node_modules/@xenova/transformers/src/utils/hub.js:471:24) 2024-01-05 09:46:17     at async getModelJSON (file:///app/collector/node_modules/@xenova/transformers/src/utils/hub.js:575:18) 2024-01-05 09:46:17     at async AutoProcessor.from_pretrained (file:///app/collector/node_modules/@xenova/transformers/src/processors.js:1752:44) 2024-01-05 09:46:17     at async Promise.all (index 2) 2024-01-05 09:46:17     at async loadItems (file:///app/collector/node_modules/@xenova/transformers/src/pipelines.js:2694:5) 2024-01-05 09:46:17     at async pipeline (file:///app/collector/node_modules/@xenova/transformers/src/pipelines.js:2640:19) 2024-01-05 09:46:17     at async LocalWhisper.client (/app/collector/utils/WhisperProviders/localWhisper.js:34:14) { 2024-01-05 09:46:17   cause: SocketError: other side closed 2024-01-05 09:46:17       at TLSSocket.onSocketEnd (node:internal/deps/undici/undici:8280:26) 2024-01-05 09:46:17       at TLSSocket.emit (node:events:529:35) 2024-01-05 09:46:17       at endReadableNT (node:internal/streams/readable:1400:12) 2024-01-05 09:46:17       at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { 2024-01-05 09:46:17     code: 'UND_ERR_SOCKET', 2024-01-05 09:46:17     socket: { 2024-01-05 09:46:17       localAddress: '<IP_ADDRESS>', 2024-01-05 09:46:17       localPort: 40518, 2024-01-05 09:46:17       remoteAddress: undefined, 2024-01-05 09:46:17       remotePort: undefined, 2024-01-05 09:46:17       remoteFamily: undefined, 2024-01-05 09:46:17       timeout: undefined, 2024-01-05 09:46:17       bytesWritten: 263, 2024-01-05 09:46:17       bytesRead: 0 2024-01-05 09:46:17     } 2024-01-05 09:46:17   } 2024-01-05 09:46:17 } 2024-01-05 09:46:17  2024-01-05 09:46:17 Node.js v18.19.0

Okay, here is the root cause.
On AMD machines, Docker runs AnythingLLM in an x86 emulator on the container. This emulator overhead causes ONNX models to run very slowly if at all. This slowness also leads to timeouts on the HTTPS call to HF when downloading the models needed to embed,whisper, etc.
The fix for this is to publish an AMD64 image for AnythingLLM so that AMD machines can download an appropriate image and not run in a compatibility mode.

[ ] Implement amd64 build and publish for anythingllm docker image

This support causes issues with lanceDB, who require Rust binaries for it to run.

[ ] We need to bump LanceDB to a version that publishes support for linux_amd64 binaries so that the non-emulated docker container can use LanceDB.


Commit mode suggestions

As a developer I'd like to have suggestions not in the editor, but in the commit mode also.
Clear suggestions can contain:

Overall description of changes has been made in the changed files
More detailed description of change for any file(s)
Possibility to select large or short commit message suggestion

Do you mean this?

@intitni Yep, exactly


Scripts converted to bash

I'm not sure if I should create a pull request, so I thought I just create an issue and let you guys decide if you are interested.
I converted the ssl, discspace and hddtemp script from nodejs to bash, which lightens this whole thing as nodejs would not be required to run this and still get a beautifull MOTD output.
The hddtemp script currently does not display the temperatures in columns, but this could probably implemented.
You can find the scripts here.
What is missing is a simple configuration wizzard, but I added all the configuration options to ./scripts/config.sh which is included by the scripts when they run.

I'm a fan.  All of these work for me as well.

A friend and me converted a few of the scripts into bash, too. They're mostly work in progress, but may be interesting for you.
https://github.com/nullbleed/fancy-motd

Hey @darookee and @dj95 (@nullbleed), you are always welcome to send pullrequests. It's to goal to convert the nodejs scripts into bash as well because it promises to be more performant and less dependencies. My suggestion would be to keep both versions of the scripts in seperate folders.

Sounds good, I'd love to get more bash scripts rather than nodejs. It saves hundreds of milliseconds in runtime and is more performant like you said.
Please open a PR so we can review them for a while before merging :)

I got my branch out of scope, so I recreated it (in case anyone is wondering why the history is different now...) and created a pull request.
When/If that is merged I'd open another PR for services.sh also using the config.sh used by these three scripts. The other shell scripts don't have anything to configure at the moment...

@dj95 I love your version, minimal dependencies, super simple to customize and fast!


Fixes #377 Group create/edit & store create/edit: check if name alrea…

…dy exists
https://github.com/yunity/foodsaving-frontend/issues

@tiltec what's the best way to resolve the below? The logic works, but i'd like to get a preliminary review and some guidance on how to resolve this issue before i continue working on the rest of the tests.
If I use the array syntax to inject dependencies, the linter complains
.directive("groupnameValidator", [GroupService, "$q", (GroupService, $q) => { - 'You should use the function syntax for DI'
If I use the function syntax for DI, i get an error
.directive("groupnameValidator", (GroupService, $q) => { - 'Error: [$injector:strictdi] function(GroupService, $q) is not using explicit annotation and cannot be invoked in strict mode'

Good question! We are using ng-annotate, have a look around in our code base for the ngInject strings that appear to have no functionality.

Codecov Report

Merging #495 into master will decrease coverage by 1.55%.
The diff coverage is 37.03%.

@@            Coverage Diff             @@
##           master     #495      +/-   ##
==========================================
- Coverage   98.37%   96.82%   -1.56%
==========================================
  Files         217      207      -10
  Lines        3020     2962      -58
==========================================
- Hits         2971     2868     -103
- Misses         49       94      +45

Impacted Files
Coverage Δ

.../group/_groupEditCreateForm/groupEditCreateForm.js
12.5% <0%> (-87.5%)
:arrow_down:

.../store/_storeEditCreateForm/storeEditCreateForm.js
11.11% <0%> (-88.89%)
:arrow_down:

client/app/services/group/groupService.spec.js
69.23% <100%> (-30.77%)
:arrow_down:

client/app/services/store/store.spec.js
81.63% <100%> (-18.37%)
:arrow_down:

client/app/services/store/store.service.js
76.47% <50%> (-23.53%)
:arrow_down:

client/app/services/group/groupService.service.js
84.61% <50%> (-11.22%)
:arrow_down:

... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 13d2c99...f614590. Read the comment docs.


3156 - Add Amido.Stacks.Messaging.Azure.ServiceBus dependency to project

[3156 - Add Amido.Stacks.Messaging.Azure.ServiceBus dependency to project]
📲 What
Add a dependency to the Amido.Stacks.Messagings.Azure.ServiceBus library
🤔 Why
That library is used to facilitate functionality for publishing and listening of events to Azure Service Bus

Approved, informally


Bugfix for uninterrupted receive loop

Replace the uninterrupted receive loop (non-blocking) with
a blocking loop with a default timeout of 1000ms.
Restores compatibility with zeromq output plugin for util/zeromq.rb as well, if logstash-plugins/logstash-output-zeromq#13 is merged.
Closes #17
May also resolve #5

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run; then  say 'jenkins, test it'.

Could please someone give me some feedback?

This would be really good if any maintainer could have a look on this.

OK, had a look. The patch looks straight forward. I do need to run some tests so it will take a few days to close this. Thanks!

I'm really interested in this fix. I'm having two zeromq inputs taking up half of the server cpu capacity. So if there is any thing I can do to help, feel free to delegate.

@erikthorselius Have you tried this PR in your setup? Did it solve your problem?

Looks good! I'm new to logstash but here is how I tested. My server is running coreos so I built a docker container.
My dockerfile:
FROM logstash
COPY logstash-input-zeromq-2.0.2.gem /tmp/
RUN /opt/logstash/bin/plugin uninstall logstash-input-zeromq
RUN /opt/logstash/bin/plugin install /tmp/logstash-input-zeromq-2.0.2.gem
CMD ["logstash", "agent"]

gem build logstash-input-zeromq.gemspec
docker build -t logstash-with-zeromq-patch .
docker save -o logstash-with-zeromq-patch logstash-with-zeromq-patch
scp logstash-with-zeromq-patch server:

and on my server
docker load -i logstash-with-zeromq-patch
docker run --rm --name logstash -p 2120:2120 -v /home/logstash/conf:/data logstash-with-zeromq-patch logstash -f /data/logstash.conf

Same process is now down to 1.8% instead of 200%
input {
  zeromq {
    topology => "pushpull"
    tags => ["pushpull"]
  }
  zeromq {
    mode => "client"
    address => ["tcp://<IP_ADDRESS>:7555"]
    topology => "pubsub"
    topic => "sensor"
    tags => ["pubsub"]
  }
}
filter {
  date {
    match => ["unix_timestamp", "UNIX"]
  }
}
output {
  elasticsearch {
    hosts => ["<IP_ADDRESS>:9200"]
  }
}

Given @erikthorselius's success report, I'm OK merging this. Code LGTM except for my one comment about the hash default value (should use hash syntax, not array).
@breml Can you address my comment? I'm happy to merge once done. :)

It fixed my cpu problem but it seems that I don't receive any messages.

I'm at work right now but I can try to debug when I'm home.

@erikthorselius Please see my comment regarding lib/logstash/util/zeromq.rb. This file was shared in Logstash prior to version 1.5. Now both plugins, logstash-input-zeromq and logstash-output-zeromq do provide this file. Therefore it is important to update this file in both plugins at the same time because otherwise it could happen that the logstash-input-zeromq plugin is using the wrong version of this file. (see my reference to logstash-plugins/logstash-output-zeromq#13 above)
@jordansissel this is quiet an annoying issue. I suggest to either really duplicate the code by moving lib/logstash/util/zeromq.rb to lib/logstash/inputrespectively lib/logstash/output or by putting lib/logstash/util/zeromq.rb into a separate gem file, which is really shared and therefore present only once.

 Would it work to uninstall logstash-output-zeromq?

@erikthorselius Uninstalling the logstash-output-zeromq plugin may work. If it doesn't, I recommend to rename ./vendor/bundle/jruby/1.9/gems/logstash-output-zeromq-2.0.2/lib/logstash/util/zeromq.rb for example to zeromq.rb.bak. This will prevent ruby from loading this file which will result in ruby loading the correct one from the logstash-input-zeromq plugin.

I'm so sorry forgetting this. I blame work.
I have tested this again and it works perfect! The cpu goes from 200% to 0.3 % and i still receive logs.

i'm going to start working on refactoring common zeromq code to a new gem, this should ease maintenance of logstash zeromq plugins. i'll update soon.

@breml Thanks for this. Could you please bump the version in the gemspec and add a changelog? I can merge this, then and publish a new version

@suyograo Sure I would like to update gemspec and changelog, but this change should not be merged without either https://github.com/logstash-plugins/logstash-output-zeromq/pull/13 or the refactoring mentioned by @avishai-ish-shalom, because of https://github.com/logstash-plugins/logstash-input-zeromq/pull/18/files#r47354652.

This PR is on hold until logstash-mixin-zeromq is integrated. cc @breml


Added software-properties-common in order to install php on aegis

make demo broken, need to add ppa:ondrej/php in order to make it work on agegis Dockerfile

Thanks!


Bugfix/migration avoid owner as collab + notification changes

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.0 out of 9 committers have signed the CLA.:x: shalini-peeka:x: ajayKrishnankm:x: 6-Sushma:x: eshaani-bhandary:x: boonbaluri:x: aniruddha-k:x: aravindh45:x: Aluri:x: VardhanDevaAluri seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


feat(tds-dropdown): added a clear filter button

Snapshot tests fail since there now is a new button with cross icon where there is a dropdown with a filter.
Added a clear filter button
Added a clear filter button since users gets confused how to use the filter ie that they have to delete the existing text in the input.
Solving issue
How to test
Provide detailed steps for testing, including any necessary setup.

Go to...
Check in...
Run ...

Checklist before submission

[ ] I have added unit tests for my changes (if applicable)
[ ] All existing tests pass
[ ] I have updated the documentation (if applicable)

Suggested test steps

[ ] Browser testing (Chrome, Safari, Firefox)
[ ] Keyboard operability
[ ] Interactive elements have labels.
[ ] Storybook controls
[ ] Design/controls/props is aligned with other components
[ ] Dark/light mode and variants
[ ] Input fields – values should be displayed properly
[ ] Events

Screenshots
Selecting the dropdown

After clicking the reset button

Additional context
Add any other context or feedback requests about the pull-request here.

@finnlake thanks a lot for your contribution! However there are some things to first consider. We don't have a design on this functionality in Figma, so it needs to be looked at by a designer, ping @charlottelorensson. Personally, I don't see why this could not be a nice addition of functionality to the component. However I think we should have some regression tests added, as it is functionality visible only with filter.

@finnlake thanks a lot for this nice contribution!
I did take a look on Figma and comments so this is what I changed:

Rebased the branch with latest changes from develop branch
Color updates for icon and border according to Figma file
Added reset method to handleFilterReset so it clears dropdown menu selection
Improved a11y for using keyboard to navigate clear and menu button

@finnlake, @timrombergjakobsson, @mJarsater please let me know what do you think of it :)

Looks great! Just one thought, now when we hit esc, we close the dropdown, but it also breaks the filter, as we can't resume to type and filter. Should that be intended behavior? If not, maybe we should reset the filter when hitting escape, and not merely set the open state to false?

Looks great! Just one thought, now when we hit esc, we close the dropdown, but it also breaks the filter, as we can't resume to type and filter. Should that be intended behavior? If not, maybe we should reset the filter when hitting escape, and not merely set the open state to false?

Good spotting, the issue exists in the current production code. I would say that we make a ticket out of it and sort it in another PR and leave this one within area of contribution.

Looks great! Just one thought, now when we hit esc, we close the dropdown, but it also breaks the filter, as we can't resume to type and filter. Should that be intended behavior? If not, maybe we should reset the filter when hitting escape, and not merely set the open state to false?

Good spotting, the issue exists in the current production code. I would say that we make a ticket out of it and sort it in another PR and leave this one within area of contribution.

Yes it does! Let's create a ticket for it.

Thank you for accepting the pr!
Nice work @theJohnnyMe , there was a considerable amount of work needed that I didnt think of... :)

Thank you for accepting the pr! Nice work @theJohnnyMe , there was a considerable amount of work needed that I didnt think of... :)

Great work @finnlake!


VRT Audit for Updated Spectrum Tokens

Description
The work to fix deprecated tokens is likely resulting in visually incorrect components. This issue is to record progress on auditing each component visually (with regression testing where possible) to find and fix visual inconsistencies with the current design language and released previously released versions of Spectrum where necessary.
Components that have been verified:

[ ] accordion
[ ] actionbar
[ ] actiongroup
[ ] actionmenu
[ ] asset
[ ] assetlist
[ ] avatar
[ ] breadcrumb
[ ] button
[ ] buttongroup
[ ] calendar
[ ] card
[ ] checkbox
[ ] coachmark
[ ] colorarea
[ ] colorhandle
[ ] colorloupe
[ ] colorslider
[ ] colorwheel
[ ] cyclebutton
[ ] dial
[ ] dialog
[ ] divider
[ ] dropindicator
[ ] dropzone
[ ] fieldgroup
[ ] fieldlabel
[ ] icon
[ ] illustratedmessage
[ ] inputgroup
[ ] label
[ ] link
[ ] menu
[ ] miller
[ ] modal
[ ] overlay
[ ] page
[ ] pagination
[ ] picker
[ ] popover
[ ] progressbar
[ ] progresscircle
[ ] quickaction
[ ] radio
[ ] rating
[ ] search
[ ] searchwithin
[ ] sidenav
[ ] slider
[ ] splitbutton
[ ] splitview
[ ] statuslight
[ ] steplist
[ ] stepper
[ ] switch
[ ] table
[ ] tabs
[ ] tags
[ ] textfield
[ ] thumbnail
[ ] toast
[ ] tooltip
[ ] tray
[ ] treeview
[ ] typography
[ ] underlay
[ ] vars
[ ] well

Closing in favor of the individual issues https://github.com/adobe/spectrum-css/milestone/3


LoadError: cannot load such file -- sinatra/activerecord/rake/activerecord_7

It gets error with activesupport 7
gem "activesupport", "7.0.0"

rake db:migrate
rake aborted!
LoadError: cannot load such file -- sinatra/activerecord/rake/activerecord_7
<internal:/home/fudo/.rbenv/versions/3.0.3/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:148:in `require'
<internal:/home/fudo/.rbenv/versions/3.0.3/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:148:in `require'
<internal:/home/fudo/.rbenv/versions/3.0.3/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:96:in `require'
<internal:/home/fudo/.rbenv/versions/3.0.3/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:96:in `require'
/home/fumi/dev/souls-app/apps/api/Rakefile:3:in `<top (required)>'
(See full trace by running task with --trace)

Rakefile LINE 3
require "sinatra/activerecord/rake"

Having this issue as well.

Using Active Record 6 for now: gem 'activerecord', '~> 6.0'

https://stackoverflow.com/questions/30656858/undefined-method-namespace-for-mainobject-nomethoderror-active-record-r

Thanks for reporting this, it was because there is a new major release of ActiveRecord (7), and the gem haven't add the relevant file for it, I will look into it soon.
https://github.com/sinatra-activerecord/sinatra-activerecord/blob/80e833b31f93eb4d893f0e522e2670b94d583097/lib/sinatra/activerecord/rake.rb#L2

Hi @POPPIN-FUMI  and @lxmrc , I have just added support for ActiveRecord 7 in the latest version of this gem (2.0.25), https://rubygems.org/gems/sinatra-activerecord/versions/2.0.25 .
Would you mind updating the gem to the latest version and see if it works?

@cupnoodle Thank you for your rapid updates. I tried it and it works now 🎊


History Dock leaving an empty hole when it's turned off

Tested versions
v4.2.1.stable.official [b09f793f5]
System information
Godot v4.2.1.stable - macOS 14.3.1 - Vulkan (Forward+) - integrated Apple M1 - Apple M1 (8 Threads)
Issue description
If the History Dock is the only one on the left side of the Godot IDE, turning it off leaves an empty panel that can't be closed.
Steps to reproduce
https://archive.org/download/ShowFlowFleetingWorld/画面収録 2024-02-24 20.28.50.mov
Minimal reproduction project (MRP)
Godot 4.2.1

It didn't leave any empty space when I tried it with v4.3.dev. It may have been fixed in 4.3.

Looks like this is not happening in
v4.3.dev5.official [89f70e98d]
(or at least I can't reproduce it anymore).
Thank you to whoever fixed it!


Resources for artifact input init-containers

Summary
Users should be able to specify the resources for artifact input init-containers
Use Cases
In our multi-tenant environment we strictly enforce quotas and have limitranges to ensure that containers set memory and cpu limits+requests by default.
The limitrange is set low by default to encourage developers to set them up properly, while still providing a low-friction developer experience when iterating on new projects.
In our particular use case a git artifact pulling a large-ish repo is causing problems. Right now you can specify init containers as part of a workflow, but you cannot specify resources used for the init containers responsible for fetching workflow inputs (from git, s3, etc)

Message from the maintainers:
Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.

You should be able to set a global value for this in the controller config-map. Alternatively, you could use podSpecPatch to set it.

ah excellent, that will work perfectly. for those following: https://github.com/argoproj/argo-workflows/blob/master/examples/pod-spec-patch.yaml


unwrap panic on string conversion

Getting below error:
thread 'main' panicked at 'called Option::unwrap()on aNone value', C:\Users\thefl\.cargo\registry\src\github.com-1ecc6299db9ec823\lnk-0.5.1\src\strings.rs:3:34
I don't think I can catch this error on my side but yell at me if I can.
This is the Rust source file I'm using it in: https://github.com/theflakes/reg_hunter/blob/main/src/file.rs. In the get_link_info function.
Thanks again for your work!

Had some time to look at this more. The two programs I'm using lnk in both crash when pointed at: "C:\Users\user\AppData\Roaming\microsoft\windows\start menu\programs\windows powershell\windows powershell.lnk" to analyze.
Both are crashing when opening the lnk file.
    let symlink= match ShellLink::open(link_path) {
        Ok(l) => l,
        Err(_e) => return Ok((link, false))
    };

These are the default lnk files installed with Windows.
Would this fix it?
pub fn trim_nul_terminated_string<S: Into<String>>(s: S) -> String {
    let s = s.into();
    let end_index = s.find('\0').unwrap();
    let end_index = match  s.find('\0') {
        Ok(l) => l,
        Err(_e) => return "".to_string()
    };
    s[..end_index].to_string()
}

Submitted a pull request for this. I don't trust myself so please let me know if I missed anything and thanks.


Cannot read property '_' of undefined

Steps to reproduce
I upgraded my application to feathers 4.3.0.
I guessed that I need to update feathers-vuex to 2.0 to support featherjs/authentication-client 4.3.0.
Expected behavior
I tried to add feathers-vuex 2.0 to my QuasarJS project but I have a syntax error.
Actual behavior
Uncaught TypeError: Cannot destructure property `_` of 'undefined' or 'null'.
    at eval (service-module.getters.js?403b:12)
    at Module../node_modules/feathers-vuex/dist/service-module/service-module.getters.js (app.js:5005)

the line is
const { _ } = commons
Module Loader:
Webpack

Note: the same error now with version 1.7.0

I solved replacing in service-modules.getters.js
import commons from '@feathersjs/commons'
const _ = commons;

with
import { _ } from '@feathersjs/commons'

I've released<EMAIL_ADDRESS>with the import fixed. Please let me know how it goes.

I have a new error:
Class constructor BaseModel cannot be invoked without 'new' at new User (users.service.js?afbf:6)
My code:
import feathersClient, { makeServicePlugin, BaseModel } from '../../feathers-client'
import hooks from './users.hooks.js'

  // Extend the base class
  class User extends BaseModel {
...
}

@marshallswain @adrianofoschi is there a fix/workaround that doesn't require upgrading to 2.0?

@MichaelJCole yes.  A PR with the edit described in this commit: https://github.com/feathers-plus/feathers-vuex/issues/247#issuecomment-525827334

Sure, funny.  I'm writing up an article about a starter project I put together integrating Quasar and Feathers with FeathersVuex, and I can't recommend using your package - it's unreliable, undocumented, and all over the place.
Luckily, I found a yarn.lock file in git.

What is your intention with telling me that you can't recommend my package?

@MichaelJCole What is your intention with telling me that you can't recommend my package?

@MichaelJCole this bug was brought on by a breaking change in another package. I have time to maintain 2.0, which is production ready if you follow the instructions and will save you so much time.  If you want 1.0 to work for you, please submit that one-line PR that I recommended and I will gladly accept and release.  I don't work for free for things that I do not require for myself.  I am gladly sharing 2.0, in all of its awesomeness, with the community, for free, because it supports me.  I have no contract with anybody, including myself, to support v1.0, so your desires will need to be supported by you.  I'll gladly assist you in supporting yourself.  :)

@MichaelJCole I am using feathers and feathers-vuex with QuasarJs and all work perfectly. You can find documentation here:
https://github.com/feathers-plus/feathers-vuex/pull/216
We are waiting for the official documentation updated according with version 2.0.
So you can wait or alternatively you should contribute :)

@adrianofoschi I am using quasar as well.  I assume that since quasar doesn't have a vue.config.js that the transpile property should be an added key/prop in quasar.conf.js?
transpileDependencies: ['feathers-vuex']

Hi @dkebler I put together a starter project/prototype using feathers-vuex.
If you're interested, this file is a good place to start reading how I did it.
https://github.com/MichaelJCole/Quathers/blob/master/src/boot/feathers.js
There may be better ways :-)

I'll take a look but I've been using quasar and feathers-vuex in a production build since April.  Only yesterday when I went back to work on updating everything (quasar, featherjs, etc) have I run into this issue.  Despite what @marshallswain noted using feathersjs4 and fv 2pre77 with the transpile key now at least gives me a warning but the issue persists.
So despite being "fixed" this issue remains for me with 2pre77
after adding transpile property I get
 warning  in ./src/store/index.js

"export 'default' (imported as 'feathersVuex') was not found in 'feathers-vuex'

as I root around I see @marshallswain

f-v 2pre77 lists   "@feathersjs/errors": "^3.3.6", but feathersjs4 loads errors 4
STILL! f-v 2 pre 77 is loading a default commons

import sift from 'sift'
import commons from '@feathersjs/commons'
import dbCommons from '@feathersjs/adapter-commons'
import { omit as _omit } from 'lodash'

const { _ } = commons

which should according to @adrianofoschi should be
import {_} from '@feathersjs/commons'
Bottom line...I am stuck.  After becoming totally dependent on f-v now I can't get 1.7 or 2.0 to honor the commons dependency from feathersjs 3 or 4.

To make the warnings go away, try adding sourceType: 'unambiguous' to your babel.config.js file. Mine looks like:
module.exports = {
  presets: [
    '@vue/app'
  ],
  sourceType: 'unambiguous'
}

HT this comment by #t2t2

that took away the warning but the issue remains.
I edited the transpiled code to
var _ = require('@feathersjs/commons')
and now my app at least loads.  Can't say if _ is the proper object the f-v needs.  @marshallswain should I open another issue or can you reopen this issue?
looks like this.
_: Object { each: each(), some: some(), every: every(), … }

__esModule: true

createSymbol: function createSymbol()

hooks: Object { ACTIVATE_HOOKS: Symbol(__feathersActivateHooks), createHookObject: createHookObject(), defaultMakeArguments: defaultMakeArguments(), … }

isPromise: function isPromise()

makeUrl: function makeUrl()

stripSlashes: function stripSlashes()

<prototype>: Object { … }
service-module.getters.js:15

In the meantime I guess I will fork it and make the correction.  Don't use typescript but hopefully the build script will make it easy?
(BTW I don't use typescript (use latest node es6 with esm, https://www.npmjs.com/package/esm) to avoid the cryptic messages like typeError: commons_1.default  I wish the sourcemaps could refer back to the original ts.  On top of that by the time transpiled ts runs through babel as well it's hopeless.

@dkebler if you want to message me in slack, we can jump on a call to investigate the issue.

let me just see if I can fix this one thing and refactor my old code for 2.0 and see if I can get it working first

totally weird.  I forked and cloned and opened the ts branch.  I see that commit that fixed it and it remains so in 77,

import sift from 'sift'
import { _ } from '@feathersjs/commons'
import dbCommons from '@feathersjs/adapter-commons'
import { globalModels as models } from './global-models'
import _get from 'lodash/get'
import _omit from 'lodash/omit'

BUT if I look in node_modules in my app clearly 77 is installed
{
  "_from"<EMAIL_ADDRESS>  "_id"<EMAIL_ADDRESS>  "_inBundle": false,
  "_integrity": "sha512-hBbwwxfjDqkSoQVM0sajZB+guTiuOamDXAcnbdNFPpVbUfPX6xS3J3SQHDjNLsEkawBFmNuKQ4e20NtvDH9tKg==",
  "_location": "/feathers-vuex",
  "_phantomChildren": {
    "debug": "4.1.1"
  },

YET the ts code for that getter module in node_modules is the old code
/*
eslint
@typescript-eslint/explicit-function-return-type: 0,
@typescript-eslint/no-explicit-any: 0
*/
import sift from 'sift'
import commons from '@feathersjs/commons'
import dbCommons from '@feathersjs/adapter-commons'
import { omit as _omit } from 'lodash'

const { _ } = commons

So I made a little test app that just loads f-v package correctly and all is as it should be.
So I did a npm i<EMAIL_ADDRESS>again
and now the correct code is in node_modules
Wow some weird npm/cache issue??  I'm using 6.10.3.  I had scrubbed node_modules and did a complete install that had the old code but similarly to some others who said they re-installed just f-v again and it went away it did for me as well :-)
So still need to refactor for 2.0  but you can close this issue again. @marshallswain

@dkebler Ah.  Well that's really weird, but also good news for me.  Thank you for the update.

This is now fixed in<EMAIL_ADDRESS> Thank you @falkodev for making the PR.


ALIAS: doesn't show the stack effect in the bottom of the listener

Type ``n*quot'' and tab-complete it. You won't see the stack effect in the bottom of the UI, but you could...

Did we ever resolve this?


Unsure how to make env vars available for use in the .meshrc.yaml.

I'm trying to use env vars in the .meshrc.yaml file, like it shows in the docs: https://www.graphql-mesh.com/docs/handlers/graphql#from-environmental-variable
I've tried using tools like dotenv, but I don't know where to run the needed code to initial the variables (ie, require('dotenv').config()). I don't know if there is a place where preliminary code can be run, or if there is another way to make this happen. I haven't tried passing in the env vars via the command line, but that isn't really viable for me.

You can use require flag.
mesh xxx --require dotenv/config

Awesome! That did it, thanks!

@Davenporten do you think there is something we could add to the docs that would make it more obvious to other who would follow?

@Urigo That's a good question. I guess I'm not sure where this is in the documentation. (I'm thinking it is https://www.graphql-mesh.com/docs/api/interfaces/types_src.yamlconfig.config/#require but am unsure; if this is the source of the answer maybe having this link by the examples that use env vars would be helpful.) I guess the difficulty is that I have a specific use case of needing dotenv/config, but it could very well be that someone else wants to provide env vars via a different method from dotenv. Would there be a different method to make env vars available in a different way? Maybe the questions to answer are

How did @santino know to suggest ---require dotenv/config? Is there a clear path to figuring this out with the docs? I chalked up my not knowing to not being familiar with yarn and/or not reading the docs sufficiently (which could definitely still be the case), but maybe it isn't clear.
Does --require need certain arguments passed to it? dotenv/config makes sense to me after getting the suggestion because it would be like a require('dotenv/config') but I wouldn't have necessarily guessed that nor that config() would be called implicitly.

With all that being said I am not a javascript/node master, maybe a lot of this stuff would be obvious to others and it is just my lack of knowledge that was the obstacle.

I believe this is very specific to the application and the infrastructure setup people have.
From my personal point of view, the thing to know from Mesh is that it allows you to use your environment variables in the .meshrc.yaml file like this ${MY_ENV_VARIABLE}, and this is documented.
Then, if my CI is setup with environment variables, as an engineer I know that in order to work locally I can use tools such as dotenv to load those environment variables in my local environment; there are other approaches of course and is totally up to the development team to make the choice.
The way environment variables are provided to the actual application is completely separate from Mesh, it depends on how you set your service either in AWS or others; hence why I am not sure how Mesh could help on this, but I might be missing something of course.

Exactly, that's why I'm unsure if there is something that can be added to the docs. For example, what I'm working on with mesh will ultimately be deployed into a container where the environment is set up and the env vars available; making my whole issue a non-issue. But while developing it is impractical for me (just given environment constraints the nature of the env vars) to explicitly set them when spinning up my mesh instance to test, hence why I wanted to use dotenv. But I'm not sure I would have come to the conclusion to use --require dotenv/config without the help from @santino. But again, that might simply be my lack of knowledge or misunderstanding something that is clear to others in the docs.

cc @Urigo

mmm I just see a lot of these questions pop up..
We have sections around that for some handlers like here: https://www.graphql-mesh.com/docs/handlers/openapi/#from-environmental-variable
But I wonder if we could add a section to Recipes and link together some valuable things to be aware of?

@Urigo Maybe that would be helpful, but I wouldn't worry about it too much unless you start getting a lot of other issues about it. I'm not the best developer in the world and a lot of my issues are mostly like from user error, haha. I appreciate you taking the time to look at this. Same for @santino


Release new version?

Hey!
It would be nice to get the logF typescript declaration at https://github.com/jetersen/express-ipfilter/commit/83c806c09883ef3d24919c73ecc955165db0b1d8 in a published version of this library!

Done https://github.com/jetersen/express-ipfilter/releases/tag/v1.2.0 :)

Thanks! :D


command examples

"After you've signed in, make sure everything works by testing a few Windows Virtual Desktop PowerShell cmdlets with the service principal."  -- some examples of commands that are valid for this would be useful.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 156a552a-fe64-e5dc-b86a-b7d8ccbeee1d
Version Independent ID: d83f235e-3ae2-ad71-30c6-0bd7f95fb894
Content: Windows Virtual Desktop service principal role assignment - Azure
Content Source: articles/virtual-desktop/create-service-principal-role-powershell.md
Service: virtual-desktop
GitHub Login: @Heidilohr
Microsoft Alias: helohr

@jrv116psu Thank you for sharing your feedback, our team will further look into this and get back to you at the earliest.

@jrv116psu
I have assigned the issue to content author to check and add the requested examples to the document as appropriate.

@Heidilohr
Can you please check and add your comments on this doc update request as applicable.

I have the same request. Could you please update the documentation with some PowerShell commands to check if everything works fine at this step?

Closing the loop on this - I've changed this line to remove this statement because if you can run the commands (e.g. Add-RdsAccount), then it's working fine. The change should be visible later today.
#please-close


Decoupling: Platform-agnostic media upload

Background
The story editor package uses useMediaPicker() in multiple places to handle media uploads.
That hook allows using the WordPress media modal for upload.
To decouple the editor from WordPress, this logic needs to be made more platform-agnostic, with the WordPress media modal handling moved to wp-story-editor, allowing other implementations on other platforms.
Proof-of-concept
@sayedtaqui has worked on a POC for decoupling the media upload, making MediaUpload a React component that needs to be passed via the ConfigProvider:
https://github.com/rtCamp/web-stories-wp/pull/731
Context
Gutenberg uses the @wordpress/hooks package to allow filtering the MediaUpload component. By default it renders nothing, but in the WordPress block editor it's filtered to render a component using the WordPress media modal.
https://github.com/WordPress/gutenberg/blob/3e14c5e812d8077f28e080522f20763bbfb1dccc/packages/block-editor/src/components/media-upload/README.md
https://github.com/WordPress/gutenberg/blob/3da717b8d0ac7d7821fc6d0475695ccf3ae2829f/packages/block-editor/src/components/media-upload/index.js
https://github.com/WordPress/gutenberg/blob/3da717b8d0ac7d7821fc6d0475695ccf3ae2829f/packages/edit-post/src/hooks/components/index.js
We can't use @wordpress/hooks directly due to the GPL license, which is incompatible with our Apache license.
Implementation Brief
We need to find a good way for making media upload platform-agnostic.
The POC above works, but the config React context is better suited for passing data/settings and not React components. So it doesn't feel right to pass MediaUpload that way.
There's also another POC for using hooks/filters: https://github.com/rtCamp/web-stories-wp/pull/765
But we can't really use that as-is due to licensing restrictions (uses parts of GPL-licensed code).
An alternative could perhaps be to use a Slot & Fill technique.
But maybe there are other, better ways.

@choumx @barklund Would love to hear your thoughts on this

Current plan is to proceed with the POC and refine later based on feedback. The code needs to be moved anyway.

yea I think with the way the context menu is setup, the way we've decided to setup the MediaPicker api, and the dependency structure, this will be weird to implement.
I think you're either going to have to:

Augment the context menu items to take a render prop and pass in the MediaPicker to relevant items. This will most likely feel hacky but might be the path of least resistance
Create some wrapper technique around MediaPicker that renders it, but reaches in and exposes its open method, which you can then consume and pass into the context menu.

I'm not in love with either option, but those are the only two routes I see with the given MediaPicker spec and existing context menu setup.
I do agree that making the config spec take a hook instead of a component would be drastically better to use in this instance, but I'm assuming the component based spec has a lot of other benefits elsewhere (including lots of prior adoption).
I guess my only critique of the MediaPicker component is that it feels a little less in vogue to use a render prop called render when you can just use the children prop, but that's such a minor detail 🤷  ie:
function MediaUpload({ children, ...props }) {
  const open = useMediaPicker(props);
  return children({ open });
}

anyway, tldr; I think it will be messy, but I don't see any better options

Closing this one for now as we decided to continue with the status quo


Copy pasting text in editor keeps formatting from source

Right now, copy pasting text directly in the editors does not use the font of the editor. The text is well understood, and rendered in markdown, but it would be better to fix this.

yes, I noticed this, too. It copies the formatting from the source. In many programs there's an option to just paste the text itself without any formatting. It looks like we have to build this for our editors