FP8 Conversion failure when using Mixtral 8x7B with use_fp8_rowwise

System Info
CPU architecture: x86_64
GPU: 8 NVIDIA H100
Libraries
TensorRT-LLM: v0.12.0
TensorRT: 10.3.0
Modelopt: 0.1.1
CUDA: 12.4
NVIDIA driver version: 550.90.12
Who can help?
No response
Information

[x] The official example scripts
[ ] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[x] My own task or dataset (give details below)

Reproduction
Steps to reproduce the behavior:

Get a Mixtral 8x7B model
Convert it to FP8 using the convert_checkpoint.py script. The command is
convert_checkpoint.py --model_dir=<MODEL DIR> --dtype=float16 --tp_size=2 --output_dir=<CHECKPOINT DIR> --use_fp8_rowwise

Expected behavior
The expected output would be a correctly FP8 row-wise converted Mixtral 8x7B checkpoint.
actual behavior
The checkpoint conversion crashes with the following message:
[TensorRT-LLM] TensorRT-LLM version: 0.12.0
0.12.0
8it [00:00, 11.22it/s]
Traceback (most recent call last):
  File "convert_checkpoint.py", line 487, in <module>
    main()
  File "convert_checkpoint.py", line 479, in main
    convert_and_save_hf(args)
  File "convert_checkpoint.py", line 421, in convert_and_save_hf
    execute(args.workers, [convert_and_save_rank] * world_size, args)
  File "convert_checkpoint.py", line 428, in execute
    f(args, rank)
  File "convert_checkpoint.py", line 410, in convert_and_save_rank
    llama = LLaMAForCausalLM.from_hugging_face(
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/model.py", line 363, in from_hugging_face
    loader.generate_tllm_weights(model)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/model_weights_loader.py", line 330, in generate_tllm_weights
    tllm_weights.update(self.load(tllm_key))
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/model_weights_loader.py", line 279, in load
    if not v.is_contiguous():
AttributeError: 'NoneType' object has no attribute 'is_contiguous'
Exception ignored in: <function PretrainedModel.__del__ at 0x7061e3970f70>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/modeling_utils.py", line 449, in __del__
    self.release()
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/modeling_utils.py", line 446, in release
    release_gc()
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/_utils.py", line 469, in release_gc
    torch.cuda.ipc_collect()
  File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 904, in ipc_collect
    _lazy_init()
  File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 333, in _lazy_init
    raise DeferredCudaCallError(msg) from e
torch.cuda.DeferredCudaCallError: CUDA call failed lazily at initialization with error: 'NoneType' object is not iterable

CUDA call was originally invoked at:

  File "convert_checkpoint.py", line 8, in <module>
    from transformers import AutoConfig
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/dist-packages/transformers/__init__.py", line 26, in <module>
    from . import dependency_versions_check
  File "<frozen importlib._bootstrap>", line 1078, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/dist-packages/transformers/dependency_versions_check.py", line 16, in <module>
    from .utils.versions import require_version, require_version_core
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/dist-packages/transformers/utils/__init__.py", line 27, in <module>
    from .chat_template_utils import DocstringParsingException, TypeHintParsingException, get_json_schema
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/dist-packages/transformers/utils/chat_template_utils.py", line 39, in <module>
    from torch import Tensor
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/dist-packages/torch/__init__.py", line 1694, in <module>
    _C._initExtension(_manager_path())
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 1470, in <module>
    _lazy_call(_register_triton_kernels)
  File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 256, in _lazy_call
    _queued_calls.append((callable, traceback.format_stack()))

additional notes
The conversion works when using quantize.py script with command
quantize.py --model_dir=<MODEL DIR> --dtype=float16 --tp_size=2 --output_dir=<CHECKPOINT DIR> --qformat=fp8 --kv_cache_dtype=fp8 --calib_dataset=<CALIB DATASET> --calib_size=512 --batch_size=8 --calib_max_seq_length=1024
It also works using the convert_checkpoint.py command if applied to a Mistral model or a LLaMA-3.1 model.

Hi @ValeGian, thanks for your interest in TRT-LLM.
Unfortunately, MOE models do not currently support FP8 rowwise quantization. We do not have a plan to add this currently, but if you are able to share more details about your use case we would be happy to consider this further.
I have submitted a pull request to the internal development branch that should make the error much clearer in the future.

Hi @djns99, thanks for the clarification!
Our use case is as follows: we have fine-tuned the Mixtral 8x7B model and are aiming to optimize inference time and throughput. The base model is in FP16, and we attempted quantization as one of our optimization strategies. So far, we’ve successfully quantized the model with TensorRT-LLM 11 using:

FP8 (QuantAlgo.FP8) via quantize.py with the following:
quantize.py --model_dir=<MODEL DIR> --dtype=float16 --output_dir=<CHECKPOINT DIR> --qformat=fp8 --kv_cache_dtype=fp8 --calib_dataset=<CALIB DATASET> --calib_size=512 --calib_max_seq_length=1024
INT8 (QuantAlgo.W8A16) via convert_checkpoint.py with the following:
convert_checkpoint.py --model_dir=<MODEL DIR> --dtype=float16 --output_dir=<CHECKPOINT DIR> --use_weight_only --weight_only_precision=int8

We observed noticeable improvements in inference time and throughput with FP8 compared to INT8. However, there’s a drawback: we’re experiencing a quality drop where FP8 < INT8 < FP16. The difference isn’t substantial but is consistently noticeable across different test cases. This was unexpected based on this documentation, which suggests FP8 should have a "Very Low" impact on quality, while INT8 has a "Low" impact, implying INT8 < FP8.
One of our tracks to address this is exploring new quantization strategies, including FP8 rowwise (QuantAlgo.FP8_PER_CHANNEL_PER_TOKEN), which should be more accurate. Any insights you could share to help with our investigation would be greatly appreciated!
Do you have any practical advice on how best to approach this FP8 quality issue?

Hi @ValeGian thanks for sharing, It's hard to give specific recommendations. Our internal experiments show that FP8 Mixtral scores very similar on MMLU to the default BF16 model.
My understanding is that base Mixtral uses bfloat16, so there may be some extra conversion cost incurred by using float16 as in your commands.
You could also try experiment with a different calibration dataset, or an increased sample size during calibration.

I see that convert_checkpoint.py now integrates a lot of FP8 conversion logic. Is there any major difference in using such script wrt using quantize.py?

To the best of my knowledge both convert_checkpoint and quantize call the same function, so both should give largely equivalent results.
You mentioned that you are using a custom fine tuned version of Mixtral, does base Mixtral without fine tuning also display the same behavior with FP8 < INT8 < BF16 on your test dataset?

Thank you for clarifying the two scripts.
As for the second point, because base Mixtral doesn’t perform well on our use case without fine-tuning, we haven’t measured the quality impact of quantization on it. The only way we could evaluate this would be by using standard public test sets, and for those, we trust the results you've provided.
Do you have any best practices at a lower level for detecting potential issues during quantization, such as monitoring scale factors or identifying outliers?

I can't seem to find any specific recommendations in any of our documentation. Checking for outliers, and understanding what the dynamic range is for your weight and activation values are good starting points. Perhaps comparing intermediate values (see troubleshooting.md) between FP8 and FP16 to identify which layers are contributing the most to the drop in accuracy will help focus your efforts.
Sorry I can't be of more help.

Thank you @djns99 for the guidance!
I’ll look into outliers and compare FP8 and FP16 values as suggested.
For now, I'll close the issue. Your help is much appreciated!


Changing tabs cause a refresh or re-render in the web view for extensions

When viewing an extension, I noticed that it refreshes or re-renders the web view if you switch to a different tab. The scrollbar position is also reset as well and starts back at the top. I was wondering if the state can be saved somehow? As an alternative I just split but still would be nice to have the option to switch between tabs and not have to deal with a refresh or re-render.

/duplicate of #27498


Not Deployable on IIS

I am worried there is no documentation that demonstrates how to deploy the app on IIS. After you create a new app, you cannot deploy to IIS for it to work straight away. It will be helpful if you could demonstrate how to run the IdentityServer without Azure but on IIS because there are intranets applications too

Hello @kwakuduahc1 ... We have a lot of general coverage for IIS in the Host and Deploy node of the docs starting with ...
https://docs.microsoft.com/aspnet/core/host-and-deploy/iis/
... and we have one set up as a tutorial for step-by-step coverage (for an app) ...
https://docs.microsoft.com/aspnet/core/tutorials/publish-to-iis
... is that what you're looking for?

run the IdentityServer without Azure but on IIS

That one is beyond the scope of our coverage. We would leave that to the IdentityServer folks ... https://identityserver.io/

I'll close as answered, but you can add a further comment if that coverage isn't what you're looking for.

@guardrex , thank you, I will look into those documents. Thanks once again.


EIP-6475: Add SSZ Optional[T] type (and drop selector byte in Union None case)

Add specification for EIP-6475 support for SSZ.
Remerkleable impl: https://eips.ethereum.org/assets/eip-6475/optional.py
Tests: https://eips.ethereum.org/assets/eip-6475/tests.py
We could possibly change all planned usage of Union with Optional, and introduce the conceptually more complex Union once needed. Optional serialization can be more optimized than Union.
Discussion: https://ethereum-magicians.org/t/eip-6475-ssz-optional/12891
This PR builds on prior work from:

@zah at https://github.com/ethereum/consensus-specs/issues/1916

I would like to update all Python Optional to PyOptional in this PR to avoid confusions.
@etan-status Thanks for adding remerkleable implmentation. Do you want to open a PR in remerkleable too?

Sure, I can open a PR against Remerkleable as well, once the specs are accepted.
With the Union also dropping the selector byte for the None case, it actually becomes a superset of Optional, so the implementation is a bit easier than in the EIP (which targets the current Remerkleable).

StableContainer (EIP-7495) supports Optional in an even more compact format, and is also forward-compatible; if containers are extended later, proofs for known fields can still be processed even if the verifier code is not updated to support later added fields.

Add specification for EIP-6475 support for SSZ.
Remerkleable impl: https://eips.ethereum.org/assets/eip-6475/optional.py
Tests: https://eips.ethereum.org/assets/eip-6475/tests.py
We could possibly change all planned usage of Union with Optional, and introduce the conceptually more complex Union once needed. Optional serialization can be more optimized than Union.
Discussion: https://ethereum-magicians.org/t/eip-6475-ssz-optional/12891
This PR builds on prior work from:

@zah at https://github.com/ethereum/consensus-specs/issues/1916

#3789 supersedes this, but as long as Verkle still depends on this PR here will leave this open.


Fix typos in NGINX configuration.

This fixes two typos in NGINX outer and inner configuration:

/22 should be /16 for internal RENCI addresses.
Forwarding to inner nginx from outer should be http, not https.

Pull Request Checklist:

[ ] Positive Test Case Written by Dev

[ ] Automated Testing

[ ] Sufficient User and Developer Documentation

[ ] Passing Jenkins Build

[ ] Peer Code review and approval

Positive Test Case

[Enter positive test case here]

Will be merged back from RC-1.41.

Will be merged back from RC-1.41.


SQL integration test

This change is

I guess we should disable this for:

Silverlight
Mono?

and enable the SQL service on AppVeyor? https://www.appveyor.com/docs/services-databases/

Superseded by https://github.com/NLog/NLog/pull/1637


Charging States dashboard reporting updating when car is not updating

Describe the bug
Charging States dashboard reporting updating when car is not updating
Expected behavior
Charging States dashboard should be reporting that the car is asleep or online instead of updating.
Relevant entries from the logs

teslamate_1  | 16:54:24.086 [info] Connection was closed (a:1|t:9|d:1). Reconnecting …
teslamate_1  | 16:54:54.314 [info] Stream.Timeout / 9
teslamate_1  | 16:54:54.360 [info] Connection was closed (a:1|t:10|d:1). Reconnecting …
teslamate_1  | 16:55:50.180 [info] Stream.Timeout / 10
teslamate_1  | 16:55:50.225 [info] Connection was closed (a:1|t:11|d:2). Reconnecting …
teslamate_1  | 16:56:20.456 [info] Stream.Timeout / 11
teslamate_1  | 16:56:20.500 [info] Connection was closed (a:1|t:12|d:2). Reconnecting …
teslamate_1  | 16:57:41.807 [info] Stream.Timeout / 12
teslamate_1  | 16:57:41.850 [info] Connection was closed (a:1|t:13|d:4). Reconnecting …
teslamate_1  | 16:58:12.101 [info] Stream.Timeout / 13
teslamate_1  | 16:58:12.124 [info] Connection was closed (a:1|t:14|d:4). Reconnecting …
teslamate_1  | 16:58:42.366 [info] Stream.Timeout / 14
teslamate_1  | 16:58:42.390 [info] Connection was closed (a:1|t:15|d:4). Reconnecting …
teslamate_1  | 16:59:12.655 [info] Stream.Timeout / 15
teslamate_1  | 16:59:12.701 [info] Connection was closed (a:1|t:16|d:4). Reconnecting …
teslamate_1  | 17:00:09.495 [info] Stream.Timeout / 16
teslamate_1  | 17:00:09.539 [info] Connection was closed (a:1|t:17|d:5). Reconnecting …
teslamate_1  | 17:00:39.795 [info] Stream.Timeout / 17
teslamate_1  | 17:00:39.817 [info] Connection was closed (a:1|t:18|d:5). Reconnecting …
teslamate_1  | 17:01:10.046 [info] Stream.Timeout / 18
teslamate_1  | 17:01:10.091 [info] Connection was closed (a:1|t:19|d:5). Reconnecting …
teslamate_1  | 17:01:40.307 [info] Stream.Timeout / 19
teslamate_1  | 17:01:40.346 [info] Connection was closed (a:1|t:20|d:5). Reconnecting …
teslamate_1  | 17:02:10.576 [info] Stream.Timeout / 20
teslamate_1  | 17:02:10.620 [info] Connection was closed (a:1|t:21|d:5). Reconnecting …
teslamate_1  | 17:03:21.392 [info] Stream.Timeout / 21
teslamate_1  | 17:03:21.435 [info] Connection was closed (a:1|t:22|d:6). Reconnecting …
teslamate_1  | 17:04:32.269 [info] Stream.Timeout / 22
teslamate_1  | 17:04:32.315 [info] Connection was closed (a:1|t:23|d:7). Reconnecting …
teslamate_1  | 17:05:02.531 [info] Stream.Timeout / 23
teslamate_1  | 17:05:02.574 [info] Connection was closed (a:1|t:24|d:7). Reconnecting …
teslamate_1  | 17:05:32.800 [info] Stream.Timeout / 24
teslamate_1  | 17:05:32.845 [info] Connection was closed (a:1|t:25|d:7). Reconnecting …
teslamate_1  | 17:06:43.687 [info] Stream.Timeout / 25
teslamate_1  | 17:06:43.730 [info] Connection was closed (a:1|t:26|d:8). Reconnecting …
teslamate_1  | 17:07:55.553 [info] Stream.Timeout / 26
teslamate_1  | 17:07:55.598 [info] Connection was closed (a:1|t:27|d:9). Reconnecting …
teslamate_1  | 17:09:06.366 [info] Stream.Timeout / 27
teslamate_1  | 17:09:06.389 [info] Connection was closed (a:1|t:28|d:10). Reconnecting …
teslamate_1  | 17:09:24.182 [info] GET /
teslamate_1  | 17:09:24.194 [info] Sent 200 in 11ms
teslamate_1  | 17:09:36.629 [info] Stream.Timeout / 28
teslamate_1  | 17:09:36.677 [info] Connection was closed (a:1|t:29|d:10). Reconnecting …
teslamate_1  | 17:09:36.883 [info] control:hello – 0
teslamate_1  | 17:09:56.926 [info] GET /settings
teslamate_1  | 17:09:56.949 [info] Sent 200 in 22ms
teslamate_1  | 17:10:05.203 [info] GET /
teslamate_1  | 17:10:05.214 [info] Sent 200 in 10ms
teslamate_1  | 17:10:05.682 [info] GET /
teslamate_1  | 17:10:05.700 [info] Sent 200 in 17ms
teslamate_1  | 17:10:06.884 [info] Stream.Timeout / 29
teslamate_1  | 17:10:06.917 [info] Connection was closed (a:1|t:30|d:10). Reconnecting …
teslamate_1  | 17:10:18.015 [warn] Too many disconnects …
teslamate_1  | 17:10:18.016 car_id=1 [info] Creating new connection …
teslamate_1  | 17:10:18.016 car_id=1 [info] Disconnecting ...
teslamate_1  | 17:10:18.040 car_id=1 [info] Connecting ...
teslamate_1  | 17:10:28.274 [info] Connection was closed (a:1|t:1|d:0). Reconnecting …
teslamate_1  | 17:10:38.538 [info] Connection was closed (a:1|t:2|d:0). Reconnecting …
teslamate_1  | 17:10:48.808 [info] Connection was closed (a:1|t:3|d:0). Reconnecting …
teslamate_1  | 17:10:59.030 [info] Connection was closed (a:1|t:4|d:0). Reconnecting …
teslamate_1  | 17:11:15.316 [info] Connection was closed (a:1|t:5|d:0). Reconnecting …
teslamate_1  | 17:11:45.561 [info] Stream.Timeout / 5
teslamate_1  | 17:11:45.583 [info] Connection was closed (a:1|t:6|d:0). Reconnecting …
teslamate_1  | 17:12:15.817 [info] Stream.Timeout / 6
teslamate_1  | 17:12:15.841 [info] Connection was closed (a:1|t:7|d:0). Reconnecting …
teslamate_1  | 17:12:46.078 [info] Stream.Timeout / 7
teslamate_1  | 17:12:46.123 [info] Connection was closed (a:1|t:8|d:0). Reconnecting …
teslamate_1  | 17:12:47.863 [info] GET /
teslamate_1  | 17:12:47.875 [info] Sent 200 in 10ms
teslamate_1  | 17:13:16.342 [info] Stream.Timeout / 8
teslamate_1  | 17:13:16.386 [info] Connection was closed (a:1|t:9|d:0). Reconnecting …
teslamate_1  | 17:13:46.641 [info] Stream.Timeout / 9
teslamate_1  | 17:13:46.685 [info] Connection was closed (a:1|t:10|d:0). Reconnecting …
teslamate_1  | 17:14:16.890 [info] Stream.Timeout / 10
teslamate_1  | 17:14:16.934 [info] Connection was closed (a:1|t:11|d:0). Reconnecting …
teslamate_1  | 17:15:12.666 [info] Stream.Timeout / 11
teslamate_1  | 17:15:12.710 [info] Connection was closed (a:1|t:12|d:1). Reconnecting …
teslamate_1  | 17:15:42.928 [info] Stream.Timeout / 12
teslamate_1  | 17:15:42.950 [info] Connection was closed (a:1|t:13|d:1). Reconnecting …
teslamate_1  | 17:16:40.390 [info] Stream.Timeout / 13
teslamate_1  | 17:16:40.434 [info] Connection was closed (a:1|t:14|d:2). Reconnecting …
teslamate_1  | 17:17:10.682 [info] Stream.Timeout / 14
teslamate_1  | 17:17:10.728 [info] Connection was closed (a:1|t:15|d:2). Reconnecting …
teslamate_1  | 17:17:40.996 [info] Stream.Timeout / 15
teslamate_1  | 17:17:41.019 [info] Connection was closed (a:1|t:16|d:2). Reconnecting …

Screenshots
Image
Environment

TeslaMate version: 1.20.1
Type of installation: Docker
OS TeslaMate is installed on: Raspberry OS Lite
User OS & Browser: Chrome

So I upgraded my install to the latest edge version of 1.21.0-dev and it seems to have resolved my issue. I think it may have related to the new 2fa that tesla is rolling out which the latest version seems to fix.

So I upgraded my install to the latest edge version of 1.21.0-dev and it seems to have resolved my issue. I think it may have related to the new 2fa that tesla is rolling out which the latest version seems to fix.


Update launch.json schema to add "python" and remove "pythonPath"

Now that we no longer have to worry about debug configs being compatible with both ptvsd and debugpy, there's no reason to use the legacy spelling and semantics for the property. The schema should be updated:

Remove "pythonPath" entirely.

Add "python", and reflect the fact that it can be either a simple string, or an array of strings (to specify command-line arguments for the Python interpreter itself).

#13218  is now intended as the one and only way of supplying arguments to the interpreter, so "python": [...] shouldn't be exposed in the extension - only scalar string values.
(debugpy will still support the array syntax for backwards compatibility)


SupportedLanguages protocol V2 \w aliases.

Signed-off-by: kuba--<EMAIL_ADDRESS>Closes https://github.com/bblfsh/python-client/issues/168
This PR:

upgrades SDK to v3.2.2
SupportedLanguages requests go over v2 protocol with aliases

This change is

As we discussed, I'm closing this PR. The new PR (only with v2 protocol) to another branch (v4)


AzGovViz v6_major_20220930_2

Changes since last PR
Changes (2022-Sep-30 / Major)

Fix issue #135

Embedded GitHub Actions OIDC (Open ID Connect) specific functionality to reconnect and get new token (AzAPICall)
New parameter -GitHubActionsOIDC which is only to be used for GitHub Actions /.github/workflows/AzGovViz_OIDC.yml
Updated /.github/workflows/AzGovViz_OIDC.yml to use the new parameter -GitHubActionsOIDC

Fix issue #136

Handle return for Storage Accounts located in managed Resource Groups
🌸 Call for contribution: Please review the list of known managed Resource Groups and contribute if you can, thanks!

Added missing variable NoStorageAccountAccessAnalysis in .azuredevops/pipelines/AzGovViz.variables.yml
Use AzAPICall PowerShell module version 1.1.30

Changes (2022-Sep-28 / Major)

New feature 'Storage Account Access Analysis' - provides insights on Storage Accounts with focus on anonymous access (containers/blobs and 'Static website' feature). Data is provided in the HTML TenantSummary (Subscriptions, Resources & Defender) and as CSV export

New parameter -NoStorageAccountAccessAnalysis - do not execute the feature
New parameter -StorageAccountAccessAnalysisSubscriptionTags - define the Subscription tags that should be added to the CSV output
New parameter -StorageAccountAccessAnalysisStorageAccountTags - define the Storage Account (resource) tags that should be added to the CSV output
Updated .azuredevops/pipelines/AzGovViz.variables.yml accordingly

Rename 'ALZ EverGreen' feature to 'Azure Landing Zones (ALZ) Policy Version Checker'

Replaced parameter -NoALZEverGreen with -NoALZPolicyVersionChecker

Use AzAPICall PowerShell module version 1.1.24
Optimizations

Changes (2022-Sep-17 / Major)

Fix Azure DevOps Pipeline correct addressing of NoDefinitionInsights variable in YAML
Fix issue #132
Add Contribution Guide

Changes (2022-Sep-12 / Major)

New feature 'ALZ EverGreen' - Azure Landing Zones EverGreen for Policy and Set definitions. AzGovViz will clone the ALZ GitHub repository and collect the ALZ policy and set definitions history. The ALZ data will be compared with the data from your tenant so that you can get lifecycle management recommendations for ALZ policy and set definitions that already exist in your tenant plus a list of ALZ policy and set definitions that do not exist in your tenant. The ALZ EverGreen results will be displayed in the TenantSummary and a CSV export *_ALZEverGreen.csv will be provided. Thanks! ALZ Team

New parameter -NoALZEverGreen - Do not execute the ALZ EverGreen feature

Update: Per default DefinitionInsights will be written to a separate HTML file. This will improve the html file handling (browser memory usage /response time / user experience).

Note: Please update your Azure DevOps and GitHub YAML files with the latest versions if you are using the webApp publishing feature
New parameter -NoDefinitionInsightsDedicatedHTML (DefinitionInsights will NOT be written to a separate HTML file *_DefinitionInsights.html)

Add Resource fluctuation detailed (*_ResourceFluctuationDetailed.csv) CSV output (add/remove, scope details, resource details)

Fix consumption reporting for large tenants with more than 3k subscriptions (Management Group abc has too many subscriptions , exceeding CCM API Current Limit 3000)

Fix CSV export *_PolicySetDefinitions.csv - Builtin Policy definitions contained in PolicySet definitions will only show the GUID instead of the full ID as for large PolicySet definitions the field size limit in Excel may be exceeded (column: PoliciesUsed4CSV)

BuiltIn definitions collection - add 'Static' Policy definitions (part of DefinitionInsights and *_PolicyDefinitions.csv)

Fix HierarchyMap image quality (now .png (aka 'peng')). Thanks! Brooks Vaughn

Use AzAPICall PowerShell module version 1.1.23

Optimizations
Changes (2022-Aug-17 / Major)

Update: IMPORTANT Fix for custom Role definitions / missing DataActions and NotDataActions

Update API reference roleDefinitions use API version 2018-07-01 (API version 2022-04-01 not available in sovereign clouds)

BugFix

Changes (2022-Aug-03 / Major)

IMPORTANT Fix for custom Role definitions / missing DataActions and NotDataActions

Update API reference roleDefinitions use API version 2022-04-01

BugFix

Changes (2022-Jul-31 / Major)

Update on feature 'PIM (Privileged Identity Management) eligible Role assignments'

Integrate with RoleAssignmentsAll (HTML, CSV)

New parameter -NoPIMEligibilityIntegrationRoleAssignmentsAll - Prevent integration of PIM eligible assignments with RoleAssignmentsAll (HTML, CSV)

Fix: PIM 'Assigned' and 'Activated' Role assignments now also reflect inheritance for lower scopes
Bugfixes & optimizations

Changes (2022-Jul-28 / Major)

Update on feature 'PIM (Privileged Identity Management) eligible Role assignments'

new parameter -PIMEligibilityIgnoreScope - By default will only report for PIM Elibility for the scope (ManagementGroupId) that was provided. If you use the new switch parameter then PIM Eligibility for all onboarded scopes (Management Groups and Subscriptions) will be reported.
Add CSV output
Add inheritance information

Use AzAPICall PowerShell module version 1.1.21
Bugfixes

Changes (2022-Jul-26 / Major)

New feature 'PIM (Privileged Identity Management) eligible Role assignments' (TenantSummary)
⛔ Breaking Change! requires API permissions update!

Get a full report of all PIM eligible Role assignments for Management Groups and Subscriptions, including resolved User members of AAD Groups that have assigned eligibility
Spoiler: Next iteration will include ScopeInsights, showing entire eligible Role assignments on Subscriptions including from upper Management Group scopes
💡 Note: this feature requires to execute as Service Principal with Application API permission PrivilegedAccess.Read.AzureResources

Use AzAPICall PowerShell module version 1.1.19
Bugfixes

merging following @jocontr review.


Remove reactarmory from the courses page

Their SSL cert has been expired for 8 months.  I don't believe its a great idea to send new users to them at this point.

Seems like it would be nice to contact the author first? I'll reach out to James.

Seems like it would be nice to contact the author first? I'll reach out to James.

Looks like at some point a different PR was merged to fix this


Update version to 1.0.0-beta-1

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] I have signed the Adobe Open Source CLA.
[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

@nporter-adbe are we doing this for core extensions already? I will look into this

We're not currently doing this, but we're not really versioning the beta as is being done here. Rather just have customers point to the main branch of the repo to get the latest beta build.

@nporter-adbe updated the framework version to 1.0.0-beta-1 as well


Question on how to use APIs with proof generation

I'm submitting a ...
[ ] bug report
[ ] feature request
[ ] question about the decisions made in the repository
[X] question about how to use this project

Summary

Hi,
I would like to use pe-js with BBS+ in which I generate a proof from a credential.  Is this possible?  When looking at the example at https://github.com/Sphereon-Opensource/pe-js#credential-query, it is not clear where I make the call to generate the proof.  Is it before or after the call to pejs.submissionFrom, or is there some option to that call which allow passing in the function which generates the proof?
Thanks in advance,
Keith

Other information (e.g. detailed explanation, stack traces, related issues, suggestions how to fix, links for us to have context, eg. StackOverflow, personal fork, etc.)

Hi Keith,
You question is outside of the scope of the Presentation Exchange (https://identity.foundation/presentation-exchange/) and thus our library. What PE is essentially about is having a system in place where wallets/agents can negotiate with verifiers about what the verifier would like to receive and what the holder has present in terms of VCs that match these requirements in their wallet.
Actual issuance and/or verification of a VC is outside of the scope of this library. This library provides a means to check/evaluate whether VCs conform to the rules as required by the verifying party. If you want to create BBS proofs, for instance the library created by Mattr might help: https://github.com/mattrglobal/jsonld-signatures-bbs/
Niels

Hi Niels,
Yes, I am already using the Mattr module.  My question is how to use your module and the Mattr module to implement the verification flow.  Here is what I think I should do.  Does this look correct?
In the verifier:
Send presentationDefinition to prover
In the prover:

Call pejs.selectFrom to find all matchingCredentials for a given presentationDefinition
Allow the user to select all or a subset of these credentials. Call them selectedCredentials
For each of the selectedCredentials, call the mattr deriveProof method (see https://github.com/mattrglobal/jsonld-signatures-bbs/blob/master/sample/ts-node/src/demo_single.ts#L91) to get the derivedProof.  I assume I'll have to build the reveal document myself from the presentationDefinition somehow.  Assuming I can do this, call the resulting array derivedProofs, one for each of the selectedCredentials.
Call pejs.submissionFrom(presentationDefinition, derivedProofs) to get the verifiablePresentation to send to the verifier

In the verifier:

Call pejs.evaluate(presentationDefinition, verifiablePresentation) to make sure the prover sent back what I asked for
Iterate over all of the derivedProofs and call the mattr verify method at https://github.com/mattrglobal/jsonld-signatures-bbs/blob/master/sample/ts-node/src/demo_single.ts#L99

Does this sound correct?
Thanks,
Keith

Ah sorry, yes your question makes total sense.
The order of operations you specify is indeed correct. Step 4 at the holder/prover calls into evaluate internally as well. Right now that method only return the presentation submission data.
In https://github.com/Sphereon-Opensource/pe-js#credential-query you can see the example how/where that data is combined into a Presentation.
We already have discussed internally that it would make sense to return the Unsigned Presentation there as well, with an optional callback parameter to do the signing with whatever function you have available to sign a VP.
Unfortunately we are not there yet. We expect to include that in our first release.
To answer your question:
Currently it needs to be done after the submissionFrom (step 4) by yourself. Where you first have to construct the presentation from the data you have available at that point.
In the future you could get the unsigned VP directly in the response, so you don't have to construct it anymore and can sign it afterwards. Unless you specify a callback function, which means the VP would be signed.
Does that make sense?

Sorry, but it is not clear to me if you are saying that the algorithm I provided is not quite right now or later.
Rather than taking up your time now, I can wait until you release.  If possible, it would nice if you could document how to do this at that time.
Thanks

One last comment for now.  The Mattr module does not currently support composite proofs, meaning that I have to derive a proof on each selected credential.  In the future, I expect that it will support composite proofs, meaning that I call deriveProof once and get a single proof to send to the verifier.  I assume your library would work in both cases.
Thanks again

Hi Niels @nklomp,
Is there any update on when the first release will be ready?
Thanks,
Keith

Is there any update on when the first release will be ready?

Hi Keith,
You have an impeccable timing.
Yesterday we started working on getting a first alpha out of the door today. The first beta is planned in 1.5 weeks.

Thanks Niels @nklomp.
Will your library provide a way to convert a DIF presentation definition to a JSON-LD frame as is required by Mattr BBS+ deriveProof method?  Or is this something that I will need to handle myself?
Also, if you could let me know when the alpha is ready, I'd appreciate it.

Almost certainly not in the beta. It is something we will be looking into, but our focus currently is on getting every single requirement/rule of the spec implemented. Once that is done we will focus more on the quality of life aspects in using the library.
The alpha has been released (see the updated README). You can now install version 0.0.1 from npmjs directly

Hi Niels,
I just got the following typescript compilation errors when trying to use the alpha.  Any ideas?
src/lib/node_modules/@sphereon/pe-js/build/main/lib/validation/bundlers/fieldsVB.d.ts:14:5 - error TS2416: Property 'getValidations' in type 'FieldsVB' is not assignable to the same property in base type 'ValidationBundler<Field[]>'.
Type '(fields: Field[]) => Validation[]' is not assignable to type '(t: Field[] | Field[][]) => Validation[]'.
Type 'Validation[]' is not assignable to type 'Validation[]'.
Type 'Validation' is not assignable to type 'Validation'.
Types of property 'predicate' are incompatible.
Type 'ValidationPredicate' is not assignable to type 'ValidationPredicate'.
Type 'unknown' is not assignable to type 'Field'.
14     getValidations(fields: Field[]): Validation[];
~~~~~~~~~~~~~~
src/lib/node_modules/@sphereon/pe-js/build/main/lib/validation/bundlers/submissionRequirementVB.d.ts:13:5 - error TS2416: Property 'getValidations' in type 'SubmissionRequirementVB' is not assignable to the same property in base type 'ValidationBundler'.
Type '(srs: SubmissionRequirement[]) => Validation[]' is not assignable to type '(t: SubmissionRequirement | SubmissionRequirement[]) => Validation[]'.
Type 'Validation[]' is not assignable to type 'Validation[]'.
Type 'Validation' is not assignable to type 'Validation'.
Types of property 'predicate' are incompatible.
Type 'ValidationPredicate' is not assignable to type 'ValidationPredicate'.
Type 'unknown' is not assignable to type 'SubmissionRequirement'.
13     getValidations(srs: SubmissionRequirement[]): Validation[];
~~~~~~~~~~~~~~

Hi,
I would suspect build did not succeed on the local machine. This would noramally happen when we have a formatting / prettier related error. Or if there is a a TypeScript config issue.

This git-repo shows a passing test suite.
Also following project can use the library successfully: https://github.com/Sphereon-Opensource/pe-js-consumer-example
I have just checked on a freshly cloned project : Following succeeds

git clone git@github.com:Sphereon-Opensource/pe-js.git
cd pe-js/
yarn install
yarn fix
yarn build
yarn test

Test Suites: 21 passed, 21 total
Tests:       206 passed, 206 total
Ran all test suites.

The cause of the 2 tsc compilation errors which I noted above could be fixed by changing the line at https://github.com/Sphereon-Opensource/pe-js/blob/develop/lib/validation/bundlers/validationBundler.ts#L13 from
public abstract getValidations(t: T | T[]): Validation<unknown>[];

to
public abstract getValidations(t: T | T[]): Validation<T>[];

Would you pls consider making this change?  I am not sure why you are not seeing these errors but would assume the yarn build above is not testing a tsc compilation as a user of this package.

@nklomp @hrehman-sphereon I opened a pull request which fixes this for me at https://github.com/Sphereon-Opensource/pe-js/pull/40

Hi @smithbk, The interfaces you mention have been introduced, but it could be that these are not used correctly at all places. We have a PR for which these issues should be checked, and if found, should be fixed, we will merge this soon.
Most importantly when we run a tcs compiler in the pe-js root, we do not find any errors. This has been tested on multiple machines. We would suggest to have a look at the tsconfig.json (in root folder) to check if you are using a correct configuration since this can cause problems.

In order to reproduce the tsc compiler failure, please unzip the following into a clean directory and then run npm install and tsc:
pejs-tsc-error.zip
You should see the following errors:
$ tsc
node_modules/@sphereon/pe-js/build/main/lib/validation/bundlers/fieldsVB.d.ts:14:5 - error TS2416: Property 'getValidations' in type 'FieldsVB' is not assignable to the same property in base type 'ValidationBundler<Field[]>'.
  Type '(fields: Field[]) => Validation<Field>[]' is not assignable to type '(t: Field[] | Field[][]) => Validation<unknown>[]'.
    Type 'Validation<Field>[]' is not assignable to type 'Validation<unknown>[]'.
      Type 'Validation<Field>' is not assignable to type 'Validation<unknown>'.
        Types of property 'predicate' are incompatible.
          Type 'ValidationPredicate<Field>' is not assignable to type 'ValidationPredicate<unknown>'.
            Type 'unknown' is not assignable to type 'Field'.

14     getValidations(fields: Field[]): Validation<Field>[];
       ~~~~~~~~~~~~~~

node_modules/@sphereon/pe-js/build/main/lib/validation/bundlers/submissionRequirementVB.d.ts:13:5 - error TS2416: Property 'getValidations' in type 'SubmissionRequirementVB' is not assignable to the same property in base type 'ValidationBundler<SubmissionRequirement>'.
  Type '(srs: SubmissionRequirement[]) => Validation<SubmissionRequirement>[]' is not assignable to type '(t: SubmissionRequirement | SubmissionRequirement[]) => Validation<unknown>[]'.
    Type 'Validation<SubmissionRequirement>[]' is not assignable to type 'Validation<unknown>[]'.
      Type 'Validation<SubmissionRequirement>' is not assignable to type 'Validation<unknown>'.
        Types of property 'predicate' are incompatible.
          Type 'ValidationPredicate<SubmissionRequirement>' is not assignable to type 'ValidationPredicate<unknown>'.
            Type 'unknown' is not assignable to type 'SubmissionRequirement'.

13     getValidations(srs: SubmissionRequirement[]): Validation<SubmissionRequirement>[];
       ~~~~~~~~~~~~~~

Found 2 errors.

Coming back too your suggestion on having strict type checking, this is something we agree with, but because of the nature of verifiableCredentials we have not yet been able to enforce strict type checking everywhere.
verifiableCredentials  have different "valid" shapes. For example we might have the following which is valid:
{
  "verifiableCredential":{
    "credentialSubject": {...},
     ....,
     "someAspect":"value"
  }
}

And we can have the following which is also valid:
{
  "verifiableCredential":{
    "vc":{
       ....,
       "someAspect":SomeObject[]
    },
    "someOtherAspect":"value"
}

There's only some parts that are always present, for example type & uri, but in their case we don't know the exact place. So we currently look at presentationDefinition which is a more strict object as a starting point. There could/should be a better solution indeed, we will be looking into this in the coming weeks. In the meantime you should be able to use the latest version where you can use the library (sphereon/pe-js:^0.1.1 or newer) with strict types checking enabled.

Hi, I'm curious if you've tried using the logical OR typescript syntax as follows where interface1 and interface2 are various valid formats?
type VerifiableCredential = interface1 | interface2;

We have multiple interfaces for different representations of VP (we have CHAPIVP, DIDCommVP, OpenIDConnectVP, and a general VP) the main problem is that is we can not be sure what is inside the VC, it can be something related to the age of the holder, a university degree, or a pet vaccination with different properties. This means using the logical OR typescript syntax as you suggest would help us in some parts like interchangeable types ( string | string[]) but not in all the cases.

Actually the above is not really the problem, as chapi, didcomm and openid connect are simply wrappers of VP data. We simply need to do some refactoring for the different ways we see VC/VPs in the wild. They do not all fully conform to the specs (jay)


Edit spanish translation

This pull request update several lines in spanish and eliminates other redundant lines.

Thanks!


+page and +layout will break the current adapter

Sveltekit changed to using +page and +layout files. As I just converted my application to the latest version, I found that +page css files and +layout css files are not loaded correctly as s3 doesn't support "+" in the uri.
Instead, the plus sign is converted to a "%2b".
One simple fix would be to change the uri in the router.js:
request.uri = uri.replace("+", "%2b")

Hi @philippschladitz thank-you for raising this issue, I will publish a fix in the next few days to address this.

See https://github.com/sveltejs/kit/issues/5843 and https://github.com/vitejs/vite/pull/9737
This is likely a vite build issue/bug you are experiencing and not this adapter.

@philippschladitz This is a bug, + is considered a reserved character in a URL.

Hi @yarbsemaj how do I resolve this issue?
We migrated to the new Sveltekit routing system and nothing is working and it's Friday. Please if someone could help me out.

Hi @souravjamwal77
Unfortunately, for a proper fix, You need to wait for the next vite release where this bug has been addressed. Alternatively, this commenter has written a vite plugin that will fix the issue https://github.com/sveltejs/kit/issues/5843#issuecomment-1220222362

Hi @yarbsemaj I'm getting below errors after successful deployment
{
    "errorType": "TypeError",
    "errorMessage": "Cannot read properties of null (reading 'handleError')",
    "stack": [
        "TypeError: Cannot read properties of null (reading 'handleError')",
        "    at Object.handle_error (/var/task/build/server/serverless.js:17138:28)",
        "    at respond (/var/task/build/server/serverless.js:17094:13)"
    ]
}

Hi @souravjamwal77
This looks to be an issue related to a svelte change introduced 3 days ago, I have published a new adapter (v0.12.0) that fixes this issue.
It should now be possible to deploy and use the latest sveltekit version.


ChunkedEncodingError when reading data about ticker

I am trying to read tickers info using the following code:
tickers_list = ['NIO',  'PLTR', 'MSFT']
tickers = '  '.join(tickers_list)
 tickers = yf.Tickers(tickers)
 print(tickers.tickers['NIO'].info)
I get this error:
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(82 bytes read, 1069 more expected)', IncompleteRead(82 bytes read, 1069 more expected))
Environment:
python 3.6.1
MacOS BigSur

I'm getting the same when trying to download price history for stocks...
I have been running these scripts several times daily for last years without much problem - but suddently they started to fail for well above 50% of the tickers I try to download since Thursday (1st of April).
I'm running Python 3.8.
I am experiencing these problems both in Windows and Linux environments...
Would really appreciate if someone has any kind of input!

I also have this problem.

Have been trying various settings the last two days and I seem to get slightly more data with "threads"=False and I could probably live with the fact that you get some errors and that it is somewhat slow - but the primary problem right now is that the "ChunkedEncodingError" isn't returned until after almost 2 minutes...so it takes ages to run through a loop with try /except... Would be happy just to find a faster workaround at this stage, suggestions?

I've also been getting this error a lot since April 1st. I'm running Python 3.8.7 on macOS Big Sur.

Thanks! Will try that workaround while yahoofinance behaves like it does currently!

Latest version has timeout, can someone confirm problem gone.


[Request] Raspberry Pi

Raspberry pi compatible?
Hi just wondering. I already have a raspberry pi with a touch screen.
Could this project just be ran in kiosk mode to have it as a media player without having an actual Car Thing?
Would be a nice addition.

Yes! It would
You can go to the Clients tab and there's a QR code button that will have the URL you can go to from any device on your network (phone, tablet, fridge, etc)
Basically anything with a browser should work

Oh neat!!!! Thanks!!


R 3.3.1 version

Would it be possible to also get a R 3.3.1 version with a higher build number than in the r channel? This old in in the R channel has problems with plotting and breaks a lot of things.
@johanneskoester @lecorguille

I doubt that his would help bioconda, because the conda-forge channel has a lower build number in our current setup. Even if it would help, it would mean that the we would have to adjust all the patches that are used here, and try to not make the mistakes that caused the broken stuff in the R channel version of 3.3.1. At least for me, that would mean too much work at the moment for a version that will become obsolete once the R port to conda-forge is finished. Of course, I am completely open to PRs in this direction.

Finally, the different underlying library versions between conda-forge and R could introduce incompatibilities between the R packages from the R channel (that we would still use) and the potentially fixed r-base from here.

This is yet another reminder how super urgent the port of all the packages from the r channel is. I am in the final stages to fix building r packages on conda-forge/staged-recipes, see conda-forge/staged-recipes#2098. If that PR is merged, adding the other packages should be straightforward. Currently the last error in that PR only occurs on windows. Maybe I'll just skip windows to speed everything up... I"m not interested in it anyways.

I'm not talking strictly about bioconda. The latest version of r 3.3.1 is broken and I don't think it is good to keep these version and don't offer a workaround.
For bioconda, we actually would need to rebuild everything against 3.3.2, which again is a major undertaking :(

I agree, but why is it not fixed in the R channel? Regarding R 3.3.2, does that mean you would be in favor of having R 3.3.1 in conda-forge and not R 3.3.2 at all? When I started, I thought we would move to 3.3.2 at some point and therefore considered it worthless to first support 3.3.1 at a point where it was already outdated...

The R channel is not really responsive. They created R 3.3.2 and that was it - and thats the fix :(
We should go for 3.3.2, this is completely fine, but I think we should not leave 3.3.1 broken. But I see your point, maybe this is the wrong repository. I should complain more in the r-channel :(


[Bug] Remove \n from generated code payload

Operating System:  Windows 7

Details
I'm doing a quite simple query

Then generating python code, either http-client or request, here it is
import requests

url = "http://<IP_ADDRESS>/api/index.php"

payload = "{\n    \"action\": \"SHOW\",\n    \"object\": \"STPL\"\n}\n"
headers = {
    'content-type': "application/json",
    'centreon-auth-token': "bla bla bla"
    }

response = requests.request("POST", url, data=payload, headers=headers)

print(response.text)

As you can see, the payload is somehow messed up. I've the same issue for all the others language, like go 	payload := strings.NewReader("{\n    \"action\": \"SHOW\",\n    \"object\": \"STPL\"\n}\n") & so on.
And thanks for the awesome client :+1:
Best regards,

Hi @rgarrigue,
I'm not sure I understand the problem. The \n characters represent the newlines in your request body. Is there an alternate way you'd like to see them represented?

I'm sorry to refute your comment, but I don't believe that's correct.
Newlines were part of your original data. The difference is that the Insomnia editor chooses to show them visually (as new rows) instead of literally (as \n). However, the underlying data is the same in both cases.
To demonstrate this, removing the \n characters from your original data would look like this:

Uh, okay. Yeah my bad obviously. It bugged me since json is usually a big one liner, and I've to ask my browser via an extension or my IDE to display this nicely, without the formating being a visible part of the data.
Well, I'm gonna close this one then. Sorry for the disturbance

No worries. If you (or anyone else) needs/wants to send compressed (minimal whitespace) request bodies, it's possible to write a Request Hook Plugin that looks for all application/json requests, parses and compresses the request body, and overwrites the request body with the compressed one.


Git for Windows 2.41.0 stuck on push with gitea 1.19.3

Description
With the latest git package, the push doesn't work anymore and stay stuck for ever at TOTAL xxxx.
Reverting to 2.40.1 solves the issue
Gitea Version
1.19.3
Can you reproduce the bug on the Gitea demo site?
Yes
Log Gist
No response
Screenshots
No response
Git Version
official dockerfile 1.19.3
Operating System
Linux
How are you running Gitea?
Official Dockerfile 1.19.3
Database
PostgreSQL

Linux or Windows?

windows client
linux server

This isn't related to git version
On my gentoo desktop
git -v git version 2.41.0
My alpine headless:
`
apk info gitea
gitea-1.19.3-r0 installed size:
138 MiB
`
So this could being git-for-win v2.41.0. I can book my work machine and upgrade git-for-win (it is  v2.39.x) and see if the issue exists as : win-git-2.41 -> linux-gitea-1.19

Bear in mind that I am using the container provided by gitea.
There might be a matching git issue between server and client...

Bear in mind that I am using the container provided by gitea.
There might be a matching git issue between server and client...

There shouldn't be. git is actually quite stable in terms of ABI with regards to data interchange
my alpine headless (running gitea-1.19.3) is also using git-2.40.1-r0.     My gentoo desktop is using git-2.41
There might be a "bug" in git-for-win-2.41 and I can independently check shortly

Apologies for the delay...
Upgraded git-for-windows to 2.41.0 and cloning/pushing to to Gitea-1.19.3

 git -v
git version 2.41.0.windows.1

Something else is going on here.

I am pretty sure the "stuck" is caused by incorrect log messages during startup.
It should have been fixed in 1.20/1.21
Feel free to reopen if there is still a problem

Sorry but <IP_ADDRESS> did not help either...
Works well with github... not with gitea... Happy to help, but don't know how to debug right now.

No issues here anymore...

No issues here anymore...

How did you resolve the problem?

The lastest git exe on windows and 1.21.x work so far


Need the ability to blacklisting/whitelisting of characters

Discussed in https://github.com/mindee/doctr/discussions/888

Originally posted by Xargonus April  7, 2022
Hello, is there currently a way to blacklist or whitelist characters used by the text recognition model?

Thanks for opening the issue, let's first focus the discussions in #888 to avoid duplicates :)

That's a great one too. Again, callback idea might shine here, since you can boost/deboost some characters, not completely blacklist them. For example, I often see 1 recognized as l or Q as 0/O, or : as 0. I'd like to avoid complete blacklist, but if I can, for example, boost and prioritize digits over letters or : over . it can solve the problem.

What if we make it a bit broader and instead provide an option to multiply the logits to some weight.
For example, l and 1 and I can often be wrongly identified, and as soon as I expect mostly digits, I can upvote "digits" to break the almost-tie situations without killing all the rest of characters.

@dchaplinsky Mhhh this would need any mapping to "close" characters ? :thinking:

Hey everyone :wave:
The use case here is to help out the text recognition part when you have more info about a subvocab, so now we need to assess whether that's worth addressing (I think it would be useful), what would be the API, and at which step this should take place.
My two cents:

This will be quite useful to better leverage wide vocab model checkpoints without retraining them. We should work on this
For the API, the issue with passing whitelist/blacklist of characters is that you need both and then you'll need an arbitrary order of priority. With minimal snippet, we can easily convert that to a weight vector which will be used by the model.
For simplicity, I think this should be passed to the model (or post processor) and made accessible as an instance attribute as a Tensor. At inference time, this should be used in the call of the post processor either here https://github.com/mindee/doctr/blob/main/doctr/models/recognition/crnn/pytorch.py#L224 if at model level, or here https://github.com/mindee/doctr/blob/main/doctr/models/recognition/crnn/pytorch.py#L75 if at postprocessor level

Here is my suggested design for blacklist:
import torch
from doctr.models import crnn_vgg16_bn

blacklisted_chars = {str(num) for num in range(10)}
# Set the mask
vocab_mask = torch.tensor((0 if char in blacklisted_chars else 1 for char in vocab), dtype=torch.float32)
model = crnn_vgg16_bn(pretrained=True, vocab_mask=vocab_mask)

input_tensor = torch.rand(1, 3, 32, 128)
out = model(input_tensor)

and whitelist:
whitelisted_chars = {str(num) for num in range(10)}
vocab_mask = torch.tensor((1 if char in whitelisted_chars else 0 for char in vocab), dtype=torch.float32)

What do you think?


Simple skip test function

Closes #1149.

Hi @andrew-aladev, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
The agreement was validated by Microsoft and real humans are currently evaluating your PR.
TTYL, MSBOT;

PR is added to the queue for testing as 2. task. (e849c33)

Test (Success) 86743101 build is completed (e849c33)
See https://github.com/ThaliTester/TestResults/tree/86743101e849c33_Simple_skip_test_function_andrew-aladev/ for the logs

Test 86743101e849c33(e849c33) has failed
See https://github.com/ThaliTester/TestResults/tree/86743101e849c33_Simple_skip_test_function_andrew-aladev/ for the fail logs

I think some scaffolding got left in SimpleTape. Also please edit UnitTestFramework, lines 61 and 68 to indicate if tests were skipped or not. A simple boolean says "test were skipped" or "tests weren't skipped" is fine. For now we can search the logs to see which tests were skipped. Once you fix those two then

Reviewed 4 of 5 files at r1, 1 of 1 files at r2.
Review status: all files reviewed at latest revision, 5 unresolved discussions, some commit checks failed.

test/www/jxcore/lib/SimpleTape.js, line 99 at r2 (raw file):

}

SimpleThaliTape.prototype._runTest = function (test) {

Isn't this whole function ignored because _runTest is redefined on line 116?

test/www/jxcore/lib/SimpleTape.js, line 102 at r2 (raw file):

  var self = this;

  return Promise.try(function () {

Isn't this code redundant with the code below in tape()?

test/www/jxcore/lib/SimpleTape.js, line 104 at r2 (raw file):

  return Promise.try(function () {
    if (test.canBeSkipped) {
      return test.canBeSkipped();

NB: I don't think it's a good idea to depend on the default return value, I know it's null and I know the right thing would happen but I think it's much cleaner to be explicit. E.g. return test.canBeSkipped ? test.canBeSkipped() : false;

test/www/jxcore/lib/SimpleTape.js, line 109 at r2 (raw file):

  .then(function (canBeSkipped) {
    if (canBeSkipped) {
      return self._skipTest(test);

Where is this function defined?

test/www/jxcore/lib/SimpleTape.js, line 111 at r2 (raw file):

      return self._skipTest(test);
    } else {
      return self._processTest(test);

Where is this function defined?

Comments from Reviewable

Review status: all files reviewed at latest revision, 5 unresolved discussions, some commit checks failed.

test/www/jxcore/lib/SimpleTape.js, line 99 at r2 (raw file):
Previously, yaronyg (Yaron Y Goland) wrote…
> Isn't this whole function ignored because _runTest is redefined on line 116?

Done.

test/www/jxcore/lib/SimpleTape.js, line 102 at r2 (raw file):
Previously, yaronyg (Yaron Y Goland) wrote…
> Isn't this code redundant with the code below in tape()?

Done.

test/www/jxcore/lib/SimpleTape.js, line 104 at r2 (raw file):
Previously, yaronyg (Yaron Y Goland) wrote…
> NB: I don't think it's a good idea to depend on the default return value, I know it's null and I know the right thing would happen but I think it's much cleaner to be explicit. E.g. return test.canBeSkipped ? test.canBeSkipped() : false;

Done.

test/www/jxcore/lib/SimpleTape.js, line 109 at r2 (raw file):
Previously, yaronyg (Yaron Y Goland) wrote…
> Where is this function defined?

Done.

test/www/jxcore/lib/SimpleTape.js, line 111 at r2 (raw file):
Previously, yaronyg (Yaron Y Goland) wrote…
> Where is this function defined?

Done.

Comments from Reviewable


Implement logic to validate dmbs set at changeset and store procedure levels

Impact

[x] Bug fix (non-breaking change which fixes expected existing functionality)
[ ] Enhancement/New feature (adds functionality without impacting existing logic)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Description
Implement validateDefinitions() methods to validate single or a list of dbms attribute values set for a given changeset or store procedure.
Fixes #1795
Things to be aware of

Validation logic basically relies on looking for a database a given definition (dbms).

Things to worry about

Nothing

Additional Context

Nothing

Code review and test results:
Things to be aware of:

Changes make sense to me
Can see it catching mistyped dbms attributes in integration tests

Things to worry about:

Nothing


enforce readability over preference.

Master and slave are conventions that have stuck with all kinds of technology for years. For someone that's new to the Swift source, the Leader/follower convention makes little sense.
I see no reason to change a practice that has been the norm for many years.

SlaveString must be slaveString

@phimage I realised this right before you said it, and have fixed it.

just an advice, squash your commit into one if you can, I see  this request on other PR

This just reverts the change in #165 which has been discussed, merged and locked already.

See #218. There's no hierarchical relationship between these threads.


ignite a tab-view : Command failed: yarn add react-native-MODULENAME

im trying to add a plugin. However when i'm running,

ignite a tab-view

DEPRECATION WARNING: Heads up! index.js in plugins is deprecated. Please use plugin.js instead!
Found in /PizzaApp/node_modules/ignite-tab-view/index.js.
✖ Command failed: yarn add react-native-MODULENAME
error An unexpected error occurred: "Couldn't find package "react-native-MODULENAME" on the "npm" registry.".

Seems like you need to change your plugin's code.  That's the default that the template spits out.
If you don't have any extra node_modules to install with your plugin, you can just remove that.


update weight default in route to service

What
Update route default to 100.
Based on my investigations the default value for the Route -> spec -> to -> weight is 100 when unspecified.

/assign @derekwaynecarr

/hold

@laurafitzgerald I think you might be missing running make generate to update generated files.

@soltysh yes I didn't do that. I'm new to operators. Will run now and push the change.

@soltysh make generate is looking for a specific version of protoc = 3.0.x. I've installed the latest as I assumed it would be happy with => but it still doesn't like it. Is also mentions that this step can be skipped. What's the correct way to proceed? Skip it or go to the specific version? Is it possible that newer versions would work but the check is very specific?

Looks like this import is not used

EDIT: generated changes now pushed: I also see some additional files in the vendor folder. Not sure if I should include those?

You don't touch vendor directory which holds dependencies. There's a separate script for bumping those but that's rarely used. The CI is green so the PR is good. Thanks.

@soltysh One final check failing on do-not-merge/hold labels.

@soltysh One final check failing on do-not-merge/hold labels.

Yes, that's intentional since we're closing 4.1 and don't want any change to our temporarily. Only critical bugs are being let in.

@soltysh One final check failing on do-not-merge/hold labels.

Yes, that's intentional since we're closing 4.1 and don't want any change to our temporarily. Only critical bugs are being let in.

Thanks for the info @soltysh Should I do anything else here then?

@laurafitzgerald nope, just ping me in a week or so in case this doesn't move, I might forget about it

/hold cancel


Incorporate display of " xxx performance tests run!"

Current Behavior
Meshery users send performance test results to the Meshery Cloud provider, but the total number of results collected isn't displayed on the site.
Desired Behavior
Similar to how https://layer5.io/meshery uses a public endpoint to display how many performance tests have been run, let's find a spot on meshery.io to display and explain this statistic.

@leecalcote sir, is this right place to show the performance test of Meshery?

@gunjanmimo @leecalcote , I think that fits in perfectly with the rest of the UI! I Think, we could go ahead with this.

I agree with @kanishkarj. Double check, if you would, @gunjanmimo, is there space that we might eliminate between the current number and the rest of the "performance tests run!" sentence?


That's odd, we don't have any control over the browser at all. Are you sure that's not a browser extension that's closing it? Can you try disabling all browser extensions and trying to repro?

This doesn't happen often but sometimes. and only when the browser is opened via cli. If the browser is already open nothing happens. Also I have no extensions in my browser currently.
I have no clue why that happens

@filipesilva . I was unable to pinpoint the problem. But this happens in almost all my projects. And at times, this is a pain. closing all other tabs

My guess would be that since the browser is not open before hand, the ng serve node process becomes the owner of the web browser process when the open option is used.  This may be an OS specific quirk.  The CLI uses the opn package to provide this functionality.

any workaround? :confused:


Build windows versions in Mac OS X Catalina (10.15.4)

After upgrading to Mac OS X 10.15.4 we noticed that wine 32 bits is not longer compatible with system. This module is dependent to winresourcer which executes 32 bits wine command.
When you try to execute wine in Mac OS X 10.15.4 you will get:
wine
bad CPU type in executable: wine

On the other hand the error dispatched by nw-builder is:

Unhandled rejection Error: spawn Unknown system error -86
    at _errnoException (util.js:1003:13)
    at ChildProcess.spawn (internal/child_process.js:340:11)
    at exports.spawn (child_process.js:499:9)
    at module.exports (/Users/user/projects/desktop/node_modules/winresourcer/lib/WinResourcer.js:70:14)
    at tryCatcher (/Users/user/projects/desktop/node_modules/bluebird/js/release/util.js:16:23)
    at ret (<anonymous>:15:23)
    at /Users/user/projects/desktop/node_modules/grunt-nw-builder/node_modules/nw-builder/lib/index.js:658:13
    at Promise._execute (/Users/user/projects/desktop/node_modules/bluebird/js/release/debuggability.js:303:9)
    at Promise._resolveFromExecutor (/Users/user/projects/desktop/node_modules/bluebird/js/release/promise.js:483:18)
    at new Promise (/Users/user/projects/desktop/node_modules/bluebird/js/release/promise.js:79:10)
    at /Users/user/projects/desktop/node_modules/grunt-nw-builder/node_modules/nw-builder/lib/index.js:654:22
    at /Users/user/projects/desktop/node_modules/grunt-nw-builder/node_modules/nw-builder/lib/index.js:761:16
    at /Users/user/projects/desktop/node_modules/grunt-nw-builder/node_modules/lodash/lodash.js:4938:15
    at baseForOwn (/Users/user/projects/desktop/node_modules/grunt-nw-builder/node_modules/lodash/lodash.js:2988:24)
    at /Users/user/projects/desktop/node_modules/grunt-nw-builder/node_modules/lodash/lodash.js:4907:18
    at Function.forEach (/Users/user/projects/desktop/node_modules/grunt-nw-builder/node_modules/lodash/lodash.js:9319:14)
    at NwBuilder._forEachPlatform (/Users/user/projects/desktop/node_modules/grunt-nw-builder/node_modules/nw-builder/lib/index.js:760:7)
    at NwBuilder.handleWinApp (/Users/user/projects/desktop/node_modules/grunt-nw-builder/node_modules/nw-builder/lib/index.js:650:10)
    at NwBuilder.tryCatcher (/Users/user/projects/desktop/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/Users/user/projects/desktop/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/Users/user/projects/desktop/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/Users/user/projects/desktop/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/Users/user/projects/desktop/node_modules/bluebird/js/release/promise.js:693:18)
    at Promise._fulfill (/Users/user/projects/desktop/node_modules/bluebird/js/release/promise.js:638:18)
    at PromiseArray._resolve (/Users/user/projects/desktop/node_modules/bluebird/js/release/promise_array.js:126:19)
    at PromiseArray._promiseFulfilled (/Users/user/projects/desktop/node_modules/bluebird/js/release/promise_array.js:144:14)
    at Promise._settlePromise (/Users/user/projects/desktop/node_modules/bluebird/js/release/promise.js:574:26)
    at Promise._settlePromise0 (/Users/user/projects/desktop/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/Users/user/projects/desktop/node_modules/bluebird/js/release/promise.js:693:18)
    at Async._drainQueue (/Users/user/projects/desktop/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/Users/user/projects/desktop/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues [as _onImmediate] (/Users/user/projects/desktop/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:756:18)
    at tryOnImmediate (timers.js:717:5)
    at processImmediate [as _immediateCallback] (timers.js:697:5)

For developers using mac to generate builds of an application for mac and windows this is an important issue because they no longer can use the nw-builder module to do it.

:+1: Running into the same thing on macOS 10.15.3
Here is a related issue for electron-forge, https://github.com/electron-userland/electron-forge/issues/1122#issuecomment-565099198 which suggeted installing the latest wine from homebrew to get wine64 installed

Let me know if this issue still happens.


Compiled code no longer backwards

Resolves the issues brought up in https://github.com/tangrams/tangram/pull/425
Our unicode range strings to detect RTL text were being converted to their textual equivalents upon bundling. This caused some very bizarre errors, such as the appearance of unicode characters such as ﷽ in our source, in addition to much of our source code being backwards.
The fix is to create the RegEx directly, instead of building it out of these strings that will get corrupted upon bundling. Thanks to @batje for spotting this!

Niiiiiice. 🔍 👀 👌

DUDE! :)
On Fri, Oct 14, 2016 at 9:11 AM, Peter Richardson<EMAIL_ADDRESS>wrote:

Niiiiiice. 🔍 👀 👌
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tangrams/tangram/pull/426#issuecomment-253847383, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA0EO3tNK-ZFdsRCNV2myXbwCIqKOLe2ks5qz6mcgaJpZM4KXKSU
.

tnellecxe

@dmvaldman does this PR supersede #425?

It should be used instead of #425

Released in https://github.com/tangrams/tangram/releases/tag/v0.10.6

Thanks for that explanation @dmvaldman, I finally understand what I reported now!

I'm glad you submitted the issue! That was a wild rabbit hole to go down.


Problems with loading pretrained models

When I try to run the test code I got
ModuleNotFoundError: No module named 'networks.relay_net'
It seems the "relaynet_pytorch" folder used to be called "networks" in your pretrained model.
Could you change the folder names back to "networks" to avoid such confusions? Thank you!

sorry ,so how can you solve the error "No module named 'networks.relay_net"? can you tell me you solution in your spare time?

sorry ,so how can you solve the error "No module named 'networks.relay_net"? can you tell me you solution in your spare time?

When I try to run the test code I got ModuleNotFoundError: No module named 'networks.relay_net' It seems the "relaynet_pytorch" folder used to be called "networks" in your pretrained model. Could you change the folder names back to "networks" to avoid such confusions? Thank you!

sorry ,so how can you solve the error "No module named 'networks.relay_net"? can you tell me you solution in your spare time?


How to pass parameters to flutter web app

This query was originally asked on StackOverflow but since I have got no reply and there is no particular answer resolving this query I am posting it here
https://stackoverflow.com/q/59698200/9236994
After hours of searching about the topic and due to lack of documentation on Flutter Web I am asking this question.
I was trying to create a web app using flutter and had an requirement where URL such as below
website.com/user/someUserCode
would be called and an page will be launched where the data (someUserCode) will be passed to the page
but haven't got any solutions yet to resolve it.
so just rounding it all up,
How to pass and fetch the data using (get / post) methods to flutter web app?
What all I know / have tried yet
I am using below code to read if some parameters are being to some class file
final Map<String, String> params = Uri.parse(html.window.location.href).queryParameters;
String data = params["userData"];

all this actually solves the Fetch part of my question (maybe)
but the part where that data will be passed to the page via URL is still missing.

You can get with:
Uri.base.queryParameters;  =>  Map<String,String>

So.... any updates at ALL ?

need this too

need this too

This will be very useful when you are adding flutter to existing web-app's
If there is any mechanism to send the parameter to "main" function from the HTML page it loads, it will be awesome.
Then we can have one flutter app with multiple HTML's that can define the HOME ROUTE (without parsing the query string as its not reliable for us)
Example of what I mean:
Currently my web app contains following pages (based on ASP.NET legacy) (based on the menu configs and roles, user can navigate to any page):
Page1.aspx
Page2.aspx
....
Page99.aspx

I can create a single flutter web app that can have say Page1 and Page 5 that we decide to move to flutter web
If the "main" function in flutter can receive a parameter then I can define in
a. Page1.aspx can send parameter to "loadMainDartJs('Page1')"
b. Page5.aspx can send parameter to "loadMainDartJs('Page5')"

Someone get a workaround ?

I have a similar requirement as well. We already have a React web site which users can log in. Once they are logged in they will be redirected to the account page which is written with Flutter Web. So;
mysite.com/auth -> React
mysite.com/account -> Flutter web
And I want to pass the session info from React to Flutter web since we can't expect users to sign in with Flutter again.
I couldn't find any resources like the OP. So what is the suggested way to have a hybrid website with Flutter.
We want to convert our website page by page to Flutter web. Any help is appreciated. thanks


Development trc

Description
Add workflow to create an image with trc
Checklist

[ ] Pre-commit hook is installed to do formatting checks before committing code...etc
[ ] Tests included
[ ] Build pass
[ ] Documentation
[ ] Code format and docstrings

Codecov Report

Merging #1891 (a3f4323) into development (109ef77) will increase coverage by 0.10%.
The diff coverage is n/a.

@@               Coverage Diff               @@
##           development    #1891      +/-   ##
===============================================
+ Coverage        32.36%   32.46%   +0.10%
===============================================
  Files              134      136       +2
  Lines            12851    13168     +317
===============================================
+ Hits              4159     4275     +116
- Misses            8692     8893     +201

Impacted Files
Coverage Δ

jumpscale/tools/notificationsqueue/__init__.py
33.33% <0.00%> (-66.67%)
:arrow_down:

jumpscale/packages/admin/services/stellar.py
37.03% <0.00%> (-51.43%)
:arrow_down:

jumpscale/tools/servicemanager/servicemanager.py
62.88% <0.00%> (-20.45%)
:arrow_down:

jumpscale/tools/notificationsqueue/queue.py
30.43% <0.00%> (-14.50%)
:arrow_down:

jumpscale/clients/stellar/__init__.py
29.82% <0.00%> (-7.02%)
:arrow_down:

jumpscale/sals/nginx/nginx.py
85.06% <0.00%> (-2.15%)
:arrow_down:

jumpscale/packages/admin/actors/admin.py
29.90% <0.00%> (-1.17%)
:arrow_down:

jumpscale/packages/tfgrid_solutions/utils.py
16.17% <0.00%> (-1.02%)
:arrow_down:

jumpscale/sals/zos/kubernetes.py
26.66% <0.00%> (-0.92%)
:arrow_down:

jumpscale/packages/auth/bottle/auth.py
25.82% <0.00%> (-0.89%)
:arrow_down:

... and 17 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 109ef77...a3f4323. Read the comment docs.


[Feature] - Relying Party Entity Configuration

According to the Trust Model section, the Verifier is a Federation Entity and MUST expose a .well-known endpoint containing its Entity Configuration.
Refer to entity_configuration method and backend configuration.
Below a non-normative example of the request made by the Wallet Instance to the openid-federation .well-known endpoint to obtain the Relying Party Entity Configuration:

[X] Add metadata to configuration
[x] Method Yaml to Json
[x] Return json metadata
[ ] Update Readme.md with the parameters to change for the correct execution

@peppelinux  the issue is complete but we have a dubt regarding to create a file readme for documents methods, suggest to use a swagger file for this scope.
confirm It?

the endpoints are documented in the technical specs
before doing a swagger (similarly made in projects like this) we need a short, clear and concrete documentation that explain how to setup, configure and execute this project
the api will be refactored and probably change before deciding to have a oas3 specs, that should come after the release 0.6.0


Use js to show status of gh issues and PR

See #1895.
Not that I heve the right skills and am good at desing, just show what could be done.
Need to be pulled into a separate JS file and properely desinged.

With my less than optimal design skills.

Can we add @isabela-pf to the repo (even only with read access) so that we can assign her to review ?

I wonder if the tool that mystmd.org uses for GitHub hover cards could be repurposed here?
@agoose77 any thoughts on that?

Can we add @isabela-pf to the repo (even only with read access) so that we can assign her to review ?

invitation sent!

Thanks for the links to #1085, I missed it. I think this is indeed much narrower in scope.

2. The addition of icons also makes sense considering GitHub as a reference. Do the icons get announced as a part of the link or in the tooltip, or are they visual only?

Currently those are just visual, we can change if we want.

3. I think it was a good idea to map the status colors in the GitHub reference to the ones within the theme, and I think appropriate colors were chosen. However, in this situation, I do worry that they are close in value (light/darkness) and could be easily mixed up due to colorblindness, low or bright light situations, and any other situation where color is harder to discern. If the status is reiterated in the tooltip or in a text precursor to the #xxxx I think this is acceptable. If color is the only way to tell this status, I'd like to advise an additional way of communicating.

I think that's my main question, I'm really not good ad design/color. If you think there are changes we need to do and you can come up with colors that work with PST that would help me. I've mostly use red/green/purple because I know that's what GitHub is doing.
I don't want to bias the design process, but should the link be colors the same way as the icon, and should for example closed issues be strikethrough. I'm tempted to think no for strikethrough as most issue reference in documentation will be to merged PR and closed issues.

I likely need to use font-awesome 6 instead of 5.

I'm happy to use octicon, I just wanted to avoid an extra dependnecy by default.
I'klk see if I can make it work

We should be using FA 6 (this is the one we are vendoring with the theme).

We should be using FA 6 (this is the one we are vendoring with the theme).

Also despite my earlier comment about octicons, I won't object to using FA6 equivalents --- we can always switch to octicons later if necessary.

Sorry this was meant more at @Carreau since he is currently using FA 5

here's a screenshot of how this will look:

IMO we're not there yet:

open and merged PRs have same color
underline on the icon looks odd
I think I prefer the issue icon to be the "bullseye" (like github uses) rather than the "i in a circle", even though the FontAwesome bullseye doesn't perfectly match the Octicon one
not sure the "how to update kitchen sink" page is the right place in the docs to demo this. Maybe this section? https://pydata-sphinx-theme--1896.org.readthedocs.build/en/1896/user_guide/theme-elements.html#link-shortening-for-git-repository-services
viewing the link-shortening section in the prior bullet shows that the new JS is not applying correctly site-wide (possibly related to issues/ vs pull/ in the URL?)

open and merged PRs have same color

Nah, that is because the PR I linked to has been merged since then...

underline on the icon looks odd

I did not figured out how to not underline it
I'll have a look at the rest.


Timelapse MP4 not completed at end of night

Hi,
Would anyone be able to point me in the right direction to solve the Timelapse generated being only 4kB and not containing any image sequence?  It seems to have been generated at the right time looking at the file timestamp, but just nothing in the file to view.
Keograms, startrails and general images are saved and viewable, so I'm just a little stuck on the reason the timelapse isn't working.
All the best
Mark

Mark,
Try running it manually to see what happens.  If you have a model 3 you may need to use swap space.
To run:  scripts/timelapse.sh date        where "date" is the day you want to process, e.g., 20210811

Windows ‘preview’ each image to find and delete a bad one that prevents mp4 creation.
Sent from my iPad

On Aug 13, 2021, at 2:45 PM, EricClaeys @.***> wrote:

Mark,
Try running it manually to see what happens. If you have a model 3 you may need to use swap space.
To run: scripts/timelapse.sh date where "date" is the day you want to process, e.g., 20210811
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.

If you do find a bad one that prevents mp4 creation, please upload it before deleting it.  I want to see if "REMOVE_BAD_IMAGES" would catch it.

Thanks Eric and Bill,
All images look fine and appear to be 'good' jpegs.
I ran the script manually and still no joy, but looking at the output window of the movie process, I see the following output (just final processing section included) with a possible issue (highlighted in the output text) that I can't relate to the camera or script settings - does this look normal?  But also, there's a lot of 0 bytes in the frame selection, which is puzzling as well?
Many thanks
Mark
Input #0, image2, from 'images/20210813/sequence/%04d.jpg':
Duration: 00:00:51.00, start: 0.000000, bitrate: N/A
Stream #0:0: Video: mjpeg, yuvj420p(pc, bt470bg/unknown/unknown), 3096x2080 [SAR 1:1 DAR 387:260], 25 fps, 25 tbr, 25 tbn, 25 tbc
Stream mapping:
Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[swscaler @ 0x1c8f3b0] deprecated pixel format used, make sure you did set range correctly
[libx264 @ 0x1c82f80] using SAR=1/1
[libx264 @ 0x1c82f80] using cpu capabilities: ARMv6 NEON
[libx264 @ 0x1c82f80] profile High, level 5.1
[libx264 @ 0x1c82f80] 264 - core 155 r2917 0a84d98 - H.264/MPEG-4 AVC codec - Copyleft 2003-2018 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=abr mbtree=1 bitrate=2000 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'images/20210813/allsky-20210813.mp4':
Metadata:
encoder         : Lavf58.20.100
Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 3096x2080 [SAR 1:1 DAR 387:260], q=-1--1, 2000 kb/s, 25 fps, 12800 tbn, 25 tbc
Metadata:
encoder         : Lavc58.35.100 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 0 vbv_delay: -1
frame=    2 fps=0.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=    3 fps=2.1 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=    4 fps=2.1 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=    5 fps=2.1 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=    6 fps=2.1 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=    8 fps=2.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=    9 fps=2.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   10 fps=2.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   12 fps=2.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   13 fps=2.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   15 fps=2.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   16 fps=2.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   17 fps=2.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   18 fps=2.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   19 fps=2.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   20 fps=1.9 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   21 fps=1.9 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   22 fps=1.9 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   23 fps=1.9 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   24 fps=1.9 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   25 fps=1.9 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   26 fps=1.9 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   28 fps=1.9 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   29 fps=1.9 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   30 fps=1.9 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   31 fps=1.9 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   32 fps=1.9 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   33 fps=1.9 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   34 fps=1.9 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   35 fps=1.9 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   36 fps=1.9 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   37 fps=1.9 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   38 fps=1.9 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   ./timelapse.sh: line 45: 24594 Killed                  ffmpeg -y -f image2 -r $FPS -i images/$1/sequence/%04d.$EXTENSION -vcodec libx264 -b:v 2000k -pix_fmt yuv420p -movflags +faststart $SCALE images/$1/allsky-$1.mp4

Deleting sequence
Timelapse was created

The "deprecated pixel format" message can be ignored.  The 0's are troublesome.
It looks like you are running the older verdion 0.7 release?
Create a directory in allsky/images called "test" and copy three images to it.  Then run "scripts/timelapse.sh test" and see what you get.
In the output should be messages for the "ln -sv" commands for each input file.  Are you seeing those?

Thanks Eric,
It does appear I'm running 0.7 - this was just the install from the Github instructions that I followed, so I may need an update?  Forgive me on this as I'm not really very familiar with Raspberry Pi applications.
I ran 'test' with just 3 images and an MP4 was created.  It seems all three images were in the timelapse video, and the output was as follows:
Input #0, image2, from 'images/test/sequence/%04d.jpg':
Duration: 00:00:00.12, start: 0.000000, bitrate: N/A
Stream #0:0: Video: mjpeg, yuvj420p(pc, bt470bg/unknown/unknown), 3096x2080 [SAR 1:1 DAR 387:260], 25 fps, 25 tbr, 25 tbn, 25 tbc
Stream mapping:
Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[swscaler @ 0x152f6b0] deprecated pixel format used, make sure you did set range correctly
[libx264 @ 0x15277f0] using SAR=1/1
[libx264 @ 0x15277f0] using cpu capabilities: ARMv6 NEON
[libx264 @ 0x15277f0] profile High, level 5.1
[libx264 @ 0x15277f0] 264 - core 155 r2917 0a84d98 - H.264/MPEG-4 AVC codec - Copyleft 2003-2018 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=abr mbtree=1 bitrate=2000 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'images/test/allsky-test.mp4':
Metadata:
encoder         : Lavf58.20.100
Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 3096x2080 [SAR 1:1 DAR 387:260], q=-1--1, 2000 kb/s, 25 fps, 12800 tbn, 25 tbc
Metadata:
encoder         : Lavc58.35.100 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 0 vbv_delay: -1
frame=    2 fps=0.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=    3 fps=2.5 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   [mp4 @ 0x151ffd0] Starting second pass: moving the moov atom to the beginning of the file
frame=    3 fps=0.5 q=-1.0 Lsize=      65kB time=00:00:00.00 bitrate=6856000.0kbits/s speed=1.4e-05x
video:64kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.347848%
[libx264 @ 0x15277f0] frame I:1     Avg QP:28.85  size: 48688
[libx264 @ 0x15277f0] frame P:1     Avg QP:29.85  size: 13051
[libx264 @ 0x15277f0] frame B:1     Avg QP:32.42  size:  3513
[libx264 @ 0x15277f0] consecutive B-frames: 33.3% 66.7%  0.0%  0.0%
[libx264 @ 0x15277f0] mb I  I16..4: 38.9% 60.2%  0.9%
[libx264 @ 0x15277f0] mb P  I16..4:  7.7% 10.6%  0.0%  P16..4: 10.3%  0.8%  3.0%  0.0%  0.0%    skip:67.6%
[libx264 @ 0x15277f0] mb B  I16..4:  0.1%  0.0%  0.0%  B16..8: 13.9%  0.1%  0.0%  direct: 0.1%  skip:85.8%  L0:14.7% L1:84.8% BI: 0.5%
[libx264 @ 0x15277f0] final ratefactor: 36.50
[libx264 @ 0x15277f0] 8x8 transform intra:59.8% inter:99.7%
[libx264 @ 0x15277f0] coded y,uvDC,uvAC intra: 10.9% 28.9% 3.4% inter: 1.0% 4.2% 0.0%
[libx264 @ 0x15277f0] i16 v,h,dc,p: 41% 30%  7% 22%
[libx264 @ 0x15277f0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 36% 15% 32%  5%  2%  1%  1%  2%  6%
[libx264 @ 0x15277f0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 17% 24% 19%  9%  5%  3%  4%  3% 14%
[libx264 @ 0x15277f0] i8c dc,h,v,p: 73% 16%  9%  2%
[libx264 @ 0x15277f0] Weighted P-Frames: Y:100.0% UV:100.0%
[libx264 @ 0x15277f0] kb/s:4350.13

Deleting sequence
Timelapse was created

I then tried adding a further 30 images to test.  Running the script again produced the same result as my original problem.
So, I then tried 5 and 10 frames, which worked!  Making the total frames 20-off broke it again.
I still see 0 bytes for even the apparently successful lower number of frames in the process output, so there is something a little odd here.
Many thanks for your help.
Cheers
Mark

Mark,
I suspect the ffmpeg command is running out of memory and you'll need to add swap space, but just to make sure it's not a bad image causing the problem, take the 10 images that worked and copy them a couple times so you have 30 images that you know work.  Then try running it.  If it fails, it's not the images themselves.  Google how to add swap space.  Some people on the AllSky forum have mentioned it but I didn't pay attention.

I didn't read all the messages for this problem but like all Linux/Unix,
everything is case sensitive.  Make sure your image output is in lower
case, jpg and not JPG, The FFMPEG command is looking for lower case jpg
files only.     This was a problem I experienced long ago.  Bill
On Sat, Aug 14, 2021 at 12:29 PM EricClaeys @.***>
wrote:

Mark,
I suspect the ffmpeg command is running out of memory and you'll need to
add swap space, but just to make sure it's not a bad image causing the
problem, take the 10 images that worked and copy them a couple times so you
have 30 images that you know work. Then try running it. If it fails, it's
not the images themselves. Google how to add swap space. Some people on the
AllSky forum have mentioned it but I didn't pay attention.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/thomasjacquin/allsky/issues/420#issuecomment-898949196,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEELBXQ6ZDOAQQUCZHCU5Y3T427YXANCNFSM5CDLF3PA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
.

In the next update to capture.cpp I'll convert uppercase extensions to lowercase, e.g., "image.JPG" to "image.jpg".  That will eliminate the issue with ffmpeg.

Thanks Eric,
Increasing the swap size worked perfectly, so it now generates whole night timelapse.  It does take over an hour to generate though!!
As far as I can see, the capture code is correctly saving .jpgs , but it is a good idea to trap all logical text case possibilities.
Many thanks for your help on this - for anyone else reading this with a Pi3 Model B, setting the swap file to 2048 Mb works well, but I wouldn't set it higher than this due to speed and memory card limitations.
All the best
Mark

Mark,
Glad the change fixed your problem.
Do you have a link to directions on how to set the swap space?  I would like to include them in the README.md file.

Hi Eric,
Yes, I used this procedure and it worked well:
https://pimylifeup.com/raspberry-pi-swap-file/
All the best
Mark


Spport for ChatGLM3

Great work! What's your suggestion if I would like to test it on ChatGLM3?

Thanks for your interest! You can modifiy the modeling_chatglm.py in huggingface similar to our implementation in llama_kivi.py or mistral_kivi.py.
We are planning to extend KIVI to more models and benchmark datasets. Meanwhile, we welcome contributions from the research community.


Simplify Testsuite Customization docs into a checklist

The goal is to make onboarding a much simpler, less reading-intensive process.
Perhaps this could be the Cassandra example we use for onboarding session?

Related to the work @galenmarchetti is doing with https://github.com/kurtosis-tech/kurtosis-onboarding-experience

Closing this in favor of the work @leoporoli is doing on https://github.com/kurtosis-tech/kurtosis-onboarding-experience


Remove direct dependency on ttf_parser

Because ttf_parser updated, rustybuzz was using an older version.

It looks like tests failed, could you please fix that?