Help me

hello everyone I have an HP hybrid azzurro zcs inverter, I tried to install the most updated version, but if I select the template for zcs hybrid nothing works I don't receive readings.  can someone help me?

Maybe it's better to put these messages in discussions and not in issues


Feature/update php support

Description
Drop support for php 7.2 and added support for php 8.0
Fixes # 33324, 33326
How Has This Been Tested?
Build is green

Looks good!


NewtonSoft.Json crash

NewtonSoft.Json crashed yesterday please help guys
https://i.gyazo.com/230cf59d6c4d22dba197f56ea3d29728.png
https://i.gyazo.com/b48d3e88cafe207e6db59b92734a64a5.png

This should be fixed by #913
I'm still waiting on anyone to verify that this PR works so we can get it merged.

https://i.gyazo.com/a39fac31fb0e1928555e442e1e63bdc8.png
https://i.gyazo.com/b05e758b567c0ee0c1ee3a93c3b3d2d7.png
Problem not fixed pls help bro

Those screenshots clearly indicate you didn't merge the PR into your own repo.  See https://github.com/Jessecar96/SteamBot/issues/919#issuecomment-179528263

I cant PR because no have permissions.

I think I'm going to have to close this with #427 #527, this is an API meant for developers, not a pre-built bot.  Sorry.


SSL: error:14094123:SSL routines:ssl3_read_bytes:application data after close notify

xrdp version
0.10.0
Detailed xrdp version, build options
Paste the result between `~~~`.  Please DO NOT remove `~~~`!

Operating system & version
Rocky Linux 8
Installation method
dnf / apt / zypper / pkg / etc
Which backend do you use?
tigvnc   1.13.1-11.el8_10
What desktop environment do you use?
macbook
Environment xrdp running on
Rocky Linux 8
What's your client?
MS remote desktop form AppStore.
Area(s) with issue?
Network
Steps to reproduce

installd xrdp
configure in MS remote desktop
connect

✔ Expected Behavior
open remote destkop
❌ Actual Behavior
MS remote desktop shutdown the opened window.
Jul 23 18:57:18  localhost xrdp[122532]: [ERROR] VNC server closed connection
Jul 23 18:57:18  localhost xrdp[122532]: [ERROR] SSL_shutdown: Failure in SSL library (protocol error?)
Jul 23 18:57:18  localhost xrdp[122532]: [ERROR] SSL: error:14094123:SSL routines:ssl3_read_bytes:application data after close notify
Anything else?
No response

@wahello - we need some more information.

We need to know how you have installed xrdp. Filling in the Detailed xrdp version, build options section above will help.
can you post the contents of /var/log/xrdp.log from around the time the problem was encountered?
can you post the contents of /var/log/xrdp-sesman.log from around the time the problem was encountered?

Finally, are you sure the RDP user is not logged in on the console? (see the FAQ)

@matt335672

dnf install xrdp, from epel.
/var/log/xrdp.log

[2024-07-24T11:36:06.826+0800] [INFO ] Socket 14: connection accepted from [::ffff:<IP_ADDRESS>]:58750
[2024-07-24T11:36:06.846+0800] [INFO ] Using default X.509 certificate: /etc/xrdp/cert.pem
[2024-07-24T11:36:06.861+0800] [INFO ] Using default X.509 key file: /etc/xrdp/key.pem
[2024-07-24T11:36:06.878+0800] [INFO ] Security protocol: configured [SSL|RDP], requested [SSL|HYBRID|HYBRID_EX|RDP], selected [SSL]
[2024-07-24T11:36:06.926+0800] [INFO ] Connected client computer name: macbookp
[2024-07-24T11:36:06.945+0800] [INFO ] client supports gfx protocol
[2024-07-24T11:36:06.962+0800] [WARN ] Received [MS-RDPBCGR] TS_UD_HEADER type 0xc006 is unknown (ignored)
[2024-07-24T11:36:06.978+0800] [WARN ] Received [MS-RDPBCGR] TS_UD_HEADER type 0xc00a is unknown (ignored)
[2024-07-24T11:36:06.998+0800] [INFO ] xrdp_load_keyboard_layout: Keyboard information sent by the RDP client, keyboard_type:[0x04], keyboard_subtype:[0x00], keylayout:[0x00000409]
[2024-07-24T11:36:07.206+0800] [INFO ] xrdp_load_keyboard_layout: model [] variant [] layout [us] options []
[2024-07-24T11:36:07.371+0800] [INFO ] TLS connection established from [::ffff:<IP_ADDRESS>]:58750 TLSv1.3 with cipher TLS_AES_256_GCM_SHA384
[2024-07-24T11:36:07.768+0800] [INFO ] Client supports single-screen resizes by xrdp
[2024-07-24T11:36:07.954+0800] [INFO ] xrdp_caps_process_pointer: client supports new(color) cursor
[2024-07-24T11:36:07.112+0800] [INFO ] xrdp_process_offscreen_bmpcache: support level 0 cache size 0 MB cache entries 0
[2024-07-24T11:36:07.128+0800] [INFO ] xrdp_caps_process_codecs: NSCodec(CA8D1BB9-000F-154F-589F-AE2D1A87E2D6), codec id [1], properties len [3]
[2024-07-24T11:36:07.145+0800] [INFO ] Client Capability: LARGE_POINTER_FLAG_96x96 supported
[2024-07-24T11:36:07.165+0800] [INFO ] Loading keymap file /etc/xrdp/km-00000409.ini
[2024-07-24T11:36:07.187+0800] [WARN ] local keymap file for 0x00000409 found and doesn't match built in keymap, using local keymap file
[2024-07-24T11:36:07.204+0800] [INFO ] Login screen monitor height is 1080 pixels over 335 mm (81 DPI)
[2024-07-24T11:36:07.223+0800] [INFO ] xrdp_egfx_create: error 0 channel_id 1
[2024-07-24T11:36:07.245+0800] [INFO ] xrdp_mm_egfx_caps_advertise:
[2024-07-24T11:36:07.262+0800] [INFO ]   version 0x00080004 flags 0x00000002 (index: 0)
[2024-07-24T11:36:07.279+0800] [INFO ]   version 0x00080105 flags 0x00000002 (index: 1)
[2024-07-24T11:36:07.295+0800] [INFO ]   version 0x000a0002 flags 0x00000022 (index: 2)
[2024-07-24T11:36:07.312+0800] [INFO ]   version 0x000a0200 flags 0x00000022 (index: 3)
[2024-07-24T11:36:07.329+0800] [INFO ]   version 0x000a0301 flags 0x00000020 (index: 4)
[2024-07-24T11:36:07.345+0800] [INFO ]   version 0x000a0400 flags 0x00000002 (index: 5)
[2024-07-24T11:36:07.362+0800] [INFO ]   version 0x000a0701 flags 0x00000082 (index: 6)
[2024-07-24T11:36:07.379+0800] [INFO ]   version 0x000b0101 flags 0x00000082 (index: 7)
[2024-07-24T11:36:07.395+0800] [INFO ] unknown version 0x000b0101
[2024-07-24T11:36:07.412+0800] [INFO ]   replying version 0x000a0701 flags 0x00000082
[2024-07-24T11:36:07.429+0800] [INFO ] xrdp_mm_egfx_caps_advertise: xrdp_egfx_send_capsconfirm error 0 best_index 6
[2024-07-24T11:36:07.445+0800] [INFO ] xrdp_egfx_reset_graphics:
[2024-07-24T11:36:07.462+0800] [INFO ] xrdp_egfx_reset_graphics: width 1920 height 1080 monitorcount 1
[2024-07-24T11:36:07.479+0800] [INFO ] xrdp_mm_egfx_caps_advertise: xrdp_egfx_send_reset_graphics error 0 monitorCount 0
[2024-07-24T11:36:07.496+0800] [INFO ] xrdp_mm_egfx_create_surfaces: map surface_id 0 left 0 top 0 width 1920 height 1080
[2024-07-24T11:36:07.512+0800] [INFO ] xrdp_encoder_create: starting gfx rfx pro codec session
[2024-07-24T11:36:07.529+0800] [INFO ] xrdp_mm_egfx_invalidate_wm_screen:
[2024-07-24T11:36:07.546+0800] [INFO ] xrdp_mm_egfx_caps_advertise: egfx created.
[2024-07-24T11:36:07.562+0800] [WARN ] Ignoring obsolete SCP port value '3350'
[2024-07-24T11:36:07.579+0800] [INFO ] xrdp_wm_log_msg: sesman connect ok
[2024-07-24T11:36:07.596+0800] [INFO ] xrdp_wm_log_msg: Performing login request for xxx
[2024-07-24T11:36:07.721+0800] [INFO ] xrdp_wm_log_msg: login was successful - creating session
[2024-07-24T11:36:08.854+0800] [INFO ] xrdp_wm_log_msg: session is available on display 10 for user xxx
[2024-07-24T11:36:08.871+0800] [INFO ] xrdp_wm_log_msg: Connecting to session
[2024-07-24T11:36:08.888+0800] [INFO ] loaded module 'libvnc.so' ok, interface size 4832, version 4
[2024-07-24T11:36:08.913+0800] [INFO ] VNC: Clipboard (if available) is provided by chansrv facility
[2024-07-24T11:36:08.929+0800] [INFO ] xrdp_wm_log_msg: Connecting to chansrv
[2024-07-24T11:36:08.963+0800] [INFO ] Layout from ClientLayout (geom=1920x1080 #screens=1) : 0:(1920x1080+0+0)
[2024-07-24T11:36:09.295+0800] [INFO ] Layout from OldServerLayout (geom=1920x1080 #screens=1) :<PHONE_NUMBER>:(1920x1080+0+0)
[2024-07-24T11:36:09.581+0800] [ERROR] VNC server closed connection
[2024-07-24T11:36:09.679+0800] [ERROR] SSL_shutdown: Failure in SSL library (protocol error?)
[2024-07-24T11:36:09.804+0800] [ERROR] SSL: error:14094123:SSL routines:ssl3_read_bytes:application data after close notify

3./var/log/xrdp-sesman.log
[2024-07-24T11:36:07.579+0800] [INFO ] Socket 13: connection accepted from AF_UNIX
[2024-07-24T11:36:07.621+0800] [INFO ] Received system login request from xrdp for user: xxx IP: ::ffff:<IP_ADDRESS>
[2024-07-24T11:36:07.639+0800] [INFO ] starting xrdp-sesexec with pid 324928
[2024-07-24T11:36:07.680+0800] [INFO ] TerminalServerUsers group tsusers doesn't exist. Access granted for xxx
[2024-07-24T11:36:07.704+0800] [INFO ] Access permitted for user: xxx
[2024-07-24T11:36:07.721+0800] [INFO ] Received sys login status for xxx : logged in
[2024-07-24T11:36:07.737+0800] [INFO ] Received request from xrdp to create a session for user xxx
[2024-07-24T11:36:07.765+0800] [INFO ] Starting X server on display 10: Xvnc :10 -auth .Xauthority -geometry 1920x1080 -depth 32 -rfbauth /home/xxx/.vnc/sesman_passwd-xxx@yyyy:10 -bs -nolisten tcp -localhost -dpi 96
[2024-07-24T11:36:08.767+0800] [INFO ] X server :10 is working
[2024-07-24T11:36:08.787+0800] [INFO ] Starting window manager for display :10
[2024-07-24T11:36:08.804+0800] [INFO ] Starting the xrdp channel server for display :10
[2024-07-24T11:36:08.805+0800] [INFO ] Using the default window manager on display 10: /usr/libexec/xrdp/startwm-bash.sh
[2024-07-24T11:36:08.821+0800] [INFO ] Session in progress on display :10. Waiting until the window manager (pid 324960) exits to end the session
[2024-07-24T11:36:09.357+0800] [INFO ] Window manager (pid 324960, display 10) finished normally in 1 secs
[2024-07-24T11:36:09.446+0800] [WARN ] Window manager (pid 324960, display 10) exited quickly (1 secs). This could indicate a window manager config problem
[2024-07-24T11:36:09.512+0800] [INFO ] Terminating X server (pid 324932) on display :10
[2024-07-24T11:36:09.580+0800] [INFO ] Terminating the xrdp channel server (pid 324961) on display :10
[2024-07-24T11:36:09.679+0800] [INFO ] X server pid 324932 on display :10 finished
[2024-07-24T11:36:10.107+0800] [INFO ] xrdp channel server pid 324961 on display :10 finished
[2024-07-24T11:36:10.465+0800] [INFO ] Session on display 10 has finished.

Thanks.
Your session manager is exiting quickly:-
[2024-07-24T11:36:08.821+0800] [INFO ] Session in progress on display :10. Waiting until the window manager (pid 324960) exits to end the session
[2024-07-24T11:36:09.357+0800] [INFO ] Window manager (pid 324960, display 10) finished normally in 1 secs
[2024-07-24T11:36:09.446+0800] [WARN ] Window manager (pid 324960, display 10) exited quickly (1 secs). This could indicate a window manager config problem

Can you confirm the RDP user is not logged in on the console?

This is a newly added part of xrdp-sesman.log:
[2024-07-25T09:27:20.544+0600] [INFO ] X server :10 is working
[2024-07-25T09:27:20.564+0600] [INFO ] Starting window manager for display :10
[2024-07-25T09:27:20.582+0600] [INFO ] Starting the xrdp channel server for display :10
[2024-07-25T09:27:20.582+0600] [INFO ] Using the default window manager on display 10: /usr/libexec/xrdp/startwm-bash.sh
[2024-07-25T09:27:20.598+0600] [INFO ] Session in progress on display :10. Waiting until the window manager (pid 416204) exits to end the session
[2024-07-25T09:27:21.119+0600] [INFO ] Window manager (pid 416204, display 10) finished normally in 1 secs
[2024-07-25T09:27:21.216+0600] [WARN ] Window manager (pid 416204, display 10) exited quickly (1 secs). This could indicate a window manager config problem
[2024-07-25T09:27:21.298+0600] [INFO ] Terminating X server (pid 416166) on display :10
[2024-07-25T09:27:21.374+0600] [INFO ] Terminating the xrdp channel server (pid 416205) on display :10
[2024-07-25T09:27:21.465+0600] [INFO ] X server pid 416166 on display :10 finished
[2024-07-25T09:27:21.707+0600] [INFO ] xrdp channel server pid 416205 on display :10 finished
[2024-07-25T09:27:21.740+0600] [INFO ] Session on display 10 has finished.

Window manager pid is changed, once I connect to xrdp.

It was resolved by the following
systemctl set-default  multi-user.target
reboot

It from this.
It's amazing.


drivers: clock_control: Allow PLL clock configuration with HSI on STM…

…32F1xx
The STM32F1xx allow to use the built-in oscillator to be used as PLL
input. This change sets sets both the fixed /2 prescaling divisor if HSI
is selected as input and prevents waiting for PLL2 to come online (which
only works for HSE input and will never happen with HSI input).
Signed-off-by: Daniel Egger<EMAIL_ADDRESS>

Codecov Report

Merging #9300 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #9300   +/-   ##
=======================================
  Coverage   52.46%   52.46%
=======================================
  Files         201      201
  Lines       25290    25290
  Branches     5286     5286
=======================================
  Hits        13268    13268
  Misses       9938     9938
  Partials     2084     2084

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a500e7e...844f8c7. Read the comment docs.

@therealprof , any update on this point?
I hope you didn't feel like I was trying to block you on this. I haven't tested my proposal of using LL_RCC_PLL_GetMainSource(HSI) to by pass PLL2 while loop (and I don' recall all details of the discussion), but how do you feel about this proposal?

@erwango I haven't looked into it any further due to a lot of other things on my plate at the moment. If I recall correctly your proposal will not work but I'd have to check again to say for sure.

ST internal bug tracker: 51414

@therealprof, a fix for this issue will be available in F1 cube packaged release V1.8.0 (Current 1.6.1).
This will be fixed as follows:
#if defined(RCC_PLL2_SUPPORT)
    if(LL_RCC_PLL_GetMainSource() != LL_RCC_PLLSOURCE_HSI_DIV_2)
    {
      /* Enable PLL2 */
      LL_RCC_PLL2_Enable();
      while (LL_RCC_PLL2_IsReady() != 1U)
      {
        /* Wait for PLL2 ready */
      }
    }
#endif /* RCC_PLL2_SUPPORT */

@therealprof, closing the ticket.
Sum up: this has been fixed in STM32 HAL and will be integrated in zephyr on the package is available

Perfect!


statfs() added

statfs(self, path) added to class Dropbox.
The command df now retrieves usable information.

Pulled within the last development


-P and -W options don't work because of clearok() call with uninitialised curses

At least on NetBSD, using the -P or -W command line options cause a seg fault because the clearok() call in main() happens with an uninitialised curses.  Adding a "if (usecurses)" check for clearok() gets things working.
New to github, so instead of a pull request here's a simple patch.  At least it's short :)
diff --git a/sc.c b/sc.c
index d2a4e1b..e4bacd2 100644
--- a/sc.c
+++ b/sc.c
@@ -395,7 +395,8 @@ main (int argc, char  **argv)
        exit (0);
     }

-    clearok(stdscr, TRUE);
+    if (usecurses)
+       clearok(stdscr, TRUE);
     EvalAll();

     if (mopt)

Thank you for fixing it.
A pull request is less work in total. Just clone the repo to your account at GitHub, then clone to your computer, do the fix, push to GitHub and do the pull request. I could patch it as well but then it's commited as @n-t-roff and not as @snarkophilus, which might not be what you want.

Cool, let me try that out.  Hopefully you'll receive a pull request soon!

I hope the two pull requests for both issues I created today worked.

Something did went wrong. I reverted it. I'll try to apply the second pull request within 24 h, maybe you need to reopen the first pull request again. I'll inform you.

GitHub did use my ID for the commit which never did happen for any pull request merge before. Did you provide an email address for your GitHub account? If not this could be the problem. (https://help.github.com/en/articles/why-are-my-commits-linked-to-the-wrong-user)

Looks like my email addr was set to private.  Not sure if that was the problem?  Changed that to public now.

Did you want me to re-submit the pull request for this one?

It seems that you need to re-submit both requests. Lets try it with the first one at first.

I think I'm making a nice mess of my fork :).  Will give up for the night and try again tomorrow.

Had to try one more time.  Hopefully this one works...

It does somehow seem to have worked, although you're not on the contributors list(???). Maybe this is a bug caused by the new Microsoft GitHub ownership....
So lets simply also apply the second pull request (I had deleted it, so please also re-submit it too.).

Not too fussed about the contributor bit yet.  Thanks for your help along the way.  git is very different to cvs that I'm used to!
I've hopefully submitted a working pull request for the other change.


MT baseline training error

I'm trying to run bash train.sh facebook/mbart-large-cc25 mbart MT_EN_HI but I'm getting the following error:
Fine-tuning facebook/mbart-large-cc25 on MT_EN_HI
03/22/2021 19:49:10 - WARNING - __main__ -   Process rank: -1, device: cuda:0, n_gpu: 1distributed training: False, 16-bits training: True
03/22/2021 19:49:10 - INFO - __main__ -   Training/evaluation parameters Seq2SeqTrainingArguments(output_dir='/tmp/mt_model', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=False, evaluation_strategy=<IntervalStrategy.STEPS: 'steps'>, prediction_loss_only=False, per_device_train_batch_size=4, per_device_eval_batch_size=2, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=5.0, max_steps=-1, lr_scheduler_type=<SchedulerType.LINEAR: 'linear'>, warmup_ratio=0.0, warmup_steps=0, logging_dir='runs/Mar22_19-49-10_iiitb-ThinkStation-P920', logging_strategy=<IntervalStrategy.STEPS: 'steps'>, logging_first_step=False, logging_steps=1500, save_strategy=<IntervalStrategy.STEPS: 'steps'>, save_steps=6000, save_total_limit=None, no_cuda=False, seed=42, fp16=True, fp16_opt_level='O1', fp16_backend='auto', fp16_full_eval=False, local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=1500, dataloader_num_workers=0, past_index=-1, run_name='/tmp/mt_model', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None, ignore_data_skip=False, sharded_ddp=[], deepspeed=None, label_smoothing_factor=0.0, adafactor=False, group_by_length=False, report_to=['tensorboard'], ddp_find_unused_parameters=None, dataloader_pin_memory=True, skip_memory_metrics=False, sortish_sampler=False, predict_with_generate=True)
03/22/2021 19:49:11 - WARNING - datasets.builder -   Using custom data configuration default-48b337e1e0f30e1a
03/22/2021 19:49:11 - WARNING - datasets.builder -   Reusing dataset csv (/home/vibhav/.cache/huggingface/datasets/csv/default-48b337e1e0f30e1a/0.0.0/2dc6629a9ff6b5697d82c25b73731dd440507a69cbce8b425db50b751e8fcfd0)
loading configuration file https://huggingface.co/facebook/mbart-large-cc25/resolve/main/config.json from cache at /home/vibhav/.cache/huggingface/transformers/36135304685d914515720daa48fc1adae57803e32ab82d5bde85ef78479e9765.b548f7e307531070391a881374674824b374f829e5d8f68857012de63fe2681a
Model config MBartConfig {
  "_num_labels": 3,
  "activation_dropout": 0.0,
  "activation_function": "gelu",
  "add_bias_logits": false,
  "add_final_layer_norm": true,
  "architectures": [
    "MBartForConditionalGeneration"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 0,
  "classif_dropout": 0.0,
  "classifier_dropout": 0.0,
  "d_model": 1024,
  "decoder_attention_heads": 16,
  "decoder_ffn_dim": 4096,
  "decoder_layerdrop": 0.0,
  "decoder_layers": 12,
  "dropout": 0.1,
  "encoder_attention_heads": 16,
  "encoder_ffn_dim": 4096,
  "encoder_layerdrop": 0.0,
  "encoder_layers": 12,
  "eos_token_id": 2,
  "forced_eos_token_id": 2,
  "gradient_checkpointing": false,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1",
    "2": "LABEL_2"
  },
  "init_std": 0.02,
  "is_encoder_decoder": true,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1,
    "LABEL_2": 2
  },
  "max_length": 1024,
  "max_position_embeddings": 1024,
  "model_type": "mbart",
  "normalize_before": true,
  "normalize_embedding": true,
  "num_beams": 5,
  "num_hidden_layers": 12,
  "output_past": true,
  "pad_token_id": 1,
  "scale_embedding": true,
  "static_position_embeddings": false,
  "task_specific_params": {
    "translation_en_to_ro": {
      "decoder_start_token_id": 250020
    }
  },
  "transformers_version": "4.4.2",
  "use_cache": true,
  "vocab_size": 250027
}

loading configuration file https://huggingface.co/facebook/mbart-large-cc25/resolve/main/config.json from cache at /home/vibhav/.cache/huggingface/transformers/36135304685d914515720daa48fc1adae57803e32ab82d5bde85ef78479e9765.b548f7e307531070391a881374674824b374f829e5d8f68857012de63fe2681a
Model config MBartConfig {
  "_num_labels": 3,
  "activation_dropout": 0.0,
  "activation_function": "gelu",
  "add_bias_logits": false,
  "add_final_layer_norm": true,
  "architectures": [
    "MBartForConditionalGeneration"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 0,
  "classif_dropout": 0.0,
  "classifier_dropout": 0.0,
  "d_model": 1024,
  "decoder_attention_heads": 16,
  "decoder_ffn_dim": 4096,
  "decoder_layerdrop": 0.0,
  "decoder_layers": 12,
  "dropout": 0.1,
  "encoder_attention_heads": 16,
  "encoder_ffn_dim": 4096,
  "encoder_layerdrop": 0.0,
  "encoder_layers": 12,
  "eos_token_id": 2,
  "forced_eos_token_id": 2,
  "gradient_checkpointing": false,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1",
    "2": "LABEL_2"
  },
  "init_std": 0.02,
  "is_encoder_decoder": true,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1,
    "LABEL_2": 2
  },
  "max_length": 1024,
  "max_position_embeddings": 1024,
  "model_type": "mbart",
  "normalize_before": true,
  "normalize_embedding": true,
  "num_beams": 5,
  "num_hidden_layers": 12,
  "output_past": true,
  "pad_token_id": 1,
  "scale_embedding": true,
  "static_position_embeddings": false,
  "task_specific_params": {
    "translation_en_to_ro": {
      "decoder_start_token_id": 250020
    }
  },
  "transformers_version": "4.4.2",
  "use_cache": true,
  "vocab_size": 250027
}

loading file https://huggingface.co/facebook/mbart-large-cc25/resolve/main/sentencepiece.bpe.model from cache at /home/vibhav/.cache/huggingface/transformers/83d419fb34e90155a8d95f7799f7a7316a327dc28c7ee6bee15b5a62d3c5ca6b.00628a9eeb8baf4080d44a0abe9fe8057893de20c7cb6e6423cddbf452f7d4d8
loading file https://huggingface.co/facebook/mbart-large-cc25/resolve/main/added_tokens.json from cache at None
loading file https://huggingface.co/facebook/mbart-large-cc25/resolve/main/special_tokens_map.json from cache at None
loading file https://huggingface.co/facebook/mbart-large-cc25/resolve/main/tokenizer_config.json from cache at None
loading file https://huggingface.co/facebook/mbart-large-cc25/resolve/main/tokenizer.json from cache at /home/vibhav/.cache/huggingface/transformers/16e85cac0e7a8c2938ac468199d0adff7483341305c7e848063b72dcf5f22538.39607a8bede9bcd2666ea442230a9d382f57e4fea127c9cc5b6fc6caf527d682
loading weights file https://huggingface.co/facebook/mbart-large-cc25/resolve/main/pytorch_model.bin from cache at /home/vibhav/.cache/huggingface/transformers/58963b41815ac5618d9910411e018d60a3ae7d4540a66e6cf70adf29a748ca1b.bef0d2e3352d6c4bf1213c6207738ec5ecf458de355c65b2aead6671bc612138
All model checkpoint weights were used when initializing MBartForConditionalGeneration.

All the weights of MBartForConditionalGeneration were initialized from the model checkpoint at facebook/mbart-large-cc25.
If your task is similar to the task the model of the checkpoint was trained on, you can already use MBartForConditionalGeneration for predictions without further training.
03/22/2021 19:50:10 - WARNING - datasets.arrow_dataset -   Loading cached processed dataset at /home/vibhav/.cache/huggingface/datasets/csv/default-48b337e1e0f30e1a/0.0.0/2dc6629a9ff6b5697d82c25b73731dd440507a69cbce8b425db50b751e8fcfd0/cache-a5481e6d57bfbce0.arrow
03/22/2021 19:50:10 - WARNING - datasets.arrow_dataset -   Loading cached processed dataset at /home/vibhav/.cache/huggingface/datasets/csv/default-48b337e1e0f30e1a/0.0.0/2dc6629a9ff6b5697d82c25b73731dd440507a69cbce8b425db50b751e8fcfd0/cache-41fcbb9a83397c4b.arrow
Using amp fp16 backend
***** Running training *****
  Num examples = 8060
  Num Epochs = 5
  Instantaneous batch size per device = 4
  Total train batch size (w. parallel, distributed & accumulation) = 4
  Gradient Accumulation steps = 1
  Total optimization steps = 10075
  0%|                                                                                | 0/10075 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/home/hdd1/vibhav/Thesis/GLUECoS/Code/run_seq2seq.py", line 584, in <module>
    main()
  File "/home/hdd1/vibhav/Thesis/GLUECoS/Code/run_seq2seq.py", line 529, in main
    train_result = trainer.train()
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/transformers/trainer.py", line 1053, in train
    tr_loss += self.training_step(model, inputs)
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/transformers/trainer.py", line 1441, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/transformers/trainer.py", line 1475, in compute_loss
    outputs = model(**inputs)
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/transformers/models/mbart/modeling_mbart.py", line 1303, in forward
    return_dict=return_dict,
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/transformers/models/mbart/modeling_mbart.py", line 1166, in forward
    return_dict=return_dict,
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/transformers/models/mbart/modeling_mbart.py", line 803, in forward
    output_attentions=output_attentions,
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/transformers/models/mbart/modeling_mbart.py", line 317, in forward
    output_attentions=output_attentions,
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/transformers/models/mbart/modeling_mbart.py", line 181, in forward
    query_states = self.q_proj(hidden_states) * self.scaling
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 94, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/torch/nn/functional.py", line 1753, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasCreate(handle)`
  0%|                                                                                | 0/10075 [00:00<?, ?it/s]```

Btw, this is running perfectly on CPU, but on GPU, getting this error.

I fixed this by downgrading the pytorch to 1.6 but I'm not able to load it even with batch size 1 on my 2080 TI GPU. @Genius1237 ?

The first issue that you've encountered is likely due to some mismatch/error with cuda versions and the version of pytorch you've used. Please try to upgrade to the CUDA 11 and Pytorch 1.7+ and see if the error goes away. The code was tested on pytorch 1.7.
I believe that batch size of 4 worked on a 16 GB Tesla P100. Could you check if the fp16 flag is set? A batch size of 2 should definitely work on your 2080Ti. Try upgrading to CUDA 11 as well, as there are a lot of improvements that come in with that.

fp16 flag is set, my cuda version is 10.2 and this works fine for pytorch 1.7 as well (but not for 1.8). I'm just skeptical of upgrading the cuda version as it messes up sometimes. But is there anything else that I should check?

Could you please share the stdout/logs once? I really cannot think of anything else that could be the issue.
How did you install pytorch? If you know the exact command, please share that. If not, atleast the source from where it came from (anaconda/pypi repos/torch website repo).

PyTorch was installed as conda install pytorch==1.6.0 cudatoolkit=10.2 -c pytorch
Fine-tuning facebook/mbart-large-cc25 on MT_EN_HI
03/25/2021 12:58:33 - WARNING - __main__ -   Process rank: -1, device: cuda:0, n_gpu: 1distributed training: False, 16-bits training: True
03/25/2021 12:58:33 - INFO - __main__ -   Training/evaluation parameters Seq2SeqTrainingArguments(output_dir='/tmp/mt_model', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=False, evaluation_strategy=<IntervalStrategy.STEPS: 'steps'>, prediction_loss_only=False, per_device_train_batch_size=1, per_device_eval_batch_size=1, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=5.0, max_steps=-1, lr_scheduler_type=<SchedulerType.LINEAR: 'linear'>, warmup_ratio=0.0, warmup_steps=0, logging_dir='runs/Mar25_12-58-33_iiitb-ThinkStation-P920', logging_strategy=<IntervalStrategy.STEPS: 'steps'>, logging_first_step=False, logging_steps=1500, save_strategy=<IntervalStrategy.STEPS: 'steps'>, save_steps=6000, save_total_limit=None, no_cuda=False, seed=42, fp16=True, fp16_opt_level='O1', fp16_backend='auto', fp16_full_eval=False, local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=1500, dataloader_num_workers=0, past_index=-1, run_name='/tmp/mt_model', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None, ignore_data_skip=False, sharded_ddp=[], deepspeed=None, label_smoothing_factor=0.0, adafactor=False, group_by_length=False, report_to=['tensorboard'], ddp_find_unused_parameters=None, dataloader_pin_memory=True, skip_memory_metrics=False, sortish_sampler=False, predict_with_generate=True)
03/25/2021 12:58:34 - WARNING - datasets.builder -   Using custom data configuration default-48b337e1e0f30e1a
03/25/2021 12:58:34 - WARNING - datasets.builder -   Reusing dataset csv (/home/vibhav/.cache/huggingface/datasets/csv/default-48b337e1e0f30e1a/0.0.0/2dc6629a9ff6b5697d82c25b73731dd440507a69cbce8b425db50b751e8fcfd0)
loading configuration file https://huggingface.co/facebook/mbart-large-cc25/resolve/main/config.json from cache at /home/vibhav/.cache/huggingface/transformers/36135304685d914515720daa48fc1adae57803e32ab82d5bde85ef78479e9765.b548f7e307531070391a881374674824b374f829e5d8f68857012de63fe2681a
Model config MBartConfig {
  "_num_labels": 3,
  "activation_dropout": 0.0,
  "activation_function": "gelu",
  "add_bias_logits": false,
  "add_final_layer_norm": true,
  "architectures": [
    "MBartForConditionalGeneration"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 0,
  "classif_dropout": 0.0,
  "classifier_dropout": 0.0,
  "d_model": 1024,
  "decoder_attention_heads": 16,
  "decoder_ffn_dim": 4096,
  "decoder_layerdrop": 0.0,
  "decoder_layers": 12,
  "dropout": 0.1,
  "encoder_attention_heads": 16,
  "encoder_ffn_dim": 4096,
  "encoder_layerdrop": 0.0,
  "encoder_layers": 12,
  "eos_token_id": 2,
  "forced_eos_token_id": 2,
  "gradient_checkpointing": false,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1",
    "2": "LABEL_2"
  },
  "init_std": 0.02,
  "is_encoder_decoder": true,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1,
    "LABEL_2": 2
  },
  "max_length": 1024,
  "max_position_embeddings": 1024,
  "model_type": "mbart",
  "normalize_before": true,
  "normalize_embedding": true,
  "num_beams": 5,
  "num_hidden_layers": 12,
  "output_past": true,
  "pad_token_id": 1,
  "scale_embedding": true,
  "static_position_embeddings": false,
  "task_specific_params": {
    "translation_en_to_ro": {
      "decoder_start_token_id": 250020
    }
  },
  "transformers_version": "4.4.2",
  "use_cache": true,
  "vocab_size": 250027
}

loading configuration file https://huggingface.co/facebook/mbart-large-cc25/resolve/main/config.json from cache at /home/vibhav/.cache/huggingface/transformers/36135304685d914515720daa48fc1adae57803e32ab82d5bde85ef78479e9765.b548f7e307531070391a881374674824b374f829e5d8f68857012de63fe2681a
Model config MBartConfig {
  "_num_labels": 3,
  "activation_dropout": 0.0,
  "activation_function": "gelu",
  "add_bias_logits": false,
  "add_final_layer_norm": true,
  "architectures": [
    "MBartForConditionalGeneration"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 0,
  "classif_dropout": 0.0,
  "classifier_dropout": 0.0,
  "d_model": 1024,
  "decoder_attention_heads": 16,
  "decoder_ffn_dim": 4096,
  "decoder_layerdrop": 0.0,
  "decoder_layers": 12,
  "dropout": 0.1,
  "encoder_attention_heads": 16,
  "encoder_ffn_dim": 4096,
  "encoder_layerdrop": 0.0,
  "encoder_layers": 12,
  "eos_token_id": 2,
  "forced_eos_token_id": 2,
  "gradient_checkpointing": false,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1",
    "2": "LABEL_2"
  },
  "init_std": 0.02,
  "is_encoder_decoder": true,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1,
    "LABEL_2": 2
  },
  "max_length": 1024,
  "max_position_embeddings": 1024,
  "model_type": "mbart",
  "normalize_before": true,
  "normalize_embedding": true,
  "num_beams": 5,
  "num_hidden_layers": 12,
  "output_past": true,
  "pad_token_id": 1,
  "scale_embedding": true,
  "static_position_embeddings": false,
  "task_specific_params": {
    "translation_en_to_ro": {
      "decoder_start_token_id": 250020
    }
  },
  "transformers_version": "4.4.2",
  "use_cache": true,
  "vocab_size": 250027
}

loading file https://huggingface.co/facebook/mbart-large-cc25/resolve/main/sentencepiece.bpe.model from cache at /home/vibhav/.cache/huggingface/transformers/83d419fb34e90155a8d95f7799f7a7316a327dc28c7ee6bee15b5a62d3c5ca6b.00628a9eeb8baf4080d44a0abe9fe8057893de20c7cb6e6423cddbf452f7d4d8
loading file https://huggingface.co/facebook/mbart-large-cc25/resolve/main/added_tokens.json from cache at None
loading file https://huggingface.co/facebook/mbart-large-cc25/resolve/main/special_tokens_map.json from cache at None
loading file https://huggingface.co/facebook/mbart-large-cc25/resolve/main/tokenizer_config.json from cache at None
loading file https://huggingface.co/facebook/mbart-large-cc25/resolve/main/tokenizer.json from cache at /home/vibhav/.cache/huggingface/transformers/16e85cac0e7a8c2938ac468199d0adff7483341305c7e848063b72dcf5f22538.39607a8bede9bcd2666ea442230a9d382f57e4fea127c9cc5b6fc6caf527d682
loading weights file https://huggingface.co/facebook/mbart-large-cc25/resolve/main/pytorch_model.bin from cache at /home/vibhav/.cache/huggingface/transformers/58963b41815ac5618d9910411e018d60a3ae7d4540a66e6cf70adf29a748ca1b.bef0d2e3352d6c4bf1213c6207738ec5ecf458de355c65b2aead6671bc612138
All model checkpoint weights were used when initializing MBartForConditionalGeneration.

All the weights of MBartForConditionalGeneration were initialized from the model checkpoint at facebook/mbart-large-cc25.
If your task is similar to the task the model of the checkpoint was trained on, you can already use MBartForConditionalGeneration for predictions without further training.
03/25/2021 12:59:25 - WARNING - datasets.arrow_dataset -   Loading cached processed dataset at /home/vibhav/.cache/huggingface/datasets/csv/default-48b337e1e0f30e1a/0.0.0/2dc6629a9ff6b5697d82c25b73731dd440507a69cbce8b425db50b751e8fcfd0/cache-a5481e6d57bfbce0.arrow
03/25/2021 12:59:25 - WARNING - datasets.arrow_dataset -   Loading cached processed dataset at /home/vibhav/.cache/huggingface/datasets/csv/default-48b337e1e0f30e1a/0.0.0/2dc6629a9ff6b5697d82c25b73731dd440507a69cbce8b425db50b751e8fcfd0/cache-41fcbb9a83397c4b.arrow
Using amp fp16 backend
***** Running training *****
  Num examples = 8060
  Num Epochs = 5
  Instantaneous batch size per device = 1
  Total train batch size (w. parallel, distributed & accumulation) = 1
  Gradient Accumulation steps = 1
  Total optimization steps = 40300
  0%|                                                                                | 0/40300 [00:00<?, ?it/s]/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:123: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
  0%|                                                                                | 6/40300 [00:01<2:40:49,  4.18it/s]Traceback (most recent call last):
  File "/home/hdd1/vibhav/Thesis/GLUECoS/Code/run_seq2seq.py", line 584, in <module>
    main()
  File "/home/hdd1/vibhav/Thesis/GLUECoS/Code/run_seq2seq.py", line 529, in main
    train_result = trainer.train()
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/transformers/trainer.py", line 1053, in train
    tr_loss += self.training_step(model, inputs)
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/transformers/trainer.py", line 1441, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/transformers/trainer.py", line 1475, in compute_loss
    outputs = model(**inputs)
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/transformers/models/mbart/modeling_mbart.py", line 1303, in forward
    return_dict=return_dict,
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/transformers/models/mbart/modeling_mbart.py", line 1189, in forward
    return_dict=return_dict,
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/transformers/models/mbart/modeling_mbart.py", line 1057, in forward
    use_cache=use_cache,
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/transformers/models/mbart/modeling_mbart.py", line 412, in forward
    output_attentions=output_attentions,
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/hdd1/vibhav/anaconda3/envs/gluecos/lib/python3.7/site-packages/transformers/models/mbart/modeling_mbart.py", line 268, in forward
    .reshape(bsz, tgt_len, embed_dim)
RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 10.76 GiB total capacity; 9.86 GiB already allocated; 3.94 MiB free; 9.88 GiB reserved in total by PyTorch)
  0%|                                                                                | 6/40300 [00:01<2:45:28,  4.06it/s]

I've never used the cuda/cudnn installs that come in conda. I hope that they aren't old and are up to date. Please check that. Also, given that you're installing them inside a conda env, there should be no harm in trying out a newer version of cuda in a separate environment.
At this point, I really do not have any suggestion than updating the cuda and torch to the latest versions and trying a smaller batch size. Let me know how this goes. If things still do not work out, I can look into reducing the memory footprint in some way.

Are you using the 2080Ti for video output as well? It says that only 10.7 gigs is the total capacity. Xorg and the desktop environment take up some vram in that case. You can claw back a bit more VRAM if you are able to run the system headless.

True, that system has video outputs as well. I'll have a look at your suggestions and let you know.

Hey @Genius1237 just wanted to update you that cuda update is having the same out of memory error. Also seems like the seq2seq script is not working just like that, I had to change the Autotokenizer to MBARTtokenizer in order to run it. Batch size of 1 is also not being loaded on 2080 TI (12gb). :/

I do not have a lot of free time at hand now, so I am not able to look into this currently. I know that atleast one other person has been able to get this running, so I would suggest that you try looking into this yourself or try running it on a different system.

As for the issue with AutoTokenizer/MBARTTokenizer, please check if the model name (facebook/mbart-large-cc25) is being passed correctly. It uses this to pick the correct tokenizer. That or it could be an issue with the version of the transformers library.

I understand that @Genius1237. I was able to run it only on a 16GB VRAM GPU, and it filled up 15GB space with just the batch size of 4.


Add snapshot generation wait metric

What is changed and how it works?

Issue Number: Close #14940
What's Changed:

Add snapshot generation wait duration metric

Related changes

PR to update pingcap/docs/pingcap/docs-cn: n/a
Need to cherry-pick to the release branch n/a

Check List
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression

Consumes more CPU
Consumes more MEM

Breaking backward compatibility

Release note
None

/cc @Connor1996

Please update grafana json as well

Done, I imported the latest tikv_details.json into the grafana (v 7.5.10) dashboard in tiup playground. And added a panel for the new metric, then export the dashboard to a json.
I took a glance at the generated json, there are several types of changes:

Deprecated field, e.g. hash_key.
Fields reorder, this contributes to the most of the changes.
Version, from 7.5.11 to 7.5.10, since the latter one is used in 7.1.0 TiDB release
Relevant changes, "snapshot generation wait duration"

/test


fixed the pitty diff in perplexity result

Very small diff, smaller than what is noticable by git, but noticable on our dev envs, when running test_metrics

Not sure why the test results on git are different from those in our dev env.


Remove default value from all redirectUri parameters

We currently use urn:ietf:wg:oauth:2.0:oob as default parameter for a redirect URI both when first registering an app with an instance (AppMethods.createApp()) and when performing OAuth (multiple methods in OAuthMethods).
This URI is a special case in that it does not actually redirect anywhere (most likely to a URL that will be intercepted by the app starting the OAuth flow), but will just display a valid code in a browser window. This might be useful in some development contexts, but makes no sense for production code.
I already argued in #68 that we shouldn't use the URI in any sample code. I now think we should not use this URI as default parameter in any of the methods mentioned above, either, but instead make the library user explicitly define a redirect URI in every case. Keeping it as a default parameter might lead to hard-to-debug situations where an actually sensible parameter is omitted and authentication fails as a result of this.
This would not remove the possibility of using this URI, its use would just need to be made explicit - we could even define it as a constant for easy access, if considered useful.

@bocops Lets change this.


Software becomes unusable after setting change.

Describe the bug
The software becomes unusable and crashes after "show in tray" option is enabled and requires re-install. It becomes impossible to change other settings after this because when the "Apply" button is clicked on the configurator it crashes both apps.
To Reproduce
Steps to reproduce the behavior:

Go to 'Extra' in the configurator
Click on 'Show tray icon’
Software breaks

Expected behavior
The software icon shows in the Windows tray.
Desktop (please complete the following information):

OS: Windows 10 Home
Version 20H2
Website Build
No other taskbar mods

To be able to focus on major/current bugs I had to close all issues. The issues section has gotten very messy to a point that I couldn't follow it anymore. Most issues are duplicates, very old or are already fixed/solved. Feel free to reopen this issue if you still have this issue on the latest version of TaskbarX to get my attention. All feature requests have been put in a list here: 💡| Feature requests #870
I hope you understand ❤


Remove version: CeVIO.CeVIOAI version 9.1.5

[Automated] It returns code over 400 in all urls
Microsoft Reviewers: Open in CodeFlow

URL: https://cevio.jp/storage/cevio_ai/CeVIO_AI_Setup_x64_(<IP_ADDRESS>).msi
Response Error: The remote server returned an error: (404) Not Found.
(Automated message - build 911)


proposal: make istio more friendly with Percentage Blue Green deployments

Describe the feature request
At Mercadolibre we use Blue Green deployments with percentage traffic shifting to newer versions. This means we have the two versions up and running and gradually shift traffic to the newer version until it reaches 100% of the traffic, and only then we terminate the older version. We can expand more on why we do this if desired.
We also have use cases where a single service can be exposed in multiple VirtualServices and some cases where this could even scale to hundreds. We can also expand more on this.
Given this use case, currently in Istio, we would need to update all those VirtualServices specifying the traffic percentages to the given subsets, and do this for every step of the deployment where we progress the traffic shift. Doing this is cumbersome and requires lots of API calls to do.
Traffic Shifting between versions of a service is conceptually part of routing and makes sense to have it in VirtualService, but we think it generally makes sense to handle traffic shifting between versions of a service as a whole. If a services is exposed in even two VirtualServices it is kind of weird to be updating both at a time or have VirtualServices routing different weights to different versions.
So, we think, Istio should provide a more straight forward approach to shift traffic between multiple versions (subsets) of the same service.
Describe alternatives you've considered
We've thought a while about this and considered this could be probably implemented within DestinationRule, this is were subsets are specified, and subsets could be accompanied with their weight or percentage of traffic.
This would work by having VirtualServices targeting the entire service (not specifying individual subsets), and then having Istio notice this and generating a weighted clusters configuration to the subsets with the configurations specified in the DestinationRule.
If the DestinationRule does not contain traffic percentages or the VirtualService is targeting subsets, then current behaviours can be maintained, and so, backwards compatibility.
This approach seems pretty straight forward, but is probably using DestinationRule for things that it's not meant to be doing, as per the documentation, DestinationRule applies after traffic routing has occurred. But it might be tolerable, since DestinationRule also defines the targets (subsets) of routes.
Affected product area (please put an X in all that apply)
[ ] Docs
[ ] Installation
[x] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Affected features (please put an X in all that apply)
[ ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane
Additional context

Have you evaluated rollout tools that integrate with Istio to do much of what you described like Argo Rollouts for example

I was thinking of another Idea around this Issue using Envoy's Runtime Layers.
Envoy has the concept of Runtime Layers which can provide runtime values for thinks like traffic percentages, and this are provided through a separate XDS API.
We could expose in Istio a way to generate runtime values for traffic percentages, and be able to reference those values from a VirtualService. With this approach we would be able to reference the same Runtime Values from multiple Virtual Services and update the values of traffic routing without ever touching Virtual Services.
This seems like a good way to separate responsabilities, but it would involve adding a new Istio API for generating Runtime Values and modifying VirtualService to be able to reference Runtime Values. This could eventually prove useful for other features as well.
@howardjohn

Ok. Thanks for the details. Now I understand this better. So every time you want traffic shifting to happen in your proposal you update the Destination Rule with changed weights (if provided at subsets) and you are expecting Istio to update the corresponding Virtual Service or change routes with weighted clusters?

Yes, precisely

Not sure subsets in DR to add traffic shifting properties is the right approach? But I do not have any good alternative other than doing via updating VS with right weights?

Yes, I know it's probably not the right approach, but it seems the more straight forward. The other approach I mentioned about using Runtime Features could prove useful too and not messing with DR.

Is your concern Istio route update scalability? I think we did similar large route updates for knative scenarios - @howardjohn has idea about knative scale.

We have 2 possible concerns:

Our first and top concern is API Server scalability, since each deployment would trigger updates through lots of VirtualServices simultaneously, and several times in a row each time we want to update weights.
Secondly, Route update scalability could be a concern, but it wouldn't be resolved with the DR approach I mentioned first. If we were to use Runtime Features, then this concern could be mitigated, because a change in weights would not trigger a change in Routes.

Apart from those concerns, it seems Istio is missing some concepts for handling traffic to a destination as a whole and also this types of Blue Green deployments.
Specifying weights in VirtualServices is good and provides lots of functionality but requires "manual" updates. When deploying a service progressively changing weights, you would want all routes pointing to that service to reflect the same weights at the same time and manage the service as a whole, no matter the amount of VirtualServices it is routed from.
If you take the example of Argo, it needs to be specified which VirtualService to update when performing a rollout, it doesn't even seem to provide support for updating multiple VirtualServices, which seems kind of restrictive. If we were to need to update multiple VS, then we would need to perform multiple rollouts or again, manually change them.
Instead, with our proposed approach, Argo could be able to just receive the DR (if that were the case) and don't even care what VS the services is routed from.

FWIW I think the disconnect is the vast majority of users have 1 or 2
(ingress + mesh) VS per service which is why this hasn't come up before.
On Sat, Oct 30, 2021, 10:06 AM sschepens @.***> wrote:

Ok. Thanks for the details. Now I understand this better. So every time
you want traffic shifting to happen in your proposal you update the
Destination Rule with changed weights (if provided at subsets) and you are
expecting Istio to update the corresponding Virtual Service or change
routes with weighted clusters?
Yes, precisely
Not sure subsets in DR to add traffic shifting properties is the right
approach? But I do not have any good alternative other than doing via
updating VS with right weights?
Yes, I know it's probably not the right approach, but it seems the more
straight forward. The other approach I mentioned about using Runtime
Features could prove useful too and not messing with DR.
Is your concern Istio route update scalability? I think we did similar
large route updates for knative scenarios - @howardjohn
https://github.com/howardjohn has idea about knative scale.
We have 2 possible concerns:

Our first and top concern is API Server scalability, since each
deployment would trigger updates through lots of VirtualServices
simultaneously, and several times in a row each time we want to update
weights.
Secondly, Route update scalability could be a concern, but it
wouldn't be resolved with the DR approach I mentioned first. If we were to
use Runtime Features, then this concern could be mitigated, because a
change in weights would not trigger a change in Routes.

Apart from those concerns, it seems Istio is missing some concepts for
handling traffic to a destination as a whole and also this types of Blue
Green deployments.
Specifying weights in VirtualServices is good and provides lots of
functionality but requires "manual" updates. When deploying a service
progressively changing weights, you would want all routes pointing to that
service to reflect the same weights at the same time and manage the service
as a whole, no matter the amount of VirtualServices it is routed from.
If you take the example of Argo, it needs to be specified which
VirtualService to update when performing a rollout, it doesn't even seem to
provide support for updating multiple VirtualServices, which seems kind of
restrictive. If we were to need to update multiple VS, then we would need
to perform multiple rollouts or again, manually change them.
Instead, with our proposed approach, Argo could be able to just receive
the DR (if that were the case) and don't even care what VS the services is
routed from.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/35611#issuecomment-955493724, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEYGXKXCGEJAOOLHFNJ2H3UJQQ23ANCNFSM5F73PQWA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Secondly, Route update scalability could be a concern, but it wouldn't be resolved with the DR approach I mentioned first. If we were to use Runtime Features, then this concern could be mitigated, because a change in weights would not trigger a change in Routes.

Does Envoy have run time values for traffic weights already?  or are you proposing we add that support in routing components of Envoy?

nvm, looks like Envoy has that support already https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/traffic_splitting#traffic-splitting-across-multiple-upstreams

Yes, Envoy already supports that, we're just setting a fixed value for the runtime feature probably.
We would probably need to add a Istio Resource for defining Runtime Values and be able to reference them from Route weights as well as a default value as we do today.

@ramaraochavali @howardjohn what do you think of these proposals, could this be viable? should other people be involved in this discussion?

@howardjohn @ramaraochavali We've found that it's possible to delegate routes to another VirtualService, that basically allows what we are proposing here.
We can create a VirtualService which only defines routes with weights for the service, and then just make every other VirtualServices delegate to this one.

@sschepens Cool. Thank you for the update. Would it be possible to share your VS and Delegate example so that any one coming to look at this can use it?


<EMAIL_ADDRESS>on npm missing dist

The github release tarball has it however.
1.2.3 release is fine.
$ npm install<EMAIL_ADDRESS>...
└──<EMAIL_ADDRESS>
$ ls node_modules/react-responsive
CHANGELOG.md  LICENSE  README.md  package.json

$ npm install<EMAIL_ADDRESS>...
└──<EMAIL_ADDRESS>
$ ls node_modules/react-responsive
CHANGELOG.md  LICENSE  README.md  book.json  dist  package.json  samples  src  test  webpack.config.js  webpack.config.samples.js

Publishing a fix new, should see it on npm update and 1.2.4 will be deprecated. Something went haywire on a publish.

1.2.5 should have it


[JS] Topic: External Libraries: Material

Description 📜
Added the documentation of Material library with some examples.
Fixes #7540

Type of change 📝

[x] Documentation (Content Creation in the form of codes or tutorials)

Domain of Contribution 📊

[x] JavaScript_Domain

Checklist ✅

[x] I follow Contributing Guidelines & Code of conduct of this project.
[x] I have performed a self-review of my own code or work.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] My changes generates no new warnings.
[x] I'm GWOC'21 contributor

@paramsiddharth @amino19 @dakshsethi @s4-2 @Creatoon Kindly review this PR.


[engineering blog] devx / devx action / devx codespace

We should write a engineering.iog.io article about the devx repo that covers:

motivation (why do we care at all?)
basic haskell.nix understanding
usage as GitHub action (example)
nix usage of devx (example)
usage of the devx CodeSpaces (example)

https://github.com/input-output-hk/engineering/pull/65


I like to donwload to the best resolution, i can?

Sorry my question is;
can i download the video on the best resolution, if i go to the video url the best resolution is 1280 x 720 but when i donwload the video is worst resolution... i use windows or linux
destreamer-v3.exe -i "https://web.microsoftstream.com/video/35111ad6-241f-44b1-b6ce-09b4e05f26a1"

Not sure about that branch, try running destreamer without arguments see if there's a parameter for resolution.
The master branch (destreamer v2) automatically picks the best available resolution to download.

I you are using this branch it should by default, if it's not you can manually select the resolution by adding -q 0 to your command and selecting the desired quality when prompted


Error on symfony 2.8.* upgrade

Hi! I just try to upgrade my project with symfony 2.8.*, here is my composer.json piece:
"require": {
    "symfony/symfony": "2.8.*",
    [...]
    "lexik/jwt-authentication-bundle": "~1.3.1",
}

And the error of composer update command :
lexik/jwt-authentication-bundle v1.3.1 requires symfony/symfony ~2.3 -> no matching package found.

May the problem comes from a mistake on your composer.json?
Thanks

Hi,
I just checked on a fresh Symfony 2.8 installation and there is no problem installing the bundle. The last tag of the 1.x branch is 1.3.1 and is compatible Symfony ~2.3. You could try to replace the constraint in your composer.json by 1.3.* instead of ~1.3.1.
Let me know if you solved this.

It works! My bad ...
Thanks for your help


Immunization.statusReason not being converted from RXA.18

Immunization.statusReason is not being converted from RXA.18 for vxu-v04-sample04.json (when .status="not done"?)
Segment is:
RXA|0|1|20170513|20170513|62^HPV quadrivalent^CVX|999|||||^^^MIICSHORTCODE|||||||00^Parental Refusal^NIP002||RE

Decision was to NOT do a mapping because the mapping is imperfect and we lose data.  For now we will accept the way the converter creates the CWE as the ground truth.  We will abandon this PR.
Perhaps in the future we will do two reasons, one that preserves the content and one that maps to the best mapping we can for the new codes.
IF we do this, we cannot this PR 341 as is.
The way to do it in the future is more like:
Do a mapping something like this, or maybe two mappings, one specifically for NIP002
V3ActReason:
  IMMUNE: IMMUNE
  MEDPREC: MEDPREC
  OSTOCK: OSTOCK
  PATOBJ: PATOBJ
  00: PATOBJ
  01: RELIG
  03: PATOBJ

This code in SimpleDataValueResolver.java needs to check for NIP002 first and then map from 00, 01, 03.  All other codings should pass through CWE processing.
    public static final ValueExtractor<Object, SimpleCode> IMMUNIZATION_STATUS_REASON = (Object value) -> {
       // System.out.println("test");
        if (value instanceof CWE) {
            CWE cwe = (CWE) value;
            String actReason = cwe.getCwe1_Identifier().toString();
            String code = getFHIRCode(actReason, V3ActReason.class);
            String system = cwe.getCwe3_NameOfCodingSystem().toString();
            String display = cwe.getCwe2_Text().toString();

            if ( actReason.equals("IMMUNE") || actReason.equals("MEDPREC") || actReason.equals("OSTOCK") ||  actReason.equals("PATOBJ")) {
                V3ActReason ar = V3ActReason.fromCode(code);
                System.out.println(code);
                return new SimpleCode(code,ar.getSystem(),ar.getDisplay());
            }
        }
         return null;
    };

This change to CodeableConcept.yml should be removed.   We don't want to have a specialized CodeableConcept.  Instead we want the IMMUNIZATION_STATUS_REASON to be processed on the statusReason and give us a mapped value or a pass-through value.
coding_6:
   type: IMMUNIZATION_STATUS_REASON
   condition: $customCoding NOT_NULL
   valueOf: $customCoding
   specs: $customCoding
   vars:
      customCoding: CWE

In Immunization.yml we want something like this which preprocesses RXA.18 to get the right coding.
statusReason_2:
condition: $coding NOT_NULL
valueOf: datatype/CodeableConcept
expressionType: resource
specs: RXA.18
vars:
coding: IMMUNIZATION_STATUS_REASON, RXA.18


Side bar buttons missing in full screen

Type: Bug
I have not tested this on other platforms, but on MacOS, when a window is made fullscreen, the buttons to manipulate the sidebars are no longer visible. Normally, on MacOS these are drawn on the title bar. Once in fullscreen, however, they are no longer visible.
Screenshots rather than steps will be easier to demonstrate.
VS Code version: Code 1.74.2 (e8a3071ea4344d9d48ef8a4df2c097372b0c5161, 2022-12-20T10:26:09.430Z)
OS version: Darwin x64 22.2.0
Modes:
Sandboxed: No

With buttons (Expected):

And fullscreen:

And fullscreen with mouse hovering to show title bar:

This appears to be fixed as of 1.82.0. Closing.


Replace binary log in segmentation_optimize with a polynomial approximation in Q11

Adapted from oc_blog32_q10() in libtheora with blessing from @tterribe.
AWCY results on ojective-1-fast at default speed:

PSNR Y
PSNR Cb
PSNR Cr
CIEDE2000
SSIM
MS-SSIM
PSNR-HVS Y
PSNR-HVS Cb
PSNR-HVS Cr
PSNR-HVS
VMAF
VMAF-NEG

0.4305
0.3834
-0.2535
0.1353
-0.2666
-0.2792
0.3404
0.4239
-0.1232
0.4060
0.1488
0.1788

Sure, I tried to translate as faithfully to the original code as possible while meeting language, formatting and linting constraints.


feat(module:date-picker): add nzShowWeekNumber property

PR Checklist
Please check if your PR fulfills the following requirements:

[x] The commit message follows our guidelines: https://github.com/NG-ZORRO/ng-zorro-antd/blob/master/CONTRIBUTING.md#commit
[x] Tests for the changes have been added (for bug fixes / features)
[x] Docs have been added / updated (for bug fixes / features)

I have not added docs in Chinese, and I'm unsure if I'm expected to do so?
PR Type
What kind of change does this PR introduce?
[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Documentation content changes
[ ] Application (the showcase website) / infrastructure changes
[ ] Other... Please describe:

What is the current behavior?
Week numbers are only shown in the week picker component.
Issue Number: Fixes #3764
What is the new behavior?
[nzShowWeekNumber] can control whether week numbers are shown in the date picker. Week numbers are still always shown in the week picker.
Does this PR introduce a breaking change?
[ ] Yes
[x] No

@wenqi73 I know you guys have a huge stack of open PRs, but do you have any sort of estimate for when this PR might be reviewed?

@UncleDave CI failed, you should make it run successfully first.

@wenqi73 I'm not familiar with your CI pipeline, but I don't see how my changes could have caused this failure, nor can I reproduce it locally, do you have any clues about it?
✖ Compiling with Angular sources in Ivy partial compilation mode.
components/core/testing/component-bed.ts:38:5 - error TS2322: Type 'TestBed' is not assignable to type 'TestBedStatic'.
  Type 'TestBed' provides no match for the signature 'new (...args: any[]): TestBed'.

38     bed,
       ~~~

  components/core/testing/component-bed.ts:13:3
    13   bed: TestBedStatic;
         ~~~
    The expected type comes from property 'bed' which is declared here on type 'ComponentBed<T>'

[07:33:54] 'library:build-zorro' errored after 44 s
[07:33:54] Error: Process failed with code 1
    at ChildProcess.<anonymous> (/home/vsts/work/1/s/scripts/gulp/util/task-helpers.ts:34:25)
    at ChildProcess.emit (node:events:513:28)
    at ChildProcess.emit (node:domain:552:15)
    at maybeClose (node:internal/child_process:1093:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)
    at Process.callbackTrampoline (node:internal/async_hooks:130:17)

@UncleDave Run npm run test and npm run build can reproduce this.

@wenqi73 I ran those before my commit, they both succeed for me.

@UncleDave Maybe you should merge the latest code of master and re-install dependencies?

@wenqi73 Is there no package-lock.json file? It looks like my build is now failing due to dependency type mismatches. You can also see that every other PR is now failing with the same issue that I originally had on this branch. Does this project have a solution for locking dependencies? It's very hard to make a change and have confidence it will build on the CI server.

@wenqi73 Is there anything else you need from me here? The code cov changes are weird since it says the coverage of ellipsis.directive.ts has changed, but I don't really see how. I can try to write a test for that if needed?


npm audit

Hi,
A quick npm audit update I had to run trough trying out the project.
Please let me know if there is anything I've missed out,
& have a great day.

@krzepah thanks, for your help... currently we use yarn for this project, so we need to update the yarn lock file as well, otherwise all the old dependencies will still be installed..

Here's how I updated the yarn.lock
git remote add aichbauer https://github.com/aichbauer/express-rest-api-boilerplate
git fetch
git checkout aichbauer/master yarn.lock
yarn
git add yarn.lock
git commit -m 'updated yarn.lock'
[security-update 480a3e5] updated yarn.lock
 **1 file changed, 1368 insertions(+), 830 deletions(-)**

The numbers there seem a bit high to me, so if you find any trouble again please let me know,
Thanks for you time


put locale:"zh-CN" in locale.json   run ./scripts/code.sh but result is software‘s menu not the Chinese language

put locale:"zh-CN" in locale.json   run ./scripts/code.sh but result is software‘s menu not the Chinese language
run ./scripts/code.sh --locale=zh-cn
The result is the same.

VS Code running from dev has not localization support. It comes only with built/shipped VS Code.


Code blocks using different font-sizes

Code blocks font-size in this issue are different. I guess this is some kind of bug right?
https://github.com/FullHuman/purgecss/issues/67

Bug Report Dump (Auto-generated)

Version 1.19.0<PHONE_NUMBER>)
Device: iPad7,5 (iOS 11.3)
TestFlight: true

Aha, great call out. We’re using Highlightr.js to parse code, and that lib uses a fixed font size. After the styled text refactor this should be a lot easier.
Great starter task for someone!
Sent with GitHawk


Fix multiple issues

This PR resolves some of the issues raised and other quality of life stuff

Fixes #145
Fixes #177
Fixes #138
Remove duplicate links in the reference section

Looks awesome thank you ❤


Convert ndp2 into Orville

We need to:

[x] Update IPMI address
[x] Update IPMI password
[x] Move /home off NFS
[x] Change the existing user account/add new users
[x] Update the ntp time source
[x] Add disks for storing images
[x] Move the orville software around to match LWA-SV
[x] Rewire the networking at the site (main and IPMI)
[x] Update the NDP configuration to rename or remove ndp2 where necessary

There's also a branch for this:  https://github.com/lwa-project/ng_digital_processor/tree/ndp2_is_orville

This just needs to be put in place now.

I'm waiting on https://github.com/lwa-project/lwana-issues/issues/21 to be resolved so I can finish my testing and close this.

It's done.


Better mocking of request state (Controller::curr())

At the moment there are many places around the code that rely on Controller::curr() to access the request / session.
This leads to code, such as the below, in test bootstrapping.
// Set dummy controller
$controller = Controller::create();
$controller->setRequest($request);
$controller->pushCurrent();
$controller->doInit();

I'm not sure what the solution would look like, but I would hope it's more elegant than relying on Controller state.
Best suggestion at the moment is to add $request to nearly every method argument, but investigation is required.

I wonder if a static instance of HTTPRequest would be a better reference point for these things than a static instance of Controller? Perhaps Application::inst()->getRequest()?

Sounds worth exploring. :) I agree it shouldn't be bound to Kernel, but maybe something specific to HTTPApplication?

Another option would be to provide it as a service in Injector: Injector::inst()->get(HTTPRquest::class)

That would require a much less intense API change. :)

It's a bit of a hack that is simply there to acknowledge that we won't fully separate ourselves from static references in SS4, so I think keeping the API surface designed specific for it to a minimum makes a lot of sense.

I did this in the middleware PR

Yep good job. :D


weird course info json for lab hours and lecture hours

As you can see, the course info JSON for SOEN 423 (let's call the JSON object obj) contains both obj["lab"]["hours"] = "two hours per week" and obj["labHours"] = "two hours per week".  Same is true from lectureHours.
This is clearly wrong.
The JSON output of the r scraper when ran on my personal machine does not contain these incorrect redundancies.  I suspect that this is "feature" of MongoDB where it avoids entirely replacing old data with new data and instead just tries to add the difference between the old and new versions of data when possible.

This is very confusing. I noticed this issue as well and dropped the course info database for dev and prod then reran the scraper and issues seemed to have been resolved. It must be that the old scraper has been run again

Misclick

I just cleared the DB and re-ran the webscraper and it's all good now

Do we want to try to fix the fact that properties end up getting duplicated when they're renamed? I think it's a non-issue any nothing needs to be changed. If there are no objections I will close this issue.

It can't hurt to clean up the JSON, we might as well do it.

The json is already clean

as per my comment above: "I just cleared the DB and re-ran the webscraper and it's all good now"

Yes I saw that, however the first sentence of your next comment thereafter confused me and made it seem like there is still this issue of redundancy.
"Do we want to try to fix the fact that properties end up getting duplicated when they're renamed?" makes it sound like there's still a problem.

yes, the problem is that when you rename or remove a property from the json spec and then store the new data to the DB, the old property will remain as the DB write merges what's currently contained in the DB with what's currently being added. This means that the old property ends up lingering. The workaround is to simply clear the DB and re-run the scraper so the old property actually gets removed. Due to the existence of this workaround and the fact that we aren't going to be updating the json spec often in the long run, I see this as a non-issue.

I also see it as a non-issue. It's probably good that the dp has an 'add-only' by default behaviour on insert.  Clearing and then inserting seems like a logical way to handle it.


Fixed #247. Added complexConjugate to Number and removed isCompleConjugateOf:

…ugateOf:

This is better. Those tests you deleted didn't seem useful and were not well named.
The coverage has gone down. Do check whether the loss represents something significant before merging.


[Template] Remove unused perf and stress folders

Summary
The focus of these changes is to remove the unused perf and stress folders from the template.  The Azure SDK team no longer requires performance tests for each library and partner teams do not use the
scaffolding when generating a new library.  Likewise, the stress folder is empty, save a README, which simply points to a wiki location that our onboarding guide also references.

/azp run net - template - tests

@m-nash, @AlexanderSher :  Do either of you know of any code generation areas that this would impact?
@hallipr: Do you know of any EngSys tooling that this would impact?

@m-nash, @AlexanderSher : Do either of you know of any code generation areas that this would impact? @hallipr: Do you know of any EngSys tooling that this would impact?

I think it is good to understand if this is used in DPG tooling or not. @lirenhe could you confirm if this is used in DPG tooling and if so who can help us with validation of those scenarios?

@m-nash, @AlexanderSher : Do either of you know of any code generation areas that this would impact? @hallipr: Do you know of any EngSys tooling that this would impact?

I think it is good to understand if this is used in DPG tooling or not. @lirenhe could you confirm if this is used in DPG tooling and if so who can help us with validation of those scenarios?

As far as I know, we don't use them in DPG.  I approved this PR.

Given the lack of responses, I'm going to merge and see what happens.  If anything breaks as a result, we'll adjust.


Default Read and Request timeout with Apache Client

Hi.
In other clients I do see some explicit 120 seconds to connect, read, write timeout.
Wondering if for the Apache client we're defaulting to org.apache.hc.client5.http.config.RequestConfig#DEFAULT or something else?
And what would be CT's recommended way to set default request configuration timeouts?
Say, something like this?
return new CtApacheHttpClient(
        httpAsyncClientBuilder ->
            httpAsyncClientBuilder
                .setDefaultRequestConfig(RequestConfig.custom()
                    .setConnectionRequestTimeout(10, TimeUnit.SECONDS)
                    .setConnectTimeout(10, TimeUnit.SECONDS)
                    .setResponseTimeout(10, TimeUnit.SECONDS)
                    .build())
    );

Or would the Queuing middleware be the way to go as explained in https://commercetools.github.io/commercetools-sdk-java-v2/javadoc/com/commercetools/docs/meta/ClientTuning.html ?
Thanks.

The Apache Client module creates a client instance with it's default parameters. I took a look and it seems it's 3 minutes for response and connect timeouts.
To manually configure them you can directly instanciate the Client instance and configure it according to the Apache HTTP client docs:
        ApiRootBuilder.of(new CtApacheHttpClient(
                64,
                64,
                builder -> builder
                        .setIOReactorConfig(IOReactorConfig.custom() // e.g. set the socket timeout
                                .setSoTimeout(10, TimeUnit.SECONDS)
                                .build())
                        .setDefaultRequestConfig(RequestConfig.custom() // or configure the request parameters
                                .setResponseTimeout(10, TimeUnit.SECONDS)
                                .setConnectTimeout(3, TimeUnit.SECONDS)
                                .build())
        ));

Thanks for the confirmation, Jens.


Optimal transport with torch + GPU?

In the current implementation, tensors are moved to numpy + CPU before calling the optimal transport solver, see e.g. https://github.com/atong01/conditional-flow-matching/blob/main/torchcfm/optimal_transport.py#L88.
Since version 0.8, the POT package supports backends beyond numpy, and GPU acceleration, see https://pythonot.github.io/quickstart.html#solving-ot-with-multiple-backends-on-cpu-gpu. This can speed up the OT solver especially for large batchsize, and enables new features like differentiation through the solver. Is there a reason why torchcfm uses the numpy + cpu policy?
I am successfully using the torch + GPU support of POT and am happy to file a PR if there is interest in including this in torchcfm.

Hi,
Thanks for opening this issue! I haven't tried this, and am very curious if you've found any performance benefits?
From my understanding, there do not exist fast exact OT GPU solvers, only Sinkhorn-based ones. If you look behind the hood of POT, I believe even using a backend with GPU support still solves exact OT on the CPU:
https://pythonot.github.io/quickstart.html#gpu-acceleration
When implementing this package there was a bug in POT that allocated extra GPU memory when using pytorch backend (see https://github.com/PythonOT/POT/issues/523). I also found that when doing DDP POT seemed to be doing operations only on one of the GPUs (so incurring GPU-GPU transfer cost). So I didn't explore this option at the time and manually disabled the backend.
I haven't checked if they are doing anything smarter than what I implemented here. Would be happy to include in torchcfm if you file a PR at least as optional. Before making default would like to know that

It's at least not slower
Works seamlessly on multi-gpu setups

Do you know of any methods that are making use of back-propping through the solver in this setting? As you mention it could be useful there.
Thanks again for your
Alex


Extend Asn1Parser to parse UTF8Strings

@markt-asf, thank for merging. I will come up with my addtional X.509 username retriever idea shortly.


[RFC][Do Not Merge] RPC protocols

Description
This is a draft protocols for data.
Automatic authentication
Adding @need_authentication in the comment of a service will enable this feature. Each request in the service needs to have a field for the credential required cred_proto.Credential creds.
The struct authentication::Credential has a method,  "pub fn auth(&self) -> bool".  Developers can implement their customized authentication logic.
see mesatee_services/common/authentication/src/lib.rs#L44
When there is a RPC request, the  Credential::auth() will be invoked first to enforce the authentication.
See mesatee_services/tdfs/external_proto/src/prost_generated/tdfs_proto.rs#L157
Different types of data
Classification

private input
private output
fusion output
fusion input

Differences

If an input or output belongs to only one client, it's private.  Owners can directly register private input or private output.
If a data belongs to multiple clients, it's fusion data. Owners need to consent to a multiparty task with fusion output.  Once the output is written by the task, it becomes a fusion input. Owners can use it in other tasks with all owners' approvals.

Information
A data may contain following information

meta information: hash/size before encryption and after encryption
logical access information: key config for file encryption/decryption
physical access information: the storage information for accessing the file

For private data, the logical access information and physical access information are provided by the owner and the owner can't retrieve the these information from the platform. That means, no cryptographic keys nor storage access tokens will be sent to clients.
For fusion data, the logical access information and physical access information are managed by the platform and the owners can't access these information.
Operations on data
private input

registration: Owner needs to register the input before using in a task.  Owner needs to provide meta information, logical access information, physical access information
used as task input:  Owner needs to approve the task.  Platform will fetch the meta information, logical access information and physical access information for the Worker
used as task output: disallowed
read by owner: disallowed, unless using a task to copy the input to an output

private output

registration: Owner needs to register the output before using in a task. Owner needs to provide physical access information and logical access information.
used as task input: disallowed. Owner has to read the output and re-register an input.
used as task output: Owner needs to approve the task. Platform will fetch the logical access information and physical access information for the Worker. After the task is executed, Platform will  update the meta information for the output.
read by owner: Partial information. Platform only tells the Owner the meta information and status.  Owner can read the content with  physical access information and logical access information, which are provided by the owner.

fusion output

registration: Platform will register the fusion output according to the task information.  Data identifier will be returned as part of the task result.
used as task input: After a task writes the result to the fusion output, Platform will register it as a fusion input.  See operations on fusion input.
used as task output: Platform will send the logical access information and physical access information to the Worker.
read by owner: Partial information.   Platform only tells the Owner the meta information and status.  See operations on fusion input.

fusion input

registration: After a task writes result to the fusion output, Platform will register it as a fusion input.
used as task input: All the owners need to approve the task.
used as task output: disallowed.
read by owner: disallowed, unless using a task to copy the input to an output. All the owners need to approve the task.

RPC
DataStoreService for user
see mesatee_services/tdfs/external_proto/src/proto/data.proto
service DataStoreService {
  // Data owner can use this request to register private input
  rpc RegisterPrivateInput(RegisterPrivateInputRequest) returns (RegisterPrivateInputResponse);

  // Data owner can use this request to register private output
  rpc RegisterPrivateOutput(RegisterPrivateOutputRequest) returns (RegisterPrivateOutputResponse);

 // Data owner can use this request to retrieve the meta information and status of a private output.
  rpc ReadPrivateOutput(ReadPrivateOutputRequest) returns (ReadPrivateOutputResponse);

  // Data owner can use this request to delete a private input or output.
  rpc DeletePrivateData(DeletePrivateDataRequest) returns (DeletePrivateDataResponse);
}

DataStoreService for platform
see mesatee_services/tdfs/internal_proto/src/proto/data.proto
service DataStoreService {
  // TMS can use this request to update the output's meta information
  rpc WritePrivateOutput(WritePrivateOutputRequest) returns (WritePrivateOutputResponse);

  // TMS can use this request to fetch the input for a worker
  rpc FetchPrivateInput(FetchPrivateInputRequest) returns (FetchPrivateInputResponse);

  // TMS can use this request to fetch the output for a worker.
  rpc FetchPrivateOutput(FetchPrivateOutputRequest) returns (FetchPrivateOutputResponse);
}

FusionDataService for user
see mesatee_services/tdfs/external_proto/src/proto/data.proto
service FusionDataService {
  // Data owner can use this request to retrieve the meta information and status of a fusion data
  rpc ReadFusionData(ReadFusionDataRequest) returns (ReadFusionDataResponse);

  // Data owner can use this request to delete a fusion data.
  rpc DeleteFusionData(DeleteFusionDataRequest) returns (DeleteFusionDataResponse);
}

FusionDataService for platform
see mesatee_services/tdfs/internal_proto/src/proto/data.proto
service FusionDataService {
  // TMS can use this request to register a fusion output
  rpc RegisterFusionOutput(RegisterFusionOutputRequest) returns (RegisterFusionOutputResponse);

  // TMS can use this request to update the fusion output's meta information
  rpc WriteFusionOutput(WriteFusionOutputRequest) returns (WriteFusionOutputResponse);

  // TMS can use this request to fetch a fusion data as an input for the worker.
  rpc FetchFusionData(FetchFusionDataRequest) returns (FetchFusionDataResponse);
}

Type of change (select applied and DELETE the others)

[X] Breaking change (fix or feature that would cause existing functionality to not work as expected)

How Has This Been Tested?
Successfully compiled.
Checklist (check ALL before submitting PR, even not applicable)

[X] Fork the repo and create your branch from master.
[X] If you've added code that should be tested, add tests.
[X] If you've changed APIs, update the documentation.
[X] Ensure the tests pass (see CI results).
[X] Make sure your code lints/format.

Description for Function Proto
This is the draft protocol for function.
In traditional Faas, there is a step to setup code or action. So we propose a similar step, function registration.
With function, there are following advantages:

provide a clear description of a computational logic, including input name, input description, output name, output description, function description and so on.
support script attestation/audit: because a script  (like python script for MesaPy)  is provided by client, the script should be attested or audited to ensure that it will not leak client's data.
make it easier for clients to consent to a multiparty task.
reuse functions.

Function Structure
see mesatee_services/tdfs/common_proto/src/proto/function_common.proto
// The defintion of the function.
message FunctionDefinition {
    // function identifier.
    required string function_id = 1;
    // the description of the function
    required string description = 2;
    // function owner
    required string owner = 3;
    // script content
    required bytes script = 4;
    // hash of the script
    required string script_hash = 5;
    // whether the function is public or private.
    // if the function is private, the task need the owner's approval.
    required bool public = 6;
    // input arguments
    repeated Arg input_args = 7;
    // output arguments
    repeated Arg output_args = 8;
    // relationship between input and output
    repeated ArgRelationship arg_relationships = 9;
    // hash of the function defintion (script content are not included).
    required string function_hash = 10;
}

How to attest or audit a function
Function can't be changed after registration, and the platform should verify the hash of a function.
Participants just need to check the function id or function hash
If a function is public, all the users can get the function definition and the script content.
If a function is private, the owner can get the function definition and the script content.
How to verify a hash of a private function for a multiparty task
It needs discussion
Potential ways

The platform signs the function definition, so the owner can share it to participants.
The platform provides an api to verify a function's hash
Include the hash in a task and platform will verify it.

How to audit a private function for a multiparty task

The owner shares the script to the participants. The platform will verify the hash of the function.
A trusted third party signs the function.

RPC
FunctionStore Service for users
see mesatee_services/tdfs/external_proto/src/proto/function.proto
service FunctionStoreService {
    \\ Function owner can use this request to register a function
    rpc RegisterFunction(RegisterFunctionRequest) returns (RegisterFunctionResponse);

    \\ Clients can use this request to read a function if the client is the owner or the function is public
    rpc ReadFunction(ReadFunctionRequest) returns (ReadFunctionResponse);

    \\ Function owner can use this request to delete a function
    rpc DeleteFunction(DeleteFunctionRequest) returns (DeleteFunctionResponse);
}

FunctionStore Service for platform
see mesatee_services/tdfs/internal_proto/src/proto/function.proto
service FunctionStoreService {
   \\ TMS can use this request to fetch the function for the worker
    rpc FetchFunction(FetchFunctionRequest) returns (FetchFunctionResponse);
}

Description for Task protocols
This is a draft protocol for Task.
Task defines a computation process, that means, the computation logic (Function) and the input/output data (Data).
Task in Clients' View
message TaskAgreement {
    // Task identifier
    required string task_id = 1;
    // Function identifier
    required string function_id = 2;
    // Input arguments
    map<string, InputArg> input_arguments = 3;
    // Output arguments
    map<string, OutputArg> output_argments = 4;
    // Participants
    repeated string participants = 5;
    // Signatures of the participants
    map<string, bytes> signatures = 6;
}

Participants need to consent to the TaskAgreement.  They can know what function will be executed, who provides the certain input, who will get the certain output.
TaskAgreement doesn't contain the script content or input content, which are represented by function_id and data_id.
// input argument in TaskAgreement
message InputArg {
    // private input or fusion input
    required bool is_private = 1;
    // owners of the input: private - single owner, fusion - multiple owners
    repeated string owners = 2;
    // data id of the input
    required string data_id = 3;
}
// output argument in TaskAgreement
message OutputArg {
    // private output or fusion output
    required bool is_private = 1;
    // owners of the input: private - single owner, fusion - multiple owners
    repeated string owners = 2;
    // data id of the output
    // fusion output may not have a data id before the TaskAgreement is accepted by the TMS
    optional string data_id = 3;
}

Because there is no sensitive information in the TaskAgreement, multiple participants can consent to the TaskAgreement offline. Participants need to approve the Task by signing the TaskAgreement
And an output can belong to multiple users (data fusion), no one can use or access it without all the owners' approvals. Owners can get the data id of a fusion output after the Task is accepted by the platform.
Task in Workers' View
message ExecutableTask {
  // Task identifer
  required string task_id = 1;
  // Task token is used for the worker to update the Task status and output meta
  required string task_token = 2;
  // Function information
  required Function function = 3;
  // Input Arguments
  map<string, InputArg> input_arguments = 4;
  // Output Arguments
  map<string, OutputArg> output_argments = 5;
}

For workers, ExecutableTask provides all the required information (function and input/output) to finish a computation process without the need to query additional information from the platform.
Function

If Function is a built-in function, then the Function is represented by a pre-defined function id.
If Function is a script, then the Function is represented by the bytes of the script content.

// Function information
message Function {
  // Built-in function only needs function_id
  optional string function_id = 1;
  // script content
  required bytes script = 2;
}

Input Data
Workers don't need to know the data id of an input data. But they need to know where to access the data, how to decrypt the data and how to verify the data integrity before use.
So in ExecutableTask, the input data is represented by the meta information, physical access information and logical access information.
// Input Arugment
message InputArg {
  // Meta information is used to verify the data integrity.
  required data_common.DataMeta meta = 1;
  // Storage information is used to access the physical data.
  required data_common.DataStorageInfo storage_info = 2;
  // Key config is used to decrypt the data.
  required kms_proto.KeyConfig config = 3;
}

Output Data
Workers don't need to know the data id of an output data. But they need to know where to to access the data and how to encrypt the data.
So in ExecutableTask, the output data is represented by the physical information and logical access information.
// Output Argument
message OutputArg {
  // Storage information is used to access the physical data.
  required data_common.DataStorageInfo storage_info = 1;
  // Key config is used to encrypt the data.
  required kms_proto.KeyConfig config = 2;
}

Stateless
Because the ExecutableTask contains all the required information and a worker doesn't need to query additional information from the platform, a worker is considered as a stateless computation component.
Once the function and data are prepared, the worker can be used to do the computation without the platform. Thus, we can directly test a worker without the platform.
RPC
TaskService for User
see mesatee_services/tms/tms_proto/src/proto/task_external.proto
service TaskService {
    // Client can use this request to propose a Task
    rpc RegisterTask(RegisterTaskRequest) returns (RegisterTaskResponse);
    // Client can use this request to query Task information.
    rpc GetTask(GetTaskRequest) returns (GetTaskResponse);
    // Client can use this request to invoke a Task.
    rpc InvokeTask(InvokeTaskRequest) returns (InvokeTaskResponse);
}

TaskService for Platform
see mesatee_services/tms/tms_proto/src/proto/task_internal.proto
service TaskService {
  // Worker can use this request to ask the scheduler to update the task status and output meta.
  // Scheduler can use this request to ask TMS to update the task status and output meta
  rpc UpdateTask(UpdateTaskRequest) returns (UpdateTaskResponse);
}


The \Error class must not have a protected $trace attribute

Code:
class a extends \Error { private $trace; }

Result:
[daniil@daniil-arch jav]$ php src/y.php

[daniil@daniil-arch jav]$ jppm start
-> linux
-> app:run
-> install

Fatal error: Uncaught Error: Access level to a::$trace must be protected (as in class Error) or weaker in res://y.php on line 3, position 34
Stack Trace:
#0 {main}
  thrown in res://y.php on line 3

Fixed.


Migrate ReverseSequence from contrib op to ONNX opset 10

As the contrib op was only used internally in the last few weeks for testing, remove it (can revert last commit to re-instate if needed).
Only change is to use int64_t for sequence_lens.

I don't see the schema removed from contrib_defs

core\graph\contrib_ops\reverse_sequence_schema_defs.cc contains the schema for this op. Shouldn't it be removed?

Good catch. Removed.


Trying to add panel and update to grapesJS 0.16

Hello,
To be able to see generated Mavo code, I tried to adapt the GrapesJS export example from https://grapesjs.com/docs/getting-started.html#panels-buttons to the Create index.js syntax, I can display the export button using:
pnm.addButton('options', {
  id: 'export',
  className: 'btn-open-export',
  command: 'export-template',
      context: 'export-template',
  attributes: {
    'title': 'Show code to export',
    'data-tooltip-pos': 'bottom',
  }
});

But nothing happens when clicked, no error in browser or server, it seems that the internal function export was not implemented back then. The local GrapesJS of Create is 0.12 and nowadays GrapesJS is 0.16.
Replacing local  /grapesjs calls with:
<link rel="stylesheet" href="https://unpkg.com/grapesjs/dist/css/grapes.min.css">

didn't do the trick :smirk:
I'm not confident/good/reckless enough to adapt Create and/or grapesjs-mavo to GrapesJS 0.16 but it would be needed I guess.
@LeaVerou, what do you think?

You can try to adapt it locally and see what happens. After all, this project is currently unmaintained, so the only way you can make it go is forwards!

I did and it failed.
With my limited coding skills, can't go forwards :(

The best way to develop coding skills is to work on something you like.

You're right @LeaVerou, but I also need some help and drilling a 2 years old plugin and 2 projects that evolved greatly is not really something I can do, or even like :smile:
I tested a little bit of Mavo and liked the idea though.
BTW, I didn't find how to deal with arrays in Mavo, nor how to link records and local storage is limited to 6MB, so I wonder about sqlite for offline and a REST API, with a preact or Vue for my PWA need.
If I'm wrong and PouchDB does the job (it seems abandoned, 22 PR waiting, last commit is sept 2019), or another way, please let me know.

Hello @LeaVerou
I tried re-implemeting Mavo with GrapesJS and it doesn't show "Edit" in the top bar, only "Save".
Adding
 and container: '#gjs', into grapesjs.init({ breaks Mavo.
Can't go further.
Would you please test it on your side?
Voici un très court exemple:
<!DOCTYPE html>

    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link rel="stylesheet" href="https://get.mavo.io/mavo.css" />

    <title>Document</title>

                Lea
                Verou