Pdb mode

Excuse me for asking again and again.
"bash scripts/spot3.sh".
After running the above code, terminal goes into Pdb mode.
What should I enter here?

Could you attach the full terminal outputs? This suggests loss goes to nan.

 Please use QuaternionCoeffOrder.WXYZ instead.
warnings.warn("XYZW quaternion coefficient order is deprecated and"
/home/kana/anaconda3/envs/lasr/lib/python3.8/site-packages/torch/nn/functional.py:3385: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
warnings.warn("Default grid_sample and affine_grid behavior has changed "
/home/kana/lasr/nnutils/train_utils.py:76: RuntimeWarning: invalid value encountered in true_divide
timg = (timg-timg.min())/(timg.max()-timg.min())

/home/kana/lasr/nnutils/train_utils.py(295)train()
-> self.optimizer.step()
(Pdb)

Interesting, it stops at the first few iterations. This might be caused by improperly generated data.
Were you able to run optimization for camel?

No, Camel's optimization also fails.
bash scripts/render_result.sh camel
Setting up model..
loading log/camel-5/pred_net_10.pth..
Traceback (most recent call last):
File "extract.py", line 253, in
app.run(main)
File "/home/shiori/anaconda3/envs/lasr/lib/python3.8/site-packages/absl/app.py", line 303, in run
_run_main(main, args)
File "/home/shiori/anaconda3/envs/lasr/lib/python3.8/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "extract.py", line 237, in main
predictor = pred_util.MeshPredictor(opts)
File "/home/shiori/lasr/nnutils/predictor.py", line 60, in init
self.load_network(self.model, 'pred', self.opts.num_train_epoch)
File "/home/shiori/lasr/nnutils/predictor.py", line 108, in load_network
states = torch.load(save_path)
File "/home/shiori/anaconda3/envs/lasr/lib/python3.8/site-packages/torch/serialization.py", line 581, in load
with _open_file_like(f, 'rb') as opened_file:
File "/home/shiori/anaconda3/envs/lasr/lib/python3.8/site-packages/torch/serialization.py", line 230, in _open_file_like
return _open_file(name_or_buffer, mode)
File "/home/shiori/anaconda3/envs/lasr/lib/python3.8/site-packages/torch/serialization.py", line 211, in init
super(_open_file, self).init(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: 'log/camel-5/pred_net_10.pth'
ffmpeg version 3.4.8-0ubuntu0.2 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
libavutil      55. 78.100 / 55. 78.100
libavcodec     57.107.100 / 57.107.100
libavformat    57. 83.100 / 57. 83.100
libavdevice    57. 10.100 / 57. 10.100
libavfilter     6.107.100 /  6.107.100
libavresample   3.  7.  0 /  3.  7.  0
libswscale      4.  8.100 /  4.  8.100
libswresample   2.  9.100 /  2.  9.100
libpostproc    54.  7.100 / 54.  7.100
[image2 @ 0x56294c2da8e0] Pattern type 'glob_sequence' is deprecated: use pattern_type 'glob' instead
[image2 @ 0x56294c2da8e0] Could not open file : log/camel-5/render-.png
[image2 @ 0x56294c2da8e0] Could not find codec parameters for stream 0 (Video: png, none(pc)): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, image2, from 'log/camel-5/render-%.png':
Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
Stream #0:0: Video: png, none(pc), 25 tbr, 25 tbn, 25 tbc
Output #0, mp4, to 'log/camel-5/camel-camel-5-10.mp4':
Output file #0 does not contain any stream
ffmpeg version 3.4.8-0ubuntu0.2 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
libavutil      55. 78.100 / 55. 78.100
libavcodec     57.107.100 / 57.107.100
libavformat    57. 83.100 / 57. 83.100
libavdevice    57. 10.100 / 57. 10.100
libavfilter     6.107.100 /  6.107.100
libavresample   3.  7.  0 /  3.  7.  0
libswscale      4.  8.100 /  4.  8.100
libswresample   2.  9.100 /  2.  9.100
libpostproc    54.  7.100 / 54.  7.100
log/camel-5/camel-camel-5-10.mp4: No such file or directory
log/camel-5/
camel/0
no mesh found
camel/1
no mesh found
camel/2
no mesh found
camel/3
no mesh found
camel/4
no mesh found
camel/5
no mesh found
camel/6
no mesh found
camel/7
no mesh found
camel/8
no mesh found
camel/9
no mesh found
camel/10
no mesh found
camel/11
no mesh found
camel/12
no mesh found
camel/13
no mesh found
camel/14
no mesh found
camel/15
no mesh found
camel/16
no mesh found
camel/17
no mesh found
camel/18
no mesh found
camel/19
no mesh found
camel/20
no mesh found
camel/21
no mesh found
camel/22
no mesh found
camel/23
no mesh found
camel/24
no mesh found
camel/25
no mesh found
camel/26
no mesh found
camel/27
no mesh found
camel/28
no mesh found
camel/29
no mesh found
camel/30
no mesh found
camel/31
no mesh found
camel/32
no mesh found
camel/33
no mesh found
camel/34
no mesh found
camel/35
no mesh found
camel/36
no mesh found
camel/37
no mesh found
camel/38
no mesh found
camel/39
no mesh found
camel/40
no mesh found
camel/41
no mesh found
camel/42
no mesh found
camel/43
no mesh found
camel/44
no mesh found
camel/45
no mesh found
camel/46
no mesh found
camel/47
no mesh found
camel/48
no mesh found
camel/49
no mesh found
camel/50
no mesh found
camel/51
no mesh found
camel/52
no mesh found
camel/53
no mesh found
camel/54
no mesh found
camel/55
no mesh found
camel/56
no mesh found
camel/57
no mesh found
camel/58
no mesh found
camel/59
no mesh found
camel/60
no mesh found
camel/61
no mesh found
camel/62
no mesh found
camel/63
no mesh found
camel/64
no mesh found
camel/65
no mesh found
camel/66
no mesh found
camel/67
no mesh found
camel/68
no mesh found
camel/69
no mesh found
camel/70
no mesh found
camel/71
no mesh found
camel/72
no mesh found
camel/73
no mesh found
camel/74
no mesh found
camel/75
no mesh found
camel/76
no mesh found
camel/77
no mesh found
camel/78
no mesh found
camel/79
no mesh found
camel/80
no mesh found
camel/81
no mesh found
camel/82
no mesh found
camel/83
no mesh found
camel/84
no mesh found
camel/85
no mesh found
camel/86
no mesh found
camel/87
no mesh found
camel/88
no mesh found
database/DAVIS/JPEGImages/Full-Resolution/camel/00000.jpg
Traceback (most recent call last):
File "render_vis.py", line 292, in
main()
File "render_vis.py", line 207, in main
refmesh = all_mesh[i]
IndexError: list index out of range

You were attaching the output of the rendering scrips. Do you have the output of the optimization script?
bash scripts/template.sh camel

Failed to execute.
bash scripts/template.sh camel

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

Jitting Chamfer 3D
Jitting Chamfer 3D
Loaded JIT 3D CUDA chamfer distance
Loaded JIT 3D CUDA chamfer distance
Traceback (most recent call last):
File "optimize.py", line 59, in
app.run(main)
File "/home/kana/anaconda3/envs/lasr/lib/python3.8/site-packages/absl/app.py", line 303, in run
_run_main(main, args)
File "/home/kana/anaconda3/envs/lasr/lib/python3.8/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "optimize.py", line 42, in main
torch.distributed.init_process_group(
File "/home/kana/anaconda3/envs/lasr/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 436, in init_process_group
store, rank, world_size = next(rendezvous_iterator)
File "/home/kana/anaconda3/envs/lasr/lib/python3.8/site-packages/torch/distributed/rendezvous.py", line 179, in _env_rendezvous_handler
store = TCPStore(master_addr, master_port, world_size, start_daemon, timeout)
RuntimeError: Address already in use
Traceback (most recent call last):
File "optimize.py", line 59, in
app.run(main)
File "/home/kana/anaconda3/envs/lasr/lib/python3.8/site-packages/absl/app.py", line 303, in run
_run_main(main, args)
File "/home/kana/anaconda3/envs/lasr/lib/python3.8/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "optimize.py", line 40, in main
torch.cuda.set_device(opts.local_rank)
File "/home/kana/anaconda3/envs/lasr/lib/python3.8/site-packages/torch/cuda/init.py", line 263, in set_device
torch._C._cuda_setDevice(device)
RuntimeError: CUDA error: invalid device ordinal
Traceback (most recent call last):
File "/home/kana/anaconda3/envs/lasr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/kana/anaconda3/envs/lasr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/kana/anaconda3/envs/lasr/lib/python3.8/site-packages/torch/distributed/launch.py", line 260, in
main()
File "/home/kana/anaconda3/envs/lasr/lib/python3.8/site-packages/torch/distributed/launch.py", line 255, in main
raise subprocess.CalledProcessError(returncode=process.returncode,
subprocess.CalledProcessError: Command '['/home/kana/anaconda3/envs/lasr/bin/python', '-u', 'optimize.py', '--local_rank=1', '--name=camel-0', '--checkpoint_dir', 'log/', '--only_mean_sym', '--nouse_gtpose', '--subdivide', '3', '--n_bones', '21', '--n_hypo', '16', '--num_epochs', '20', '--dataname', 'rcamel', '--sil_path', 'none', '--ngpu', '2', '--batch_size', '2', '--opt_tex', 'yes']' returned non-zero exit status 1.

This is related to pytorch distributed data parallel:
Seems the previous process hangs and occupies the port (and likely gpu memory) and you'll need to force kill them before launching a new process
pkill -f xxx

where xxx is a substring of the command that hangs
https://github.com/NVIDIA/tacotron2/issues/181#issuecomment-481607690

Overall, I'd suggest to use the pre-rendered spot data and run optimization again. The code was tested on different machines and I didn't observe similar issues. Let me know if it still produces nan loss.

I ran the optimization again, using the pre-trained data.
However, I get the following.
bash scripts/spot3.sh
Jitting Chamfer 3D
Loaded JIT 3D CUDA chamfer distance
1/0
workers: 1
pairs: 1
init:0, end:-1
198 paris of images
Only the mean shape is symmetric!
found mean v
found tex
found ctl rotation
found rest translation
found ctl points
found log ctl
running k-means on cuda:0..
[running kmeans]: 17it [00:00, 341.07it/s, center_shift=0.000000, iteration=18, tol=0.000100]running k-means on cuda:0..
running k-means on cuda:0..
[running kmeans]: 13it [00:00, 337.02it/s, center_shift=0.000000, iteration=14, tol=0.000100]running k-means on cuda:0..
running k-means on cuda:0..
[running kmeans]: 13it [00:00, 335.13it/s, center_shift=0.000000, iteration=14, tol=0.000100]running k-means on cuda:0..
running k-means on cuda:0..
[running kmeans]: 20it [00:00, 340.19it/s, center_shift=0.000000, iteration=21, tol=0.000100]running k-means on cuda:0..
new bone locations
scores:g kmeans]: 14it [00:00, 350.38it/s, center_shift=0.000000, iteration=15, tol=0.000100]
tensor([0., 0., 0., 0., 0., 0., 0., 0.], device='cuda:0')
selecting 0eans]: 13it [00:00, 340.97it/s, center_shift=0.000000, iteration=14, tol=0.000100]
[running kmeans]: 18it [00:00, 44.70it/s, center_shift=0.000000, iteration=18, tol=0.000100]
[running kmeans]: 14it [00:00, 39.79it/s, center_shift=0.000000, iteration=14, tol=0.000100]
[running kmeans]: 14it [00:00, 44.92it/s, center_shift=0.000000, iteration=14, tol=0.000100]
[running kmeans]: 21it [00:00, 77.39it/s, center_shift=0.000000, iteration=21, tol=0.000100]
[running kmeans]: 15it [00:00, 71.10it/s, center_shift=0.000000, iteration=15, tol=0.000100]
[running kmeans]: 14it [00:00, 82.74it/s, center_shift=0.000000, iteration=14, tol=0.000100]
[running kmeans]: 19it [00:00, 146.67it/s, center_shift=0.000000, iteration=19, tol=0.000100]
[running kmeans]: 26it [00:00, 347.56it/s, center_shift=0.000000, iteration=26, tol=0.000100]
/home/kana/anaconda3/envs/lasr/lib/python3.8/site-packages/kornia/geometry/conversions.py:506: UserWarning: XYZW quaternion coefficient order is deprecated and will be removed after > 0.6. Please use QuaternionCoeffOrder.WXYZ instead.
warnings.warn("XYZW quaternion coefficient order is deprecated and"
/home/kana/anaconda3/envs/lasr/lib/python3.8/site-packages/torch/nn/functional.py:3385: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
warnings.warn("Default grid_sample and affine_grid behavior has changed "

/home/kana/lasr3/nnutils/train_utils.py(295)train()
-> self.optimizer.step()
(Pdb)

I'm sorry.
I forgot that I was getting the following error when building docker.
Is it possible that this is the cause?
Since I got this error, I am using Build with conda to build the environment.
sudo docker build --tag lasr:latest -f docker/Dockerfile ./
Sending build context to Docker daemon  20.41MB
Step 1/15 : FROM nvidia/cuda:11.0-devel-ubuntu18.04
---> d89f75c1799d
Step 2/15 : ENV CONDA_DIR /anaconda3
---> Using cache
---> 952b951d6c2c
Step 3/15 : COPY third_party/softras /workspace/softras
---> Using cache
---> a28e87572e82
Step 4/15 : COPY lasr.yml /workspace/lasr.yml
---> Using cache
---> 34095cc757ec
Step 5/15 : RUN apt-get update -q
---> Running in 6f3373585c02
Ign:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease
Ign:2 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease
Err:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release
Could not handshake: Error in the pull function. [IP: <IP_ADDRESS> 443]
Err:4 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release
Could not handshake: Error in the pull function. [IP: <IP_ADDRESS> 443]
Err:5 http://security.ubuntu.com/ubuntu bionic-security InRelease
Connection failed [IP: <IP_ADDRESS> 80]
Err:6 http://archive.ubuntu.com/ubuntu bionic InRelease
Connection failed [IP: <IP_ADDRESS> 80]
Err:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Connection failed [IP: <IP_ADDRESS> 80]
Err:8 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Connection failed [IP: <IP_ADDRESS> 80]
Reading package lists...
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release' does not have a Release file.
E: The repository 'https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release' does not have a Release file.
The command '/bin/sh -c apt-get update -q' returned a non-zero code: 100

Since I'm not able to reproduce this issue, I can only give suggestions for debugging here.
To further debug the cause of total_loss being nan, I would print out the values of individual losses in the forward pass of the model, and check where it becomes nan , specifically from mask_loss to the [auxiliary losses]. (https://github.com/google/lasr/blob/a22780b533079befd29d0bc5e0f9ca8b95f43873/nnutils/mesh_net.py#L522)
Given the observations, my best guess is that the versions of packages went wrong. This is not likely to happen with conda environment. Please let me know if conda build works for you.

conda build is working properly.

If you try again from "Data preparation", you will see the following output after running auto_gen.py.
Could this be related to the problem?
Traceback (most recent call last):
File "preprocess/auto_gen.py", line 191, in
main()
File "preprocess/auto_gen.py", line 155, in main
print('%s/%s'%(test_left_img[inx],test_left_img[jnx]))
IndexError: list index out of range

Were you able to run optimization on spot? You don't need to run auto_gen.py for optimization on spot.

No, I get the same error as camel.

I'm still confused about how to reproduce your issue.
If you plan to investigate further, please make a summary about the issue here or in a new thread.
Please include your system hardware, conda&pip package list if you use conda install, and steps to reproduce your issue.

This is my experimental environment.
 What GPU card were you using ?

I use
TITAN X (Pascal)

Ok, I've validated on almost the same setting as yours.
If the same error exists, the best suggestion I can give is to print out the variable self.total_loss in the nnutils/mesh_net.py as mentioned earlier, and see where it becomes nan. For example, add print(self.total_loss) after this line. It should not be a valid number if everything is correct.

I inserted print(self.total_loss) and ran bash scripts/spot3.sh.
When I continue in PDB mode, I get the following output.

/home/shiori/lasr/nnutils/train_utils.py(295)train()
-> self.optimizer.step()
(Pdb) c
/home/shiori/lasr/nnutils/train_utils.py:76: RuntimeWarning: invalid value encountered in true_divide
timg = (timg-timg.min())/(timg.max()-timg.min())
tensor(0.1620, device='cuda:0', grad_fn=)
tensor(nan, device='cuda:0', grad_fn=)
/home/shiori/lasr/nnutils/train_utils.py(294)train()
-> pdb.set_trace()
(Pdb) c
tensor(0.1632, device='cuda:0', grad_fn=)
/home/shiori/lasr/nnutils/train_utils.py(295)train()
-> self.optimizer.step()

Thanks. The code block l382-l522 in nnutils/mesh_net.py sequentially computes and adds the following losses: (1) mask loss, (2) flow loss, (3) rgb loss, (4) shape loss (5) deformation loss (6) bone symmetry loss (7) camera loss (8) auxiliary losses.
Given the information you provided, self.total_loss is valid after (1) but becomes nan after (8). To figure out when it becomes nan, you could print out the value of self.total_loss after adding each loss.

Thank you.
It was nan from the output of (3).
/home/kana/anaconda3/envs/lasr/lib/python3.8/site-packages/torch/nn/functional.py:3385: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
warnings.warn("Default grid_sample and affine_grid behavior has changed "
tensor(nan, device='cuda:0', grad_fn=)

/home/kana/lasr/nnutils/train_utils.py(295)train()
-> self.optimizer.step()
(Pdb) c
/home/kana/lasr/nnutils/train_utils.py:76: RuntimeWarning: invalid value encountered in true_divide
timg = (timg-timg.min())/(timg.max()-timg.min())
tensor(0.1620, device='cuda:0', grad_fn=)
tensor(0.1620, device='cuda:0', grad_fn=)
tensor(nan, device='cuda:0', grad_fn=)
tensor(nan, device='cuda:0', grad_fn=)
tensor(nan, device='cuda:0', grad_fn=)
tensor(nan, device='cuda:0', grad_fn=)
tensor(nan, device='cuda:0', grad_fn=)
tensor(nan, device='cuda:0', grad_fn=)
/home/kana/lasr/nnutils/train_utils.py(294)train()
-> pdb.set_trace()

Hi, are you able to further localize which line produces invalid value, that made the self.total_loss become nan?

The output of self.texture_loss_sub and self.texture_loss is as follows.
self.texture_loss_sub : tensor([[   nan,    nan,    nan,    nan,    nan,    nan, 0.1660, 0.1662],
[0.1692, 0.1693, 0.1693, 0.1695, 0.1692, 0.1693, 0.1694, 0.1695]],
device='cuda:0', grad_fn=)
self.texture_loss : tensor(nan, device='cuda:0', grad_fn=)
texture_loss: tensor(nan, device='cuda:0', grad_fn=)

Thanks, the self.texture_loss_sub is the sum of (1) rgb loss and (2) perceptual loss, it would help to localize further.

Thank you.
I got the following output.
rgb_loss: tensor(nan, device='cuda:0', grad_fn=)
rgb_loss: tensor(nan, device='cuda:0', grad_fn=)
rgb_loss: tensor(nan, device='cuda:0', grad_fn=)
rgb_loss: tensor(nan, device='cuda:0', grad_fn=)
rgb_loss: tensor(nan, device='cuda:0', grad_fn=)
rgb_loss: tensor(nan, device='cuda:0', grad_fn=)
rgb_loss: tensor(0.2586, device='cuda:0', grad_fn=)
rgb_loss: tensor(0.2586, device='cuda:0', grad_fn=)
rgb_loss: tensor(0.2601, device='cuda:0', grad_fn=)
rgb_loss: tensor(0.2601, device='cuda:0', grad_fn=)
rgb_loss: tensor(0.2601, device='cuda:0', grad_fn=)
rgb_loss: tensor(0.2601, device='cuda:0', grad_fn=)
rgb_loss: tensor(0.2601, device='cuda:0', grad_fn=)
rgb_loss: tensor(0.2601, device='cuda:0', grad_fn=)
rgb_loss: tensor(0.2601, device='cuda:0', grad_fn=)
rgb_loss: tensor(0.2601, device='cuda:0', grad_fn=)
percept_loss: tensor([   nan,    nan,    nan,    nan,    nan,    nan, 2.1181, 2.1905, 2.1126,
2.1569, 2.1562, 2.1891, 2.1188, 2.1328, 2.2336, 2.2318,    nan,    nan,
nan,    nan,    nan,    nan, 2.0471, 2.0863, 1.9113, 1.9823, 1.9965,
2.0729, 1.9108, 1.9763, 1.9714, 2.0456], device='cuda:0',
grad_fn=)

I'm not sure what happens. Could you add these lines before computing perceptual loss
data_save = {}
data_save['obspair'] =  obspair.detach().cpu().numpy()
data_save['rndpair'] = rndpair.detach().cpu().numpy()
data_save['verts_pre'] = verts_pre.detach().cpu().numpy()
data_save['faces']=faces.detach().cpu().numpy()
data_save['tex']=tex.detach().cpu().numpy()
np.save('./data.npy', data_save)

and share the saved data.npy file in the current folder with me? I will check whether it's a rendering problem or data loading problem.

OK.
The email has been sent.

Hi, the problem I found is that the variable "verts_pre" contains identical values.
To be more specific, verts_pre is the Nx3 matrix where each row is the view space (x,y,z) coordinate of a mesh vertex.
I don't know which part went wrong during the transformation from rest shape coordinate to view space. Could you share more information as follows?
import json
data_save = {}
data_save['pred_v'] =  pred_v.detach().cpu().numpy().tolist() # rest
data_save['verts_tex'] = verts_tex.detach().cpu().numpy().tolist() # camera
data_save['verts_pre'] = verts_pre.detach().cpu().numpy().tolist()  # view
data_save['offset'] = offset.detach().cpu().numpy().tolist()
with open('data.json', 'w') as f: json.dump(data_save, f)

Thank you.
I will share the information you mentioned.
https://drive.google.com/file/d/1fZXvDXO_QE6A2yGhdLH9h4AvWkHj3MoU/view?usp=sharing
2021年7月8日(木) 8:26 Gengshan Yang @.***>:

Hi, the problem I found is that the variable "verts_pre
https://github.com/google/lasr/blob/a22780b533079befd29d0bc5e0f9ca8b95f43873/nnutils/mesh_net.py#L347"
contains identical values.
To be more specific, verts_pre is the Nx3 matrix where each row is the
view space (x,y,z) coordinate of a mesh vertex.
I don't know which part went wrong during the transformation from rest
shape coordinate to view space. Could you share more information as follows?
import json
data_save = {}
data_save['pred_v'] =  pred_v.detach().cpu().numpy().tolist() # rest
data_save['verts_tex'] = verts_tex.detach().cpu().numpy().tolist() # camera
data_save['verts_pre'] = verts_pre.detach().cpu().numpy().tolist()  # view
data_save['offset'] = offset.detach().cpu().numpy().tolist()
with open('data.json', 'w') as f: json.dump(data_save, f)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/google/lasr/issues/3#issuecomment-875999672, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AUNDDDSG57XYI5DQAWP47RTTWTPBLANCNFSM47T55AMA
.

Thanks, we can nail down the problem to this block
The output after blend skinning and camera projection was wrong. I guess that's due to wrong transformation and camera parameters.
I will need the following information to further look into it.
import json
data_save = {}
data_save['Rmat_tex'] =  Rmat_tex.detach().cpu().numpy().tolist() # rotations
data_save['quat'] =  quat.detach().cpu().numpy().tolist() # rotations
data_save['Tmat'] =  Tmat.detach().cpu().numpy().tolist() # translation xyz
data_save['trans'] =  trans.detach().cpu().numpy().tolist() # translation xy
data_save['depth'] =  depth.detach().cpu().numpy().tolist() # translation z
data_save['ppoint'] =  ppoint.detach().cpu().numpy().tolist() # principle points
data_save['scale'] =  scale.detach().cpu().numpy().tolist() # focal length
data_save['cams'] =  self.cams.detach().cpu().numpy().tolist() # camera calibration
data_save['pp'] =  self.pp.detach().cpu().numpy().tolist() # camera calibration
with open('data.json', 'w') as f: json.dump(data_save, f)

Can you also share the checkpoint file log/spot3-0/pred_net_0.pth?

Thank you.
I will share the two files below.
https://drive.google.com/drive/folders/123NpMzh_4ZJnsTO0TQ_DXGdzf0ViTvSK?usp=sharing
2021年7月10日(土) 4:37 Gengshan Yang @.***>:

Thanks, we can nail down the problem to this block
https://github.com/google/lasr/blob/a22780b533079befd29d0bc5e0f9ca8b95f43873/nnutils/mesh_net.py#L343-L345
The output after blend skinning and camera projection was wrong. I guess
that's due to wrong transformation and camera parameters.
I will need the following information to further look into it.
import json
data_save = {}
data_save['Rmat_tex'] =  Rmat_tex.detach().cpu().numpy().tolist() # rotations
data_save['quat'] =  quat.detach().cpu().numpy().tolist() # rotations
data_save['Tmat'] =  Tmat.detach().cpu().numpy().tolist() # translation xyz
data_save['trans'] =  trans.detach().cpu().numpy().tolist() # translation xy
data_save['depth'] =  depth.detach().cpu().numpy().tolist() # translation z
data_save['ppoint'] =  ppoint.detach().cpu().numpy().tolist() # principle points
data_save['scale'] =  scale.detach().cpu().numpy().tolist() # focal length
data_save['cams'] =  self.cams.detach().cpu().numpy().tolist() # camera calibration
data_save['pp'] =  self.pp.detach().cpu().numpy().tolist() # camera calibration
with open('data.json', 'w') as f: json.dump(data_save, f)
Can you also share the checkpoint file log/spot3-0/pred_net_0.pth?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/google/lasr/issues/3#issuecomment-877413701, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AUNDDDW6EFLRPLCJVJZ2WMDTW5FXFANCNFSM47T55AMA
.

Thanks, the rotation, translation, focal length, and principal points all look correct. It has to be the problem of skinning weights.  However, I'm still not able to reproduce the problem even with the checkpoint you provided.
Could you also provide skinning weights?
import json
data_save = {}
data_save['skin'] =  skin.detach().cpu().numpy().tolist() # skinning weights
with open('data.json', 'w') as f: json.dump(data_save, f)

Thank you.Share the file.
https://drive.google.com/drive/folders/123NpMzh_4ZJnsTO0TQ_DXGdzf0ViTvSK?usp=sharing
2021年7月10日(土) 13:17 Gengshan Yang @.***>:

Thanks, the rotation, translation, focal length, and principal points all
look correct. It has to be the problem of skinning weights. However, I'm
still not able to reproduce the problem even with the checkpoint you
provided.
Could you also provide skinning weights?
import json
data_save = {}
data_save['skin'] =  skin.detach().cpu().numpy().tolist() # skinning weights
with open('data.json', 'w') as f: json.dump(data_save, f)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/google/lasr/issues/3#issuecomment-877561297, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AUNDDDUR7IW2VINT2VPLGYDTW7CVFANCNFSM47T55AMA
.

The file does not contain all the variables listed. Could you check the updated code and share again?

I'm sorry. I've updated it.
https://drive.google.com/drive/folders/123NpMzh_4ZJnsTO0TQ_DXGdzf0ViTvSK?usp=sharing
2021年7月11日(日) 4:07 Gengshan Yang @.***>:

The file does not contain all the variables listed. Could you check the
updated code and share again?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/google/lasr/issues/3#issuecomment-877688546, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AUNDDDRPE6AME2MJU3GEL5DTXCK5XANCNFSM47T55AMA
.

The rest bone location, orientation, scale all look correct. I guess it is a numerical issue.
Could you try replace this line with
skin = (-10 * dis_norm.sum(3)).double().softmax(1).float()[:,:,:,None] # h,j,n,1

If it still does not fix the issue, can you share with me the following?
import json
data_save = {}
data_save['skin'] =  skin.detach().cpu().numpy().tolist() # skinning weights
data_save['ctl_ts'] = self.ctl_ts.detach().cpu().numpy().tolist() # bone centroid
data_save['ctl_rs'] = self.ctl_rs.detach().cpu().numpy().tolist() # bone orientation
data_save['log_ctl'] = self.log_ctl.detach().cpu().numpy().tolist() # bone scale
data_save['dis_norm'] = dis_norm.detach().cpu().numpy().tolist() # bone scale
with open('data.json', 'w') as f: json.dump(data_save, f)

Thank you.
I fixed the code and ran it
However, it did not solve the problem.
https://drive.google.com/drive/folders/123NpMzh_4ZJnsTO0TQ_DXGdzf0ViTvSK?usp=sharing
2021年7月11日(日) 15:15 Gengshan Yang @.***>:

The rest bone location, orientation, scale all look correct. I guess it is
a numerical issue.
Could you try replace this line
https://github.com/google/lasr/blob/a22780b533079befd29d0bc5e0f9ca8b95f43873/nnutils/mesh_net.py#L260
with
skin = (-10 * dis_norm.sum(3)).double().softmax(1).float()[:,:,:,None] # h,j,n,1
If it still does not fix the issue, can you share with me the following?
import json
data_save = {}
data_save['skin'] =  skin.detach().cpu().numpy().tolist() # skinning weights
data_save['ctl_ts'] = self.ctl_ts.detach().cpu().numpy().tolist() # bone centroid
data_save['ctl_rs'] = self.ctl_rs.detach().cpu().numpy().tolist() # bone orientation
data_save['log_ctl'] = self.log_ctl.detach().cpu().numpy().tolist() # bone scale
data_save['dis_norm'] = dis_norm.detach().cpu().numpy().tolist() # bone scale
with open('data.json', 'w') as f: json.dump(data_save, f)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/google/lasr/issues/3#issuecomment-877748008, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AUNDDDROEDULOCIWK5LEUVDTXEZI7ANCNFSM47T55AMA
.

What about replacing this line with
            from pytorch3d import transforms
            ctl_rs = torch.cat([self.ctl_rs[:,3:4], self.ctl_rs[:,:3]],-1)
            dis_norm = dis_norm.matmul(transforms.quaternion_to_matrix(ctl_rs).view(opts.n_hypo,-1,3,3)) # h,j,n,3

Thank you.
The result did not change.
https://drive.google.com/drive/folders/123NpMzh_4ZJnsTO0TQ_DXGdzf0ViTvSK?usp=sharing
2021年7月11日(日) 15:58 Gengshan Yang @.***>:

What about replacing this line
https://github.com/google/lasr/blob/a22780b533079befd29d0bc5e0f9ca8b95f43873/nnutils/mesh_net.py#L258
with
        from pytorch3d import transforms
        ctl_rs = torch.cat([self.ctl_rs[:,3:4], self.ctl_rs[:,:3]],-1)
        dis_norm = dis_norm.matmul(transforms.quaternion_to_matrix(ctl_rs).view(opts.n_hypo,-1,3,3)) # h,j,n,3

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/google/lasr/issues/3#issuecomment-877751833, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AUNDDDTPF27PG3IHSTYA4VTTXE6HTANCNFSM47T55AMA
.

There is something apparently wrong in these lines that computs skinning weights, because you are getting dis_norm = 0.
dis_norm is the mahalanobis distance computed by (1) subtraction from bone position by vertex position (2) rotation and (3) scaling.
Can you debug a little bit and let me know when dis_norm becomes zeros?

I ran the following code to output dis_norm.
Only the output of (1) was not zero, does that mean that all outputs need to be zero?
GMM
        dis_norm = (self.ctl_ts.view(opts.n_hypo,-1,1,3) - pred_v.view(2*local_batch_size,opts.n_hypo,-1,3)[0,:,None].detach()) # p-v, H,J,1,3 - H,1,N,3
        print('(1)',dis_norm)
        #dis_norm = dis_norm.matmul(kornia.quaternion_to_rotation_matrix(self.ctl_rs).view(opts.n_hypo,-1,3,3)) # h,j,n,3
        from pytorch3d import transforms
        ctl_rs = torch.cat([self.ctl_rs[:,3:4], self.ctl_rs[:,:3]],-1)
        dis_norm = dis_norm.matmul(transforms.quaternion_to_matrix(ctl_rs).view(opts.n_hypo,-1,3,3)) # h,j,n,3
        print('(2)',dis_norm)
        dis_norm = self.log_ctl.exp().view(opts.n_hypo,-1,1,3) * dis_norm.pow(2) # (p-v)^TS(p-v)
        print('(3)',dis_norm)

The output of dis_norm should not be zero in any case. Does the output of transforms.quaternion_to_matrix(ctl_rs) look reasonable? They should be identity matrices of shape 3x3.

The outputs of ( 2) and (3 ) are both zero.

The following code was used to save the output of (2) to a json file.
It turned out that zero was output.
https://drive.google.com/file/d/1LqO9NTBZGmc9Qt0jppJqDVbkXc9JxSmb/view?usp=sharing
from pytorch3d import transforms
ctl_rs = torch.cat([self.ctl_rs[:,3:4], self.ctl_rs[:,:3]],-1)
dis_norm = dis_norm.matmul(transforms.quaternion_to_matrix(ctl_rs).view(opts.n_hypo,-1,3,3)) # h,j,n,3
        import json
        data_save = {}
        data_save['dis_norm.matmul(transforms.quaternion_to_matrix(ctl_rs).view(opts.n_hypo,-1,3,3))'] = dis_norm.matmul(transforms.quaternion_to_matrix(ctl_rs).view(opts.n_hypo,-1,3,3)).detach().cpu().numpy().tolist() # mahalanobis distance
        with open('data_2.json', 'w') as f: json.dump(data_save, f)

Ok, multiplying an identity matrix should not change the values from non-zero to zero. Can you verify this? Something was wrong going from (1) to (2).

The dimension of dis_norm should be HxBxNx3 before multiplication with the HxBx3x3 rotation matrix transforms.quaternion_to_matrix(ctl_rs). Maybe you could verify this by doing matrix multiplication at element level.
rotmat = transforms.quaternion_to_matrix(ctl_rs)
print(dis_norm[0,0].matmul(rotmat[0,0]))

I inserted the requested t-code between (1) and (2).
 Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
warnings.warn("Default grid_sample and affine_grid behavior has changed "

/home/kana/lasr4/nnutils/train_utils.py(295)train()
-> self.optimizer.step()
(Pdb)

This is in contradiction to the previous output. You mentioned the output of (2) are zeros. But shouldn't this be equivalent to the output of (2)?

I ran it with the following code.
The output of (2) is ZERO.
GMM
        dis_norm = (self.ctl_ts.view(opts.n_hypo,-1,1,3) - pred_v.view(2*local_batch_size,opts.n_hypo,-1,3)[0,:,None].detach()) # p-v, H,J,1,3 - H,1,N,3
        print('(1)',dis_norm)

        from pytorch3d import transforms
        ctl_rs = torch.cat([self.ctl_rs[:,3:4], self.ctl_rs[:,:3]],-1)
        rotmat = transforms.quaternion_to_matrix(ctl_rs)
        print(dis_norm[0,0].matmul(rotmat[0,0]))

        dis_norm = dis_norm.matmul(rotmat.view(opts.n_hypo,-1,3,3)) # h,j,n,3

        print('(2)',dis_norm)

        dis_norm = self.log_ctl.exp().view(opts.n_hypo,-1,1,3) * dis_norm.pow(2) # (p-v)^TS(p-v)
        print('(3)',dis_norm)

Hopefully you can see my logic for debugging. You need to figure out why dis_norm becomes zero on your end. I couldn't help further unless this is solved.

I can't figure out why it keeps going to zero.
The type of dis_norm was (8x20x642x3) both before and after multiplying the rotation matrix.
The type of the rotation matrix was (8x20x3x3).
What could be the cause of dis_norm being zero, for example?

If the rotation matrix is identity matrix, then there is no reason the output could be zeros. The only possibility I can see is that torch.matmul() function went wrong. For example, the first two dimensions are not treated as batch dimensions.

I'm sorry. I've lost track of the role of this sentence.
dis_norm = dis_norm.matmul(transforms.quaternion_to_matrix(ctl_rs).view(opts.n_hypo,-1,3,3)) # h,j,n,3
I think this sentence is making the rotation, but if I multiply by the identity matrix, the value will not change.
How does the value of dis_norm in (1) and (2) change?
Sorry for asking such a basic question.

ctl_rs is the orientation of gaussians, which is identity matrices at initialization but will change during gradient updates.

I think I can do the rotation with this sentence, but I don't think multiplying the dis_norm in (1) by the identity matrix will change the value.

So this is the conflicting part. The dis_norm in (1) are not zeros, but multiplying rotation matrices turns dis_norm to zeros in (2).  I've never seen this happening.

Thank you.
I understand now. I still don't understand why it goes to zero, but I will continue to work on it.

dis_norm[0,0].matmul(rotmat[0,0]) worked fine, but dis_norm[0].matmul(rotmat[0]) did not.
Therefore, I wrote the code to dis_norm[i,j].matmul(rotmat[i,j]), and used a for loop to prevent dis_norm from going to zero.
I was able to fix the error of dis_norm going to zero, but still when I run bash scripts/spot3.sh, it starts pdb mode and stops working.
How should I improve this?

This is the output that was obtained.
bash scripts/spot3.sh
Jitting Chamfer 3D
Loaded JIT 3D CUDA chamfer distance
1/0
workers: 1
pairs: 1
init:0, end:-1
198 paris of images
Only the mean shape is symmetric!
found mean v
found tex
found ctl rotation
found rest translation
found ctl points
found log ctl
running k-means on cuda:0..
[running kmeans]: 14it [00:00, 340.51it/s, center_shift=0.000000, iteration=15, tol=0.000100]running k-means on cuda:0..
running k-means on cuda:0..
[running kmeans]: 14it [00:00, 356.84it/s, center_shift=0.000000, iteration=15, tol=0.000100]running k-means on cuda:0..
running k-means on cuda:0..
[running kmeans]: 19it [00:00, 358.95it/s, center_shift=0.000000, iteration=20, tol=0.000100]running k-means on cuda:0..
running k-means on cuda:0..
[running kmeans]: 17it [00:00, 357.37it/s, center_shift=0.000000, iteration=18, tol=0.000100]running k-means on cuda:0..
new bone locations
scores:g kmeans]: 18it [00:00, 357.66it/s, center_shift=0.000000, iteration=19, tol=0.000100]
tensor([0., 0., 0., 0., 0., 0., 0., 0.], device='cuda:0')
selecting 0eans]: 16it [00:00, 344.69it/s, center_shift=0.000000, iteration=17, tol=0.000100]
[running kmeans]: 15it [00:00, 39.38it/s, center_shift=0.000000, iteration=15, tol=0.000100]
[running kmeans]: 15it [00:00, 44.24it/s, center_shift=0.000000, iteration=15, tol=0.000100]
[running kmeans]: 20it [00:00, 67.03it/s, center_shift=0.000000, iteration=20, tol=0.000100]
[running kmeans]: 18it [00:00, 73.80it/s, center_shift=0.000000, iteration=18, tol=0.000100]
[running kmeans]: 19it [00:00, 97.66it/s, center_shift=0.000000, iteration=19, tol=0.000100]
[running kmeans]: 17it [00:00, 119.30it/s, center_shift=0.000000, iteration=17, tol=0.000100]
[running kmeans]: 19it [00:00, 201.61it/s, center_shift=0.000000, iteration=19, tol=0.000100]
[running kmeans]: 14it [00:00, 342.31it/s, center_shift=0.000000, iteration=14, tol=0.000100]
/home/shiori/anaconda3/envs/lasr/lib/python3.8/site-packages/kornia/geometry/conversions.py:506: UserWarning: XYZW quaternion coefficient order is deprecated and will be removed after > 0.6. Please use QuaternionCoeffOrder.WXYZ instead.
warnings.warn("XYZW quaternion coefficient order is deprecated and"
mask_loss: tensor(0.1626, device='cuda:0', grad_fn=)
flow_loss: tensor(0.1626, device='cuda:0', grad_fn=)
texture_loss: tensor(0.3304, device='cuda:0', grad_fn=)
deformation_loss_bones: tensor(1.3323, device='cuda:0', grad_fn=)
deformation_loss_skins: tensor(1.3445, device='cuda:0', grad_fn=)
bone_symmetry_loss: tensor(1.3586, device='cuda:0', grad_fn=)
camera_loss: tensor(1.3599, device='cuda:0', grad_fn=)
/home/shiori/anaconda3/envs/lasr/lib/python3.8/site-packages/torch/nn/functional.py:3385: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
warnings.warn("Default grid_sample and affine_grid behavior has changed "
aux_losses: tensor(2.6152, device='cuda:0', grad_fn=)
/home/shiori/lasr/nnutils/train_utils.py:76: RuntimeWarning: invalid value encountered in true_divide
timg = (timg-timg.min())/(timg.max()-timg.min())
mask_loss: tensor(0.1621, device='cuda:0', grad_fn=)
flow_loss: tensor(0.1621, device='cuda:0', grad_fn=)
texture_loss: tensor(nan, device='cuda:0', grad_fn=)
deformation_loss_bones: tensor(nan, device='cuda:0', grad_fn=)
deformation_loss_skins: tensor(nan, device='cuda:0', grad_fn=)
bone_symmetry_loss: tensor(nan, device='cuda:0', grad_fn=)
camera_loss: tensor(nan, device='cuda:0', grad_fn=)
aux_losses: tensor(nan, device='cuda:0', grad_fn=)

/home/shiori/lasr/nnutils/train_utils.py(295)train()
-> self.optimizer.step()
(Pdb)

dis_norm[0,0].matmul(rotmat[0,0]) worked fine, but dis_norm[0].matmul(rotmat[0]) did not.

The fact that torch.matmul() breaks in your environment is surprising. This would lead to errors in multiple places of the code. You may need to fix it in a more fundamental way, instead of fixing it only for this piece of code.
I don't have an idea of why would torch.matmul() break, which needs a lot more involved debugging. The only solution I can think of is to use another machine, or try docker.

When I reproduced the multiplication of rank4 x rank4, I was able to calculate it correctly.
Therefore, I don't think that torch.matmul is broken.
However, I don't know why.
Thank you very much.
"""
import kornia
import torch
opts=8
dis_norm=torch.rand([opts, 20, 642, 3])
a1=torch.zeros(160,1)
a2=torch.zeros(160,1)
a3=torch.zeros(160,1)
b=torch.ones(160,1)
ctl=torch.cat((a1,a2,a3,b),1).reshape(160,4)
dis_norm = dis_norm.matmul(kornia.quaternion_to_rotation_matrix(ctl).view(opts,-1,3,3)) # h,j,n,3
print(dis_norm)
"""

I recreated the environment using docker.
As a result, torch.matmul is now working properly and the code can be executed.
Thanks so much for all your help and advice!
If I have any more questions, I'll be sure to ask.


Add new text facet type

Is your feature request related to a problem or area of OpenRefine? Please describe.
Add a new facet of type Text (as opposed to current "List" facet which is usually used for text values). This is in order to have a facet that facets on textual values while not attempting to automagically transform non-text value cells into text. The new Text facet should list all cell values that are text, and create buckets (with counts) for non-text values
Describe the solution you'd like

[ ] Add support for a new facet type of "Text"
[ ] Change option to "Create custom text facet" in menu to use this new facet (rather than the current use of List facet)
[ ] Convert pre-defined facets to use new Text facet type where appropriate

Describe alternatives you've considered
See discussion in #1957 and #1662 and other linked PRs and issues
Additional context
See discussion in #1957 and #1662 and other linked PRs and issues

Just to note progress. I didn't manage to get as much time as I hoped in the last week to look at this,  but I did get some time to think through the problem. My plan currently is to:

Create a new class for the current ListFacet behaviour which extends the ListFacet class
Create a new class for a new text fact which extends the ListFacet class
Shared code between these will continue to live in the ListFacet class. Where there are differences the new classes will override methods to implement the appropriate behaviour

I may do some further refactoring of the ListFacet class once I've done this initial work, but it is this initial work that needs to be completed to get us to a point where we have the current List Facet behaviour and the new Text Facet behaviour supported with a reasonably minimum amount of code duplication.

In terms of actual progress I've written some extended tests for the current List Facet behaviour to make sure this continues to work in the current way

@ostephens what is your timeline for this? Is your current work pushed on some branch? If so maybe I can help?

@wetneb I've got time to look at this over this weekend - If I don't get the work completed, I'll share a branch with what I've done so we can share the work

Feel free to share your branch if you need a hand on this :)

Hi @wetneb - sorry I've been so slow on this.
You can see the progress (or lack of) I've made so far at https://github.com/ostephens/OpenRefine/tree/list-facet-refactor
I think the following commits are good & sensible steps:
https://github.com/ostephens/OpenRefine/commit/02501db87f21e2094cfa62f903b0bc02c0fe3e09
https://github.com/ostephens/OpenRefine/commit/4ce6769fc5a52fcb6263742fe2c9b384bf040e68
But apart from that I seem to end up going in circles. My original idea was to have an abstract class for a 'list facet' which would support grouping by any of the supported data types then be implemented by different types of list facet including the existing 'list facet' and a new 'text list facet' which would exhibit the new desired behaviour. But each time I look at this I start to think the whole way facets are structured feels not quite right, and end up fluctuating between just doing something v simple but duplicating loads of code, and revisiting the whole structure of the facet code (at which point I have to lie down in a dark room for half an hour).
So - sorry I've not really made progress with this and holding back 3.2 :(

btw code as committed to this branch fails as follows:
[ERROR] Failures:
[ERROR]   TextListFacetTests.testBooleanSelection:340 NullPointer
[ERROR]   TextListFacetTests.testDateSelection:214 NullPointer
[ERROR]   TextListFacetTests.testIntegerSelection:277 NullPointer
[INFO]
[ERROR] Tests run: 597, Failures: 3, Errors: 0, Skipped: 0

@ostephens Would this new text facet type also differentiate between (blank) and (null) values?  Currently the Text Facet treats (null) values as (blank), when it really should not.


String/Number object support

Fix for cases when table data contains String and Number objects (not primitives). In this case typeof returns "object" that causes wrong calculation in MEDIAN and errors when AlaSQL creates copies of objects.
Not sure I found all cases with the issue.

👀

Tank you!


[TRY] Define PHP version in env file.

Context
Allow developers to easily define the PHP version with an env variable.
Summary

Relevant Technical Choices

To-do

User-facing changes

Testing Instructions

QA

[ ] This is a non-user-facing change and requires no QA

This PR can be tested by following these steps:

UAT

[ ] UAT should use the same steps as above.

This PR can be tested by following these steps:

Reviews
Does this PR have a security-related impact?

Does this PR change what data or activity we track or use?

Does this PR have a legal-related impact?

Checklist

[ ] This PR addresses an existing issue and I have linked this PR to it in ZenHub
[ ] I have tested this code to the best of my abilities
[ ] I have verified accessibility to the best of my abilities (docs)
[ ] I have verified i18n and l10n (translation, right-to-left layout) to the best of my abilities
[ ] This PR contains automated tests (unit, integration, and/or e2e) to verify the code works as intended (docs)
[ ] I have added documentation where necessary
[ ] I have added a matching Type: XYZ label to the PR

Fixes #

Running PHP 8.

Running PHP 8.


Switch to new REQUEST_MESSAGE but fall back to deprecated request message commands if required

This implements the logic of requesting messages using the new command as well as the deprecated commands consolidated in the RequestMessage class.
Currently WIP implementing the functionality in REQUEST_MESSAGE. The next step is to use this class consistently everywhere.
Closes #1605, closes #2013.

@JonasVautherin do you mind double checking the last commit please, thanks! Better to get that one right...

It looks good to me! Though I haven't tested it, I only read the commit (carefully, though). Let me know if you need me to test something.

It fixes the system_test failure, so that's enough testing for me :)


Replace CodeListener with RedwoodView.Root

This moves loading/error/ready tracking to live beside the content, rather than as something that side-effects the content.
This is not compatible with the old API and there is no shim.
Closes: https://github.com/cashapp/redwood/issues/2174

[x] CHANGELOG.md's "Unreleased" section has been updated, if applicable.

Yeah it's... Big. I started today but had to leave early. Will finish tomorrow morning.


Updates dependencies to work with Commanded 1.0.0

While looking at using this in a project that uses Commanded 1.0.0, I was encountering clashes on the version of postgrex required 0.14.0 vs 0.15.0.
So I have taken this opportunity to bump all the neccesary versions to bring them all in line.
The tests that currently exist are passing, but if further work is required let me know, I'm approaching this naively as a "simple" version bump as I couldn't see anything code wise that this may trip up over.

Thanks for the pull request @hoodsuphopeshigh. I will publish this to Hex as v1.0.0


Suggestion: cacheDirectory should default to something local

Using the OS tmp directory for Jest's ephemeral generated assets means that most CIs can't reliably cache it and keep test runs consistently fast.
Using a default cacheDirectory like node_modules/.jest-cache would stash away those files in a spot that most (if not all) devs have git-ignored and workspace-ignored by default, but also remain completely cacheable by container-based CI services like Travis.

cacheDirectory is configurable for this reason. We used to store it inside of Jest but it was always a huge issue to people so we changed to a temp folder. Please configure Jest the way you like for your CI :)


Update inside a for loop use old value

My Environment

ArangoDB Version:  3.10.2
Deployment Mode:   Single Server
Deployment Strategy:  Manual Start in Docker
Infrastructure:  own
Operating System:  MacOs Ventura 13.1
Total RAM in your machine: 8gb

Component, Query & Data
Affected feature:

AQL query using web interface
AQL query (if applicable):
for i in 0..2
for element in Prova
Update {_key:element._key, number:element.number-1} in Prova

AQL explain and/or profile (if applicable):

This is just an example to explain the problem. The idea is subtracting the attributes number n times, depending on how many times the cycle is repeated.
Dataset:

This is just an easy example to explain the problem. There is an element in the collection prova, having just number as attribute.
Problem:
The value is not updated correctly and  is subtracted only one times. I think that element.number, inside the cycle,  refer to a cached value, corresponding to the values before the query is started, but i don't know how to manage this situation.

Hi,
AQL is not designed that way. Values can only travel top down, left to right. And with each iteration you're in a fresh state.
You can use subqueries to achieve that.
So you create a subquery, iterate, get a count, then update all documents with the new count.

All clear, many thanks!


estimating MS-DSGE models

Hi Junior,
Is there any example to estimate MS-DSGE in the toolbox? because I haven't found any, it seems that all of them uses calibrated parameters and when I try to use the estimate() function in a MS-DSGE models i get errors "parameter rho_x is not controlled by const", for example. I know that to calculate the likelihood for the estimation is with a different filter than the Kalman filter but I do not know if one has to implement that option in the estimate() function or so.
Thanks a lot and regards.
Pedro.

Hi Pedro,
This just a syntax error. You most likely declared "rho_x" to be governed by some Markov chain and when it came to parameterizing it you forgot to put the name of the Markov chain in parentheses.
For example the code below will issue an error
endogenous X Y Z
parameters alpha beta, gamma
parameters(policy,2) rho_x
model
X=Y*log(Z);
...

parameterization
alpha, 0.7;
beta, 0.99;
gamma, 1;
rho_x, 0.85;

This is because rho_x is declared to be controlled by Markov chain policy but then in the parameterization block, we treat it in the same way as alpha, beta and gamma. The correct way to go about is to rewrite the parameterization block as follows:
parameterization
alpha, 0.7;
beta, 0.99;
gamma, 1;
rho_x(policy,1), 0.85;
rho_x(policy,2), 0.65;

Let me know if this helps,
Cheers,
Junior

Thanks a lot for answered me really fast, unfortunately the problem is not that. That problem of syntaxis was already fixed by me before.
To be specific, what i'm doing is trying to run the code "master.m" in the directory "C:...\RISE_toolbox-master\examples\VariousModels\PeterIreland\NKperspective_JMCB2011" but instead of using the code "nkp.rs" I'm using "nkp_switch.rs" because i want to estimate the MS-DSGE model and the code "master_switch.m" just solves the MS-DSGE model with calibration NOT ESTIMATES IT.
So to sum up I replace the line "m=rise('nkp');" in "master.m" with the line "m=rise('nkp_switch');" and i got that error, what i can infer is that the problem comes from the file "create_parameters.m" because now i should define two priors for the two states of the variable and also for the transition probabilities but the problem is that i don't know which syntax i should use.
Do you have an example which estimates a MS-DSGE NOT CALIBRATES IT? I haven't found any in the RISE toolbox
Thanks a lot and best,
Pedro.

All the errors are shown below:
**_Error using generic_tools.parameter_position_and_regimes/retrieval_engine (line 62)
parameter rho_x is not controlled by const
Error in generic_tools.parameter_position_and_regimes (line 32)
[position(ipar),regime_states{ipar}]=retrieval_engine(pnames{ipar});
Error in generic/setup_calibration (line 82)
[position,regime_states]=generic_tools.parameter_position_and_regimes(pnames,...
Error in dsge/setup_calibration (line 46)
obj=setup_calibration@generic_switch(obj,Calibration);
Error in generic/set/set_one_at_a_time (line 130)
obj=setup_calibration(obj,propval);
Error in generic/set (line 116)
set_one_at_a_time();
Error in dsge/set (line 105)
obj=set@generic_switch(obj,varargin{:});
Error in master (line 14)
m=set(m,'parameters',p);_**

Hi Pedro,
That example was never meant for estimation. I recently added it to answer a question by a user who, I thought, needed to run some historical decomposition.
I you want to see an example with estimation, take a look at \examples\VariousModels\TaoZha\Tutorials\DSGE\Tutorial1
In there you will find examples of estimation of dsge models with regime switching. You will still need to set up your own problem but hopefully this can help you started.
Cheers,
Junior

Hi Junior,
Thanks a lot! That’s what i wanted since the begining but i thought it wasn’t in the toolbox, I should have checked more carefully.
I really appreciate yor help.
Kind regards.
Pedro.

Hi Junior
how do I modify the file \examples\VariousModels\TaoZha\Tutorials\DSGE\Tutorial1 to add exchange rate to the feedback rule

Dear Junior
Where can a find a guide to specifying observation equations for the MS-dsge model. I have to write mine own equations but is becoming more confusing for me.
Best,
Sam

Hi Sam,

How to modify ... to add exchange rate to the feedback rule: Do you have a model with exchange rate? The equations of your model should be the ones that you code up in RISE. You do not just add a new variable in the system without providing an extra equation.

There is no need to specify observation equations. All you have to do is to list the observable variables, RISE will figure out what the measurement equations should look like.

Cheers,
Junior

Thank you will work on it.
Best,
Sam
On Fri, Nov 11, 2016 at 2:31 PM, Junior Maih<EMAIL_ADDRESS>wrote:

Hi Sam,

How to modify ... to add exchange rate to the feedback rule: Do you
have a model with exchange rate? The equations of your model should be the
ones that you code up in RISE. You do not just add a new variable in the
system without providing an extra equation.
2.
There is no need to specify observation equations. All you have to do
is to list the observable variables, RISE will figure out what the
measurement equations should look like.
Cheers,
Junior
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/jmaih/RISE_toolbox/issues/24#issuecomment-259947830,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AVNwKLTobHru6WGT6FLpZ92I3wds3f0-ks5q9GAEgaJpZM4JsUQB
.

Version Name: 6.0.2, Version Code: 124w.wr33I.'ren33sse3t3as3a


Remove validation cookie support code

Removes the cookie support code added as part of #2437, but retains the optional field in STValidation message. This ensures the cookie field will be available in all servers running 1.0 and gives time to refine the implementation of the cookie for monitoring validations.

Jenkins Build Summary
Built from this commit
Built at 20180504 - 17:43:35
Test Results

Build Type
Log
Result
Status

docs
logfile
1 cases, 0 failed, t: 2s
PASS :white_check_mark:

clang.debug.unity
logfile
1021 cases, 0 failed, t: 251s
PASS :white_check_mark:

coverage
logfile
1021 cases, 0 failed, t: 620s
PASS :white_check_mark:

msvc.debug
logfile
1017 cases, 0 failed, t: 561s
PASS :white_check_mark:

clang.debug.unity, PARALLEL_TESTS=false
logfile
1021 cases, 0 failed, t: 486s
PASS :white_check_mark:

clang.debug.nounity
logfile
1019 cases, 0 failed, t: 450s
PASS :white_check_mark:

gcc.debug.unity
logfile
1021 cases, 0 failed, t: 245s
PASS :white_check_mark:

gcc.debug.nounity
logfile
1019 cases, 0 failed, t: 104s
PASS :white_check_mark:

clang.release.unity
logfile
1020 cases, 0 failed, t: 328s
PASS :white_check_mark:

gcc.release.unity
logfile
1020 cases, 0 failed, t: 346s
PASS :white_check_mark:

gcc.debug.unity -Dstatic=true
logfile
1021 cases, 0 failed, t: 245s
PASS :white_check_mark:

msvc.debug, PROJECT_NAME=rippled_classic
logfile
1017 cases, 0 failed, t: 1165s
PASS :white_check_mark:

msvc.release
logfile
1016 cases, 0 failed, t: 505s
PASS :white_check_mark:

Codecov Report

Merging #2506 into develop will decrease coverage by 0.03%.
The diff coverage is 75%.

@@             Coverage Diff             @@
##           develop    #2506      +/-   ##
===========================================
- Coverage    70.69%   70.66%   -0.04%
===========================================
  Files          696      696
  Lines        54440    54404      -36
===========================================
- Hits         38489    38442      -47
- Misses       15951    15962      +11

Impacted Files
Coverage Δ

src/ripple/protocol/impl/Feature.cpp
95.45% <ø> (-0.07%)
:arrow_down:

src/ripple/app/misc/ValidatorKeys.h
100% <ø> (ø)
:arrow_up:

src/ripple/app/misc/impl/ValidatorKeys.cpp
87.87% <ø> (-0.36%)
:arrow_down:

src/ripple/app/consensus/RCLValidations.h
93.1% <ø> (+6%)
:arrow_up:

src/ripple/protocol/STValidation.h
60.6% <ø> (ø)
:arrow_up:

src/ripple/app/consensus/RCLConsensus.h
80% <ø> (ø)
:arrow_up:

src/ripple/protocol/Feature.h
100% <ø> (ø)
:arrow_up:

src/ripple/app/consensus/RCLValidations.cpp
83.01% <0%> (+0.81%)
:arrow_up:

src/ripple/consensus/Validations.h
97.33% <100%> (+0.13%)
:arrow_up:

src/ripple/app/consensus/RCLConsensus.cpp
66.99% <100%> (-0.24%)
:arrow_down:

... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a442d3f...68c33bc. Read the comment docs.

👍  LGTM

:+1: from me as well.

Merged as 6156ff3eb78dc8ee0cada76dde3d4f1f321e8239


Deprecate Legacy APIs and CLIs

The goal of this epic is to deprecate legacy pkg/model package, along with legacy CLIs (e.g. bacalhau list) and APIs (e.g. /api/v1/requester/...) in favour of newer CLIs (e.g. bacalhau job list) and APIs (e.g. /api/v1/orchestrator/...)
### CLI & API Migration
- [ ] https://github.com/bacalhau-project/bacalhau/issues/3837
- [ ] https://github.com/bacalhau-project/bacalhau/issues/4021
- [ ] https://github.com/bacalhau-project/bacalhau/issues/4043
- [ ] https://github.com/bacalhau-project/bacalhau/issues/4042
- [ ] https://github.com/bacalhau-project/bacalhau/issues/4044
- [ ] https://github.com/bacalhau-project/bacalhau/issues/4046
- [ ] https://github.com/bacalhau-project/bacalhau/issues/4045
- [ ] https://github.com/bacalhau-project/bacalhau/issues/4048
- [ ] https://github.com/bacalhau-project/bacalhau/issues/3971
- [ ] https://github.com/bacalhau-project/bacalhau/issues/4047
- [ ] https://github.com/bacalhau-project/bacalhau/issues/4058
- [ ] https://github.com/bacalhau-project/bacalhau/issues/4070
- [ ] https://github.com/bacalhau-project/bacalhau/issues/3832

When the above list is complete we will be able to completely remove the V1 Client, API, and the model package. When those pieces of code are removed this issue is considered complete.
### Misc
- [ ] https://github.com/bacalhau-project/bacalhau/issues/3839
- [ ] https://github.com/bacalhau-project/bacalhau/issues/3863
- [ ] https://github.com/bacalhau-project/bacalhau/issues/3719

Final PR that completes this work was merged via https://github.com/bacalhau-project/bacalhau/pull/4156


created meditation model

created meditation model with id, title, instructor, duration, and description

All gravy!


ctrl + any_key not working

I have used karabiner elements for years, but recently (not sure when it started), (left) ctrl on my macbook pro does not modify the key that is pressed with it anymore.
Although Karabiner Eventviewer clearly recognizes it as "left_control" and the modified key plus "flags:left_control", respectively.
But if I go into e.g. Terminal and want to do ctrl + r for reverse search or ctrl + c for interrupting a running process, it just gives me "r" or "c". Only when turning off Karabiner Elements does it work again, so it is clearly an issue with Karabiner.
Edit:
System: MacOS Sierra 10.12.6, Late 2013 Macbook Pro
Karabiner Elements version: 12.7.0
Only Key I have modified so far is right_alt, but the issue persists even if I remove all modifiers

I have this issue too. no modifier keys (cmd ctrl opt) work with a remapped key.

I think I have a similar issue.
I am trying to remap my end key to ctrl + right arrow to switch desktops.
{"description": "End to right Desktop",
                        "manipulators": [
                            {
                                "from": {
                                    "key_code": "end",
                                    "modifiers": {
                                        "optional": [
                                            "any"
                                        ]
                                    }
                                },
                                "to": [
                                    {
                                        "key_code": "left_control",
                                        "modifiers": [
                                            "right_arrow"
                                        ]
                                    }
                                ],
                                "type": "basic"
                            }
                        ]
                    }

Or is my rule just wrong?
Mac OS Mojave 1.14.6,  Early 2015 MacBook Pro
Karabiner Elements 12.7.0

I solved my problem by going back to an earlier version of my system and resetting all keyboard related settings, then reinstalling karabiner. No idea what it was, though.


Add GitHub badges

Add badges for Travis CI, Maven Central and ALv2.

The Travis Badge will become active once https://issues.apache.org/jira/browse/INFRA-12156has been resolved.
The Maven Central badge will show the latest release version once the first release has been published.

Thank @britter for your contribution, I'm +1.
Nit: There is a typo at commit message tarvis -> travis

@sundapeng fixed the typo. Do we want to merge this now or wait until the Travis Build has been activated by INFRA?

Thank @britter , I think both are okay. How about wait for INFRA, so that the status at github would be right.

@sundapeng okay. Will merge when INFRA-12156 has been resolved.

Travis could work now, committed to master, thank @britter for your contribution.


add xa rpc interface

add xa rpc interface

Codecov Report

Merging #310 into release-rc4 will increase coverage by 0.14%.
The diff coverage is 30.35%.


ABC iView HTTP Error 404: Not found

youtube-dl reports HTTP:404 errors on iView video URLS.  other sites work fine.

can successfully download youtube videos + yahoo news videos (eg, https://au.news.yahoo.com/york-cats-become-first-us-pets-contract-coronavirus-210024387--spt.html and https://au.tv.yahoo.com/person-dresses-t-rex-shops-055357444.html)

Tried all the options I can think of in a number of combinations.
--no c:\users\user1.cache dir found, even hidden or system.
--ignore-config
--deleted c:\users\user1\youtube-dl.conf
--no-cache and --rm-cache-dir
--config-location as a freshly created zero-sized file.
-U update switch
--no-geo-bypass --no-cache-dir

tried SSL switching off certificate checking

overwriting youtube-dl.exe with the latest version downloaded
-g yields same error HTTP:404.

tried installed ffmpeg latest version and testing path calling.

typing python at command prompt. launches a windows store python stub/bootstrapper which apparently was added to the May 10 Windows 10 update https://devblogs.microsoft.com/python/python-in-the-windows-10-may-2019-update/

manually installed python3.8 and successfully tested correct version running by typing python.  This no longer launches the windows store stub.

youtube-dl verbose information reports python v3.4, but the static version I installed is v3.8.2(tested calling path via command line was v3.8.2).  Python was not installed prior to this.  I was using the static version of python, do I need to use the shared object/dll version instead?

--cookies file
--disabled firewall and tested.  socket error reported so this s a different error and ∴ probably not related to firewall.
This used to work seamlessly with iView but something has changed.  Is there a way to get more verbosity so I can troubleshoot which (if any settings have changed)?
C:\TEMP\aBC-IVIEW>youtube-dl --no-check-certificate --no-geo-bypass --ignore-config --no-cache-dir --verbose https://iview.abc.net.au/show/catalyst/series/20/video/SC1802H004S00
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--no-check-certificate', '--no-geo-bypass', '--ignore-config', '--no-cache-dir', '--verbose', 'https://iview.abc.net.au/show/catalyst/series/20/video/SC1802H004S00']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2020.03.24
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.18362
[debug] exe versions: none
[debug] Proxy map: {}
[abc.net.au:iview] SC1802H004S00: Downloading JSON metadata
[abc.net.au:iview] SC1802H004S00: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\extractor\common.py", line 627, in _request_webpage
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\YoutubeDL.py", line 2238, in urlopen
File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default

ffmpeg version git-2020-04-22-2e38c63 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 9.3.1 (GCC) 20200328
configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --disable-w32threads --enable-libmfx --enable-ffnvcodec --enable-cuda-llvm --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf
libavutil      56. 42.102 / 56. 42.102
libavcodec     58. 80.100 / 58. 80.100
libavformat    58. 42.101 / 58. 42.101
libavdevice    58.  9.103 / 58.  9.103
libavfilter     7. 79.100 /  7. 79.100
libswscale      5.  6.101 /  5.  6.101
libswresample   3.  6.100 /  3.  6.100
libpostproc    55.  6.100 / 55.  6.100
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...
Use -h to get full help or, even better, run 'man ffmpeg'

Remove --no-geo-bypass

Similar issue https://github.com/ytdl-org/youtube-dl/issues/24680 - may be related to Python being on the system, given that's where the error appears in the log

Similar issue #24680 - may be related to Python being on the system, given that's where the error appears in the log

Disabled the windows python.exe stub in settings->apps->app execution aliases and uninstalled python.  Same issue.
Also tried both --no-geo-bypass and normal mode.
Perhaps this is related to SSL or some other settings?  Is there a way to get more verbosity/information on specifically what youtube-dl is communicating to see why it is not found?

--print-traffic, --write-pages, and -j would be your best options:
https://github.com/ytdl-org/youtube-dl#verbosity--simulation-options

Looks like the main links are not working.  I tried using with and without -g options for some of the .m3u8 files but am not sure which .m3u8 files to use.
End up getting an error.  I have attached 2 files.
See SC1802H008S00_https_-_iview.abc.net.au_api_programs_catalyst_SC1802H008S00.dump for the main file.
See additional_information for output of what happens when trying grab video using .m3u8 files.
Not sure what else I can try at this point or what else I can test.  Any ideas?
additional_information.txt

I get the same errors using https://iview-vod.akamaized.net/prod/cla/classification_PG_2019/20191126150417/fairplay/master.m3u8
Available streams for https://iview.abc.net.au/show/catalyst/series/20/video/SC1802H004S00:
streams: {mpegdash: {,…}, fairplay: {,…}, hls: {,…}}
fairplay: {,…}
protected: true
sd: "https://iview-vod.akamaized.net/prod/sc1/SC1802H004S00CE1D1/20190219155946/fairplay/master.m3u8"
sd-low: "https://iview-vod.akamaized.net/prod/sc1/SC1802H004S00CE1D1/20190219155946/fairplay/master.m3u8"
hls: {,…}
protected: false
sd: "https://iviewhlsvod-vh.akamaihd.net/i/playback/_definst_/_video/sc1/SC1802H004S00CE1D1_20190219155946_,1500000,1000000,650000,500000,220000,.mp4.csmil/master.m3u8?subtitle_file_pattern=identifier_langcode&subtitle_identifier=SC1802H004S00CE1D1&subtitle_location=%2Fplayback%2Fcaptions%2Fprod%2Fsc"
sd-low: "https://iviewhlsvod-vh.akamaihd.net/i/playback/_definst_/_video/sc1/SC1802H004S00CE1D1_20190219155946_,650000,500000,220000,.mp4.csmil/master.m3u8?subtitle_file_pattern=identifier_langcode&subtitle_identifier=SC1802H004S00CE1D1&subtitle_location=%2Fplayback%2Fcaptions%2Fprod%2Fsc"
mpegdash: {,…}
protected: true
sd: "https://iview-vod.akamaized.net/prod/sc1/SC1802H004S00CE1D1/20190219155946/mpegdash/master.mpd"
sd-low: "https://iview-vod.akamaized.net/prod/sc1/SC1802H004S00CE1D1/20190219155946/mpegdash/master.mpd"

YTDL can only download the unprotected streams. For me, it uses https://iviewhlsvod-vh.akamaihd.net/i/playback/_definst_/_video/sc1/SC1802H004S00CE1D1_20190219155946_,1500000,1000000,650000,500000,220000,.mp4.csmil/master.m3u8 but then adds a different id string each time (reusing an old one fails), e.g.:
https://iviewhlsvod-vh.akamaihd.net/i/playback/_definst_/_video/sc1/SC1802H004S00CE1D1_20190219155946_,1500000,1000000,650000,500000,220000,.mp4.csmil/master.m3u8?null=0&id=AgBSDPQXAhVLHaj5ol7Orzjb1cu1pG7mZHsP7qtbKUoK+NjWQ7CBVnKOsO9VULx6phyj926EFEz+rw%3d%3d

For me this is entirely automated, so I'm not sure why it doesn't do that for you, or how to work around it.
Have you tried - https://github.com/ytdl-org/youtube-dl#developer-instructions
Download, unzip, open cmd in folder, run python -m youtube_dl https://iview.abc.net.au/show/catalyst/series/20/video/SC1802H004S00
Do you have another computer you can try it on? One that hassn't had Python or youtube-dl on it?

Strangely I seem to get two errors and one warning, indicated in bold below:
Is the failed HEAD request causing the 405 and/or 403 errors?  Is there a different mode for this?
Will try a fresh install of Windows 7 in a VM.

C:\TEMP\aBC-IVIEW>youtube-dl --rm-cache-dir --ignore-config --no-geo-bypass https://iviewhlsvod-vh.akamaihd.net/i/playback/definst/video/sc1/SC1802H004S00CE1D1_20190219155946,1500000,1000000,650000,500000,220000,.mp4.csmil/master.m3u8
Removing cache dir C:\Users\bates/.cache\youtube-dl ..
[generic] master: Requesting header
WARNING: Could not send HEAD request to https://iviewhlsvod-vh.akamaihd.net/i/playback/definst/video/sc1/SC1802H004S00CE1D1_20190219155946,1500000,1000000,650000,500000,220000,.mp4.csmil/master.m3u8: HTTP Error 405: Method Not Allowed
[generic] master: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

I get the same warning and errors if I just use https://iviewhlsvod-vh.akamaihd.net/i/playback/_definst_/_video/sc1/SC1802H004S00CE1D1_20190219155946_,1500000,1000000,650000,500000,220000,.mp4.csmil/master.m3u8, it needs the id string after it.
I tried --add-header but nothing I used worked. Also played around with using ffmpeg directly, but to no avail.
Fingers crossed for your VM.

I don't seem to have any trouble with the command as written. youtube-dl version 2020.12.26 on Win 10 machine. Can you confirm that the issue still exists?
PS H:\Downloads> & '.\youtube-dl(1).exe' --no-check-certificate --no-geo-bypass --ignore-config --no-cache-dir --verbose https://iview.abc.net.au/show/catalyst/series/20/video/SC1802H004S00
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--no-check-certificate', '--no-geo-bypass', '--ignore-config', '--no-cache-dir', '--verbose', 'https://iview.abc.net.au/show/catalyst/series/20/video/SC1802H004S00']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2020.12.26
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.18362
[debug] exe versions: ffmpeg git-2020-01-08-5bd0010, ffprobe git-2020-01-08-5bd0010
[debug] Proxy map: {}
[abc.net.au:iview] SC1802H004S00: Downloading JSON metadata
[abc.net.au:iview] SC1802H004S00: Downloading webpage
[abc.net.au:iview] SC1802H004S00: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'http://iviewhls-i.akamaihd.net/i/playback/definst/video/sc1/SC1802H004S00CE1D1_20190219155946,1500000,1000000,650000,500000,220000,.mp4.csmil/index_0_av.m3u8?null=0&id=AgByS2dGF8BstSqS6V+IrDoWNl9z6duGdxXAGsd0KRnECpOEwl9FQ75or2YYfkDo5DdOritYZ7Y8%2fw%3d%3d'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 351
[download] Destination: Series 20 The Mindfulness Experiment-SC1802H004S00.mp4
[download] 100% of 519.98MiB in 03:25
[debug] ffmpeg command line: ffprobe -show_streams "file:Series 20 The Mindfulness Experiment-SC1802H004S00.mp4"
[ffmpeg] Fixing malformed AAC bitstream in "Series 20 The Mindfulness Experiment-SC1802H004S00.mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel "repeat+info" -i "file:Series 20 The Mindfulness Experiment-SC1802H004S00.mp4" -c copy -f mp4 "-bsf:a" aac_adtstoasc "file:Series 20 The Mindfulness Experiment-SC1802H004S00.temp.mp4"


[Feature][FileSystem] Support huawei cloud OBS

Search before asking

[X] I searched in the issues and found nothing similar.

Motivation
Paimon support huawei cloud  obs fileSystem.
Solution
No response
Anything else?
No response
Are you willing to submit a PR?

[X] I'm willing to submit a PR!

The related PR has been closed, I will close this issue.


Recommend storing the config locally

Often developers work on more than one RDE, therefore the recommend way of setting the configuration should affect the current directory only.
Screenshots (if appropriate):
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] I have signed the Adobe Open Source CLA.
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.

Hi @kwin thank you for the improvements to the README. While I think it's good to highlight the -l option, I think we should not make it the default shown as it complicates the commandline (a little) and using it is not a necessity. Whether or not you need -l depends really on your workflow and situation.
Therefore I would leave the example command aio config:set commands as they are but just mention below it, as you do, that there exist a -l flag to store it locally.

@kwin how does #38 look to you as an alternative?


New icons: user-search and user-search-2

What is the purpose of this pull request?

[x] New Icon
[ ] Bug fix
[ ] New Feature
[ ] Documentation update
[ ] Other:

Description
New icons:

user-search
user-search-2

Icon use case

Searching for users
Recruitment

Alternative icon designs

Icon Design Checklist
Concept

[x] I have provided valid use cases for each icon.
[x] I have not added any a brand or logo icon.
[x] I have not used any hate symbols.
[x] I have not included any religious or political imagery.

Author, credits & license

[ ] The icons are solely my own creation.
[ ] The icons were originally created in # by @
[x] I've based them on the following Lucide icons: user-check, user-plus, user-x etc. + corresponding -2 variations, plus search
[ ] I've based them on the following design:

Naming

[x] I've read and followed the naming conventions
[x] I've named icons by what they are rather than their use case.
[x] I've provided meta JSON files in icons/[iconName].json.

Design

[x] I've read and followed the icon design guidelines
[x] I've made sure that the icons look sharp on low DPI displays.
[x] I've made sure that the icons look consistent with the icon set in size, optical volume and density.
[x] I've made sure that the icons are visually centered.
[x] I've correctly optimized all icons to two points of precision.

Before Submitting

[x] I've read the Contribution Guidelines.
[x] I've checked if there was an existing PR that solves the same issue.

https://lucide-site-git-feat-added-edit-page-jguddas.vercel.app/edit?PHN2ZwogIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIKICB3aWR0aD0iMjQiCiAgaGVpZ2h0PSIyNCIKICB2aWV3Qm94PSIwIDAgMjQgMjQiCiAgZmlsbD0ibm9uZSIKICBzdHJva2U9ImN1cnJlbnRDb2xvciIKICBzdHJva2Utd2lkdGg9IjIiCiAgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIgogIHN0cm9rZS1saW5lam9pbj0icm91bmQiCj4KICA8Y2lyY2xlIGN4PSIxMCIgY3k9IjciIHI9IjQiIC8+CiAgPGNpcmNsZSBjeD0iMTciIGN5PSIxNyIgcj0iMyIgLz4KICA8cGF0aCBkPSJtMjEgMjEtMS45LTEuOSIgLz4KICA8cGF0aCBkPSJNMTAuMyAxNUg3YTQgNCAwIDAgMC00IDR2MiIgLz4KPC9zdmc+=

@jguddas that looks good, did that exist somewhere already? missed it if so
had a go at making a similar version for the user-2 style:

updated this PR with the new icons, looks much better, thanks for having a look!

It's based on an existing icon, so it should inherit the contributors.

gotcha, missed that – fixed

IMO adding icons with the same use case but with different styles is something we should avoid, it works against having one consistent feel across all icons.
So 👍 for user-search and 👎 for user-search-2 from me.

Thanks for having a look @jguddas and @karsa-mistmere ❤

I highly recommend creating one PR per icon.

From CONTRIBUTING.md:

If you want submit multiple icons, please separate the icons and group them. That makes reviewing the icons easier and keep the thread clean and scoped. So don't submit multiple icons in one PR that have noting to do with each other. So for example don't create one PR with icons: arrow-up, bicycle, arrow-down. Seperate them by two PRs; 'pr-01' arrow, arrow-down and 'pr-02' bicycle.

As-per these guidelines I think I was correct in making this one PR.

Re the user*-2 naming I'm aware that the Naming conventions advise against -2 for "what makes the alternate unique" but I opted for naming consistency instead – I agree with @karsa-mistmere that it makes more sense than deviating.
I like the sound of user-round-*! Should take into account though that this would be a breaking change for anyone currently using user*-2 icons.

As for what to do with this PR, it looks like the options are:

Keep the user-search-2 name for now
Rename user-search-2 to user-round-search
Remove user-search-2 entirely

Option 2 isn't ideal since it introduces naming inconsistency, and option 3 isn't ideal since it also introduces inconsistency considering every other user-* icon has a -2 variant.
Hoping we can go with option 1 but let me know what you think

We can add backwards compatible aliases for current user-*-2 icons.

We can add backwards compatible aliases for current user-*-2 icons.

Ah, awesome! I'd be happy to have a go at setting that up

We can add backwards compatible aliases for current user-*-2 icons.

Ah, awesome! I'd be happy to have a go at setting that up

If you are up for it, you can give my new renameIcon.sh script a try.
https://github.com/lucide-icons/lucide/pull/1630
How to use it
Make sure you have the gh cli installed.
Don't think this will work on Windows.
gh pr checkout 1630 # go to my PR with the new rename script
./scripts/renameIcon.sh user-2 user-round # run the script
git checkout main # go back to main
git checkout -b rename/user-2-to-user-round # create a new branch
git commit -m 'Renamed user-2 to user-round' # commit the changes
gh pr create --title 'Renamed user-2 to user-round' # create a pull request
git checkout main # go back to main

Done! #1638


Adding "total_count" in "meta" for data collections.

Coverage decreased (-0.2%) to 25.424% when pulling 4c50787590df72c726b9fb241ae3d2825ba23564 on grahamu:add_total_count into b485afd20b26cf8ad19e8d463c3c9df4da6f0154 on pinax:master.

Coverage decreased (-0.1%) to 25.46% when pulling 0e24795f907a56cf3dd3d67f20230180e9d09e90 on grahamu:add_total_count into b485afd20b26cf8ad19e8d463c3c9df4da6f0154 on pinax:master.

Coverage decreased (-0.1%) to 25.46% when pulling 0e24795f907a56cf3dd3d67f20230180e9d09e90 on grahamu:add_total_count into b485afd20b26cf8ad19e8d463c3c9df4da6f0154 on pinax:master.

Coverage decreased (-0.1%) to 25.46% when pulling 07017ed67839a1c45692c33dfeca610330622148 on grahamu:add_total_count into b485afd20b26cf8ad19e8d463c3c9df4da6f0154 on pinax:master.

@brosner this fixes the "meta" dictionary existence issue. I believe meta works as desired and page[size] as well, but forthcoming tests will prove their correctness.

Coverage increased (+10.1%) to 35.704% when pulling 80dc1f6983433761dc7b41374378aa325f84f60c on grahamu:add_total_count into b485afd20b26cf8ad19e8d463c3c9df4da6f0154 on pinax:master.

Yippee @brosner this is ready for review and merging

Coverage increased (+10.1%) to 35.704% when pulling fa0d8dd3dd4c67a4e75b230a0e8057005c30af30 on grahamu:add_total_count into b485afd20b26cf8ad19e8d463c3c9df4da6f0154 on pinax:master.


Create a View utility method for removing a group from a View

In some cases it is useful to remove a group from a View. As Views are unmodifiable we would need to return a new View instance with the modification. This should go in the ViewUtil class.
Something like:
public static View removeGroups(final View view, final String... groups) {
  final LinkedHashMap<String, ViewElementDefinition> entities = new LinkedHashMap<>(view.getEntities());
  final LinkedHashMap<String, ViewElementDefinition> edges = new LinkedHashMap<>(view.getEdges());
  for (String group : groups) {
    entities.remove(group);
    edges.remove(group);
  }

  return new View.Builder()
        .merge(view)
        .entities(entities)
        .edges(edges)
        .build();
}

Merged into develop.


[AIRFLOW-2181] Convert password_auth and test_password_endpoints from DOS to UNIX

JIRA

[x] My PR addresses the following Airflow JIRA

https://issues.apache.org/jira/browse/AIRFLOW-2181

Description

This PR contains no actual changes to the code. It only contains file format changes to password_auth and test_password_endpoints converting from DOS to UNIX.

Tests

I have re-run the unittests and confirmed the password_auth backend still works with this change (using the patch here: https://github.com/apache/incubator-airflow/pull/3048). Travis build/tests pass successfully.

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

[x] Passes git diff upstream/master -u -- "*.py" | flake8 --diff

@dan-sf Please fix your line endings. Now the full file is marked as a diff. Cheers

@Fokko I'm not sure I understand, the files are currently in DOS format in master and this change updates them to UNIX format. Since every line changes git marks the full file diffs, is there a way to force git to only diff line endings?

 Li|

Currently it is 0D 0A, which is Carriage return and a newline. I think this is caused by @NielsZeilemaker his Windows environment. So currently it is DOS\Windows and we want to move to UNIX style.

Thanks @dan-sf


[3.x] Fix physics tick counter

The counter is now incremented at the start of a physics tick rather than the end.
Fixes #92903
Notes

Full details on the problem in the issue.
This also fixes a knock-on potential regression in the Input.is_action_just_pressed() that occurs when the counter is changed.
Some uint counters are initialized to -1 to ensure they register tick 0 as a change.
It turns out there's actually very little in core that depends on the tick count, so it may be less prone to regressions than I initially thought.

Discussion
There are some input considerations for changing the tick increment timing: the pressed_physics_frame of the action is used to determine whether input has just been pressed (i.e. is_action_just_pressed()).
This was previously set to the current physics tick, which was fine providing this was not incremented until after the tick. However, now the tick is incremented at the start of the tick, we have to account for two situations:

Agile input being flushed at the start of the tick
Input coming in during the current physics tick

The key problem with (2), which was already present (but may not have occurred in the wild) is that multithread input coming in part way through a physics tick could have been missed if the _physics_process() to detect it had already run.
This PR sets the action physics tick to the current + 1. This ensures that input coming in within the physics tick will never be missed. As a result of this, the regular agile flushing with flush_buffered_events() is (as before) set to take place before the physics tick begins (especially the tick increment). This ensures that input from flush_buffered_events() is processed as quickly as possibly, on the tick that begins straight after the flush.
Future considerations
I am not absolutely sure that agile flushing should logically best take place outside the physics tick, however that is the position that @RandomShaper added it originally, so to keep this PR as simple as possible and reduce risk of regressions I'm keeping that order for now.

I can't really spot a visual difference before and after this PR though.

Luckily it seems (so far) in engine internals little depends on the tick count being correct, so ideally there will be no change / before after. :+1:
The input is most affected (particularly the just pressed logic), so I want to double check that when I get a spare moment, before moving out of draft.
It's good to fix though because user projects may depend on this, and the tick count is more important to be correct when using physics interpolation, to prevent bugs.

I've had a proper look through the input now, and it seems to make logical sense in the different scenarios.
There is some potential for regressions in the input (particularly slight timing differences), but I think everything should operate as before with the +1 change.
I'm slightly in two minds about whether to merge before 3.6 or leave to 3.7, however it is a bug fix, and quite easy to revert if necessary, so maybe we should go for it. The worst likely problem is slight timing difference in rare circumstance.

I'm slightly in two minds about whether to merge before 3.6 or leave to 3.7

And there's my cue.  Enter Stage Left: ME in the role of THAT GUY.  Looking uncomfortable and clearly not wanting to be THAT GUY, I point to an overhead sign that reads:

Waiting 4 Godot 3.6
 445
DAYS

One one hand there's the unwavering, awe-inspiring, and endlessly amusing "commitment to the bit" that comes with the engine name and not shipping 3.6 for 445 days with no end in sight :)  On the other hand that's 445 days waiting for my own little physics fix to enter an official 3.x build :(
Given this fix is more of an edge case and likely to make no difference to most projects, I favor getting 3.6 out the door and leaving this one to 3.7.  BUT, I don't deny the Chaotic Neutral in me wants this endless 3.6 beta to become a thing of legend whispered around the campfire by my grandchildren's children.
p.s. You personally are amazing.  Everyone still working on 3.x is amazing.  A release this year would be amazing!

Haha, yeah sorry it's taking ages. Things have improved quite a bit lately but there are still review bottlenecks.
There's no right / wrong answers but I'm inclined to think we should test this in the next RC. There's actually just one PR waiting for a review (#92105) before we can make the RC (possibly a couple of small bug fixes too but not necessary).

Let's give this a spin. If there are any input regressions we can revert easily.

Thanks!

@belzecue Your plea for Godot 3.6 sound like a poem. When it comes out, I promise I'll ask ChatGPT to white another one of these ;-)
In the digital vast, where ideas collide,
Godot stands tall, with open source pride.
A beacon of freedom in game design's night,
A tool of power, of might.
Where nodes connect in a dance of fate,
Scenes come alive, opening the gate.
To worlds unimagined, stories untold,
In Godot's hands, they boldly unfold.
No king's ransom to use its might,
For Godot's gift is free as light.
In this engine's glow, developers find,
A community strong, generous, and kind.
So let's craft a tale in pixels and hues,
With Godot's aid, there's nothing to lose.
From the simplest sprite to the grandest scene,
In the realm of Godot, we all are seen.
@lawnjelly  Happy to see we're getting close to the RC. Thanks.

@rburing When I try MRP of #93339 with this PR there is now a new befavior when you click on the "Show NinePatchRect" button when the ball is visible. You can see the NinePatchRect streaking from the left. When the ball spawns no streaks are visible, only when you manually click the button. You can see it much more clearly when you set common/physics_fps=10 and settings/fps/force_fps=0. Here is a video:
https://github.com/godotengine/godot/assets/499286/09a9b722-d67a-4b76-b47d-9311723af30e
The streaks were not visible before this PR.

@matmas This is not related directly to this PR I think.
The reason is because:

when you first call get_global_transform_interpolated(), this sets up client interpolation system on the ball
but the ball at this point has no information on the previous transform of the ball (this is only stored in VisualServer so is not accessible)
so currently the only thing the ball can do is perform a reset on the client interpolated data the first time you call get_global_transform_interpolated()

If you want to do this with a moving start, you would currently have to "prime" the client interpolation by calling get_global_transform_interpolated() on the previous tick. This limitation is mentioned in the classref:

Note: This function creates an interpolation pump on the Spatial the first time it is called, which can respond to physics interpolation resets. If you get problems with "streaking" when initially following a Spatial, be sure to call get_global_transform_interpolated() at least once before resetting the Spatial physics interpolation.

There is no easy way around this unfortunately. We don't want to store previous transforms on every node just in case you start the interpolation pump (as it has performance implications), so the user has to manually specify which nodes will be "followed". Currently you do this by calling get_global_transform_interpolated() at an earlier stage.
I have some ideas about just setting a bool property on the node to indicate the node should always stay primed, which may be a more user friendly solution (but risks overuse).

This is a dev discussion area, but since there is no kudos area, I am chirping this feedback here.
The 3.6-rc1 is very good and the FTI work + bugfixing is especially appreciated. Due to the propagation of the FTI logic down the viewports (fixed in this release), it solved a number of visual glitches in our project and identified places where we had to add reset_physics_interpolation() calls for objects changing position drastically (like global_/position.x += delta_x). That's all. The result is a smooth output on 60 and 120 fps rendering devices (Apple) where previously we had "vision split" at 120 fps - the objects movement was delayed by 1 frame on odd frames, and was on time on even frames. So all in all - terrific work here, especially by @lawnjelly ! Thanks.
We did notice however some new streaking effects like the ones reported by @matmas but since we do not use get_global_transform_interpolated() I believe this streaking is due to some missing reset interpolation calls as explained above.   There is no MRP to report as it happens sporadically and a reset call usually solves it. Testing so far reveals a noticeable, not to say, marvellous improvement.
We have yet to test this extensively on VRR Apple Pro devices where things get trickier, but that's for another day. No bugs to report at this point, which is pretty good news.
Keep up the good work!


Add source control section

This adds a source control section with some helpful resources on working with Git :)

Great idea for a new section. Thank you!


GraphQL Playground works only while db-sync container is stopped

I made fresh install with recent snapshot. After 28-30 hours later, I realized that cardano-db-sync-extended container keeps restarting again and again. It reports Error: validateGenesisDistribution: Expected initial block to have 14505 but got 0. In addition playground is not working. When I stopped cardano-db-sync-extended container and restart graphql, playground start serving. I made test queries, { cardano { tip { number slotNo epoch { number } } } } and { cardanoDbMeta { initialized syncPercentage }} it returns logical responses.
Is there anything wrong? Does GraphQL container depends db-sync?

 Since they depend on each other, you need to bring all of them down. When you bring it back up, it will take about 15 - 30 minutes to start up (depending on IOPS). It will then most likely rollback to the start of the most recent epoch that the db-sync has. Since the rewards are kept in memory, it needs to go back to the start of the epoch to re-calulate them.
After some time (depending on how far back your snapshot was), it can take some time syncing up to the network Tip. It could take an hour, maybe more.


[Bug] Table row is not rendering the example data properly when table widget is dropped on the canvas the first time.

Description

@somangshu
Steps to repro the issue:

Drag and drop Table 1
Do binding with any DB query/API
Now drag and drop Table 2
Observe that Table 2 data is not coming

This happens until the old table is removed