Bootable iso example

I'm trying to create a bootable iso. The gist of it, is I want to recreate this command:
genisoimage \
   -V my-volume \
   -c isolinux/boot.cat \
   -b isolinux/isolinux.bin \
   -no-emul-boot \
   -boot-load-size 4 \
   -boot-info-table \
   -eltorito-alt-boot \
   -e images/efiboot.img \
   -no-emul-boot \
   -o test-iso.iso \
   /tmp/iso-test/

I think the issue is in my finalize options which currently look like this:
options := iso9660.FinalizeOptions{
        VolumeIdentifier: "my-volume",
        RockRidge:        true,
        ElTorito: &iso9660.ElTorito{
                BootCatalog: "isolinux/boot.cat",
                Entries: []*iso9660.ElToritoEntry{
                        &iso9660.ElToritoEntry{
                                Platform:    iso9660.BIOS,
                                Emulation:   iso9660.NoEmulation,
                                BootFile:    "isolinux/isolinux.bin",
                                LoadSegment: 4,
                        },
                        &iso9660.ElToritoEntry{
                                Platform:  iso9660.EFI,
                                Emulation: iso9660.NoEmulation,
                                BootFile:  "images/efiboot.img",
                        },
                },
        },
}

After running my code I get an iso that file shows as bootable but doesn't actually boot in a VM:

isoinfo -i test-iso.iso -d shows:
    Eltorito validation header:
    Hid 1
    Arch 0 (x86)
    ID 'https://github.com/disk'
    Key 55 AA
    Eltorito defaultboot header:
        Bootid 88 (bootable)
        Boot media 0 (No Emulation Boot)
        Load segment 4
        Sys type 0
        Nsect 4C
        Bootoff 0 0

Any hints?

Also, I would be happy to add an example ... once I get this working :sweat_smile:

I assume the isoinfo output from above is from the one generated via this library. What does the output of an iso generated using genisoimage look like?

Also, I would be happy to add an example ... once I get this working 😅

And very happy to take it! 😁

Also, what do you get when running the actual bootable image? What is in /tmp/iso-test/?

Here's the whole output for the image from this library for reference:
CD-ROM is in ISO 9660 format
System id:
Volume id: my-volume
Volume set id:
Publisher id:
Data preparer id: https://github.com/diskfs/go-diskfs
Application id:
Copyright File id:
Abstract File id:
Bibliographic File id:
Volume set size is: 1
Volume set sequence number is: 1
Logical block size is: 2048
Volume size is: 224690
El Torito VD version 1 found, boot catalog is in sector 29
NO Joliet present
Rock Ridge signatures version 1 found
Eltorito validation header:
    Hid 1
    Arch 0 (x86)
    ID 'https://github.com/disk'
    Key 55 AA
    Eltorito defaultboot header:
        Bootid 88 (bootable)
        Boot media 0 (No Emulation Boot)
        Load segment 4
        Sys type 0
        Nsect 4C
        Bootoff 0 0

What does the output of an iso generated using genisoimage look like?

CD-ROM is in ISO 9660 format
System id: LINUX
Volume id: my-image
Volume set id:
Publisher id:
Data preparer id:
Application id: GENISOIMAGE ISO 9660/HFS FILESYSTEM CREATOR (C) 1993 E.YOUNGDALE (C) 1997-2006 J.PEARSON/J.SCHILLING (C) 2006-2007 CDRKIT TEAM
Copyright File id:
Abstract File id:
Bibliographic File id:
Volume set size is: 1
Volume set sequence number is: 1
Logical block size is: 2048
Volume size is: 224840
El Torito VD version 1 found, boot catalog is in sector 30
NO Joliet present
NO Rock Ridge present
Eltorito validation header:
    Hid 1
    Arch 0 (x86)
    ID ''
    Key 55 AA
    Eltorito defaultboot header:
        Bootid 88 (bootable)
        Boot media 0 (No Emulation Boot)
        Load segment 0
        Sys type 0
        Nsect 4
        Bootoff 1F 31

Also, what do you get when running the actual bootable image?

A live OS

What is in /tmp/iso-test/?

$ tree /tmp/iso-test/
/tmp/iso-test/
├── EFI
│   └── redhat
│       └── grub.cfg
├── images
│   ├── efiboot.img
│   ├── ignition.img
│   ├── my_image.img
│   └── pxeboot
│       ├── initrd.img
│       ├── rootfs.img
│       └── vmlinuz
├── isolinux
│   ├── boot.cat
│   ├── boot.msg
│   ├── isolinux.bin
│   ├── isolinux.cfg
│   ├── ldlinux.c32
│   ├── libcom32.c32
│   ├── libutil.c32
│   └── vesamenu.c32
└── zipl.prm

5 directories, 16 files

There isn't a lot of difference between the two, but a few: number of sectors, total size, boot offset, a few.
I would like to try and recreate all of this. As far as I can tell, your isolinux dir is standard. Where is the rest of the directory from? And can you share the entire go program that includes the above? Obviously sanitized if needed. I can recreate it easily, but I might not do exactly what you do.
Heads up, going to be a few days.

Sure, let me clean it up and see if I can publish something.

Just went and made a repo https://github.com/carbonin/iso-edit

Great! I will get to it. Hopefully, it is just a documentation issue, but if it is a real bug, then I will be glad we found it.
It will take a few days; in any case, sitting there looking at hex structures in xxd takes time.

Fantastic, much appreciated!

So I'm not sure if this is contributing to the problem, but I also found that the files extracted are significantly different sizes than the ones I can see when mounting the iso...
Mounted:
14:00:43:~/Source/iso-edit/isos/rhcos$ ls -lR
.:
total 7
dr-xr-xr-x. 3 root root 2048 Oct  9 14:01 EFI
dr-xr-xr-x. 3 root root 2048 Oct  9 14:01 images
dr-xr-xr-x. 2 root root 2048 Oct  9 14:01 isolinux
-r--r--r--. 1 root root  132 Oct  9 13:18 zipl.prm

./EFI:
total 2
dr-xr-xr-x. 2 root root 2048 Oct  9 14:01 redhat

./EFI/redhat:
total 2
-r--r--r--. 1 root root 1061 Oct  9 13:18 grub.cfg

./images:
total 8240
-r--r--r--. 1 root root 8173158 Oct  9 14:01 efiboot.img
-r--r--r--. 1 root root  262144 Oct  9 13:52 ignition.img
dr-xr-xr-x. 2 root root    2048 Oct  9 14:01 pxeboot

./images/pxeboot:
total 887826
-r--r--r--. 1 root root  80239428 Oct  9 14:01 initrd.img
-r--r--r--. 1 root root 819969024 Oct  9 14:01 rootfs.img
-r--r--r--. 1 root root   8924528 Oct  9 13:52 vmlinuz

./isolinux:
total 383
-r--r--r--. 1 root root   2048 Oct  9 14:01 boot.cat
-r--r--r--. 1 root root     58 Oct  9 13:18 boot.msg
-r-xr-xr-x. 1 root root  38912 Oct  9 14:01 isolinux.bin
-r--r--r--. 1 root root   2047 Oct  9 13:18 isolinux.cfg
-r-xr-xr-x. 1 root root 116320 Oct  9 14:01 ldlinux.c32
-r-xr-xr-x. 1 root root 180476 Oct  9 14:01 libcom32.c32
-r-xr-xr-x. 1 root root  23924 Oct  9 14:01 libutil.c32
-r-xr-xr-x. 1 root root  26724 Oct  9 14:01 vesamenu.c32

Extracted (using the code in https://github.com/carbonin/iso-edit):
13:59:58:/tmp$ ls -lR iso-test
iso-test:
total 4
drwxr-xr-x. 3 ncarboni ncarboni  60 Dec 10 13:59 EFI
drwxr-xr-x. 3 ncarboni ncarboni 120 Dec 10 13:59 images
drwxr-xr-x. 2 ncarboni ncarboni 200 Dec 10 13:59 isolinux
-rw-rw-r--. 1 ncarboni ncarboni 132 Dec 10 13:59 zipl.prm

iso-test/EFI:
total 0
drwxr-xr-x. 2 ncarboni ncarboni 60 Dec 10 13:59 redhat

iso-test/EFI/redhat:
total 4
-rw-rw-r--. 1 ncarboni ncarboni 1082 Dec 10 13:59 grub.cfg

iso-test/images:
total 4196
-rw-rw-r--. 1 ncarboni ncarboni 4128768 Dec 10 13:59 efiboot.img
-rw-rw-r--. 1 ncarboni ncarboni  163840 Dec 10 13:59 ignition.img
-rw-rw-r--. 1 ncarboni ncarboni    1056 Dec 10 13:59 my_image.img
drwxr-xr-x. 2 ncarboni ncarboni     100 Dec 10 13:59 pxeboot

iso-test/images/pxeboot:
total 444000
-rw-rw-r--. 1 ncarboni ncarboni  40140800 Dec 10 13:59 initrd.img
-rw-rw-r--. 1 ncarboni ncarboni 410025984 Dec 10 13:59 rootfs.img
-rw-rw-r--. 1 ncarboni ncarboni   4489216 Dec 10 13:59 vmlinuz

iso-test/isolinux:
total 328
-rw-rw-r--. 1 ncarboni ncarboni   2048 Dec 10 13:59 boot.cat
-rw-rw-r--. 1 ncarboni ncarboni     58 Dec 10 13:59 boot.msg
-rw-rw-r--. 1 ncarboni ncarboni  38912 Dec 10 13:59 isolinux.bin
-rw-rw-r--. 1 ncarboni ncarboni   2068 Dec 10 13:59 isolinux.cfg
-rw-rw-r--. 1 ncarboni ncarboni  98304 Dec 10 13:59 ldlinux.c32
-rw-rw-r--. 1 ncarboni ncarboni 131072 Dec 10 13:59 libcom32.c32
-rw-rw-r--. 1 ncarboni ncarboni  23924 Dec 10 13:59 libutil.c32
-rw-rw-r--. 1 ncarboni ncarboni  26724 Dec 10 13:59 vesamenu.c32

Created a separate branch to try to track down the size issue.
https://github.com/carbonin/iso-edit/tree/extract_test
This clearly shows that the size reported by ReadDir is different from what is written.
@deitch should I open a separate bug for this? It might be related to the boot issue, but it might not also...

Opened a PR to fix the file size bug https://github.com/diskfs/go-diskfs/pull/81
Will update if it fixes my entire use case

It seems that #81 does not solve the boot problem I was experiencing originally.

@carbonin trying to get your genisoimage command to work. Which efiboot.img are you using? The rest doesn't matter much, as it isn't used in your genisoimage command, but the efiboot.img is needed (and isn't referenced in your linked repo). We need to recreate this precisely.

It's from this iso https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.6/4.6.1/rhcos-4.6.1-x86_64-live.x86_64.iso
My use case is extracting that iso, making some edits, and repacking it.

Also, did you get a chance to take a look at https://github.com/diskfs/go-diskfs/pull/81?

Managed to get a much simpler version to recreate the issue.
https://github.com/diskfs/isotester

Well, this has been entertaining. Took long enough to dig back into it, but there are two pieces missing. The irony is, I am positive I remember working with these when iso9660 was implemented. Which goes to show how much one must trust their memory.

the boot catalog is not being inserted properly into the filesystem. There are tests for it, so something is missing.
the boot file is not being modified to create the boot information table

No, it is even stranger. The boot catalog is being inserted. My Mac won't see it on mounting, but I can see it when inspecting the directory entry looking at the iso in hex.

Phew, got it. It looks like the boot catalog file is added after the directory size has been calculated. So the directory entry for the parent of wherever the boot catalog goes has a size that stops just short of the actually catalog entry.

This should not be closed yet.

@carbonin can you take another look with the latest? I don't think it is a full fix, but we should see improvement.

Sure, but I think #81 is also very important. Do you want me to open a separate issue for that?

No need to open a separate issue with #81 in place.

After the latest change the boot offset is set (according to isoinfo) but it still doesn't boot.
$ isoinfo -d -i my-rhcos.iso
CD-ROM is in ISO 9660 format
System id:
Volume id: rhcos-46.82.202010091720-0
Volume set id:
Publisher id:
Data preparer id: https://github.com/diskfs/go-diskfs
Application id:
Copyright File id:
Abstract File id:
Bibliographic File id:
Volume set size is: 1
Volume set sequence number is: 1
Logical block size is: 2048
Volume size is: 448259
El Torito VD version 1 found, boot catalog is in sector 27
NO Joliet present
NO Rock Ridge present
Eltorito validation header:
    Hid 1
    Arch 0 (x86)
    ID 'https://github.com/disk'
    Key 55 AA
    Eltorito defaultboot header:
        Bootid 88 (bootable)
        Boot media 0 (No Emulation Boot)
        Load segment 4
        Sys type 0
        Nsect 4C
        Bootoff 6D642 448066

@carbonin I found an error in the example:
                        &iso9660.ElToritoEntry{
                                Platform:    iso9660.BIOS,
                                Emulation:   iso9660.NoEmulation,
                                BootFile:    "isolinux/isolinux.bin",
                                LoadSegment: 4,
                        },

Why are you setting the LoadSegment to 4? This causes the boot catalog to have an entry that says to load segment 4 from the boot file, see https://pdos.csail.mit.edu/6.828/2014/readings/boot-cdrom.pdf , page 10.
Removing this doesn't solve it completely, but a step forward.
See the latest update at https://github.com/diskfs/isotester

Why are you setting the LoadSegment to 4?

No idea :sweat_smile: I thought that was the equivalent of the genisoimage arg -boot-load-size 4

Ah no. That is which segment to load, not how many sectors to load. Something I wrong with that too, still figuring it out.

One more #83 adds it, and I updated the example at https://github.com/diskfs/isotester along with the documentation.
We still have a checksum issue, but a big improvement. Please try it.

We still have a checksum issue, but a big improvement. Please try it.

yup seems about right

I know what it is, too. :-)

@carbonin I think it finally got it. Take a gander at the latest. If that is clean, we can close this, and maybe cut a new patch release.

Just did a quick test and it seems to work! Thanks for all the work @deitch
The only other thing I think I'll need to do is also run isohybrid on the resulting image, but that's likely a separate issue.


SD3 cannot achieve good fitting results on large data sets

I conducted several tests. DAdaptAdam achieved similar fitting results to SD15/SDXL on a dataset consisting of a single image. However, on datasets with hundreds of images, the fitting speed of DAdaptAdam significantly decreased. The figure below shows three loss reduction curves: a small dataset with a single image, a large dataset using Cosine With Restarts, and Cosine..
Not smoothed:

Smoothed(Smooth strength 1):

When using conventional schedulers like Adam, the loss reduction curve is better than with DAdaptAdam. If the final loss with DAdaptAdam is 0.142, Adam can reach 0.135, but this loss is still somewhat high. I am currently testing higher learning rates to observe the trend of the loss, but I seem to observe that the loss is difficult to reduce on large datasets across multiple training scripts, resulting in poor learning performance. Could this be an issue with the SD3 model itself?

Are you finetune LoRA or full model?

Sd-scripts only support full finetune by now

I think you should try to examine the data set

same

hi dude, what's the training parameters you used, do you run into the increasing memory and caused crash finally? i tried to train sd3 use 50k data, and the memory is increasing all the time, finally after the first epoch, it reached the memory line and crashed

hi dude, what's the training parameters you used, do you run into the increasing memory and caused crash finally? i tried to train sd3 use 50k data, and the memory is increasing all the time, finally after the first epoch, it reached the memory line and crashed

accelerate launch --mixed_precision="bf16" --num_processes=1 --num_machines=1 --num_cpu_threads_per_process=2 "/root/sd3_train.py"  --bucket_reso_steps=64 --caption_extension=".txt" --enable_bucket --min_bucket_reso=256 --max_bucket_reso=2048 --learning_rate="3e-6" --loss_type="l2" --lr_scheduler="cosine" --lr_scheduler_num_cycles="1" --max_data_loader_n_workers="0" --max_grad_norm="1" --resolution="1024,1024" --max_train_steps=""  --optimizer_type="AdamW" --output_name="last" --output_dir="/root/181" --pretrained_model_name_or_path="/root/SD3.safetensors" --save_every_n_epochs="1" --save_model_as=safetensors --save_precision="fp16" --train_batch_size="1" --train_data_dir="/root/181/full" --cache_text_encoder_outputs --cache_text_encoder_outputs_to_disk --cache_latents --cache_latents_to_disk --lr_warmup_steps=
Hi, my training was conducted on a 48GB graphics card with a batch size of 1, so I did not encounter any memory issues. You need to use BF16 precision and a batch size of 1. If you still experience memory overflow, you will need to switch to a graphics card with larger memory capacity.
Additionally, how is the trend of your training loss? Could you show me the trend of your loss decrease?

hi dude, what's the training parameters you used, do you run into the increasing memory and caused crash finally? i tried to train sd3 use 50k data, and the memory is increasing all the time, finally after the first epoch, it reached the memory line and crashed

accelerate launch --mixed_precision="bf16" --num_processes=1 --num_machines=1 --num_cpu_threads_per_process=2 "/root/sd3_train.py" --bucket_reso_steps=64 --caption_extension=".txt" --enable_bucket --min_bucket_reso=256 --max_bucket_reso=2048 --learning_rate="3e-6" --loss_type="l2" --lr_scheduler="cosine" --lr_scheduler_num_cycles="1" --max_data_loader_n_workers="0" --max_grad_norm="1" --resolution="1024,1024" --max_train_steps="" --optimizer_type="AdamW" --output_name="last" --output_dir="/root/181" --pretrained_model_name_or_path="/root/SD3.safetensors" --save_every_n_epochs="1" --save_model_as=safetensors --save_precision="fp16" --train_batch_size="1" --train_data_dir="/root/181/full" --cache_text_encoder_outputs --cache_text_encoder_outputs_to_disk --cache_latents --cache_latents_to_disk --lr_warmup_steps=
Hi, my training was conducted on a 48GB graphics card with a batch size of 1, so I did not encounter any memory issues. You need to use BF16 precision and a batch size of 1. If you still experience memory overflow, you will need to switch to a graphics card with larger memory capacity.
Additionally, how is the trend of your training loss? Could you show me the trend of your loss decrease?

oh, i mean the cpu memory not the gpu memory, i didn't run into the OOM error. As for the loss trend, because of the cpu memory problem, i can only train like 2000 steps, the training loss seems decrease normally (like from 0.22 to 0.17)

hi dude, what's the training parameters you used, do you run into the increasing memory and caused crash finally? i tried to train sd3 use 50k data, and the memory is increasing all the time, finally after the first epoch, it reached the memory line and crashed

accelerate launch --mixed_precision="bf16" --num_processes=1 --num_machines=1 --num_cpu_threads_per_process=2 "/root/sd3_train.py" --bucket_reso_steps=64 --caption_extension=".txt" --enable_bucket --min_bucket_reso=256 --max_bucket_reso=2048 --learning_rate="3e-6" --loss_type="l2" --lr_scheduler="cosine" --lr_scheduler_num_cycles="1" --max_data_loader_n_workers="0" --max_grad_norm="1" --resolution="1024,1024" --max_train_steps="" --optimizer_type="AdamW" --output_name="last" --output_dir="/root/181" --pretrained_model_name_or_path="/root/SD3.safetensors" --save_every_n_epochs="1" --save_model_as=safetensors --save_precision="fp16" --train_batch_size="1" --train_data_dir="/root/181/full" --cache_text_encoder_outputs --cache_text_encoder_outputs_to_disk --cache_latents --cache_latents_to_disk --lr_warmup_steps=
Hi, my training was conducted on a 48GB graphics card with a batch size of 1, so I did not encounter any memory issues. You need to use BF16 precision and a batch size of 1. If you still experience memory overflow, you will need to switch to a graphics card with larger memory capacity.
Additionally, how is the trend of your training loss? Could you show me the trend of your loss decrease?

oh, i mean the cpu memory not the gpu memory, i didn't run into the OOM error. As for the loss trend, because of the cpu memory problem, i can only train like 2000 steps, the training loss seems decrease normally (like from 0.22 to 0.17)

My memory is 100GB, so I haven't encountered training failures due to memory increase. It sounds like an out-of-memory issue. You can provide detailed information to Kohya for his assessment.  @kohya-ss
A loss of 0.22 is indeed too high, and even 0.17 is high. Such a decrease doesn't indicate much. I hope you can complete the training soon. I look forward to your results as they may help me, since I am currently unable to obtain sufficiently fitting results on smaller datasets.

Thank you very much, I hope to see your test results as soon as possible.

I fixed a bug the gradients are in bf16 in mixed precision training even without --full_bf16 option. This might improve the training result in mixed precision training.

Thanks for the code update and follow-up on this question, I will complete a test training in a few hours. Although I don't think this will help much, because I have trained in fp32 precision before, and the underfitting phenomenon is also serious.

I fixed a bug the gradients are in bf16 in mixed precision training even without --full_bf16 option. This might improve the training result in mixed precision training.

I have completed the tests, and your updated code does not result in a greater loss reduction compared to the previous training with bf16 precision. I am curious about your tests with larger datasets. Have you encountered any issues with reduced learning effectiveness due to the larger dataset?

I've trained on a 16k dataset, but it seems that mixed precision training is problematic, and my average loss is in the 0.9 range

I've trained on a 16k dataset, but it seems that mixed precision training is problematic, and my average loss is in the 0.09 range

If you don't encounter NaN, kohya shouldn't encounter it either. Can you tell us the learning rate and repeat you used? Maybe kohya just used too high lr? Before today, mixed precision was problematic. Bf16 will cause --full_bf16 to be enabled before today. Also, what is the starting loss of your training?

我已经在 16k 数据集上进行了训练，但混合精度训练似乎存在问题，我的平均损失在 0.09 范围内

如果您没有遇到 NaN，kohya 也不应该遇到它。您能告诉我们您使用的学习率和重复率吗？也许 kohya 只是使用了太高的 lr？在今天之前，混合精度是有问题的。Bf16 将导致在今天之前启用 --full_bf16。另外，您的训练的起始损失是多少？

mixed_precision = "bf16" full_bf16 = false weighting_scheme = "logit_normal" sdpa = true optimizer_type = "AdamW"
learning_rate = 1e-5

At around 2000 steps, the loss is 0.0932, and as you mentioned earlier, the loss remains at 0.09 by the end of the training. The loss curve indeed reflects this, as the loss doesn't seem to decrease significantly.
What is your dataset about? How do the images generated by the model look to you? Do those images represent your dataset well?

At around 2000 steps, the loss is 0.0932, and as you mentioned earlier, the loss remains at 0.09 by the end of the training. The loss curve indeed reflects this, as the loss doesn't seem to decrease significantly.
What is your dataset about? How do the images generated by the model look to you? Do those images represent your dataset well?

Based on my previous experience with fine-tuning sdxl as long as the loss is within the normal range, in reality, the drop in loss does not necessarily represent a fitting of the data, my training data is a real photo

在 2000 步左右，损失为 0.0932，正如您之前提到的，训练结束时损失仍为 0.09。损失曲线确实反映了这一点，因为损失似乎没有显着下降。
你的数据集是什么？模型生成的图像在你看来是什么样的？这些图像能很好地代表你的数据集吗？

My loss went down successfully after repairing mixed accuracy training

My data set is 200 pictures but my loss value cannot be reduced very well

My data set is 200 pictures but my loss value cannot be reduced very well。it from 0.103 to 0.113（after 19hours on4090）

Although I know the current training is difficult to fit, the loss value shouldn't be increasing. What repeat, optimizer, and learning rate are you using?

My data set is 200 pictures but my loss value cannot be reduced very well。it from 0.103 to 0.113（after 19hours on4090）我的数据集是200张图片，但是我的损失值不能很好地降低。它从0.103到0.113（4090上19小时后）

Although I know the current training is difficult to fit, the loss value shouldn't be increasing. What repeat, optimizer, and learning rate are you using?虽然我知道目前的训练很难拟合，但损失值不应该增加。您使用什么重复率、优化器和学习率？

repeat:4
optimizer:PagedAdamW8bit
learning rate:1e-5

My data set is 200 pictures but my loss value cannot be reduced very well。it from 0.103 to 0.113（after 19hours on4090）

Although I know the current training is difficult to fit, the loss value shouldn't be increasing. What repeat, optimizer, and learning rate are you using?

Do you have some suggestions for fine tuning the sd3, thanks a lot

Do you have some suggestions for fine tuning the sd3, thanks a lot

I have never used PagedAdamW8bit, but 1e-5 is not considered high for AdamW series optimizers, so the learning rate is probably not the reason your final loss is increasing.
Assuming your epoch is 10, because I see many people like to use 10 epochs, then your single image learning times are 4x10=40 times. Assuming the batch size is 1, that's a total of 8000 steps. Completing 8000 steps on an A40 48GB GPU only takes 1.57 hours, while your training took 19 hours. Given that the 4090's computation speed is similar to the A40, is the high time consumption due to the 4090's lower memory capacity? This far exceeds the normal training time, so there is an issue.
Can you provide all the parameters you are using so I can help you further diagnose the problem? However, my current guess is that you should test using a GPU with 48GB of memory.

$lr = "1e-5"
$unet_lr = "6e-3"
$text_encoder_lr = "2e-5"
$lr_scheduler = "warmup_stable_decay"

The left is the original image, and the right is the result after 300 epochs.

The left is the original image, and the right is the result after 300 epochs.

Given the very high unet lr you're using, the training results look pretty good! It looks like a good fit, but your loss is going up? That's weird.

The left is the original image, and the right is the result after 300 epochs.

Given the very high unet lr you're using, the training results look pretty good! It looks like a good fit, but your loss is going up? That's weird.

thank you!,but there is still some gap between this and the original image. I am looking for better training methods to start larger-scale fine-tuning.

The left is the original image, and the right is the result after 300 epochs.
I initially thought that your learning rate was way too high and definitely wouldn't work, but after seeing your results, I am a bit hesitant.

However, here are some diagnostics based on conventional parameters. They can help you save a lot of training time and make the process more reasonable.
The learning rate (lr) for model training is divided into two parts: the Unet lr and the Text encoder lr. The lr settings will be overridden by these two. A unet lr of 6e-3 is ridiculously high, and 2e-5 is also an excessively high lr for the Text encoder. A max_train_epoches of 320 is also a ludicrously high total epoch count. Maybe PagedAdamW8bit would work on a 6e-3 lr? I haven't used it so I don't know.
Now you could try to set lr = 1e-5, unet_lr = "1e-5", text_encoder_lr = "1e-6", max_train_epoches = 20, and lr_scheduler = "cosine", optimizer = "AdamW8bit".
For the parameters mentioned above, the lr for the text encoder is usually 1%-100% of the unet lr. The unet lr should be consistent with the overall lr, and 1e-5 is a good starting point. As the amount of data increases, the lr needs to be decreased. The maximum training epochs and repeats determine how many times a single image is learned, and 50-300 times is a good range. Your previous 320x4=1280 times is too exaggerated, causing extreme overfitting of the model. However, due to your excessively high lr, the model did not overfit but instead resulted in the collapse of the unet. For SD15 or SDXL, such lr would result in loss=nan, but even though the SD3 loss did not equal nan, you can still observe that the severely blurred unet cannot produce reasonable images. The lr_scheduler can be set to cosine. cosine_with_restarts requires additional setup, so cosine is fine.
Due to issues with SD3 or current training scripts, using the settings I suggested above, although you will observe a normal decrease in loss, the underfitting will still be quite severe. Please wait for updates from kohya and other developers.

I am testing on a medium sized dataset (20k) and am struggling with NaN. It does not occur on a very small toy dataset, so there must be something wrong. I will continue to investigate.

I use a small size dataset which just have 290 images,but the loss is NaN.I try to use the smaller dataset, the loss become normal


[BUG] changing multiselect medium height

🐞 Bug report
Description
If you change variable --tui-height-m to 40px, multiselect will be 44px anyway.
Reproduction
Works for multiselect with [tuiTextfieldLabelOutside]="true" [expandable]="false".
https://stackblitz.com/edit/angular-ms3ukr
Expected behavior
Changing height through variables should change multiselect height properly.
Versions

OS: [MacOS]
Browser [Chrome 89.0.4389.114 ARM64, Safari 14.0.3 (166<IP_ADDRESS>.7)]
Angular [11.2.3]
Package [<EMAIL_ADDRESS>
Additional context
It happens due to tui-scrollbar height inside tui-input-tag. I can fix it with adding CSS class for tui-scrollbar depending on controller size. If it is OK, I will push an PR.

Hi, @KarimovDev
Unfortunately, there are reasons why we cannot make m-size input less than 44px. You can only make it more than 44px.
But we added support size s for the multi-select (it will be released soon :tada:).
S-size input can  be32px or more.
Use tuiTextfieldSize = s and configure --tui-height-s to desirable height (>= 32px).

Thank you! Will wait the release

@nsbarsukov In my case I've fixed this with
.tags {   max-height: var(--tui-height-m); }


Add a simple system time example

Overview
Add a very simple example for using the Time module from motoko-base.

Ah, I see. The Motoko base in dfx is not in sync with master. Maybe we need to wait the example for the next release. The current Time module is not the interface we want to expose.

Closing this for now since there's a disconnect between the time exposed in dfx and the current motoko-base


feat: add DeepCopy() annotations to enterprise entities

This PR adds annotations to the Enterprise Entities added in previous patches in addition to a newly generated deepcopy output.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thank you for your contribution. Please fill out the following form to claim your contributor swag:
https://github.com/Kong/kong/blob/master/CONTRIBUTING.md#contributor-t-shirt.


Feature/dropdown

👋 Thanks for creating a pull request! 🚀 Checkout the storybook we created for it here.

:tada: This PR is included in version<EMAIL_ADDRESS>:tada:
The release is available on:

npm package (@beta dist-tag) :package::rocket:
GitHub release :package::rocket:

:tada: This PR is included in version<EMAIL_ADDRESS>:tada:
The release is available on:

npm package (@beta dist-tag) :package::rocket:
GitHub release :package::rocket:

:tada: This PR is included in version<EMAIL_ADDRESS>:tada:
The release is available on:

npm package (@beta dist-tag) :package::rocket:
GitHub release :package::rocket:

:tada: This PR is included in version<EMAIL_ADDRESS>:tada:
The release is available on:

npm package (@beta dist-tag) :package::rocket:
GitHub release :package::rocket:


Refactor all standard tdd / bdd tests to not use prebound functions

Mocha does not support prebound functions as in
describe('suite', () => {
   it ('something', () => {});
});

instead one must use unbound functions, as in
describe('suite', function () {
   it ('something', function () {});
});

Also we must check whether we use prebound functions for our object oriented suites...

I think it supports lambda functions, but you loose the this context, it is ok as long as you don't need it.

Yes, that is why. If you are ok with that, then we can leave it as is.

Closing as it is not an issue.


Unhandled top-level exception: "failed to initialize transition frontier: ..."

After installing version
Commit [DIRTY]9652f8ee092ea77e29f5ab49fa0a295e36743e8b on branch HEAD

Had this crash after second coda daemon start.
coda demon output:
2020-12-04 23:08:13 UTC [Info] Daemon ready. Clients can now connect
2020-12-04 23:08:13 UTC [Info] Created GraphQL server at: http://localhost:3085/graphql
2020-12-04 23:08:14 UTC [Info] Coda daemon is connecting
2020-12-04 23:08:14 UTC [Info] Coda daemon is bootstrapping
2020-12-04 23:08:23 UTC [Info] Coda daemon is bootstrapping
2020-12-04 23:08:23 UTC [Info] Getting some extra initial peers to start
2020-12-04 23:08:24 UTC [Info] Already connected to enough peers, start initialization
2020-12-04 23:08:24 UTC [Info] Fast forward successful
2020-12-04 23:08:24 UTC [Fatal] Unhandled top-level exception: $exn
Generating crash report
	exn: {
  "sexp": [
    "Failure",
    "failed to initialize transition frontier: Staged_ledger.of_scan_state_and_ledger : did not connect with snarked ledger hash"
  ],
  "backtrace": [
    "Raised at file \"stdlib.ml\", line 33, characters 17-33",
    "Called from file \"src/deferred1.ml\", line 17, characters 40-45",
    "Called from file \"src/job_queue.ml\" (inlined), line 131, characters 2-5",
    "Called from file \"src/job_queue.ml\", line 171, characters 6-47"
  ]
}
2020-12-04 23:08:24 UTC [Info] Requesting peers for their best tip to do initialization
2020-12-04 23:08:24 UTC [Fatal] Exception when generating crash report: $exn
	exn: {
  "exn_name": "Rocks__Rocks_common.OperationOnInvalidObject",
  "exn": "Rocks__Rocks_common.OperationOnInvalidObject"
}

  ☠  Coda Daemon crashed.
   The Coda Protocol developers would like to know why!

    Please:
      Open an issue:
        <https://github.com/CodaProtocol/coda/issues/new>

      Briefly describe what you were doing and include the last 20 lines from .coda-config/coda.log and then paste the following:
Summary:
{"num_accounts":null,"blockchain_length":null,"highest_block_length_received":1,"uptime_secs":55,"ledger_merkle_root":null,"state_hash":null,"chain_id":"ef95774fb2ed657da61bfc40a8a148a9c9a202476b539aaf915e6cd81d1ef268","commit_id":"[DIRTY]9652f8ee092ea77e29f5ab49fa0a295e36743e8b","conf_dir":"/home/greg/.coda-config","peers":["<IP_ADDRESS>:10516","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:10509","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:10502","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:10512","<IP_ADDRESS>:10514","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:10507","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:10504","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:10001","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302","<IP_ADDRESS>:8302"],"user_commands_sent":0,"snark_worker":null,"snark_work_fee":25000000,"sync_status":"Bootstrap","block_production_keys":["B62qkBCmzSHcALfdnPhnYKheN32rQ8FPtgyAWUpqbzgL6C8M1jFJmVF"],"histograms":{"rpc_timings":{"get_staged_ledger_aux":{"dispatch":null,"impl":null},"answer_sync_ledger_query":{"dispatch":null,"impl":null},"get_ancestry":{"dispatch":null,"impl":null},"get_transition_chain_proof":{"dispatch":null,"impl":null},"get_transition_chain":{"dispatch":null,"impl":null}},"external_transition_latency":null,"accepted_transition_local_latency":null,"accepted_transition_remote_latency":null,"snark_worker_transition_time":null,"snark_worker_merge_time":null},"consensus_time_best_tip":null,"next_block_production":null,"consensus_time_now":{"slot_number":"7999","slots_per_epoch":"7140"},"consensus_mechanism":"proof_of_stake","consensus_configuration":{"delta":0,"k":290,"slots_per_epoch":7140,"slot_duration":180000,"epoch_duration":1285200000,"genesis_state_timestamp":"1605683480000","acceptable_network_delay":180000},"addrs_and_ports":{"external_ip":"<IP_ADDRESS>","bind_ip":"<IP_ADDRESS>","peer":{"host":"<IP_ADDRESS>","libp2p_port":8302,"peer_id":"12D3KooWAdo1R53rSbMXTXxDXEohuXdyV44qrv4PKK2g6oHWD8ZG"},"libp2p_port":8302,"client_port":8301}}
Status:
{"OS_type":"Unix","Release":"4.9.0-8-amd64","Machine":"x86_64","Sys_name":"Linux","Exception":{"sexp":["Failure","failed to initialize transition frontier: Staged_ledger.of_scan_state_and_ledger : did not connect with snarked ledger hash"],"backtrace":["Raised at file \"stdlib.ml\", line 33, characters 17-33","Called from file \"src/deferred1.ml\", line 17, characters 40-45","Called from file \"src/job_queue.ml\" (inlined), line 131, characters 2-5","Called from file \"src/job_queue.ml\", line 171, characters 6-47"]},"Command":"Command: (coda daemon -peer-list-file /home/greg/peers.txt -generate-genesis-proof\n true -block-producer-key /home/greg/keys/my-wallet -block-producer-password\n zbos)","Coda_branch":"HEAD","Coda_commit":"[DIRTY]9652f8ee092ea77e29f5ab49fa0a295e36743e8b"}

Closing this issue as it is stale for a long time.


[SUGGESTION] Don't render data-turbo-permanent elements + solution

So I've been using Turbo and I love it. But I've been struggling with some basic things. For example I have a live chat that appears on every page and it has to load all the messages and their user relation (2 querys). I've added a "data-turbo-permanent" attribute (and an ID) to the live chat element and it works, but the server still has to render those elements.
My solution was to add a header to every Turbo request by searching for every "data-turbo-permanent" attribute on the page
// app.js
document.addEventListener('turbo:before-fetch-request', (e) => {
    let permanentElements = []
    document.querySelectorAll('[data-turbo-permanent]').forEach(function(el) {
        permanentElements.push(el.id)
    })
    e.detail.fetchOptions.headers['Turbo-Permanent'] = JSON.stringify(permanentElements)
});

Then in my blade templates I check if this header exists and return nothing if it does
// app.blade.php

    // Check if header exists && has the "live-chat" id
    @if(!(request()->hasHeader('Turbo-Permanent') && in_array('live-chat', json_decode(request()->header('Turbo-Permanent')))))
        // My life chat html & logic
    @endif

If I visit the page the first time or visit the page with Turbo and the element appears the first time, it gets rendered. And if I visit another page with Turbo and the element still exists, it doesn't get rendered and the response has an empty
 so Turbo knows not to delete the element, like this:
// response
...

    ...

    ...

...

I bet this could be programmed in a much better way without having to check the request in the blade template and also the render method also needs the request checking logic which I didn't include here.

I'd also have a solution for persisting scroll positions with a custom "data-turbo-scroll" attribute
// app.js
Turbo['scrollPositionsToSave'] = []
document.addEventListener('turbo:before-visit', function () {
    let elements = document.querySelectorAll('[data-turbo-scroll]')
    elements.forEach(function(el) {
        if (el.scrollTop) {
            Turbo['scrollPositionsToSave'].push({ element: el,  scrollPosition : el.scrollTop })
        }
    })
}, false)
document.addEventListener('turbo:load', function () {
    Turbo['scrollPositionsToSave'].forEach(function(el) { el.element.scrollTo(0, el.scrollPosition) })
    Turbo['scrollPositionsToSave'] = [];
}, false)

For the chat example, I think you could have a Turbo Frame inside the data-turbo-permanent element and load its contents after the page load (not using lazy loading turbo frames, manually triggering a form or something)

  <form
    data-turbo-frame="chatroom"
    action="{{ route('chatroom.index') }}"
    method="GET"
    class="hidden"
    x-data
    x-ref="loadRoomsForm"
    x-init="$refs.loadRoomsForm.submit()"
  >
    <button type="submit">Load</button>
  </form>

  <turbo-frame id="chatroom">

Loading...

  </turbo-frame>

Subsequent requests would contain the turbo permanent element, but its contents would only load the first time the form enters the DOM.

(maybe event just a hidden link instead of the form?)

@tonysm I'm going to try that, thanks. To be honest I didn't understand any of those  or  elements.

@tonysm It's not really what I'm looking for, the chat room has to be loaded from the start, without user input and without lazy loading. It also has to persist on every page. The "data-turbo-permanent" attribute is working like it should, but the server doesn't know that and still sends the data and does all the queries. With my method it would be possible to persist elements without rendering them multiple times and with little to no effort.

Hey, I don't think we should implement something custom in the package that is not part of the Turbo.js or Turbo Rails source (unless something like this already exists there?).
Consider bringing up this issue on the Hotwire Discuss forum, please. Others might have a solution for this that feels more like a 1st-party.

I understand. This is just more or less laravel specific.

It sounds like it's more generic. I can totally see the same problem occurring in Rails. But there they use fragment caching, so this is maybe the only difference?


Revisiting #3815 (Update Racket color)

In #3815, I wanted to update the Racket color to match the blue color used in the logo. Unfortunately, Go was very close in the colorspace. In #4331, Go was changed to a lighter blue. Now, with a minor tweak, the Racket color can be successfully updated from #22228f to #3c5caa.
This won't turn out like the #4319 (Rust) debacle for the following reasons:

The blue color is official, not just a random opinion.
We are going from a dark and not easily visible shade of blue to a lighter shade of the same color.
Racket is not as popular as Rust, so an army of Reddit users will not show up armed with negative emojis.

Of course, if a Racket developer shows up and explains why this is a disaster, it will be easy enough to revert. But I think a nicer color will help Racket, an underappreciated Lisp dialect, look more attractive.

Anywhere we could address the Racket community at large to get their opinion (e.g., mailing list)?

Yes, that's a good idea. I've mailed the users list: https://groups.google.com/forum/#!topic/racket-users/suv7Kv7qUBw

From an email sent directly to me 😢 :

Is there a reason the changes have not been merged?

Because I've not had the time to do it :-D - Linguist is very low on my list of priorities. And when I approve and merge a PR isn't really much of an issue as changes won't appear on GitHub until such time as I make a release, which in itself needs waaay more time than you'd think.
As an aside, it's not particularly nice emailing maintainers directly asking about an issue when you could have simply commented on the issue.

Sorry about that. Thanks Colin.


cert-manager 1.4.0

Thanks submitting your Operator. Please check below list before you create your Pull Request.
New Submissions

[x] Are you familiar with our contribution guidelines?
[x] Have you packaged and deployed your Operator for Operator Framework?

[ ] Have you tested your Operator in all supported installation modes?

[x] Is your submission signed?
[x] Is operator icon set?

Updates to existing Operators

[x] Did you create a ci.yaml file according to the update instructions?
[x] Is your new CSV pointing to the previous version with the replaces property if you chose replaces-mode via the updateGraph property in ci.yaml?
[x] Is your new CSV referenced in the appropriate channel defined in the package.yaml or annotations.yaml ?
[x] Have you tested an update to your Operator when deployed via OLM?
[x] Is your submission signed?

Your submission should not

[x] Modify more than one operator
[x] Modify an Operator you don't own
[x] Rename an operator - please remove and add with a different name instead
[x] Submit operators to both upstream-community-operators and community-operators at once
[x] Modify any files outside the above mentioned folders
[x] Contain more than one commit. Please squash your commits.

Operator Description must contain (in order)

[x] Description about the managed Application and where to find more information
[x] Features and capabilities of your Operator and how to use it
[x] Any manual steps about potential pre-requisites for using your Operator

Operator Metadata should contain

[x] Human readable name and 1-liner description about your Operator
[x] Valid category name1
[x] One of the pre-defined capability levels2
[x] Links to the maintainer, source code and documentation
[ ] Example templates for all Custom Resource Definitions intended to be used
[x] A quadratic logo

Remember that you can preview your CSV here.
--
1 If you feel your Operator does not fit any of the pre-defined categories, file an issue against this repo and explain your need
2 For more information see here

Not ready for review. This is a draft PR. I'm still working through the PR checklist and attempting to test locally.

While testing on a local Kind cluster I see an error during the bundle unpacking
kubectl -n olm  logs 69eb921e5868a606ccbf4c588b0af829f838c007446ae623e255748bdb77h5h
time="2021-06-17T10:19:02Z" level=info msg="Using in-cluster kube client config"
time="2021-06-17T10:19:02Z" level=info msg="Reading file" file=/bundle/manifests/acme.cert-manager.io_challenges.yaml
time="2021-06-17T10:19:02Z" level=info msg="Reading file" file=/bundle/manifests/acme.cert-manager.io_orders.yaml
time="2021-06-17T10:19:02Z" level=info msg="Reading file" file=/bundle/manifests/cert-manager-cainjector_v1_serviceaccount.yaml
time="2021-06-17T10:19:02Z" level=info msg="Reading file" file=/bundle/manifests/cert-manager-edit_rbac.authorization.k8s.io_v1_clusterrole.yaml
time="2021-06-17T10:19:02Z" level=info msg="Reading file" file=/bundle/manifests/cert-manager-view_rbac.authorization.k8s.io_v1_clusterrole.yaml
time="2021-06-17T10:19:02Z" level=info msg="Reading file" file=/bundle/manifests/cert-manager-webhook_v1_service.yaml
time="2021-06-17T10:19:02Z" level=info msg="Reading file" file=/bundle/manifests/cert-manager-webhook_v1_serviceaccount.yaml
time="2021-06-17T10:19:02Z" level=info msg="Reading file" file=/bundle/manifests/cert-manager.clusterserviceversion.yaml
time="2021-06-17T10:19:02Z" level=info msg="Reading file" file=/bundle/manifests/cert-manager.io_certificaterequests.yaml
time="2021-06-17T10:19:02Z" level=info msg="Reading file" file=/bundle/manifests/cert-manager.io_certificates.yaml
time="2021-06-17T10:19:02Z" level=info msg="Reading file" file=/bundle/manifests/cert-manager.io_clusterissuers.yaml
time="2021-06-17T10:19:02Z" level=error msg="File with size 692874 exceeded 1048576 limit, aboring" file=/bundle/manifests/cert-manager.io_clusterissuers.yaml
Error: error loading manifests from directory: file cert-manager.io_clusterissuers.yaml bigger than total allowed limit
Usage:
  opm alpha bundle extract [flags]

Flags:
  -c, --configmapname string   name of configmap to write bundle data
  -l, --datalimit uint         maximum limit in bytes for total bundle data (default 1048576)
      --debug                  enable debug logging
  -h, --help                   help for extract
  -k, --kubeconfig string      absolute path to kubeconfig file
  -m, --manifestsdir string    path to directory containing manifests (default "/")
  -n, --namespace string       namespace to write configmap data (default "openshift-operator-lifecycle-manager")

Global Flags:
      --skip-tls   skip TLS certificate verification for container image registries while pulling bundles or index

Same problem is described in https://github.com/operator-framework/operator-lifecycle-manager/issues/1523 and a solution is described in https://github.com/operator-framework/enhancements/pull/40

I've worked around the bundle size issue and moved all the make scripts into our own repo in https://github.com/jetstack/cert-manager-olm/pull/26, but I'm now trying to work around some problems:

with the -service suffix that OLM adds to the  DNS name of the cert-manager webhook service  OR
trying to get the cert-manager webhook to use the OLM supplied serving TLS certificate rather than the one it uses by default.

richard   release-1.5  ~  projects  cert-manager  cert-manager  1   ./devel/run-e2e.sh --ginkgo.focus '[Conformance].*SelfSigned Issuer'
...
        Internal error occurred: failed calling webhook "mutate.webhooks.cert-manager.io": Post "https://cert-manager-webhook-service.operators.svc:443/mutate?timeout=10s": x509: certificate is valid for cert-manager-webhook, cert-manager-webhook.cert-manager, cert-manager-webhook.cert-manager.svc, not cert-manager-webhook-service.operators.svc
    occurred

    test/e2e/suite/conformance/certificates/selfsigned/selfsigned.go:54
------------------------------
STEP: Retrieving logs for global addons
STEP: Cleaning up the provisioned globals

I fixed the webhook problem  in https://github.com/operator-framework/community-operators/compare/d3ff3db4bacdeae7cefd17b74001c6bd074cd1ec..f72a400c50d09cc4e38cc93359e846f336b1758e and now able to run a selection of the cert-manager conformance tests after installing it with OLM.
I've also tested an upgrade from a 1.3.1 bundle which I generated locally in https://github.com/jetstack/cert-manager-olm/pull/26
$ kubectl operator install cert-manager -c stable -v 1.3.1 --create-operator-group -n operators
subscription "cert-manager" created
operator "cert-manager" installed; installed csv is "cert-manager.v1.3.1"

kubectl operator upgrade  cert-manager  -n operators
operator "cert-manager" upgraded; installed csv is "cert-manager.v1.4.0"

kubectl get csv
NAME                  DISPLAY        VERSION   REPLACES              PHASE
cert-manager.v1.4.0   cert-manager   1.4.0     cert-manager.v1.3.1   Succeeded

/merge possible

/merge possible

/merge possible

@wallrj since addReviewers: true is not critical i am merging it. You can fix it next PR


julia dataframe changing one cell changes entire row

Consider the following code, where I want to change only 1 cell, but the entire row gets changed:
df=DataFrames.DataFrame(A=[1,2],B=[3,4])

df[2,:A]=7 # this is OK, changes only 1 cell

df[:,1:end]=0.0 # this line somehow makes the change in the next line behave unexpectedly

df[2,:A]=7 # entire 2nd row is 7

It's as if the df[:,1:end]=0.0 sets all the cells of that row to the same reference; but I am setting it to 0.0, so I expect this to be a value copy, not reference copy
Versions: julia version 0.4.6-pre DataFrames v"0.7.8"
Btw, I first put this on stackoverflow: http://stackoverflow.com/questions/39320783/julia-dataframe-changing-one-cell-changes-entire-row

Thanks for the report. Indeed, df[:,1:end]=0.0 makes both columns share the same underlying array: after runing it, df[:A] === df[:B]. This also happens with df[1:end]=0.0, which that method calls under the hood. It's kind of worrying nobody noticed this bug before.
Looks like we just need a copy call in the latter method when calling insert_single_column!.

BTW, please post the link to this issue on StackOverflow. In general, it's a good idea to post only in one place, and cross-post only if you don't get a reply after a few days; and bug reports better go to GitHub than SO.

You are right, I thought of github second after stackoverflow, else I would have only posted here


A/B Button

Sometimes I like to compare models that are not in sequential order or the same folder, therefore the arrows do not suffice.
It would be great to have an "A/B" or "Last Model" button that goes back to the model previously loaded.

A workaround would be to copy the two into a temporary folder.
This is going to stay as low-priority for now (and might slip into a won't-do)--I can see potential complexity in this that makes it possible this would be a bit of trouble for what it's worth.

This will be very good addition!


Increase code coverage of stream_data.js

Covered Function exports.name_in_home_view, and exports.notification_in_home_view

You should have stream_data.name_in_home_view and so on in the commit messages.

LGTM! @lonerz can you check and approve?


Validate project name

If the wrong project name is used, this is all we have to go on:
if project_list:
    UnboundLocalError: local variable 'project_list' referenced before assignment

Fixed here :https://github.com/anteater/anteater/commit/d2c1af2300895e3eefc35e97febc0b1f66aa535c


Support custom rewrites / lemmas

Allow users to define their own rewrite rules or prove lemmas, which they can apply in the proofs. This requires more thinking and is not urgent, but I am opening this issue as I think this is an important feature for a realistic use-case of zxlive.

So I agree that something like this would be great to have. I see two different ways we could go about this.
One is to allow people to write custom python scripts that implement a matcher and a rewriter in just the way that pyzx does it. We could for instance make it that people put these in a standard location and that they are then automatically found. Or that people manually import these custom rewrites.
Two is to have an interface inside of ZXLive where people define a left-hand side and a right-hand side, and we use some kind of subgraph isomorphism matching to find out where to match it. The question here is how general we make this. I at least vote no bang-boxes.

I vote for the second option. We should start by making it not general at all. To apply a rewrite, the user has to select the exact nodes of the LHS of the rewrite. Then, we simply need to check if selection matches the LHS exactly, which is the graph isomorphism problem. In practice, the diagrams can be arbitrary large but rewrites usually don't grow as much, so this will not slow down zxlive.


Remove unnecessary code from test

Jenkins, test this please.

Jenkins CI test (for commit 79a00b9effea1e848482161a6a007c1926a7635e, target branch master) failed with status FAILURE.
(For contributors, please wait until the reviewer confirms the details of the error.)

Could you perform "rebase master" ?
When I ran unittests, I got the errors.
It was solved by https://github.com/chainer/chainer/pull/6636


CI: Add MacOS and Windows build flows

See this advice from @mcevoypeter

https://github.com/urbit/new-mars/pull/31#discussion_r1120361134
tl;dr no windows


Add more info about FirebaseAppConfig in docs

Could you add more information on where exactly do we find values for FirebaseAppConfig? I've looked at Firebase Console, but I'm not sure what goes where...

apiKey - Where do I get that?
authDomain - Is this OAuth redirect domains value? From here: https://console.firebase.google.com/project/<PROJECT_NAME>/authentication/providers? There are two Authorized domains here: localhost and <PROJECT_NAME>.firebaseapp.com. Do I use localhost for development and change to the other one in production?
databaseURL - This is https://<PROJECT_NAME>.firebaseio.com/? From https://console.firebase.google.com/project/<PROJECT_NAME>/database/data?
storageBucket - gs://<PROJECT_NAME>.appspot.com from https://console.firebase.google.com/project/<PROJECT_NAME>/storage/files?

I think some additional info about getting these values should be in Installation and Setup #Bootstrap...

OK, now I feel silly (: There's a WEB SETUP button in the Authentication section of the console at:
https://console.firebase.google.com/project/<PROJECT_NAME>/authentication/users, that opens a popup with all values for FirebaseAppConfig...
Maybe mention that in docs?

don't feel silly, I was just trying to figure this out too until I found this post, thanks :)

many thanks for selfansered 👍

Reopening so we can review the docs and see if there's an appropriate fix here.


[cleanup][build] Remove unused docker build scripts

From the current build and release guide I think the source of related logics is now under:
$PULSAR_HOME/docker/build.sh
$PULSAR_HOME/docker/publish.sh

[x] doc-not-needed

cc @nodece @lhotari @codelipenghui

Sorry I made the mistake that these files are for pulsar-build, not the official pulsar/pulsar-all images.


✨ updated transport controller code

Summary
added code to transport controller
Related issue(s)
Fixes #

/cc @vMaroon
/cc @shmuelk


Ensure all tests pass on Windows

This fixes #88.
Introduced AppVeyor CI for Windows builds on the latest Elixir/Erlang. This is just as piece of mind to ensure that we're not missing test issues on Windows.
Incidentally, fixes a couple of things causes tests to fail due to the BEAM impl on Windows - nothing serious.

Coverage remained the same at 100.0% when pulling bcd45fff0f5da283b3b7b205adb0fe7deafdcafd on issue-88-2 into 9b4255e9f0716589bb8819f888d707a85f88d678 on master.

Coverage remained the same at 100.0% when pulling a138e27af09f438e736631e2ad88f67f1fb19a75 on issue-88-2 into 9b4255e9f0716589bb8819f888d707a85f88d678 on master.

Coverage remained the same at 100.0% when pulling bf68b327e858df4b794dc8f073fa42adc36e16be on issue-88-2 into 9b4255e9f0716589bb8819f888d707a85f88d678 on master.


Sources for Wolfmax4k torrents

Options to improve Torrentio access to Wolfmax4k torrents:
wolfmax4k - BTDigg DHT Search Engine https://btdig.com/search?order=2&q=wolfmax4k
Cajon de Torrents V4 https://t.me/+K06LBYuA0wNkZTRk

Btdigg added cf captcha, so it's much harder to scrape

Btdigg added cf captcha, so it's much harder to scrape

hello, I just want to ask if you can publicly share the method used to overcome the cf captcha of btdigg so that other addons similar to torrentio have access to wolfmax4k torrents again
please and thank you

Options to improve Torrentio access to Wolfmax4k torrents:
wolfmax4k - BTDigg DHT Search Engine https://btdig.com/search?order=2&q=wolfmax4k
Cajon de Torrents V4 https://t.me/+K06LBYuA0wNkZTRk

Cajon de Torrents 202410 (V5) https://t.me/+lIz5M_1uzhxlY2Nk


[Performance] Onyx.clear() takes a long time and can leave stale data in storage

What performance issue do we need to solve?
When Onyx.clear() takes a long time to work, and if the browser is refreshed or closed during the process, it can leave stale data in storage.
What is the impact of this on end-users?
The next user that signs in can then see data from the previous user.
List any benchmarks that show the severity of the issue
TBD
Proposed solution (if any)
Provide a better mechanism to ensure Onyx is cleared fully before redirecting users to the sign out page. Possibly put something in place that prevents the browser from being refreshed while clear() is happening. Also possibly speed up the time it takes for Onyx.clear() to work.
List any benchmarks after implementing the changes to show impacts of the proposed solution (if any)
TBD
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[x] Android / Chrome
[ ] iOS / native
[x] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

Previous mentions of bugs caused by this:

https://github.com/Expensify/App/issues/11671
https://github.com/Expensify/App/issues/11671#issuecomment-1302761477
https://github.com/Expensify/App/issues/11901
https://github.com/Expensify/App/issues/12055

cc @fedirjh

Note to self: there is a plugin for LocalForage which provides multiRemove() functionality that could be quicker than the for loop we are using now to call removeItem() for each key needing removed.

I'm going to move this to be weekly since I need to do some performance testing on it and we've got a busy week in the office.

@tgolen I've captured some metrics for localForage methods : setItem , removeItem and clear , results show :

Very huge difference between Chrome and Safari
Clear method is faster than multiple removeItem
removeItem and setItem  are so slow on chrome

note: These tests are related to localForage and not Onyx , this to prove that performance issue is related to localForage and not Onyx

Method
Records
Chrome
Safari

setItem
1K
4926.37 ms
142.10 ms

removeItem
1K
4858.75 ms
155.44 ms

clear
1K
15.21 ms
0.79 ms

setItem
4K
20054.76 ms
608.83 ms

removeItem
4K
19988.10 ms
444.84 ms

clear.
4K
38.94 ms
2.07 ms

Problem
From above results we can tell that Onyx.clear performance issue on chrome is due to removeItem which is taking much time to clear storage . Users usually close the browser tab right after logout (when they see the login page).
If user refresh or close the tab after logout,  in this case Onyx.clear will be interrupted and will cause data persist in storage causing above issues.
Solution
We can use localForage.clear to clear data instead of using multi removeItem , I did checked Onyx implementation and found that we have this limitation
    // If we just call Storage.clear other tabs will have no idea which keys were available previously
    // so that they can call keysChanged for them. That's why we iterate and remove keys one by one
    this.clear = () => Storage.getAllKeys()
        .then(keys => _.map(keys, key => this.removeItem(key)))
        .then(tasks => Promise.all(tasks));

@kidroca is this the only limitation for using localForage.clear directly , or there are other limitations ? if this is the only one I can provide another solution for this limitation and we can use localForage.clear directly .

@fedirjh Thanks for that initial investigation. There has been a string of problems that we have run into which makes LocalForage.clear() not a valid option for us.
Primarly, it's because there are a small handful of keys that we don't want to be removed (preferred locale for example). What we were doing before was:

Remember the locale version
Clear Onyx
Re-set the locale version

This unfortunately, leads to a flash of text as it changes back and forth from different languages.
The idea then is that we should try to clear every key except the ones we want to keep. That's why it uses setItem() or removeItem().
I think we should continue down this path rather than going back to trying to use LocalForage.clear().
What I would like to do next is see about implementing the removeItems() plugin for LocalForage here: https://github.com/localForage/localForage-removeItems to see if that will speed up things to give us the same performance as clear()

defaultKeys also needs to be accounted for, yes. What I am talking about are these here:
https://github.com/Expensify/App/blob/main/src/libs/actions/SignInRedirect.js#L40-L43

You might be right though that once the cache is updated with the correct values, we can clear onyx and then reset the keys (at the bottom of the Onyx.clear() method here). Is that what you're saying?

Yes that's the point , we only update the clear method to reset keys after clear
   return Storage.clear(defaultKeyValuePairs);

and in here
this.clear = (keysToDefault) => Storage.clear().then(() => Storage.multiSet(keysToDefault));

OK, got it, thanks! How would you update that to account for the values I pointed to (just so I am clear on what you are proposing)?

I think those values are stored locally before we call Onyx.clear

OK, I don't think I'm totally following you, but regardless, I've tried this:
return Storage.clear()
                .then(() => Storage.multiSet(keyValuesToPreserve))

and it appears to be several times slower (like 2-3x) than using Storage.multiRemove() (which is a loop that is calling removeItem() on each key).
So maybe there is something that I'm not quite grasping with the timing of Storage.clear()

@tgolen I think you missed my comment , here is diff
Changes in Onyx.js

-            return Storage.multiSet(keyValuesToReset);
+            return Storage.clear(defaultKeyValuePairs);;

Changes in WebStorage.js
-       this.clear = () => Storage.getAllKeys()
-           .then(keys => _.map(keys, key => this.removeItem(key)))
-           .then(tasks => Promise.all(tasks));
+       this.clear = (keysToDefault) => Storage.clear().then(() => Storage.multiSet(keysToDefault));

Ah, thank you for the clear diffs. That makes more sense now! I keep forgetting of WebStorage.js. I'm wondering if I'm still doing something wrong though, because with ~4000 reports, it still takes ~35 seconds to complete Onyx.clear(). You can see my branch here in this WIP https://github.com/Expensify/react-native-onyx/pull/220/files (ignore the multiRemove() stuff for now).
I'm timing this in App with this branch: https://github.com/Expensify/App/pull/13886

@tgolen can you set this account<EMAIL_ADDRESS>) for 4K reports ? I did tested with 1K account and results are ~200 ms

Hm, I don't think I can. I'm just testing this in my local environment. I
can send you credentials for a high traffic account on staging through.
Please send me a DM in slack to @tgolen
On Mon, Jan 2, 2023 at 4:34 PM Fedi Rajhi @.***> wrote:

@tgolen https://github.com/tgolen can you set this account (
@.*** ) for 4K reports ? I did tested with 1K account and
results are ~200 ms
—
Reply to this email directly, view it on GitHub
https://github.com/Expensify/App/issues/13884#issuecomment-1369294126,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJMAB4WAMODHUEKSN2LSFDWQNXYDANCNFSM6AAAAAATMGUI7U
.
You are receiving this because you were mentioned.Message ID:
@.***>

OK, the results are looking pretty good now. It appears that the times when it takes a while depends on how quickly you sign out after signing in. For these times that are high, I am signing out about 2-3 seconds after signing in. There must just be a lot of data being put into Onyx which slows things down. If I wait about 15-20 seconds before signing out, then it is very fast to clear.

Unfortunately, even with this improvement, if you signout quickly, and refresh (before the timing event has fired) then you still end up with report data in Onyx :disappointed: But, I guess this is still better than what we had before, even if it doesn't fully solve the problem.

i think the actual implementation of the logout is split on two parts , 1st part is to discard auth token ( this will redirect  to login page) then we call Onyx.clear , you can check it here https://github.com/Expensify/App/blob/d9b25e20846f5ea0670223ed2b06570e8c75a0c0/src/libs/actions/Session/index.js#L56-L60
I did tracked back the code changes , there were a PR that implemented this behaviour because of Onyx.clear taking longer time , this is the PR https://github.com/Expensify/App/pull/7397
I think with actual results we can revert those change hence the redirection to login page will be done once Onyx.clear.
you can test with this diff , I just commented the part that's making redirection happen before Onyx.clear is called
diff --git a/src/libs/actions/Session/index.js b/src/libs/actions/Session/index.js
index b564efd5e7..3b84d67082 100644
--- a/src/libs/actions/Session/index.js
+++ b/src/libs/actions/Session/index.js
@@ -29,7 +29,7 @@ Onyx.connect({
 function signOut() {
     Log.info('Flushing logs before signing out', true, {}, true);

-    const optimisticData = [
+    /* const optimisticData = [
         {
             onyxMethod: CONST.ONYX.METHOD.SET,
             key: ONYXKEYS.SESSION,
@@ -40,7 +40,7 @@ function signOut() {
             key: ONYXKEYS.CREDENTIALS,
             value: {},
         },
-    ];
+    ]; */
     API.write('LogOut', {
         // Send current authToken because we will immediately clear it once triggering this command
         authToken: NetworkStore.getAuthToken(),
@@ -48,7 +48,7 @@ function signOut() {
         partnerName: CONFIG.EXPENSIFY.PARTNER_NAME,
         partnerPassword: CONFIG.EXPENSIFY.PARTNER_PASSWORD,
         shouldRetry: false,
-    }, {optimisticData});
+    }, {/* optimisticData */});

     Timing.clearData();
 }

You can also check this conversation , the root cause is the same , I bet that issue appeared when you switched from clear to removeItem for Onyx.clear.

I did some more exploring this problem today, and here is what I found is happening with the signout process.

When you sign in on a large account, the OpenApp API command happens. This has a large Onyx instruction for mergeCollection with all the report details (eg. 4000 reports).
In LocalForage.mergeCollection() there is a SyncQueue which is filled with a bunch of setItem operations (one for each report)
If you sign out while that SyncQueue is in process, Onyx will get cleared, but the SyncQueue continues to function and data keeps getting put into Onyx
This leaves a lot of stale data in Onyx after signing out

You can avoid this by signing out after all the SyncQueue is finished, but as a user, you never know when that could be (it's like 30-40s for 4000 reports).
cc @kidroca @marcaaron @chrispader to get more eyes on this problem.
@chrispader and I were discussing a possible to solution to add a method to the SyncQueue called abort() which would be called during an Onyx.clear(). This would stop all of the setItem calls from happening.
We should also do something similar for the SQLite data provider if possible (is it possible? cc @Szymon20000).

Could we also somehow check the SyncQueue to see if it is not empty when calling clear - and if so, wait for it to complete?

I think you could, yes. I don't think it's really valuable to wait for the queue to empty though because you're just waiting to clear it out. It's best to just stop putting stuff into Onyx I think.

I think the abort idea is a good one. It should be pretty simple to do.
I would also be up for revisiting the "sync queue" in general if it is giving us problems. Maybe we can do a better solution to IndexedDB's performance problems. Remember, this is the only reason we have the sync queue.

I agree it would be best to stop the queue
I don't think we should access the queue from Onyx.clear, it's not possible unless we expose it, but we don't have to - We can call SyncQueue.clear() from the providers/LocalForage.clear method

As far as testing goes, there are no SyncQueue tests.
Since the queue is only used in providers/LocalForage it can be tested in the tests for the provider.
A test can queue 4-5 items and mock localforage.setItem to resolve using Jest's fake timers
Let's say we setup setItem calls to resolve after a timeout of 1000ms

We call Storage.setItem 5 times
Then advance the time by 1000ms
Then call Storage.clear
Then advance the time by 4000ms

The test is successful if only one setItem call was deleagated to localForage (the 3rd party lib mock)
This simulates engaging some writes, but calling Storage.clear before 4 of the pending writes are complete
Another test can verify no setItem call is made after localForage.clear

@tgolen I have a better option here , we can improve these 3 methods : SyncQueue , multiMerge and multiSet , I will create a Draft PR for the required changes.
These are tests with account that have 1774 key :
Before

After

@tgolen Could you assign me and pull BZ to the issue? Thanks!

@davidcardoza I think you can ignore some of the above. I just needed to assign it to BZ so that @mollfpr could get paid (I think)

bump @davidcardoza

@mollfpr I extended a contract to you via Upwork
https://www.upwork.com/jobs/~012e3f7c4d43a1722c

@davidcardoza Accepted, thank you! Also, is a bonus apply to internal PR review?

Oh, maybe I am a little confused here. @mollfpr why are you requesting payment for this? I don't see you providing any proposals or helping out at all.
@fedirjh I am fine with you having some compensation for your comments in this GH which were helpful for me in writing the solution. I am OK with compensating you with $250 for your help.

@tgolen I got assigned to review the internal PR.

Oh, right. I was only looking at the react-native-onyx PR: https://github.com/Expensify/react-native-onyx/pull/220. I see you reviewed the PR for App 👍

Thanks for flagging @fedirjh I should have caught your contribution. My fault I will process the contract in Upwork.

@mollfpr sent payment!
@fedirjh sent offer!

@davidcardoza mind tackling payment to @fedirjh when they accept?

@tgolen, @davidcardoza, @mollfpr 12 days overdue. Walking. Toward. The. Light...

@fedirjh did you accept? I am having trouble location the offer sent by @dylanexpensify

Hmmm...Can you link me to the post you accepted? I am not seeing you in the hired or proposal sections of the job posting
https://www.upwork.com/jobs/~012e3f7c4d43a1722c

@davidcardoza That’s my Upwork profile 😅
I’ll already get the payment from Dylan.

Sorry my comment comment was for @fedirjh you can ignore.

great, closing this out then!

Payment sent.


Speedup pipeline

only build preview images for arm64 and x86
do not run test and web on default branch

Codecov Report
Patch coverage: 52.63% and project coverage change: -0.16 :warning:

Comparison is base (6c58e9d) 39.28% compared to head (05ec72d) 39.12%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1966      +/-   ##
==========================================
- Coverage   39.28%   39.12%   -0.16%
==========================================
  Files         177      176       -1
  Lines       10734    10680      -54
==========================================
- Hits         4217     4179      -38
+ Misses       6223     6209      -14
+ Partials      294      292       -2

Impacted Files
Coverage Δ

pipeline/backend/docker/docker.go
0.00% <0.00%> (ø)

pipeline/backend/kubernetes/pod.go
0.00% <0.00%> (ø)

pipeline/frontend/yaml/compiler/convert.go
0.00% <0.00%> (ø)

pipeline/frontend/yaml/types/container.go
77.77% <0.00%> (ø)

server/model/pipeline.go
0.00% <ø> (ø)

server/model/step.go
0.00% <ø> (ø)

server/model/workflow.go
0.00% <ø> (ø)

...tastore/migration/020_parent_steps_to_workflows.go
59.45% <ø> (ø)

pipeline/frontend/yaml/compiler/compiler.go
7.10% <20.00%> (+0.28%)
:arrow_up:

pipeline/stepBuilder.go
77.57% <67.27%> (+0.87%)
:arrow_up:

... and 1 more

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Could we even skip arm completely for normal PR tests as they are still pretty slow due to the emulation part. If we want to test them / provide test images we could add a step with a label filter.


Warnings about missing selectedKind and selectedStory

Steps to reproduce:

Go through installation guide (https://storybook.js.org/basics/slow-start-guide/)
Open http://localhost:9001/ (without query string!)
Open console

Result:

Expected result: clean console.
Additional information:
These warnings don't break the app, but add some extra noise, that sometimes confuses.
Apparently the problem is here: https://github.com/storybooks/storybook/blob/a9944e1533de79ce4a1a2fd8d2b25aa450998147/app/react/src/client/preview/init.js#L7
Probably it's not a good idea to run init function before stories had been added, or if it's an acceptable case, then selectedKind and selectedStory props of Stories component should become optional.

I have the same issue.
Prev version (no issues) : 3.1.7
Actualy version (with issue): 3.2.3

Looks like we jus should select the first story of the first kind by default. Feel free to file a PR

Yup, seeing this error too in Storybook 3.2.3. Also seeing an error about the prop-types package:
Warning: Accessing PropTypes via the main React package is deprecated. Use the prop-types package from npm instead.

Is the issue still open? I'd like to take a shot at this one.

I just ran into it this weekend after upgrading to the newest Storybook (and doing a clean install), so I believe it's still an issue.

Cool, I'll check it out and try to make fix


Add variate generation for Wiener First Passage Time Distribution

The current implementation of the Wiener First Passage Time Distribution lacks a generation step for a variate following this distribution. This feature should be analogous to the _rng methods that other distributions have and which can only be used in generated quantities or transformed data blocks.

I have written the wiener_rng function in Stan at https://discourse.mc-stan.org/t/alternative-to-wiener-rng-in-stan/28443/8?u=spinkney. It should be pretty straightforward for someone to put this into stan-math.

And now it actually works too! https://discourse.mc-stan.org/t/alternative-to-wiener-rng-in-stan/28443/11?u=spinkney


chore(feat-multi-module-components): release 0.76.1

:robot: I have created a release beep boop
0.76.1 (2022-10-31)
BugFixes

Read terraformVersion from ComponentCommon.Common (#15) (af6b9d4)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/vincenthsh/fogg/releases/tag/v0.76.1 :sunflower:


Custom React hooks

I can not use the data return from a custom hook to update my state, this was not explained in the Docs.

I have to defend hook to other functions.

I have to defend hook to other functions.

I have to defend hook to other functions.

Please show you code via a GitHub repository or codesandbox.


Qa

Codecov Report

Merging #47 (fb62257) into master (fa83bd1) will decrease coverage by 2.91%.
The diff coverage is 48.48%.

@@            Coverage Diff             @@
##           master      #47      +/-   ##
==========================================
- Coverage   81.16%   78.24%   -2.92%
==========================================
  Files          13       13
  Lines         775      786      +11
==========================================
- Hits          629      615      -14
- Misses         96      117      +21
- Partials       50       54       +4

Impacted Files
Coverage Δ

api/v1alpha1/elastalert_types.go
100.00% <ø> (ø)

controllers/observer/observer.go
79.09% <20.00%> (-2.99%)
:arrow_down:

controllers/deployment_controllers.go
55.93% <25.00%> (-2.25%)
:arrow_down:

controllers/elastalert_controller.go
51.04% <40.00%> (-11.81%)
:arrow_down:

controllers/podspec/deployment.go
84.21% <71.42%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fa83bd1...fb62257. Read the comment docs.


Add optional sentry configuration

cookiecutter.json must have a prompt that says, "Sentry for monitoring? (y/n)". If the user clicks yes, then template adds in the Raven package and configuration.

Closed via #292


Can not run the docker image of version 0.1.14

Hi There,
Thanks for sharing the project. I am trying to run the project in docker. I pulled the image successfully.
docker pull dromni/nerfstudio:0.1.14

However, when I try to run the docker image with the following command:
docker run --gpus all -v /home/songlin/data:/workspace/ -v /home/songlin/.cache:/home/user/.cache/ -p 7007:7007 --rm -it dromni/nerfstudio:0.1.14

I met the error:
==========
== CUDA ==
==========

CUDA Version 11.7.1

Container image Copyright (c) 2016-2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

[13:55:54] 🤷 .zshrc not found, skipping.                                                                 install.py:210
           🔍 Found .bashrc!                                                                              install.py:212
           ✔ Wrote new completion to /home/user/nerfstudio/scripts/completions/bash/_ns-install-cli!      install.py:117
           ✔ Wrote new completion to /home/user/nerfstudio/scripts/completions/bash/_ns-dev-test!         install.py:117
           ✔ Wrote new completion to /home/user/nerfstudio/scripts/completions/bash/_ns-process-data!     install.py:117
[13:55:55] ✔ Wrote new completion to /home/user/nerfstudio/scripts/completions/bash/_ns-download-data!    install.py:117
[13:55:56] ✔ Wrote new completion to /home/user/nerfstudio/scripts/completions/bash/_ns-eval!             install.py:117
           ❌ Completion script generation failed: ['ns-train', '--tyro-print-completion', 'bash']        install.py:107
           Matplotlib created a temporary config/cache directory at /tmp/matplotlib-fz54s6k1 because the  install.py:111
           default path (/home/user/.cache/matplotlib) is not a writable directory; it is highly
           recommended to set the MPLCONFIGDIR environment variable to a writable directory, in
           particular to speed up the import of Matplotlib and to better support multiprocessing.
           Traceback (most recent call last):
             File "/home/user/.local/bin/ns-train", line 5, in <module>
               from scripts.train import entrypoint
             File "/home/user/nerfstudio/scripts/train.py", line 50, in <module>
               from nerfstudio.configs.method_configs import AnnotatedBaseConfigUnion
             File "/home/user/nerfstudio/nerfstudio/configs/method_configs.py", line 46, in <module>
               from nerfstudio.field_components.temporal_distortions import TemporalDistortionKind
             File "/home/user/nerfstudio/nerfstudio/field_components/__init__.py", line 17, in <module>
               from .encodings import Encoding, ScalingAndOffset
             File "/home/user/nerfstudio/nerfstudio/field_components/encodings.py", line 34, in <module>
               import tinycudann as tcnn
             File "/home/user/.local/lib/python3.10/site-packages/tinycudann/__init__.py", line 9, in
           <module>
               from tinycudann.modules import free_temporary_memory, NetworkWithInputEncoding, Network,
           Encoding
             File "/home/user/.local/lib/python3.10/site-packages/tinycudann/modules.py", line 35, in
           <module>
               raise EnvironmentError(f"Could not find compatible tinycudann extension for compute
           capability {system_compute_capability}.")
           OSError: Could not find compatible tinycudann extension for compute capability 75.

Traceback (most recent call last):
  File "/home/user/.local/bin/ns-install-cli", line 8, in <module>
    sys.exit(entrypoint())
  File "/home/user/nerfstudio/scripts/completions/install.py", line 282, in entrypoint
    tyro.cli(main, description=__doc__)
  File "/home/user/.local/lib/python3.10/site-packages/tyro/_cli.py", line 127, in cli
    _cli_impl(
  File "/home/user/.local/lib/python3.10/site-packages/tyro/_cli.py", line 328, in _cli_impl
    out, consumed_keywords = _calling.call_from_args(
  File "/home/user/.local/lib/python3.10/site-packages/tyro/_calling.py", line 194, in call_from_args
    return unwrapped_f(*args, **kwargs), consumed_keywords  # type: ignore
  File "/home/user/nerfstudio/scripts/completions/install.py", line 251, in main
    completion_paths = list(
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/user/nerfstudio/scripts/completions/install.py", line 253, in <lambda>
    lambda path_or_entrypoint_and_shell: _generate_completion(
  File "/home/user/nerfstudio/scripts/completions/install.py", line 112, in _generate_completion
    raise e
  File "/home/user/nerfstudio/scripts/completions/install.py", line 99, in _generate_completion
    new = subprocess.run(
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ns-train', '--tyro-print-completion', 'bash']' returned non-zero exit status 1.
           ✔ Wrote new completion to /home/user/nerfstudio/scripts/completions/bash/_ns-render!           install.py:117

Can somebody help?

It seems like this may be the same issue as #1056?

I met the issue while installed docker version for
sudo docker run --gpus all -v /home/ubuntu/data:/workspace/  -v /home/ubuntu/.cache/:/home/user/.cache/ -p 7007:7007 --rm  -it dromni/nerfstudio:0.1.16

solved https://github.com/nerfstudio-project/nerfstudio/issues/1056#issuecomment-1410456290


Support passing env vars to tasks

I've yet to need this, but I expect that sooner or later I will.

This is done.


write call is returning Mono.empty() when request failed with 404 Not Found

Our app tried to write to vault and vault returned 404 because the path doesn't exist.
Expected behaviour: We assumed in such scenario spring-vault would result mono error
Actual behaviour: it actually returns Mono.empty().
Have a look at this method in vault/core/ReactiveVaultTemplate.java
	private static <T> Mono<? extends T> mapOtherwise(ClientResponse response, String path) {

		if (response.statusCode() == HttpStatus.NOT_FOUND) {
			return Mono.empty();
		}

NOTE: the problem is when vault returns 204 No Content, spring-vault also results in Mono.empty(), which makes it almost impossible for downstream to handle 204 and 404 responses.

Good catch. That is a bug that we should fix in write and delete. Mono.empty() should apply only to 404's received through read(). 404's in write and delete should propagate an error.

Thanks @mp911de
Can I also ask when 2.1.5 will be out?

Releases are mostly driven by the content. We released 2.1.4 and 2.2.0 recently so it may take to January until we ship another release.

@mp911de how do we want to fix this, i can do the pr just want to see if you have figured it out or it need investigation.

We need to refine ReactiveVaultTemplate.mapResponse(…) and mapOtherwise(…). Currently, mapResponse(…) considers HttpStatus.NOT_FOUND to be a non-error outcome. We need to retain that functionality for the read(…) and doRead(…) cases but not for write and delete.
Happy to review a pull request and help you with the next steps.


DataCollection: added reverse config

LGTM @sjiang2018 @JasonXu12 Please double check if the criteria set for reverse gear makes sense.


use instanceof instead of isInstance() on the hot path

That's surely NBD, or it would have shown up in Sanne's performance work long ago, but still, every time I see it I'm just left with that nagging doubt.
And it's such an easy thing to fix in IntelliJ..

Does anyone object to this change? @sebersole?

Nobody has objected, and all those warnings just mess with my OCD.


WIP: [CSL-2383] Extend testing infrastructure

When done this PR will pave the way to hook up @uroboros 's work on the wallet to the inductive wallet generator. Indeed, a large part of this PR will consist of a reworked version of the work that @uroboros has already done.

I think that it is more correct to ignore epoch-boundary blocks: they are only convenience devices, not semantic

Sure, that's fair, but the very first block does point to the "genesis block" as its predecessor. So conceivably something odd happens at epoch boundaries too. Not disagreeing with you btw, but I just wanted to make sure :)

Yessir :)

@uroboros CI is failing due to warnings in the Kernel module. Would you mind fixing those, so that we can merge this PR? Thanks! :)

Addressed review comments as indicated in the comments above, with the exception of nitpicks on formatting issues in @uroboros 's code. I don't necessarily disagree with them but I don't think it's the best use of my time right now :)

Rebased on develop in the hope of making CI go green.


[Bug]: 1.20 hVers0.0.3

Bug Description
Heracles 0.0.3 complains of mitchmatched mod from server https://mclo.gs/gn7dLvW
How to Reproduce?
install heracles on client and server
join server
Expected Behavior
Join server without a popup kicking me back to main menu
Version
0.0.3
Mod Loader Version
0.14.21
Mod Loader
Fabric
Logs or additional context
https://mclo.gs/gn7dLvW
Code of Conduct

[X] I have searched the issue tracker and confirmed that the issue has not been reported
[X] I have checked the FAQ (if one is present) and my issue is not listed
[X] I have verified that I am using the latest version of the mod
[X] I have verified that I have installed all the required dependencies for the mod
[X] I have verified that I do not have Optifine installed. We will close issues if we detect Optifine is in your pack. We cannot fix Optifine issues. Please do not report issues with Optifine present.

the server is using 0.0.2 not 0.0.3

it isnt.


⚠ Bofrost.be has degraded performance

In 642edb0, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 2373 ms

Resolved: Bofrost.be performance has improved in ce05c1d.


concatenate bytes with bytes

Pinging @rsignell-usgs who needs this to fix https://github.com/USGS-CIDA/pyGDP/pull/141
output_content will be bytes in Python 3 and concatenating it with string causes OWSLib to fail when we do:
content = content + output_content

Latest master status on Travis-CI is:
Python 2.7 LXML=true :: 9 failed, 73 passed, 9 warnings
Python 2.7 LXML=false :: 9 failed, 73 passed, 7 warnings
Python 3.4 LXML=true :: 16 failed, 66 passed
Python 3.4 LXML=false :: 16 failed, 66 passed

I got slightly less failures with Python 3.4 LXML=false (14 failed, 68 passed) also a few less failure for Python 3.6 in both cases. However, I don't think it is related to this change b/c I don't think that part of the code is covered with a test.
PS: I also remove Python 2.6 from the Travis-CI configuration and added Python 3.6 to it.

@tomkralidis we could really use this!

We need a new release :smile:


Offset angle radar chart

Is it possible to give an offset angle for a radar chart, so that the first axis would not start at the top?

This is not currently possible in v1.
In v2, this is easy to expose, as we can simply refactor the scale to take a start angle when generating point positions. As a result, targeting for v2.1

Done in #2984


fix viewport events

I was a bit over eager with the events, it does indeed make more sense to not fire the inBounds again when going out of viewport.
Related to:
#231

Closing this PR since we have a better working solution on the dev branch


[4.3dev] Exported node path in child scene are reflected in the parent scene file

Tested versions

Reproductible in any 4.3 dev

System information
macOS Monterey 12.6.5
Issue description
Exported nodes in a child scene will be reflected in the parent scene file when saving a scene, BUT only if the parent scene is opened in the editor.
Steps to reproduce

Create an empty scene called MyLevel
Create a scene called Menu. Create a node with a script that exports some other nodes from the same scene (example: @export var nodes: Array[Label])
Instantiate the Menu scene in MyLevel as a child scene
Keep both scenes open in the editor and save
Using an external tool (VSCode, vim, cat, etc...), inspect the MyLevel scene and observe that exported nodes from Menu are reflected in the MyLevel file

Minimal reproduction project (MRP)
exported-node-issue.zip

If I understand correctly, what you've reported is about properties of the instance being copied to properties of the base scene.
It also applies in reverse, though. It's turning out to be quite the headache. I tried removing a node export on the base scene, but since, for some reason, the NodePath is also saved for the instance despite the two paths matching, I can't just do this like I used to be able to...

If I understand correctly, what you've reported is about properties of the instance being copied to properties of the base scene.

That's it! That's the issue I wanted to describe. This wasn't the case in Godot 4.2.

This wasn't the case in Godot 4.2.

Indeed, can't reproduce in v4.2.2.stable.official [15073afe3] and earlier.
Can reproduce since v4.3.dev1.official [9d1cbab1c].
That's a regression, hence marking it as such (should be addressed before 4.3.stable).

menu.tscn
my_level.tscn

menu.tscn contents (same in v4.2.2.stable.official [15073afe3] / v4.3.dev6.official [89850d553]):
[gd_scene load_steps=2 format=3 uid="uid://bdp4gvtih2bec"]

[ext_resource type="Script" path="res://menu.gd" id="1_vb7hi"]

[node name="Menu" type="Node2D" node_paths=PackedStringArray("nodes")]
script = ExtResource("1_vb7hi")
nodes = [NodePath("VBoxContainer/Label"), NodePath("VBoxContainer/Label2"), NodePath("VBoxContainer/Label3")]

[node name="VBoxContainer" type="VBoxContainer" parent="."]
offset_right = 40.0
offset_bottom = 40.0

[node name="Label" type="Label" parent="VBoxContainer"]
layout_mode = 2
text = "Label 1"

[node name="Label2" type="Label" parent="VBoxContainer"]
layout_mode = 2
text = "Label 2"

[node name="Label3" type="Label" parent="VBoxContainer"]
layout_mode = 2
text = "Label 3"

my_level.tscn contents:

v4.2.2.stable.official [15073afe3]:

[gd_scene load_steps=2 format=3 uid="uid://dakw7h8ta0aa4"]

[ext_resource type="PackedScene" uid="uid://bdp4gvtih2bec" path="res://menu.tscn" id="1_tkt5r"]

[node name="MyLevel" type="Node2D"]

[node name="Menu" parent="." instance=ExtResource("1_tkt5r")]

v4.3.dev6.official [89850d553]:

[gd_scene load_steps=2 format=3 uid="uid://dakw7h8ta0aa4"]

[ext_resource type="PackedScene" uid="uid://bdp4gvtih2bec" path="res://menu.tscn" id="1_tkt5r"]

[node name="MyLevel" type="Node2D"]

[node name="Menu" parent="." node_paths=PackedStringArray("nodes") instance=ExtResource("1_tkt5r")]
nodes = [NodePath("VBoxContainer/Label"), NodePath("VBoxContainer/Label2"), NodePath("VBoxContainer/Label3")]

This leads to potential desyncing of the properties, e.g. after adding a new entry to nodes within the menu.tscn and saving it, the nodes saved in my_level.tscn remains unchanged.

Exported nodes in a child scene will be reflected in the parent scene file when saving a scene, BUT only if the parent scene is opened in the editor.

"BUT only if the parent scene is opened in the editor"
I can reproduce with no other scenes opened (v4.3.dev6.official [89850d553]):

Open the MRP, no scenes are auto opened.
Create new scene (root node), instantiate menu.tscn as child.
Save the scene.
Inspect created <new_scene>.tscn file, the node paths from menu.tscn are duplicated in there.

@Sauermann Unlabelling topic:export as it's not for @export annotation related issues, it's for the export system and templates (exporting the project to executable etc.). :slightly_smiling_face:

Thank you for digging into it! I wasn't able to reproduce this when the scenes are closed, but maybe I missed something. That's good to know.

If someone has time to bisect, this should be done between 4.2-stable (good) and 9d1cbab1c432b6f1d66ec939445bec68b6af519e (bad). Interestingly, 4.3-dev1 was a very conservative snapshot with mostly the same contents as 4.2.1-rc1, so it must be one of the few commits which weren't cherry-picked (we did merge a handful of "riskier" ones in the first snapshot to get a headstart on regression testing).

Bisecting points to #83343 as the culprit

CC @warriormaster12 @KoBeWi

Minor note from my own run-in and experimentation with this issue: It does not seem to affect @exports of NodePaths, only @exports of Nodes themselves.

I've run into this as well and my interpretation of the issue is that when instancing a scene (child.tscn in parent.tscn) with exported nodepaths on child.tscn it now always saves the nodepaths in parent.tscn as well, as if they had set overrides in parent.tscn. For example, even without overriding the my_node_path export var parent.tscn will contain:
[node name="Child" parent="." node_paths=PackedStringArray("my_node_path") instance=ExtResource("1_v8w3q")]
my_node_path = NodePath("Node1")

This is problematic since if I then change the my_node_path export var in child.tscn the instance in parent.tscn in will not update as it will use the nodepath set in parent.tscn as an override. However, if I open parent.tscn in the editor and resave it, it will update the nodepath to match child.tscn. So somehow the editor is keeping track of that this nodepath should not be an override even if it's saved as one and applied as one when the scene is run. But the real issue to me seems to be that it's saved there at all when there's no override set.
If instead I edit parent.tscn in a text editor and just remove the my_node_path = NodePath("Node1") line it will behave as expected, using the nodepath set in child.tscn as expected with no overrides without needing to open or re-save parent.tscn on changes. This is also consistent with how overrides of other export var types work, and how I'd expect it to work for nodepaths as well.
I think this could also be the root issue behind the node duplication issues https://github.com/godotengine/godot/pull/83343 above was originally meant to fix. Since if the nodepaths are seen as overridden in instances when they are not this could I think cause exactly these types of issues when duplicating these instances.

I'm experiencing the same issue with the 4.3.beta.2. Exported nodes are persisted in nested scenes.

That's what I did. Here is a screencast; maybe i do something wrong.
https://github.com/godotengine/godot/assets/1923206/f208002d-d6d9-4862-a0d3-c313f54104fd
Thank you for the fast answer 🙏

Your scenes still show revert icon though. Revert them before saving.

The ones in the video are exported from the root node and they are of type string; it doesn't contain the labels. Here is the correct one:

As you can see, there is no revert icon.

Ok looks like the only fixed case is when a Node is exported, but not Array of nodes.

Many many thanks for the quick investigation and MR @KoBeWi 🙏

I confirm the issue has been solved for good with beta 3! Again, many thanks to everyone for this fix!


made USB_DPU optional

made USB_DPU optional. Just comment out USB_DPU if not needed / tied to 3V3 via 1.5K

actually, separating the DPU pin configuration creates extra code that bring the bootloader really close to its 2k limit. I might rework that to be more efficient (but also more ugly). I will need the saved space for the next feature 🙈

Well #ifdefs within the same "line" don't look nice, but in the end it saved 20 bytes 🤷♂