VRAM issues

Hi!
The repo and finetuning tutorial mention >16GB as a minimum amount of VRAM. Has this been verified somehow?
Running this on an RTX3090 (24GB VRAM) and CUDA is still OOM:ing.
With that result running this on Google Colab doesn't seem particularly possible?
The tutorial mentions reducing batch_size which I've done (by changing batch_size and num_workers to 1 in the pokemon.yaml conf) but still CUDA OOM.
Are there any other params one can change to reduce memory use?
Were batch_size and num_workers the correct parameters to tweak in the first place?
Thanks!

Same here. 3x 3090, No joy.

One thought. The environment.yaml file specifies Cuda 11.0. The 3090 requires 11.1 or better. (see https://forums.developer.nvidia.com/t/cuda-10-1-on-rtx-3090/185255/2)
I'm not sure what the consequences are wrt the supported PyTorch and then cascading set of dependency changes that happen, but were going to try 11.1.
The forked master ( presser/stable-diffusion) has 11.0, but strangely compvis/stable-diffusion has cuda 11.3
and the Yaml file was updated a month ago, which seems to have different dependencies pinned. Just an observation, but this repo at 11.0 is likely going to be an issue for newer coda hardware.

Hmm yeah maybe related? I'm having these issues in an environment with 11.3 anyways.

I'm also unable to run this in an environment with 11.3.
I think the environment.yaml file is definitely outdated, as it mentions pytorch=1.7.0, which is incompatible with the dependencies inside requirements.txt (specifically, kornia==0.6 requires torch > 1.8)
@justinpinkney it would be great if you could specify the torch and torchvision versions in the requirements.txt.
FWIW, I tried reducing the batch size to 1, and training and validation image sizes to 128, but still got OOM errors.
Also, I'm happy to run any tests needed to help get this running on a 3090. Any help is much appreciated. Thanks!

Yeah the environment.yml is old an inherited from the base repo, I should get rid of that.
The version I'm using are:
torch==1.12.1
torchvision==0.13.1
cuda 11.3.1

Thanks @justinpinkney.
I setup a fresh docker container with the mentioned requirements via docker pull pytorch/pytorch:1.12.1-cuda11.3-cudnn8-runtime, then installed deps from the requirements.txt. I had to uninstall torchtext to get the script working, but then failed with the following OOM error:
RuntimeError: CUDA out of memory. Tried to allocate 58.00 MiB (GPU 0; 23.70 GiB total capacity; 21.90 GiB already allocated; 13.62 MiB free; 22.07 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I modified image size and batch size as follows:
data:
  target: main.DataModuleFromConfig
  params:
    batch_size: 1
    num_workers: 0
    num_val_workers: 0
    train:
      target: ldm.data.simple.hf_dataset
      params:
        name: lambdalabs/pokemon-blip-captions
        image_transforms:
        - target: torchvision.transforms.Resize
          params:
            size: 64
            interpolation: 3
        - target: torchvision.transforms.RandomCrop
          params:
            size: 64
        - target: torchvision.transforms.RandomHorizontalFlip
    validation:
      target: ldm.data.simple.TextOnly
      params:
        captions:
        - A pokemon with green eyes, large wings, and a hat
        - A cute bunny rabbit
        - Yoda
        - An epic landscape photo of a mountain
        output_size: 128
        n_gpus: 1

But still failed with the same error. Curious if you're able to replicate this.
Any ideas how to proceed?

@justinpinkney The full stack trace is:

Epoch 0:   0%|                                                                          | 0/833 [00:00<00:00, 8943.08it/s]/opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed.  This is not an error, but may impair performance.
grad.sizes() = [1280, 1280, 1, 1], strides() = [1280, 1, 1280, 1280]
bucket_view.sizes() = [1280, 1280, 1, 1], strides() = [1280, 1, 1, 1] (Triggered internally at  /opt/conda/conda-bld/pytorch_1659484809535/work/torch/csrc/distributed/c10d/reducer.cpp:312.)
  allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
Summoning checkpoint.

Traceback (most recent call last):
  File "main.py", line 893, in <module>
    raise err
  File "main.py", line 875, in <module>
    trainer.fit(model, data)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 553, in fit
    self._run(model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 918, in _run
    self._dispatch()
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 986, in _dispatch
    self.accelerator.start_training(self)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/accelerators/accelerator.py", line 92, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 161, in start_training
    self._results = trainer.run_stage()
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 996, in run_stage
    return self._run_train()
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1045, in _run_train
    self.fit_loop.run()
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/loops/fit_loop.py", line 200, in advance
    epoch_output = self.epoch_loop.run(train_dataloader)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 150, in advance
    "on_train_batch_end", processed_batch_end_outputs, batch, self.iteration_count, self._dataloader_idx
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1217, in call_hook
    trainer_hook(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/callback_hook.py", line 189, in on_train_batch_end
    callback.on_train_batch_end(self, self.lightning_module, outputs, batch, batch_idx, dataloader_idx)
  File "/stable-diffusion/main.py", line 409, in on_train_batch_end
    self.log_img(pl_module, batch, batch_idx, split="train")
  File "/stable-diffusion/main.py", line 377, in log_img
    images = pl_module.log_images(batch, split=split, **self.log_images_kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/stable-diffusion/ldm/models/diffusion/ddpm.py", line 1323, in log_images
    with ema_scope("Sampling"):
  File "/opt/conda/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/stable-diffusion/ldm/models/diffusion/ddpm.py", line 182, in ema_scope
    self.model_ema.store(self.model.parameters())
  File "/stable-diffusion/ldm/modules/ema.py", line 62, in store
    self.collected_params = [param.clone() for param in parameters]
  File "/stable-diffusion/ldm/modules/ema.py", line 62, in <listcomp>
    self.collected_params = [param.clone() for param in parameters]
RuntimeError: CUDA out of memory. Tried to allocate 58.00 MiB (GPU 1; 23.70 GiB total capacity; 21.90 GiB already allocated; 10.56 MiB free; 22.07 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

It looks like EMA is happening on GPU, which would explain the OOM issue.
I'm gonna attempt making it happen on CPU, will post any new insights here as I go along.
Curious if you have any more insights regarding this

Hmmm I think it used around 40gb of vram when training with a batch size of 4.
Yeah looks like the ema is adding some extra weight which might be the issue.
You could also try reduced precision training. The repo uses pytorch lightning so should be straightforward to try mixed precision training. Couldn't comment on the stability of that though...

I did try precision=16 but failed with some hard to diagnose errors. Didn't bother trying to solve that.

40gb of vram when training with a batch size of 4

I see. I think I can confidently say that setting up the model + EMA on GPU alone takes > 24 GB ram in its current state. Moving EMA to CPU would probably allow you to increase batch size by a reasonable amount

Modified EMA as follows:

import torch
from torch import nn

class LitEma(nn.Module):
    def __init__(self, model, decay=0.9999, use_num_upates=True, device="cpu"):
        super().__init__()
        if decay < 0.0 or decay > 1.0:
            raise ValueError('Decay must be between 0 and 1')

        self.device = device
        self.m_name2s_name = {}
        self.register_buffer('decay', torch.tensor(decay, dtype=torch.float32))
        self.register_buffer('num_updates', torch.tensor(0,dtype=torch.int) if use_num_upates
                             else torch.tensor(-1,dtype=torch.int))

        for name, p in model.named_parameters():
            if p.requires_grad:
                #remove as '.'-character is not allowed in buffers
                s_name = name.replace('.','')
                self.m_name2s_name.update({name:s_name})
                self.register_buffer(s_name,p.clone().detach().data)

                # Move above buffer to device
                setattr(self, s_name, getattr(self, s_name).to(self.device))

        self.collected_params = []

    def forward(self,model):
        decay = self.decay

        if self.num_updates >= 0:
            self.num_updates += 1
            decay = min(self.decay,(1 + self.num_updates) / (10 + self.num_updates))

        one_minus_decay = 1.0 - decay

        with torch.no_grad():
            m_param = dict(model.named_parameters())
            shadow_params = dict(self.named_buffers())

            for key in m_param:
                if m_param[key].requires_grad:
                    sname = self.m_name2s_name[key]
                    shadow_params[sname] = shadow_params[sname].type_as(m_param[key]).to(self.device)
                    shadow_params[sname].sub_(one_minus_decay * (shadow_params[sname] - m_param[key]))
                else:
                    assert not key in self.m_name2s_name

    def copy_to(self, model):
        m_param = dict(model.named_parameters())
        shadow_params = dict(self.named_buffers())
        for key in m_param:
            if m_param[key].requires_grad:
                m_param[key].data.copy_(shadow_params[self.m_name2s_name[key]].data)
            else:
                assert not key in self.m_name2s_name

    def store(self, parameters):
        """
        Save the current parameters for restoring later.
        Args:
          parameters: Iterable of `torch.nn.Parameter`; the parameters to be
            temporarily stored.
        """
        # breakpoint()
        self.collected_params = [param.clone().to(self.device) for param in parameters]

    def restore(self, parameters):
        """
        Restore the parameters stored with the `store` method.
        Useful to validate the model with EMA parameters without affecting the
        original optimization process. Store the parameters before the
        `copy_to` method. After validation (or model saving), use this to
        restore the former parameters.
        Args:
          parameters: Iterable of `torch.nn.Parameter`; the parameters to be
            updated with the stored parameters.
        """
        for c_param, param in zip(self.collected_params, parameters):
            param.data.copy_(c_param.data)

But now I fail later in the forward pass with an OOM error. I'm not sure if this can be run on 3090s without heavier modifications which I suspect would be a bit gnarly...

For anyone else trying to run this on 3090s, I was able to run huggingface's WIP script on a single 3090. Couldn't quite get the right accelerate config for multi-GPU yet.
https://github.com/huggingface/diffusers/pull/356
python train_text_to_image.py \
    --dataset_name lambdalabs/pokemon-blip-captions \
    --use_auth_token \
    --mixed_precision fp16 \
    --resolution 512 \
    --gradient_accumulation_steps 4 \
    --train_batch_size 1 \
    --num_train_epochs 1 \
    --output_dir sd-pokemon

haven't it tried it yet, but this might be another avenue for 3090 gang: https://github.com/XavierXiao/Dreambooth-Stable-Diffusion

To launch on multi-gpu you can use the following command:
accelerate launch --multi_gpu train_text_to_image.py \
    --dataset_name lambdalabs/pokemon-blip-captions \
    --use_auth_token \
    --mixed_precision fp16 \
    --resolution 512 \
    --gradient_accumulation_steps 4 \
    --train_batch_size 1 \
    --num_train_epochs 1 \
    --output_dir sd-pokemon

I have a machine with 6x GPU of 12GB ram. Is it possbile launch finetune on such kind of hardware configuration? It seems even 24G 3090 do not working.

I just simply set the use_ema=False. Then, the default main.py can be run, but ema model lost.
So the 3090 can train the model with a bit more VRAM.
I also have another idea，precalculate the condition text to embed space to save GPU VRAM.
Crying from a poor lab.

I just simply set the use_ema=False. Then, the default main.py can be run, but ema model lost.
So the 3090 can train the model with a bit more VRAM.
I also have another idea，precalculate the condition text to embed space to save GPU VRAM.
Crying from a poor lab.

Sadly, The clip model has little effect on GPU memory.

@patil-suraj Is there any way to train on multiple GPU without using accelerate launch

For anyone else trying to run this on 3090s, I was able to run huggingface's WIP script on a single 3090.

@StrugglingForBetter I did, yes. There's some GPU memory overhead, so it's not possible to train the full model. But, if you freeze the params for part of the model, it fits in memory.

Using EMA on multi-GPU
I have a 3x 3090 setup, and I hard-coded the EMA model to load on torch.device(2) and train on only the first two gpus.
In addition to the changes in the script, I set up my accelerate config to only use gpu_ids 0,1 for training

@StrugglingForBetter I did, yes. There's some GPU memory overhead, so it's not possible to train the full model. But, if you freeze the params for part of the model, it fits in memory.
Using EMA on multi-GPU
I have a 3x 3090 setup, and I hard-coded the EMA model to load on torch.device(2) and train on only the first two gpus. In addition to the changes in the script, I set up my accelerate config to only use gpu_ids 0,1 for training

could you share the code of putting the EMA model on different gpu?

@StrugglingForBetter I did, yes. There's some GPU memory overhead, so it's not possible to train the full model. But, if you freeze the params for part of the model, it fits in memory.
Using EMA on multi-GPU
I have a 3x 3090 setup, and I hard-coded the EMA model to load on torch.device(2) and train on only the first two gpus. In addition to the changes in the script, I set up my accelerate config to only use gpu_ids 0,1 for training

Thanks for your reply! Can you show us how to set accelerate config? Is there any way to train without accelerate config since I have never used accelerate when training on multiple GPUs before ?
BTW, can you give me your email so that I can contact you more conveiently? Thx!

But now I fail later in the forward pass with an OOM error. I'm not sure if this can be run on 3090s without heavier modifications which I suspect would be a bit gnarly...

I was able to run the code on the 3090 this way, but the results are far from the same, can you get similar results for Pokémon?

For anyone else trying to run this on 3090s, I was able to run huggingface's WIP script on a single 3090.


Feature | Filler templates switch

Make switching absolute and relative paths logic.

Made with simple menu option.


Pin bleach to version less than 5

Bleach 5.0.0 doesn't except the keyword argument 'styles' for clean() anymore.
So we get the error clean() got an unexpected keyword argument 'styles'.
So I pinned the Bleach version to bleach >= 2, <5.
Bleach 5.0.0 also doesn't support Python 3.6 anymore.
See https://github.com/mozilla/bleach/blob/main/CHANGES.

I had also to upgrade black to 22.3.0 and rerun it, as the old version was broken in the CI.
See psf/black#2964.

Could we switch to bleach > 5 instead? I would prefer to stay up to date instead of locking ourselves into an old version

Sure we can try.
Probably this will be a breaking change.
You could also merge this first to make a patch release to have a working version of Mezzanine 5 and I will prepare the move to bleach 5, which will be a major release - Mezzanine 6.

Well, I made a huge mess while trying to merge this from my phone (currently on vacation). I'll wait until next week to merge into stable.
Thanks for contributing!

What do you mean? It looks quite ok.
Can I use the master branch now for the next PR or do you want to change something?


Duplicate result of selecting td with classname

I'm try to get data from td with specific classname but the result is just the first one and duplicate.  Here is my code:-
var Xray = require("x-ray");
var xray = new Xray();
xray('http://www.barchart.com/quotes/stocks/$ONE', 'td', [{
name: '.qname_shad',
value: '.qval_shad'
}]
)
(function(err, obj) {
console.dir(obj);
});
Regards,

I'm still learning to use x-ray efficiently.  The following is what I came up with that might be close to what you're looking for:

var Xray = require("x-ray");
var xray = new Xray();
xray('http://www.barchart.com/quotes/stocks/$ONE', 'td', [{
name: xray('.qname_shad', ''),
value: xray('.qval_shad', '')
}]
)(function(err, obj) {
console.dir(obj);
});

The result would be:

[ { name: 'High', value: '922.33\t\t\t\t\t\t\t\t\t' },
  { name: 'High', value: '922.33\t\t\t\t\t\t\t\t\t' },
  { name: 'High', value: '922.33\t\t\t\t\t\t\t\t\t' },
  { name: 'High' },
  { value: '922.33\t\t\t\t\t\t\t\t\t' },
  { name: 'Low' },
  { value: '913.38\t\t\t\t\t\t\t\t\t' },
  { name: 'Open' },
  { value: '916.59\t\t\t\t\t\t\t\t\t' },
  { name: 'Prev Close' },
  { value: '916.50\t\t\t\t\t\t\t\t\t' },
  { name: '\r\n\t\t\t\t\t\t\t\t\t\t20-Day Average   924.89\t\t\t\t\t\t\t\t\t\t\r\n\t\t\t\t\t\t\t\t\t\r\n\t\t\t\t\t\t\t\t\t' },
  { name: '100-Day Average   899.15\t\t\t\t\t\t\t\t\t\t\r\n\t\t\t\t\t\t\t\t\t\r\n\t\t\t\t\t\t\t\t\t' },
  { name: 'Trend\r\n\t\t\t\t\t\t\t\t\t\tSpotter' },
  { value: 'Hold\t\t\t\t\t\t\t\t\t' },
  { name: 'Trend Strength' },
  { value: ' \t\t\t\t\t\t\t\t\t' } ]

I'm a novice for javascript & node.js world.  However, I think the result should have only 1 { name: 'High', value: '922.33' }.  Here is my code which using 'cheerio' & 'request' to something like the above x-ray code:-
// Load the request module
var request = require("request");
var cheerio = require("cheerio");

var url = 'http://www.barchart.com/quotes/stocks/$ONE';
request(url, function(error, response, body) {
	// check for error
	if (error) {
		return console.log('Error:', error);
	}

	// check for right status code
	if (response.statusCode !== 200) {
		return console.log('Invalid Status Code: ', response.statusCode);
	}

	if (body) {
		// parse data from web page
		var $ = cheerio.load(body);
		$('.qval_shad').each(function(i, e) {
			console.log("> " + i + ": " + $(e).text());
		});
		console.log('----------------------------------');
		$('.qname_shad').each(function(i, e) {
			console.log("> " + i + ": " + $(e).text().trim());
		});
	}
});

The result has only 1 High as I've expected.  So I think x-ray should do something wrong.

Ahh I see, you're only looking for one result.  Try this instead:

var Xray = require("x-ray");
var xray = new Xray();
xray('http://www.barchart.com/quotes/stocks/$ONE', 'tr', {
name: '.qname_shad',
value: '.qval_shad'
})
(function(err, obj) {
console.dir(obj);
});

The result would be:

{ name: 'High', value: '922.33\t\t\t\t\t\t\t\t\t' }

No, I don't looking for 1 result.  I want all of .qname_shad and .qval_shad pairs.  In fact, I want Open, High, Low, Close price/value on this page.  And I think x-ray will do this task for me as in my first code.  But I got multiple High which should be only 1.  So I think x-ray would be do wrong thing in its algorithm and I should report it as an issue.
I don't know how x-ray works and why I get multiple 'High' in my result which I think it should return only one.
Thanks for your suggestion.

@klingkie
The problem with the original question was selecting a class (qname_shad) within a TD that was already selected.  What you should be doing is starting with the TR since the class is on the TD.  It would look something like:

xray('http://www.barchart.com/quotes/stocks/$ONE', 'table#main-content div table table tr', [
	{
		name: '.qname_shad',
		value: '.qval_shad'
	}
])
(function(err, obj) {
	console.dir(obj);
});

This loops over each TR and finds a .qname_shad and .qval_shad.  However some TR elements have more than one .qname_shad:

	High
	920.07
	Low
	911.27

And other .qname_shad elements don't have .qval_shad at all:

	20-Day Average
	  915.62

Therefore x-ray isn't suitable because your selector doesn't fit a pattern.  You could perform subrequests or parse the results in cheerio, but the default methods will not resolve arrays within arrays or inconsistent class naming.  Regardless, this is not an issue with x-ray.


The install path for Google Chrome is hard-coded

The mdfind kMDItemCFBundleIdentifier = 'com.google.Chrome' can be used to locate where Chrome is installed, in case it is a non-standard location.

Another command that looks helpful is lsappinfo list

Satisfied by #109


anchor heatmap did not run

Hi, I have encountered a problem in running SPLASH in heatmap mode. I have run SPLASH successfully and I want to visualise the results using heatmap. My input was:
nextflow run kaitlinchaung/nomad -r main -latest --input sheet.csv --element_annotations_samplesheet ../samplesheet.csv --run_anchor_heatmaps true --results_dir <results_dir> --use_heatmap_anchor_list true --heatmap_anchor_list anchor.tsv
but the pipeline ends in seconds and I saw no update the result folder:
[d8/537650] process > RUN_NOMAD:ANCHOR_HEATMAPS (h... [100%] 1 of 1 ✔     <results_dir>  -[salzmanlab/nomad] Pipeline completed successfully-
I also tried to add a command that should induce an error message:
nextflow run kaitlinchaung/nomad -r main -latest --input sheet.csv --element_annotations_samplesheet ../samplesheet.csv --run_anchor_heatmaps true --results_dir <results_dir> --use_heatmap_anchor_list true --heatmap_anchor_list anchor.tsv **--additional_summary test**
but again the pipeline run without an error but gave no output. Could you please help with this?

Hi @chstellar,
The anchor heatmaps should be output in "${params.results_dir}/anchor_heatmaps". It won't be located in ${params.outdir}.
Also, we recommend using the newer version of the codebase here: https://github.com/refresh-bio/SPLASH


Packages on npm are missing documentation

For example, https://www.npmjs.com/package/@oasislabs/client:
"Unable to find a readme for<EMAIL_ADDRESS>
Keywords
none

Also link to GitHub is missing.

The solution here is to add separate READMEs for each subpackage.

It can also be something very basic, just linking to the main documentation.
GitHub link is also very important.

Addressed by https://github.com/oasislabs/oasis.js/pull/285.

I think important things reported here are still missing:

Link to GitHub repositories for packages are missing.
Keywords are missing.

Also, the link to main README does not work on NPM.


JSON Parse Error: Unrecognised token '<'

This error has started showing up since this morning (15/June). I have not changed the script in any way. Was working great till yesterday. Is it something I did on my phone?

Resolved by itself


Cannot install on Node 0.8.x because npm doesn't understand ^1.0.0

Trying to install tap 0.6.0 in Node 0.8.x doesn't work since the new dependencies require<EMAIL_ADDRESS>which apparently that version of npm doesn't understand?
npm ERR! Error: No compatible version found<EMAIL_ADDRESS>npm ERR! Valid install targets:
npm ERR! ["0.0.0","0.1.0","0.1.1","0.1.2","0.2.0","0.2.1","0.2.2","1.0.0"]
npm ERR!     at installTargetsError (/home/travis/.nvm/v0.8.28/lib/node_modules/npm/lib/cache.js:719:10)
npm ERR!     at /home/travis/.nvm/v0.8.28/lib/node_modules/npm/lib/cache.js:641:10
npm ERR!     at saved (/home/travis/.nvm/v0.8.28/lib/node_modules/npm/node_modules/npm-registry-client/lib/get.js:138:7)
npm ERR!     at Object.oncomplete (fs.js:302:15)
npm ERR! If you need help, you may report this log at:
npm ERR!     <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR<EMAIL_ADDRESS>
Example build: https://travis-ci.org/fastest963/ChainLoading/jobs/50595075
Thanks!

+1

Use a more recent version of npm. npm i -g npm


Multiple targets revisited

Hi,
I have read #22 and #24, but I believe that multiple targets support (without decision making) is still makes sense.
The problem is that dpinger creates 4 threads for single host (+ main thread), and I believe that this is a waste of resources and context switches if one has to monitor (say) 100 targets, and even for 10 targets this is a little bit too much, IMHO, especially on embedded systems (async I/O based on libev, for instance, would be more appropriate, but this is another story).
The idea is just to allow multiple targets to be processed by single instance of dpinger, and every report has to be prefixed with target ip/name.
So, the question is - do you have any interest to implement this feature?

I have considered this, but I honestly don’t find the thread argument compelling.
Dpinger creates a recv thread, a send thread, and a thread to support each type of reporting activity requested. In the default configuration there are 4 threads: main, send, recv, and report.
There really are no excess context switches. The send and recv threads spend 99.9% of their time on the sleep queue. They wake up, execute a system call and a small number of instructions (much less than a quantum), then return to the sleep queue. The only context switch is the system call and sleep queue transition. These are inherent and unavoidable.
Similarly, the report thread spends 99.9% of it’s time on the sleep queue. It wakes up, executes a relatively small number of instructions (again less than a quantum), followed by 1-3 system calls depending upon the IO implementation, and then returns to the sleep queue.
The main thread spends 100% of its time on the sleep queue until exit.
In short, thread context switching itself is not an issue, regardless of whether it is an embedded system or not. And I would not expect the total number of threads is much of an issue either. I have a few embedded systems lying around, of which my firewall is a good example… It has around 500 threads in normal operation. My NAS runs over 400 threads. And even my barest of bare bone systems, which runs nothing but NTP, has 100 threads.
Dpinger is designed to monitor reliability and latency on communication links. I.E. WAN connections. The need to monitor fifty or more targets is something that I would view as a less than 1% use case. The 95% use case is one or two links. Even with monitoring both IPv4 and IPv6, this gives 4 instances of dpinger, each of which spends almost 100% of its time on the sleep queue. I really don’t see this as a thread resource issue.
In addition to the above, it should be noted that changing dpinger to handle multiple targets per process raises other issues:

Multiple targets per process requires a hash to index the targets and associate the echo reply with the echo request. This means an increased CPU requirement for each packet sent/received. Alternatively, the association could be encoded in the data payload, but this would increase the minimum send/recv packet size which is even less desirable.

The report/socket threads have to include the identity of the targets in the output, requiring the reader to parse this for each and every report. This is undesirable, particularly since the reader has already effectively done this parsing in order to determine which file or socket to open.

Either all the targets must use the same parameters, intervals, thresholds, and such, or a configuration file must be introduced. And if different intervals are to be supported, then the additional cost and complexity of an event manager is required.

Restarting the dpinger process to change parameters for a target results in loss and rebuild of state for all targets rather than just the one target. This is seldom desirable.

While some of these tradeoffs might be appropriate for a use case requiring the monitoring of hundreds of targets, for tens of targets they are not appropriate. I don’t want to penalize the 99% in order to optimize for the 1%. This path leads to apinger.

Just for grins, I spun up a hundred dpinger instances on my little NTP system with one second send and reporting intervals. Total CPU utilization for the box was 8%.

Denny, thank you for such detailed explanation. You make good arguments, and if you want to keep dpinger simple and for small amount of targets (and low packet rates) - ok, this makes sense.
As to context switches - yes, they are insignificant if you have only few targets and 1 pps rate, but things will surely change if you have tens of targets and at least 10 pps rates - and please do not forget about related interrupts generated in process of handling all those packets.
If you increase reporting interval, the report thread will consume significantly more CPU - and on small devices (like RPi) this is not something that you can ignore.
Just try 10 pps in your test with 100 instances - and check the CPU utilization (especially kernel).

The test I ran was 100 instances with a 1 second probe interval. This is already 100 pps in and out, or 200 pps aggregate. As noted, total CPU, interrupt, system, and user time combined was 8%.
It should be noted that the concept of interrupt handling cost resulting from N pps is actually unrelated to the discussion of multiple target support. Interrupt handling costs are essentially a constant regardless of whether dpinger is running as 1 process or 100.
The 100 target test was only of theoretical value as I don't have 100 WAN links to monitor, nor do I know of anyone who does. And if someone does have 100 links to monitor, I seriously doubt that they would be using a small embedded system to do it with.
The frequency of reporting is also unrelated to the discussion of multiple target support. The CPU cost for calculating the average latency and loss is determined by the size of the array it is processing and the frequency with which it is done. This is the same whether dpinger is running as 1 process or 100.
On a general note, why would you Increase the reporting frequency to more than once per second? Many people decrease the reporting frequency to several seconds rather than increasing it. Many also decrease the probe frequency as well. Their comments usually come in the form of "Seriously, how frequently do you need an update on the average latency/loss of a WAN link?"

And if someone does have 100 links to monitor, I seriously doubt that they would be using a small embedded system to do it.

Not everyone could afford powerful system for every situation, sometimes peope have to work with what they already have.

On a general note, why would you Increase the reporting frequency to more than once per second?

Because with 1pps I do not get any reliable link quality measurement over 10s interval - this is only 10 packets, after all. I have links where where I could detect issues only with 10+ pps.
For simple cases, like "host up/down/significantly disrupted link" 1 pps is (probably) ok, but in my case disruptions are so short and irregular that 1 pps will simply miss them. For regular communications (web, mail etc) this is something that you will never feel, but for near-reatime applications (VoIP, telemetry) this makes a huge difference.
Though, since my use case does not fit even in 1% - please forget about my request, and thank you for your time. I just thought that optimizing and extending dpinger for such "advanced" usage would be more popular.

On a general note, why would you Increase the reporting frequency to more than once per second?

Because with 1pps I do not get any reliable link quality measurement over 10s interval - this is only 10 packets, after all. I have links where where I could detect issues only with 10+ pps.

I think you are conflating reporting interval (-r) with send interval (-s) and time period (-t). See dpinger usage statement and NOTES.md for an explanation.
Remember that dpinger is reporting average latency and loss over time rather than individual packets. Even if you have short term loss, it will be averaged over the time period when a report is generated.
If you would like help in designing a configuration that will work in your usage case, please contact me via email and I will do my best to help.
As these issues are unrelated to the single vs. multi process question, I am closing this issue.

FWIW, I ran a test based on my (admittedly minimal) understanding of your use case. 100 instances of dpinger each with the following parameters: -s 100m -t 20s -r 5s. In English, this is each instance conducting 10 probes per second, results averaged over 20 seconds, and status reported every 5 seconds. This gives a loss resolution of approximately .5%. On my NTP system this results in a CPU utilization of around 65%. Upping the report interval to 1s takes it to around 70%.
While CPU wise this should not be a problem even for a Pi, you may run into a problem with IO. 100 instances at 10 pps each is almost a megabit per second of ICMP traffic. That may be asking a bit much from a single Pi. You may want two of them. :)
Please contact me via email if you want to explore more.


[ISSUE #4749] make topic length , consumer group length configable

Make sure set the target branch to develop
What is the purpose of the change
fix https://github.com/apache/rocketmq/issues/4749
Brief changelog
XX
Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

 void setMaxTopicLength(int maxLength);
    int getMaxTopicLength();

IMO,  Put those two method into ClientConfig will be better. MQConsumer,LitePullConsumer and MQConsumerInner contains code about send/consume message only.  Set/get config should use ClientConfig.

I also think that it is a very good choice to put configuration-related variables in ClientConfig. At the same time, it is necessary to consider how to deal with the inconsistency of the MAX_TOPIC_LENGTH between the producer and the consumer.

@francisoliverlee following test would not throw exception
AbstractMessingActivityTest.testValidateTopic()
        assertThrows(GrpcProxyException.class, () -> messingActivity.validateTopic(Resource.newBuilder().setName(createString(128)).build()));


Facility_type_Processing_type for OS Hub

Overview (state of the OAR world):
In OAR, since all facilities are within the apparel sector, there is an established taxonomy for mapping processign type values to a facility type. Regardless of whether facility type is contributed, if there is a processing value that matches our taxonomy, the associated value for facility type will be displayed on the profile under the facility type field. Any values contributed as part of processing_type_faciltiy_type that do not match our taxonomy are displayed under the processing type only.
At the start of OS Hub Beta, this logic still applied, but since facilities were outside of apparel all values contributed only displayed under processing type.
Solution
On OS Hub:

For facilities contributed as apparel sector, the OAR logic for processing_type_faciltiy_type is applied.
For facilities contributed as any other sector, values contributed in the processing_type_facility_type field should be accepted and displayed under processing type.

@mariel-oar Contributors can specify more than one sector. As an example, if a contributor specifies Apparel and Electronics (or any number of non-Apparel sectors), should we use the OAR logic for the parsing the facility_type_processing_type values?
Tagging client action needed since wee need an answer to this question to build the feature

yes, if apparel is a sector value that's being contributed, then parse all processing_type_facility_type values as they are for the apparel sector; else leave them tagged as processing type.

@mariel-oar Two questions to confirm my current understanding:

Should all items submitted with processing_type_facility_type (on non-apparel items) be labeled as processing_type?
Should items submitted with processing_type_facility_type be matched to the processing_type taxonomy, or should they be saved as raw data without formatting? (The downsides of just using the raw data are that it won't necessarily be formatted well when it displays on facility details, and that these values won't be available in the Processing Types filter.)

@mariel-oar I think that actually answers the question - if the processing types in the taxonomy definitely don't apply for the non-apparel items, attempting to match to them doesn't make sense, so we should just use the raw data.
E.G., if someone submits 'seam tape' for a 'Home Furnishings' sector item, we will assign the processing_type as 'seam tape' without trying to match it to the Apparel taxonomy's 'Seam Taping'.


Create TrappingRainWater.cpp

Added solution for LeetCode Hard problem: https://leetcode.com/problems/trapping-rain-water/

Hey Raghavarya2002, I am Rahul Verma, a third year B. Tech. student from DTU. I am a competitive programmer (2000+ on Codeforces and 2200+ on CodeChef), you can check about me more on LinkedIn (ID: vrintle)

pls give proper description about problem

pls give proper description about problem

Please check the updated pr

pls merge all in single file and i will commit it

good effort


Add support for preemptive basic auth

From<EMAIL_ADDRESS>on March 22, 2011 16:21:55
Groovy impl:
def http = new HTTPBuilder('http://myhost.com'
http.headers[ 'Authorization' ] =
"$user:$pass".getBytes('iso-8859-1').encodeBase64()
Original issue: http://code.google.com/p/rest-assured/issues/detail?id=45

From<EMAIL_ADDRESS>on March 22, 2011 11:06:57
Status: Fixed


SOMA-collection sketching [WIP]

I'm splitting this once-big, now-shrinking PR up into more easily reviewable pieces. Once all those pieces are created, reviewed, accepted, and merged, and as I rebase this PR atop them, eventually this PR will be zero-length and will be closed.
Context: #95

#157 replaces this.


Add files via upload

Html Full Course Is Coming Soon Please follow my Github Account and learn more thing . I upload  lectures on daily basis


Please retain the Cellpose copyright as required by the BSD-3 license

The Cellpose license (BSD-3) is very permissive. However, it requires you to retain the copyright notice here when you copy portions of the code.
The copyright notice must be maintained and distributed with every copy of your codebase.
In addition, we ask that you include this copyright statement at the top of your file:
"""
Copyright © 2022 Howard Hughes Medical Institute, Authored by Carsen Stringer and Marius Pachitariu.
"""
Thank you.

Thank you for the clarification regarding the BSD-3 license requirements. We have updated the LICENSE file to include the copyright notice and proper attribution for the portions of code We’ve used from the Cellpose project.

Great, thanks!


Fix bug allowing validation of bool types with coerce_numbers_to_str=True

Change Summary
Setting the coerce_numbers_to_str flag shouldn't allow the validation of bool types, so this is fixed here. Though bool is a subclass of int we didn't specify (nor intend, to the best of my understanding) for this behavior to be allowed.
Related issue number
Fix https://github.com/pydantic/pydantic/issues/7820
Checklist

[x] Unit tests for the changes exist
[ ] Documentation reflects the changes where applicable
[x] Pydantic tests pass with this pydantic-core (except for expected changes)
[x] My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

Please review

I think the failing tests are just xfail tests waiting on the newest version of pydantic-core to be released - perhaps we could do that with the next pydantic minor release @samuelcolvin?


[SPARK-38999][SQL] Refactor FileSourceScanExec: file scan physical node

What changes were proposed in this pull request?
The PR refactors FileSourceScanExec case class into a base trait FileSourceScanLike which is then subclassed by FileSourceScanExec. FileSourceScanLike contains basic functionality like metrics and file listing while the FileSourceScanExec contains execution specific code.
Why are the changes needed?
Currently the code for FileSourceScanExec class, the physical node for the file scans is quite complex and lengthy making it slightly difficult to reason about.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Code refactor, existing tests should suffice.

@cloud-fan @gengliangwang Can you please help review this?

Can one of the admins verify this patch?

thanks, merging to master!


Update text errors from README.md

Hyphens are necessary for compound adjectives

@jorgesumle  Thanks for your first contribution to git-quick-stats!  You rock!


lpcxpresso55s69: Support for the flash hardware is broken

Describe the bug
If to enable the FLASH support for lpcxpresso55s69 or lpcxpresso55s16 , an application is stuck and never reaches its main() function.
To Reproduce
Steps to reproduce the behavior:

Use zephyr\samples\basic\blinky sample example.
Enable the FLASH hardware support in the application, by adding CONFIG_FLASH=y to the project configuration.
Build and run it for lpcxpresso55s69_cpu0 or lpcxpresso55s16.
The application is stuck and never reaches its main() function.

If to delete CONFIG_FLASH=y, the application is functional as expected.
Expected behavior
Enabling of FLASH should not break the application functionality.
Impact
Any application, that requires the FLASH support, is not functional.
Logs and console output
N/A
Environment
OS: Windows
Toolchain: Zephyr SDK 0.16.1
Commit: bf830ba780b8a76bf8962857b72b551c41f628ee Date: 9/22/2023
Additional context
N/A

I've bisected this issue down to c6e3bac4f20a659. It appears the LPC55S69 requires the CPU core clock be less than 100MHz when flash operations are being performed:

(section 5.3 of the LPC55S69 reference manual)

The #62963 is flash initialization issue. No write/erase is done yet.

@butok This is a good point- however, I see flash API tests able to pass prior to c6e3bac. I wonder if the issue is that we call the flash_init ROM API during flash init? Perhaps the ROM code there runs into issues if the clock is over 100MHz. The SDK FLASH_Init function seems to suggest 96MHz is expected: https://github.com/zephyrproject-rtos/hal_nxp/blob/master/mcux/mcux-sdk/drivers/iap1/fsl_iap.c#L237

Hi Daniel (@danieldegrasse )

I see flash API tests able to pass
What test application are you using?

The SDK FLASH_Init function seems to suggest 96MHz
It is true. Checked on practice for LPC55S69, it should be <100 for a flash write.
But for reading, it had no issue.

Now, I am working on LPC55S36 & FLASH & MCUboot. I have  a working prototype, I need to prepare/clean code for upstreaming and to add final improvements.
When it is finished (during next week), I will switch to this LPC55S69/16/06 issue.
BTW: 55S69/16/06 and 55S36 are too different.
Do we want to split them to separate source files to have cleaner code and to avoid repetitive #ifdef ?

What test application are you using?

tests/drivers/flash/common

BTW: 55S69/16/06 and 55S36 are too different.
Do we want to split them to separate source files to have cleaner code and to avoid repetitive #ifdef ?

Yes, we should probably split the SOC initialization for the LPC5536 into another file, if that is what you mean.

Tested on my side, on LPC55S16/69.
Confirm, it is fixed.


[CSL-44] [Research] Address structure

@georgeee
From Charles:
For address structure, let's do a double hash with sha3 and then blake2 with checksum in a human readable base58 format
we also have to consider the idea of address compression. For example, we could potentially have a large amount of addresses aggregate to a single address via a merkle tree
this would be relevant for redemption from RSCoin to Cardano SL
https://en.bitcoin.it/wiki/Technical_background_of_version_1_Bitcoin_addresses
Have your guys review this document to gain a fuller understanding of addresses
I'd like to repeat using two hashes in the process of address generation
it's a client side cost and doesn't impact the core protocol so it's worth spending
let me know about checksum preference
as for address format
human readable addresses is a vogue topic in bitcoin land
I'd like to see some ideas about what is possible

@georgeee
BIP: [13|https://github.com/bitcoin/bips/blob/master/bip-0013.mediawiki]
Title: Address Format for pay-to-script-hash

@georgeee
P.S. AFAIK multisigs are to be handled by scripts. Consult w/ @neongreen

@georgeee
Also consider multisigs: we don't need them now, but imagine for 5-10m how hard it would be to add them to CSL
https://github.com/bitcoin/bips/blob/master/bip-0011.mediawiki

@flyingleafe
https://docs.google.com/document/d/1PKiRVYiJe8rKW78hUc59EFA9qWP2juPsGkXP2DJcjCg/edit?usp=sharing


Add an integration test for VNC + websockify

This tests the following components:

The VNC server
websockify
Jupyter Server Proxy
The desktop environment itself

It does so by:

Building the container image
Starting a container with that image, with jupyter server running
inside.
Connecting to the container via websockets, authenticating and
exposing the websocket connection as a regular TCP socket on
localhost via websocat
Taking a screenshot of the remote desktop by connecting to it
through vncdotool. This requires a tcp connection, hence the
double proxying with websocat
Verify that the image produced by the screenshot is the same as a
reference image we have.

This only misses the noVNC + JS from testing, but otherwise represents
a massive improvement to the status quo!

This works perfectly fine locally, but unfortunately something is causing it to hang when the VNC client is trying to connect to the server on GitHub actions. will have to debug next time i get a chance.

Not needed, now that we have #119


adding new release action to send release information to pyxis

Info

Adding new workflow to send release information to pyxis.

This workflow does not get triggered on pre-releases (ie the checkbox on the release page). This will allow us to release a alpha/beta release that we can test in lower environments, but is not enabled/enforced by pyxis.

Tests

Success of a release

Prerelease getting ignored

coverage: 81.752%. remained the same
when pulling af2971b7e0bb6aea11f26dd09c77e9cdf58e2028 on acornett21:update_release_action
into<PHONE_NUMBER>11035b05e5bc7e228a36557269a562 on redhat-openshift-ecosystem:main.

/ok-to-test

/test 4.15-e2e

/test 4.15-e2e


Creating new multi tenant environment using this script is broken

[Enter feedback here]
This is very broken. Creating an empty database and mounting it to the app database gives an error about missing tables... But it's an empty database right?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 369ef361-8740-4777-2f6d-32e73df71e43
Version Independent ID: e2d76464-650f-3a97-d15c-f6d50245bfa4
Content: Manage Application and Tenant Databases in a Multitenant Deployment - Business Central
Content Source: dev-itpro/administration/multitenant-setup-guide.md
Service: dynamics365-business-central
GitHub Login: @jswymer
Microsoft Alias: solsen

Hello again,
I followed the instructions in the article and didn't run into any problems. The issue mentions broken script, but the article doesn't explicitly include a script, so perhaps I misunderstood something. Would it be possible to provide a bit more information pointing to where the issue occurred? Thanks

Hi there @jswymer,
I created a new multi-tenant environment and wanted to make a new tenant.
The web-page states you can create an empty database, mount it and sync it and it should be working.
Though mounting an empty database does not work as it gives an error.
In this instance I meant the 4 steps on the webpage when using the word 'Script'.

Hello,
Sorry, I'm not sure what the issue could be. More detailed steps can be found here, although I'm not sure this will solve the problem.
[Create databases]https://docs.microsoft.com/en-us/dynamics365/business-central/dev-itpro/deployment/devenv-create-databases#tenant)
It could something to contact support about. If there is a documentation issue, we'll be notified.

Hello again, I just wanted to follow up on this issue. I tried to repro it on version 19.2, but I didn't get the error. I recommend that you contact support, so they can better analyze the issue. Sorry we couldn't help more. Kind regards.


Add more tests to MarkedSource rendering in Java and fix bugs [phab:D1985]

Created by zarko at 2017-10-20 22:58:06:

schroederc accepted this revision at 2017-10-20 23:01:37

zarko closed via commit aa6de13fb72c5b8c22cd53ff35f5ee1edd4e16d4 at 2017-10-20 23:10:35


Flag to turn on/off tracing task spend time

When I apply plugin com.archinamon:android-gradle-aspectj to my project automatically starts task spent time tracing:
122 actionable tasks: 7 executed, 115 up-to-date
Task spend time:
     58ms       :presentation:compileDebugJavaWithJavac
    330ms       :app:mergeDebugResources
    975ms       :app:processDebugResources
   1229ms       :app:packageDebug

If I correctly analyze source of plugin that behavior implemented in class https://github.com/Archinamon/android-gradle-aspectj/blob/master/android-gradle-aspectj/src/main/kotlin/com/archinamon/api/BuildTimeListener.kt
I don't need this tracing in everytime. Maybe this helpfull when error occurs but not always.
I think plugin should support flag to turn on this tracing and turn it off by default.

Hi!
It already has this option:
buildTimeLog true // default value' in the aspectj` extension closure in build.gradle file.

I know about this flag but when I add it to all my modules with aspectj plugin to build.gradle file there is no change. Lines with time still printing:
...

> Task :app:packageRelease
:app:packageRelease spend 2039ms
:app:packageRelease spend 2039ms
:app:packageRelease spend 2039ms
:app:packageRelease spend 2039ms

...

BUILD SUCCESSFUL in 40s
299 actionable tasks: 91 executed, 208 up-to-date
Task spend time:
     51ms       :app:fabricGenerateResourcesDebug
   1958ms       :presentation:packageDebugResources
    306ms       :presentation:mergeDebugResources
    663ms       :presentation:processDebugResources
    508ms       :app:mergeDebugResources
    993ms       :app:processDebugResources
    658ms       :presentation:kaptGenerateStubsDebugKotlin
   8853ms       :presentation:kaptDebugKotlin
    158ms       :presentation:compileDebugKotlin
   2659ms       :presentation:compileDebugJavaWithJavac

...

I doing it right? Or I should write something like
allProjects {
    if (enabledPlugins.contains('com.archinamon.aspectj')) {
        aspectj {
            buildTimeLog false
        }
    }
}

in my root build.gradle file?

Try flag on empty project with applied plugin (no aspects added).
Without flag:
...

> Task :app:processReleaseJavaRes
:app:processReleaseJavaRes spend 0ms

> Task :app:transformResourcesWithMergeJavaResForRelease
:app:transformResourcesWithMergeJavaResForRelease spend 121ms

> Task :app:packageRelease
:app:packageRelease spend 761ms

> Task :app:assembleRelease
:app:assembleRelease spend 1ms

> Task :app:assemble
:app:assemble spend 0ms

BUILD SUCCESSFUL in 10s
55 actionable tasks: 22 executed, 33 up-to-date
Task spend time:
    835ms       :app:compileDebugKotlin
    375ms       :app:compileDebugJavaWithJavac
    581ms       :app:transformClassesWithAspectjForDebug
    311ms       :app:transformClassesWithDexBuilderForDebug
    601ms       :app:transformDexArchiveWithDexMergerForDebug
   1264ms       :app:transformNativeLibsWithMergeJniLibsForDebug
    529ms       :app:transformResourcesWithMergeJavaResForDebug
   1317ms       :app:packageDebug
     83ms       :app:mergeReleaseResources
   1315ms       :app:compileReleaseKotlin
    179ms       :app:compileReleaseJavaWithJavac
    258ms       :app:lintVitalRelease
    133ms       :app:transformClassesWithAspectjForRelease
    134ms       :app:transformClassesWithPreDexForRelease
    328ms       :app:transformDexWithDexForRelease
    122ms       :app:transformNativeLibsWithMergeJniLibsForRelease
    121ms       :app:transformResourcesWithMergeJavaResForRelease
    761ms       :app:packageRelease

With flag (false value):
...

> Task :app:transformNativeLibsWithStripDebugSymbolForRelease
:app:transformNativeLibsWithStripDebugSymbolForRelease spend 2ms

> Task :app:processReleaseJavaRes
:app:processReleaseJavaRes spend 0ms

> Task :app:transformResourcesWithMergeJavaResForRelease
:app:transformResourcesWithMergeJavaResForRelease spend 10ms

> Task :app:packageRelease
:app:packageRelease spend 13ms

> Task :app:assembleRelease
:app:assembleRelease spend 1ms

> Task :app:assemble
:app:assemble spend 0ms

BUILD SUCCESSFUL in 1s
55 actionable tasks: 4 executed, 51 up-to-date
Task spend time:
    251ms       :app:lintVitalRelease

Less lines but :app:lintVitalRelease still printed don't know why.

Nope, you're implementing this config in a wrong place. You have to put it into the concrete project's build.gradle file on top level (only if-closures allowed).
Example:
if (isAspectedBuild()) {
    apply plugin: 'com.archinamon.aspectj'

    aspectj {
        buildTimeLog false
    }
}


Swift throws "Use of undeclared type 'Note' " Error

var _fetchedResultsController: NSFetchedResultsController<Note>? = nil
Note is not referenced before, so the compiler cannot resolve it.

Nim104. We have completely refactored the starter app and it no longer uses the NSFetchedResultsController and Core Data has been removed. Please download or clone the latest and let us know if you have any issues.


verify_headless_installation fails

Hello,
I noticed that running fails with the following error:
(.venv) michel@debian:/tmp$ python -m metadrive.examples.verify_headless_installation
[INFO] Environment: MetaDriveEnv
[INFO] MetaDrive version: <IP_ADDRESS>
[INFO] Sensors: [lidar: Lidar(), side_detector: SideDetector(), lane_line_detector: LaneLineDetector()]
[INFO] Render Mode: none
[INFO] Horizon (Max steps per agent): None
[INFO] Assets version: <IP_ADDRESS>
[INFO] Known Pipes: glxGraphicsPipe
[INFO] Start Scenario Index: 0, Num Scenarios : 1
[INFO] Episode ended! Scenario Index: 0 Reason: out_of_road.
Bullet physics world is launched successfully!
[INFO] Environment: MetaDriveEnv
[INFO] MetaDrive version: <IP_ADDRESS>
[INFO] Sensors: [lidar: Lidar(), side_detector: SideDetector(), lane_line_detector: LaneLineDetector(), rgb_camera: RGBCamera(256, 256), depth_camera: DepthCamera(256, 256)]
[INFO] Render Mode: offscreen
[INFO] Horizon (Max steps per agent): None
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/examples/verify_headless_installation.py", line 11, in <module>
    verify_installation(args.cuda, args.camera)
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/tests/test_installation.py", line 84, in verify_installation
    capture_headless_image(cuda)
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/tests/test_installation.py", line 14, in capture_headless_image
    env = MetaDriveEnv(
          ^^^^^^^^^^^^^
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/envs/metadrive_env.py", line 103, in __init__
    super(MetaDriveEnv, self).__init__(config)
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/envs/base_env.py", line 299, in __init__
    self.agent_manager = self._get_agent_manager()
                         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/envs/base_env.py", line 396, in _get_agent_manager
    return VehicleAgentManager(init_observations=self._get_observations())
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/envs/base_env.py", line 393, in _get_observations
    return {DEFAULT_AGENT: self.get_single_observation()}
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/envs/base_env.py", line 661, in get_single_observation
    o = ImageStateObservation(self.config) if img_obs else LidarStateObservation(self.config)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/obs/image_obs.py", line 26, in __init__
    self.img_obs = ImageObservation(config, config["vehicle_config"]["image_source"], config["norm_pixel"])
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/obs/image_obs.py", line 61, in __init__
    self.state = np.zeros(self.observation_space.shape, dtype=np.float32 if self.norm_pixel else np.uint8)
                          ^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/obs/image_obs.py", line 67, in observation_space
    sensor_cls = self.config["sensors"][self.image_source][0]
                 ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/utils/config.py", line 225, in __getitem__
    self._check_and_raise_key_error(item)
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/utils/config.py", line 211, in _check_and_raise_key_error
    raise KeyError(
KeyError: "'main_camera' does not exist in existing config. Please use config.update(...) to update the config. Existing keys: dict_keys(['lidar', 'side_detector', 'lane_line_detector', 'rgb_camera', 'depth_camera'])."

After switching to the rgb camera, I get a different error:
(.venv) michel@debian:/tmp$ python -m metadrive.examples.verify_headless_installation --camera rgb
[INFO] Environment: MetaDriveEnv
[INFO] MetaDrive version: <IP_ADDRESS>
[INFO] Sensors: [lidar: Lidar(), side_detector: SideDetector(), lane_line_detector: LaneLineDetector()]
[INFO] Render Mode: none
[INFO] Horizon (Max steps per agent): None
[INFO] Assets version: <IP_ADDRESS>
[INFO] Known Pipes: glxGraphicsPipe
[INFO] Start Scenario Index: 0, Num Scenarios : 1
[INFO] Episode ended! Scenario Index: 0 Reason: out_of_road.
Bullet physics world is launched successfully!
[INFO] Environment: MetaDriveEnv
[INFO] MetaDrive version: <IP_ADDRESS>
[INFO] Sensors: [lidar: Lidar(), side_detector: SideDetector(), lane_line_detector: LaneLineDetector(), rgb_camera: RGBCamera(256, 256), depth_camera: DepthCamera(256, 256)]
[INFO] Render Mode: offscreen
[INFO] Horizon (Max steps per agent): None
[INFO] Assets version: <IP_ADDRESS>
Xlib:  extension "XFree86-DGA" missing on display ":0".
[INFO] Known Pipes: glxGraphicsPipe
[WARNING] You are using too large buffer! The height is 256, and width is 256. It may lower the sample efficiency! Consider reducing buffer size or use cuda image by set [image_on_cuda=True]. (base_camera.py:46)
[WARNING] You are using too large buffer! The height is 256, and width is 256. It may lower the sample efficiency! Consider reducing buffer size or use cuda image by set [image_on_cuda=True]. (base_camera.py:46)
Assertion failed: _current_shader_context != nullptr at line 6939 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx
Traceback (most recent call last):
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/component/sensors/depth_camera.py", line 142, in _dispatch_compute
    self.engine.graphicsEngine.dispatch_compute(
AssertionError: _current_shader_context != nullptr at line 6939 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/examples/verify_headless_installation.py", line 11, in <module>
    verify_installation(args.cuda, args.camera)
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/tests/test_installation.py", line 86, in verify_installation
    capture_headless_image(cuda, image_source="rgb_camera")
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/tests/test_installation.py", line 33, in capture_headless_image
    env.reset()
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/envs/base_env.py", line 519, in reset
    self.lazy_init()  # it only works the first time when reset() is called to avoid the error when render
    ^^^^^^^^^^^^^^^^
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/envs/base_env.py", line 406, in lazy_init
    initialize_engine(self.config)
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/engine/engine_utils.py", line 12, in initialize_engine
    cls.singleton = cls(env_global_config)
                    ^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/engine/base_engine.py", line 88, in __init__
    self.warmup()
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/engine/base_engine.py", line 741, in warmup
    self.taskMgr.step()
  File "/tmp/.venv/lib/python3.11/site-packages/direct/task/Task.py", line 504, in step
    self.mgr.poll()
  File "/tmp/.venv/lib/python3.11/site-packages/metadrive/component/sensors/depth_camera.py", line 142, in _dispatch_compute
    self.engine.graphicsEngine.dispatch_compute(
AssertionError: _current_shader_context != nullptr at line 6939 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx

For the first error, you should explicitly declare that you need the main_camera as a sensor by putting it in the sensor config:
"sensors":{"main_camera": ()
    ...
}

So you can access it even in offscreen mode.
The second error is raised by the depth camera because the GPU doesn't support OpenGL > 4.0 and the compute shader is not available. It usually happens for users using an integrated graphics card.

My GPU should support OpenGL 4.6. Is there any specific extension that is required that I could check for?

Sorry for the late reply. You would be better to try some compute shader that is irrelevant to MetaDrive and Panda3D to figure out whether it is the OpenGL/hardware problem. Then we can do further debugging on Panda3D and MetaDrive side.