CUDA out of memory upon initiating neuralangelo training

I am a NeRF, conda and python noob, so go easy on me, lol.
I am trying to use sdfstudio to train a custom dataset.  The dataset consists of 90 .jpg images with resolution 2028x1520.  I processed the images through colmap successfully (from within sdfstudio), but I get constant "CUDA out of memory" errors when I attempt to train the model.    I tried training the model directly using the following command (note that the command includes changes to several parameters, i.e. rays per batch and rays per chunk, that I hoped might allow the process to complete):

ns-train neuralangelo --pipeline.datamanager.train-num-rays-per-batch=32 --pipeline.datamanager.eval-num-rays-per-batch=32 --pipeline.model.sdf-field.inside-outside False --vis viewer --viewer.num-rays-per-chunk=512 --experiment-name neuralangelo-xfrmr-small nerfstudio-data --data C:\Users\fjsch\sdfstudio\outputs\xfrmr-small

Here is the error I received:

Setting up training dataset...
Caching all 81 images.
Setting up evaluation dataset...
Caching all 9 images.
No checkpoints to load, training from scratch
Printing profiling stats, from longest to shortest duration in seconds
VanillaPipeline.get_train_loss_dict: 0.7051
Traceback (most recent call last):
File "C:\Users\fjsch.conda\envs\sdfstudio\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\fjsch.conda\envs\sdfstudio\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Users\fjsch.conda\envs\sdfstudio\Scripts\ns-train.exe_main.py", line 7, in
File "C:\Windows\System32\sdfstudio\scripts\train.py", line 250, in entrypoint
main(
File "C:\Windows\System32\sdfstudio\scripts\train.py", line 236, in main
launch(
File "C:\Windows\System32\sdfstudio\scripts\train.py", line 175, in launch
main_func(local_rank=0, world_size=world_size, config=config)
File "C:\Windows\System32\sdfstudio\scripts\train.py", line 90, in train_loop
trainer.train()
File "C:\Windows\System32\sdfstudio\nerfstudio\engine\trainer.py", line 151, in train
loss, loss_dict, metrics_dict = self.train_iteration(step)
File "C:\Windows\System32\sdfstudio\nerfstudio\utils\profiler.py", line 43, in wrapper
ret = func(*args, **kwargs)
File "C:\Windows\System32\sdfstudio\nerfstudio\engine\trainer.py", line 321, in train_iteration
self.grad_scaler.scale(loss).backward()  # type: ignore
File "C:\Users\fjsch.conda\envs\sdfstudio\lib\site-packages\functorch_src\monkey_patching.py", line 77, in _backward
return old_backward(*args, **kwargs)
File "C:\Users\fjsch.conda\envs\sdfstudio\lib\site-packages\torch_tensor.py", line 396, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "C:\Users\fjsch.conda\envs\sdfstudio\lib\site-packages\torch\autograd_init.py", line 173, in backward
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA out of memory. Tried to allocate 1.67 GiB (GPU 0; 8.00 GiB total capacity; 5.03 GiB already allocated; 366.00 MiB free; 5.87 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I have a 3060Ti video card with 8GB of VRAM and I have 32GB of system memory.  I am able to process and train models using far more higher resolution images just fine using plain old nerfstudio...I just can't get sdfstudio to work.  Any ideas???  Any input you may be able to provide would be greatly appreciated!

Try BakedSDF, Neuralangelo requires a lot of VRAM.

How do I reduce the number of training rays?

Hi, you can try to use a small hash grids with --pipeline.model.sdf-field.hash-features-per-level 2 --pipeline.model.sdf-field.log2-hashmap-size 19. Or you could try bakedangelo and reduce the number of training rays.

But it seems to slow down the convergence? I haven't try --pipeline.model.sdf-field.hash-features-per-level 8 --pipeline.model.sdf-field.log2-hashmap-size 22 which is the default settings, because my 32G VRAM is not enough. I am wondering why it will consume such a large amount of VRAM and whether the convergence is inherently slow?


Add support for workflow_call property for GitHub Actions

The GitHub Action renderer fails when a file contains a workflow_call property. It should be supported. This issue was  reported via Discord

Also make it possible to use something other than the main branch?


fix clone wiki failed via ssh

will fix #5291

Codecov Report

Merging #5503 into master will decrease coverage by <.01%.
The diff coverage is 5.26%.

@@            Coverage Diff             @@
##           master    #5503      +/-   ##
==========================================
- Coverage   37.63%   37.62%   -0.01%
==========================================
  Files         317      318       +1
  Lines       46826    46845      +19
==========================================
+ Hits        17624    17627       +3
- Misses      26701    26717      +16
  Partials     2501     2501

Impacted Files
Coverage Δ

routers/private/wiki.go
0% <0%> (ø)

routers/private/internal.go
55.73% <100%> (+0.73%)
:arrow_up:

modules/sync/unique_queue.go
78.57% <0%> (-10.72%)
:arrow_down:

models/repo_list.go
64.55% <0%> (+1.26%)
:arrow_up:

modules/process/manager.go
81.15% <0%> (+4.34%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7c0c965...c44fee8. Read the comment docs.

Does user need code rights for wiki?


Additional function that allows passing pthread_attr_t during thread pool initialization

Hello,
First of all, I'd like to say thank you for your library.
I want to propose extension of current thread pool initialization process in order to be able to pass pthread attribute to underneath threads which are created in a pool.
It might be done via additional method like:
const pthread_attr_t *internal_pattr_variable = NULL;
//...
void thpool_set_attr(const pthread_attr_t *pattr)
{
    internal_pattr_variable = pattr;
}

By doing so we can check whether our internal_pattr_variable is not NULL in the thread_init call and if so, apply the provided attributes to the brand new threads.
If you do not mind I can do that and send a pull request.

Closing this due to little traction


feature request: simplify host interface

I am not great with C/C++ so this may not be a reasonable expectation, but I think the WAMR host interface is simpler, as it is, than exposing a host-function with extism. Here is an example in wamr:
// host implementation, basically a thin-wrapper around null0_random_int that adds env param
static int32_t wamr_null0_random_int(wasm_exec_env_t exec_env, int32_t min, int32_t max) {
  return null0_random_int(min, max);
}

// later in setup
{"random_int", wamr_null0_random_int, "(ii)i"}

No raising/lowering of values into a shared-buffer, WAMR sees the function is (ii)i meaning "takes 2 i32 params, returns i32" and inserts those into the host function-call.
It does break down sometimes, like this function that returns a struct:
// Generate randomized preset sfxr params
static void wamr_null0_preset_sfx(wasm_exec_env_t exec_env, SfxParams* params, SfxPresetType type) {
  null0_preset_sfx(params, type);
}

{"preset_sfx", wamr_null0_preset_sfx, "(*i)"}

clang mangles the return in wasm, and inserts it as the first param, as a pointer in wasm-space. But otherwise, all the pointers in the host-function are already setup to work with the wasm memory, and so you just work with those pointers like they are plain old C pointers, in host-space.
I think it is similar in c-pdk, like to keep my simple user-api, I will need to wrap every function with extism stuff to raise/lower values into the buffer and then call user/host functions.
I am not good enough with C macros to know how to do it myself, but it'd be cool if I could just expose a function and have it work with no extra trouble, like this in host:
SfxParams null0_preset_sfx(SfxPresetType type) {
  // implemented here, just normal C function that can use pointers  that are pulled out alread
}

extism_expose_host(plugin, null0_preset_sfx, "null0.preset_sfx");

and this in wasm:
extism_import_host("null0.preset_sfx", SfxParams preset_sfx(SfxPresetType type));

so I can use the host-function without any wrapping or pulling/pushing pointers. Like here, it would look at null0_preset_sfx and turn it into extism stuff.
On host:

input is SfxPresetType in shared-mem buffer
output is SfxParams in shared-mem buffer
pull inputs into regular params from shared-mem buffer
call null0_preset_sfx with those regular params
take return and push into shared-mem buffer

In wasm:

input is SfxPresetType in shared-mem buffer
output is SfxParams in shared-mem buffer
when user calls preset_sfx, raise input, lower output, and return it

Here is another simple example:
In wamr
static void wamr_null0_trace(wasm_exec_env_t exec_env, char* str) {
  printf("%s\n", str);
}

// ... in setup function
{"trace", wamr_null0_trace, "($)"}

but in extism, I need to juggle the vars around:
auto trace_in = std::vector<extism::ValType>{extism::ValType::ExtismValType_I64};
  auto null0_host_trace = extism::Function(
      "trace",
      trace_in,
      {},
      [](extism::CurrentPlugin plugin, void* user_data) {
        cout << plugin.inputStringView(0) << endl;
      },
      NULL,
      NULL);

This isn't too bad, since I can just call plugin.inputStringView(0) to get the string val, but if I could just point it directly at the plain function it'd be much simpler. Also, on the wasm-side, it's more complicated, because I need to raise the memory first, so I need an extra layer if I want to keep it simple for the user (no handles and storing):
EXTISM_IMPORT("extism:host/user", "trace")
extern void _null0_trace_real(ExtismHandle);

// Log a string
void trace(const char* null0_traceBuffer) {
  int l = strlen(null0_traceBuffer);
  ExtismHandle handle = extism_alloc(l);
  extism_store_to_handle(handle, 0, null0_traceBuffer, l);
  _null0_trace_real(handle);
}

A more convenient host function interface would be nice, but I'm not sure how much can be done portability. Currently, I don't believe libextism exposes the linear memory of the plugin, but the plugin's (kernel) memory. This could probably be done with wasmtime and WAMR, but on the host side I'm not sure if the plugin's linear memory is always accessible, for example, in JS unless it's exported I don't think it is. Without this feature, the wasm must handle copying into the plugin's kernel memory unless it was exporting it's copy routines for the host function wrapper to use.
Still thought. on the host side potentially a host function interface could be created that allows the host function to take pointer parameters or string_view, etc. The old api might still be necessary for multi-value returns or maybe those can use std::tuple.
Please share if you experiment and get something working!


How to use this VAE on LCM

Thanks to open your work.
Btw, I ran the code:
from diffusers import DiffusionPipeline,StableDiffusionPipeline
import torch
from consistencydecoder import ConsistencyDecoder, save_image, load_image

pipe = DiffusionPipeline.from_pretrained("SimianLuo/LCM_Dreamshaper_v7", custom_pipeline="latent_consistency_txt2img", custom_revision="main", revision="fb9c5d")

decoder_consistency = ConsistencyDecoder(device="cuda:0") # Model size: 2.49 GB

pipe.to(torch_device="cuda", torch_dtype=torch.float32)

prompt = "Self-portrait oil painting, a beautiful cyborg with golden hair, 8k"

# Can be set to 1~50 steps. LCM support fast inference even <= 4 steps. Recommend: 1~8 steps.
num_inference_steps = 4

latent = pipe(prompt=prompt, num_inference_steps=num_inference_steps, guidance_scale=8.0, output_type="latent").images[0]

latent=latent.unsqueeze(0).to("cuda:0")

sample_consistency = decoder_consistency(latent)
save_image(sample_consistency, "con.png")

I got the image:

What is wrong?

you need to scale the latent using pipe.vae.config.scaling_factor

Is this right?
from diffusers import DiffusionPipeline,StableDiffusionPipeline
import torch
from consistencydecoder import ConsistencyDecoder
from PIL import Image
import numpy as np

pipe = DiffusionPipeline.from_pretrained("SimianLuo/LCM_Dreamshaper_v7", torch_dtype=torch.float32)

# To save GPU memory, torch.float16 can be used, but it may compromise image quality.
pipe.to(torch_device="cuda", torch_dtype=torch.float32)

prompt = "Self-portrait oil painting, a beautiful cyborg with golden hair, 8k"

# Can be set to 1~50 steps. LCM support fast inference even <= 4 steps. Recommend: 1~8 steps.
num_inference_steps = 4

decoder_consistency = ConsistencyDecoder(device="cuda:0") # Model size: 2.49 GB

# To save GPU memory, torch.float16 can be used, but it may compromise image quality.
pipe.to(torch_device="cuda", torch_dtype=torch.float32)

prompt = "Self-portrait oil painting, a beautiful cyborg with golden hair, 8k"

# Can be set to 1~50 steps. LCM support fast inference even <= 4 steps. Recommend: 1~8 steps.
num_inference_steps = 8

latent = pipe(prompt=prompt, num_inference_steps=num_inference_steps, guidance_scale=8.0, output_type="latent")

latent=latent.images[0]/0.18215
latent=latent.unsqueeze(0)
print(latent.size())
with torch.no_grad():
    consistent_latent = decoder_consistency(latent,schedule=[1.0])

image = consistent_latent[0].cpu().numpy()
image = (image + 1.0) * 127.5
image = image.clip(0, 255).astype(np.uint8)
image = Image.fromarray(image.transpose(1, 2, 0))

image.save("con.png")

Is this right?
from diffusers import DiffusionPipeline,StableDiffusionPipeline
import torch
from consistencydecoder import ConsistencyDecoder
from PIL import Image
import numpy as np

pipe = DiffusionPipeline.from_pretrained("SimianLuo/LCM_Dreamshaper_v7", torch_dtype=torch.float32)

# To save GPU memory, torch.float16 can be used, but it may compromise image quality.
pipe.to(torch_device="cuda", torch_dtype=torch.float32)

prompt = "Self-portrait oil painting, a beautiful cyborg with golden hair, 8k"

# Can be set to 1~50 steps. LCM support fast inference even <= 4 steps. Recommend: 1~8 steps.
num_inference_steps = 4

decoder_consistency = ConsistencyDecoder(device="cuda:0") # Model size: 2.49 GB

# To save GPU memory, torch.float16 can be used, but it may compromise image quality.
pipe.to(torch_device="cuda", torch_dtype=torch.float32)

prompt = "Self-portrait oil painting, a beautiful cyborg with golden hair, 8k"

# Can be set to 1~50 steps. LCM support fast inference even <= 4 steps. Recommend: 1~8 steps.
num_inference_steps = 8

latent = pipe(prompt=prompt, num_inference_steps=num_inference_steps, guidance_scale=8.0, output_type="latent")

latent=latent.images[0]/0.18215
latent=latent.unsqueeze(0)
print(latent.size())
with torch.no_grad():
    consistent_latent = decoder_consistency(latent,schedule=[1.0])

image = consistent_latent[0].cpu().numpy()
image = (image + 1.0) * 127.5
image = image.clip(0, 255).astype(np.uint8)
image = Image.fromarray(image.transpose(1, 2, 0))

image.save("con.png")

Not entirely. You're using the prompt twice! I edited the code and ran this locally and it worked.
from diffusers import DiffusionPipeline,StableDiffusionPipeline
import torch
from consistencydecoder import ConsistencyDecoder
from PIL import Image
import numpy as np

pipe = DiffusionPipeline.from_pretrained("SG161222/Realistic_Vision_V5.1_noVAE", torch_dtype=torch.float32)
pipe.to(torch_device="cuda", torch_dtype=torch.float32) # To save GPU memory, torch.float16 can be used, but it may compromise image quality.

prompt = "masterpiece, best quality, realistic photo of a cat jumping after a string, backlit, 8k"
num_inference_steps = 8 # 1~50 steps, recommended 1~8 steps. LCM support fast inference even <= 4 steps.

decoder_consistency = ConsistencyDecoder(device="cuda:0") # Model size: 2.49 GB

latent = pipe(prompt=prompt, num_inference_steps=num_inference_steps, guidance_scale=8.0, output_type="latent")

latent=latent.images[0]/0.18215
latent=latent.unsqueeze(0)
print(latent.size())
with torch.no_grad():
    consistent_latent = decoder_consistency(latent,schedule=[1.0])

image = consistent_latent[0].cpu().numpy()
image = (image + 1.0) * 127.5
image = image.clip(0, 255).astype(np.uint8)
image = Image.fromarray(image.transpose(1, 2, 0))

image.save("diffused.png")

Thanks.

Any observed quality increase/performance gain when using consistency-decoder instead of LCM's decoder (which is just SD 1.5 VAE decoder)?


Monitoring icon is missing

Hi, thanks for this. I've just noticed the monitoring icon is missing. Would it be possible to add it?

Hi, monitoring icon is available in Material Symbols. This package is for Material Icons.
For Material Symbols, use material-symbols package.


Support for go 1.9

Hello,
I am using xgo to build and release several of my projects and would like to start using go 1.9 features and fixes, unfortunately when I try with xgo --go 1.9 I get:
Checking for required docker image karalabe/xgo-1.9... not found!

I am not sure what is involved in getting a new xgo supported version, hopefully it is not too hard, or perhaps you can point me in a direction to help provide you a PR?
Thanks for xgo, it has been extremely useful!

Is there anything I can do to help push this forward?

+1, waiting for the PR to be merged, please make it forward.


Move snippet version to sdkVersion

Ideally you need to provide a way for extensions to add this postfix as well.


[BUG] nvt.Dataset(df).to_parquet(tmp, partition_on=col) saves parquet files without .parquet extension

Describe the bug
I run this script nvt.Dataset(df).to_parquet(tmp, partition_on=col) to save preprocessed dask-cudf or cudf file by partitioning on a given column but this does not save the files with .parquet extension. It saves them as part.0 instead of part.0.parquet.
Expected behavior
A clear and concise description of what you expected to happen.
Environment details (please complete the following information):

Environment location: [Bare-metal, Docker, Cloud(specify cloud provider)] Docker

Method of NVTabular install: [conda, Docker, or from source] Docker- 0.5.3 image with latest nvtabular main branch and also tested with nvtabular-merlin-pytorch nightly container.

If method of install is [Docker], provide docker pull & docker run commands used

@rjzamora @benfred looks like adding method="worker" param in the to_parquet(..) function solves this issue. @rjzamora can you confirm if that's the solution. thanks.

@rjzamora can you confirm if that's the solution

Thanks - This is good information, but we probably want to fix the default method setting to work as well.


Feat: #298 Integrate with sentry

Summary (Please recap what you changed or fixed)

Integrate with sentry to catch error in

Background
API
UI

Checklist (If you won't pass this checklist please comment why)

[x] I removed all commented or unnecessary code.
[ ] Provide the screenshots due to UI changed or bug fixed
[ ] My code has covered these test cases (please list down your tested cases):

leave this to next release

Just removed from release/1.8.0, for focusing to dApp feature


Connection failed: Servname not supported for ai_socktype

I am getting the following error when using aerc with hotmail:
Connection failed: Servname not supported for ai_socktype

Can I see your connection string? Obsfucate your password, please. Also, prepare to be underwhelmed when you do get it connected.

[Hotmail]
source=imaps://myemailaddress@hotmail.com[:mypassword]@imap-mail.outlook.com[:993]

You have to URL escape the @ in your username.

@SirCmpwn does this apply to passwords? Tried both, still doesn't work

Yep. Try something like this:
<EMAIL_ADDRESS>
Where your email address is<EMAIL_ADDRESS>and your pw is p@ssword.

@SirCmpwn so the square brackets are optional?

Don't use the square brackets at all. They're just there to explain what fields are optional, the brackets themselves aren't included.

@SirCmpwn on first run, my entire Xubuntu system froze, tried switching to terminal to kill X but system did not respond, tried REISUB without much success, had to do a force restart, I am hesitant to try again...

That... does not even remotely sound like it could be aerc's fault.

@SirCmpwn thank you for your most encouraging words, my system has gone through a second hard reboot now. It seems that the problem kicks in after the words (null) -- connected appears. What log files do you need? Should I send you an strace dump etc.?

I don't even know where to start.

@SirCmpwn why is there sway in a terminal app?

I misspoke and edited my comment

Working on sway right now, it's on the tip of my tongue :P

@SirCmpwn I guess I will revisit aerc again in future, I need to get some work done now and I can't afford another hard lock. The OS I am currently using is an almost fresh install of Xubuntu 16.04. Apps that were running when aerc was started were Chromium and redshift


⚠ TorrentLeech has degraded performance

In f3afe2e, TorrentLeech ($TL) experienced degraded performance:

HTTP code: 200
Response time: 8341 ms

Resolved: TorrentLeech performance has improved in ddd3ab4.


Makefile target deb fails when executed in a network behind a proxy

Description
Similar to the issue described in #29031, when trying to generate the debian packages using make deb, the script build-deb fails due to the absence of the DOCKER_BUILD_ARGS variable when building some temporary container. Since there is no proxy information available, the step defined by RUN ./hack/dockerfile/install-binaries.sh runc-dynamic containerd-dynamic proxy-dynamic tini fails as it tries to access some online repositories.
Steps to reproduce the issue:
1.make DOCKER_BUILD_ARGS="--build-arg http_proxy=http://<IP_ADDRESS>:8088/ --build-arg https_proxy=http://<IP_ADDRESS>:8088/ --build-arg HTTP_PROXY=http://<IP_ADDRESS>:8088 --build-arg HTTPS_PROXY=http://<IP_ADDRESS>:8088" (ensure that you are in a network behind a proxy)
Describe the results you received:
Step 6/9 : RUN ./hack/dockerfile/install-binaries.sh runc-dynamic containerd-dynamic proxy-dynamic tini
 ---> Running in 6bfc2535262f
+ dirname ./hack/dockerfile/install-binaries.sh
+ . ./hack/dockerfile/binaries-commits
+ TOMLV_COMMIT=9baf8a8a9f2ed20a8e54160840c492f937eeaf9a
+ RUNC_COMMIT=51371867a01c467f08af739783b8beafc154c4d7
+ CONTAINERD_COMMIT=03e5862ec0d8d3b3f750e19fca3ee367e13c090e
+ TINI_COMMIT=949e6facb77383876aeff8a6944dde66b3089574
+ LIBNETWORK_COMMIT=0f534354b813003a754606689722fe253101bc4e
+ VNDR_COMMIT=f56bd4504b4fad07a357913687fb652ee54bb3b0
+ RM_GOPATH=0
+ TMP_GOPATH=
+ [ -z  ]
+ mktemp -d
+ export GOPATH=/tmp/tmp.cb8hCl0p68
+ RM_GOPATH=1
+ RUNC_BUILDTAGS=apparmor selinux
Install runc version 51371867a01c467f08af739783b8beafc154c4d7
+ install_runc
+ echo Install runc version 51371867a01c467f08af739783b8beafc154c4d7
+ git clone https://github.com/docker/runc.git /tmp/tmp.cb8hCl0p68/src/github.com/opencontainers/runc
Cloning into '/tmp/tmp.cb8hCl0p68/src/github.com/opencontainers/runc'...
fatal: unable to access 'https://github.com/docker/runc.git/': Could not resolve host: github.com
The command '/bin/sh -c ./hack/dockerfile/install-binaries.sh runc-dynamic containerd-dynamic proxy-dynamic tini' returned a non-zero code: 128
---> Making bundle: .integration-daemon-stop (in bundles/1.14.0-dev/build-deb)
+++++ cat bundles/1.14.0-dev/build-deb/docker.pid
++++ kill 1094
++++ /etc/init.d/apparmor stop
Clearing AppArmor profiles cache:.

Describe the results you expected:
Docker debian packages.
Additional information you deem important (e.g. issue happens only occasionally):
diff --git a/hack/make/build-deb b/hack/make/build-deb
index d3c2859..33f9240 100644
--- a/hack/make/build-deb
+++ b/hack/make/build-deb
@@ -82,7 +82,7 @@ set -e
                        RUN dpkg-buildpackage -uc -us -I.git
                EOF
                tempImage="docker-temp/build-deb:$version"
-               ( set -x && docker build -t "$tempImage" -f "$DEST/$version/Dockerfile.build" . )
+               ( set -x && docker build ${DOCKER_BUILD_ARGS} -t "$tempImage" -f "$DEST/$version/Dockerfile.build" . )
                docker run --rm "$tempImage" bash -c 'cd .. && tar -c *_*' | tar -xvC "$DEST/$version"
                docker rmi "$tempImage"
        done

Output of docker version:
Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 21:44:32 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 21:44:32 2016
 OS/Arch:      linux/amd64

Output of docker info:
Containers: 3
 Running: 1
 Paused: 0
 Stopped: 2
Images: 74
Server Version: 1.12.3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 54
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 4.4.0-51-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.797 GiB
Name: ecosystem-040.br.rdlabs.hpecorp.net
ID: LSX3:LZMH:VEHM:G7IB:6UB3:RW53:D7T6:4GAC:XHR7:6RH3:SYM6:VO6B
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: http://<IP_ADDRESS>:8088/
Https Proxy: http://<IP_ADDRESS>:8088/
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 <IP_ADDRESS>/8

Additional environment details (AWS, VirtualBox, physical, etc.):

Thanks! Are you interested in opening a pull request, or want a maintainer to take care of that?
/cc @vdemeester @justincormack

Sure! I can submit a PR containing the fix. I just have to check my environment to see if it meets your contributor's requirements.

Add env GIT_SSL_NO_VERIFY=true to man/Dockerfile，as Described:
https://github.com/docker/docker/issues/29031

@supergis I'll comment here as well; that sounds like a bad idea; https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables

GIT_SSL_NO_VERIFY tells Git not to verify SSL certificates. This can sometimes be necessary if you’re using a self-signed certificate to serve Git repositories over HTTPS, or you’re in the middle of setting up a Git server but haven’t installed a full certificate yet.

Basically that allows git to pull content from any server, which could be an rogue server, serving content that was tampered with.


Fix asynchonous camera transforms

This reworks the condition variables and mutexes around the point cloud transformation. It should now be a) safer and easier to maintain than before b) lower latency and c) lower CPU usage.
It gets rid of the poll for when new transforms arrive in the transform buffer, uses a single mutex for the each camera's point cloud instead of 2, and gets rid of all (I think) non-threadsafe access to the camera_ vector.
Tested in SITL, needs testing on a real vehicle.

Tested on a lower-compute platform and found an issue where it would run at 50% update rate if a planning session took slightly over the desired compute time. I've fixed this by decoupling the planner update info and the waypoint generator, it now runs with both waypoint generator at 10Hz and planner at the maximum rate it can, on this platform around 9.2Hz. I've also added some mutexes around other things I noticed might be problematic.

Successfully test flown on iMX8M-mini and Up Core based companion computers.
Up Core flight:
https://logs.px4.io/plot_app?log=8f95bd6b-150c-4d8f-8606-7b4707b5fcf1


Downgrade requirement to python 3.6

Conda tests runs on python 3.6.10: here

pip._internal.exceptions.UnsupportedPythonVersion: Package 'seiqrdp-model' requires a different Python: 3.6.10 not in '>=3.7'

I have downgrade the python requirement to 3.6
Added the requirement.txt and LICENSE to the source distribution.

And I have tested it on python 3.6.10, the launcher worked perfectly.

Does that mean that our codes works with Python 3.6 only or Python 3.6+ (and all more recent versions) ?
I have the answer but this should be clarified for a reader.

Does that mean that our codes works with Python 3.6 only or Python 3.6+ (and all more recent versions) ?
I have the answer but this should be clarified for a reader.

The requirement is Python=>3.6. So in the description we should mention to the readers that Python 3.6 and above are OK.

Thanks for taking care of this!
I may have one small remark: we should keep the version 7.5, since no changes were made to the code itself.

Thanks for the clarification!
In this case, I approve of all your changes.


fix(lambda): avoid UTF-8-encoding binary data

So far, the AWS Lambda adapter has used atob to convert a Base64-encoded request body to a binary. However, atob returns a string, and passing that to the Request constructor (at least on Node.js, which Lambda runs on) results in the binary string being encoded as UTF-8. This means that if the request body contains bytes which are not in the ASCII range, they are replaced with UTF-8 multi-byte encodings, which leads to c.req.blob(), c.req.formData(), etc. returning the wrong binary data!
This commit fixes the issue by replacing atob(body) with Buffer.from(body, 'base64'), which does not have that problem. The Buffer can be directly passed as body for new Request() because it implements the Uint8Array interface. While this makes the AWS Lambda adapter depend directly on the "node:buffer" module, this should be fine because AWS Lambda functions always run on Node.

Hi @denisw !
Thanks for the PR.I think you are right.

Hey @watany-dev,
I believe the Lambda@Edge adapter is also using atob(). Could you please check this and if it is the case, consider replacing it with Buffer?

@yusukebe
Yes, that's right, I made it as a reference and it has the same process. I'll remove it for reference.


[IIIF-857] Version Fester

Note the re-formatting commit, the result of
black *.py

Not sure how [IIIF-826] got in this branch since it's not in the change-set. and was there before the merge commit :question: :question:


Multiple spark-context instances found in cdh cluster

With the Kylin running on cdh cluster, there will be multiple spark-context which are used for sql query. Even stop the Kylin, only one spark-context will be killed.

Not a bug, it is because someone starts another kylin instance.


ENH Exclude a list of tables for checking and repairs

Issue https://github.com/silverstripe/silverstripe-framework/issues/10495
I accidentally click the wrong button which closed the old PR :(
Noted @GuySartorelli Why not use DataObject class names and infer the table names from that? Yes will have a look at this one.

For reference: the original PR was https://github.com/silverstripe/silverstripe-framework/pull/10496
@ssmarco in the future please refrain from opening a new PR - it would have been preferable to reopen the original one instead.

This seems like it would solve our super long dev/builds. What are the next steps to get this over the finish line?

@IsaacInsoll Thank you for bringing this back to my attention. I've added it to our project board now so it'll be easier to keep track of.

Closed in favour of https://github.com/silverstripe/silverstripe-framework/pull/10746 (targeting CMS 5 and implementing requested changes)


Issue with k8s.io/docs/concepts/policy/resource-quotas/

This is a...

[ ] Feature Request
[X] Bug Report

Problem:
This doc page contradicts itself, at one point stating that multiple ResourceQuota objects may be created per namespace, here:

Resource quotas work like this:

Different teams work in different namespaces. Currently this is voluntary, but support for making this mandatory via ACLs is planned.
The administrator creates one or more Resource Quota objects for each namespace.
Users create resources (pods, services, etc.) in the namespace, and the quota system tracks usage to ensure it does not exceed hard resource limits defined in a Resource Quota.

Then, later:

Resource Quota is enforced in a particular namespace when there is a ResourceQuota object in that namespace. There should be at most one ResourceQuota object in a namespace.

Proposed Solution:
Either removing one of the statements or clarify which is considered "best practice" and which is actually not supported.
Page to Update:
http://kubernetes.io/docs/concepts/policy/resource-quotas/

/remove-lifecycle stale


Changed [A-Z] to [:alpha:] for intl

Changed the alphanum regex to account for international characters

oops.


Schema not work with kebabCase key

Reproduction
https://formkit.link/e8cee4506dae46999f13b811204b1248
Describe the bug
Schema not work with kebabCase key.
Environment
• OS: WIN10
• Browser: chrome
• Version next.7

No schema does not support kebab-case variables (similar to javascript) — they should be provided as camelCase only. For performance reasons I’m not sure its worth the transform cost when devs can provide them the formatted way.
Is there a common use case youve found where defining kebab-case properties is really helpful?

@justin-schroeder
The custom attribute data-* is an HTML specification. Sections schema should be used in combination with the value of attrs.

Just like in JavaScript our compiler cannot accept - in variable names. For this reason, I would recommend converting any kebab-case variables to camelCase — you can even use a utility provided in @formkit/utils to do so:
https://formkit.link/1de8ff9de07887f1e3e940349c8480ea


[FEATURE] Zero downtime upgrade after v1.0.0

The epic issue to continue tracking zero downtime upgrade related issues and test plan after v1.0.0 release
This extend the development and testing status from #1022 to perform more testing in different stages to fully check the zero downtime on bumped rke2, rancher and longhorn version.

pre-1.0.0 dependencies

[x] https://github.com/longhorn/longhorn/issues/3234 (Enhancement included in v1.2.3-rc1)
[ ] https://github.com/harvester/harvester/issues/1707

(For one node zero downtime upgrade image from URL)

[x]  [Fixed in Rancher, need to test more times] https://github.com/harvester/harvester/issues/1717

(For four nodes zero downtime upgrade image from File)

v1.0.0 dependencies

[x] https://github.com/harvester/harvester/issues/1852
[x] https://github.com/harvester/harvester/issues/1855
[ ] https://github.com/harvester/harvester/issues/1810
[x] https://github.com/harvester/harvester/pull/1832
[ ] Remove settings vlan: see https://github.com/harvester/harvester/issues/1739
[x] https://github.com/rancher/rancher/issues/36536
[ ] https://github.com/harvester/harvester/issues/1982
[ ] https://github.com/harvester/harvester/issues/1983
[x] [Fixed in Rancher] https://github.com/harvester/harvester/issues/1984
[ ] https://github.com/harvester/harvester/issues/1985
[ ] https://github.com/harvester/harvester/issues/2029

Additional context

Please refer to #1635 for upgrade design and test execution steps

I drop the item:

Remove settings vlan: see https://github.com/harvester/harvester/issues/1739

The setting vlan doesn't exist in the previous version, it's a misuse of the installer config. We have the following installer PR to avoid this happening in the future.

Close this epic one, we can track upgrade issues with area/upgrade-related label and release.


Add new exercise: Roman numerals

Here's the classic Roman numerals exercise.

@rchavarria looks nice!  I like the tests on one line!

I would favor readability, but in this particular case, lines are short enough to look nice. Anyway, it's neat to be able to write them, uh? :wink:


Skip compression of assets during development

Is it possible to skip the compression step when watching files during development?
I don't use the compressed assets during this phase and have my own minification step later, so I really don't need this processing delay.

You will need to fork the gulp tasks in watch.js and remove the bits about compressed assets. You can then import these tasks in your gulp file using this method:
https://github.com/Semantic-Org/Semantic-UI/tree/master/tasks#how-to-use


Code format

Run JuliaFormatter.jl to comply with Julia Blue style.

Blue or SciML?

See also: https://github.com/control-toolbox/CTBase.jl/issues/253.


Kids first HP terms

Set of terms from the KidsFirst project that have been added to HPO. We would like to determine if these exist in the MP and if not is it appropriate to add the term.
Term list: https://docs.google.com/spreadsheets/d/11EB7Kkbkeubtz3qnxfTqBpmXgyoj3uBYnzY_Jlr7HLw/edit?usp=sharing

New terms to consider adding (red on spreadsheet)
Left ventricular outflow tract obstruction child of "abnormal heart left ventricle outflow tract morphology" (MP:0010429)

Terms I have questions about (purple on spreadsheet)
Tetralogy of Fallot - collection of phenotypes

Terms with multiple possible MP matches (yellow on spreadsheet)
Microtia - could match small ears (broader?) or outer ear hypoplasia (narrower?)

Green on spreadsheet are terms that could use a new synonym
Orange - tumor terms. The MP deals with most tumor terms as increased tumor type incidence not just tumor type. This was done as populations have a baseline incidence of tumors and in most cases we are focused on a change in that incidence. It also provides a clear distinction between the tumor phenotype and the tumor as an anatomical entity. Will the mapping from an HPO tumor term to and MP tumor incidence work for KidsFirst?

There are some cases of a mismatch between the upper structure of the MP and HPO. HPO has terms for abnormal X that could cover both morphology and physiology.  The MP does not have abnormal X that covers both.
Examples:
Abnormality of limbs (HP:0040064) vs limbs/digits/tail phenotype (MP:0005371) and abnormal limb morphology (MP:0002109) (no physiology term in this case)
Abnormality of the face (HP:0000271) vs abnormal facial morphology (MP:0003743) (child of craniofacial branch, no physiology branch for the face)

More possible new terms
Single umbilical artery - I would be inclined to add a term for "decreased umbilical artery number" with a more species neutral definition, child of MP:0003230

more questions
Strabismus exists in both MP and HPO but the definitions are different. I currently have these as "broad" since the MP term covers a broader range of phenotypes (no restricted by "bifoveal fixation"). Should these be exact? The additional restriction in the HPO is really defining the standard alignment of human eyes. Can we ignore this?
Hypothyroidism in HPO maps to decreased activity of thyroid gland in MP (exact synonym hypothyroidism) but the MP has hypothyroidism but defines this as "reduced function of this endocrine gland that normally produces hormones that regulate the metabolic rate of the body"  and has separate terms for abnormal thyroid homone levels. HPO has this term defined as "Deficiency of thyroid hormone." and also has terms for thyroid hormone levels but the decreased terms do not relate to hypothyroidism. This makes me question the definition used by HPO.

Pulmonic stenosis (HP:0001642) I think this is an exact match to heart right ventricle outflow tract stenosis (MP:0010449). MP also has pulmonary valve stenosis (MP:0006128) as a child of heart right ventricle outflow tract stenosis. Searching for pulmonic stenosis brings back both but I think the outflow tract is better.
The MP terms are located under morphology but the HPO terms are under physiology.

More new terms
Bronchopulmonary sequestration (HP:0010960) - child of lung cysts (MP:0004880)?

Ureterocele/ureterocele - I suspect these are an exact match but the definitions have details that do not match. Unclear if these are essential.
HP definition - A ureterocele is a congenital saccular dilatation of the distal segment of the ureter.
MP definition - saccular dilation of the terminal portion of the ureter that protrudes into the lumen of the urinary bladder, probably due to a congenital stenosis of the ureteric orifice meatus; most often associated with a duplicated collection system, where two ureters drain their respective kidney instead of one
How essential is:
congenital
protrudes into the lumen of the urinary bladder
I think we could ignore "probably due to a congenital stenosis of the ureteric orifice meatus; most often associated with a duplicated collection system, where two ureters drain their respective kidney instead of one" This information should be moved into a comment.

Coloboma/coloboma - while the text labels are identical the definitions are not
HP (HP:0000589) - A developmental defect characterized by a cleft of some portion of the eye or ocular adnexa.
MP (MP:0005262) - anomaly in which some of the structures of the eye are absent due to incomplete fusion of the fetal intraocular fissure during gestation
cleft vs absent seems like a big difference to me, but the child terms in MP mostly refer to "some part of the structure is absent" which seems closer to cleft so possibly it is just a language issue?

Review the definition of seizures (MP:0002064)
"sudden and often acute manifestation of epileptic attack, sometimes convulsive"
Compare to HP definition - "A seizure is an intermittent abnormality of nervous system physiology characterised by a transient occurrence of signs and/or symptoms due to abnormal excessive or synchronous neuronal activity in the brain"
Is it correct to specify "epileptic attack" in the definition? What exactly does this mean?

Accessory spleen/accessory spleen
HP definition - An accessory spleen is a round, iso-echogenic, homogenic and smooth structure and is seen as a normal variant mostly on the medial contour of the spleen, near the hilus or around the lower pole. This has no pathogenic relevance.
MP definition - the splenic tissue is divided into equal masses; often related to situs inversus
despite identical labels these do not seem the same to me. Which is correct?


规范License信息

Ⅰ. Describe what this PR does
Ⅱ. Does this pull request fix one issue?

Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews

Please fix DCO using git commit -s -m "<commit_message>"


[C++][Parquet] Incompatibility due to moving from Parquet to Arrow IO interfaces

We moved to the Arrow IO interfaces as part of https://issues.apache.org/jira/browse/PARQUET-1422
However, the BufferedInputStream implementations between Parquet and Arrow are different.
Parquet's BufferedInputStream used to takes a RandomAccessSource. Arrow's implementation takes an InputStream. As a result, the ::arrow::io::BufferedInputStream::Peek(which invokes Read()) implementation causes the raw source (input to BufferedInputStream) to change its offset on Peek(). This did not happen in the Parquet's BufferedInputStream implementation.
Reporter: Deepak Majeti / @majetideepak
Assignee: Wes McKinney](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=wesm) / @wesm
Related issues:

[C++] Create InputStream that is an isolated reader of a segment of a RandomAccessFile (relates to)

Note: This issue was originally created as PARQUET-1636. Please see the migration documentation for further details.

Deepak Majeti / @majetideepak:
CC: @czxrrr

Deepak Majeti / @majetideepak:
I might be missing something here. Peek() moving the raw offset might not be the core issue. The old ReadAt changes the offset too. @czxrrr can you clarify this? Thanks.

Zherui Cao / @czxrrr:
I checked out previous version and found that BufferedInputStream::Peek() did move offset when it calls Read or ReadAt.
But the previous InMemoryInputStream::Read() is calling its Peek() and then called ReadAt(offset), since it always specify offset, so the previous one works well, that is to say, this does not care about where the offset went, since every call of Read() will specify the position,
But currently InMemoryInputStream(old) was replaced by BufferedInputStream(new) is using inputstream(non seekable) rather than randomAccessFile(seekable). inputstream has no function like Seek() or ReadAt(), therefore, calling Read has no control of the offset.

Wes McKinney / @wesm:
If you are doing column read buffering in different threads, that could definitely be an issue

Zherui Cao / @czxrrr:
The raw file position is changed in columnReader::NextPage(), because of the call of BufferedInputStream->Peek().

Do you think it would work to change the type of BufferedInputStream::raw_() to RandomAccessFile, so it can use ReadAt

Wes McKinney / @wesm:
I would like to understand the nature of the problem.
The stream is created here:
https://github.com/apache/arrow/blob/master/cpp/src/parquet/file_reader.cc#L119
The raw file position should not be moved outside of the span of the ColumnChunk in the file. If that's happening, that is a bug. The intention of the buffering option is to do read buffering through the ColumnChunk instead of reading the entire chunk into memory (the alternative).

Wes McKinney / @wesm:
Resolved by https://github.com/apache/arrow/commit/2c808a2cbd62300a36d682ebd7bd25ad8b6cd500

Wes McKinney / @wesm:
I just created https://issues.apache.org/jira/browse/ARROW-6180, that might be a solution if your problem results from multithreading

Wes McKinney / @wesm:
Why is changing the file position a problem, though? That's the part that I don't understand.


Rebranding

As @seangwright mentioned in A comment in #5, We just announced a rebrand of the product. When you get a chance it'd be great if you could update this plugin to reflect that new brand, "Kentico Kontent"
On an unrelated note, I've been working with the plugin in my free time the last couple of weeks converting the Vue travel blog sample I built and I'm nearly complete. I've run into very few issues that I couldn't resolve quickly and they were all mostly due to my lack of understanding of Gridsome or GraphQL. I've been super pleased with this plugin!

Thanks for the comments @ChristopherJennings - I'm really pleased the plugin has been working well for you!
I saw the announcement of the rebrand - congrats on that! I will bring the plugin in line with the new brand in the next release, which I am planning to also include #7 and #9.

Hi @ChristopherJennings - with the rebrand, would it be better to rename the plugin:

gridsome-source-kontent; or
gridsome-source-kentico-kontent

Guidelines: https://gitter.im/Kentico/Kontent?at=5d8b643d692d464f7969c6d9
I'd go for gridsome-source-kentico-kontent

Hi,
Sorry for the delay in my response.
We've just published guidelines for the community here.
As your plugin is outside of our github organization, naming would be gridsome-source-kentico-kontent, since there's an established pattern for Gridsome plugins already, we defer to that.

Great, thanks @petrsvihlik and @ChristopherJennings :)
I'm pretty much done with changes related to the rebrand and upgrade to the latest js sdk. I'll hopefully get some time in the coming days to finalise and push.


about Squeeze-And-Excite in that block

I Notice that you don't add Squeeze-And-Excite in some block  which is different from paper. why not add it?

I Notice that you don't add Squeeze-And-Excite in some block which is different from paper. why not add it?

I tried to implement the network architecture as described in the paper. Please specify the location of the problem you mentioned, I will check it.

怎么训练

Did you reproduce the paper's acc?

Did you reproduce the paper's acc?

Not yet, I just got 65% accuracy on v3_small and 70% accuracy on v3_large. I think there may be some training strategies that are not in the way mentioned in the paper.

hello, i want to kown why layer"conv2-1/dwise/scale" end with layer"conv2-1/dwise/scale-relu", while other "se" layer only have layer "scale"?


Unknown record type: 3

Tried to flash latest router firmware (CC1352P2_CC2652P_launchpad_router_20221102) to Sonoff Zigbee 3.0 dongle plus using Flash Programmer 2, but received error: Unknown record type: 3.
I was able to flash "CC1352P2_CC2652P_launchpad_router_20220125"

Could you please check:
https://github.com/Koenkk/Z-Stack-firmware/issues/423 ?

and/or:
https://github.com/Koenkk/Z-Stack-firmware/issues/397

Duplicate of https://github.com/Koenkk/Z-Stack-firmware/issues/397


Mayastor just plainly does not start under Microk8s 1.24/stable

Describe the bug
Multiple pods in the mayastor namespace just stuck in the pending state. No mayastorpool ever becomes ready.
To Reproduce

Preconfigure 4 Debian11 machines as stated in https://microk8s.io/docs/addon-mayastor (huge pages, kernel modules).
snap install microk8s --classic --channel=1.24/stable on the 4 machines
Join the 4 nodes into one cluster with microk8s.add-node, verify that the HA cluster is up with microk8s.status
Try to enable the core/mayastor addon as stated in https://microk8s.io/docs/amicrok8s.add-nodeddon-mayastor, with a 6G default pool size.

Expected behavior
The mayastor operator should start, the 6G pool should become available.
Screenshots

** OS info (please complete the following information):**

Distro: Debian 11 bullseye
Kernel version: 5.10.0-18-cloud-amd64
MayaStor revision or container image: stock from microk8s 1.24

Additional context
I am so overwhelmed by so many containers pending, I don't even know whose logs to show and where to start debugging. The microk8s cluster seems operational otherwise:
# /snap/bin/microk8s.kubectl get nodes
NAME            STATUS   ROLES    AGE   VERSION
vas-k3s-test4   Ready    <none>   18h   v1.24.4-2+2f38f78fa07274
vas-k3s-test3   Ready    <none>   18h   v1.24.4-2+2f38f78fa07274
vas-k3s-test1   Ready    <none>   18h   v1.24.4-2+2f38f78fa07274
vas-k3s-test2   Ready    <none>   18h   v1.24.4-2+2f38f78fa07274

If this is a microk8s issue and not a mayastor issue, please redirect me to their bug tracker.

Just picking randomly into some pending pods

Here is what it prints when I enable the addon:
# /snap/bin/microk8s.enable core/mayastor --default-pool-size 6G
Checking for HugePages (>= 1024)...
Checking for HugePages (>= 1024)... OK
Checking for nvme_tcp module...
Checking for nvme_tcp module... OK
Checking for addon core/dns...
Checking for addon core/dns... OK
Checking for addon core/helm3...
Checking for addon core/helm3... OK
Error from server (AlreadyExists): namespaces "mayastor" already exists
Default image size set to 6G
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/snap/microk8s/3719/credentials/client.config
Getting updates for unmanaged Helm repositories...
...Successfully got an update from the "https://raw.githubusercontent.com/canonical/mayastor-control-plane/develop/chart" chart repository
...Successfully got an update from the "https://raw.githubusercontent.com/canonical/mayastor/develop/chart" chart repository
...Successfully got an update from the "https://raw.githubusercontent.com/canonical/etcd-operator/master/chart" chart repository
Saving 3 charts
Downloading etcd-operator from repo https://raw.githubusercontent.com/canonical/etcd-operator/master/chart
Downloading mayastor from repo https://raw.githubusercontent.com/canonical/mayastor/develop/chart
Downloading mayastor-control-plane from repo https://raw.githubusercontent.com/canonical/mayastor-control-plane/develop/chart
Deleting outdated charts
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/snap/microk8s/3719/credentials/client.config
NAME: mayastor
LAST DEPLOYED: Thu Sep 22 23:38:04 2022
NAMESPACE: mayastor
STATUS: deployed
REVISION: 1
TEST SUITE: None

=============================================================

Mayastor has been installed and will be available shortly.

Mayastor will run for all nodes in your MicroK8s cluster by default. Use the
'microk8s.io/mayastor=disable' label to disable any node. For example:

    microk8s.kubectl label node vas-k3s-test3 microk8s.io/mayastor=disable

But its pods never really come up.

@victor-sudakov To be honest, I'm not sure what the root cause is based on the above but, given the failure to resolve the service name of the etcd instance you've shown above, I'm fairly certain that many of the other Mayastor pods wouldn't init anyway, since they check for the availability of this service.
For that matter, neither is your etcd pod running anyway.
So, what do you see if you kubectl describe the pods that are pending?  Perhaps start with checking why the etcd pod is stuck?
I think a bit of generic k8s troubleshooting is in order here, before going to potential mayastor-specific issues.

Perhaps start with checking why the etcd pod is stuck?

Does Mayastor bring its own etcd service or does it expect some external one? I may be mistaken but I don't think a default Microk8s installation contains any etcd:
 254563 ?        Ss     0:39 /bin/bash /snap/microk8s/3719/apiservice-kicker
 254944 ?        Ssl   57:52 /snap/microk8s/3719/bin/k8s-dqlite --storage-dir=/var/snap/microk8s/3719/var/kubernetes/backend/ --listen=unix:///var/snap/microk8s/3719/var/kubernetes/backend/kine.sock:12379
 255252 ?        Ss     0:00 /bin/bash /snap/microk8s/3719/run-cluster-agent-with-args
 255279 ?        Sl     0:02 /snap/microk8s/3719/bin/cluster-agent --bind <IP_ADDRESS>:25000 --keyfile /var/snap/microk8s/3719/certs/server.key --certfile /var/snap/microk8s/3719/certs/server.crt --timeout 240
 255380 ?        Ssl    9:35 /snap/microk8s/3719/bin/containerd --config /var/snap/microk8s/3719/args/containerd.toml --root /var/snap/microk8s/common/var/lib/containerd --state /var/snap/microk8s/common/run/containerd --address /var/snap/microk8s/common/run/containerd.sock
 255512 ?        Ssl   85:33 /snap/microk8s/3719/kubelite --scheduler-args-file=/var/snap/microk8s/3719/args/kube-scheduler --controller-manager-args-file=/var/snap/microk8s/3719/args/kube-controller-manager --proxy-args-file=/var/snap/microk8s/3719/args/kube-proxy --kubelet-args-file=/var/snap/microk8s/3719/args/kubelet --apiserver-args-file=/var/snap/microk8s/3719/args/kube-apiserver --kubeconfig-file=/var/snap/microk8s/3719/credentials/client.config --start-control-plane=true
 255783 ?        Sl     3:45 /snap/microk8s/3719/bin/containerd-shim-runc-v2 -namespace k8s.io -id 996ce86ce832eef2c90653628b1ee1819c8a6a73e4cd312c52d9b567a58c93dc -address /var/snap/microk8s/common/run/containerd.sock
 256677 ?        Sl     0:37 /snap/microk8s/3719/bin/containerd-shim-runc-v2 -namespace k8s.io -id 6cded02142c29be7c80ad851476ef4a2e77a89d8c1058ce4f5901514d7c3fd0a -address /var/snap/microk8s/common/run/containerd.sock
 928704 pts/0    R+     0:00 grep /snap/microk8s/

The cluster uses dqlite as a configuration storage by default.

Does Mayastor bring its own etcd service or does it expect some external one?

It depends.  If the deployment is made following the example we provide in the official Mayastor documentation, then the etcd instance required is deployed at the same time, using a Helm chart prepared by the OpenEBS maintainers.
What is happening in your case, is that the providers of the Canonical Mayastor add-on are providing their own installation route, using an etcd operator.
So my suggestion to progress your issue is a) use some basic k8s troubleshooting to ascertain why the etcd pod isn't starting and b) if that isn't down to something directly related to your node setup (i.e. resource shortage), then maybe raise an issue with the core add-ons community at Canonical .
Potentially another option: remove the etcd and etcd-operator deployments and instead deploy etcd on your node using the directions from the Mayastor documenation.  If you do that, be sure to scale down the replica count to one in the statefulset.yaml

@GlennBullingham Do you suggest I should start debugging with researching why the etcd pod in the mayastor namespace is not starting? I will have time to do it hopefully on Wednesday, we can put this issue on hold till then. Thank you for the hint.

Do you suggest I should start debugging with researching why the etcd pod in the mayastor namespace is not starting?

Yes, absolutely.  I think that's the correct starting point.
The control and data plane Mayastor pods will not start until they can find their expected etcd instance is available.  And if there's something causing the scheduling of the etcd instance to fail, it may well impact Mayastor pods directly too.

Oh well, the etcd pod is stuck in the PodInitializing state, and its initContainer "check-dns" is failing the check (see below). Does this make any sense to you?
 Are you running debian OS on bare metal server or on a vm of some sort?
I ask because I also have same problem when trying to run microk8s/1.24 + mayastor on ubuntu 22.04 in a vm managed by multipass (qemu). Pods always pending.
However, if i install microk8s with mayastor in ubuntu os running on bare metal. they all come up running.
But then I hit a known issue in that mayastor take 100% of a cpu.
I suspect  either the vm doesn't have a spare cpu for mayastor,  or, because mayastor does thick-provisioning it conflicts with multipass that use thin-provisioning for the vm storage.
Hope i don't appear to be hijacking your thread.

@ivanix1 I think it is a smart question. I'm experimenting in AWS EC2 instances, not on bare metal. I wonder if it matters.

Quite stale and seems like it was not a mayastor issue.
Please reopen if otherwise, thanks!


Bad owner or permissions on ssh config when using private GIT repo

I noticed some weird behaviour when TF code to be deployed is depending on private GIT repositories. The terraform init step fails with the following error:
Connecting to raw.githubusercontent.com (<IP_ADDRESS>:443)
saving to '/home/tfo-runner/generations/6/init.sh'
init.sh              100% |********************************|  2762  0:00:00 ETA
'/home/tfo-runner/generations/6/init.sh' saved
Initializing modules...
Downloading git@github.com:xxx/lqt-gitops.git?ref=DOC-1171 for db...

│ Error: Failed to download module
│
│ Could not download module "db" (main.tf:29) source code from
│ "git@github.com:xxx/gitops.git?ref=DOC-1171": error downloading
│ 'ssh://git@github.com/xxx/gitops.git?ref=DOC-1171': /usr/bin/git
│ exited with 128: Cloning into '.terraform/modules/db'...
│ Bad owner or permissions on /home/tfo-runner/.ssh/config
│ fatal: Could not read from remote repository.
│
│ Please make sure you have the correct access rights
│ and the repository exists.

Interestingly the setup step runs successful although it depends on an internal GIT repository. The error above is a result of terraform init trying to download further modules from the same GIT repo.
Here is my Terraform manifest:
apiVersion: tf.isaaguilar.com/v1alpha2
kind: Terraform
metadata:
  name: example
  namespace: lqt-srv-1
spec:
  terraformVersion: 1.0.0
  # Pull this module to execute
  terraformModule:
    source: "git@github.com:xxx/gitops.git//some/module?ref=BRANCH"

  # Use kubernetes as a backend which is available for terraform >= v0.13
  backend: |-
    terraform {
      backend "s3" {
        region         = "eu-central-1"
        bucket         = "tf-state"
        key            = "terraform.tfstate"
      }
    }
  ignoreDelete: false

  # Create a tfvar env for the terraform to use
  taskOptions:
  - for:
    - '*' # The following config affects all task pods
    env:
    - name: TF_VAR_vpc_name
      value: blah
    - name: TF_VAR_client_id
      value: acme
    - name: TF_VAR_environment
      value: test

  scmAuthMethods:
    - git:
        ssh:
          sshKeySecretRef:
            key: github-ssh-key
            name: github-ssh-key-aws-managed-key
            namespace: lqt-srv-1
      host: github.com

  keepCompletedPods: true
  keepLatestPodsOnly: true
  serviceAccount: tf-operator-service-account

I defines a preinit step to investigate the /home/tfo-runner/.ssh directory and found this:
bash-5.1$ ls -al /home/tfo-runner/.ssh/
total 16
drwxrwsr-x    2 tfo-runn 2000          4096 Oct  7 12:40 .
drwxrwsr-x    6 root     2000          4096 Oct  6 18:14 ..
-rw-rw----    1 tfo-runn 2000           126 Oct  7 12:40 config
-rw-rw----    1 tfo-runn 2000           399 Oct  7 12:40 github.com

All attempts to change the permissions within preinit failed in init with same result.
I'm using the Helm chart v0.2.15 from https://galleybytes.github.io/helm-charts which installs terraform-operator:v0.9.0-pre3.

I was able to replicate the issue on the ARM64v8 architecture, but not on the AMD64. I'll have to dig to see why the different builds have different results for ssh keys.
I ran a test as similar as possible to yours.  This is the config that works for amd and not for arm:
apiVersion: tf.isaaguilar.com/v1alpha2
kind: Terraform
metadata:
  name: example
  namespace: default
spec:
  terraformVersion: 1.0.0
  # Pull this module to execute
  terraformModule:
    source: "git@github.com:isaaguilar/simple-aws-tf-modules.git//private-github-module"

  # Use kubernetes as a backend which is available for terraform >= v0.13
  backend: |-
    terraform {
      backend "s3" {
        region         = "us-east-1"
        bucket         = "my-terraform-state-bucket"
        key            = "terraform-operator/my/awesome/example.tfstate"
      }
    }
  ignoreDelete: true

  # Create a tfvar env for the terraform to use
  taskOptions:
  - for:
    - '*' # The following config affects all task pods
    env:
    - name: TF_VAR_vpc_name
      value: blah
    - name: TF_VAR_client_id
      value: acme
    - name: TF_VAR_environment
      value: test
    envFrom:
    - secretRef:
        name: aws-session-credentials # temp creds for my bucket

  scmAuthMethods:
    - git:
        ssh:
          sshKeySecretRef:
            key: key
            name: gitsshkey
            namespace: default
      host: github.com

  keepCompletedPods: true
  keepLatestPodsOnly: true
  serviceAccount: tf-operator-service-account

The main module is simple:
# https://github.com/isaaguilar/simple-aws-tf-modules/blob/master/private-github-module/main.tf
output "static" {
  value = "static"
}

module "private" {
  // This source is private
  source = "git@github.com:isaaguilar/terraform-do-something-awesome.git?ref=main"
}

terraform {
  required_version = "> 0.12"
}

And indeed my logs match yours:
Connecting to raw.githubusercontent.com (<IP_ADDRESS>:443)
saving to '/home/tfo-runner/generations/1/init.sh'
init.sh              100% |********************************|  2762  0:00:00 ETA
'/home/tfo-runner/generations/1/init.sh' saved
Initializing modules...
Downloading git@github.com:isaaguilar/terraform-do-something-awesome.git?ref=main for private...
╷
│ Error: Failed to download module
│
│ Could not download module "private" (main.tf:7) source code from
│ "git@github.com:isaaguilar/terraform-do-something-awesome.git?ref=main":
│ error downloading
│ 'ssh://git@github.com/isaaguilar/terraform-do-something-awesome.git?ref=main':
│ /usr/bin/git exited with 128: Cloning into '.terraform/modules/private'...
│ error: cannot run ssh: No such file or directory
│ fatal: unable to fork
│
╵

Any help determining how terraform

Found the issue. In the arm build I don't have ssh installed.
~/generations/2/main$ ssh
bash: ssh: command not found

Fix should be relatively easy, but the tftask pods usually take a long time to build for all the versions. Hopefully I'll have them all updated for next week.

Oh, but in the error you posted, it has to do with Bad owner or permissions on /home/tfo-runner/.ssh/config. Then I haven't replicated that issue yet. Ok still looking then. :(

@odise I made some changes in the task scripts to attempt to fix the .ssh dir. I hope this fixes the issue you're having.
https://github.com/GalleyBytes/terraform-operator-tasks/pull/9
To make use of the changes, the following additions to the spec may be used:
spec:
  # ...
  taskOptions:
  - for: [ init, plan, apply, init-delete, plan-delete, apply-delete ]
    script:
      source: https://raw.githubusercontent.com/GalleyBytes/terraform-operator-tasks/always-attempt-to-fix-ssh/tf.sh
  - for: [ setup ]
    script:
      source: https://raw.githubusercontent.com/GalleyBytes/terraform-operator-tasks/always-attempt-to-fix-ssh/setup.sh

The preinit should have worked. Perhaps the chmod in the preinit used 660 in order to produce  -rw-rw---- permissions:
-rw-rw----    1 tfo-runn 2000           399 Oct  7 12:40 github.com

and the fix uses 600 which produces -rw------- permissions:
-rw------- 1 tfo-runner 2000 1.7K Oct 10 20:17 /home/tfo-runner/.ssh/github.com


OCPBUGS-34491: fix: adjust topolvm controller probes

Which issue(s) this PR addresses:
Changes the readinessProbe of topolvm-controller to use /readyz instead of /metrics
Introduces Startup / liveness probes for the csi sidecar containers for their leader election checks
Closes #

/jira refresh

/test all

/test all

/test all

/test all

had to rebase...

/test test-rebase


Add Group Permissions for 'Addons' in Admin Center

In my case specifically Simpilot's Event Booking module....my events manager can modify events but can't create them

This one should go to simpilot, this isn't a phpvms core issue. Sorry for the super-late reply....


Title is overlaying

I am sure this was working earlier. But on a new project, when I navigate between screen, the title is overlaying on previous one. and back button is not clearing, I am using IOS emulator

Visited home screen
Then to next Page
Then used 'Back' to come back to Home screen

Check the last image, it is overlaying with previous one

Please refer to https://github.com/react-native-simple-router-community/react-native-simple-router.

Is it a new implementation than this?  Does have support to android?

I got it, i didnt know that @t4t5  is not maintaining this. Thanks anyway. Please update the Readme too... As this is the second option on google search results..


Frontespizio isn't updating

Hi, thank you a lot for this template, it's really useful but i've got a problem...
I'm trying to edit the frontespizio but i can't, i tried on visual studio and i edited it once and now i can't anymore (my repository if you want to try build it with the recipe: mega, arara isn't working with minted) and i tried to upload the file on overleaf but i get the template frontespizio and not the one i wrote (https://www.overleaf.com/read/kfdhkrrkftyz)
(The file tesi-frn.pdf is genereted correctly but seems that is not linked with tesi.pdf)
I have no clue why this isn't working anymore
Waiting for a response, good Evening

Hi Matteo, I would like to help you, but it's difficult for me to diagnose the problem without some log or any detail about your setup!
Ignoring Overleaf for now, what OS and LaTeX distribution are you working on? Also, LaTeX is logging any errors about frontespizio package, or any other package?
Also, the phrase

arara isn't working with minted

sounds very strange to me, it may be a problem with the environment setup, it should work just fine. The recipe you wrote uses lualatex directly, which works just fine with the main document structure, but it doesn't invoke frontespizio command to refresh the frontspice, nor it invokes biber for the bibliography generation
Finally, opening the Overleaf link everything looks good to me:

Why you say that you that you get the template and not the one you wrote? What's missing?

Is your problem resolved? Can I close this issue?

I'm closing this issue, feel free to reopen it if needed


Cannot auth with password / private access token

I tried #46 but I couldn't login with my password or a newly created private access token.
Is this issue on a device? I uses Huaway mediapad T1 7.0.

Same. Megafon Login 3. Android version 4.4.2.

Probably related to https://github.com/jonan/ForkHub/issues/345

The problem still persists. I am on huawei android4.4.2. Cannot login.


Coraza not logging logs and audit logs

Hi,
I am using coraza with caddy and trying to find the audit logs noted that it does not leave the audit log registers and when change the config SecAuditLog got a error:
If used a rute /var/log/audit/coraza_audit.log I got Permission deny
and If I used another rute in my case I try with /var/log/caddy/coraza_audit.log I got file don't found (if I create log file back permission deny)
All logs that i receive came from caddy log and not from the coraza
I try seting the config "SecDebugLog /var/log/coraza/coraza.log" with "SexDebugLogLevel 6"  but not generate any file.
and if I using audit log "SecAuditLogDir /var/log/audit/audit_coraza.log", it's generate a permission error even with permission 777, only if using on /tmp/ folder not geting a error but it's not generate any file or log.
anyone knows how to fix the permission deny?

this logs is from coraza ?  If that is, why all logs are generate on /var/log/syslog ?
Feb  6 19:47:04 lab caddy[585]: {"level":"error","ts":1675730824.3300385,"logger":"http.handlers.waf","msg":"[client "<IP_ADDRESS>"] Coraza: Warning. SQL Injection Attack Detected via libinjection [file "/usr/share/caddy/waf/coreruleset/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "0"] [id "942100"] [rev ""] [msg "SQL Injection Attack Detected via libinjection"] [data "Matched Data: n&1 found within ARGS:name: carlos or 1=1"] [severity "critical"] [ver "OWASP_CRS/4.0.0-rc1"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/248/66"] [tag "PCI/6.5.2"] [hostname ""] [uri "/?name=carlos%20or%201=1"] [unique_id "hEliunlRvPlEKAvH"]\n[client "<IP_ADDRESS>"] Coraza: Warning. SQL Injection Attack Detected via libinjection [file "/usr/share/caddy/waf/coreruleset/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "0"] [id "942100"] [rev ""] [msg "SQL Injection Attack Detected via libinjection"] [data "Matched Data: n&1 found within ARGS:name: carlos or 1=1"] [severity "critical"] [ver "OWASP_CRS/4.0.0-rc1"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/248/66"] [tag "PCI/6.5.2"] [hostname ""] [uri "/?name=carlos%20or%201=1"] [unique_id "hEliunlRvPlEKAvH"]\n[client "<IP_ADDRESS>"] Coraza: Warning. SQL Injection Attack Detected via libinjection [file "/usr/share/caddy/waf/coreruleset/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "0"] [id "942100"] [rev ""] [msg "SQL Injection Attack Detected via libinjection"] [data "Matched Data: n&1 found within ARGS:name: carlos or 1=1"] [severity "critical"] [ver "OWASP_CRS/4.0.0-rc1"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/248/66"] [tag "PCI/6.5.2"] [hostname ""] [uri "/?name=carlos%20or%201=1"] [unique_id "hEliunlRvPlEKAvH"]\n[client "<IP_ADDRESS>"] Coraza: Warning. SQL Injection Attack Detected via libinjection [file "/usr/share/caddy/waf/coreruleset/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "0"] [id "942100"] [rev ""] [msg "SQL Injection Attack Detected via libinjection"] [data "Matched Data: n&1 found within ARGS:name: carlos or 1=1"] [severity "critical"] [ver "OWASP_CRS/4.0.0-rc1"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/248/66"] [tag "PCI/6.5.2"] [hostname ""] [uri "/?name=carlos%20or%201=1"] [unique_id "hEliunlRvPlEKAvH"]\n"}
Feb  6 19:47:04 lab caddy[585]: {"level":"error","ts":1675730824.3382945,"logger":"http.handlers.waf","msg":"[client "<IP_ADDRESS>"] Coraza: Warning. Inbound Anomaly Score Exceeded (Total Score: 20) [file "/usr/share/caddy/waf/coreruleset/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "0"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 20)"] [data ""] [severity "emergency"] [ver "OWASP_CRS/4.0.0-rc1"] [maturity "0"] [accuracy "0"] [tag "anomaly-evaluation"] [hostname ""] [uri "/?name=carlos%20or%201=1"] [unique_id "hEliunlRvPlEKAvH"]\n"}

@jptosso @jcchavezs Any inputs here?

@jcchavezs @jptosso ping.

@jcchavezs @jptosso ping 2.

we completely rewrite the connector. Do you mind testing again with latest commit? At least the debug logging should be working fine. Audit we will tackle soon.

Hello @jcchavezs when try to update caddy it's take v1.2.2 this is the final version of coraza-caddy? it's not will be 1.2.3 ?
"SecAuditLogDir /var/log/audit_coraza.log" provision http.handlers.waf: invalid WAF config: open /var/log/audit_coraza.log: permission denied
same error.

The version with the rewritten connector is not yet tagged, you should be able to try it pointing directly to the commit (34daaf87f9ddaca2833461de59ebada21c902598)

Hello @M4tteoP  if used xcaddy with build 34daaf87f9ddaca2833461de59ebada21c902598 i got error invalid
go: github.com/caddyserver/caddy/v2@34daaf87f9ddaca2833461de59ebada21c902598: invalid version: unknown revision 34daaf87f9ddaca2833461de59ebada21c902598
I used  xcaddy build 34daaf87f9ddaca2833461de59ebada21c902598 --with github.com/corazawaf/coraza-caddy

I think the right syntax is xcaddy build --with
@.***
On Tue, 4 Apr 2023, 19:45 Carlos Herrera, @.***> wrote:

Hello @M4tteoP https://github.com/M4tteoP if used xcaddy with build
34daaf8
https://github.com/corazawaf/coraza-caddy/commit/34daaf87f9ddaca2833461de59ebada21c902598
i got error invalid
go:
@.***:
invalid version: unknown revision 34daaf8
https://github.com/corazawaf/coraza-caddy/commit/34daaf87f9ddaca2833461de59ebada21c902598
I used xcaddy build 34daaf8
https://github.com/corazawaf/coraza-caddy/commit/34daaf87f9ddaca2833461de59ebada21c902598
--with github.com/corazawaf/coraza-caddy
—
Reply to this email directly, view it on GitHub
https://github.com/corazawaf/coraza-caddy/issues/42#issuecomment-1496361490,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAXOYAV2CB3NFMJA4P23R23W7RM2TANCNFSM6AAAAAAUPQYRVU
.
You are receiving this because you were mentioned.Message ID:
@.***>

Should be this one: xcaddy build --with github.com/corazawaf/coraza-caddy@34daaf87f9ddaca2833461de59ebada21c902598
In your attempt, you are trying to use the commit like it was a caddy commit, not a coraza-caddy one

Hello @M4tteoP  @jcchavezs
You have right I can compile using xcaddy build --with github....coraza-caddy@build_hash
But now the error change and I got a error with the CRS loaded.
["/usr/share/caddy/waf/coreruleset/rules/REQUEST-901-INITIALIZATION.conf","/usr/share/caddy/waf/coreruleset/rules/]

The coreruleset library isn't the problem. Loading filesystem is and that
is what I am fixing.
On Wed, 5 Apr 2023, 14:28 Juan Pablo Tosso, @.***>
wrote:

@jcchavezs https://github.com/jcchavezs I would remove your coreruleset
library from coraza-caddy until it is fixed. It's not such an important
feature for the connector, and it's not even documented
—
Reply to this email directly, view it on GitHub
https://github.com/corazawaf/coraza-caddy/issues/42#issuecomment-1497405210,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAXOYAWDPIAWJ23JFDCCWL3W7VQNFANCNFSM6AAAAAAUPQYRVU
.
You are receiving this because you were mentioned.Message ID:
@.***>

Yeah I will remove that. And reassess the os filesystem.

In the other hand, the coreruleset library eases testing in this repo which
we really need it to avoid poor coverage. I'd rather make that work to not
to have to download CRS for ftw.
On Wed, 5 Apr 2023, 14:28 José Carlos Chávez, @.***> wrote:

The coreruleset library isn't the problem. Loading filesystem is and that
is what I am fixing.
On Wed, 5 Apr 2023, 14:28 Juan Pablo Tosso, @.***>
wrote:

@jcchavezs https://github.com/jcchavezs I would remove your
coreruleset library from coraza-caddy until it is fixed. It's not such an
important feature for the connector, and it's not even documented
—
Reply to this email directly, view it on GitHub
https://github.com/corazawaf/coraza-caddy/issues/42#issuecomment-1497405210,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAXOYAWDPIAWJ23JFDCCWL3W7VQNFANCNFSM6AAAAAAUPQYRVU
.
You are receiving this because you were mentioned.Message ID:
@.***>

@carlos-herrer please do try this branch https://github.com/corazawaf/coraza-caddy/pull/52

Hello @jcchavezs, I got the same error Permission deny.
with "SecDebugLog /var/log/coraza/coraza.log" with "SexDebugLogLevel 6" I got invalid sintax.
and if I using audit log "SecAuditLogDir /var/log/audit/audit_coraza.log", it's generate a permission error.

Hey there,
any updates on this? Still can't get logs working for coraza.

Still having the same problem. I builded caddy by xcaddy build --with github.com/corazawaf/coraza-caddy/v2@latest . All log just show on screen console. Can't find in any log file.

I will look into this again.

I changed the example to show logs https://github.com/corazawaf/coraza-caddy/pull/173. Unfortunately I did not face any issue with logs.

check https://github.com/corazawaf/coraza-caddy/pull/173/files


[BUG] MLP always need RBAC for creating namespaces

During the MLP deploy phase there is this code:
if _, err := helper.Create(namespace, false, namespaceInfo[0].Object); err != nil && !apierrors.IsAlreadyExists(err) {
		return nil, err
	}

for ensuring the existence of the namespace.
For a variety of reasons, this is suboptimal, it si best to at least first, do a GET to see if exists and/or mlp don't have the right permissions to do this action, because is cannot be always granted.
Best solution is to add a new flag to skip the process entirely, something like "--skip-ensure-namespace" and add an if on the entire ensureNamespace function.

There is a --ensure-namespace flag already handled, which skip the namespace creation if not needed. It is sufficient for the usecase?


Update and propagate md5 hash of existing file

Example: When I change a file in a way that doesn't affect processing of the dependent files (e.g. adding a comment to a script file, adding an extra column to a CSV file). I want to tell DVC that the file content (=md5 hash) changed but that I don't want to process the dependent files again.
I think there are two options:

Running a command on the changed file:

For data files, this could be done by running dvc add again. Currently, when you run dvc add file.csv on a data file, the existing file.csv.dvc is overwritten. Instead, you could overwrite just the output md5 field.
The md5 change would propagate into directly dependent .dvc files.
However, we don't want to run dvc add on non-data files (e.g. scripts). Maybe this would have be a whole new command like dvc propagate file.csv?

Running a command on the dependent .dvc files:

Update the .dvc md5 fields based on the current deps and outs file content. This is basically the reverse of dvc repro, maybe it could be added as an argument like dvc repro --refresh?
Probably easier to implement, but harder to use since the user would have to run on all dependencies separately (but again, maybe that's desirable?)

Do you think that would be useful?
Thanks! I'm excited about this tool very much! 👏

Thank you @prihoda! It looks like a useful and very practical feature. I feel it like git squash - hack by nature but sometimes it is very useful. I don't think we can avoid this kind of hacks in real life.
A couple more open questions:

Should DVC write additional information about this trick? Something like propagated: true.
Adding a comment to a script file is a good scenario. But editing data file is questionable. For data sources - why don't we just remove and add a new data file? For data derivatives - we probably should not support this feature since it breaks reproducibility chain. So, should DVC support that for data files at all?

About no. 2: As you said, it's a bit of a hack, the user would have to know about the impact. I think it's useful even for data files, in case they are really expensive to reproduce, you changed their dependencies, but the change can be applied by hand instead.
E.g. I realized I want a different column name in a CSV file -> change the script that generates the file, apply the change to the file by hand, run dvc propagate.

This would also be useful when using DVC for an existing project. Currently there is no way to create valid .dvc files (with dependencies and build command) from existing files without reproducing them.

Hi @prihoda !
Thank you for your feedback! These are indeed some very interesting ideas.

This would also be useful when using DVC for an existing project. Currently there is no way to create valid .dvc files (with dependencies and build command) from existing files without reproducing them.

There is actually a way to do that right now. If you really need that kind of hack, you can simply write dvc files by yourself, without dvc run. Dvcfiles have very simple yaml format, so there should be absolutely no problem to write it by-hand. I.e. lets say I have my script.py, input.data and output.data files that are already there. Now I can hack my way in by simply creating script.dvc and calculating md5 of imput.data, output.data and script.py with md5sum utility myself and pasting the checksums into the script.dvc's deps and outs(just look at any existing dvc file to get the idea where to put what).
What worries me the most about dvc propagate is just how easy it is to shoot yourself in the leg, if you mess up in any of the steps and break the reproducibility only to realize that your project is broken some time later when you actually need to reproduce it. Thus I don't think that we should encourage that kind of behavior, instead we should simply suggest rebuilding your pipeline with dvc. Yes, it will take some time to rebuild the pipeline, but at least you will be 100% sure that your project actually works, which is the most important thing in my opinion. That being said, I do realize that there are use cases for dvc propagate, but I'm not yet convinced that we should actually embrace it and include such feature into core dvc. Lets leave this issue open for now, we will discuss it more carefully soon and come back to it.

Great to know! I am editing .dvc files but didn't realize I can generate the hashes so easily.
After using DVC for a while I see that the dvc propagate command would violate the guarantees you want to create in the first place. I guess this would only make sense as an extension for the dvc add command - to be able to specify dependencies (and also the build command if possible).


MAINT: remove more NumPy 1.6 compatibility hacks

Thanks!

FTR, for me it's the same whether it's the same PR or a separate one.


Feature Request - Sort Media

Discussed in https://github.com/CyferShepard/Jellystat/discussions/47

Originally posted by NeuroDawg June  7, 2023
Now that Jellystat is detecting all my media, it would be nice that when I look at the media, in a library, to see it sorted. Right now it's not clear that there is any type of sort order. The screenshots below shows the media of one of my libraries; it would be nice to have it sorted properly. Right now it looks like some items are sorted by collection title, but then other collections aren't sorted at all.

This has been added in the unstable branch


Login error (2401): username or password wrong

Hello,
I get the following error when configuring the application:
Login error (2401): username or password wrong
I am using the "Smart Life" credentials with the same country as the app.
I tried changing the password and it doesn't work either.
I have other tuya devices integrated from the official integration.

I am having the same issue. I tried the same credentials I am using with Tuya Local, what am I missing here?

Hi, provide please debug logs of the problem. In order to enable debug log for the integration, just add next lines to your configuration.yaml file:
logger:
  default: info
  logs:
    custom_components.tuya_ble: debug

Hello,
Sorry for replying so late, I didn't get anything in the logs, but I managed to solve the problem.
For this I followed the official documentation of the Tuya integration, specifically this section: 2406: skill id invalid
Maybe it would be nice to add this information somehow in the integration documentation.


ReferenceError: process is not defined

My project is a vite react app, with tailwindcss
below is the simple code of rendering table
I don't know what it's not working
import Table from "react-tailwind-table";
import "react-tailwind-table/dist/index.css";

export default function TableTailwind() {
  return <Table columns={[]} rows={[]} />;
}

Identical setup and error:
Uncaught ReferenceError: process is not defined
at i2 (factoryWithTypeCheckers.js:170:9)
at l2 (factoryWithTypeCheckers.js:241:12)
at factoryWithTypeCheckers.js:116:12
at factoryWithTypeCheckers.js:590:10
at a (react-tailwind-table.js?v=3402fe69:26:10)
at factoryWithThrowingShims.js:16:1

I am currently experiencing the same issue with the library, using CRA.

Same Here i installed forcefully and when i run it, Uncaught ReferenceError: process is not defined


Nav dropdown blurry

The nav text in the nav dropdown seems to be blurry, I turned off the transform property in the element (    transform: translate3d(5px, 41px, 0px)) and it went back to normal, but this property seems to be added inline by js.

Same issue with me, it would be good if there will be any option to disable the popper.js on particular element.
Like:
$('.dropdown-toggle').dropdown({     "popperjs" : false })

You can do this with display: static.

Or you can disable gpu Acceleration in Popper.js with:
Popper.Defaults.modifiers.computeStyle.gpuAcceleration = false;

Closed because it's not a Bootstrap issue and you have solutions 😉


Allow anchors to be asssigned without breaking ObservablePoint

While trying to switch our games to PIXI 4 we came across loads of cases where we did
sprite.anchor = new PIXI.Point(0.5, 0.5);

which causes v4 to break in weird ways.
This PR uses the same approach used in https://github.com/pixijs/pixi.js/blob/dev/src/extras/BitmapText.js#L366 to make assignments save again and to make updating from v3 to v4 a little easier.

rebased, I hope this fixes the eslint failures

@marekventur looks like the unit-tests are failing. You can run npm run test:debug to use DevTools to debug what's happening there.

Yep, that's what I talked about long time ago :)

I'm sorry for the failing tests, I've forgot to update one reference.

Looks good @marekventur !
My one comment would be, how do you guys feel about the (typeof value === 'number') check as this is not consistent with our other point setters we have in pixi.

Happy to change it (both here and in BitmapText) if that's preferred.

I think its optional, but why not :) Anchors aren't set that often

Updated to only do the copy() bit without the typeof value === 'number' check

Nice one @marekventur 👍

thanks!


Add syrupy library for snapshot tests

Proposed change

SSIA
Additional information

This PR fixes or closes issue: fixes #

Checklist

[ ] I have updated the documentation if needed.
[ ] I have updated the tests if needed.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (3780e91) 100.00% compared to head (2879738) 96.36%.

Additional details and impacted files
@@             Coverage Diff             @@
##              main     #337      +/-   ##
===========================================
- Coverage   100.00%   96.36%   -3.64%
===========================================
  Files            5        5
  Lines          165      165
  Branches        31       31
===========================================
- Hits           165      159       -6
- Misses           0        6       +6

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


SAK-32572 Messages Tool: Allow all Member Roles from the Groups of wh…

…ich the current user is a member in the 'To' field

Hi Earl,
Changes have been addressed.
Thanks.


Create KandaneAlgorithm.java

Added kandane algorithm

Sorry to inform but this repository has been excluded from hacktoberfest. So inorder to make your PR count towards hacktoberfest, request you to make your contribution in this repository -> https://github.com/SR-Sunny-Raj/Hacktoberfest2022-for-everyone


usbhost: hid_parser.c is compiled unconditionally and is possibly dead code

in usbhost/Make.defs hid_parser.c is added to CSRCS before ifeq ($(CONFIG_USBHOST),y) so it is always compiled, no matter if USB is used or not.
It seems that the functions are not used by any code that is publicly available.
So just moving it inside the CONFIG_USBHOST block is probably not the right thing/fix.
Maybe another config option should be added? Does anyone know where the code is used?

It is not "dead code" in that even if it is unnecessarily compiled, it will not be linked.  So no harm done other than wasted compile time.
It was moved out of the conditional logic with commit 1ba22407623cb60c54e89969c13ea2e227dff9fd in 2011 but I don't know why.  That was too long ago to reconstruct what was going on.  Perhaps it was used by other logic at some time?  Perhaps with was used by a HID device side driver (drivers/usbdev)?
This is third party code that could have some value in the future.  Any complex HID driver would need this parsing logic.


Fix a crash in Utf8PrintfString::CreateFromVaList for long strings on unit systems

Backport of (#66)

should the target branch name be release/3.5.x ?


Add PFNs4BO sampler

Contributor Agreements
Please read the contributor agreements and if you agree, please click the checkbox below.

[x] I agree to the contributor agreements.

Motivation

PFNs4BO was recently proposed by Samuel Müller, Matthias Feurer, Noah Hollmann, and Frank Hutter in https://arxiv.org/abs/2305.17535. It is highly interesting algorithm and can be applied to any BO problem. This PR aims to add the method in the OptunaHub.
Description of the changes

Add PFNs4BO sampler

@y0z Could you review this PR?

I conducted benchmark experiments of PFNs4BO against the random search. PFNs4BO looks promising.

This figure stands for the performance of PFNs4BO against the random search. The horizontal axis represents the amount of  budgets, which is equal to the number of trials. The vertical axis represents the p-value of the Mann–Whitney U test, in which the null hypothesis is that the PFNs4BO loses the random search. Therefore, don't get me wrong, but the lower is better.
The benchmark consists of 70 problems. The 8 problems are from HPO tabular benchmarks in https://github.com/automl/HPOBench (original 4 problems, and log-converted 4 problems). The 62 problems are several synthetic functions in https://github.com/sigopt/evalset (Not just use it, but I re-implement them).

@SamuelGabriel Thank you for providing several important comments. I addressed all of your suggestions. Thank you vary much.

using power transform would improve performance further

I will use apply_power_transform = True in the acquisition function optimization. Thanks for the information.

you could also have something in place to use random search after 100 or 200 trials, as PFNs4BO becomes worse then.

Sounds very good. That type of algorithm change is easily possible in Optuna. Users can change the algorithm as follows. I will add that instruction in the docstring and README.
sampler1 = PFNsBOSampler()
study = optuna.create_study(sampler=sampler)

study.optimize(objective, n_trials=200)

sampler2 = RandomSampler()
study.sampler = sampler2
study.optimize(objective, n_trials=100)

Other than that: I did not double check all hyperparameters, but if you use the ones in the code (which are the same as in the paper), you should be fine.

I used completely same hyperparameters! Thank you for the information.

@y0z Could you review this PR?


Add comments to methods in the new packages

A couple of new packages were created in the monorepo and are missing comments that will be handled by TypeDoc to build the docs (like the safe-core-sdk package: https://gnosis.github.io/safe-core-sdk/).
Namely:

safe-web3-lib
safe-ethers-lib
safe-core-sdk-utils

Hey @germartinez. Is this still necessary after we revamp the repo and merge all these packages? Are we producing any automated docs?


Get GoViral viral profiling data online and exportable

Reviewing files from GoViral. Sent query about raw data for 2014-2015 season.

Got data, need to map to report and user IDs in Firebase. Plan is to load this in a "results" field that's keyed by Firebase user ID.

Waiting to hear back regarding difficulties in mapping 2013-14 data.