[BUG] Negative tests for kubescape

Describe the bug
We need three more negative tests for kubescape
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
The following specs should pass by using negative tests to check if kubescape fails

Security 'insecure_capabilities' should fail on a cnf that has containers with insecure capabilities
Failure/Error: (/PASSED: Containers with insecure capabilities were not found/ =~ response_s).should be_nil
Expected: 2202 to be nil
spec/workload/security_spec.cr:116

Security 'dangerous_capabilities' should fail on a cnf that has containers with dangerous capabilities
Failure/Error: (/PASSED: Containers with dangerous capabilities were not found/ =~ response_s).should be_nil
Expected: 2202 to be nil
spec/workload/security_spec.cr:142

Security 'application_credentials' should fail on a cnf that allows applications credentials in configuration files
Failure/Error: (/FAILED: Found applications credentials in configuration files/ =~ response_s).should_not be_nil
Expected: nil not to be nil
spec/workload/security_spec.cr:168

@wavell @taylor is this issue still relevant for cnti?


JS file - "function" in GUI mode has red background

When editing JS code in GUI mode, 'function' word has annoying red background.
It's caused by guibg=red in darcula.vim. Seems mistake.
line 96:
 hi javaScriptFunction ctermfg=222 ctermbg=NONE cterm=NONE guifg=#d58b48 guibg=red gui=bold

Could you please fix it? (i.e. guibg=NONE).
Thanks!

Should be fixed.  Was trying to see the highlight though didn't show up with my distribution (Janus).


Initial Gantt chart

Current coverage is 99.33%

Merging #52 into master will not affect coverage as of 1619475

@@            master     #52   diff @@
======================================
  Files            2       2
  Stmts          150     150
  Branches         0       0
  Methods          0       0
======================================
  Hit            149     149
  Partial          0       0
  Missed           1       1

Review entire Coverage Diff as of 1619475

Powered by Codecov. Updated on successful CI builds.


Apps no longer supported

Hey man. Ive started to notice recently that some apps wont adhere to the dark mode but they used to before. One of them that I used today is AliExpress. I cant think of the other one I saw off the top of my head. But anyways Im on Lineage OS (10) with root.

Oops I had a setting wrong.

Oops I had a setting wrong.

Reopening because AliExpress still doesnt work for me. But its currently the only app causing issues.

Reopening because AliExpress still doesnt work for me. But its currently the only app causing issues.

Please read the FAQ, apps are able to disable force dark. The 2.0 update may improve this, if the app has simply optimised its startup, but if it's still not working, there's nothing I can do. The Xposed module may help you, but again I cannot make guarantees.

It didnt help but thanks anyways. I have no desire to use xposed again.


Execution failed for task ':react-native-maps:compileDebugJavaWithJavac'.

Hi, when i put playServicesVersion = "17.0.0" on build script, Excution failed for task error occur and build is fail.
With out playServicesVersion, build is success but app is crashed when MapView is called.
environment
rn : 0.68.0
react : 17.0.2
buildToolsVersion = "31.0.0"
        minSdkVersion = 23
        compileSdkVersion = 31
        targetSdkVersion = 31
        ndkVersion = "21.4.7075529"
        kotlinVersion = '1.5.0'
        supportLibVersion   = "23.1.1" // default: 23.1.1
        playServicesVersion = "17.0.0" // or find latest version
        androidMapsUtilsVersion = "2.3.0"

Same here i can't build too.

+1

same issue facing here  Execution failed for task ':react-native-maps:compileDebugJavaWithJavac'.
buildscript {
ext {
buildToolsVersion = "30.0.2"
minSdkVersion = 21
compileSdkVersion = 31
targetSdkVersion = 31
ndkVersion = "21.4.7075529"
supportLibVersion   = "30.0.2"
    playServicesVersion = "17.0.0" // or find latest version
    googlePlayServicesVisionVersion = "19.0.0"
    androidMapsUtilsVersion = "2.3.0"
}

I'm also having this error

I am still facing this issue on react-native-maps version 1.4.0
environment :
react-native : 0.71.2, react-native-maps : 1.4.0, react : 18.2.0
gradle settings :
    buildToolsVersion` = "33.0.0"
    minSdkVersion = 21
    compileSdkVersion = 33
    targetSdkVersion = 33
    ndkVersion = "20.1.5948944"
    playServicesVersion = "17.0.0"
    androidMapsUtilsVersion = "2.3.0"
    androidXCore = "1.7.0"

Still facing this issues, fresh install. Any word?

is this issue resolved? if yes then how?


When upgrading through suc the hosts file get extra entries from the pod

Kairos version:

PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
KAIROS_ID_LIKE="kairos-standard-ubuntu-24.04"
KAIROS_IMAGE_LABEL="24.04-standard-amd64-generic-v3.2.1-k3sv1.31.1-k3s1"
KAIROS_ARTIFACT="kairos-ubuntu-24.04-standard-amd64-generic-v3.2.1-k3sv1.31.1+k3s1"
KAIROS_FLAVOR="ubuntu"
KAIROS_FLAVOR_RELEASE="24.04"
KAIROS_FAMILY="ubuntu"
KAIROS_MODEL="generic"
KAIROS_NAME="kairos-standard-ubuntu-24.04"
KAIROS_BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues"
KAIROS_SOFTWARE_VERSION="v1.31.1+k3s1"
KAIROS_TARGETARCH="amd64"
KAIROS_GITHUB_REPO="kairos-io/kairos"
KAIROS_VERSION="v3.2.1-v1.31.1-k3s1"
KAIROS_REGISTRY_AND_ORG="quay.io/kairos"
KAIROS_HOME_URL="https://github.com/kairos-io/kairos"
KAIROS_ID="kairos"
KAIROS_PRETTY_NAME="kairos-standard-ubuntu-24.04 v3.2.1-v1.31.1-k3s1"
KAIROS_IMAGE_REPO="quay.io/kairos/ubuntu:24.04-standard-amd64-generic-v3.2.1-k3sv1.31.1-k3s1"
KAIROS_VARIANT="standard"
KAIROS_RELEASE="v3.2.1"
KAIROS_SOFTWARE_VERSION_PREFIX="k3s"
KAIROS_VERSION_ID="v3.2.1-v1.31.1-k3s1"
CPU architecture, OS, and Version:

Linux localhost 6.8.0-45-generic #45-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 30 12:02:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Describe the bug

When upgrading using the routine documented for system-upgrade-controller, the containers /etc/hosts file seemingly gets merged with the hosts file and ends up with more and more entries.
This is after two upgrades using suc since provisioning:
root@localhost:~# cat /etc/hosts
# Kubernetes-managed hosts file (host network).
# Kubernetes-managed hosts file (host network).
<IP_ADDRESS>       localhost
<IP_ADDRESS>       localhost
<IP_ADDRESS>       localhost
root@localhost:~#

To Reproduce

using the plan shown in documentation here, apply and upgrade a cluster. After reboot, check the /etc/hosts file:
root@localhost:~# cat /etc/hosts
<IP_ADDRESS>       localhost
root@localhost:~# kubectl apply -f upgrade.yaml
plan.upgrade.cattle.io/os-upgrade configured
root@localhost:~#
Broadcast message from root@localhost (Thu 2024-10-10 14:17:44 UTC):

The system will reboot now!
Connection to <IP_ADDRESS> closed.
$ ssh <IP_ADDRESS>
$ cat /etc/hosts
# Kubernetes-managed hosts file (host network).
<IP_ADDRESS>       localhost
<IP_ADDRESS>       localhost
$

Expected behavior

It should work the same way as when upgrading using the kairos-agent ugrade command directly, so result in a hostsfile equal to the one we started with:
root@localhost:~# cat /etc/hosts
<IP_ADDRESS>       localhost
root@localhost:~#

Logs

Not been able to find any logs indicating what's gone wrong here
Additional context

Does not seem to affect upgrades using kairos-agent upgrade directly.
My guess is that it's related to the suc upgrade using the containers root as a source, but haven't yet found how it could be prevented. Not aware of any other files being affected in a similar way.

yep, confirmed.
Built with master, k3s image, set a single node k8s, upgrade with system-upgrade-controller, result sin duplicated lines in /etc/hosts
kairos@kairos-k3s:~$ cat /etc/hosts
# Kubernetes-managed hosts file (host network).
<IP_ADDRESS> localhost kairos-k3s
<IP_ADDRESS> localhost kairos-k3s

Not sure whats going on, lol

can be reproduced by running the initramfs stage several times with kairos-agent run-stage initramfs
Seem like yip is not picking up or checking that the line exists?

using yip directly seems to work though??

ah seems that its the 31_host file from system/oem and its only run in initramfs.before

yes, somehow the check is failing so it recreates the hostname..

Could it be just the check in 31_host? Seems a bit strange that it would add the extra comment as well from the container then as in the example under describe the bug? 🤔
Checked on a cluster that had been upgraded a few times using suc, and ends up with one extra comment and one extra hosts line for each upgrade plus the original one

Went and tested in a vm, and seems like it's like you're saying and the extra <IP_ADDRESS> localhost line is simply the 31_hosts adding an extra line.
The extra comment line being added seems to be just Kubernetes mounting the node hosts file into the container and then adding its own comment to the top. (again)
Set up a pod that is about equal to the suc-container and there it is:
root@localhost:~# kubectl exec -it -n system-upgrade suc-busybox-test -- sh
/ # cat /etc/hosts
# Kubernetes-managed hosts file (host network).
# Kubernetes-managed hosts file (host network).
<IP_ADDRESS>       localhost
<IP_ADDRESS>       localhost
/ # exit

root@localhost:~# cat /etc/hosts
# Kubernetes-managed hosts file (host network).
<IP_ADDRESS>       localhost
<IP_ADDRESS>       localhost

I guess the second problem could be solved by dropping comments in the hosts file in yip (unless that's something you guys aim to not modify) somewhere in this loop :thinking:
https://github.com/mudler/yip/blob/master/pkg/plugins/hostname.go#L74-L82

This patch seems to alleviate it, after 2 upgrades I no longer get the duplicated entries: https://github.com/kairos-io/packages/pull/1113/
I do get duplicated comments though. I still dont get why. If k8s mounts stuff under /etc/hosts from the host into the container, thats ok but the upgrade should just ignore that and copy it. Plus, /etc is ephemeral so after a reboot it should go away?
The only thing I can see touching that file is that yaml file... No idea where the duplication comes from, could it be that the plugin is adding extra lines somehow? But maybe the underlying /etc in the image does have the /etc/hosts duplication ??
Im really confused over this one

Looking at the Kairos-container available in the kairos registries it contains an etc/hosts file, but this is empty so that explains why it is reset once you upgrade with kairos-agent upgrade $image
On the other hand when it's ran as pod in kubernetes, then kubernetes will give it a hosts file based on either cluster-network or node (host-network). This seems to be what is causing problems here, I don't really have a good solution for this though as long as the pod-fs is mounted and used to upgrade.
My suggestion about overwriting existing file or copying the file from host doesn't seem to work though based on some testing since I couldn't find a way to delete/overwrite the file from the pods.
Not sure how you could get around this? I assume the upgrade command when provided the --source just takes the entire OS there and packs it up?
kubernetes doc: https://kubernetes.io/docs/tasks/network/customize-hosts-file-for-pods/

Not sure how you could get around this? I assume the upgrade command when provided the --source just takes the entire OS there and packs it up?

Yep, it does. Maybe we should either skip the hosts file or overwrite it on each boot before filling the hostname? so we start from initramfs with a clean hosts file that we know its "clean" on each boot?

mmmh, going into the upgrade container I can see this:
/dev/disk/by-label/COS_PERSISTENT on /etc/hosts type ext4 (rw,relatime)
so its storing the hosts file in the persistent partition.

somehow somewhere, with the patch this suddenly seems to be fixed. There is also another patch that may affect this, that changes the config read paths as we were not reading the current system paths for configs (https://github.com/kairos-io/kairos-agent/pull/579)
I could not reproduce it anymore with framework 2.14.1 (latest agent and cloud configs). I need to try it again tomorrow freom a clean image though, but it may have gone away.


Crop images command fails

Running ruby-2.0.0-p247 and PhantomJS 1.9.2
Error occurs when running full capture task or crop command individually.
Full stack trace below:
cropping images
 /Users/matt/Code/wraith/lib/wraith/wraith.rb:72:in ``': No such file or directory - convert (Errno::ENOENT)
	from /Users/matt/Code/wraith/lib/wraith/wraith.rb:72:in `crop_images'
	from /Users/matt/Code/wraith/lib/wraith/crop.rb:32:in `crop_images'
	from /Users/matt/Code/wraith/lib/wraith/cli.rb:61:in `crop_images'
	from /Users/matt/Code/wraith/lib/wraith/cli.rb:90:in `capture'
	from /Users/matt/.rvm/gems/ruby-2.0.0-p247/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
	from /Users/matt/.rvm/gems/ruby-2.0.0-p247/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
	from /Users/matt/.rvm/gems/ruby-2.0.0-p247/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
	from /Users/matt/.rvm/gems/ruby-2.0.0-p247/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
	from /Users/matt/Code/wraith/bin/wraith:7:in `<top (required)>'
	from /Users/matt/.rvm/gems/ruby-2.0.0-p247/bin/wraith:23:in `load'
	from /Users/matt/.rvm/gems/ruby-2.0.0-p247/bin/wraith:23:in `<main>'
	from /Users/matt/.rvm/gems/ruby-2.0.0-p247/bin/ruby_executable_hooks:15:in `eval'
	from /Users/matt/.rvm/gems/ruby-2.0.0-p247/bin/ruby_executable_hooks:15:in `<main>'

I'll have a look into it but I thought I'd drop the error in here first for reference.

@zeroxme this also occurred for me when running the standard config with a single domain, same as @nikglavin

@jonnywyatt can you please post your exact config and the output of the failing run (including the command you ran)?
In capture mode, Wraith is intended to run against two domains. If you want to compare differences over time on a single domain, you probably want to use history mode.


Added script to create missing workflow tags

Added "text conversion needed" tags for all references, mods with "file uploaded"

Thanks @sweng66 !


Disable cron on version 3 hosts

Disable the whenever gem or its capistrano configuration so that if we deploy any updates to version 3, the crontabs will not be updated.

@whereismyjetpack just cleared out the crontabs in each SS3 host. If ever deploy SS3 again, we'd need to remove the cap job that updates the tabs, but that is unlikely to happen.


AssertionError: Cookie ./.auth/gitlab_state.json expired.

Hi Thanks for your great work!
I've followed the instructions to set up the docker environments. But when I run this file: python browser_env/auto_login.py . It continually yields this Assertion error: AssertionError: Cookie ./.auth/gitlab_state.json expired.
I checked and found that only this cookie file expires. Could you help me to look into this problem? Many thanks!

Hi @Yangyi-Chen, unfortunately, I wasn't able to reproduce the error from my end. Are you still facing this error? If so, can you check if you could login via http://ec2-3-131-244-37.us-east-2.compute.amazonaws.com:8023/explore?

Thanks for your quick response. The amazing thing is I do not have this error when I rerun the same code. so i close this issue.


epic-kitchens_55 error

Hi,
I'm having this errorError running Epic-Kitchens-55. Have you encountered this before? Thanks
Save file name  anti_mod_rgb_span_6_s1_5_s2_3_s3_2_recent_2_r1_1.6_r2_1.2_r3_0.8_r4_0.4_bs_10_drop_0.3_lr_0.0001_dimLa_512_dimLi_512_epoc_15_vb_nn
Printing Arguments
Namespace(add_noun_loss=True, add_verb_loss=True, alpha=1, batch_size=10, best_model='best', debug_on=False, display_every=10, dropout_rate=0.3, ek100=False, epochs=15, img_tmpl='frame_{:010d}.jpg', json_directory='tempAgg_ant_rec//models_anticipation/', latent_dim=512, linear_dim=512, lr=0.0001, modality='rgb', mode='train', noun_class=352, noun_loss_weight=1.0, num_class=2513, num_workers=0, past_attention=True, path_to_data='/content/drive/MyDrive/Individual_Project/Models/RULSTM/rulstm-master/RULSTM/data/ek55', path_to_models='models_anticipation/ek55', recent_dim=2, recent_sec1=1.6, recent_sec2=1.2, recent_sec3=0.8, recent_sec4=0.4, resume=False, scale=True, scale_factor=-0.5, schedule_epoch=10, schedule_on=1, span_dim1=5, span_dim2=3, span_dim3=2, spanning_sec=6, task='action_anticipation', topK=1, trainval=False, verb_class=125, verb_loss_weight=1.0, verb_noun_scores=True, video_feat_dim=1024, weight_flow=0.1, weight_obj=0.25, weight_rgb=0.4, weight_roi=0.25)
Populating Dataset: 100% 23493/23493 [00:33<00:00, 694.22it/s]
Populating Dataset: 100% 4979/4979 [00:07<00:00, 689.38it/s]
Add verb losses
Add noun losses
/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of lr_scheduler.step() before optimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order: optimizer.step() before lr_scheduler.step().  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [1,0,0] Assertion t >= 0 && t < n_classes failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [4,0,0] Assertion t >= 0 && t < n_classes failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [7,0,0] Assertion t >= 0 && t < n_classes failed.
Traceback (most recent call last):
File "main_anticipation.py", line 674, in
main()
File "main_anticipation.py", line 531, in main
start_epoch, start_best_perf, schedule_on)
File "main_anticipation.py", line 400, in train_validation
loss.backward()
File "/usr/local/lib/python3.7/dist-packages/torch/_tensor.py", line 307, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/usr/local/lib/python3.7/dist-packages/torch/autograd/init.py", line 156, in backward
allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: CUDA error: device-side assert triggered

This error occurs when the labels are outside the range of the outputs of the model.
I ran the code again and didn't encounter such a problem.
Could you check if you are mistakenly using EPIC-100 annotations?
You can print the max label number for verb/noun/action and compare them with the classifier head sizes.


Unescaped query parameter

In Daedalus, here, the query parameters do not seem to be escaped.
Since it looks like we already depend on purescript-contrib, it might be useful to also depend on purescript-form-urlencoded to fix this problem.
@akegalj

Thanks @toshipiazza . Initial discussion was done over email:
by @toshipiazza :
Hi all,

Sorry if I messaged the wrong people, I still have some catching up to
do with names.

As referenced in the chat earlier today, I did a simple once-over of
Daedalus (understanding it's not yet complete) and noticed an
un-escaped query-parameter,
here: https://github.com/input-output-hk/cardano-sl/blob/master/daedalus/src/Daedalus/BackendApi.purs#L45.

This could simply be fixed by escaping the l and r variables. Since I
believe we're already using purescript contrib, it looks like there
exists a purescript-contrib package for url-encoding
strings, https://github.com/purescript-contrib/purescript-form-urlencoded.

I will submit an issue to the issue tracker. Is there a particular
format with which I should submit an issue? All issues I can see are
of the form "[CSL-XXX] <commit message>". Should I use this numbering
scheme as well?

Thanks,
Toshi

response by me:

> Sorry if I messaged the wrong people, I still have some catching up to
do with names.
I am very sorry if this is stupid question but are you part of one of
IOHK teams ? (its not clear to me at this point)

> As referenced in the chat earlier today
can you point me to the chat. Is it in some IOHK Slack channel ? (if you
are part of the IOHK)

> noticed an un-escaped query-parameter
yes, you are right. We are not escaping query parameters. Most of
sensitive data is sent over HTTP body so there was no use case to escape
it yet. This is a bad practice and we should escape query params.

> This could simply be fixed by escaping the l and r variables. Since I
believe we're already using purescript contrib, it looks like there
exists a purescript-contrib package for url-encoding
strings, https://github.com/purescript-contrib/purescript-form-urlencoded
thank you for the suggestion. I wasn't aware of this package. This is
exactly what fixes the problem

> I will submit an issue to the issue tracker. Is there a particular
format with which I should submit an issue? All issues I can see are of
the form "[CSL-XXX] <commit message>". Should I use this numbering
scheme as well?
You can submit the issue to
https://github.com/input-output-hk/cardano-sl repo . You don't have to
use particular scheme. We use internally numbering scheme because we
have internal issue tracker that manages this. You can just create no
name issue and we will edit the issue accordingly when we mange its
numeration internally (as I am not aware are you part of the IOHK atm)
.  You can assign issue to me (github alieas: akegalj) and we can
continue discussion there.

@toshipiazza this was just merged to cardano-sl-0.5-rc branch https://github.com/input-output-hk/cardano-sl/pull/991#pullrequestreview-45914378
Thanks for reporting it


copr: vectorize Inet6Aton

Signed-off-by: mak<EMAIL_ADDRESS>
PCP #5751
What have you changed?
Add vectorized Inet6Aton
What is the type of the changes?
Improvement (a change which is an improvement to an existing feature)
How is the PR tested?
Please select the tests that you ran to verify your changes:

Unit test

Does this PR affect documentation (docs) or should it be mentioned in the release notes?
No. It should not change the behavior.
Does this PR affect tidb-ansible?
No

@sre-bot /run-integration-copr-test copr-test=pr/80

@overvenus @5kbpers @rleungx PTAL, thanks!

Please fix conflicts. Thanks!

/test


Automatic build + download not working on Windows host

On a Windows  host the automatic build + download is not working. The soure/oopsy.js script performs a "make -n" i.s.o. a "make" (line 254).

Hi Stephen,
The problem I was experiencing had to do with os.EOL being used to split the make -n output into executable commands. Using '\n' instead corrects this  (possibly 'cause I had my Cygwin bin in my path)..
When following the new installataion instrcutions for Windows this problem was solved.
But still a glitch remains in the build and flash sequence (see the Max console log below).
Regards,
Marcel du Bois
newtools.txt

"(see the Max console log below)" -- can't see the log here.

There's a file attached (newtools.txt) to the previous comment

Just to clarify things: building and flashing works fine now. The remaining glitch just causes the flashing to be attempted twice.


Adds the guide page custom-entity-full

Oh yeah, same branch means this just updated the PR...anyway, adds a jekyll include for youtube videos plus fixes the formatted on the json parts of my previous translation.


Allow users to configure insecure registries

This is useful when users want to pull images from private registries.
Signed-off-by: Kiefer Chang<EMAIL_ADDRESS>How to test
Use sesdev to prepare an env and stop before deploy.
Bring up a docker registry and push an image into it.
e.g. sync latest Ceph image to <IP_ADDRESS>:5000
skopeo copy --dest-tls-verify=false docker://ceph/daemon-base:latest-master-devel docker://<IP_ADDRESS>:5000/ceph/daemon-base:latest-master-devel

Add registries to pillar:
ceph-salt config /Containers/Insecure_Registries add <IP_ADDRESS>:5000

Specify an image from the insecure registry
ceph-salt config /Containers/Images/ceph set <IP_ADDRESS>:5000/ceph/daemon-base:latest-master-devel

Deploy
ceph-salt -ldebug deploy --non-interactive

What is the need that this feature is addressing? It's already possible to point ceph-salt at the latest upstream master container image.

What is the need that this feature is addressing? It's already possible to point ceph-salt at the latest upstream master container image.

I need this feature for:

Speedup the testing of deployment. Every node pulls image from a local registry, not from the Internet.
Patch my WIP codes into images for testing.

I'm not 100% sure that this feature should be part of ceph-salt.
I see ceph-salt as a tool that uses a registry, but setting up a registry seems a bit out of scope to me.
ATM this feature is mainly useful for development environment so I wonder if we can simply add instructions to README on how to set up local/insecure registries that can then be used on ceph-salt config /Containers/Images/ceph set ..., instead of adding support for automatically setup those registries.
I might be missing wrong, thoughts?

@bk201 It seems like this feature is only for developers, so maybe sesdev would be a better place for it?

Thanks for the feedback.

I'm not 100% sure that this feature should be part of ceph-salt.
I see ceph-salt as a tool that uses a registry, but setting up a registry seems a bit out of scope to me.
ATM this feature is mainly useful for development environment so I wonder if we can simply add instructions to README on how to set up local/insecure registries that can then be used on ceph-salt config /Containers/Images/ceph set ..., instead of adding support for automatically setup those registries.
I might be missing something, thoughts?

The formula doesn't set up the registry. It just adds the registries to the exception list for podman.

@bk201 It seems like this feature is only for developers, so maybe sesdev would be a better place for it?

Agree. Close this and do it in the sesdev.

This is not only for developers. It's super useful for eg. airgaped installations, registry pull through caches, ...
The idea here is, that you configure the clients (podman on all nodes) to use another registry or use multiple registries (in case one is unavailable). With that, you could setup a local registry (or a pull through cache) and that would reduce the needed bandwith a lot. Eg. having a cluster with 200 nodes would require to download the ceph image 200 times for every update.

@toabctl If it's superuseful (and I'm not disputing that), why implement the feature only for Salt users? Why not have it in cephadm itself?

@toabctl If it's superuseful (and I'm not disputing that), why implement the feature only for Salt users? Why not have it in cephadm itself?

Fine for me. But imo definitely not a sesdev thing. But you could argue that chrony and friends could also be configured by cephadm .

@toabctl If it's superuseful (and I'm not disputing that), why implement the feature only for Salt users? Why not have it in cephadm itself?

Changing the system configuration wasn't really in scope for cephadm till now. (Well, with the exception of prepare-host which is not used by us.

I'm trying to test this PR without success.
My /etc/containers/registries.conf is configured, but podman still refuses to connect due to HTTP vs HTTPS issue.
Any idea what's wrong here?
node3:~ # cat /etc/containers/registries.conf
# For more information on this configuration file, see containers-registries.conf(5).
#
# Registries to search for images that are not fully-qualified.
# i.e. foobar.com/my_image:latest vs my_image:latest
[registries.search]
registries = ["docker.io"]

# Registries that do not use TLS when pulling images or uses self-signed
# certificates.
[registries.insecure]

registries = [

  "<IP_ADDRESS>:500",

]

# Blocked Registries, blocks the `docker daemon` from pulling from the blocked registry.  If you specify
# "*", then the docker daemon will only be allowed to pull from registries listed above in the search
# registries.  Blocked Registries is deprecated because other container runtimes and tools will not use it.
# It is recommended that you use the trust policy file /etc/containers/policy.json to control which
# registries you want to allow users to pull and push from.  policy.json gives greater flexibility, and
# supports all container runtimes and tools including the docker daemon, cri-o, buildah ...
[registries.block]
registries = []

node3:~ # podman pull <IP_ADDRESS>:5000/ceph/daemon-base:latest-master-devel
Trying to pull <IP_ADDRESS>:5000/ceph/daemon-base:latest-master-devel...ERRO[0000] Error pulling image ref //<IP_ADDRESS>:5000/ceph/daemon-base:latest-master-devel: Error initializing source docker://<IP_ADDRESS>:5000/ceph/daemon-base:latest-master-devel: pinging docker registry returned: Get https://<IP_ADDRESS>:5000/v2/: http: server gave HTTP response to HTTPS client
Failed
Error: error pulling image "<IP_ADDRESS>:5000/ceph/daemon-base:latest-master-devel": unable to pull <IP_ADDRESS>:5000/ceph/daemon-base:latest-master-devel: unable to pull image: Error initializing source docker://<IP_ADDRESS>:5000/ceph/daemon-base:latest-master-devel: pinging docker registry returned: Get https://<IP_ADDRESS>:5000/v2/: http: server gave HTTP response to HTTPS client
``'

Any idea what's wrong here?

My bad. I've missed one zero when setting the insecure repository (should be add <IP_ADDRESS>:5000 instead of <IP_ADDRESS>:500)

So I asked the caasp ppl about support for version2 of container-registry.conf. And V2 is supported . So I would prefer to use v2. According to caasp, that's also the only way to support air-gapped installations currently.

Codecov Report

Merging #113 into master will not change coverage by %.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #113   +/-   ##
=======================================
  Coverage   77.93%   77.93%
=======================================
  Files          10       10
  Lines        2116     2116
=======================================
  Hits         1649     1649
  Misses        467      467

Impacted Files
Coverage Δ

ceph_salt/config_shell.py
69.68% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 71c823d...fbdb2e0. Read the comment docs.

@ricardoasmarques @sebastian-philipp are you still fine with this change?

@bk201 could you rebase to get a CI run, please?

@bk201 could you rebase to get a CI run, please?

Done. But test failed when installing packages.

@bk201 needs another rebase...

Executing remove without location parameter results in a Traceback. Can we fail more gracefully here?

Fixed by raising appropriate exceptions. Thanks!


Failed to load NavMesh.asset because it was serialized with a newer version of Unity.

Hello
I opened the latest version of master in unity 2018.4.4 and whenever I open a test scene I encounter errors similar to this one:
Failed to load 'D:/Code/virtualhome_unity/Assets/Story Generator/TestScene/TestScene_6/NavMesh.asset' because it was serialized with a newer version of Unity. (Has a higher SerializedFile version)
One of the authors mentioned that the latest build was made using 2019.X.
Can you please specify the exact unity version that was used ?
Also, you should update the documentation and specify to open the project in unity 2019.X and not in 2018.4.
Thank you

Hi @simonobi! The current codebase uses Unity 2019.4.29 LTS. I will be updating the documents this week, thank you for letting me know!


Fix zoom when map container is scaled

Closes #5386, since that solution didn't work.
Fixes #2795.
@mourner mentioned, that reading transform can impact performance. So I'm calculating scale instead of reading it from stylesheet. I believe it will not impact performance. But still I would like to add memoization for scale calculation, since it rather wouldn't change often, and getMousePosition is fired every mousemove.
I'll also add a debug page with scaled map later.

Looks good to me!
P.S. Seeing a lot of well-thought out PRs from you recently, I thought it's time to invite you to the contributor team in Leaflet. Hopefully this will make working on Leaflet a bit easier — now you can work directly in this repo in separate branches, and it will be easy for others to add commits to your branches when working on something. You now also have the right to manage/close issues. Welcome!

@mourner thank you! ;)


ScottPlot5: basic Signal plot implementation

not optimized for speed - mostly just exploring the API, interfaces, and primitives...


feat: remove argocd api dependency

Removed the Argo CD API dependency from the repo
Bring all the dependencies up to date.

Tested and it seems fine 2024-11-14T03:49:34Z	INFO	setup	starting manager with unstructured Application

@xiangjingli PTAL Thanks!
This is the upstream community staging repo for the argocd pull model propagation controller. It's not directly linked to the multicloud integration repo in ocm-io or stolostron.
This might be the perfect repo for you to POC with the Maestro integration because it's nicely independent from Search API, GitOpsCluster and all the other multicloud-integration stuffs. I've already updated all the dependencies in this PR and if you look at the new go.mod file, you can see that it's quite clean. Hope that it helps.
As for the ACM issue regarding outdated argocd api, once you think this PR is merged. I will start bringing the changes over to the multicloud integration repo.

/lgtm


Logs are still redirected to stderr by default

What version of descheduler are you using?
descheduler version: v0.24.0
Does this issue reproduce with the latest release?
Yes
Which descheduler CLI options are you using?
None
Please provide a copy of your descheduler policy config file
What k8s version are you using (kubectl version)?
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.9", GitCommit:"b631974d68ac5045e076c86a5c66fba6f128dc72", GitTreeState:"clean", BuildDate:"2022-01-19T17:51:12Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.11-gke.1100", GitCommit:"20da4c21b3a6b1a56ff6ad5ecb7dee013aaf1b83", GitTreeState:"clean", BuildDate:"2022-04-01T09:40:07Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}
What did you do?
I installed the latest v0.24.0 version using the provided helm chart (also v0.24.0).
The chart created a CronJob on the cluster.
The pods created by that CrobJob are indeed using v0.24.0 but they still log everything using stderr instead of stdout.
If I add the argument --logtostderr=false  it works but that's throws a deprecation warning and also misses the purpose of the fix.
Moreover, downloading the image locally and running it without flags, prints the following:
      --logtostderr                              log to standard error instead of files (default true) (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)

which seems like it still defaults to true
What did you expect to see?
After upgrading to v0.23.1 and v0.24.0, I expect to see logs being sent to stdout by default as #680 was fixed and was part of the v0.23.0 release notes.
What did you see instead?
Logs are still sent to stderr and considered as Errors in Datadog.

Mentioned in #815, we should investigate why --logtostderr was deprecated and what it is replaced with. I imagine we just need to change how our flags are being instantiated. @JaneLiuL do you have a chance to look at it? Open to anyone

The suggestion in KEP regarding the flag deprecation of klog was to use something like go-runner
xref: KEP-2845: Deprecate klog specific flags in Kubernetes Compnents
/go-runner --redirect-stderr=true /bin/descheduler <args> could be done. That is how I am running it currently.

/remove-lifecycle rotten
Sorry this got stale, going to try to revive this as it might be related to https://github.com/kubernetes-sigs/descheduler/issues/965 (thanks @harshanarayana for pointing this out)


Furnaces need separate error messages to report being out of ingredients and being out of fuel

If I recall correctly, one of the error messages is being returned for both cases at the moment, which causes confusion.

Fix ready for 0.4.0


Document follower functions

How to follow another render node to keep the distributed simulation in sync

Documented enough for now...


Enhencement The trino-connector catalog supports pushdown predicates to the connector

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall

run p0 cloud_p1

run buildall

run buildall

run buildall

run buildall

run performance


Inquiry about Using  Additional Tools for Chromosome Reordering after HapHiC  analysis

Hi Xiaofei,
Thanks so much for the great tools! The flexibility and efficiency of HapHiC have been incredibly impressive and have significantly streamlined my workflow.
I have a question regarding the refsort function. As per my understanding, HapHiC has introduced a separate command, haphic refsort, in version 1.0.4 to order and orient whole scaffolds according to a reference genome.
Because this function is NOT reference-based scaffolding and will NOT alter your scaffolds; it only changes the way of presentation through overall ordering and orientation of the entire scaffolds. Given this, if I want to reorder the scaffolds according to chromosomes and perform JCVI alignment, which software can be used after HapHiC? Would "RagTag -scaffold" be a suitable option using 04.build/scaffolds.fa?
I would greatly appreciate your guidance on this matter. Thank you for your time and assistance!

By default, HapHiC outputs scaffolds (typically chromosome-level or nearly chromosome-level) with an ordering sorted by scaffold length and a random orientation. If you only want to order and orient your entire scaffolds based on a reference genome, please use haphic refsort. The ordered and oriented scaffolds are still fully supported by Hi-C data. RagTag (ragtag scaffold) can also generate similar results, but it may join multiple scaffolds into a single one according to its principle, resulting in the inclusion of evidence from the reference genome in the scaffolds, not just Hi-C data.

Thanks so much for your help. Could you please advise on how I can determine the chromosomal assignment for each contig in my dataset? (Which contig belong to which chromosome) My command line is :" minimap2 -x asm20 T2T_CHM13_V2/hs1.2bit.fa ./COLO829.asm.hifiasm.fa --secondary=no -t 28 -o asm_to_ref.paf
$HapHiC/haphic refsort 04.build/scaffolds.raw.agp asm_to_ref.paf > scaffolds.refsort.agp
$HapHiC/haphic refsort 04.build/scaffolds.raw.agp asm_to_ref.paf --ref_order chr1,chr2,chr3,chr4,chr5,chr6,chr7,chr8,chr9,chr10,chr11,chr12,chr13,chr14,chr15,chr16,chr17,chr18,chr19,chr20,chr21,chr22,chrX,chrY > scaffolds.refsort.chr.agp "
I have attached a sample of my output for your reference.
scaffolds.refsort.zip

Could you please advise on how I can determine the chromosomal assignment for each contig in my dataset?

This information should have been included in the logs, something like:
chr1, group3 +/-

Truly appreciate your guidance. I'm wondering if I replace "group3" with "chr1" in the "scaffolds.refsort.agp" file would be sufficient. Would that be correct?

It depends. Current scaffolding tools cannot guarantee perfect results, so manual curation in Juicebox is often necessary. The output of HapHiC (scaffolds.raw.agp) may contain some chromosomal misassignments. It would be better if you manually adjust the results in Juicebox before running haphic refsort. Another issue is when the assembly contains multiple haplotypes; one chromosome in the reference genome can have more than one corresponding scaffold, so these scaffolds should be renamed to something like "chr1_1," "chr1_2," etc.

Thank you so much for your insightful advice. I will try to use manual curation to adjust the results. Thanks!

I would like to ask, if we give a haplotype resolved assembly for instance
cat hap1.fa hap2.fa > asm.fa
and then we get the final genome out_JBAT.FINAL.fa after post review in JBAT of the asm.fa using the scaffold.refsort.agp.
The contigs are now named
Scaffold_1
Scaffold_2
Scaffold_3
First question is, can you change to name from 01 02 03 04 instead of 1 2 3 4 because in bash it will order from 1 10 11 12 .. 2 20 21 22 ..  and its a bit messy in the command line ?
The second question is how to get the phasing information again, for instance for very small contigs that showed haplotype switches patterns (the hic pattern was strong with the opposite chromosome, then we get this in the final agp:
BWA_MEM_5SP_HiC/04.build> cat out_JBAT.FINAL.agp | cut -f1,2,6 scaffold_1      1       HiC_scaffold_68_hap1_1_212635 scaffold_1      212636  100 scaffold_1      212736  HiC_scaffold_21_hap2_9085232_24725964 scaffold_1      15853469        100 scaffold_1      15853569        HiC_scaffold_21_hap2_3501833_9085231 scaffold_1      21436968        100 scaffold_1      21437068        HiC_scaffold_21_hap2_1848886_3501832 scaffold_1      23090015        100 scaffold_1      23090115        HiC_scaffold_258_hap1_1_86110 scaffold_1      23176225        100 scaffold_1      23176325        HiC_scaffold_21_hap2_724403_1848885 scaffold_1      24300808        100 scaffold_1      24300908        HiC_scaffold_221_hap1.1_1_51883 scaffold_1      24352791        100 scaffold_1      24352891        HiC_scaffold_21_hap2_1_724402 scaffold_2      1       HiC_scaffold_117_hap1_1_148072 scaffold_2      148073  100 scaffold_2      148173  HiC_scaffold_23_hap1_6797813_24363869 scaffold_2      17714230        100 scaffold_2      17714330        HiC_scaffold_23_hap1_1_6797812 scaffold_2      24512142        100 scaffold_2      24512242        HiC_scaffold_218_hap1_1_97978 scaffold_2      24610220        100 scaffold_2      24610320        HiC_scaffold_489_hap2_1_50833 scaffold_3      1       HiC_scaffold_1_hap2_582962_43028588 scaffold_3      9387657 100 scaffold_3      9387757 HiC_scaffold_126_hap1_1_145795 scaffold_3      9533552 100 scaffold_3      9533652 HiC_scaffold_108_hap1_1_158383 scaffold_3      9692035 100
is it possible to give 2 fasta files to HapHic instead of 1 ? so we can recover the belongings of namings at the end ?

The first question:
I currently do not have a plan to change the scaffold ID because:

Scaffolds are similar to chromosomes; the chromosome IDs of human genomes are named chr1~22, chrX, and chrY.
If I were to change them to chr01, chr02, when the number of scaffolds exceeds one hundred, I would need to change them to chr001, chr002 to keep them correctly ordered in the command line, which feels a little strange.
Some concerns about backward compatibility.

The second question:
You can find the contig IDs in the final AGP file. If you did not rename them, they can reflect the haplotype to which they were assigned by hifiasm. The phasing results of hifiasm may not always be correct. By default, HapHiC may cluster contigs from different haplotypes into a single group (scaffold). Of course, HapHiC is not always correct either. If you just want to ensure that contigs from different haplotypes are never clustered together, you can use the parameter --gfa *.hap1.p_ctg.gfa,*.hap2.p_ctg.gfa (for more information, please refer to: https://github.com/zengxiaofei/HapHiC#hifiasm)

Hello thank you for the quick answers it is informative and helpful.
I have another question, so if the HiC pattern intra contig ( inside of a
green square) show like a strong interraction with the opposite haplotype
it means that this little part is a haplotype switch error right ? When at
MAPQ1.
So it makes sense to cut the contig with the scissors and to move it to the
desired location in the opposite haplotype contig in Juicebox. So nothing
special here. But then when we save the scaffolds it will show the changes
in the AGP file, as you explained.
So my question is, I can simply rename the scaffolds by myself as you said,
right? Because my starting contigs are from hifiasm but I already polished
them and went in JBAT and so on before thats why the names of the contigs
are different than the hifiasm GFAs so I cannot use the GFA files. Because
I believe my assembly now is much better than hifiasm.
I went from 8,000 contigs in hifiasm, QV 31 to QV 51 and 500 gaps.. more or
less. But there was still some regional errors thats why I run HapHiC for
the last and fourth round of JBAT instead of 3DDNA. But I also noticed that
at MAPQ1  HapHiC basically removed my telomeres ( it was kind of expected,
maybe if I choose no contig correction and MAPQ0 it would preserve it? )
Or maybe I should process each Haplotype separately in HapHiC?
It's like while HapHiC improved the chromosome arms it also damaged the
assembly. So I need to find a way to fill gaps to  bridge assembly gaps in
HapHiC jbat Final fasta and use the long contigs to scaffold my First
assembly or idk..
All of this is complicated.. I don't understand why in JBAT the chromosomes
looked linear and great and after HapHiC it looked like it needed a lot of
manual curation? So you think that the contig correction could have damaged
the assembly ? Or that the JBAT from the round 3 that looked correct on
both separate haplotypes was actually  wrong because it didn't use the -5SP?
On Sun, Oct 20, 2024, 1:21 PM Xiaofei Zeng @.***> wrote:

The first question:
I currently do not have a plan to change the scaffold ID because:

Scaffolds are similar to chromosomes; the chromosome IDs of human
genomes are named chr1~22, chrX, and chrY.
If I were to change them to chr01, chr02, when the number of
scaffolds exceeds one hundred, I would need to change them to chr001,
chr002 to keep them correctly ordered in the command line, which feels a
little strange.
Some concerns about backward compatibility.

The second question:
You can find the contig IDs in the final AGP file. If you did not rename
them, they can reflect the haplotype to which they were assigned by
hifiasm. The phasing results of hifiasm may not always be correct. By
default, HapHiC may cluster contigs from different haplotypes into a single
group (scaffold). Of course, HapHiC is not always correct either. If you
just want to ensure that contigs from different haplotypes are never
clustered together, you can use the parameter --gfa
.hap1.p_ctg.gfa,.hap2.p_ctg.gfa (for more information, please refer to:
https://github.com/zengxiaofei/HapHiC#hifiasm)
—
Reply to this email directly, view it on GitHub
https://github.com/zengxiaofei/HapHiC/issues/80#issuecomment-2424643847,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ASYS5TEESKDBJEMOJ4GKM7DZ4NDXXAVCNFSM6AAAAABP2TVOFOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRUGY2DGOBUG4
.
You are receiving this because you commented.Message ID:
@.***>

So, you used the results of 3D-DNA for HapHiC scaffolding, right? I'm not sure how you did this. For example, did you use the output scaffolds of 3D-DNA as the input asm.fa for HapHiC? If the 3D-DNA scaffolds are already at the chromosome level and look good in Juicebox, there is no need to use HapHiC. Just adjusting them in Juicebox is okay. If the 3D-DNA scaffolds are very short, HapHiC may be helpful (equivalent to considering these scaffolds as longer contigs). However, if the 3D-DNA has many scaffolding errors, HapHiC may not be able to correct all these errors using the assembly correction function. In this case, it would be better if you use HapHiC to perform scaffolding from the original contigs. The contig IDs in the sixth column of the *.Final.agp should be consistent with the sequence IDs of input asm.fa. I don't know how you renamed these sequence IDs, so I don't know how to map them to the original hifiasm contig IDs.

But I also noticed that at MAPQ1 HapHiC basically removed my telomeres ( it was kind of expected, maybe if I choose no contig correction and MAPQ0 it would preserve it? )

If these telomeres are not anchored to any groups (scaffolds), it could be due to MAPQ>=1 filtering. The contigs containing these sequences may be very similar to some other contigs, resulting in multiple alignments during Hi-C read mapping, and these alignments are designated a MAPQ of 0. Actually, the 3D-DNA pipeline also uses the MAPQ>=1 criterion, however, the Hi-C read mapping pipeline (Juicer pipeline) is somewhat different from the mapping and filtering approach we recommended. We find that the Juicer pipeline will retain more incorrectly mapped reads. So these contigs may have Hi-C signals in the .hic files generated by 3D-DNA but cannot be seen in the .hic file generated by HapHiC. You can try different filtering criteria in the visualization steps, with an awareness of possible mapping errors. In addition, the correct placements of these telomeres may also be retrieved based on the assembly graph in the GFA file if contig IDs are consistent.

Or maybe I should process each Haplotype separately in HapHiC?

If Hi-C signals are still strong enough after MAPQ filtering, we would recommend scaffolding two haplotypes simultaneously. Scaffolding each haplotype separately will ignore possible phasing errors in the upstream analysis or the inherent SVs between haplotypes. However, if this is impossible, we may need to compromise.
If these answers do not fully resolve your confusion, I may need more information, such as how you specifically integrate 3D-DNA and HapHiC, and how HapHiC makes the results worse from the contact map. This way, I might be able to provide more detailed advice.

Because I thought that HapHic was more precise and that it would be better
to solve the small gaps at the regional level not structural. But it seems
that yes after manual curation the automatic scaffolders wether it is
HapHiC or 3DDNA are not going to improve it.
So id I just want to have the beautiful plots, I can run take the output
from 3DDNA/ JBAT and simply run HapHiC in quickview mode right?
On Sun, Oct 20, 2024, 3:06 PM Xiaofei Zeng @.***> wrote:

So, you used the results of 3D-DNA for HapHiC scaffolding, right? I'm not
sure how you did this. For example, did you use the output scaffolds of
3D-DNA as the input asm.fa for HapHiC? If the 3D-DNA scaffolds are already
at the chromosome level and look good in Juicebox, there is no need to use
HapHiC. Just adjusting them in Juicebox is okay. If the 3D-DNA scaffolds
are very short, HapHiC may be helpful (equivalent to considering these
scaffolds as longer contigs). However, if the 3D-DNA has many scaffolding
errors, HapHiC may not be able to correct all these errors using the
assembly correction function. In this case, it would be better if you use
HapHiC to perform scaffolding from the original contigs. The contig IDs in
the sixth column of the *.Final.agp should be consistent with the sequence
IDs of input asm.fa. I don't know how you renamed these sequence IDs, so I
don't know how to map them to the original hifiasm contig IDs.
But I also noticed that at MAPQ1 HapHiC basically removed my telomeres (
it was kind of expected, maybe if I choose no contig correction and MAPQ0
it would preserve it? )
If these telomeres are not anchored to any groups (scaffolds), it could be
due to MAPQ>=1 filtering. The contigs containing these sequences may be
very similar to some other contigs, resulting in multiple alignments during
Hi-C read mapping, and these alignments are designated a MAPQ of 0.
Actually, the 3D-DNA pipeline also uses the MAPQ>=1 criterion, however, the
Hi-C read mapping pipeline (Juicer pipeline) is somewhat different from the
mapping and filtering approach we recommended. We find that the Juicer
pipeline will retain more incorrectly mapped reads. So these contigs may
have Hi-C signals in the .hic files generated by 3D-DNA but cannot be seen
in the .hic file generated by HapHiC. You can try different filtering
criteria in the visualization steps, with an awareness of possible mapping
errors. In addition, the correct placements of these telomeres may also be
retrieved based on the assembly graph in the GFA file if contig IDs are
consistent.
Or maybe I should process each Haplotype separately in HapHiC?
If Hi-C signals are still strong enough after MAPQ filtering, we would
recommend scaffolding two haplotypes simultaneously. Scaffolding each
haplotype separately will ignore possible phasing errors in the upstream
analysis or the inherent SVs between haplotypes. However, if this is
impossible, we may need to compromise.
If these answers do not fully resolve your confusion, I may need more
information, such as how you specifically integrate 3D-DNA and HapHiC, and
how HapHiC makes the results worse from the contact map. This way, I might
be able to provide more detailed advice.
—
Reply to this email directly, view it on GitHub
https://github.com/zengxiaofei/HapHiC/issues/80#issuecomment-2424717547,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ASYS5TCFV56TZWBQGJF4OCLZ4NQCFAVCNFSM6AAAAABP2TVOFOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRUG4YTONJUG4
.
You are receiving this because you commented.Message ID:
@.***>

Because at first when I did 3DDNA and the manual curation the QV was 31.
But now it's 51 so I thought that using HapHiC with the 5SP filtering
method and the increase of QV from the polishing would give me more
resolution or power to rescaffold the small contigs. But it seems more
complicated that that. Now I am trying to run it on the separate
haplotypes. It may give better results? Let's see.
On Sun, Oct 20, 2024, 3:06 PM Xiaofei Zeng @.***> wrote:

So, you used the results of 3D-DNA for HapHiC scaffolding, right? I'm not
sure how you did this. For example, did you use the output scaffolds of
3D-DNA as the input asm.fa for HapHiC? If the 3D-DNA scaffolds are already
at the chromosome level and look good in Juicebox, there is no need to use
HapHiC. Just adjusting them in Juicebox is okay. If the 3D-DNA scaffolds
are very short, HapHiC may be helpful (equivalent to considering these
scaffolds as longer contigs). However, if the 3D-DNA has many scaffolding
errors, HapHiC may not be able to correct all these errors using the
assembly correction function. In this case, it would be better if you use
HapHiC to perform scaffolding from the original contigs. The contig IDs in
the sixth column of the *.Final.agp should be consistent with the sequence
IDs of input asm.fa. I don't know how you renamed these sequence IDs, so I
don't know how to map them to the original hifiasm contig IDs.
But I also noticed that at MAPQ1 HapHiC basically removed my telomeres (
it was kind of expected, maybe if I choose no contig correction and MAPQ0
it would preserve it? )
If these telomeres are not anchored to any groups (scaffolds), it could be
due to MAPQ>=1 filtering. The contigs containing these sequences may be
very similar to some other contigs, resulting in multiple alignments during
Hi-C read mapping, and these alignments are designated a MAPQ of 0.
Actually, the 3D-DNA pipeline also uses the MAPQ>=1 criterion, however, the
Hi-C read mapping pipeline (Juicer pipeline) is somewhat different from the
mapping and filtering approach we recommended. We find that the Juicer
pipeline will retain more incorrectly mapped reads. So these contigs may
have Hi-C signals in the .hic files generated by 3D-DNA but cannot be seen
in the .hic file generated by HapHiC. You can try different filtering
criteria in the visualization steps, with an awareness of possible mapping
errors. In addition, the correct placements of these telomeres may also be
retrieved based on the assembly graph in the GFA file if contig IDs are
consistent.
Or maybe I should process each Haplotype separately in HapHiC?
If Hi-C signals are still strong enough after MAPQ filtering, we would
recommend scaffolding two haplotypes simultaneously. Scaffolding each
haplotype separately will ignore possible phasing errors in the upstream
analysis or the inherent SVs between haplotypes. However, if this is
impossible, we may need to compromise.
If these answers do not fully resolve your confusion, I may need more
information, such as how you specifically integrate 3D-DNA and HapHiC, and
how HapHiC makes the results worse from the contact map. This way, I might
be able to provide more detailed advice.
—
Reply to this email directly, view it on GitHub
https://github.com/zengxiaofei/HapHiC/issues/80#issuecomment-2424717547,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ASYS5TCFV56TZWBQGJF4OCLZ4NQCFAVCNFSM6AAAAABP2TVOFOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRUG4YTONJUG4
.
You are receiving this because you commented.Message ID:
@.***>

contact_map.pdf
After sorting to Clarias fuscus ( a closely related species to my catfish ) ^
And the scaffolds looked like this: after manual review:
A:

B:

C:

D:

E:

F:

G:

H:

I:

J:

K:

L:

M:

N:

O:

I don't understand why you say that the Juicer pipeline is different from
the 3DDNA pipeline?
The juicer pipeline just aligns reads using simple BWA but we can use the
--assemby tag to output it in a format adequate for JBAT and then we can
use the 3DDNA run assembly visualizer to filter for MAPQ1 and redo the same
for MAPQ0 and run a third time at MAPQ30 ten in JBAT we overlay the hic
maps MAPQ1 and MAPQ0 ( as control )
The MAPQ 0 will show no signal on the regions without reads mapping but
will show signals on the repeats. The MAPQ1 hic map will show the signals
only on the non repetitive regions. It would be great if HapHiC could also
output the HiC maps at MAPQ0 by default in addition to MAPQ1. Maybe post
assembly it could filter again the reads ? Or I'm not sure how to do that.
But yes the 3DDNA is a scaffolding method but in my case I used Greenhill
and then JBAT but the run assembly visualizer of 3DDNA after Juicer for
read mapping. Sorry for the oversight.
I will try HapHiC for the hybrid catfish genome then. It seems that this
genome that I work with now didn't improve much after HapHiC but I need to
verify more in detail. Because the quality values and everything that is
not structural is the same. So maybe I will just use HapHiC to make the HiC
map figures because your custom script to plot juicebox is beautiful.
However for the Hybrid catfish I am not sure wether the genome is
tetraploid or wether it is a diploid that is the sum of the two parental
species ( 27+28 chromosomes per haplotype). There is something I need to
figure out. Maybe I can try both setting as you recommended two weeks ago
with allelic links = 2 and starting from hifiasm.
There seems to be a challenge because even though we can scaffold the
chromosomes and the HiC maps look correct, it doesn't mean that the
assembly is correct in reality especially in very particular genomes.
On Sun, Oct 20, 2024, 3:06 PM Xiaofei Zeng @.***> wrote:

So, you used the results of 3D-DNA for HapHiC scaffolding, right? I'm not
sure how you did this. For example, did you use the output scaffolds of
3D-DNA as the input asm.fa for HapHiC? If the 3D-DNA scaffolds are already
at the chromosome level and look good in Juicebox, there is no need to use
HapHiC. Just adjusting them in Juicebox is okay. If the 3D-DNA scaffolds
are very short, HapHiC may be helpful (equivalent to considering these
scaffolds as longer contigs). However, if the 3D-DNA has many scaffolding
errors, HapHiC may not be able to correct all these errors using the
assembly correction function. In this case, it would be better if you use
HapHiC to perform scaffolding from the original contigs. The contig IDs in
the sixth column of the *.Final.agp should be consistent with the sequence
IDs of input asm.fa. I don't know how you renamed these sequence IDs, so I
don't know how to map them to the original hifiasm contig IDs.
But I also noticed that at MAPQ1 HapHiC basically removed my telomeres (
it was kind of expected, maybe if I choose no contig correction and MAPQ0
it would preserve it? )
If these telomeres are not anchored to any groups (scaffolds), it could be
due to MAPQ>=1 filtering. The contigs containing these sequences may be
very similar to some other contigs, resulting in multiple alignments during
Hi-C read mapping, and these alignments are designated a MAPQ of 0.
Actually, the 3D-DNA pipeline also uses the MAPQ>=1 criterion, however, the
Hi-C read mapping pipeline (Juicer pipeline) is somewhat different from the
mapping and filtering approach we recommended. We find that the Juicer
pipeline will retain more incorrectly mapped reads. So these contigs may
have Hi-C signals in the .hic files generated by 3D-DNA but cannot be seen
in the .hic file generated by HapHiC. You can try different filtering
criteria in the visualization steps, with an awareness of possible mapping
errors. In addition, the correct placements of these telomeres may also be
retrieved based on the assembly graph in the GFA file if contig IDs are
consistent.
Or maybe I should process each Haplotype separately in HapHiC?
If Hi-C signals are still strong enough after MAPQ filtering, we would
recommend scaffolding two haplotypes simultaneously. Scaffolding each
haplotype separately will ignore possible phasing errors in the upstream
analysis or the inherent SVs between haplotypes. However, if this is
impossible, we may need to compromise.
If these answers do not fully resolve your confusion, I may need more
information, such as how you specifically integrate 3D-DNA and HapHiC, and
how HapHiC makes the results worse from the contact map. This way, I might
be able to provide more detailed advice.
—
Reply to this email directly, view it on GitHub
https://github.com/zengxiaofei/HapHiC/issues/80#issuecomment-2424717547,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ASYS5TCFV56TZWBQGJF4OCLZ4NQCFAVCNFSM6AAAAABP2TVOFOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRUG4YTONJUG4
.
You are receiving this because you commented.Message ID:
@.***>

There are indeed some switch errors that can be observed on the contact maps you uploaded. It seems that the green rectangles on the maps are 3D-DNA scaffolds. These errors could be introduced during the 3D-DNA scaffolding process. HapHiC is usually unable to correct them as they are too short.
My suggestion is as follows:

Use HapHiC to scaffold raw hifiasm contigs without polishing. I think the polishing step will not have much effect on your contact map, but it will cause contig lengths to change, resulting in an exception when running HapHiC. You can polish your assembly after scaffolding. (Although the authors of nextpolish2 claim that this tool can alleviate switch errors in the human genome HG002, I have not tested it in other species yet and have not used it in constructing any haplotype-resolved assembly, sorry!)
I notice that many regions have nearly no Hi-C signal. You can scaffold each haplotype separately (you will get two AGP files), and then validate your results in Juicebox (align Hi-C reads to the two haplotypes simultaneously, rename the first column of the two AGP files to avoid duplication, and concatenate them together).

I don't understand why you say that the Juicer pipeline is different from
the 3DDNA pipeline?

Sorry, there could be some misunderstanding. I meant that the contact maps generated using the Juicer pipeline for Hi-C read mapping and filtering are different from the method we recommend here. The Juicer pipeline typically retains much stronger diagonally distributed Hi-C links between homologous chromosomes. We suspect these might be mapping errors, so we generally do not recommend using the Juicer pipeline for scaffolding a haplotype-resolved assembly.

I am not sure but in my case I processed each haplotype separately in juicer even the BWA mapping so there was two BAM files. It may not be optimal but in species with low heterozygosity it may still work somehow?
But yes HapHic is better but it was not really famous one year ago🤧.

There are indeed some switch errors that can be observed on the contact maps you uploaded. It seems that the green rectangles on the maps are 3D-DNA scaffolds. These errors could be introduced during the 3D-DNA scaffolding process. HapHiC is usually unable to correct them as they are too short.
My suggestion is as follows:

Use HapHiC to scaffold raw hifiasm contigs without polishing. I think the polishing step will not have much effect on your contact map, but it will cause contig lengths to change, resulting in an exception when running HapHiC. You can polish your assembly after scaffolding. (Although the authors of nextpolish2 claim that this tool can alleviate switch errors in the human genome HG002, I have not tested it in other species yet and have not used it in constructing any haplotype-resolved assembly, sorry!)
I notice that many regions have nearly no Hi-C signal. You can scaffold each haplotype separately (you will get two AGP files), and then validate your results in Juicebox (align Hi-C reads to the two haplotypes simultaneously, rename the first column of the two AGP files to avoid duplication, and concatenate them together).

In Hifiasm, does the --duak-scaff option impacts the results of HapHiC?

I am not sure but in my case I processed each haplotype separately in juicer even the BWA mapping so there was two BAM files. It may not be optimal but in species with low heterozygosity it may still work somehow?

Yes. Animals usually have fewer SVs between haplotypes. Many haplotype-resolved assemblies of diploids were constructed using this approach and acceptable.

In Hifiasm, does the --dual-scaff option impacts the results of HapHiC?

Sorry, I have not used this option before because it also hypothesizes that there are no SVs between haplotypes.

Ok thank you. And maybe I would kindly like to ask you the last question
for today,
Do you think that it makes sense to use Redundans or something like Purge
dups  and give the two haplotype resolves assemblies after HapHiC for
instance, to generate a primary assembly at the chromosome level?
Because for instance in my case. Each haplotype has 92% buscos and 89%
completeness
But when together the results are 95% buscos and 96% completeness.
Because on NCBI most people use Primary as reference genome but in my
opinion it would make a chimeric assembly which is wrong in reality but at
the same time if people want to do comparative genomics and use one
haplotype they may miss the 3% of genome that is haplotype specific but if
they take the chimeric primary assembly reduced for duplications after
Redundans they will get most of the genes?
What do you usually upload  on NCBI or do after HiC scaffolding a haplotype
resolved assembly with HapHic?
Thank you so much for your suggestions it really helps and gave me many
insights and a better global understanding.
Have a great day Xiaofei.
On Sun, Oct 20, 2024, 5:22 PM Xiaofei Zeng @.***> wrote:

I am not sure but in my case I processed each haplotype separately in
juicer even the BWA mapping so there was two BAM files. It may not be
optimal but in species with low heterozygosity it may still work somehow?
Yes. Animals usually have fewer SVs between haplotypes. Many
haplotype-resolved assemblies of diploids were constructed using this
approach and acceptable.
In Hifiasm, does the --dual-scaff option impacts the results of HapHiC?
Sorry, I have not used this option before because it also hypothesizes
that there are no SVs between haplotypes.
—
Reply to this email directly, view it on GitHub
https://github.com/zengxiaofei/HapHiC/issues/80#issuecomment-2424824503,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ASYS5TF2DHCM4A5VBQN2P4DZ4N77RAVCNFSM6AAAAABP2TVOFOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRUHAZDINJQGM
.
You are receiving this because you commented.Message ID:
@.***>

Do you think that it makes sense to use Redundans or something like Purge
dups  and give the two haplotype resolves assemblies after HapHiC for
instance, to generate a primary assembly at the chromosome level?

Many studies have included both a primary assembly and a haplotype-resolved assembly. This is particularly beneficial for researchers in population genetics and gene expression regulation who may not require the haplotype information in the reference genome. Therefore, it is recommended to provide both assembly versions and allow users to choose which one to use.

Have a great day Xiaofei.

Thank you, you too!


Could not search on remote source after changed from GH-771

This happens because of the wrong parameters now on LoadPackages. Better changing this to SearchForPackages and CanSearchForPackages to prevent such a mistake in the future.

This was a bug that was introduced with a feature that is being added to the 0.18.0 release, as such, this bug doesn't need to be assigned to the 0.18.0 milestone.
This is related to this issue: https://github.com/chocolatey/ChocolateyGUI/issues/769 and this PR: https://github.com/chocolatey/ChocolateyGUI/pull/771


Bug lc 190 update wcag

Replaced by https://github.com/Civil-Service-Human-Resources/identity-service/pull/117


Input data sets for IM-fusion

I have three questions for using IM-fusion:

Your article identify TE insertions from RNA-Seq based on Sleeping Beauty (SB) transposon model. But, I have several RNA-Seq data sets from Normal vs Tumor paired samples. Are my data sets suitable for insertion detection by using IM-fusion?
I wonder any other transposon elements are suitable for IM-fusion?
And how can I prepare for these files?

Best,

Hi @xflicsu,
If your files contain transposon insertions from an insertional mutagenesis screen, you should be able to analyse them with IM-Fusion. Note that you need to supply two files for IM-Fusion to be able to recognise your insertion:

A sequence file containing the transposon sequence.
A feature file (CSV) describing the positions of splice-donor and splice-acceptor sequences in your transposon sequence.

Example files for the PB and SB transposons are given here:
https://github.com/NKI-CCB/imfusion/tree/develop/data
Best,
Julian


Register J2SEPlatform using reflection

Related to #21. Seems to me an impl dependency is a bit brittle and for such a simple API call we could just take the classloader and use reflection.
For newer NetBeans versions, that make that API public (assuming the PR goes through) we could use normal API calls.

Great plan. If we can avoid impl dependency and friends, etc, that's great.


How to View your Outside Guest Access in Desktop App when working from Different Organization

I was invited to a different organizations teams page as a guest. In the web portal i can easily change between my organization and the outside organization. However, I can't figure out how to view the outside guest access from my desktop app. Is this possible?

@aheff01 Thank you for submitting feedback.
I think the best way forward is if you open a service ticket in your tenant so this can get resolved ASAP. Based on the outcome let me know if it is something that can be called out in the docs.
Please follow this link to contact support for business products:https://docs.microsoft.com/office365/admin/contact-support-for-business-products
Please keep us posted here on the resolution so we can feed whatever information you discover into the content.


readme: note re sbt plugins on sonatype central

publishing sbt plugins to sonatype central requires a workaround
See also: https://github.com/sbt/sbt/issues/3410

cc @xerial

Thanks. Merged


Issues loading forever in folder with git not initialized

Testing https://github.com/microsoft/vscode-pull-request-github/issues/2581 with a folder open that does not have git initialized:

The pull requests view correctly displays a "No git repositories found" message, but the issues view still displays "Loading..."

Looks like both viewlets are stuck on "Loading..." now.
Also, though the menu identifier warning shows up, I noticed it's as-designed so that's not an issue here.


Anachronistic protocol ledger view is the wrong concept

The anachronistic ledger view returns a LedgerView within a certain window from the current ledger tip
                 <---------|--------->
                           |
---------------------------X-----------------

If we use Byron as our running example, the LedgerView is a Delegation.Map, and the range is ±2k. This is fine as long this LedgerView is only on the same fork, but this is wrong when used across forks. Suppose we have
                 <---------|--------->
                           |
--------------------A------X-----------------
                    \
                     -------------Y---

where we have a point Y on a different fork than X, where Y still falls within the bounds that the anachronistic ledger views, but is outside that same 2k range from A. Then the Delegation.Map as X may well not apply anymore to Y; after all, there could have been some delegation certificates on Y's branch between A and Y that have come into effect.
The primary use case of the anachronistic ledger view is to use our current ledger to validate block headers received by the chain sync client, which may indeed well live on a different chain. This means that we may currently accept some headers as valid that aren't (although they will then subsequently be rejected by the chain DB), or -- potentially worse -- reject some headers as invalid that in fact are valid.
The right approach is to split this into two steps: in principle at least, we should first go back to the intersection point A, get the delegation state there (that is, the current mapping as well as any scheduled delegations), then use that to derive a "forecast" which we can use to derive conclusions about Y; moreover, the range of that forecast should be determined by A, not by X. We don't literally have to implement that (though we might want to, see #1935 ), but what we do implement should be equivalent to that.

Note that if the intersection point is 2k slots back (worst case scenario, both because it's the most distant possible fork and the lowest possible chain density), then the range of the forecast is only up to the current slot, which means we can't look ahead at all. This implies that we would never adopt such a fork, because we would not be able to validate enough of it to get a chain that is longer than our current one, and so we would not prefer it. This means the effective maximum rollback is k - 1. We consider this acceptable.

Note that if the intersection point is 2k slots back, then the range of the forecast is only up to the current slot, which means we can't look ahead at all. This implies that we would never adopt such a fork, because we would not be able to validate enough of it to get a chain that is longer than our current one, and so we would not prefer it.

This doesn't seem right to me. In particular, I think that use of "look ahead" is too focused on slots. Is this following counterexample correct?
Suppose k is 3 and our current chain X is D16, which is as sparse as the chain density invariant allows.
A10 > B12 > C14 > D16
Suppose we suddenly learn about a chain Y = U14.
A10 > R11 > S12 > T13 > U14
Because Y is more dense than X, it has more blocks within 2k slots of A10, even though its tip is from an earlier slot. So we'll be able to project the A10 ledger view forward enough to validate Y. And then we'll switch to it. Am I missing something, or does that contradict the blockquote? @edsko

Sure, there are cases when we can switch, but if the alternative chain is also at it's lowest density, we won't be able to.

Draft PR fixing this at #1942.

Sure, there are cases when we can switch, but if the alternative chain is also at it's lowest density, we won't be able to.

Ah, I see. So when the intersection is 2k slots and k blocks back from both tips. 👍
A10 > B12 > C14 > D16
A10 > R12 > S14 > T16


Stabilize the build server

Fixes AB#141

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: nemrism sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

merged in wrong repository, sorry guys


Sonoff Slampher issue

Please help bro, sonoff slampher support ?

Hi @febriyan1302, the logs looks fine, would you please elaborate the issue in more detail?

Hi @febriyan1302, the logs looks fine, would you please elaborate the issue in more detail?

Hi, thank you for your reply, if you see carefully it's not fine, Problem device name : [TypeError: Cannot read property '-1' of undefined]
once i pair with homekit, they don't recognize device name (sonoff slampher).

Hi @febriyan1302, that "error" won't do any harm. It was part of metarutaiga's pull request https://github.com/howanghk/homebridge-ewelink/pull/58 which add support for device names returned from ewelink api.
https://github.com/howanghk/homebridge-ewelink/commit/5b5697d3945c57111547aa026729b4d97e53b555#diff-168726dbe96b3ce427e7fedce31bb0bcR540-R546

Closing due to inactivity.


Reduce duplication when writing

We replicate writes to N (typically 3) ingesters to reduce data loss from random failures of machine or process. However this means in stable operation each metric is written to store three times, along with index entries for each label (at least in the DynamoDB back-end).  Say 10 index entries per chunk, for a total of 33 writes.  Nearly one third of these writes are pointless and, worse, they slow down queries because all those entries get re-fetched and finally deduplicated.
One alternative idea:
When an ingester is about to write a chunk, query the index to see if an overlapping chunk already exists. If so, subtract the overlapping part and merge the remainder back into the chunks being accumulated.  This would replace the 33 writes with 11 writes and 2 reads.
This idea doesn't require any particular consistency from the store - if it under-reports we'll just store some pointless chunks as before.

The details have gone hazy, but I believe there's a trade-off with how many ingesters we need to read from. The Dynamo paper section 4.5 has some discussion on this.

This proposal makes every write into a read + a write, and potentially a delete.  As Cortex tends to do many orders-of-magnitude more writes than reads, making queries cheaper by making writes more expensive seems like the wrong direction...
It might be worth considering TCO here; for systems like BigTable & Cassandra, writes are significantly cheaper than reads (in turns of IOPS and $$$), so doing read-on-write is a big no.   Its interesting that this isn't the case for DynamoDB - reads seem to be 1/5th the cost - but does at least suggest that this needs to be done on a per-backend basis.  Although the existing abstractions won't nicely support that.
Alternatives to consider:

The 3x index cost should probably be tackled by indirecting via a timeseries ID, and not having the entries point straight at chunks.  This is something we started with @aaron7 in #442, using the sha256 of the label-set of a series as its fingerprint.  This might also be necessary to do proper stream processing of queries anyway, but won't cover the 3x chunk cost.
On a query you could always get the range of chunks and then read at most 2 of the 3 chunks for every overlapping range, but this seems like it might be challenging (fun?) to compute.
We don't actually need to write to 3 ingesters in normal operation, we should only write to two.  This is discussed in #100, would get you 50% of the wins, and is probably relatively easy to implement.
We could potentially do the merge as form of "Read Repair", writing back deduped chunks after a query.  This would work for backends like Cassandra and BigTable, as you've already paid the read cost.
Moving to something like TSDB for the ingesters, and guaranteeing the last N hours are ingester, would make >80% of queries not touch the chunk store.  I think users are willing to accept slower queries for older data.

As a less rational fear, getting the write path to perform reliably was really challenging; I'd be pretty apprehensive to adding a read into that pipeline.

Thanks, I appreciate the insights.
Bit more background: the main use-case I'm targeting is when you see some data in the last hour and think "what does that look like over the last month?". Depending on the metric, the index reads for 30 days can be very expensive.
I would not change the write path lightly.

Interesting - I wouldn't have thought hour-long queries in the past would have been any more expensive than hour-long now queries, modulo caching effects.  Are the historical tables adequately provisioned in DynamoDB?  Do you know if its index lookups or chunk fetches that are dominating?
If its index, I remember an optimisation I added to make index lookups from [now-1hr, now) cheaper - the index always looks up to the end of the "row".  This is indicated by the inclusion of only a greater-than comparison here.  Given we know the maximum size of chunks, it would be possible to add a less-than comparison to the same place (and plumbing through the logic for the schemas), and terminate the scan early.  Would be a minor improvement for queries that happen in the morning...
If its chunk fetches taking the majority of time (which I guess it is by your recent PRs),   IIRC is the shear number of chunks that need fetching, more than the size of the data, that causes problems.  Do you see a lot of DynamoDB throttling? You could consider:

proactively prefetching month-ago historic chunks on queries in the background
When I get round to doing the TSDB work the average chunk size will increase significantly (and should always hit max chunk age), which should massively reduce the number of chunks fetched.
Parallelising chunk fetches across multiple parallel workers, and have them stream deduped series back to the querier.

Alternatively, instead of putting this kind of deduping and processing on the write path, you could instead do it as a separate process which reads back data and rewrites it asynchronously, taking it off the critical path.  The original design doc included a mention of this IIRC.

Index queries can be several seconds each. The data for one bucket for one index all comes from one hash key, so I'm not sure more provisioning helps - just fetching 100,000 data items takes time. Even filtering them takes 1-2 seconds.
In many cases that number can be reduced with better query formation; I have added some thoughts to #209. This was another thought that came to me.
I am working on many fronts; the PRs you see are just the parts that are ready now.

To clarify, "what does that look like over the last month" is a 30-day query not an hour-long query.
But an hour-long query in the past is also slower, because we read to the end of the bucket.

Since we're on the subject, #209 says " don't range query to end of row, but use max chunk length to query to end of query + 12hrs"; what is that "+ 12hrs" for?

Index queries can be several seconds each. The data for one bucket for one index all comes from one hash key, so I'm not sure more provisioning helps - just fetching 100,000 data items takes time. Even filtering them takes 1-2 seconds.

Indeed; indirection through a "time series" index would massively improve that, as the labels index would only contain the cardinality of the labels, not cardinality x number of chunks.

In many cases that number can be reduced with better query formation; I have added some thoughts to #209. This was another thought that came to me.

Yes! Even better, you could track and pick low cardinality indexes and only read them; you can always under read the index and over read chunks, and filter later.  Thats if the index is slower than the chunks of course.

To clarify, "what does that look like over the last month" is a 30-day query not an hour-long query.

Ah I see.  How slow?
Another idea is to pipeline index and chunk fetches, so we can start fetching chunks earlier.  Thats a fair bit of work mind you.

Since we're on the subject, #209 says " don't range query to end of row, but use max chunk length to query to end of query + 12hrs"; what is that "+ 12hrs" for?

12hrs is max chunk age

#875 now writes out (label:value) -> []seriesID and seriesID -> []chunkID indexes, which reduces the size of the index by 6-12x over the (label:value) -> []chunkID indexes of old - 3x due to replication and 4x due to average chunk size of 6hrs vs 24hrs index window.
We don't dedupe chunks yet, but this should massively reduce size of index.

Given #875 is merged, can this be closed?

The original point hasn't changed - if the chunk you're about to write is already in the store then you would benefit from not re-writing it.
Querying the disk store wasn't a good idea, but querying memcache should work better.
You have very different usage patterns if you see an "average chunk size of 6hrs".  2 hours is more like what we see as an overall average, but if you are scraping a cluster where containers start and stop a lot then the average goes down.  Chunks with just a few minutes of data are ideal candidates for de-duping, because they will have identical start and end times on all replicas.

Just noticed this:

DynamoDB - reads seem to be 1/5th the cost

From current US East pricing:
Write capacity unit (WCU)  $0.00065 per WCU
Read capacity unit (RCU)  $0.00013 per RCU
But, "DynamoDB charges one WCU for each write per second (up to 1 KB) [...] and one-half of an RCU for each eventually consistent read per second (up to 4 KB)."
So there is another factor of 2-8 in there, depending on data size.
Reads are at least 1/10th the cost of writes.

Closed by #1578 and #1475


Overthink color choice for low risks

Regarding to the latest issues with the CWA a green color in the risk panel might be a fallacy.
– You're not safe when it is green.
Maybe a neutral white or gray would be a better choice for clearness.

people are very sensitive when observing changes.
After seeing the green every day for weeks now, it changing to gray could result in people questioning if the app is working (gray currently is used when not enough data is collected yet) or worst case: It coincides with their first low-risk risk-contact.
Some people already panicked when they see one low risk risk-contact on green 1.
Now imagine people seeing a similar text 2 on gray on their device, when in the media and official communication screenshots with a green background and different text are in circulation.
How will they respond?
I totally agree that the app can give a false sense of safety and that gray would have been the more objective choice in hindsight (now knowing the low usage/acceptance in the population) - but concerning the current user base by changing the color(s) is not a good idea in my opinion, as people rightfully would ask what the change means when they notice it one day.
Communicating the change in advance is also tricky.
Note the media tendency to misunderstand and misrepresent details.
This is mend as a subtle change in representation, but it can be perceived as a major change in meaning. (I mean: that is kind of the point of this suggestion: changing it from green to gray is most concerning for people that currently had a false sense of safety due to the app. Correcting this misunderstanding is important, but the not unlikely reaction could be that people uninstall the app, because they realize that the app does not provide the personal and direct benefit they assumed it as to them.)

I would like to reference my suggestion to think about maybe a second level of information that can only be accessed (together with) a health worker, so that messaging towards the end user can remain simple and non-alarmist.
https://github.com/corona-warn-app/cwa-wishlist/issues/100#issuecomment-666186069

I would like to reference my suggestion to think about maybe a second level of information that can only be accessed (together with) a health worker, so that messaging towards the end user can remain simple and non-alarmist.

Imagine the practical logistics of a health care worker having to explain that a contact that lasted between 0 and 5 minutes (yes, the increments are about 5 minutes) in a distance of 10 to 15 meters (free field equivalent distance - the actual distance could be less if a small wall was in between the phones, but not the faces) is less risky then going to a hospital or doctors office to get a 5 minute explanation about that topic. Repeat that in a few days, because maybe an other patient in an other room tests positive that day and again triggers a low risk warning.
Also: How do you want to restrict access to the data logistically? Handing over the phone to the doctor? shared item/Indirect handshake. Letting the patient scan a QR code to unlock additional information? And then what? Let the patient read out the information? What if the patient can not speak/read German?
And then what? The patient visited the doctor and at best still knows as little as before. Or at worst the patient thinks that not having symptoms, low risk in the app and a doctors visit implies being corona free. (Even a negative PCR test is no prove for not being infected)

Correcting this misunderstanding is important, but the not unlikely reaction could be that people uninstall the app, because they realize that the app does not provide the personal and direct benefit they assumed it as to them.)

Yesterday I just had this case - but exactly the opposite. Person got one exposure but the color stayed green. Person was very confused and thought about deinstalling the App because of the green color which might give the person a feeling of wrong security. No matter what you do it always will confuse some people.

Here my few cents:
I always thougt I had no risks, since the app always showed "green" to me - but then after some months, when reading the text on the green box again, I was totally shocked to see that I had contact to an infection case with (low) risk for me! Maybe I had that before many times and I never saw the details in the text...
Why the app didn´t show low risks in "yellow" color to me, isn´t it the purpose of the app to warn their users in case of risks, instead to hide/vanish such important information, by showing always green color (grean means to me = everything is great)?
So I would suggest a traffic light similar color code, having very good UI/UX:
green: no risk
yellow: low risk
red: higher risk
That everybody shall be able to understand without the need of reading all the details...
btw. I also don´t understand why "gray" is in duscussion here - "gray" means usually "disabled" - things are grayed out so they are not applicable, e.g. when not able to calculate or similar.
Thanks! :)

Here my few cents:
I always thougt I had no risks, since the app always showed "green" to me
You are totally misunderstanding what the app can do.

It can only warn you about contacts with other app users, that have tested positive and self reported them self. The app can not warn you when had a high risk contact with an untested asymptomatic carrier. The app can not warn you, when a person that has tested positive does not have the app. The app can not warn you, when the person that has tested positive has the app, but does not want or can not report him self positive.
So, the app can never declare "no risk" for you or anyone as there are situations, it simply can not cover.

but then after some months, when reading the text on the green box again, I was totally shocked to see that I had contact to an infection case with (low) risk for me! Maybe I had that before many times and I never saw the details in the text...
Why the app didn´t show low risks in "yellow" color to me, isn´t it the purpose of the app to warn their users in case of risks, instead to hide/vanish such important information, by showing always green color (grean means to me = everything is great)?

You want a yellow warning, when the contact maybe was less then 5 minutes and 15 meters away?
And then what?
You want to be alarmed about risk? Look at the rki dashboard Plenty of red and yellow where ever you look.

So I would suggest a traffic light similar color code, having very good UI/UX:
green: no risk

There is no "no risk" state as the app can not cover a lot of situations.
There is always a latent risk.

yellow: low risk

with the "call of action" to "stay alert" meaning to continue as before? How is that useful?

red: higher risk
That everybody shall be able to understand without the need of reading all the details...

No, you can display "low risk" in green and people panic: 1.
Make it yellow and people are forced to read all the details to understand what yellow means, followed by them calling up their doctors and local health authorities "just to be sure".
Make one state "no risk" and green and people falsely assume that have "no risk" of having covid.

btw. I also don´t understand why "gray" is in discussion here - "gray" means usually "disabled" - things are grayed out because they are not applicable, e.g. when not able to calculate or similar.

yeah, the app can not calculate your risk. Not at the current rate of adoption in the population.
All the app can do is warn you when you had a high risk contact with an other app user.
At the current rate of adoption, the "low risk" status can be misleading. Gray as in "no known high risk contact with a known infected person but there are lot of people that do not use the app or do not report them self positive" would be more accurate. Note: I am against changing it to gray as the chance would scare people, but objectively in hindsight (low rate of usage in the population), I agree that it would have had been the more objective choice.
Under no circumstances (even if using the app was mandatory and the usage rate 100%) the app could actually tell you that you have a no risk of infection.
There are plenty of situations with risk of infection that the app simply can not cover even under ideal circumstances.

There were a lot of discussions about this and there is a related official statement from RKI in
https://github.com/corona-warn-app/cwa-wishlist/issues/100#issuecomment-690279530.
PS Maybe one of the moderators could change the title of this issue from "Overthink" to "Rethink".
Overthink means "to spend more time thinking about something than is necessary or productive" 😄 and it is not the right translation for the German "Überdenken".


Devcontainer fails to build

The docker file for the dev container includes copying Pipfile.lock, but this is excluded from the gitignore with a comment.
I tried removing it from the copy command but building the container is still failing.
9.768 Traceback (most recent call last):
9.768   File "/usr/local/py-utils/bin/pipenv", line 8, in <module>
9.768     sys.exit(cli())
9.768   File "/usr/local/py-utils/venvs/pipenv/lib/python3.10/site-packages/pipe
nv/vendor/click/core.py", line 1130, in __call__
9.769     return self.main(*args, **kwargs)
9.769   File "/usr/local/py-utils/venvs/pipenv/lib/python3.10/site-packages/pipe
nv/cli/options.py", line 58, in main
9.769     return super().main(*args, **kwargs, windows_expand_args=False)
9.769   File "/usr/local/py-utils/venvs/pipenv/lib/python3.10/site-packages/pipe
nv/vendor/click/core.py", line 1055, in main
9.770     rv = self.invoke(ctx)
9.770   File "/usr/local/py-utils/venvs/pipenv/lib/python3.10/site-packages/pipe
nv/vendor/click/core.py", line 1657, in invoke
9.770     return _process_result(sub_ctx.command.invoke(sub_ctx))
9.770   File "/usr/local/py-utils/venvs/pipenv/lib/python3.10/site-packages/pipe
nv/vendor/click/core.py", line 1404, in invoke
9.771     return ctx.invoke(self.callback, **ctx.params)
9.771   File "/usr/local/py-utils/venvs/pipenv/lib/python3.10/site-packages/pipe
nv/vendor/click/core.py", line 760, in invoke
9.771     return __callback(*args, **kwargs)
9.771   File "/usr/local/py-utils/venvs/pipenv/lib/python3.10/site-packages/pipe
nv/vendor/click/decorators.py", line 84, in new_func
9.778     return ctx.invoke(f, obj, *args, **kwargs)
9.778   File "/usr/local/py-utils/venvs/pipenv/lib/python3.10/site-packages/pipe
nv/vendor/click/core.py", line 760, in invoke
9.778     return __callback(*args, **kwargs)
9.778   File "/usr/local/py-utils/venvs/pipenv/lib/python3.10/site-packages/pipe
nv/cli/command.py", line 783, in requirements
9.778     lockfile = state.project.load_lockfile(expand_env_vars=False)
9.778   File "/usr/local/py-utils/venvs/pipenv/lib/python3.10/site-packages/pipe
nv/project.py", line 1044, in load_lockfile
9.782     with io.open(self.lockfile_location, encoding="utf-8") as lock:
9.782 FileNotFoundError: [Errno 2] No such file or directory: '/workspaces/Pipfi
le.lock'
------
Dockerfile-with-features:14
--------------------
  13 |     # Create Python requirements files from pipenv (lockfile)
  14 | >>> RUN pip3 install -U pip \
  15 | >>>     && pip3 install pipenv \
  16 | >>>     && pipenv requirements > /tmp/requirements.txt \
  17 | >>>     && pipenv requirements --dev > /tmp/requirements_dev.txt
  18 |
--------------------
ERROR: failed to solve: process "/bin/bash -o pipefail -c pip3 install -U pip
  && pip3 install pipenv     && pipenv requirements > /tmp/requirements.txt
&& pipenv requirements --dev > /tmp/requirements_dev.txt" did not complete succe
ssfully: exit code: 1

This is fixed on dev now.