Interpretation of sequencing data obtained from NovoGene

We got back 150 bp read data from NovoGene from the SATAY experiments. Based on a manual inspection of the data, we see that:

Each read also contains the sequence of the primers used for amplification of the DNA and part of the MiniDS sequence. Different samples can therefore be separated based on the barcode that can be found in the read (this barcode is not mentioned in the fastqc file and probably has to be used as input for demultiplexing the samples)

Each read pair can be identified from the fastqc file and can therefore be processed as a pair (which means you can match the forward and reverse reads from the same strand of DNA)

After removing the primer+MiniDS sequence, about 40-70 bp remains for alignment to a reference genomic DNA sequence, which may not be a lot (the protocol mentions 75 bp used for alignment). It might be beneficial to use 250 bp sequencing next time if we think that might significantly improve the alignment results.

I tried two different tools for demultiplexing the data;

sabre
cutadapt

Both tools work, but both also give a lot of reads which are discarded. I am not sure yet whether this is because;

there is no barcode sequence in both the forward and reverse (partner) read (which I don't think),
there is a barcode in either the forward or reverse read, but not in its paired partner. Because it is paired-end both reads will then be discarded,
there is a barcode in both a forward and reverse (partner) read, but they don't match the barcode from the same sample (e.g. forward read has barcode from sample 1 but its reverse read has a barcode from sample 2),
I am not implementing the software correctly. I followed the readme's from both tools as closely as possible, but I might misunderstood something.

I need to look more closely at the data and the results to be sure what the cause it and how to solve it.
The results of both tools are stored on the Linux machine at "~/Documents/satay/datasets/wt1_enzo_dataset" (it is still on the Linux computer because the data transfer between the Linux machine and the drives is somewhat troublesome, so I can't store it on the N-drive).

Hi Greg,
I had a look at the files from the Sabre demultiplexing and I did not really understand why the barcodes file (wt1_enzo_barcodes.txt) has only 2 of the 4 barcodes

I followed the tutorial on the Sabre github page, and from what I understood you can only enter one barcode per sample. That is probably why it didn't get many of the pairs. Maybe I should run it a second time, now with the other (unused) barcodes on the file with all the unmatched reads.

I am writing a small python code that loops over the sequencing reads and checks what the barcode is in each sample. I notice that

some reads don't have a barcode sequence in them (also not if you allow sequencing errors of 1 or 2 bp or reverse the nucleotide sequence of the barcodes).
some forward reads have the barcode from sample 1, but a reverse read from sample 2 (or vice versa).

I guess this is why the other tools discard many of the reads. The question is what to do with those reads?
This is a snippet from the output of the code for first 10 reads where first the headers of the reads are shown from the two files (those should be almost identical) and then the two barcodes that are found (if any) in the sequences are shown:
@A00153:690:HJVN5DSXY:1:1101:1470:1000 1:N:0:GAACGTGA+AACTGGTG

@A00153:690:HJVN5DSXY:1:1101:1470:1000 2:N:0:GAACGTGA+AACTGGTG

GATAGACA Sample2_rev
GAGCTGAA Sample2_for

@A00153:690:HJVN5DSXY:1:1101:3568:1000 1:N:0:GAACGTGA+AACTGGTG

@A00153:690:HJVN5DSXY:1:1101:3568:1000 2:N:0:GAACGTGA+AACTGGTG

GCCACATA Sample1_for
GCGAGTAA Sample1_rev

@A00153:690:HJVN5DSXY:1:1101:4056:1000 1:N:0:GAACGTGA+AACTGGTG

@A00153:690:HJVN5DSXY:1:1101:4056:1000 2:N:0:GAACGTGA+AACTGGTG

GCCACATA Sample1_for
GCGAGTAA Sample1_rev

@A00153:690:HJVN5DSXY:1:1101:4273:1000 1:N:0:GAACGTGA+AACTGGTG

@A00153:690:HJVN5DSXY:1:1101:4273:1000 2:N:0:GAACGTGA+AACTGGTG

GCGAGTAA Sample1_rev
GCCACATA Sample1_for

@A00153:690:HJVN5DSXY:1:1101:5267:1000 1:N:0:GAACGTGA+AACTGGTG

@A00153:690:HJVN5DSXY:1:1101:5267:1000 2:N:0:GAACGTGA+AACTGGTG

GAGCTGAA Sample2_for
[]

@A00153:690:HJVN5DSXY:1:1101:6298:1000 1:N:0:GAACGTGA+AACTGGTG

@A00153:690:HJVN5DSXY:1:1101:6298:1000 2:N:0:GAACGTGA+AACTGGTG

GCCACATA Sample1_for
GCGAGTAA Sample1_rev

@A00153:690:HJVN5DSXY:1:1101:6388:1000 1:N:0:GAACGTGA+AACTGGTG

@A00153:690:HJVN5DSXY:1:1101:6388:1000 2:N:0:GAACGTGA+AACTGGTG

GCGAGTAA Sample1_rev
GCCACATA Sample1_for

@A00153:690:HJVN5DSXY:1:1101:7256:1000 1:N:0:GAACGTGA+AACTGGTG

@A00153:690:HJVN5DSXY:1:1101:7256:1000 2:N:0:GAACGTGA+AACTGGTG

GAGCTGAA Sample2_for
GATAGACA Sample2_rev

@A00153:690:HJVN5DSXY:1:1101:8558:1000 1:N:0:GAACGTGA+AACTGGTG

@A00153:690:HJVN5DSXY:1:1101:8558:1000 2:N:0:GAACGTGA+AACTGGTG

GCGAGTAA Sample1_rev
GCCACATA Sample1_for

@A00153:690:HJVN5DSXY:1:1101:9028:1000 1:N:0:GAACGTGA+AACTGGTG

@A00153:690:HJVN5DSXY:1:1101:9028:1000 2:N:0:GAACGTGA+AACTGGTG

GCGAGTAA Sample1_rev
GAGCTGAA Sample2_for

For clarification of the above problem:
There are two raw datasets, one with the forward reads and one with the reverse reads that both contain 2 samples. These samples need to be separated. Every read has a header with a unique number so you can identify which reads are a pair (e.g. in the snippet in the previous message, the first two headers both contain the unique number 1470, meaning that they belong together so that one read is the forward read and one is reverse read). Remember that there are 2 samples in these datasets that we need to separate based on a barcode sequence (a 7bp sequence in our case). This barcode should be in the sequence read and should refer to the same sample for reads that form a pair. For example, if I find that a forward read contains a barcode for sample 1, then I expect that its paired read (the reverse read) also contains a barcode that indicates that it belongs to sample 1. But this is not what I always find (e.g. the last read pair in the snippet of the previous message). Moreover, some reads don't seem to contain any barcode at all (e.g. the read pair that contains the number 5267 in their headers). I think this is also why the software tools discard so many of our reads because they don't what to do in these cases.
See the below figure for a schematic overview of the issue.
Demultiplexing_issue.pdf

Inline figure:

For clarification of the above problem:
There are two raw datasets, one with the forward reads and one with the reverse reads that both contain 2 samples. These samples need to be separated. Every read has a header with a unique number so you can identify which reads are a pair (e.g. in the snippet in the previous message, the first two headers both contain the unique number 1470, meaning that they belong together so that one read is the forward read and one is reverse read). Remember that there are 2 samples in these datasets that we need to separate based on a barcode sequence (a 8bp sequence in our case). This barcode should be in the sequence read and should refer to the same sample for reads that form a pair. For example, if I find that a forward read contains a barcode for sample 1, then I expect that its paired read (the reverse read) also contains a barcode that indicates that it belongs to sample 1. But this is not what I always find (e.g. the last read pair in the snippet of the previous message). Moreover, some reads don't seem to contain any barcode at all (e.g. the read pair that contains the number 5267 in their headers). I think this is also why the software tools discard so many of our reads because they don't what to do in these cases.
See the below figure for a schematic overview of the issue.
Demultiplexing_issue.pdf
These are the barcode sequences:

sample 1 forward read: GCCACATA
sample 1 reverse read: GCGAGTAA
sample 2 forward read: GAGCTGAA
sample 2 reverse read: GATAGACA

Hi Greg,
Great that you could make a working Python code for this. Did you process the entire file? If so, could you please tell me where the results are stored (or copy the results to the N drive)? Then I can have a look at it. Would it be possible to separate out the cases where one of the barcodes is missing from those that seem to have mixed barcodes?

I have not yet processed the entire file, only the first few hundred reads or so. I am trying to implement the demultiplexing myself, because it can't be that hard to do. It is just deciding the what to do with the cases where the barcodes are dubious.

I'll put an update on the progress for demultiplexing the dataset.
I've written a Python code that can demultiplex the dataset which initially seemed to work just fine. But apparently it doesn't like big datafiles, so it crashed the computer when running it on the actual dataset.
To solve this, I translated this code to Bash, which has special functions for handling text files which makes it way more efficient. This also works and with this code I'm able to demultiplex the files without any trouble (it takes somewhere between 0.5 and 1 hour). I have now four datasets:

sample 1 forward reads,
sample 1 reverse reads,
sample 2 forward reads,
sample 2 reverse reads.

However, the problem which I now have is that files are in a wrong order.  Normally the alignment files for the same sample are ordered the same way (i.e. read_1 in the forward file pairs with read_1 in the reverse file, read_2 in forward with the read_2 in reverse etc.). But, this is not the case in demultiplexed files and therefore I am afraid that the alignment software is going to pair the wrong reads with each other and therefore the alignment will be inaccurate.
Now, there is a tool (that belongs to the same software package that I use for sequence alignment) that claims it can solve this issue, but it doesn't like such large files as well and so it crashes.
What I now want to try is not to create four demultiplexed read files (what I now have), but two files where the forward and reverse reads are interleaved. So you end up with the following files:

sample 1 forward & reverse reads,
sample 2 forward & reverse reads.

This would than also have a wrong order initially, but this is potentially easier to solve.

Hi Greg, do these files that you have now still have the headers connected to the reads? In that case it is fine and you do not need to worry about the order of files

I have now two demultiplexed files that includes both the forward and reverse reads from a single sample. I want to check if the demultiplexing is good, because I didn't had the opportunity to check the data as the remote control of the Linux computer is not flawless and I have some hard time connecting to it. But there should be two files in the 'dataset' folder on the Linux machine that are demultiplexed.

I have posted a question on Biostars regarding the demultiplexing of the dataset.
I don't know if it is possible to change the way the barcodes are integrated in the sequences for future experiments? If yes, maybe we should reconsider the barcode location.

I have found a potential solution for the demultiplexing problem. This solution searches for barcodes in the reads and stores both the forward and reverse reads in the same file (i.e. interleaved paired-end fastq file).
It uses the following steps:

grep --no-group-separator -h -A 2 -B 1 -E 'GCCACATA|GCGAGTAA' $file1 $file2 > $outputfile
This searches for all lines in both fastq files that contain either of the two barcodes (in this example those from sample 1). When a line contains one of the barcodes, it stores that line (the actual sequence), the previous line (the header)(-B 1) and the next two lines (the quality score of the read)(-A 2). The reads from both input files are stored in a single output file (the outputfile is thus interleaved).

cat $outputfile | paste - - - - | sort -k1,1 -S 1G | tr '\t' '\n' > $outputfilesorted
The disadvantage of the first step is that the read pairs are not ordered. All the downstream processes expect that read pairs are stored together like this:

readpair 1, forward read
readpair 1, reverse read
readpair 2, forward read
readpair 2, reverse read
etc.
To accomplish this, the reads are sorted based on their headers, so that the reads that form a pair are subsequent reads. This is stored in another output file that should have the same size as the output file from the first step.

bash bbsplitpairs.sh -Xmx1g in=$outputfilesorted out=$outputpair outs=$outputsing fint
After step 2 the fastq files are correctly formatted, but there are still singletons in the files. This indicates reads that do not have a paired partner because the paired partner either has a barcode that belongs to another sample or does not contain any barcodes at all. Those singleton reads have to be removed from the files in order for the downstream processes to work correctly. This can be done by a tool that is available in the bbmap package. This package is later used for trimming of the reads, but also contains the bbsplitpairs.sh tool. This sorts out the reads that do not have a paired partner in the same sample and save those reads separately in the file given by outs.

These steps are repeated with the barcodes from sample 2.
When this is ran for the wt dataset, it finds 53756670 pairs for sample 1 (which is 88.9% of the total number of reads that contain a barcode from sample 1) and for sample 2 it finds 59550160 pairs (91.2% of the total number of reads that contain a barcode from sample 2).
In total there are 134106512 reads in the raw files and there are 113306776 read pairs in the output files for sample 1 and sample 2 combined, which is 84.49%.

I wrote a small python script that checks for fastq files whether two subsequent reads are from the same pair and whether they contain a barcode sequence that belongs to the correct sample. If you run this, ideally it does not give you any output. It will only give warnings when either two subsequent headers are not the same (i.e. there is no pair) or when one of the barcodes is not found in a read (i.e. a read is assigned to the wrong sample).
I have ran this code on the two output files that were created with the method described in my previous post and I indeed get no warnings from this python code, meaning that the output files are likely to be correct.
I also checked the output files with the quality report determined by FASTQC and this also seems to indicate that that the output files does not contain any singleton reads anymore and that the right barcodes are present in all reads.

I have ran the processing pipeline for the first sample (bem1bem2bem3nrp1 delete + Cas9 op de HO locus) in the dataset. This works almost fine, except for a minor issue in the python_TransposonMapping.py code. The results are stored on the N-drive and can be used for analysis, but keep in mind the small issue described below.
The issue relates to the processing of the reverse reads. The mapping of the reads seem to be oke (i.e. the bam-file), but the processing in the python code seems to have a bug when analyzing the reverse reads. When the results of the python code (e.g. the wig-files) are compared with the data in the bam files (e.g. using IGV), the locations of the forward reads in the wig-files seem to match the results from the bam-files. However, the reverse reads are not correct in the wig-files. This causes the reverse reads to be unreliable. The exact cause is yet unclear.
A second point regards to the trimming of the data. I have now trimmed the data with settings that I think are reasonable (see the log-file stored together with the data on the N-drive), but some more iterations of the trimming are needed to make it more accurate. If you have any ideas for the trimming settings, please let me know.

Here I present a more thorough overview of some of the steps I have been taking in the processing pipeline the SATAY data.
This is to clarify some things I have been doing and some issues I am facing.
I take the following processing steps:

Demultiplexing
For this I have created a custom workflow specific for this dataset.
This step will likely be different for different datasets depending how the data is formatted.
The commands for the current dataset are discussed above.
This creates interleaved paired end fastq files for all samples based on the barcodes present in the sequences.
This seem to work fine.
I have written a python code to check the format of the fastq files and all reads are stored as a pair (i.e. there are not singleton reads), the right barcodes are present in each read and the general format of the files is good.
Also I use fastqc that checks the quality of the data and sequences that frequently occur (e.g. primers and barcodes).
This also shows that the demultiplexing is good as the barcodes from sample x does not occur frequently in sample y and vice versa.

Trimming
For this I use bbduk.
Choosing the right settings seems to be quite important as this influences the the quality of the alignment.
Currently I input the whole 50bp primer sequences (including the barcodes) and those are trimmed using kmers.
The kmers should not be too short as then they have low specificity and changes are that false positives occur.
Typically I use somewhere around k=20, only look for primers and adapters in the first 50bp of the sequences and trim all reads to the left when a match is found as the primer sequences occur at the start of the sequences.
The pairs are trimmed together, so that when a primer sequence was found in one of the reads of a pair the reads still have the same length after adapter trimming (which is advised to do according to the bbduk documentation).
Typically not many reads are lost (<0.5%).
Here is an example for sample 2 from Enzo dataset:

Input:                  	59550106 reads<PHONE_NUMBER> bases.
QTrimmed:               	2 reads (0.00%) 	20 bases (0.00%)
KTrimmed:               	59545334 reads (99.99%)<PHONE_NUMBER> bases (32.69%)
Trimmed by overlap:     	347645 reads (0.58%) 	7519706 bases (0.08%)
Total Removed:          	0 reads (0.00%)<PHONE_NUMBER> bases (32.77%)
Result:                 	59550106 reads (100.00%)<PHONE_NUMBER> bases (67.23%)

Time:                         	416.036 seconds.
Reads Processed:      59550k 	143.14k reads/sec
Bases Processed:       8932m 	21.47m bases/sec

What the best trimming steps are is really depending on the data.
The current steps might be improved, but how exactly I am not sure but possibly with trial-and-error I will see when the quality of the data after trimming is best.

Alignment
Alignment is done with bwa mem.
When running this in paired-end mode for the current dataset this gives some warnings.

[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (160, 286, 286)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 538)
[M::mem_pestat] mean and std.dev: (237.82, 93.12)
[M::mem_pestat] low and high boundaries for proper pairs: (1, 664)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_pestat] skip orientation FF as there are not enough pairs

I am not yet entirely convinced what this means and if this is something concerning or not.
The output file of the alignment (the sam file) can be checked using samtools flagstat.
This the flagstat for sample 2:
59692730 + 0 in total (QC-passed reads + QC-failed reads)
0 + 0 secondary
142624 + 0 supplementary
0 + 0 duplicates
32325475 + 0 mapped (54.15% : N/A)
59550106 + 0 paired in sequencing
29775053 + 0 read1
29775053 + 0 read2
17298410 + 0 properly paired (29.05% : N/A)
22173236 + 0 with itself and mate mapped
10009615 + 0 singletons (16.81% : N/A)
3923564 + 0 with mate mapped to a different chr
3494300 + 0 with mate mapped to a different chr (mapQ>=5)

This shows that only about 54% of the reads are actually mapped.
This is not much compared to the number of reads that were initially sequenced, but the absolute number of mapped reads is comparable to the number of mapped reads in the dataset from benoit.
Maybe more worrying is that only 29% of the reads are mapped as actual pairs.
This is also visible using IGV where many reads are colored (see figure below).
This indicates that the paired partner read is mapped to another chromosome.
The cause of this I don't know, but it probably has to do with one of the preceding processing steps.

Transposon mapping
For transposon mapping I still use the python code that was created based on the Matlab code from Benoit.
One issue that occurred in the code has to do with the flags assigned to each read during alignment.
These flags tells something how the reads were mapped (see this website for an overview of all possible flags).
In the code from Benoit only two different flags are defined: 0 for the forward reads and 16 for the reverse reads (e.g. see lines 188 and 189 in the current code).
But for the paired end reads the flags are different.
These different flags need yet to be included in the python code, but I am thinking how to do this in somewhat more robust way so that the code will also work with future datasets where the flags might be different again than in the current dataset.

This is a follow up on point 4 raised in the previous comment on the flag handling in the transposon mapping python code.
This issue has now been resolved and the python code can handle flags from both paired and single read datasets. For this I have created samflag.py that converts a flag (i.e. a decimal number) to a 12-bit binary number. This binary number indicated which parameters hold for a specific read. For more information see this website. Samflag is a python function that can be used for other scripts as well.
So far I have analyzed dataset1 from Enzo (dBem1dBem2dBem3dNrp1 strain) (The results are stored at N:\tnw\BN\LL\Shared\Gregory\datasets\wt1_enzo_dataset\wt1_enzo_dataset_demultiplexed_interleaved_sample1\wt1_enzo_dataset_demultiplexed_interleaved_sample1_trim2\align_out). When I compare the results from this dataset with those from Benoit (e.g. using this code that show the read distribution per chromosome) it shows that the read density is >2 times higher in the Enzo-dataset compared with the Benoit-dataset (strictly speaking I can't directly compare the datasets since its a different strain). This is even when many reads are ignored due to previous steps in the processing of the Enzo-dataset (see my previous comment). I also see that around the centromeres and in the Ade2-gene the insertion density is higher, just like in the Benoit-dataset. However, I also see that there are still reads in annotated essential genes (of course this isn't a wild type strain which might mess with the essentiality of genes) and I see that there are many insertions in Bem2, which is not what we expect (there are only few to no insertions in Bem1, Bem3 and Nrp1 which is what we expect and the few insertions that those genes have might be due to alignment uncertainty and noise).
Below I have included the read distribution for chromosome 3 where on the left I have plotted Benoit-dataset (wt1) and on the right I have plotted Enzo-dataset 1. Note the difference in y-scale (this is created using this code).

This is a follow up on point 4 raised in the previous comment on the flag handling in the transposon mapping python code.
This issue has now been resolved and the python code can handle flags from both paired and single read datasets. For this I have created samflag.py that converts a flag (i.e. a decimal number) to a 12-bit binary number. This binary number indicated which parameters hold for a specific read. For more information see this website. Samflag is a python function that can be used for other scripts as well.
So far I have analyzed dataset1 from Enzo (dBem1dBem2dBem3dNrp1 strain) (The results are stored at N:\tnw\BN\LL\Shared\Gregory\datasets\wt1_enzo_dataset\wt1_enzo_dataset_demultiplexed_interleaved_sample1\wt1_enzo_dataset_demultiplexed_interleaved_sample1_trim2\align_out). When I compare the results from this dataset with those from Benoit (e.g. using this code that show the read distribution per chromosome) it shows that the read density is >2 times higher in the Enzo-dataset compared with the Benoit-dataset (strictly speaking I can't directly compare the datasets since its a different strain). This is even when many reads are ignored due to previous steps in the processing of the Enzo-dataset (see my previous comment). I also see that around the centromeres and in the Ade2-gene the insertion density is higher, just like in the Benoit-dataset. However, I also see that there are still reads in annotated essential genes (of course this isn't a wild type strain which might mess with the essentiality of genes) and I see that there are many insertions in Bem2, which is not what we expect (there are only few to no insertions in Bem1, Bem3 and Nrp1 which is what we expect and the few insertions that those genes have might be due to alignment uncertainty and noise).
Below I have included the read distribution for chromosome 3 where on the left I have plotted Benoit-dataset (wt1) and on the right I have plotted Enzo-dataset 1. Note the difference in y-scale (this is created using this code).

I have reanalyzed sample 2 (wild type strain) from Enzo's dataset. This time I have treated the data as being single-end reads. For this I took the reads that contained the sequencing primer (TTTTACCGACCGTTACCGACCGTTTTCATCCCTA) which should correspond to the tn-insertion site. The discarded reads contained the NlaIII or the DpnII ligation sequence and corresponds to the mate reads of the reads containing the sequencing primer. With this approach I was left with about 37% of the reads (i.e. 24063554 reads) compared with the number of reads that originally were sequenced for sample 2 (that might appear to be little, but remember we lose at least half of all reads since we discard the mate reads). Those reads were trimmed for the primer and adapter sequences and aligned to the reference genome as single-end reads. This allows about 15265731 reads to be aligned and those can be used for analyzing the transposon insertions. The results are stored at "X:\tnw\BN\LL\Shared\Gregory\datasets\wt1_enzo_dataset\wt1_enzo_dataset_demultiplexed_interleaved_sample2\wt1_enzo_dataset_demultiplexed_singleend_sample2_trim1"
When I do some checking of the reads I see that in most of the annotated essential genes there are less transposons compared to other regions, but this is not always very clear (some annotated essential genes still have relatively many reads). Also our data appears to be more noisy has less 'contrast' compared to the Benoit dataset. With this I mean that on average we appear to have less reads per insertion location and there are locations that have exceptionally many reads. This might make it even more tricky to estimate the relative fitness of gene inhibitions.
Below are two figures of chromosome IX where the second graphs are zoomed in slightly compared to the first graphs. On the left is Benoit's dataset and on the right is Enzo's dataset.

Maybe other trimming and alignment settings might improve the contrast a bit, but at least there is some improvement compared with the prior methods I have tried that used paired-end mapping for the reads where I don't see much change in the number of reads in essential genes compared to the non-essential genes.
I will check the sample 1 dataset as well to see if there appear no insertions in the deleted genes.

I have reanalyzed sample 2 (wild type strain) from Enzo's dataset. This time I have treated the data as being single-end reads. For this I took the reads that contained the sequencing primer (TTTTACCGACCGTTACCGACCGTTTTCATCCCTA) which should correspond to the tn-insertion site. The discarded reads contained the NlaIII or the DpnII ligation sequence and corresponds to the mate reads of the reads containing the sequencing primer. With this approach I was left with about 37% of the reads (i.e. 24063554 reads) compared with the number of reads that originally were sequenced for sample 2 (that might appear to be little, but remember we lose at least half of all reads since we discard the mate reads). Those reads were trimmed for the primer and adapter sequences and aligned to the reference genome as single-end reads. This allows about 15265731 reads to be aligned and those can be used for analyzing the transposon insertions. The results are stored at "X:\tnw\BN\LL\Shared\Gregory\datasets\wt1_enzo_dataset\wt1_enzo_dataset_demultiplexed_interleaved_sample2\wt1_enzo_dataset_demultiplexed_singleend_sample2_trim1"
When I do some checking of the reads I see that in most of the annotated essential genes there are less transposons compared to other regions, but this is not always very clear (some annotated essential genes still have relatively many reads). Also our data appears to be more noisy has less 'contrast' compared to the Benoit dataset. With this I mean that on average we appear to have less reads per insertion location and there are locations that have exceptionally many reads. This might make it even more tricky to estimate the relative fitness of gene inhibitions.
Below are two figures of chromosome IX where the second graphs are zoomed in slightly compared to the first graphs. On the left is Benoit's dataset and on the right is Enzo's dataset.

Maybe other trimming and alignment settings might improve the contrast a bit, but at least there is some improvement compared with the prior methods I have tried that used paired-end mapping for the reads where I don't see much change in the number of reads in essential genes compared to the non-essential genes.
I will check the sample 1 dataset as well to see if there appear no insertions in the deleted genes.

That is great to kow :) Can you also plot the statistics of the library to compare it more thoroughly with Benoit dataset , like the ones in the table below

With this I mean that on average we appear to have less reads per insertion location and there are locations that have exceptionally many reads

why this is a problem? it could be a real fact from the data... that is why it is good to know the resolution of the transposition , how many transposons in average are inserted per bp? For benoit it was approximately every 160bp right?

That is great to kow :) Can you also plot the statistics of the library to compare it more thoroughly with Benoit dataset , like the ones in the table below

With this I mean that on average we appear to have less reads per insertion location and there are locations that have exceptionally many reads

why this is a problem? it could be a real fact from the data... that is why it is good to know the resolution of the transposition , how many transposons in average are inserted per bp? For benoit it was approximately every 160bp right?

Hi Greg, nice that it worked, but the uneven distribution of reads throughout the genome is indeed troublesome. Do you maybe know why 10 million of the reads that were left after taking only those that contain the sequencing primer from Benoit could aligned? And how many mismatches did you allow in the alignment of the sequencing primer?
It would be good to have the number of transposons that were mapped such that we can see if the lower amount of contrast between essential/non-essential genes is due to a difference in the number of transposon insertions or due to the amount of reads we have. What it looks like now is that we might have filtered for locations that survive this process of trimming and alignment, which could cause the peaks in your alignment plots.

Hi Greg, nice that it worked, but the uneven distribution of reads throughout the genome is indeed troublesome. Do you maybe know why 10 million of the reads that were left after taking only those that contain the sequencing primer from Benoit could aligned? And how many mismatches did you allow in the alignment of the sequencing primer?
It would be good to have the number of transposons that were mapped such that we can see if the lower amount of contrast between essential/non-essential genes is due to a difference in the number of transposon insertions or due to the amount of reads we have. What it looks like now is that we might have filtered for locations that survive this process of trimming and alignment, which could cause the peaks in your alignment plots.

I have tried to determine the statistics similar to the ones Benoit shows in his paper, but I fail to reproduce the values presented in the paper. Probably I am doing something wrong here, but nevertheless here are the number as I have them so far for Enzo's dataset. I'll update the numbers as soon as figured out what's wrong.

@EKingma

Do you maybe know why 10 million of the reads that were left after taking only those that contain the sequencing primer from Benoit could aligned?

I don't know, I haven't stored the rejected reads. I should recreate the dataset for this with the rejected reads.
More interestingly, I have also plotted the number of transposon instead the number of reads. The number of transposons in less noisy compared to the read count. This show some similarities with Benoits dataset (see figure below, note the difference in y-scales). This clearly indicates the centromere and Ade2-gene bias in transposition. Also there is a large peak in chromosome 12 around bp 460000. I looked at this location on sgd, but I couldn't find anything interesting that would explain such a large peak. There are mainly non-coding regions and ribosomal RNA sequences and two small genes which I don't expect them to cause such a transposition bias. Also in the number of reads a high density of reads are visible.

Note: I used these dataset for creating the above plot:

X-drive: Gregory\datasets\dataset_enzo\wt1_enzo_dataset_demultiplexed_interleaved_sample1\wt1_enzo_dataset_demultiplexed_singleend_sample1_trim20210127\align_out"
X-drive: Gregory\datasets\dataset_enzo\wt1_enzo_dataset_demultiplexed_interleaved_sample2\wt1_enzo_dataset_demultiplexed_singleend_sample2_trim20210122\align_out"
"X-drive: Gregory\datasets\dataset_benoit"


inferno: 0.10.6 -> 0.10.7

Motivation for this change
https://github.com/jonhoo/inferno/blob/master/CHANGELOG.md
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Change looks good and the ci is passing.
Thank you!


css linter: Report empty declarations more clearly.

Raise a CssParserException when declarations are empty.

Automated message from Dropbox CLA bot
@showell, it looks like you've already signed the Dropbox CLA. Thanks!

Merged, after removing the unused res variable linter failure.  Thanks @showell!


Improve instructions on Importing groups into Ultra

Jacob developed a guide on this specific topic and would like me to update on the Ultra Essentials.
The guide is now stored in OneNote under "Importing groups into Ultra - Jacob"

[ ] Compare the doc with the one on Ultra Essentials and merge the two
[ ] Update Ultra Essentials

Importing Groups into Ultra
What is this resource for?
This resource covers the different aspects needed to successfully import your group sets, groups and students.
Why import?
Whenever we are dealing with large cohorts and they need to be split into groups, it’s preferable to do what’s called a batch enrolment. This both saves time and minimises the risk of user error.
What do I need?
In order to enrol your student(s) into their group(s) you need to first create your Group Set(s) and Group(s).
If you are starting from scratch you can download a sample file. In the event you are going to be doing this a multitude of times, it’s recommended you rename it from its default name to a more unique identifier e.g. the module code.
The sample file has different column headers which can be described as such:
Any columns with the asterisk(*) are mandatory to fill out.

Sonia is set to do the update.


List Class getRange (and sublist) don't return single item

Not sure if it's intended behaviour, but when start and end index value is the same getRange returns and empty list. Even a list with one item is still a list and should be returned as such.
To be more specific, in my case I was using myList.getRange(0,0) and getting an empty list.
If the index is a valid index, say 0, and you call List.getRange(0,0), a List containing the single item at index 0 should be returned.

@craigsailor
Could you please update the question with your intended output?

Try using myList.elementAt(0) . If you want to check that index is a valid one ,try the code inside a try-catch block

@craigsailor documentation https://api.flutter.dev/flutter/dart-core/List/getRange.html says that the range is valid if 0 <= start <= end <= len, where len is this list's length. The range starts at start and has length end - start.
An empty range (with end == start) is valid.
I believe that it was intended to have the length of returned Iterable as end - start, and the end points on the element which will not be included.
I believe myList.getRange(0,1) solves the problem. Can you try?

myList.getRange(0,1) also solved the problem like myList.elementAt(0).

Using getRange(0,1) fails if there is only a single element in the list as 1 is an invalid index.
The same problem would happen any time you are trying to get a range that is at the end of your list.

Please provide the error log while using myList.getRange(0,1) along with your list. I am not getting an error with it.
My code:
var x=['hai'];  print(x.getRange(0,1));
this will print `(hai)'

Apologies. That does work.
I would recommend that the documentation be improved slightly to make it more clear that the end index must be one more than the index of the last time you'd like to retrieve.
Many Thanks.


[KYUUBI #2238] [WIP] Support Flink 1.15

Why are the changes needed?
Flink 1.15.0 is out, Kyuubi needs to support it.
Please clarify why the changes are needed. For instance

Rename the original kyuubi-flink-sql-engine to kyuubi-flink-sql-engine-1.14, the code inside remains unchanged
New kyuubi-flex-sql-engine-1.15 module, implementation reference  kyuubi-flex-sql-engine-1.14
Removed the dependency declaration of Flink from the parent module,  because the dependency declaration of Flink is placed in the specific sub module of kyuubi-flink-sql-engine

kyuubi-flex-sql-engine-1.15 part of the work is still in progress, At present, 28 test cases have passed and 9 have failed, You can review, put forward suggestions for improvement, I will continue to improve,  After this is passed, a Kyuubi flex SQL engine base module will be pulled out, and the public APIs that are not affected by the version will be put into this module. In the next step, the coexistence and isolation of multiple versions will be realized.
How was this patch tested?

[x]  Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[ ] Run test locally before make a pull request

@wolfboys please rebase this PR, thanks

I'm a little bit surprised that you create dedicated modules for every Flink minor version. Actually, in Kyuubi project, we try our best to avoid maintaining multi-version support in this way, we only adopt this approach in Kyuubi Spark Extension modules because all APIs that we used are private APIs.
IMO, to support both Flink 1.14 & 1.15, even if we access some Flink private API in Flink engine module, it should be handled by reflection properly. In detail, my idea is,

Introduce flink-1.14 and flink-1.15 maven profiles to handle name changes of some modules.

<profile>
  <id>flink-1.14</id>
  <properties>
    <flink.module.scala.suffix></flink.module.scala.suffix>
  </properties>
</profile>
<profile>
  <id>flink-1.15</id>
  <properties>
    <flink.module.scala.suffix>${scala.binary.version}</flink.module.scala.suffix>
  </properties>
</profile>

The code should work with flink-1.14 but may not flink-1.15, find them and extract a FlinkShim trait with implementations FlinkShim_1_14 and FlinkShim_1_15, to detect and handle the Flink version in runtime.
Enable flink-1.14 and flink-1.15 in CI
Change the default Flink version to 1.15

In fact, my approach is to refer to the implementation of shims in zeppelin and streamx. If a module is based on a specific flink version such as (flink 1.14) to support multiple different flink versions, use reflection In fact, it will be more troublesome to implement. In addition to the incompatibility of API changes, some versions such as (flink 1.15) will even use new APIs (new APIs than flink 1.14),   It's not easy to do by reflection
The current method is only an initial version, and in the final implementation, the common classes and codes in each version will be extracted upward, and only the code compatible with that version will be placed in each shims package.

I'm a little bit surprised that you create dedicated modules for every Flink minor version. Actually, in Kyuubi project, we try our best to avoid maintaining multi-version support in this way, we only adopt this approach in Kyuubi Spark Extension modules because all APIs that we used are private.
IMO, to support both Flink 1.14 & 1.15, even if we access some Flink private API in Flink engine module, it should be handled by reflection properly. In detail, my idea is,

Introduce flink-1.14 and flink-1.15 maven profiles to handle version/name changes of some modules.

<profile>
  <id>flink-1.14</id>
  <properties>
    <flink.version>1.14.4</flink.version>
    <flink.module.scala.suffix>_${scala.binary.version}</flink.module.scala.suffix>
  </properties>
</profile>
<profile>
  <id>flink-1.15</id>
  <properties>
    <flink.version>1.15.0</flink.version>
    <flink.module.scala.suffix></flink.module.scala.suffix>
  </properties>
</profile>

The code should work with flink-1.14 but may not flink-1.15, find them and extract a FlinkShim trait with implementations FlinkShim_1_14 and FlinkShim_1_15, to detect and handle the Flink version in runtime.
Enable flink-1.14 and flink-1.15 in CI
Change the default Flink version to 1.15

On other hand, if the API changes are huge enough, I agree with you that dedicated modules for each version is a better way.

Thank you for your review and so great point of view, I will seriously consider how to achieve it

@wolfboys Thanks for your work! Looking forward to your upcoming PR.


media=prepress fails to be processed by Kindle Publishing

media values of screen and print do upload and are processed with no problem.
prepress will be rejected. After adding an optimization pass with rghost (and obviously ghostcript in the background) it does then upload. I also tried the Foxit PDF Editor, applying only the cleanup optimization and it also did something which made the format good for Kindle Publishing.
Understanding that finding a very small thing in the PDF generation that could be slightly different could be a huge effort, I would propose that a small hint be added to the documentation, pointing that in case of any problem with the prepress media mode, an optimization pass can iron things out.
Best regards

This looks like a duplicate of #2476.

I would propose that a small hint be added to the documentation, pointing that in case of any problem with the prepress media mode, an optimization pass can iron things out.

I'm open to accepting a pull request to add this information to the documentation.

@mementum you can try this workaround I used to get a prepress PDF working with Lulu: https://github.com/meonkeys/print-this/blob/main/book/extensions.rb#L22

If we can confirm with local tools that this indeed fixes the validation error, we could consider adding this into the converter when media=prepress. But we have to be able to test it locally.

pdftotext appears to be happy with the workaround, as discussed in chat.
I'm using pdftotext version 22.02.0. Came to my workspace via poppler-utils 22.02.0-2ubuntu0.3 on Ubuntu 22.04.3 LTS 64-bit desktop.

Thanks for the fix.

Let me add positive feedback now that version 2.3.11 has been released: media=prepress is accepted by KDP.

Thanks for sharing @mementum! This makes me very happy. :man_dancing:


option to work with raw json object?

http://jsonviewer.stack.hu/ allows to pass in raw json data (without children / text fields, etc), eg:
{"a":[1,2,{"b":3}]}

which shows up as:

is there a similar built in way in jstree or do I have to write my own raw_json_to_jstree_json (which would add children, text nodes) ?

jsTree only understands the two documented formats - flat or nested. You need a converter.


Compatibility fixes/improvements for JSON/RPC filter polling

Description
A set of changes that improve consistency of the Edge JSON/RPC surface area for block/log filters, with other Ethereum JSON/RPC client implementations. These proposed changes vary from clear fixes, to other items that are more subjective. So I'm leaving this in draft while I work through a set of tests with event streaming framework ethconnect - which is part of the Hyperledger FireFly project.

Fix for #640
Consistent filter not found error string
Allowing filters to exist for more than one minute, when being actively used

Changes include

[x] Bugfix (non-breaking change that solves an issue)
[ ] Hotfix (change that solves an urgent issue, and requires immediate attention)
[ ] New feature (non-breaking change that adds functionality)
[ ] Breaking change (change that is not backwards-compatible and/or changes current functionality)

Breaking changes
If any users were depending on the invalid behavior of the JSON/RPC eth_getFilterChanges returning a string result (rather than an array).
TODO: This needs to stay in draft until a discussion happens on how to handle https://github.com/umbracle/ethgo, which has code that depends on the incorrect behavior. Resulting in this test failure:
--- FAIL: TestNewFilter_Logs (5.17s)
    logs_test.go:84:
                Error Trace:    /Users/pbroadhurst/dev/polygon/polygon-edge/e2e/logs_test.go:84
                Error:          Received unexpected error:
                                json: cannot unmarshal array into Go value of type string
                Test:           TestNewFilter_Logs

Checklist

[x] I have assigned this PR to myself
[x] I have added at least 1 reviewer
[x] I have added the relevant labels
[ ] I have updated the official documentation
[x] I have added sufficient documentation in code

Testing

[ ] I have tested this code with the official test suite
[x] I have tested this code manually

Manual tests
Please complete this section if you ran manual tests for this functionality, otherwise delete it
Documentation update
Please link the documentation update PR in this section if it's present, otherwise delete it
Additional comments
Please post additional comments in this section if you have them, otherwise delete it

I've found a follow-on issue, when a block filter returns an empty result set.
In this case we get [""] (rather than [].
Will extend the PR to cover this case

Hey @peterbroadhurst,
Thank you for opening up a PR.
Can you please explain what kind of functionality is expected from the filter response, that Edge is not providing?
Also, I'd like to know what kind of problem exists with the ethgo package regarding this functionality.
We are moving towards dropping this package completely from Edge in the future as it's been more of a burden to maintain than our actual JSON-RPC interface.
If you can describe the problem here with ethgo, I'd be able to get a PR out and a fix merged very quickly as I'm one of the collaborators on that repo.

Thanks you @zivkovicmilos - I'm sorry for the slow reply here.
The problem in the ethgo package is here: https://github.com/umbracle/ethgo/blob/8326bf607acf2ae4893128cacf205c67261cc411/jsonrpc/eth.go#L74-L86
This is actually working around the behavior currently on the JSON/RPC interface for eth_getFilterChanges, by unmarhsaling the string response into []*ethgo.Log within the client code.
So when the fix in this PR changes the JSON/RPC response to an array (rather than a string), the error occurs.
So the PR needed in ethgo is just to pass the []*ethgo.Log interface directly to e.c.Call() (rather than the string).
I had mean to submit this as a PR to ethgo, and got distracted by other priorities in the past couple of days. Very sorry about that 🙇 . I will do that now.

I've raised https://github.com/umbracle/ethgo/pull/209 - I'm going to check in a change while this PR is in draft, that updates the go.mod to redirect to that branch so I can check it allows the PR tests to pass.

Sadly my attempt to test with my fork of ethgo failed, as the test run against the vendor directory (and go mod vendor didn't respect the replace directive it seems). So it might be necessary to work through https://github.com/umbracle/ethgo/pull/209 before I can re-test here.

Ok - with https://github.com/umbracle/ethgo/pull/209 merged, I've updated the PR to pull in that commit of ethgo.

I have read the CLA Document and I hereby sign the CLA

Hey @peterbroadhurst,
Can you please pull in the latest develop branch into this PR so we can go ahead and merge it? 🙏
We've made some minor fixes in the meantime that you'll need to resolve.

Thanks @zivkovicmilos - I've completed the merge, and added the change suggested by @Kourin1996


Make the car recognize the distance to the obstacle.

Requirement to be covered for this Issue:

The car should be able to detect when it is about to run into an obstacle.

Requirement to be covered for this Issue:

The car should be able to detect when it is about to run into an obstacle.

This issue was resolved in PR #15


lens type

What type of lens is used in Intel realsense D415, D435 and L515 RGB camera?
rectilinear or curvilinear lens
Where can I find this information?
I need only for RGB camera
Thanks in advance

The RGB imager component used in the 400 Series cameras is the OmniVision OV2740.  An OmniVision PDF of the component's 'product brief' can be found at the link below.
https://www.ovt.com/download/sensorpdf/153/OmniVision_OV2740.pdf
The product specification has this description for the lens:
Lens chief ray angle: 33 degrees non-linear
The OV2740 is also listed in the L515's data sheet document as the imager used for RGB:


Make collector fields the same width and align left edges

I'm seeing equal-width collector fields that are left-aligned on Chrome, FF, Edge, and IE as well as Safari on BrowserStack. Did you mean full-width fields with the label sitting on top?

Must be Mac only:


Iphone XR not displaying dropdown list with location data

Issue : On my Iphone XR , as I enter location data (for Ex: zipcode), drop down list with related data not showing up.
steps:(on safari)
1.Go to Staging. https://staging.fightpandemics.work/
2.navigate to 'View Help Board'
3.Go to location
3.Enter zipcode
4.No drop-down list dispalyed  as shown in video below
https://share.getcloudapp.com/p9uPyZW7
expected: drop-down list with related data should appear as the user provides location info

@vikjoshi ,please feel free to change  release label , if needed

@rabbi08  tested .fine on iphone 11 https://fightpandemics.slack.com/files/U015XBKFRHA/F01795GFRTN/rpreplay_final1594751316.mp4
@nivetheatha tested and fine on samsung galaxy

His @sheeba1 Something similar was reported earlier https://github.com/FightPandemics/FightPandemics/issues/1119#issue-655037639 but they never responded to my question.
I had a feeling it is something related to a feature that antd is using that the browser doesn't support.  BUT now I have been trying BrowserStack I see with onboarding it works, but for filters it doesn't so that may be a clue. It would be nice to have more than 1 minute to debug (we need that partnership!).
To confirm. Which browser are you using, Safari or Chrome?

@joshmorel on both safari and chrome

Interesting. I am pretty sure it has something to do with z-index https://stackoverflow.com/a/58490946/4582054
Both the modal (from antd-mobile) and the select (from antd) utilize z-index and the modal has translateZ(1px) and might be related. Another time where using antd-mobile and antd together is causing some weird issues...

I think this is the same issue I entered on 1119 defect.

@joshmorel , I am assigning this to you for right now because it is related to geolocation but can reassign if needed. Thanks!

@vikjoshi that's okay for now. I'm getting an idea of what might be the issue.
Having an iPhone device to emulate for more than 1 minute would be helpful. Until we partner with BrowserStack or other maybe we can use AWS Device Farm. I'll ask @mannykary .

@joshmorel i have access to an XR- let me know if I can help

Thanks @vikjoshi I will ask you & @sheeba1 to confirm when I think I've got it.
If the situation is complicated (I'll need to override some the styles from one or both of 2 external dependencies) an emulator would be helpful but I can make do for now (+ hopefully it's not too complicated - maybe only a single CSS rule?).

@vikjoshi @sheeba1 @dabartlett Can you test on https://featurefix-iphone-overlay-bug.fightpandemics.xyz
I tried once on Safari on BrowserStack and it worked. I couldn't test Chrome because I ran out of time on time and valid devices*.
*It seems BrowserStack doesn't upgrade emulated devices to newer OS version.
This isn't supported (used on feed for infinite scroll) up to iOS 12.01 and I think with BrowserStack you test at where the OS was at the time of release (maybe?)
https://caniuse.com/#feat=intersectionobserver

It seems iPhone 6S are upgraded to iOS 13 so we should be fine, no? If you think we need to support older devices we'd need to open another issue but it would be a lot of work to address this.

@joshmorel tested on both safari and chrome on iPhone XR and it is working as expected!
Yeah, for now lets not worry about pre iPhone 6s

tested and worked fine on iphone XR ,safari

@rabbi08 @nivetheatha can you please test on your devices ,just to make sure its working fine and did not breaking already working code.Thanks!

@sheeba1 works fine in Android devices.

I have re-tested on iPhone-11 and it is working as expected. Thanks

Tested with iphone 6S after upgrading to version 13 and it works as well. thank you


Dependency order in tools can raise Exception in client

How to reproduce:

Create tool1
Create tool2 with tool1 as dependency
Create tool3.

First add tool1 as dependency
Then add tool2 as dependency.

The order is very important. If we add first too2 and then tool1 as dependencies to tool3, then it works.

Closing this for now. Will reopen if any issue appear.


Godoc issue with go.mongodb.org/mongo-driver

This documentation for Decode says that

DecodeBytes will return

when it should say

Decode will return

I don't think this is the right place to report this--this looks like the Godoc issue tracker and I suspect this belongs on the Mongo issue tracker. However, it wasn't obvious where to go to report a documentation error. Can someone point me in the right direction?

Looks like mongodb/mongo-go-driver is the repository you're looking for.

Thank you!


Added missing "-o" parameter to Dockerfile instructions.

Coverage remained the same at 87.52% when pulling 6e881bbfb1e05756b799a9515cd599f036976559 on njgraham:docker_instructions_typo into<PHONE_NUMBER>558ef42dc6ea75d433218471e91a0d on quantopian:master.

Hey @njgraham thanks so much for adding this flag :) we appreciate your PR


Azure pipeline use Microsoft hosted agent

Type of Change
Code-Scan, Spell-Check, UT-basic (stage: Coverage) will use Microsoft-hosted agent instead of self-hosted agent
Description

yml:
pool: ICX-16C -->     pool:  vmImage: "ubuntu-latest"
update UT-basic Coverage stage dependsOn

dockerfile:
DockerfileCodeScan.devel: move some  pip install packages to pylint.sh, use ubuntu:20.04 as devel

format some files

Expected Behavior & Potential Risk
azure pipeline will use Microsoft-hosted agent
How has this PR been tested?
pipelines succeed
Dependency Change?
no

test results:
Code-Scan Pipeline: https://dev.azure.com/lpot-inc/neural-compressor/_build/results?buildId=6337&view=results
UT-Basic Pipeline: https://dev.azure.com/lpot-inc/neural-compressor/_build/results?buildId=6308&view=results
Spell-Check Pipeline: https://dev.azure.com/lpot-inc/neural-compressor/_build/results?buildId=6310&view=results


[Suggestion]Update the CN translation of boolean.md according to the new EN doc.

Note: This repository is ONLY used to solve issues related to DOCS.
For other issues, please move to other repositories.

Is there anything that's missing or inappropriate in the docs? Please describe.
boolean.md under CN site folder
Describe your suggestion
The EN doc is updated. Please help update the CN translation accordingly.
Additional reference
The updated parts include doc overview and content under the section "operators".

/assign sreyan-ghosh

Hi @sreyan-ghosh, thank you for showing your interest in contributing to Milvus! I have reviewed your PR and it seems like you've only updated the overview part in the CN doc. However, the content under the section named "operators" also needs to be translated into Chinese.
Plus, I have added more details to the issue description. We do not accept pure machine translation but MTPE is okay. Please take a look at the issue description and feel free to let me know if you still would like to take this issue.

I see. Then @YiyunNi I wouldn't be able to take this issue up, I saw the last change made to the boolean.md file and saw a link change so implemented that. Please close my PR. Sorry to have misunderstood.

But thank you anyways! If you are still interested in contributing to Milvus, you can take a look at other non-translation issues that are still open and unassigned. @sreyan-ghosh

I can try this.

Hi @noviicee, thank you for your contribution. I have reviewed your PR and left several comments. Would you please revise the translation accordingly?

Hi @noviicee, thank you for your contribution. I have reviewed your PR and left several comments. Would you please revise the translation accordingly?

Sure thanks.


language file fr.js missing

Hello, the language file resolving does not work well for us when changing django language to french ('fr').
  Failed to load: http://localhost:8000/static/tinymce/js/tinymce/langs/fr.js

What exists in the codebase is fr_FR.js but from code i think this will never be resolved or is there any configuration parameter that i missed?
Language config from settings:
  LANGUAGE_CODE = 'en'
  LANGUAGES = [
      ('en', _('English')),
      ('fr', _('French')),
      ('de', _('German')),
  ]

My TinyMCE default config:
TINYMCE_DEFAULT_CONFIG = {
    'selector': 'textarea',
    'theme': 'modern',
    'plugins': 'link image preview codesample contextmenu table code',
    'toolbar1': 'bold italic underline | alignleft aligncenter alignright alignjustify '
                '| bullist numlist | outdent indent | table | link | codesample | preview code',
    'contextmenu': 'formats | link',
    'menubar': False,
    'inline': False,
    'statusbar': True,
    'height': 360,
}

Thanks much!

For French TinyMCE uses full "language_COUNTRY" codes: fr_FR and fr_CH. You can either use a full language code in Django: fr-fr or override automatic language configuration by specifying respective TinyMCE language options explicitly in TINYMCE_DEFAULT_CONFIG. Or you can manually rename fr_FR.js to fr.js.

Great thank you for responding that fast and providing a solution 👍 , i selected to put a fr.js file into the projects static folder but your first solution sounds better, depending on the side effects, will give it a try.


[Question]: Deleting cross compiler documentation

Way back when we first started the project, we put together some cross compiler documentation that was to be used by developers to hand create the cross compilers. This process is scripted now, and would be insane to document now that we also support docker, and will in the next coming weeks support Clang//LLVM.
So the question is... does anyone have an issue with me deleting this documentation? It will always be available in the version 1.0 tag (even though it is out-of-date in that tag too) just in case people want to refer back to it.

Fine with me..I've only ever used the script :)


Gather resources and next steps from testing group

We had a testing working group lead by @mratsim. We need to gather the results and next steps from this group.

closed by #3


Remove fetchModrinthMod from README.md

Reflect the removal of fetchModrinthMod in the README.md.

Completely slipped by me, good catch!


Analytics

Google Analytics page tracking up and running

Codecov Report

Merging #161 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #161   +/-   ##
=======================================
  Coverage   77.21%   77.21%
=======================================
  Files          15       15
  Lines         373      373
  Branches      103      103
=======================================
  Hits          288      288
  Misses         85       85

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f542cfc...9be7371. Read the comment docs.

It doesn't look like much but I spent at least 8 hours on this! That autotrack module from Google is brainless once it is working. Auto tracking URL changes without any fiddling with react-router etc.
But it is published as ES2015 code, even via the npm package.
Plus all the time in the weeds...had been using a 3rd party plugin that tied into react-router, but that required hooking into history changes, which is surprisingly difficult in react-router - should be easy...
Anyway...this is what my travails have yielded. And it is actually tracking in Google Analytics.


missing dependencies when using uikit as bower dependency

tstrap-sass-datepicker/sass/datepicker```

the uikit probably needs to include this file if it references it

fixed by #13


Extend MessageRouter to support other Serialization types (JSON, etc)

Plug in your own Serializer
Default of FsPickler

Preempted by #11


[Dubbo-2838] Fix UT failed on windows with FileNetworkerTest

What is the purpose of the change
Fix Dubbo-2838
Brief changelog
Fix FileNetworkerTest failure
Verifying this change
Run UT on windows platform
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a GITHUB_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GITHUB issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [Dubbo-XXX] Fix UnknownException when host config not exist #XXX. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn clean install -DskipTests & mvn clean test-compile failsafe:integration-test to make sure unit-test and integration-test pass.
[x] If this contribution is large, please follow the Software Donation Guide.

Codecov Report

Merging #2839 into master will decrease coverage by 0.16%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2839      +/-   ##
==========================================
- Coverage    63.8%   63.63%   -0.17%
==========================================
  Files         577      577
  Lines       25930    25930
  Branches     4541     4541
==========================================
- Hits        16544    16501      -43
- Misses       7239     7269      +30
- Partials     2147     2160      +13

Impacted Files
Coverage Δ

...onfig/spring/extension/SpringExtensionFactory.java
68.42% <0%> (-15.79%)
:arrow_down:

.../remoting/transport/netty4/NettyClientHandler.java
75% <0%> (-13.89%)
:arrow_down:

.../apache/dubbo/qos/protocol/QosProtocolWrapper.java
69.23% <0%> (-12.83%)
:arrow_down:

...ache/dubbo/remoting/transport/AbstractChannel.java
75% <0%> (-12.5%)
:arrow_down:

...org/apache/dubbo/rpc/filter/ActiveLimitFilter.java
77.77% <0%> (-11.12%)
:arrow_down:

...ubbo/rpc/protocol/dubbo/ChannelWrappedInvoker.java
41.66% <0%> (-8.34%)
:arrow_down:

.../apache/dubbo/remoting/transport/AbstractPeer.java
58.69% <0%> (-6.53%)
:arrow_down:

.../dubbo/remoting/transport/netty4/NettyChannel.java
61.17% <0%> (-4.71%)
:arrow_down:

...ing/exchange/support/header/AbstractTimerTask.java
66.66% <0%> (-4.17%)
:arrow_down:

...rpc/protocol/dubbo/telnet/InvokeTelnetHandler.java
54.21% <0%> (-3.62%)
:arrow_down:

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 46b3458...311acf5. Read the comment docs.

Fix #2838

I have seen another pr to fix this issue:
https://github.com/apache/incubator-dubbo/pull/2848
Personally think that his modification is simpler and effective.
What do u think about it?

Please refer to https://en.wikipedia.org/wiki/File_URI_scheme
Actually an additional slash is only for Windows local file which hostname is omitted.
And for Unix like system remains the same. (file:///tmp/junit2484075406115019206/junit2016855624112146186.tmp ,three slashes in total including the root path one)
In fact the UT passed, so I think both pr are fine.  😄

Hi,
We don't need to judge our system, we just need to add a '/'. The content after the third '/' will be considered the path of the URL (the current way of parsing dubbo). I know that this problem is caused by the separator of WINDOWS. But in the way of fixing, I think we only need to add a '/' to move closer to dubbo's parsing.
Another thing is that that pr is simpler. So I personally prefer that one.  Thx for your pr, too. Very nice solution. 👍


Can i use it in Vanilla javscript

I saw the documentation here and on npm.
I just want to ask , Can i use it on pure js.
Because i feel like it is built solely for react. Why I felt so was because i saw react templating (jsx) in the html.

Hello,
The project itself is made for Vanilla already. The React implementation is just an optional feature.

Hello how can I use with php


associate State variable with Client

This commit levels your approach one level higher. Because state can be modified during the connection and handlers run concurrently, functions modifying the state must be synchronized. That's why I made the State a separate type with a RWLock in it.

Oh, right! I totally forgot about such race conditions. Thanks. Code seems OK. Could you squash commits into one so I can go ahead and merge?

Done.


DocsArchive-<IP_ADDRESS>Auto archive content

Please don't merge this PR until target archive repo PR https://github.com/OPS-E2E-PPE/docs-archive-test-target/pull/461 is merged into live branch.
Auto archive content to https://github.com/OPS-E2E-PPE/docs-archive-test-target.git

Docs Build status updates of commit c7d2ae0:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error] An exception occurred during the build process. Please check the raw log and retry building your content again. If the issue persists, please open a ticket in https://SiteHelp and include the URL of publishing report as well as below information: One or more errors occurred..

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


remove cdms2

cdms2 is going to be abandoned at the end of this year, the recommended replacement is xcdat. We have deprecated our conversion functions in 2023.06.0 (release on June 23) which was less than 6 months ago, but given that cdms2 is already not installable on some architectures it makes sense to remove earlier.
This also appears to allow us to remove the special python=3.11 environment files.

[x] Follow-up to #7876, closes #8419
[x] User visible changes (including notable bug fixes) are documented in whats-new.rst

there's some mentions of cdms2 / CDAT in the faq, so this will have to be updated before merging

@dcherian, there's some groupby failures that are not visible on main (or maybe they will appear soon?). Can you make sense of these?

Should be fixed after a numbagg release.


4.9.1 jar does not exists in maven repo

Good bug reports include a failing test! Writing a test helps you to isolate and describe the problem, and it helps us to fix it fast. Bug reports without a failing test or reproduction steps are likely to be closed.
Here’s an example test to get you started.
https://gist.github.com/swankjesse/981fcae102f513eb13ed
http://maven.owfg.com/nexus/content/groups/public/com/squareup/okhttp3/okhttp/4.9.1/
My Build is failing as it complains that the Jar file is missing from the above location. Please build and publish the Jar to maven repo

This project releases to Maven central at https://repo1.maven.org/maven2/com/squareup/okhttp3/okhttp/4.9.1/. If I had to guess you're looking at a read-through proxy that only contains artifacts once they're requested.


[BUG] Well-known challenge never created.

Describe the bug
cert-manager does not create _acme-challenge.[*] record for the ACME challenge at simply.com by using simply.com implementation of dns-01 webhook .
Expected behavior
_well-know record is created based on challenged received.
Additional context
I have a Kubernetes cluster (K3S) that I have installed by using the quickstart guide and then installed cert-manager by using Helm.
I want to use your webhook, since my DNS provider is Simply.com, but unfortunately I ran into a few troubles:
First off: If I install simply-dns-webhook by calling:
helm install simply-dns-webhook simply-dns-webhook/simply-dns-webhook --version 1.5.4

Then I will run into a permission error when I try to issue a certificate using ClusterIssuer.
It appears the role that reads the secrets from the cert-manager namespace is stored in the default namespace and is therefore not allowed to read secrets from the cert-manager namespace.
This is easily mitigated by adding -n cert-manager to the helm install ... command. :-)
Which leads my to my real issue.
I want to use the simply.com webhook to get certificates issued by Let's Encrypt.
Therefore I have created letsencrypt-staging.yaml with the following content:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email<EMAIL_ADDRESS>    privateKeySecretRef:
      name: letsencrypt-staging-key
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    solvers:
    - dns01:
        webhook:
          groupName: com.github.runnerm.cert-manager-simply-webhook
          solverName: simply-dns-solver
          config:
            secretName: simply-credentials
      selector:
        dnsZones:
        - 'cluster.example.com'
        - '*.cluster.example.com'

To generate my test certificate I have a file called test-certificate.yaml with the following content:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test-certificate
spec:
  dnsNames:
  - test.cluster.example.com
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer

  secretName: test-certificate-tls

I should then be able to get a certificate by first creating the ClusterIssuer with kubectl apply -f letsencrypt-staging.yaml and request for a certificate with kubectl apply -f test-certificate.yaml.
Unfortunately this does not happen because output from kubectl get certs reveals the following result:
NAME               READY   SECRET                 AGE
test-certificate   False   test-certificate-tls   52m

Screenshots/Logs
Output from kubectl events gives:
LAST SEEN   TYPE     REASON               OBJECT                                               MESSAGE
53m         Normal   Issuing              Certificate/test-certificate                         Issuing certificate as Secret does not exist
53m         Normal   Generated            Certificate/test-certificate                         Stored new private key in temporary Secret resource "test-certificate-vjnhx"
53m         Normal   Requested            Certificate/test-certificate                         Created new CertificateRequest resource "test-certificate-1"
53m         Normal   WaitingForApproval   CertificateRequest/test-certificate-1                Not signing CertificateRequest until it is Approved
53m         Normal   WaitingForApproval   CertificateRequest/test-certificate-1                Not signing CertificateRequest until it is Approved
53m         Normal   WaitingForApproval   CertificateRequest/test-certificate-1                Not signing CertificateRequest until it is Approved
53m         Normal   WaitingForApproval   CertificateRequest/test-certificate-1                Not signing CertificateRequest until it is Approved
53m         Normal   WaitingForApproval   CertificateRequest/test-certificate-1                Not signing CertificateRequest until it is Approved
53m         Normal   cert-manager.io      CertificateRequest/test-certificate-1                Certificate request has been approved by cert-manager.io
53m         Normal   OrderCreated         CertificateRequest/test-certificate-1                Created Order resource default/test-certificate-1-1965387138
53m         Normal   OrderPending         CertificateRequest/test-certificate-1                Waiting on certificate issuance from order default/test-certificate-1-1965387138: ""
53m         Normal   Created              Order/test-certificate-1-1965387138                  Created Challenge resource "test-certificate-1-1965387138-1295925723" for domain "test.cluster.example.com"
53m         Normal   Started              Challenge/test-certificate-1-1965387138-1295925723   Challenge scheduled for processing
53m         Normal   Presented            Challenge/test-certificate-1-1965387138-1295925723   Presented challenge using DNS-01 challenge mechanism

Output from kubectl describe challenge test-certificate-1-1965387138-1295925723 reveals amongst others the following lines:
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for DNS-01 challenge propagation: DNS record for "test.cluster.example.com" not yet propagated
  State:       pending
Events:
  Type    Reason     Age   From                     Message
  ----    ------     ----  ----                     -------
  Normal  Started    59m   cert-manager-challenges  Challenge scheduled for processing
  Normal  Presented  59m   cert-manager-challenges  Presented challenge using DNS-01 challenge mechanism

And when I login Simply.com website then I cannot see any records containing the word _acme-challenge, which is used by the ACME protocol.
I can see In your code that you handoff communication with Simply.com to simply-com-client, but I cannot see any debug information about whether not the client is getting called or if there is an authentication error between the client and Simply.com?
Where can I look to debug further?

Will look at it later.😉

Please take a look at the webhook's logs and maybe set it to debug mode. 😉 Also you can provide it here if you want me to take a look.

I am quite new to kubernetes, however a little more digging by running first kubectl get pods -n cert-manager gave me that simply-dns-webhook is running in the pod simply-dns-webhook-6f87bbb8db-t5njw.
From there I delete my old test certificate then ran the test-certificate.yaml again, so it was easier to find in the log.
It gave me the following output from kubectl logs simply-dns-webhook-6f87bbb8db-t5njw -n cert-manager:
I0330 14:21:47.668912       1 handler.go:275] Adding GroupVersion com.github.runnerm.cert-manager-simply-webhook v1alpha1 to ResourceManager
I0330 14:21:47.677105       1 secure_serving.go:213] Serving securely on [::]:443
I0330 14:21:47.677170       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0330 14:21:47.677178       1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController
I0330 14:21:47.677209       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tls/tls.crt::/tls/tls.key"
I0330 14:21:47.677314       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
time="2024-03-30T14:21:47Z" level=info msg="Initializing Simply dns solver"
I0330 14:21:47.677700       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0330 14:21:47.677711       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0330 14:21:47.677733       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0330 14:21:47.677738       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0330 14:21:47.777289       1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController
I0330 14:21:47.778055       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0330 14:21:47.778060       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
time="2024-03-30T14:22:12Z" level=info msg="Presenting challenge for: _acme-challenge.test.cluster.example.com."
time="2024-03-30T14:22:13Z" level=error msg="Failed request, response code:  401"
time="2024-03-30T14:22:13Z" level=error msg="Failed request, response:  401"
time="2024-03-30T17:55:08Z" level=info msg="Cleaning up challenge for: _acme-challenge.test.cluster.example.com."
time="2024-03-30T17:55:08Z" level=error msg="Failed request, response code:  401"
time="2024-03-30T17:55:08Z" level=error msg="Failed request, response code:  401"
time="2024-03-30T17:55:08Z" level=error msg="record(0) have not been cleaned up"
time="2024-03-30T17:55:51Z" level=info msg="Presenting challenge for: _acme-challenge.test.cluster.example.com."
time="2024-03-30T17:55:51Z" level=error msg="Failed request, response code:  401"
time="2024-03-30T17:55:51Z" level=error msg="Failed request, response:  401"

I get that the 401 error code is an authentication error, but against what? :-)
I just checked to be sure that I stored the correct data for my Simply.com account by verifying that the account name in Simply.com's control page is exactly the same as:
kubectl get secrets -n cert-manager simply-credentials -ojson | jq '.data."account-name"' -r | base64 -d

And API key on Simply.com's control page is exactly the same as:
kubectl get secrets -n cert-manager simply-credentials -ojson | jq '.data."api-key"' -r | base64 -d

I suggest the same as in this issue from before: here to do a test run with the config option and then try to debug your secret my guess is that your base64 conversion is wrong. I have been there and many other people did that as well, so no worries.🙂

Strange. The way I converted was something like: echo SECRET_KEY | base64, which should give the correct base64 encoded secret.
But you are into something because when I encoded using https://www.base64encode.org/ I could see 1 char being replaced in account-name and api-key getting shortened by 4 chars.
I guess the devil is in the details...?

I looked up why the encoding would fail for you and the explanation is that, when you do this:  echo SECRET_KEY | base64 you unknowingly add a new line character with the echo command at the end of your string. Which can be mitigated with the -n flag like so  echo -n SECRET_KEY | base64 to produce the correct encoded string.  ;)
Also many services packaged for K8S read configs and secrets once on startup or only deploy in some cases. With this package it is enough to restart the pod.
Hope this helps!  ;)

It did. 👍
You learn something new every day. I have used like two months on and off trying to learn Kubernetes and having issues like not being able to generate certificate can be a bit off putting in doing your own experments.
Particularly since many experiments require a browser and for instance the Chrome browser will at times simply not show the webpage if it is NOT protected by a certificate.
The fact that non-visible characters can affect the code you are generating is very hard to debug.
The last time I had to go debugging in the extreme was when wrote a program for a USB modem many years ago that communicated via SMS.
Text was garbled up, so I did pen and paper debugging on the encoding ... at binary level!
That is the you get when you have string consisting of 7-bit characters converted into a string of 8-bit characters and encoding used BigIndian if I remember right.


JSON Serialisation Library

Should we use Jackson or try a different one?

It is probably worthwhile to test other alternatives to jackson, e.g.:

https://github.com/google/gson
https://github.com/square/moshi
https://github.com/Kotlin/kotlinx.serialization

@felix19350 @monxalo @pedrohh
Gson (https://github.com/waterdog-oss/ktor-template/pull/36)
Gson serialiser already has a converter for ktor.
It handled well on normal kotlin classes but struggled on the generic PageResponse <T> deserialisation, which required a custom deserialiser to be written.
It also did not handle well kotlin's sealed classes.
Jackson
Working fine with everything.
As for sealed classes, instead of writing custom serialisers and deserialisers  there is a annotation for the higher sealed class that registers all subclasses and it is working fine with that, for example, with Koltin's sealed class example:
@JsonTypeInfo(
    use = JsonTypeInfo.Id.NAME,
    include = JsonTypeInfo.As.PROPERTY,
    property = "type")
@JsonSubTypes(
    JsonSubTypes.Type(value = Const::class, name = "const"),
    JsonSubTypes.Type(value = Sum::class, name = "sum"),
    JsonSubTypes.Type(value = NotANumber::class, name = "notanumber")
)
sealed class Expr
data class Const(val number: Double) : Expr()
data class Sum(val e1: Expr, val e2: Expr) : Expr()
class NotANumber : Expr()

Kotlinx Serialization (https://github.com/waterdog-oss/ktor-template/pull/46)
Official kotlin serialiser, already has an implemented Ktor converter. This convert seams to have an open bug (1, 2) that struggles to serialize the PageResponse<T> on the converted itself, which can for now be fixed by converting the class to string before the response being sent to the client.
All classes must be annotated with @Serializable so it can create custom serialisers at compile time and all koltin classes seam to be working well.
It does not automatically create third party classes serialisers, so, custom ones must be written to handled them, like java's class Instant:
object InstantAsStringSerializer : KSerializer<Instant> {
     override val descriptor: SerialDescriptor = PrimitiveSerialDescriptor("Instant", PrimitiveKind.STRING)
     override fun serialize(encoder: Encoder, value: Instant) =
         encoder.encodeString(DateTimeFormatter.ISO_INSTANT.format(value))

     override fun deserialize(decoder: Decoder): Instant = Instant.parse(decoder.decodeString())
 }

There is no centralised place to register these custom converters which means that we have to annotate these third party classes with:
@Serializable
data class TestInstantLongSerialization(@Serializable(with = InstantAsStringSerializer::class) val date: Instant)

or annotate the file that contains the class with @file:UseSerializers(InstantAsStringSerializer::class)
As for sealed classes, they work out of the box when annotated with the serialisable annotation.
It also has support for the nullable and default values.
Moshi (https://github.com/waterdog-oss/ktor-template/pull/52)
Moshi does not have an out of the box converter for ktor, so a custom one must be written. A converter for ktor must be as generic as possible so it can parse any king of object and classes. I run into some problems and could not write a generic enough converter to handle the PageResponse<T> which means that when sending and receiving this type of classes on ktor must be done in string format and serialised and deserialised on the application with custom adapters.
Normal Kotlin classes do not need to be annotated for the serialisation be use Kotlin's reflection, but to auto generate adapters at compile time, kapt plugin must be enabled and all classes annotated with @JsonClass(generateAdapter = true).
Like Kotlinx-serialisation, third party classes such as Date and Instant must have custom serialisers, although they are very simple to write and can all be registered in the same place (when building the Moshi Json converter).
class InstantAdapter {
     @ToJson
     fun toJson(instant: Instant): Long {
         return instant.toEpochMilli()
     }

     @FromJson
     fun fromJson(instant: Long): Instant {
         return Instant.ofEpochMilli(instant)
     }
 }

(...)

Moshi.Builder()
         .add(InstantAdapter())
         .build()

Sealed classes also have to have a custom adapter, however, there isn't a need to write them, we just need to register a custom adapter with all the subclasses:
Moshi.Builder()
         .add(
             PolymorphicJsonAdapterFactory.of(Expr::class.java, "exp")
                 .withSubtype(Const::class.java, "const")
                 .withSubtype(Sum::class.java, "sum")
                 .withSubtype(NotANumber::class.java, "notanumber")
         )
         .build()

It also has support for the nullable and default values.
Comparison and Performance
I found a nice blog and another one that compares them all with some performance testing at the end. (note: versions might not be the latest).
Test are on Android, but I think we can have a nice ideia of the performance of each library.
It seams that Kotlinx-Serialization and Moshi have similar performances when both are generating code, then we have Gson. Although Jackson seams to be the slowest, it is the most complex and complete one.


Adding time inferred for activities.

Changes Unknown when pulling 0dfc483e134f2b2ab303e40b076268d02160361e on V0.5_RELEASE_JS into ** on V0.5_FINAL_WEIJIE**.

Comments:

Can think about changing the name of timeValue to something that is more intuitive

Reviewed. I will try and debug the helpWindow


Avahi doesn't survive container restarts (or other phenomena)

Problem:
When the server is first made with the AVAHI=1 flag, the process list looks something like this, and your computer is automatically discoverable by Finder, thus appearing on the left-hand navigation bar.
I'm starting from accessing the container (docker exec -it netatalk /bin/bash or similar).
# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.1  0.0 147912  6068 ?        Ssl  08:06   0:00 netatalk -d
message+    15  0.0  0.0  42124  2188 ?        Ss   08:06   0:00 dbus-daemon --system
avahi       18  0.0  0.0  32488  3096 ?        S    08:06   0:00 avahi-daemon: running [ubuntu.local]
avahi       19  0.0  0.0  32100   256 ?        S    08:06   0:00 avahi-daemon: chroot helper
root        20  0.4  0.0<PHONE_NUMBER>0 ?        S    08:06   0:00 /usr/sbin/afpd -d -F /etc/afp.conf
root        21  0.0  0.0  67380  5832 ?        S    08:06   0:00 /usr/sbin/cnid_metad -d -F /etc/afp.conf
root        23  0.1  0.0  20224  3204 ?        Ss   08:06   0:00 /bin/bash
root        37  0.0  0.0  17500  2072 ?        R+   08:06   0:00 ps aux

When you reboot the server, or something unexpected happens, the server doesn't appear as before in the left-hand side bar, though you can still mount it by IP address. The process list looks like this:
# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.3  0.0  74180  6232 ?        Ss   08:09   0:00 netatalk -d
root        21  0.8  0.0<PHONE_NUMBER>2 ?        S    08:09   0:00 /usr/sbin/afpd -d -F /etc/afp.conf
root        22  0.0  0.0  67380  5820 ?        S    08:09   0:00 /usr/sbin/cnid_metad -d -F /etc/afp.conf
root        23  0.5  0.0  20232  3240 ?        Ss   08:09   0:00 /bin/bash
root        32  0.0  0.0  17500  2184 ?        R+   08:09   0:00 ps aux

Therefore, the avahi-daemon and dbus services aren't being restarted along with the other services.
Temporary Solution:
Running service dbus start then service avahi-daemon start then kill -1 1 (PID 1 for netatalk -d) resolves the issue until the next boot/error. Running services:
# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.1  0.0 147912  6336 ?        Ssl  09:05   0:00 netatalk -d
root        20  1.0  0.0<PHONE_NUMBER>0 ?        S    09:05   0:00 /usr/sbin/afpd -d -F /etc/afp.conf
root        21  0.0  0.0  67380  5756 ?        S    09:05   0:00 /usr/sbin/cnid_metad -d -F /etc/afp.conf
root        22  0.0  0.0  20248  3260 ?        Ss   09:05   0:00 /bin/bash
message+    43  0.0  0.0  42124  2188 ?        Ss   09:05   0:00 /usr/bin/dbus-daemon --system
avahi       60  0.1  0.0  32364  3200 ?        S    09:06   0:00 avahi-daemon: running [ubuntu.local]
avahi       61  0.0  0.0  32100   256 ?        S    09:06   0:00 avahi-daemon: chroot helper
root        63  0.0  0.0  17500  2132 ?        R+   09:06   0:00 ps aux

20 and 21 are save to kill as well, which makes the two working models rather comparable. Obviously, however, this solution is nonideal for most use cases.
Suggested Solution:
I think it would be wise to enable these services from the Dockerfile or entrypoint. Running update-rc.d dbus defaults along with update-rc.d avahi-daemon defaults gives the following output:
# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.6  0.0  74180  6092 ?        Ss   09:10   0:00 netatalk -d
avahi       18  0.0  0.0      0     0 ?        Z    09:10   0:00 [avahi-daemon] <defunct>
avahi       19  0.0  0.0      0     0 ?        Z    09:10   0:00 [avahi-daemon] <defunct>
root        20  5.3  0.0<PHONE_NUMBER>4 ?        S    09:10   0:00 /usr/sbin/afpd -d -F /etc/afp.conf
root        21  0.0  0.0  67380  5688 ?        S    09:10   0:00 /usr/sbin/cnid_metad -d -F /etc/afp.conf
root        22  1.0  0.0  20248  3264 ?        Ss   09:10   0:00 /bin/bash
root        30  0.0  0.0  17500  2128 ?        R+   09:10   0:00 ps aux

avahi-daemon doesn't start running, try as it may, because dbus never starts. Log entry generated:
Failed to start message bus: The pid file "/var/run/dbus/pid" exists, if the message bus is not running, remove this file
Removing /var/run/dbus/pid and rebooting indeed seems to fix the issue.
# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  1.6  0.0 147912  6100 ?        Ssl  09:19   0:00 netatalk -d
message+    17  0.0  0.0  42124  2256 ?        Ss   09:19   0:00 dbus-daemon --system
avahi       20  0.3  0.0  32488  2980 ?        S    09:19   0:00 avahi-daemon: running [ubuntu.local]
avahi       21  0.0  0.0  32100   256 ?        S    09:19   0:00 avahi-daemon: chroot helper
root        22  4.3  0.0<PHONE_NUMBER>2 ?        S    09:19   0:00 /usr/sbin/afpd -d -F /etc/afp.conf
root        23  0.0  0.0  67380  5832 ?        S    09:19   0:00 /usr/sbin/cnid_metad -d -F /etc/afp.conf
root        25  2.0  0.0  20252  3040 ?        Ss   09:19   0:00 /bin/bash
root        32  0.0  0.0  17500  2076 ?        R+   09:19   0:00 ps aux

This solution isn't permanent since the file will reappear after every boot. In the Dockerfile, I tried using the command service dbus start since it seemed to gracefully remove the pid file when run manually.
# service dbus start
Removing stale PID file /var/run/dbus/pid..
Starting system message bus: dbus.

So far the results are good. Replacing avahi-daemon -D with service avahi-daemon start might be a good idea as well, given these findings.
Conclusion
I suggest using service dbus start in place of  dbus-daemon --system in the Dockerfile to more gracefully handle the destruction of the stale PID file /var/run/dbus/pid, which appears to cause issues with dbus and by extension avahi when restarting the container.

Hi @Wiegraf, thank you for this very detailed bug report. I have not had any free time to work on this and therefore are thankful for your analysis, changes will come when I get some time to work on this.
I'm thinking of removing running Avahi from the core image, since it would be better to configure and run Avahi daemon elsewhere anyways. Then since Gnome Tracker is not run by default either, dbus is not necessary to run either anymore and therefore the default container command should be changed to execute netatalk -d. A bootstrapping entrypoint script could still be a good idea, depending on what bootstrapping features which should be supported.
I have also thought of creating some other images based on this extending to eg. include Avahi daemon, or one to have libpam-ldap installed for LDAP authentication of AFP shares (I use this personally on my home server).