After sorting, part CDS entries missing

Hi alejandrogzi,
Hi Alejandro,
I've been testing the gtfsort tool on the latest axolotl GTF version (AmexT_v47-AmexG_v6.0-DD.gtf) downloaded from Axolotl-omics. However, I encountered an issue with data reduction after sorting.
Issue Description:
Input File: AmexT_v47-AmexG_v6.0-DD.gtf (1,977,265 rows).
Output File: reference_gtf_after_gtfsort.gtf (1,425,753 rows).
Before Sort:

After Sort:

Below content is what my mentor think about which part  CDS is missing:
The gtfsort program is only printing out one CDS entry (in column 3) per transcript.  Here’s an example, for two transcripts of gene AMEX60DD000031.  The first capture is from the original file, the second from the sorted file.  There should be four CDS entries for each transcript, but only the final one is included in each case.  (Seems like perhaps all entries are being written into one location, so only the final one persists, perhaps)
[jhgraber@random testGtfsort]$ grep AMEX60DD000031 AmexT_v47-AmexG_v6.0-DD.gtf | cut -b 1-100
chr10p      ambMex60DD  gene  10258638    10502225    1000  -     .     gene_id "AMEX60DD000031"; gene_name "LOC102279365
chr10p      ambMex60DD  transcript  10258638    10502225    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC
chr10p      ambMex60DD  exon  10258638    10258703    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  exon  10306400    10306498    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  exon  10403547    10403667    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  exon  10404202    10404245    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  exon  10502174    10502225    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  CDS   10306466    10306498    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC1022793
chr10p      ambMex60DD  CDS   10403547    10403667    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC1022793
chr10p      ambMex60DD  CDS   10404202    10404245    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC1022793
chr10p      ambMex60DD  CDS   10502174    10502225    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC1022793
chr10p      ambMex60DD  transcript  10284305    10502225    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC
chr10p      ambMex60DD  exon  10284305    10284358    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  exon  10306400    10306498    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  exon  10403547    10403667    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  exon  10404202    10404245    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  exon  10502174    10502225    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  CDS   10284317    10284358    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC1022793
chr10p      ambMex60DD  CDS   10306400    10306498    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC1022793
chr10p      ambMex60DD  CDS   10403547    10403667    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC1022793
chr10p      ambMex60DD  CDS   10404202    10404245    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC1022793
chr10p      ambMex60DD  CDS   10502174    10502224    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC1022793
[jhgraber@random testGtfsort]$ grep AMEX60DD000031 reference_gtf_after_gtfsort.gtf | cut -b 1-100
chr10p      ambMex60DD  gene  10258638    10502225    1000  -     .     gene_id "AMEX60DD000031"; gene_name "LOC102279365
chr10p      ambMex60DD  transcript  10258638    10502225    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC
chr10p      ambMex60DD  exon  10258638    10258703    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  exon  10306400    10306498    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  exon  10403547    10403667    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  exon  10404202    10404245    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  exon  10502174    10502225    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  CDS   10502174    10502225    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC1022793
chr10p      ambMex60DD  transcript  10284305    10502225    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC
chr10p      ambMex60DD  exon  10284305    10284358    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  exon  10306400    10306498    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  exon  10403547    10403667    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  exon  10404202    10404245    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  exon  10502174    10502225    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC102279
chr10p      ambMex60DD  CDS   10502174    10502224    1000  -     .     gene_id "AMEX60DD000031"; transcript_id "LOC1022793

Hi @dxu104,
Thanks for this! I will get to you once I solve this issue.
Best,
Alejandro

@dxu104,
I just reviewed your issue. gftsort outputs one CDS entry because your gene model has all CDS attribute entries bad formatted:
chr10p  ambMex60DD      CDS     [...]       gene_id "AMEX60DD000031"; transcript_id "LOC102279365 [nr]|GLT1D1 [hs]|AMEX60DD301000031.1";

what I mean by "bad formatted" is that they lack the "exon_number" attribute. In theory, all exon/CDS/start_codon/stop_codon features should have that. gtfsort assumes that this exist and that is how builds the most inner layer.
When you fix the GTF with this information, the result is correct:
chr10p  ambMex60DD      gene    10258638        10502225        1000    -       .       gene_id "AMEX60DD000031"; gene_name "LOC102279365 [nr]|GLT1D1 [hs]";
chr10p  ambMex60DD      transcript      10258638        10502225        1000    -       .       gene_id "AMEX60DD000031"; transcript_id "LOC102279365 [...]
chr10p  ambMex60DD      exon    10258638        10258703        1000    -       .       gene_id "AMEX60DD000031"; transcript_id "LOC102279365 [nr]|GLT1D1 [hs]|AMEX60DD301000031.1"; exon_number "1";
chr10p  ambMex60DD      exon    10306400        10306498        1000    -       .       gene_id "AMEX60DD000031"; transcript_id "LOC102279365 [nr]|GLT1D1 [hs]|AMEX60DD301000031.1"; exon_number "2";
chr10p  ambMex60DD      CDS     10306466        10306498        1000    -       .       gene_id "AMEX60DD000031"; transcript_id "LOC102279365 [nr]|GLT1D1 [hs]|AMEX60DD301000031.1"; exon_number "2";
chr10p  ambMex60DD      exon    10403547        10403667        1000    -       .       gene_id "AMEX60DD000031"; transcript_id "LOC102279365 [nr]|GLT1D1 [hs]|AMEX60DD301000031.1"; exon_number "3";
chr10p  ambMex60DD      CDS     10403547        10403667        1000    -       .       gene_id "AMEX60DD000031"; transcript_id "LOC102279365 [nr]|GLT1D1 [hs]|AMEX60DD301000031.1"; exon_number "3";
chr10p  ambMex60DD      exon    10404202        10404245        1000    -       .       gene_id "AMEX60DD000031"; transcript_id "LOC102279365 [nr]|GLT1D1 [hs]|AMEX60DD301000031.1"; exon_number "4";
chr10p  ambMex60DD      CDS     10404202        10404245        1000    -       .       gene_id "AMEX60DD000031"; transcript_id "LOC102279365 [nr]|GLT1D1 [hs]|AMEX60DD301000031.1"; exon_number "4";
chr10p  ambMex60DD      exon    10502174        10502225        1000    -       .       gene_id "AMEX60DD000031"; transcript_id "LOC102279365 [nr]|GLT1D1 [hs]|AMEX60DD301000031.1"; exon_number "5";
chr10p  ambMex60DD      CDS     10502174        10502225        1000    -       .       gene_id "AMEX60DD000031"; transcript_id "LOC102279365 [nr]|GLT1D1 [hs]|AMEX60DD301000031.1"; exon_number "5";

Hope this helps!
Best,
Alejandro

I think the reason you did not find the exon number is because the meaning of the entire command AmexT_v47-AmexG_v6.0-DD.gtf | cut -b 1-100 is: extract the first 100 bytes of each line from the AmexT_v47-AmexG_v6.0-DD.gtf file.

@dxu104,
I downloaded this file: AmexT_v47-AmexG_v6.0-DD.gtf.gz.
cat AmexT_v47-AmexG_v6.0-DD.gtf.gz | less first lines are:
chr10p  ambMex60DD      gene    313039  315424  1000    +       .       gene_id "AMEX60DD000001"; gene_name "ZFP37 [nr]|ZNF568 [hs]";
chr10p  ambMex60DD      transcript      313039  315424  1000    +       .       gene_id "AMEX60DD000001"; transcript_id "ZFP37 [nr]|ZNF568 [hs]|AMEX60DD301000001.1"; gene_name "ZFP37 [nr]|ZNF568 [hs]|AMEX60DD301000001.1";
chr10p  ambMex60DD      exon    313039  314183  1000    +       .       gene_id "AMEX60DD000001"; transcript_id "ZFP37 [nr]|ZNF568 [hs]|AMEX60DD301000001.1"; exon_number "1";
chr10p  ambMex60DD      exon    315024  315424  1000    +       .       gene_id "AMEX60DD000001"; transcript_id "ZFP37 [nr]|ZNF568 [hs]|AMEX60DD301000001.1"; exon_number "2";
chr10p  ambMex60DD      transcript      313800  315023  1000    +       .       gene_id "AMEX60DD000001"; transcript_id "ZNF79 [hs]|AMEX60DD301000001.2"; gene_name "ZNF79 [hs]|AMEX60DD301000001.2";
chr10p  ambMex60DD      exon    313800  313931  1000    +       .       gene_id "AMEX60DD000001"; transcript_id "ZNF79 [hs]|AMEX60DD301000001.2"; exon_number "1";
chr10p  ambMex60DD      exon    314856  315023  1000    +       .       gene_id "AMEX60DD000001"; transcript_id "ZNF79 [hs]|AMEX60DD301000001.2"; exon_number "2";
chr10p  ambMex60DD      gene    313679  358550  1000    -       .       gene_id "AMEX60DD000002"; gene_name "LOC114595135 [nr]|ZNF268 [hs]";

Basically the same output you have. Note that in that example (yours) there are 0 "CDS" lines.
Running grep -w "CDS" AmexT_v47-AmexG_v6.0-DD.gtf | head -4 and just extracting the CDS lines, outputs:
chr10p	ambMex60DD	CDS	313750	315750	1000	-	.	gene_id "AMEX60DD000002"; transcript_id "LOC114595135 [nr]|ZNF268 [hs]|AMEX60DD201000002.1";
chr10p	ambMex60DD	CDS	509597	512104	1000	+	.	gene_id "AMEX60DD000003"; transcript_id "LOC101953204 [nr]|ZNF850 [hs]|AMEX60DD201000003.1";

Note that here no "exon_number" attributes appear, and as I said in my previous message:

In theory, all exon/CDS/start_codon/stop_codon features should have that

In your example, only "exon" lines appear, this lines are good, the problem you are having is due to bad formatted CDS lines (like the above).
Here is an example of a complete CDS line (+ an exon line for comparison) [this is from a random assembly]:
MDVP01000001.1  ensembl exon    65995   66177   .       -       .       gene_id "ENSCPRG00005000208"; gene_version "1"; transcript_id "ENSCPRT00005000271"; transcript_version "1"; exon_number "2"; gene_name "CHPF2";
MDVP01000001.1  ensembl CDS     65995   66177   .       -       0       gene_id "ENSCPRG00005000208"; gene_version "1"; transcript_id "ENSCPRT00005000271"; transcript_version "1"; exon_number "2"; gene_name "CHPF2";

Note that both ("exon" and "CDS") lines have the "exon_number" attribute. It is not about the command (because I used just cat while inspecting it), I think is because the lack of this attribute (which is necessary for gtfsort). Kind of strange, though, because this is the standard format for a GTF file. Maybe has something to do with the person that build this gene model. Nothing that a few lines of code could fix.
Alejandro

@alejandrogzi, thank you for your help. You are correct; the axolotl omics reference GTF has all CDS attribute entries marked as bad. Consequently, I switched to the Zebrafish reference GTF, which does not have any missing information.
However, I encountered an issue after appending a new transcript found by StringTie, as shown below:
But I appended one new transcript found from StringTie, like
chr9p	StringTie	transcript	145679542	145683546	.	-	.	transcript_id "TCONS_00065193"; gene_id "XLOC_034434"; oId "STRG.28759.1"; cmp_ref "AMEX60DD201053840.1"; class_code "p"; cmp_ref_gene "RNF26_[hs"; tss_id "TSS47892"; chr9p	StringTie	exon	145679542	145683546	.	-	.	transcript_id "TCONS_00065193"; gene_id "XLOC_034434"; exon_number "1";
After using gtfsort, the appended part is missing again. Here is what I observed:
Before sort:

After sort:

Has anyone experienced a similar situation before? Thank you in advance.
Decehng

@dxu104,
Thanks for your interest!
I can download the reference GTF, would you mind to share your StringTie predicted transcripts file? I will find some time to debug that in order to make your experience with gtfsort nicer.
Alejandro

new_transcripts_from_StringTie.gtf.zip
Thank you for your help. The attached is transcripts to be appended and sorted.
Decheng

@jhgraber
Hi Joel,
Thank you for your response. Initially, I used the axolotl reference and appended new transcripts from StringTie. However, after processing with the gtfsort tool, these new transcripts were still missing. Subsequently, I attempted the same process with the zebrafish reference, but encountered the same issue.

Hi @jhgraber,
Nice to meet you! I've been talking with @dxu104 about these appending/sorting bugs you guys are having. In theory, gtfsort should handle that without any problems. I have been re-structuring all gtfsort code this morning, the goal is to publish a new release today implementing a parallel algorithm that makes it run x2 faster (~6s for 1.5GB GTF). Once I finish doing that, and before publishing it, I will take a deep look at your problems to see if something is failing with gtfsort.
I told @dxu104 that I will be writting the nf-module, galaxy-tool and docker (probably) of gtfsort soon, to be easily implemented in any pipeline. On the other hand, the Axolotl problem (person who build that gene model did not include "exon_number" in CDS lines) could be easily fix with a script that updates all those lines. With that, gtfsort will sort the file without any problem.
Once I get the solution I will contact you guys!
Best,
Alejandro

@alejandrogzi
Here's what I did:
Axolotl GTF before sort:

The contents in the blue box are from the reference axolotl GTF, where I fixed the CDS exon number issue.
The contents in the red box are from the stringtie axolotl GTF that I appended.
Axolotl GTF after sort:

The result: The appended stringtie axolotl GTF is missing.
Has anyone experienced a similar situation before? Thank you in advance.
Decehng

Hi @jhgraber and @dxu104,
I have found some workarounds and tips to solve your problems. First, that GTF file needs to be corrected, it does not follow a correct GTF format. Specifically, all the "transcript_id" attributes should only preserve 1 id, not a lot of a ids separated by "|". This is how it looks like:
chr10p  ambMex60DD  [...]  gene_id "AMEX60DD000002"; transcript_id "LOC114595135 [nr]|ZNF268 [hs]|AMEX60DD201000002.1"; gene_name "LOC114595135 [nr]|ZNF268 [hs]|AMEX60DD201000002.1"; [...];

and this is how is expected to look like:
chr10p  ambMex60DD  [...]  gene_id "AMEX60DD000002"; transcript_id "AMEX60DD201000002.1"; gene_name "AMEX60DD201000002.1"; [...];

With 10 lines of code I corrected those lines. However, now the other problem is that all "CDS" lines does not have "exon_numbers". To correct that I used:
./gtfToGenePred AmexT_v47-AmexG_v6.0-DD_fixed.gtf stdout | ./genePredToBed stdin axolotl.bed

to end with a .bed file that looks like this:
chr10p  313038  315424  AMEX60DD301000001.1     0       +       315424  315424  0       2       1145,401,       0,1985,
chr10p  313799  315023  AMEX60DD301000001.2     0       +       315023  315023  0       2       132,168,        0,1056,
chr10p  313678  358550  AMEX60DD201000002.1     0       -       313749  315750  0       2       2660,190,       0,44682,
chr10p  489231  514890  AMEX60DD201000003.1     0       +       509596  512104  0       2       155,6096,       0,19563,

Why? Because then you can use bed2gtf to help you reformat all CDS lines. Note that bed2gtf needs an isoforms file (tab-delimited file with gene to transcript paris). To create the isoforms file and then covert the .bed file to a .gtf file again and sort it I did:
 awk -F'"' -v OFS='\t' '{print $1, $2}' <(awk -F'\t' '{print $9}' <(grep -w "transcript" AmexT_v47-AmexG_v6.0-DD_fixed.gtf) | cut -d '"' -f 2,4) > axolotl.isoforms && bed2gtf --bed axolotl.bed  --isoforms axolotl.transcripts.txt --output axolotl.gtf && gtfsort -i axolotl.gtf -o axolotl.sorted.gtf

where the isoforms file looks like:
AMEX60DD000001	AMEX60DD301000001.1
AMEX60DD000001	AMEX60DD301000001.2
AMEX60DD000002	AMEX60DD201000002.1
AMEX60DD000003	AMEX60DD201000003.1
AMEX60DD000003	AMEX60DD301000003.2

and the output sorted .gtf file now looks like:
C0000568        bed2gtf gene    109816  110194  .       +       .       gene_id "AMEX60DDU001000001";
C0000568        bed2gtf transcript      109816  110194  .       +       .       gene_id "AMEX60DDU001000001"; transcript_id "AMEX60DDU001000001.1";
C0000568        bed2gtf exon    109816  110194  .       +       .       gene_id "AMEX60DDU001000001"; transcript_id "AMEX60DDU001000001.1"; exon_number "0"; exon_id "AMEX60DDU001000001.1.1";
C0000570        bed2gtf gene    39532   39934   .       +       .       gene_id "AMEX60DDU001000002";
C0000570        bed2gtf transcript      39532   39934   .       +       .       gene_id "AMEX60DDU001000002"; transcript_id "AMEX60DDU001000002.1";
C0000570        bed2gtf exon    39532   39624   .       +       .       gene_id "AMEX60DDU001000002"; transcript_id "AMEX60DDU001000002.1"; exon_number "0"; exon_id "AMEX60DDU001000002.1.1";
C0000570        bed2gtf exon    39699   39934   .       +       .       gene_id "AMEX60DDU001000002"; transcript_id "AMEX60DDU001000002.1"; exon_number "1"; exon_id "AMEX60DDU001000002.1.2";
C0000570        bed2gtf gene    128078  128467  .       -       .       gene_id "AMEX60DDU001000003";
C0000570        bed2gtf transcript      128078  128467  .       -       .       gene_id "AMEX60DDU001000003"; transcript_id "AMEX60DDU001000003.1";
C0000570        bed2gtf exon    128078  128467  .       -       .       gene_id "AMEX60DDU001000003"; transcript_id "AMEX60DDU001000003.1"; exon_number "0"; exon_id "AMEX60DDU001000003.1.1";

At this point, the original GTF file 1) has been corrected, 2) converted to .bed, 3) reconverted to .gtf and 4) sorted. Now, the second part is to append all the StringTie predicted transcripts and sort it. Another problem arises here, the predicted transcript GTF file is also bad formatted. I think this is because you used the original GTF file (that is bad-formatted) as input. What I suggest is to use this corrected GTF file: axolotl.gtf.gz; however, there is a "workaround" to make gtfsort work with that StringTie predictions. First, you need to fix that file to look like this (assuming that this information is from a novel transcript and does not need to be appended to an existent transcript):
C0002765        StringTie       transcript      153352  503168  .       +       .       transcript_id "TCONS_00000053"; gene_id "AMEX60DDU001000192"; gene_name "LOC108792107 [nr]|CPT1A [hs]"; oId "STRG.35344.4"; cmp_ref "AMEX60DDU001000192.3"; class_code "j"; tss_id "TSS44";
C0002765        StringTie       exon    153352  153518  .       +       .       transcript_id "TCONS_00000053"; gene_id "AMEX60DDU001000192"; exon_number "1";
C0002765        StringTie       exon    153605  154337  .       +       .       transcript_id "TCONS_00000053"; gene_id "AMEX60DDU001000192"; exon_number "2";
C0002765        StringTie       exon    392201  392376  .       +       .       transcript_id "TCONS_00000053"; gene_id "AMEX60DDU001000192"; exon_number "3";
C0002765        StringTie       exon    412065  412204  .       +       .       transcript_id "TCONS_00000053"; gene_id "AMEX60DDU001000192"; exon_number "4";
C0002765        StringTie       exon    419995  420169  .       +       .       transcript_id "TCONS_00000053"; gene_id "AMEX60DDU001000192"; exon_number "5";
C0002765        StringTie       exon    422863  422964  .       +       .       transcript_id "TCONS_00000053"; gene_id "AMEX60DDU001000192"; exon_number "6";
C0002765        StringTie       exon    503024  503168  .       +       .       transcript_id "TCONS_00000053"; gene_id "AMEX60DDU001000192"; exon_number "7";

Note that the "gene_id" attribute follows the same naming convention used in the big GTF file. This is mandatory because the tool (gtfsort) does not regex lines or guesses names, it expects input GTFs following the basic GTF rules (this means that all the lines belonging to the same gene must have the same gene_id, not "gene_id ABC | LOC1231231 | AMEX12312312").
Once all this have been completed, gtfsort will work correctly:
cat StringTie_transcript_fixed.gtf >> axolotl.gtf && gtfsort -i axolotl.gtf -o axolotl.sorted.gtf

here is a snippet of the output:
C0002765	bed2gtf	gene	153204	520850	.	+	.	gene_id "AMEX60DDU001000192";
C0002765	bed2gtf	transcript	153204	503239	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.1";
C0002765	bed2gtf	exon	153204	154039	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.1"; exon_number "0"; exon_id "AMEX60DDU001000192.1.1";
C0002765	bed2gtf	exon	392201	392376	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.1"; exon_number "1"; exon_id "AMEX60DDU001000192.1.2";
C0002765	bed2gtf	exon	412065	412204	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.1"; exon_number "2"; exon_id "AMEX60DDU001000192.1.3";
C0002765	bed2gtf	exon	419995	420169	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.1"; exon_number "3"; exon_id "AMEX60DDU001000192.1.4";
C0002765	bed2gtf	exon	422863	422964	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.1"; exon_number "4"; exon_id "AMEX60DDU001000192.1.5";
C0002765	bed2gtf	exon	503024	503239	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.1"; exon_number "5"; exon_id "AMEX60DDU001000192.1.6";
C0002765	bed2gtf	transcript	153312	503239	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.2";
C0002765	bed2gtf	exon	153312	154039	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.2"; exon_number "0"; exon_id "AMEX60DDU001000192.2.1";
C0002765	bed2gtf	exon	392201	392376	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.2"; exon_number "1"; exon_id "AMEX60DDU001000192.2.2";
C0002765	bed2gtf	exon	503024	503239	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.2"; exon_number "2"; exon_id "AMEX60DDU001000192.2.3";
C0002765	bed2gtf	transcript	153329	503239	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.3";
C0002765	bed2gtf	exon	153329	154337	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.3"; exon_number "0"; exon_id "AMEX60DDU001000192.3.1";
C0002765	bed2gtf	exon	392201	392376	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.3"; exon_number "1"; exon_id "AMEX60DDU001000192.3.2";
C0002765	bed2gtf	exon	412065	412204	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.3"; exon_number "2"; exon_id "AMEX60DDU001000192.3.3";
C0002765	bed2gtf	exon	419995	420169	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.3"; exon_number "3"; exon_id "AMEX60DDU001000192.3.4";
C0002765	bed2gtf	exon	422863	422964	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.3"; exon_number "4"; exon_id "AMEX60DDU001000192.3.5";
C0002765	bed2gtf	exon	503024	503239	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.3"; exon_number "5"; exon_id "AMEX60DDU001000192.3.6";
C0002765	StringTie	transcript	153352	503168	.	+	.	transcript_id "TCONS_00000053"; gene_id "AMEX60DDU001000192"; gene_name "LOC108792107 [nr]|CPT1A [hs]"; oId "STRG.35344.4"; cmp_ref "AMEX60DDU001000192.3"; class_code "j"; tss_id "TSS44";
C0002765	StringTie	exon	153352	153518	.	+	.	transcript_id "TCONS_00000053"; gene_id "AMEX60DDU001000192"; exon_number "1";
C0002765	StringTie	exon	153605	154337	.	+	.	transcript_id "TCONS_00000053"; gene_id "AMEX60DDU001000192"; exon_number "2";
C0002765	StringTie	exon	392201	392376	.	+	.	transcript_id "TCONS_00000053"; gene_id "AMEX60DDU001000192"; exon_number "3";
C0002765	StringTie	exon	412065	412204	.	+	.	transcript_id "TCONS_00000053"; gene_id "AMEX60DDU001000192"; exon_number "4";
C0002765	StringTie	exon	419995	420169	.	+	.	transcript_id "TCONS_00000053"; gene_id "AMEX60DDU001000192"; exon_number "5";
C0002765	StringTie	exon	422863	422964	.	+	.	transcript_id "TCONS_00000053"; gene_id "AMEX60DDU001000192"; exon_number "6";
C0002765	StringTie	exon	503024	503168	.	+	.	transcript_id "TCONS_00000053"; gene_id "AMEX60DDU001000192"; exon_number "7";
C0002765	bed2gtf	transcript	219148	520850	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.4";
C0002765	bed2gtf	exon	219148	219361	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.4"; exon_number "0"; exon_id "AMEX60DDU001000192.4.1";
C0002765	bed2gtf	exon	392201	392376	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.4"; exon_number "1"; exon_id "AMEX60DDU001000192.4.2";
C0002765	bed2gtf	exon	412065	412204	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.4"; exon_number "2"; exon_id "AMEX60DDU001000192.4.3";
C0002765	bed2gtf	exon	419995	420169	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.4"; exon_number "3"; exon_id "AMEX60DDU001000192.4.4";
C0002765	bed2gtf	exon	422863	422964	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.4"; exon_number "4"; exon_id "AMEX60DDU001000192.4.5";
C0002765	bed2gtf	exon	503024	503161	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.4"; exon_number "5"; exon_id "AMEX60DDU001000192.4.6";
C0002765	bed2gtf	exon	520812	520850	.	+	.	gene_id "AMEX60DDU001000192"; transcript_id "AMEX60DDU001000192.4"; exon_number "6"; exon_id "AMEX60DDU001000192.4.7";
C0002765	StringTie	transcript	219190	503168	.	+	.	transcript_id "TCONS_00000054"; gene_id "AMEX60DDU001000192"; gene_name "LOC115080999 [nr]|CPT1A [hs]"; oId "STRG.35344.5"; cmp_ref "AMEX60DDU001000192.4"; class_code "j"; tss_id "TSS45";
C0002765	StringTie	exon	219190	219361	.	+	.	transcript_id "TCONS_00000054"; gene_id "AMEX60DDU001000192"; exon_number "1";
C0002765	StringTie	exon	392201	392376	.	+	.	transcript_id "TCONS_00000054"; gene_id "AMEX60DDU001000192"; exon_number "2";
C0002765	StringTie	exon	412065	412204	.	+	.	transcript_id "TCONS_00000054"; gene_id "AMEX60DDU001000192"; exon_number "3";
C0002765	StringTie	exon	419995	420169	.	+	.	transcript_id "TCONS_00000054"; gene_id "AMEX60DDU001000192"; exon_number "4";
C0002765	StringTie	exon	422863	422964	.	+	.	transcript_id "TCONS_00000054"; gene_id "AMEX60DDU001000192"; exon_number "5";
C0002765	StringTie	exon	503024	503168	.	+	.	transcript_id "TCONS_00000054"; gene_id "AMEX60DDU001000192"; exon_number "6";

Also note that, for example, predicted transcript lines also are bad-formatted: gene_id attributes should always be at the beginning of the attribute line.
I hope this could help you, guys! Please let me know if there is another doubt or concern.
Best,
Alejandro

Alejandro @alejandrogzi - thanks for all of your effort in tracking down these issues.  The GTF was constructed not by us I appreciate the time it took.  I'd like to know what you are using as your source for defining the format. There are competing definitions available online-- this is a long standing source of frustration in the field, because with different definitions or constraints, the programs written to work with these files frequently assume or require different features.  Most of the confusion, of course, comes from column 9.
After looking at the gmod pages on these formats, I'm inclined to move towards defaulting to trying to update our files as GFF3 instead of GTF/GFF2, since the definition is more clearly stated.  Is your program designed to work with gff3 as well as gtf?
Thanks again for the efforts.

@alejandrogzi Thank you for your effort. It appears to have sorted successfully. As the gffsort tool currently lacks a container or image for integration with the Nextflow pipeline, I prefer to use my gtfinsert subworkflow (https://github.com/dxu104/gtfInsert/blob/main/nextflow/gtfinsert.nf) at the moment. I look forward to your nf-core module soon. Thanks again for all your efforts.

Hi @jhgraber and @dxu104,
Sorry for the delay response.

I'd like to know what you are using as your source for defining the format. There are competing definitions available online-- this is a long standing source of frustration in the field, because with different definitions or constraints, the programs written to work with these files frequently assume or require different features. Most of the confusion, of course, comes from column 9.

Agree. Here I am taking the Ensembl/GENCODE base convention. gtfsort assumes that - at least - a "gene_id", "transcript_id" and "exon_number" appear in that column.

After looking at the gmod pages on these formats, I'm inclined to move towards defaulting to trying to update our files as GFF3 instead of GTF/GFF2, since the definition is more clearly stated. Is your program designed to work with gff3 as well as gtf?

Sadly, no. I can modify to accept gff files, though. Another tool called "gff3sort" already exists that should work. Is not as efficient as gtfsort (takes x3-4 time) and have some problems with the attributes ordering (this is more detailed in the paper, because I considered gff3sort in the general benchmark).

I look forward to your nf-core module soon. Thanks again for all your efforts.

Thanks to you guys, it is very motivating see people using gtfsort. I would let you know when gtfsort nf-module is available.
Best,
Alejandro


📃: Add Isogram Checker:

:red_circle: Title : Isogram Checker
:red_circle: Aim : It checks if given string is isogram or not. An isogram is a word in which the letters occur an equal number of times.
:red_circle: Tech Stack Used : HTML , CSS , JS
Screenshots 📷
N/A

:white_check_mark: To be Mentioned while taking the issue :

Full name : Anshika
What is your participant role? GSSOC

Happy Contributing 🚀
All the best. Enjoy your open source journey ahead. 😎

@Anshika14528 go ahead.


Compile on ARM using ccache causes: invalid option -- 'd'

Hi,
when I try to compile Dlib on ARM running ARMBIAN 5.31 stable Ubuntu 16.04.2 LTS 4.11.6-sunxi.
There seem to be some issues using the compiler specified in the ccache.
It uses the compiler from:
-- Check for working CXX compiler: /usr/bin/ccache
-- Check for working CXX compiler: /usr/bin/ccache -- works

and the error I get is the following:
/usr/bin/ccache: invalid option -- 'd'

which then leads to CMake Error at:
*/dlib/dlib/cmake_utils/set_compiler_specific_options.cmake:31 (message):
  C++11 is required to use dlib, but the version of GCC you are using is too
  old and doesn't support C++11.  You need GCC 4.8 or newer.
Call Stack (most recent call first):

It can be solved by using:
cmake .. -DCMAKE_CXX_COMPILER=g++

I cloned Dlib from Github just yesterday. While I got it to compile I was wondering if somebody experienced a similar issue and if there is a way to get it running without changing the compiler.
Thanks,
Julian

You are just using cmake wrong, this isn't a dlib issue.  See here for instructions: https://stackoverflow.com/questions/1815688/how-to-use-ccache-with-cmake

Thank you for your fast reply. I am not sure if it is a ccache issue. It should work with ccache. It works fine with all my other projects. I think it is rather some kind of incompatibility.
I don't know where the compiler is called with the -- 'd' option. It might be some compiler incompatibility.

Follow the instructions in that stack overflow post.  It tells you how to use ccache with cmake.


OpenTelemetry Integration

Hello,
Thanks for the recent Wiki updates! I came across this section below in Monitoring & Debugging:

Aeron has been designed in an open fashion so that much of its internal state can be observed during operation. By taking an open approach we hope to simplify this challenge. If you have suggestions for improvements then we would love to hear them. We try to accommodate suggestions that will benefit the Aeron user community provided they do not violate the Design Principles.

I was wondering if adding some sort of OpenTelemetry integration would be worthwhile for Aeron. For context, OTeL is supposed to be the vendor-neutral CNCF-supported framework for generating and collecting metrics, traces, and logs.
One key benefit I can imagine is the ability for users to view the cluster as part of their broader application stack within whatever observability vendor they're using. The Aeron cluster could participate in distributed tracing and, for example, make it easy to visualize a UI client making a call to a gRPC gateway which sends a message to the cluster that finally posts something to a Kafka gateway.
I see AeronStat referenced in the wiki and the Cookbook, but I haven't checked if its output is in a format that'd be compatible for the OTeL components currently.
I'm mainly looking to see if there's even any desire to have this integration in the first place, and I'd be more than happy to contribute towards it.
Thanks!

This is not something we wish to support at present. Telemetry can add a significant performance burden and this goes against the design principles of Aeron.


  Let's do it!

Going to host on Github Pages
https://www.davideguida.com/how-to-deploy-blazor-webassembly-on-github-pages-using-github-actions/


SSl auto signed problem and solution

Hello, I have encountered this issue when using self-signed SSL on my local servers, which throws a node error. Upon investigating, I found a solution by adding an option within the code. This is a local fix, and I would like to know if I can make a pull request with this patch, adding an option to enable these lines, which would default to true.
I have never contributed before and wanted to check before making the modification and sending the pull request. I also don't know if the owner of this repository is still active.
Below are the lines I have added locally in the node_modules.

Hi @darthenis, thanks for raising this, PR is welcome. Would be nice to make this configurable instead of hardcoded :)


process managed resource conditions for health checks

How to categorize this PR?

/area robustness
/kind enhancement
/priority normal
What this PR does / why we need it:
This PR adapts the health system component checks, so that now the conditions of the corresponding managed resource is considered as opposed to checking the deployment, daemon set, etc. directly.
Release note:

NONE

/assign @rfranzke @tim-ebert

Thanks for your reviews @rfranzke @tim-ebert. The changes are ready for another round of review.

Can you check the CI failure?


fix in autorun, html instantiated before nodejs

📝 Description

Issue: Url to Github Issue
📋 List of Changes

Please provide a concise list of the main changes.
Change autorun.brs to instantiate the html after the nodejs server is started

🧪 Steps to Test

If uploading the code to brightsign it will now load html correctly

Step 1

Notes to the Reviewer

📸 Screenshots

✔ Dev Complete Checklist

[ ] PR template filled out
[ ] Change is tested by submitter
[ ] PR follows all linting and coding standards
[ ] Github Issue exists (if applicable)
[ ] Team member has been assigned
[ ] At least one commit message is in Conventional Commit format

First time making a pull request, I realise I should have ran yarn install to get the pr template to fill out. Should have run linting but it's just lines being moved;

Confirmed that I had the same issue testing a blank app until I moved the same lines


Guard against DupSort change sets

Silkworm does not yet support https://github.com/ledgerwatch/turbo-geth/pull/1342

Codecov Report

Merging #98 (59c74d7) into master (44d5b55) will decrease coverage by 0.07%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master      #98      +/-   ##
==========================================
- Coverage   80.79%   80.72%   -0.08%
==========================================
  Files          52       52
  Lines        4104     4108       +4
==========================================
  Hits         3316     3316
- Misses        788      792       +4

Impacted Files
Coverage Δ

silkworm/db/access_layer.cpp
23.84% <0.00%> (-0.41%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 44d5b55...e64397f. Read the comment docs.


Integrated gogs webhook deletion

Fixes #1603

awesome, thanks for working on this one! I think we might need to mirror this after the GitHub hook removal as well. Notice we have logic to find the matching hook:
https://github.com/drone/drone/blob/master/remote/github/github.go#L238
the matching hooks function parses the URL to remove the token from the url query string. Otherwise it won't properly match the link that is passed in
https://github.com/drone/drone/blob/master/remote/github/github.go#L314
EDIT: I'm looking at your implementation and it does look like the HasPrefix might work fine. Perhaps for consistency we can break into a helper function similar to the GitHub implementation?

I will do it the same like GitHub. And than I have to fix the tests, looks like there have been something broken because of the client update.

@bradrydzewski I don't get the tests running correctly but at least the deletion of webhooks works now. Maybe you can do a small addition afterwards to get the tests correct since you are deeper in it?

what issues are you seeing w/ the unit tests? maybe I can help

@tboerger could you please take a look at this again?

@tboerger could you please take a look at this again?

A little bit busy at the moment, but I try to come back ASAP.

any progress in this pr?

Sorry, I forgot about this PR.

does this needs to be merged before gitea driver?

Closing this in favor of go-scm


Publish Volar to Open VSX extension gallery

Hi, thank you for your incredibly-fast Vue extension.
I use Codium as an open source alternative to Visual Source Code (that is not open source and sends telemetry (by default) without explicit user consent, which is not compliant with European data protection laws).
Codium is not allowed to install extensions from the VS Code Marketplace. Hopefully, Codium uses a community driven, open source extension gallery: Open VSX (open-vsx.org).
Open VSX already distributes 1151 VSCode extensions:

https://open-vsx.org/extension/antfu/vite
https://open-vsx.org/extension/octref/vetur
…

To publish Volar to Open VSX, please follow this steps:
https://github.com/eclipse/openvsx/wiki/Publishing-Extensions

Thanks for your suggestion, but sorry I don't have enough energy to maintain more releases.
If anyone needs it, you can publish it yourself.

See #550.


Have a way for local IPFS communities to organize

We need a clear way for IPFS community to organize, or at least some guidelines, so that people that want to host an IPFS meetup or be part of one can find it and be informed of when it is happening.
We've looked into attending.io and meetup.com, both have pros and cons, but none lets us create a succinct and clean experience.
@RichardLitt you have been developing a lot of the materials, calendars and resources for the IPFS community, would you be able to make this happen? I feel that a guideline plus a community map (https://github.com/ipfs/community/issues/39) added on ipfs.io could be something really good.

(new idea) we could have meetup folder here and then meetup/lisbon, meetup/portland, etc and then have a README with orgs, location and other links + one markdown file per meetup with resources of that meetup

:+1: on a meetup folder here as a first step.

We could have a meetup folder here with assets used universally, and suggest that each chapter build their own MeetUp group. We can sync their meetups to the IPFS community calendar, and anyone who wants to can be pointed there. The issue with this is that meetup is a separate, siloed service, and it's common for assets to be used or not use, and for a community to break down if the person leading it leads.
I think we should do something similar to nodeschool or Papers-We-Love. I opened an issue on PapersWeLove here and helped them switch to the NodeSchool model; a lot of what I said there applies here, so I'll copy it over (replace Papers We Love with IPFS, and imagine that we already have a few chapters up and running):

What

I'm suggesting we adopt a NodeSchool like chapter system, where each chapter gets their own repository in order to help organizers coordinate efforts and cut down on time and resources spent figuring out management.

Why

Help Organizers coordinate
I'm currently looking into how to host a Papers We Love in Boston. It's not difficult to do - find a speaker, find a space, let people know it exists, set up a MeetUp group. However, as I'm working with a coorganizer, we have to figure out a way to talk to each other outside of Twitter direct messages. I'd like to be able to use a GitHub repository to use issues for discussion of how to do this, largely because that would allow a way for community members to vote on papers closer to the source, because it's what I'm used to doing with NodeSchool and other organizations, and because it doesn't use Meetups rather poor tools for discussion.

Cross-pollinate ideas about how to coordinate
It would be good to have a standard way of communicating with coorganizers so that chapters can learn from each other how to better organize, too.

People are already doing this
Some chapters already have their own GitHub repositories, or organizations: papers-we-love-rtp, Munich, Bangalore, Hyderabad. Some individuals set up issues for their talks, too, like these in Chicago, and London. I think keeping this in the PapersWeLove organization would be more useful.

Chapter histories could be maintained
Finally, Boston's PapersWeLove chapter used to exist, and now it doesn't. Without some sort of history - which would have happened if there were closed issues on the GitHub repo, instead of just a deleted Meetup group - I don't know why it failed to continue, what caused the chapter to fall apart, and how those issues might be alleviated (or indeed, not. Maybe Boston isn't a good place for Papers We Love.).

How
Set up a repo to help organizers, like this nodeschool one. Create a team called chapter-organizers, and give members access to edit their new chapter repo, e.g. https://github.com/paperswelove/boston.

I think this approach works particularly well for NodeSchool, and would work very well for IPFS, especially with our github-issues-all-the-things mindset. I'm happy to write up a README about proper protocol for this.
Also :+1: for the map.

@RichardLitt im all for that. do we want this in a separate org? or the same? i know people like separate orgs.
it's also nice to be able to say:
ipfs/nyc
ipfs/boston
ipfs/pdx
ipfs/paris
ipfs/lx
ipfs/london

which we could get on the same ipfs org. otherwise:
ipfs-meetups/nyc
ipfs-meetups/boston
ipfs-meetups/pdx
ipfs-meetups/paris
ipfs-meetups/lx
ipfs-meetups/london

or something...

(i.e. id prefer same org, but others may kill me if that goes on too long)

I think the same org. There's really no cost, at this point, to having more repos. The only issue is that we need to be very clear about teams for people who can contribute to repos - someone deleted all of NodeSchool the other month, which was super awkward.

someone deleted all of NodeSchool the other month, which was super awkward.

yeah this is why i want to hand out permissions granularly. oh trust!

@skylarnorris have you stumbled upon this issue before?

@diasdavid had not as I am not subscribed to this repo but @RichardLitt is handling most Meetup stuff now.

@kumavis would it have been easier to organize your meetup with a separate organization, or with a separate ipfs/seattle repo?

I am going to veto making a new organization or repositories for meetups, for now. What we have seems to be working. We can revisit this later. For now, communities can open an issue, read the guide in meetups/, and talk to me if they need help with anything. I'll keep an eye on how people organize in the future, and may come back to this repo, but I think what we currently have is working well enough to not add the overhead of a dozen repos or another org.


Question: Can people use their account to run the scratchattach python code?

Example
import scratchattach
# Original
session = scratch3.login("owner","ownerpassword")

** Replacement **
import scratchattach
# New
session = scratch3.login("newuser","otherpassword")

yes

Oh ok (Wait you have a github account??)


Didnt delete not used versions files

For each update, increase my app size, without I create new files.
There is a way to delete not used versions files?

@alamops Right now there is no way to delete them directly, but I did have plans to add more support around the local versions. I will update with more info as it becomes available.

This would be a great feature!

+1 This is the reason we cannot start using the deploy service right now

@ericb Sorry to pester, but is there any word on this issue?

@TombolaShepless Sorry! I must have missed your original comment. So this will definitely be addressed, but I don't have a timeline for you at the moment. We're gearing up to release some updates to the service clients (including deploys). Once that happens I should be able to swing around to this issue.

+1 any updates on this ?

@kcedric This is an outdated and unsupported library. Please check out the cloud-client and our latest docs for more information.


Remove DependencyInfoBlock for F-Droid

Description
We find that there is a DependencyInfoBlock in your APK.
It's a Signing block added by AGP and encrypted with the Google public key so it can't be read by anyone else except Google. You can read more about it here and here.
While this was added a while ago, we were only enforcing it for new apps, and recently we started scanning updates too.
Could you please disable it with the following code?
android {
    dependenciesInfo {
        // Disables dependency metadata when building APKs.
        includeInApk = false
        // Disables dependency metadata when building Android App Bundles.
        includeInBundle = false
    }
}

Thanks!
Acknowledgements

[X] I have searched the open and closed issues and this is NOT a duplicate.

I had no idea that was even a thing.
I released a new build under the release tag v2.0.2-1 without the dependency info block.
Let me know if there are any other issues.

Thanks!


UnsafeDestroyRange may cause panic for a concurrent generating snap when Titan enabled

When Titan enabled, after doing deletefiles(deleteblobfiles), iterating on the deleted range may encounter MissingBlobFile error. So we set a special key_only for Titan when doing the scan-delete.
Whereas, if there is a concurrent generating snap process, it will encounter the MissingBlobFile error and panic in final after trying 5 times. So it's better to delete blob files in the range after TiKV finishing scan-delete on that range.

How could it have concurrent snapshoting, after delete range issued?

Cause UnsafeDestroyRange doesn't go through the Raft, whereas called by TiDB directly. So it's possible that there is a concurrent snapshot generation issued by Raft.


Add num_workers argument to run()

Some tasks might have to run in serial, this change allowes a user to
change the number of workers for a specific task without changing the
value of Brigade.num_workers

Sorry, bad habits.. Fixed now.


[🔨for snippets]: add new mode to display snippets items as a list in the header

The main idea is to add ability to change how the list of snippets items look. We can add specific classs or attribute to change dropdown snippets menu (displays by default) to the list of items

Resolved on 2.0.0-beta.2


SAM version?

Hi I love Serverless but corporate clients are generally hesitant to use third party elements in the core flow. SAM has come along a bit in recent times -- have you thought of doing a SAM version?

Hmm yeah thats a good point on the corporate clients part. We didn't have plans but I'll have a look at SAM and see how far along they are.
Contributions are welcome though!

I’ll definitely contribute if I get far enough in this direction. This stack has been unbelievably good. When I start earning myself I’ll definitely sponsor it.

@jayair how would you prefer to see contributions in regards to SAM?

As an addition to every part when it references serverless, it would have underneath that how to do with sam`?
A copy of the whole document structure with serverless parts replaced by sam?
Use variables in the static site, so the choice of serverless and sam can be changed at site build-time (e.g. auto-framework=sam)
Something else?

Personally I think the juxtaposition of individual steps would be really good. That way people can really see the benefit of serverless.com as well.

@craigbarrau I think it's really hard to do each of the parts side-by-side and keeping them updated over time.
It might be easier to create a separate section that shows how to build out the same backend but with SAM. And then allowing the frontend section to use either backend.
We might even change the design of the homepage to indicate that you can use either backend.

Thanks @jayair. I recently went through the tutorials from the beginning and used sam instead. I think you are right that a separate section would probably be the easiest way to express the alternative option for sam at least in the short-term.
I've just pushed my sam version of the API here and I'd be keen for feedback. I've also got some detailed notes on the gotchas and limitations compared to serverless as well as some of the benefits I found for sam over the serverless framework.
If you are seriously interested in bringing SAM support into the guide, I'd be more than happy to help as a collaborator or the primary author.
As an aside, and in reply to @boxabirds comment, the tools were a lot less different in terms of functionality and syntax than I thought they would be. One of the main things that I noticed was that sam doesn't try to own the build space as much as serverless. That wasn't a big deal as I learned that the magic of the serverless-bundle plugin seems to be easy to do with plain ol' webpack + npm.

@craigbarrau amazing! This is really exciting work, thank you! Honestly as a newbie in this space, the tutorial was extremely well done and with sam it should become a mainstream seminal text.
Really, thank you, on behalf of all the budding AWS developers now and in the future.

@m0un10 Awesome! Thank you for this.

I've also got some detailed notes on the gotchas and limitations

Do you mind sharing these? Either in your repo README or here?
I'd like to go through your repo and try setting it up myself. And then put together a rough list of the steps and figure out how we can work together to get this done. Thanks again for offering to help.

I've got a gist here with my original notes. It also has some details around what is different for the first few chapters when SAM is used instead of the Serverless framework. I didn't have a chance to describe the differences in as much detail for the later chapters but it becomes pretty self explanatory and the full set of changes are available in the sam fork.

Perfect. I'll have a look. Thanks.

I didn't realize SAM requires Docker installed and running. Is there a way around this?

This is how SAM simulates the function calls locally when you run a test (e.g. sam local invoke or sam local start-api). It pulls down images to support whatever runtime is defined in the template.yaml. If you were simply wanting to build/deploy I don't think Docker would be needed. The deploy step is essentially a convenient wrapper around aws cloudformation deploy.

If you need to build native binaries I think you still need Docker though, right?


Support ESM with @remix-run/dev

What version of Remix are you using?
1.14.3
Are all your remix dependencies & dev-dependencies using the same version?

[X] Yes

Steps to Reproduce

Have an ESM TS project
Try importing from @remix-run/dev/server-build
TS throws error: Cannot find module '@remix-run/dev/server-build' or its corresponding type declarations.

Expected Behavior
@remix-run/dev should be an ESM package and thus have an explicitly defined set of entrypoints.
This fixes this issue (but then breaks the dev CLI):
"type": "module",
  "exports": {
    ".": {
      "import": "./dist/index.js"
    },
    "./server-build": {
      "import": "./server-build.js",
      "types": "./server-build.d.ts"
    }
  },

Actual Behavior
It isn't ESM.

Possibly related https://github.com/remix-run/remix/pull/4676

Is this still an issue? I am using 1.18.0 with ESM and it works as expected.

Not reproducible in 1.19. For example typechecking passes for the arc template (note: skipLibCheck: true must be set in the tsconfig.json to work around other unrelated problems)


jwt not active

  minecraft-protocol New connection:  <IP_ADDRESS>/54828 +43s
  minecraft-protocol * new status 2 +0ms
  minecraft-protocol S -> request_network_settings {"client_protocol":582} +0ms
  minecraft-protocol S <- network_settings {
  compression_threshold: 512,
  compression_algorithm: 'deflate',
  client_throttle: false,
  client_throttle_threshold: 0,
  client_throttle_scalar: 0
} +1ms
  minecraft-protocol S -> login {"protocol_version":582,"tokens":{"identity":"{\"chain\":[\"eyJhbGciOiJFUzM4NCIsIng1dSI6Ik1IWXdFQVlIS29aSXpqMENBUVlGSzRFRUFDSURZZ0FFWFpuUG8xTWorTHF2WGtnUUFDY0gwd3RDTmlnMnRmQTlCYjlaWFMrR0oxVlNkbEEzL2Fh +107ms
  minecraft-protocol Verified client with mojang key ******************* +0ms
  minecraft-protocol undefined NotBeforeError: jwt not active
    at E:\Codes\Javascript\GreenFrogMCBE\node_modules\jsonwebtoken\verify.js:181:21
    at getSecret (E:\Codes\Javascript\GreenFrogMCBE\node_modules\jsonwebtoken\verify.js:97:14)
    at module.exports [as verify] (E:\Codes\Javascript\GreenFrogMCBE\node_modules\jsonwebtoken\verify.js:101:10)
    at verifyAuth (E:\Codes\Javascript\GreenFrogMCBE\node_modules\frog-protocol\src\handshake\loginVerify.js:26:27)
    at client.decodeLoginJWT (E:\Codes\Javascript\GreenFrogMCBE\node_modules\frog-protocol\src\handshake\loginVerify.js:54:27)
    at Player.onLogin (E:\Codes\Javascript\GreenFrogMCBE\node_modules\frog-protocol\src\serverPlayer.js:82:46)
    at Player.readPacket (E:\Codes\Javascript\GreenFrogMCBE\node_modules\frog-protocol\src\serverPlayer.js:168:14)
    at Player.handle (E:\Codes\Javascript\GreenFrogMCBE\node_modules\frog-protocol\src\connection.js:172:16)
    at E:\Codes\Javascript\GreenFrogMCBE\node_modules\frog-protocol\src\server.js:93:35
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11) {
  date: 2023-05-04T07:01:59.000Z
} +27ms
  minecraft-protocol S <- disconnect {
  hide_disconnect_screen: false,
  message: 'Server authentication error'
} +10ms
  minecraft-protocol Kicked  <IP_ADDRESS>/54828 Server authentication error +201ms
  minecraft-protocol * new status 0 +313ms
  minecraft-protocol Connection closed:  <IP_ADDRESS>/54828 undefined +181ms

i tried deleting node_modules but still receiving

Why do you believe this is an issue with bedrock protocol? You need to provide more detailed information on this issue including steps to reproduce.

just created an server with offline: false and shows this

i'm able to connect to others server so not mc issue

Can't reproduce on 1.19.81, please provide more detailed steps to reproduce.

Delete your node_modules and package-lock.json then, do npm install bedrock-protocol, restart your game and try again.
If you can reproduce this issue, reopen and provide the all example code you used to replicate this issue including the steps.

initialize an new project
install bedrock-protocol
Use basicServer.js example code with debug on
Done

Can't seem to reproduce this either on latest release.

there's probably "an reason" for this issue tho

i fixed the issue, it was due to the ntp was not synced. I will open an pr for this silly issues...
it took me 3 days for fix this issue

This could be added to the FAQ


Add documentation examples for promises + async/await

In the "Promises and Callbacks" section, I mention that the rest of the README will use callbacks. We should provide a Promise-based version of the documentation that uses promises in the examples, and link to that README version from the "Promises and Callbacks" section.

Updated for new async/await usage. The ES6 usage should probably be the default in the docs, with links to promise and callback usage in a /docs/ directory.