Takes several re-compilations to reuse 100% of generated bc+obj files

Hello,
Testing platform:

Ubuntu 16.10 (amd64)
Crystal 0.20.5
LLVM 3.8.1
Local build: Crystal 0.20.5+65 [dff9b2e] (2017-02-10)

Scenario:
Compiled Crystal with 0.20.5 and then proceeded to clear the cache (~/.cache/crystal).
Subsequent builds start reusing bc+obj files but it takes several attempts until 100% of the files are reused.
The following is the console output and the commands used after the first build:
$ rm -rf ~/.cache/crystal

$ touch src/compiler/crystal.cr

$ make stats=1
Using /usr/bin/llvm-config-3.8 [version=3.8.1]
CRYSTAL_CONFIG_PATH=`pwd`/src ./bin/crystal build --stats  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
Using compiled compiler at .build/crystal
Parse:                             00:00:00.0001633 (   0.19MB)
Semantic (top level):              00:00:00.9600884 (  83.18MB)
Semantic (new):                    00:00:00.0049334 (  83.18MB)
Semantic (type declarations):      00:00:00.1009797 (  91.18MB)
Semantic (abstract def check):     00:00:00.0029139 (  91.18MB)
Semantic (cvars initializers):     00:00:00.0222071 (  91.18MB)
Semantic (ivars initializers):     00:00:00.0852713 (  99.18MB)
Semantic (main):                   00:00:14.0652180 ( 867.25MB)
Semantic (cleanup):                00:00:00.0015672 ( 867.25MB)
Semantic (recursive struct check): 00:00:00.0019160 ( 867.25MB)
Codegen (crystal):                 00:00:08.8524483 ( 876.12MB)
Codegen (bc+obj):                  00:00:10.0868879 ( 876.12MB)
Codegen (linking):                 00:00:04.0767550 ( 876.12MB)

Macro runs:
 - /home/luis/code/crystal-lang/crystal/src/ecr/process.cr: 00:00:03.4635985

Codegen (bc+obj):
 - no previous .o files were reused

As expected on first run after clean cache, no bc file was reused.
Second attempt:
$ touch src/compiler/crystal.cr

$ make stats=1
Using /usr/bin/llvm-config-3.8 [version=3.8.1]
CRYSTAL_CONFIG_PATH=`pwd`/src ./bin/crystal build --stats  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
Using compiled compiler at .build/crystal
Parse:                             00:00:00.0001430 (   0.19MB)
Semantic (top level):              00:00:00.9827208 (  82.74MB)
Semantic (new):                    00:00:00.0044674 (  82.74MB)
Semantic (type declarations):      00:00:00.1067661 (  90.74MB)
Semantic (abstract def check):     00:00:00.0030296 (  90.74MB)
Semantic (cvars initializers):     00:00:00.0219456 (  90.74MB)
Semantic (ivars initializers):     00:00:00.1206257 (  90.74MB)
Semantic (main):                   00:00:10.9776052 ( 826.74MB)
Semantic (cleanup):                00:00:00.0015307 ( 826.74MB)
Semantic (recursive struct check): 00:00:00.0018696 ( 826.74MB)
Codegen (crystal):                 00:00:10.6871095 ( 874.93MB)
Codegen (bc+obj):                  00:00:08.8601001 ( 874.93MB)
Codegen (linking):                 00:00:03.7990048 ( 874.93MB)

Macro runs:
 - /home/luis/code/crystal-lang/crystal/src/ecr/process.cr: reused previous compilation (00:00:00.0080372)

Codegen (bc+obj):
 - 247/1186 .o files were reused

Only 247 of 1186 were reused.
Third attempt:
$ touch src/compiler/crystal.cr

$ make stats=1
Using /usr/bin/llvm-config-3.8 [version=3.8.1]
CRYSTAL_CONFIG_PATH=`pwd`/src ./bin/crystal build --stats  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
Using compiled compiler at .build/crystal
Parse:                             00:00:00.0002702 (   0.19MB)
Semantic (top level):              00:00:00.9722807 (  82.74MB)
Semantic (new):                    00:00:00.0044383 (  82.74MB)
Semantic (type declarations):      00:00:00.1029496 (  90.74MB)
Semantic (abstract def check):     00:00:00.0028177 (  90.74MB)
Semantic (cvars initializers):     00:00:00.0214620 (  90.74MB)
Semantic (ivars initializers):     00:00:00.1101186 (  90.74MB)
Semantic (main):                   00:00:10.2959104 ( 826.74MB)
Semantic (cleanup):                00:00:00.0015995 ( 826.74MB)
Semantic (recursive struct check): 00:00:00.0018579 ( 826.74MB)
Codegen (crystal):                 00:00:09.0105445 ( 874.93MB)
Codegen (bc+obj):                  00:00:01.5306878 ( 874.93MB)
Codegen (linking):                 00:00:03.8219490 ( 874.93MB)

Macro runs:
 - /home/luis/code/crystal-lang/crystal/src/ecr/process.cr: reused previous compilation (00:00:00.0082830)

Codegen (bc+obj):
 - 1057/1186 .o files were reused

Number goes up but still isn't 100%
Fourth attempt:
$ touch src/compiler/crystal.cr

$ make stats=1
Using /usr/bin/llvm-config-3.8 [version=3.8.1]
CRYSTAL_CONFIG_PATH=`pwd`/src ./bin/crystal build --stats  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
Using compiled compiler at .build/crystal
Parse:                             00:00:00.0001631 (   0.19MB)
Semantic (top level):              00:00:01.0759624 (  82.74MB)
Semantic (new):                    00:00:00.0049699 (  82.74MB)
Semantic (type declarations):      00:00:00.1093231 (  90.74MB)
Semantic (abstract def check):     00:00:00.0032196 (  90.74MB)
Semantic (cvars initializers):     00:00:00.0232578 (  90.74MB)
Semantic (ivars initializers):     00:00:00.2513325 (  90.80MB)
Semantic (main):                   00:00:10.9166493 ( 826.80MB)
Semantic (cleanup):                00:00:00.0015945 ( 826.80MB)
Semantic (recursive struct check): 00:00:00.0020521 ( 826.80MB)
Codegen (crystal):                 00:00:10.0011559 ( 875.68MB)
Codegen (bc+obj):                  00:00:00.8647753 ( 875.68MB)
Codegen (linking):                 00:00:03.9348918 ( 875.68MB)

Macro runs:
 - /home/luis/code/crystal-lang/crystal/src/ecr/process.cr: reused previous compilation (00:00:00.0080509)

Codegen (bc+obj):
 - all previous .o files were reused

This time (and any subsequent one) reuses 100% of the generated bc files.
I've tried waiting several minutes between builds considering it might be a timing issue, but obtained the same result. Also considered might be the date or a macro call (like getting the git commit and version tag), but cannot confirm.
I was able to reproduce this consistently multiple times:

1st build (after rm): no reused bc+obj
2nd build: reused 247/1186
3rd build: reused 1057/1186
4th build: reused 100%
5th build: reused 100%

Thanks for making the compile process fast, which help me spot this issue :smile:
Let me know what else should I look at that might help.
Thank you :heart: :heart: :heart:

YES!! I noticed this too. It seems to happen with .o files that have the Thread type inside them. I have no idea what's going on... would be interesting to find out :-)

I can reproduce 247, 1061, 100% here on arch linux (llvm 3.9). How strange!

You can also reproduce this with an empty file. I'm getting:
 - no previous .o files were reused
 - 118/121 .o files were reused
 - all previous .o files were reused

So using an empty file should be easier for finding out what's wrong.

@asterite maybe we can get an extended list of which bc files were reused and then diff between to determine which files might the be ones changing between builds.

The comment:

I've tried waiting several minutes between builds considering it might be a timing issue...

brings to my mind a problem seen in some older versions of make.  I have no idea what crystal is doing here, so let me phrase it in terms of make.  Given:
bthing : athing
	compile -o bthing

clib : bthing
	buildlib -o clib bthing

You then touch athing, and type make clib.
The problem comes up when the compile and buildlib steps can execute in less than a second.  You'd end up with bthing and clib with the exact same timestamp (which was saved in seconds at the time).  And this meant that the next time make was run, it thought that clib needed to be rebuilt even though it did not.

@drosehn don't think make is the issue, as @asterite mentioned this also happens with an empty file:
$ touch empty.cr

$ rm -rf ~/.cache/crystal

$ bin/crystal build --stats empty.cr
Using compiled compiler at .build/crystal
...
Codegen (bc+obj):
 - no previous .o files were reused

$ bin/crystal build --stats empty.cr
Using compiled compiler at .build/crystal
...
Codegen (bc+obj):
 - 106/118 .o files were reused

$ bin/crystal build --stats empty.cr
Using compiled compiler at .build/crystal
...
Codegen (bc+obj):
 - all previous .o files were reused

Also Crystal's Makefile takes in consideration all the timestamps for all the source files against the generated executable, including libcrystal.a, so that shouldn't be the issue.

I just added some code to show which units are not reused. For an empty file I'm getting:
These modules were not reused:
 - Thread (T-hread.bc)
 - Set(Thread) (S-et40T-hread41.bc)
 - Hash(Thread, Nil) (H-ash40T-hread4432N-il41.bc)

The strange thing is that if I emit LLVM IR for these (with --ll) then there's no difference. Somehow a different .bc is generated for the same .ll...

I also went from two different .bc files for the same module, turned them into .ll (with llvm-dis) and there's no difference. I think this promotes this to an LLVM bug. Or maybe there's a reason why they generate different .bc files for the same LLVM IR. The strange thing is that it later stabilizes, so I'm not entirely sure what I say here is correct.

I got a slightly number of different for an empty file:
(pulled and compiled from master after your change)
Codegen (bc+obj):
 - 106/118 .o files were reused

These modules were not reused:
 - Thread (T-hread.bc)
 - Set(Thread) (S-et40T-hread41.bc)
 - Hash(Thread, Nil) (H-ash40T-hread4432N-il41.bc)
 - Pointer(Hash::Entry(Thread, Nil) | Nil) (P-ointer40H-ash58-5c8fd5776cf345ec94b91bf7a5f7e50f.bc)
 - Array(Int32) (A-rray40I-nt3241.bc)
 - String::Builder (S-tring5858B-uilder.bc)
 - String (S-tring.bc)
 - GC (G-C-.bc)
 - Slice(T) (S-lice40T-41.bc)
 - IO::Encoder (I-O-5858E-ncoder.bc)
 - Iconv (I-conv.bc)
 - Math (M-ath.bc)

What is also interesting is that after removal of the cache, the second build displays some other random modules that weren't reused:
Codegen (bc+obj):
 - 106/118 .o files were reused

These modules were not reused:
 - Thread (T-hread.bc)
 - Set(Thread) (S-et40T-hread41.bc)
 - Hash(Thread, Nil) (H-ash40T-hread4432N-il41.bc)
 - Pointer(Hash::Entry(Thread, Nil) | Nil) (P-ointer40H-ash58-5c8fd5776cf345ec94b91bf7a5f7e50f.bc)
 - Array(Int32) (A-rray40I-nt3241.bc)
 - String::Builder (S-tring5858B-uilder.bc)
 - String (S-tring.bc)
 - GC (G-C-.bc)
 - Slice(T) (S-lice40T-41.bc)
 - IO::Encoder (I-O-5858E-ncoder.bc)
 - Iconv (I-conv.bc)
 - Math (M-ath.bc)
 - Debug::DWARF::LineNumbers::Sequence (D-ebug5858D-W-A-R-F-5858L-ineN-umbers5858S-equence.bc)
 - Pointer(Hash::Entry(Signal, Proc(Signal, Nil)) | Nil) (P-ointer40H-ash58-ac8d12c9042874d67c1e049ddde35350.bc)

And this other time:
Codegen (bc+obj):
 - 106/118 .o files were reused

These modules were not reused:
 - Thread (T-hread.bc)
 - Set(Thread) (S-et40T-hread41.bc)
 - Hash(Thread, Nil) (H-ash40T-hread4432N-il41.bc)
 - Pointer(Hash::Entry(Thread, Nil) | Nil) (P-ointer40H-ash58-5c8fd5776cf345ec94b91bf7a5f7e50f.bc)
 - Array(Int32) (A-rray40I-nt3241.bc)
 - String::Builder (S-tring5858B-uilder.bc)
 - String (S-tring.bc)
 - GC (G-C-.bc)
 - Slice(T) (S-lice40T-41.bc)
 - IO::Encoder (I-O-5858E-ncoder.bc)
 - Iconv (I-conv.bc)
 - Math (M-ath.bc)
 - StaticArray(UInt8, 1024) (S-taticA-rray40U-I-nt84432102441.bc)
 - Char (C-har.bc)
 - Nil (N-il.bc)
 - Debug::DWARF::LineNumbers (D-ebug5858D-W-A-R-F-5858L-ineN-umbers.bc)
 - Array(Debug::DWARF::LineNumbers::Row) (A-rray40D-ebug585-1d195a7f864d2d3790ebeadd84b7a4df.bc)
 - Proc(Fiber, (IO::FileDescriptor | Nil)) (P-roc40F-iber4432-6eb246a0a45118d3c5507cc830b14a70.bc)
 - Hash(Int32, Channel::Buffered(Process::Status | Nil)) (H-ash40I-nt324432-12de0ef574784dde6ffe5b3ae93d5bba.bc)
 - Pointer(Hash::Entry(Int32, Channel::Buffered(Process::Status | Nil)) | Nil) (P-ointer40H-ash58-1ce4187043ca1e746f24acb8d0ff7a7e.bc)
 - Array(Proc(Int32, Nil)) (A-rray40P-roc40I-nt324432N-il4141.bc)
 - Pointer(Proc(Int32, Nil)) (P-ointer40P-roc40I-nt324432N-il4141.bc)

The count appears to be consistent, but the listed modules not :100: :confused:

With  --no-debug this problem disappears.

I still have no idea what's going on. If I start with an empty cache and compile an empty file, then in the cache directory there will be a T-hread.bc file. I do:
$ md5 T-hread.bc
MD5 (T-hread.bc) = ea315a5d222f2f73379e9378cb7fb12f

Now I compile again and repeat the check:
$ md5 T-hread.bc
MD5 (T-hread.bc) = 182b831f82b936d681b1e6ebdc4fccba

From there, subsequent compilations give the same md5 value. If I delete the cache and start over I get the same md5 values (first the first one, then always the second one). I can't understand what we are doing different in the compiler for LLVM to generate two different .bc files, and why LLVM manages to always generate a same .bc file starting from the second time... big mistery!

I continued investigating this. I changed Compiler#codegen_many_units to this:
    private def codegen_many_units(program, units, target_triple)
      unit1 = units.find(&.name.==("F-iber")).not_nil!
      unit2 = units.find(&.name.==("T-hread")).not_nil!

      unit1.llvm_mod.target = target_triple
      unit2.llvm_mod.target = target_triple

      # First unit
      unit1.llvm_mod.write_bitcode_to_file(unit1.bc_name)

      if ENV["FOO"]?
        target_machine.emit_obj_to_file unit1.llvm_mod, unit1.object_name
      end

      # Second unit
      unit2.llvm_mod.write_bitcode_to_file(unit2.bc_name)

      p `md5 #{unit2.bc_name}`

      [] of String
    end

So what I'm doing here is generating the .bc file for the LLVM module that corresponds to Fiber, and then doing the same for Thread. In between, I conditionally emit the .o for Fiber (by checking ENV["FOO"]?. What I can observe is that the second .bc file (the one for Thread) changes depending on whether an .o file was emited for Fiber or not. So it seems emit_obj_to_file (a method exposed by the LLVM API) has some other side effect besides writing the object file.

May no debug be a hint? Maybe some locations are slightly different, or maybe the generated DWARF changes a bit?

@ysbaddaden It's probably related to the debug info. However:

When I go from two different .bc files for the same module to .ll, I get the same .ll
The difference happens when I generate a .o file for an entirely different LLVM module

Should I be looking somewhere else? You can repeat the experiment by (using master) cleaning the cache, compiling an empty file, then doing llvm-bcanalyzer -dump ~/.cache/crystal/path/T-hread.bc > file1.txt, the compiling the file again and repeating the above command but storing it in a different file. I post the diff here. It seems the second time 3 more debug words (whatever that means) are generated. I still have no idea what's going on :-)
(I wanted to use the details tag, but apparently it stopped processing markdown...)
1c1
< <BITCODE_WRAPPER_HEADER Magic=0x0b17c0de Version=0x00000000 Offset=0x00000014 Size=0x00000f1c CPUType=0x01000007/>
---
> <BITCODE_WRAPPER_HEADER Magic=0x0b17c0de Version=0x00000000 Offset=0x00000014 Size=0x00000f28 CPUType=0x01000007/>
6c6
< <MODULE_BLOCK NumWords=957 BlockCodeSize=3>
---
> <MODULE_BLOCK NumWords=960 BlockCodeSize=3>
124c124
<   <VSTOFFSET abbrevid=6 op0=839/>
---
>   <VSTOFFSET abbrevid=6 op0=842/>
130c130
<   <METADATA_KIND_BLOCK NumWords=92 BlockCodeSize=3>
---
>   <METADATA_KIND_BLOCK NumWords=95 BlockCodeSize=3>
150a151
>     <KIND op0=20 op1=115 op2=114 op3=99 op4=108 op5=111 op6=99/>
521c522
<     <FNENTRY abbrevid=9 op0=9 op1=623 op2=42 op3=84 op4=104 op5=114 op6=101 op7=97 op8=100 op9=64 op10=82 op11=101 op12=102 op13=101 op14=114 op15=101 op16=110 op17=99 op18=101 op19=35 op20=61 op21=61 op22=60 op23=84 op24=104 op25=114 op26=101 op27=97 op28=100 op29=62 op30=58 op31=66 op32=111 op33=111 op34=108/> record string = '*Thread@Reference#==<Thread>:Bool'
---
>     <FNENTRY abbrevid=9 op0=9 op1=626 op2=42 op3=84 op4=104 op5=114 op6=101 op7=97 op8=100 op9=64 op10=82 op11=101 op12=102 op13=101 op14=114 op15=101 op16=110 op17=99 op18=101 op19=35 op20=61 op21=61 op22=60 op23=84 op24=104 op25=114 op26=101 op27=97 op28=100 op29=62 op30=58 op31=66 op32=111 op33=111 op34=108/> record string = '*Thread@Reference#==<Thread>:Bool'
523c524
<     <FNENTRY abbrevid=9 op0=1 op1=435 op2=42 op3=84 op4=104 op5=114 op6=101 op7=97 op8=100 op9=58 op10=58 op11=99 op12=117 op13=114 op14=114 op15=101 op16=110 op17=116 op18=58 op19=84 op20=104 op21=114 op22=101 op23=97 op24=100/> record string = '*Thread::current:Thread'
---
>     <FNENTRY abbrevid=9 op0=1 op1=438 op2=42 op3=84 op4=104 op5=114 op6=101 op7=97 op8=100 op9=58 op10=58 op11=99 op12=117 op13=114 op14=114 op15=101 op16=110 op17=116 op18=58 op19=84 op20=104 op21=114 op22=101 op23=97 op24=100/> record string = '*Thread::current:Thread'
526,528c527,529
<     <FNENTRY abbrevid=9 op0=18 op1=800 op2=42 op3=84 op4=104 op5=114 op6=101 op7=97 op8=100 op9=35 op10=99 op11=117 op12=114 op13=114 op14=101 op15=110 op16=116 op17=95 op18=102 op19=105 op20=98 op21=101 op22=114 op23=61 op24=60 op25=70 op26=105 op27=98 op28=101 op29=114 op30=62 op31=58 op32=70 op33=105 op34=98 op35=101 op36=114/> record string = '*Thread#current_fiber=<Fiber>:Fiber'
<     <FNENTRY abbrevid=9 op0=10 op1=654 op2=42 op3=84 op4=104 op5=114 op6=101 op7=97 op8=100 op9=64 op10=82 op11=101 op12=102 op13=101 op14=114 op15=101 op16=110 op17=99 op18=101 op19=35 op20=115 op21=97 op22=109 op23=101 op24=63 op25=60 op26=84 op27=104 op28=114 op29=101 op30=97 op31=100 op32=62 op33=58 op34=66 op35=111 op36=111 op37=108/> record string = '*Thread@Reference#same?<Thread>:Bool'
<     <FNENTRY abbrevid=9 op0=2 op1=464 op2=42 op3=84 op4=104 op5=114 op6=101 op7=97 op8=100 op9=58 op10=58 op11=110 op12=101 op13=119 op14=58 op15=84 op16=104 op17=114 op18=101 op19=97 op20=100/> record string = '*Thread::new:Thread'
---
>     <FNENTRY abbrevid=9 op0=18 op1=803 op2=42 op3=84 op4=104 op5=114 op6=101 op7=97 op8=100 op9=35 op10=99 op11=117 op12=114 op13=114 op14=101 op15=110 op16=116 op17=95 op18=102 op19=105 op20=98 op21=101 op22=114 op23=61 op24=60 op25=70 op26=105 op27=98 op28=101 op29=114 op30=62 op31=58 op32=70 op33=105 op34=98 op35=101 op36=114/> record string = '*Thread#current_fiber=<Fiber>:Fiber'
>     <FNENTRY abbrevid=9 op0=10 op1=657 op2=42 op3=84 op4=104 op5=114 op6=101 op7=97 op8=100 op9=64 op10=82 op11=101 op12=102 op13=101 op14=114 op15=101 op16=110 op17=99 op18=101 op19=35 op20=115 op21=97 op22=109 op23=101 op24=63 op25=60 op26=84 op27=104 op28=114 op29=101 op30=97 op31=100 op32=62 op33=58 op34=66 op35=111 op36=111 op37=108/> record string = '*Thread@Reference#same?<Thread>:Bool'
>     <FNENTRY abbrevid=9 op0=2 op1=467 op2=42 op3=84 op4=104 op5=114 op6=101 op7=97 op8=100 op9=58 op10=58 op11=110 op12=101 op13=119 op14=58 op15=84 op16=104 op17=114 op18=101 op19=97 op20=100/> record string = '*Thread::new:Thread'
530c531
<     <FNENTRY abbrevid=9 op0=5 op1=533 op2=42 op3=84 op4=104 op5=114 op6=101 op7=97 op8=100 op9=35 op10=105 op11=110 op12=105 op13=116 op14=105 op15=97 op16=108 op17=105 op18=122 op19=101 op20=58 op21=80 op22=111 op23=105 op24=110 op25=116 op26=101 op27=114 op28=40 op29=86 op30=111 op31=105 op32=100 op33=41/> record string = '*Thread#initialize:Pointer(Void)'
---
>     <FNENTRY abbrevid=9 op0=5 op1=536 op2=42 op3=84 op4=104 op5=114 op6=101 op7=97 op8=100 op9=35 op10=105 op11=110 op12=105 op13=116 op14=105 op15=97 op16=108 op17=105 op18=122 op19=101 op20=58 op21=80 op22=111 op23=105 op24=110 op25=116 op26=101 op27=114 op28=40 op29=86 op30=111 op31=105 op32=100 op33=41/> record string = '*Thread#initialize:Pointer(Void)'
534,535c535,536
<     <FNENTRY abbrevid=9 op0=13 op1=687 op2=42 op3=84 op4=104 op5=114 op6=101 op7=97 op8=100 op9=35 op10=102 op11=105 op12=110 op13=97 op14=108 op15=105 op16=122 op17=101 op18=58 op19=40 op20=73 op21=110 op22=116 op23=51 op24=50 op25=32 op26=124 op27=32 op28=78 op29=105 op30=108 op31=41/> record string = '*Thread#finalize:(Int32 | Nil)'
<     <FNENTRY abbrevid=9 op0=8 op1=596 op2=42 op3=84 op4=104 op5=114 op6=101 op7=97 op8=100 op9=64 op10=82 op11=101 op12=102 op13=101 op14=114 op15=101 op16=110 op17=99 op18=101 op19=35 op20=104 op21=97 op22=115 op23=104 op24=58 op25=85 op26=73 op27=110 op28=116 op29=54 op30=52/> record string = '*Thread@Reference#hash:UInt64'
---
>     <FNENTRY abbrevid=9 op0=13 op1=690 op2=42 op3=84 op4=104 op5=114 op6=101 op7=97 op8=100 op9=35 op10=102 op11=105 op12=110 op13=97 op14=108 op15=105 op16=122 op17=101 op18=58 op19=40 op20=73 op21=110 op22=116 op23=51 op24=50 op25=32 op26=124 op27=32 op28=78 op29=105 op30=108 op31=41/> record string = '*Thread#finalize:(Int32 | Nil)'
>     <FNENTRY abbrevid=9 op0=8 op1=599 op2=42 op3=84 op4=104 op5=114 op6=101 op7=97 op8=100 op9=64 op10=82 op11=101 op12=102 op13=101 op14=114 op15=101 op16=110 op17=99 op18=101 op19=35 op20=104 op21=97 op22=115 op23=104 op24=58 op25=85 op26=73 op27=110 op28=116 op29=54 op30=52/> record string = '*Thread@Reference#hash:UInt64'
543c544
<          Total size: 30944b/3868.00B/967W
---
>          Total size: 31040b/3880.00B/970W
551c552
<     Percent of file: 2.1717%
---
>     Percent of file: 2.1649%
564c565
<     Percent of file: 10.3219%
---
>     Percent of file: 10.2899%
583c584
<     Percent of file: 0.3781%
---
>     Percent of file: 0.3769%
596c597
<     Percent of file: 3.3771%
---
>     Percent of file: 3.3666%
609c610
<     Percent of file: 3.5968%
---
>     Percent of file: 3.5857%
627c628
<     Percent of file: 20.0814%
---
>     Percent of file: 20.0193%
655c656
<     Percent of file: 0.6916%
---
>     Percent of file: 0.6894%
669c670
<     Percent of file: 19.7227%
---
>     Percent of file: 19.6617%
685c686
<     Percent of file: 15.6056%
---
>     Percent of file: 15.5573%
708c709
<     Percent of file: 2.1717%
---
>     Percent of file: 2.1649%
722c723
<     Percent of file: 10.7194%
---
>     Percent of file: 10.6862%
744c745
<     Percent of file: 1.3088%
---
>     Percent of file: 1.3048%
756,757c757,758
<          Total Size: 2997b/374.62B/93W
<     Percent of file: 9.6852%
---
>          Total Size: 3093b/386.62B/96W
>     Percent of file: 9.9646%
760c761
<         Num Records: 20
---
>         Num Records: 21
765c766
<          20      2928     146.4          KIND
---
>          21      3021     143.9          KIND

OK, at this point I'm 99% sure it's a bug in LLVM. Here's some code to reproduce the issue very easily:
require "llvm"

lib LibLLVM
  fun create_memory_buffer_with_contents_of_file = LLVMCreateMemoryBufferWithContentsOfFile(path : UInt8*, out_mem_buf : MemoryBufferRef*, out_message : UInt8**) : Int32
  fun parse_ir_in_context = LLVMParseIRInContext(context : ContextRef, mem_buf : MemoryBufferRef, out_m : ModuleRef*, out_message : UInt8**) : Int32
end

module LLVM
  class MemoryBuffer
    def self.from_file(filename : String)
      ret = LibLLVM.create_memory_buffer_with_contents_of_file(filename, out mem_buf, out msg)
      if ret != 0 && msg
        raise LLVM.string_and_dispose(msg)
      end
      new(mem_buf)
    end
  end

  class Context
    def initialize
      @unwrap = LibLLVM.create_context
    end

    def parse_ir(buf : MemoryBuffer)
      ret = LibLLVM.parse_ir_in_context(self, buf, out mod, out msg)
      if ret != 0 && msg
        raise LLVM.string_and_dispose(msg)
      end
      Module.new(mod)
    end
  end

  class Module
    def initialize(@unwrap : LibLLVM::ModuleRef)
      @name = ""
      @owned = false
    end
  end
end

LLVM.init_x86

target_triple = LLVM.default_target_triple
target = LLVM::Target.from_triple(target_triple)
target_machine = target.create_target_machine(target_triple).not_nil!

ctx = LLVM::Context.global

fiber_buf = LLVM::MemoryBuffer.from_file("./F-iber.ll")
fiber_mod = ctx.parse_ir(fiber_buf)

thread_buf = LLVM::MemoryBuffer.from_file("./T-hread.ll")
thread_mod = ctx.parse_ir(thread_buf)

target_machine.emit_obj_to_file fiber_mod, "./Fiber.o"

thread_mod.write_bitcode_to_file "./Thread.bc"

What I'm doing here:

I'm extending the LLVM modules with missing functionality to create an LLVM::Module from LLVM IR in a file
I create two LLVM::Module instances, one from "F-iber.ll" and one from "T-hread.ll". You can generate these ".ll" files by executing crystal build foo.cr --ll with an empty "foo.cr" file, then finding these ".ll" files inside ~/.crystal/cache
I conditionally emit a ".o" file for "F-iber.ll"
I then emit a ".bc" file for "T-hread.ll"

The ".bc" changes depending on whether I emit a ".o" file for "F-iber.ll" or not.
What I found out is that if I use two different LLVM::Context instances, the problem goes away:
require "llvm"

lib LibLLVM
  fun create_memory_buffer_with_contents_of_file = LLVMCreateMemoryBufferWithContentsOfFile(path : UInt8*, out_mem_buf : MemoryBufferRef*, out_message : UInt8**) : Int32
  fun parse_ir_in_context = LLVMParseIRInContext(context : ContextRef, mem_buf : MemoryBufferRef, out_m : ModuleRef*, out_message : UInt8**) : Int32
end

module LLVM
  class MemoryBuffer
    def self.from_file(filename : String)
      ret = LibLLVM.create_memory_buffer_with_contents_of_file(filename, out mem_buf, out msg)
      if ret != 0 && msg
        raise LLVM.string_and_dispose(msg)
      end
      new(mem_buf)
    end
  end

  class Context
    def initialize
      @unwrap = LibLLVM.create_context
    end

    def parse_ir(buf : MemoryBuffer)
      ret = LibLLVM.parse_ir_in_context(self, buf, out mod, out msg)
      if ret != 0 && msg
        raise LLVM.string_and_dispose(msg)
      end
      Module.new(mod)
    end
  end

  class Module
    def initialize(@unwrap : LibLLVM::ModuleRef)
      @name = ""
      @owned = false
    end
  end
end

LLVM.init_x86

target_triple = LLVM.default_target_triple
target = LLVM::Target.from_triple(target_triple)
target_machine = target.create_target_machine(target_triple).not_nil!

ctx1 = LLVM::Context.new
ctx2 = LLVM::Context.new

fiber_buf = LLVM::MemoryBuffer.from_file("./F-iber.ll")
fiber_mod = ctx1.parse_ir(fiber_buf)

thread_buf = LLVM::MemoryBuffer.from_file("./T-hread.ll")
thread_mod = ctx2.parse_ir(thread_buf)

target_machine.emit_obj_to_file fiber_mod, "./Fiber.o"

thread_mod.write_bitcode_to_file "./Thread.bc"

I still consider the above to be a bug in LLVM: there's no reason why a ".bc" file should change for one module depending on whether I emitted a ".o" file for another module in the same context. I'm pretty sure that LLVM function somehow changes LLVM::Context.
In any case, to support parallel compilation without fork, like what we do right now (fork will most likely disappear from the language) we'll need to use different LLVM contexts, so I guess it's time to start working on that :-)

If I remember correctly, there is a global context in the LLVM C API, though the C++ API no longer has one; it changed in LLVM 3.7 or 3.8.

Some progress on this: I managed to refactor the compiler's codegen to use different LLVM::Context instances for each module. This opens the door for generating bc/obj files in multiple threads, and also partially fixes the problem described in this issue. Now with an empty file I get that all .bc files are reused at the second compilation. But with bigger files, like the http server example or the compiler there are still some files that generate in a different way. I'm pretty sure at this point that this is an issue in LLVM itself. One way to "fix" this would be to first generate all ".bc" files and then all ".o" files, but I'm not sure that will slow down things a bit... we can keep this issue open, but it's really minor as eventually all ".bc" files end up being reused.

Looking into the changed binary cache files provides further clues, since various debug info is legible.
I'm guessing the H-ash40T-hread4432N-il41.bc changes since debug information is referenced via hash tables - and initialising these hash tables (initial build) is treated differently than running subsequent builds - thus H-ash40T-hread4432N-il41.bc only changes once.
the files S-et40T-hread41.bc and T-hread.bc appear to change as a side-effect of H-ash40T-hread4432N-il41.bc changing.
Some useful info is here: http://llvm.org/docs/SourceLevelDebugging.html
This is of course largely speculation, but it looks like a debug trade-off against cache reuse. Of course no-debug or release negates the trade-off.
https://gist.github.com/mjago/1174481762671c8293ab8a0fc94b78e0

I believe that this issue should be closed, because fixed along #4336 issue.

@akzhan looks like is working, closing it.
Thank you for the follow up.

Actually reopening this based on last comment from @akzhan from #4336 😢

This is just a compiler detail. Not a bug. I don't think we can improve this because LLVM doesn't guarantee we can reuse modules.


Does turms support voice and video calls?

No plan for it in v1.0.
Most open-source IM projects will incorporate other video conferencing solutions, e.g. jitsi/jitsi, versatica/mediasoup, and we will incorporate other solutions, but don't have efforts for it in v1.0.


Remove redundant Transform2D initializations

As mentioned in https://github.com/godotengine/godot/pull/68627#discussion_r1021739394.

While at it, there's also:
scene/resources/skeleton_modification_2d_fabrik.h
71:     Transform2D target_global_pose = Transform2D();
72:     Transform2D origin_global_pose = Transform2D();

And some more for other vector/matrix types:
$ rg "Transform3D .* = Transform3D\(\);" --sort path
core/variant/variant_call.cpp
2441:   Transform3D identity_transform = Transform3D();

drivers/gles3/rasterizer_scene_gles3.h
102:    Transform3D cam_transform = Transform3D();
103:    Transform3D inv_cam_transform = Transform3D();

editor/plugins/node_3d_editor_plugin.cpp
3495:           Transform3D axis_angle = Transform3D();

editor/plugins/skeleton_3d_editor_plugin.cpp
1291:   Transform3D original_to_local = Transform3D();
1299:   Transform3D t = Transform3D();

scene/resources/skeleton_modification_3d_fabrik.h
71:     Transform3D target_global_pose = Transform3D();
72:     Transform3D origin_global_pose = Transform3D();

And all other non-trivial Variant types such as Vector*, String, Color, StringName, etc. all have similar occurrences. But I guess that's out of scope for this PR :)

And all other non-trivial Variant types such as Vector*, String, Color, StringName, etc. all have similar occurrences. But I guess that's out of scope for this PR :)

I can simply rename the PR to "Remove redundant initialization" to change the scope ;-)
Will see what I can find.

Thanks!


python312Packages.itk: init at 5.4.0

Description of changes
Init the Python wrapping for ITK, a major biomedical imaging software package. The wrapping is useful in its own right and is also a dependency of various other packages such as ITKElastix, MONAI-Label, MONAI-Vista, ITKWidgets, ITKWASM, ngf-zarr, etc.
Unfortunately, the SWIG/castXML wrapping is a bit of a beast to build relative to the C++-only build, though not horrible by the standards of C++ projects.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

The x86_64-darwin Hydra build times out seemingly in the same way as https://github.com/NixOS/nixpkgs/pull/277465. No errors are thrown but the build is simply incredibly slow for unknown reasons.

nixpkgs-review happy on x86_64-linux (pydicom-seg already broken on master):
1 package marked as broken and skipped:

ezminc

4 packages failed to build:

python311Packages.pydicom-seg python311Packages.pydicom-seg.dist python312Packages.pydicom-seg python312Packages.pydicom-seg.dist

34 packages built:

ants c3d elastix intensity-normalization intensity-normalization.dist itk itk_5_2 mirtk mrtrix python311Packages.intensity-normalization python311Packages.intensity-normalization.dist python311Packages.itk python311Packages.medpy python311Packages.medpy.dist python311Packages.pymedio python311Packages.pymedio.dist python311Packages.pyradiomics python311Packages.pyradiomics.dist python311Packages.simpleitk python311Packages.simpleitk.dist python311Packages.torchio python311Packages.torchio.dist python312Packages.itk python312Packages.medpy python312Packages.medpy.dist python312Packages.pymedio python312Packages.pymedio.dist python312Packages.pyradiomics python312Packages.pyradiomics.dist python312Packages.simpleitk python312Packages.simpleitk.dist python312Packages.torchio python312Packages.torchio.dist simpleitk


switch to andreiborisov docker image to test against version up to 3.1.2

As oh-my-fish/dockerfiles repo is stuck to 3.0.0 without activity for the last 1 and a half, I'm switching to @andreiborisov docker images so we can test against versions up to 3.1.2

[x] clean our dockerfile from redundancy with parent image
[x] update travis files images
[x] add new version to test in CI

Hi! Are you guys still supporting fish 2? Fisher dropped support for the second version recently, so I have a choice to pull the latest supported version of Fisher for the v2-based images or drop v2 support altogether. Let me know if you still planning to use v2 images, please.
Also, I have an example of how to set up tests with GitHub Actions, if you want to ditch Travis: https://github.com/andreiborisov/fish-package
😉

Good idea, @andreiborisov. This is my workflow for fish plugins by the way.

@andreiborisov yep we will drop support for fish 2.x as described in Versions-Support-Strategy.
Travis CI is doing the job so nothing to fix but I'm open to switch (out of curiosity as I'm still unfamiliar with github CI)

I'm dropping v2 support too then🙌

I'm open to switch (out of curiosity as I'm still unfamiliar with github CI)

It would be as easy as adding docker-compose.yml for testing and a GitHub workflow config itself, which is also quite simple and straightforward.
The only thing you'd need to tweak is to change the path to test files here: https://github.com/andreiborisov/fish-plugin/blob/89550024d7bf8438f9d9d02040f0d75da03d2760/docker-compose.test.yml#L6 and remove v2 fish versions from both files.

Thanks I will have a look shortly, TravisCI is way too slow! :angry:


Diagram support

Hello, first of all I'd like to say a big thank you to the whole team for creating such a good tool!
I'd like to know, are there any estimates, when inline diagrams would be supported?
I am writing a scientific paper with a lot of diagrams that I created using graphviz dot. I tried converting .dot files to SVG, but when inserted to the document, text labels become invisible, for some reason. Inserting them as PDF for now.

Diagram support is still a bit away. For now SVGs are the best we have. If they contain text, you need to export the text as paths because text in SVG is also not yet supported (see also #371)

I think that is better to use d2 https://d2lang.com/ generate svg and import it in typst

@giovanniberti, mermaid is fine for simpler diagrams, but for complex graphs with custom edge and node settings it can be problematic.

hey, i'm the creator of D2. If there's anything we can do on our side to make integration possible, please let me know.

+1 vote for d2, and not mermaid
mermaid is very limited.

The focus should be on implementing native Typst visualization first. Support for various external formats sounds like a job for a future package, not a part of typst itself.

The focus should be on implementing native Typst visualization first.

Whilst native solution would probably be the best, adapting existing tools seem to be an easier task with predictable result, IMO. Especially given that the surface area is quite small. Just an extension of a figure will do the trick: text in, SVG out.

FWIW if it helps anyone here, I'm using inline Python code to generate images dynamically: c.f. this comment and typst_pyimage.

It appears that D2 is fairly limited. It seems to be focused on creating box-and-connector type diagrams.
I imagine that Typst will be aiming for a generic drawing system more akin to TikZ.
Something that I value about TikZ in particular is the ability to integrate it into the layout of the page including the text and equations on the page. More specifically, I can set anchors within the text, then have, for example arrows connecting those anchors across the page.
FWIW the haskell "diagrams" package is absurdly elegant  but perhaps less relevant as a source of inspiration

+1 vote for d2, and not mermaid

I have asked here, and according to D2 creator, complex diagrams (e.g. in the context of academia) are beyond the scope of D2.
I suggest using Penrose instead. Take a look here:

Project: https://github.com/penrose/penrose
Gallery: https://penrose.cs.cmu.edu/examples

I have asked here, and according to D2 creator, complex diagrams (e.g. in the context of academia) are beyond the scope of D2.

I think that's a misrepresentation of what @alixander said:

Thanks for the reply. These tools are at different levels of abstraction. The fundamental difference is that their syntax is telling you how to draw it (segment from this coordinate to this coordinate), whereas D2's syntax is describing the model. The types of diagrams one makes with each tool is almost entirely mutually exclusive. It's out of scope for this tool/language.

I think that higher level abstraction fits nicely with how Typst abstracts away directly laying out text. D2 is the same in that it abstracts away directly laying out diagram elements. I think there are some cool possibilities that could be unlocked if Typst could influence or constrain the rendered diagram based on the page's/document's layout constraints
FWIW - D2 (generating an SVG) & Typst fit my use case where in the past I would have used Tikz & Latex. Obviously that doesn't solve every use case though.

@a7g4
Your reply is completely misleading and I do not see what value it tries to add here. The author summarized it in clear language by saying:

It's out of scope for this tool/language.

Take a look at Penrose. It has very advanced features similar to tikz et al.

+1 for penrose, i've used it in the past and it's very extensible. d2 while is on a higher level, doesn't many diagram use cases.

+1 for Penrose.

Screenshot of: https://penrose.cs.cmu.edu/try/?examples=set-theory-domain/tree-venn

Another possibility is Excalidraw as shown in this screen capture. It's nothing sophisticated but it's convenient and simple, fitting some use-cases.
https://github.com/typst/typst/assets/1690072/84d23db4-9512-4bff-a468-02a799dcd845

The svgbob direction appears potentially promising, but has some drawbacks as of writing this. The key idea of svgbob is for ascii diagrams (diagrams that are drawn with unicode characters) to be rendered into the svg format which are embeddable in Typst, and subseqeuently rendered in the output file (e.g., pdf).

Relevant links:

Samples of what svgbob can do are shown in an online editor that can convert ascii into svg in real-time so that you can get your hands dirty: https://ivanceras.github.io/svgbob-editor/

A Typst package that converts svgbob-compliant ascii into svg: https://typst.app/universe/package/bob-draw

(Current) Drawbacks:

No method (that I know of) to convert an image (say, a sketch) to svgbob-compliant ascii (which can, in turn, be input into an svgbob renderer)

No method (that I know of) to convert ascii to svgbob-compliant ascii. (which can, in turn, be input into an svgbob renderer)

It appears that some graphics (vectors?) in the svgbob standard are hard-coded, e.g., there really are only a handful of circles of different sizes available in svgbob.

The svgbob specifications appear to be quite strict at the moment – one can't just draw ascii diagrams, of say, circles, in most ascii applications like Monodraw or ASCIIflow, and expect svgbob renderers to render them as circles.

Note that the drawbacks I've listed here are current as of writing this comment, and may well be overcome in the future. Overall, I think the svgbob direction is kinda neat.

Given the discussion over various other libraries, I just wanted to mention Kroki which offers "one API to rule them all"
https://kroki.io/

There are a few things being suggested here. Any of them can be used to generate images that can be imported into a Typst.  I could also imagine
However, when it comes to properly integrating a language for drawing into the Typst typesetting system, I think that the following features would be desirable

integration with Typst scripting, so that Typst variables and state can be used in generating diagrams
integration with typesetting allowing typeset elements to be included in
integration with typesetting, allowing diagrams to anchor to points in the page layout (e.g. adding arrow between specific paragraphs)
be precise and general purpose, providing lower level primitives like polygons and lines that allow the user to express any kind of diagram in any style
yet still allow convenient abstractions (e.g. allow writing functions to draw bar charts from data)

While many of the above diagramming suggestions are well worth learning from, I can't imagine that integrating them directly could ever provide all of these features in a satisfactory way.
I imagine that, similar to  TiKZ in TeX, it makes to build a diagramming library in the Typst scripting library on top of some lower level drawing primitives.
This seems like the kind of project that some dedicated and talented individual is going to have to eat sleep and breathe for some months to bring to life.

I think that's what CeTZ is aiming for (and it's already quite usable).


Bullet gem complains about n+1 when using Model.root_comments and recursive partials

Hi,
The Bullet gem gives me the following error:
Comment => [:children]
  Add to your finder: :include => [:children]
N+1 Query method call stack
  /Users/Nadeem/rails/iQue/app/models/comment.rb:46:in `has_children?'
  /Users/Nadeem/rails/iQue/app/views/questions/_comment.html.haml:7:in `_app_views_questions__comment_html_haml___3048192702811821479_70228511044500'
  /Users/Nadeem/rails/iQue/app/views/questions/show.html.haml:22:in `block in _app_views_questions_show_html_haml___3264554324951737521_70228577037320'
  /Users/Nadeem/rails/iQue/app/views/questions/show.html.haml:21:in `_app_views_questions_show_html_haml___3264554324951737521_70228577037320'

How do we fix this?
Here is my controller code:
@comments = @question.root_comments

Here is my view code:
- @comments.each do |c|
  = render partial: 'comment', locals: {c: c, level: 0}

Here is my partial:
.row
  %div{class: "col-xs-#{12 - level} pull-right"}
    %li
      .card
        = c.body

- if c.has_children?
  - c.children.each do |child|
    = render partial: 'comment', locals: {c: child, level: level + 1}

Adding this to comment.rb worked:
def self.default_scope

  {includes: :children}

end

I am using partials in a recursive manner like you are showing.
I am not sure why your getting that error without seeing your entire setup, but I did put together a walkthrough of my setup. Let me know if that helps figure out why your getting that error.


Cannot Open Realm in Background Isolate

What happened?
See https://github.com/realm/realm-dart/issues/1055 for a similar issue...
I'm trying to follow the doc from here : https://www.mongodb.com/docs/realm/sdk/flutter/sync/sync-multiple-processes/...
The basic instructions from that are to have a single realm using the flexible sync config and then any other workers using a disconnected sync configuration.
I'm trying to do that and am getting the following exception.
RealmException (RealmException: Error opening realm at path /Users/ebenezer/.hh2mobile.realm. Error code: 2013 . Message: Realm at path '/Users/ebenezer/.hh2mobile.realm' already opened with different sync configurations.)
Repro steps
Look at the code in the code snippet below. It does the following

Starts up a flexible sync realm
adds subscriptions to listen for changes to a collection of objects
spawns an isolate that will open the same realm with a disconnected sync config

the isolate will loop to scan for new items to process

the main thread will loop and run a query every so often

Note that the way the writeRealm is initialized is the same as the issue that I mention in the "What happened?" section except that reporter used a memory config where I'm using a disconnected sync config.
What am I doing wrong?
Version
Dart SDK version: 3.2.3 (stable) (Tue Dec 5 17:58:33 2023 +0000) on "macos_arm64"
What Atlas Services are you using?
Atlas Device Sync
What type of application is this?
Dart standalone application
Client OS and version
macOS Sonoma 14.2
Code snippets
import 'dart:async';
import 'dart:io';
import 'dart:isolate';

import 'package:path/path.dart' as path;
import 'package:realm_dart/realm.dart';

import 'package:mongodart/realmmodels/models.dart';

String get _homeDir => Platform.environment['HOME'] ?? (Platform.environment['APPDATA'] != null ? path.join(Platform.environment['APPDATA']!, 'Local') : null) ?? Platform.environment['USERPROFILE'] ?? '.';
String get _realmPath => path.join(_homeDir, '.hh2mobile.realm');

final schemaObjects = [RealmReceipt.schema, RealmFileRecordLinkChunk.schema, RealmCreditCard.schema, RealmFileRecordLink.schema, RealmUser.schema];
final writeConfig = Configuration.disconnectedSync(schemaObjects, path: _realmPath);
final writeRealm = Realm(writeConfig);

class _tusConfig {
  _tusConfig(this.endpoint, this.cachePath, this.port);

  final String endpoint;
  final String cachePath;
  final SendPort port;
}

Future<void> _tusUploader(_tusConfig config) async {
  final p = ReceivePort();
  config.port.send(p.sendPort);

  do {
    final frl = writeRealm.all<RealmFileRecordLink>().where((frl) => frl.tusEndpoint != null && frl.completedOn == null);

    for (var r in frl) {
      // do some work here... but it doesn't really matter...
    }

    await Future.delayed(Duration(seconds: 60));
  } while (await p.isEmpty);
}

Future<void> _subscribeForSync<T extends RealmObject>(Realm realm) async {
  final subName = T.toString();
  final q = realm.all<T>();
  realm.subscriptions.update((mutableSubscriptions) {
    mutableSubscriptions.add(q, name: subName);
  });
}

Future<void> main() async {
  final app = App(AppConfiguration('devicesync-kjdtl'));
  final user = await app.logIn(Credentials.anonymous());
  final syncConfig = Configuration.flexibleSync(user, schemaObjects, path: _realmPath);
  final syncRealm = Realm(syncConfig);

  // subscribe to the data elements that we care about
  await _subscribeForSync<RealmUser>(syncRealm!);
  await _subscribeForSync<RealmReceipt>(syncRealm!);
  await _subscribeForSync<RealmCreditCard>(syncRealm!);
  await _subscribeForSync<RealmFileRecordLink>(syncRealm!);

  final p = ReceivePort();
  final cachePath = path.join(_homeDir, '.tuscache');
  final tusConfig = _tusConfig("https://tusd.tusdemo.net/files/", cachePath, p.sendPort);
  await Isolate.spawn(_tusUploader, tusConfig);

  final sendPort = await p.first as SendPort;

  var keepRunning = true;
  do {
    final receipts = syncRealm.all<RealmReceipt>();
    if (receipts.isNotEmpty) {
      final first = receipts.first;
    }
    await Future.delayed(const Duration(seconds: 5));
  } while (keepRunning);

  sendPort.send(null);
}

Stacktrace of the exception/crash you're getting
No response
Relevant log output
No response

This documentation explains how to open the file from multiple processes. In your case, you are opening it from the same process from different isolates. In this case, you should open it with the same config as the one you used to open it on the main isolate. Note that due to https://github.com/realm/realm-dart/issues/1433, that won't work with 1.6.0, so you either need to downgrade or wait until 1.7.0 is released.

Okay... any idea what the timeframe for 1.7 is?

I don't imagine it'll land before the holidays, so realistically - the first half of January.

@richard457 Waiting for #1470. I expect us to release this week.

when this goes to pub.dev?

I have tested this out with the following code:
``dart
static Future handleRealm(List args) async {
final rootIsolateToken = args[0] as RootIsolateToken;
final sendPort = args[1] as SendPort;
final branchId = args[2] as int;
final encryptionKey = args[3] as String;
BackgroundIsolateBinaryMessenger.ensureInitialized(rootIsolateToken);
// get isar instances
isar = await isarK();
final app = App.getById(AppSecrets.appId);
final user = app?.currentUser!;
List key = encryptionKey.toIntList();
final config = Configuration.flexibleSync(
user!,
[
RealmITransaction.schema,
RealmITransactionItem.schema,
RealmProduct.schema,
RealmVariant.schema,
RealmStock.schema,
RealmIUnit.schema
],
);
final realm = Realm(config);
// final realm = await Realm.open(config);

// Subscribe to changes for transactions
final iTransactionsCollection =
    realm.query<RealmITransaction>(r'branchId == $0', [branchId]);

iTransactionsCollection.changes.listen((changes) async {
  for (final result in changes.results) {
    final model = createmodel(result);

    if (model.action == AppActions.deleted && model.deletedAt == null) {
      model.deletedAt = DateTime.now();
    }
   sendPort.send('data ${model.id}');
  }
});

}
and Trigered the function likedart
@override
Future pull() async {
RootIsolateToken? rootIsolateToken = RootIsolateToken.instance;
int branchId = ProxyService.box.getBranchId()!;
String key = ProxyService.box.encryptionKey();
ReceivePort receivePort = ReceivePort();
await Isolate.spawn(IsolateHandler.handleRealm, [
rootIsolateToken,
receivePort.sendPort,
branchId,
key,
]);
receivePort.listen((message) {
log('Isolate: $message');
});
}
``
I can confirm that on Windows the above code works but not on Android device, when there is a data change the sendPort.send('data ${model.id}'); is not triggered on android @nielsenko

@richard457 Remember to hold onto the stream subscription so that you can call cancel when you are done with it.
final subscription = iTransactionsCollection.changes.listen(...
// and later
await subscription.cancel();

This holds for all streams, not just realm change streams. Otherwise you risk leaking native resources, such as file handles, etc. You can also end up in situations where the subscription is the only thing that prevents the entire stream from being garbage collected.
In general I recommend using await for instead of raw listen to avoid oversights such as this.
await for (final t in iTransactionsCollection.changes.listen(...)) {
  // dart will ensure subscription is cancelled no matter how you exit from here
}

but that is not always possible.
Realm actually tries to help mitigate this issue by using weak handles, and while debugging this, I found a bug that I'm fixing as I write this, so stay tuned..
However - after the fix - it becomes even more important to always remember to cancel the stream subscription, as you are otherwise certain to both leak native- and pin dart- resources.

Thanks, @nielsenko but I find on Android that data changes are not reflected when the connection to the server (flexible sync) is established while on Windows changes are detected automatically, I mean I have two device

Windows which makes changes
Android the receiver of change
Please shed more light on this as I can not get the latest android data when there are new changes..

@richard457 Could you setup logging and tell us what you see?
Realm.logger.level = RealmLogLevel.trace;

on both windows and android?
Also,

How does your models look?
How does your flexible sync subscriptions look?

My model
import 'package:realm/realm.dart';

part 'realmTransactionItem.g.dart';

@RealmModel()
class _RealmITransactionItem {
  late String id;
  @PrimaryKey()
  @MapTo('_id')
  late ObjectId realmId;
  late String name;
  late String transactionId;
  late String variantId;
  // quantity
  late double qty;
  late double price;
  late String action;

  late int branchId;
}```
```dart
 final app = App(AppConfiguration(AppSecrets.appId,
        baseUrl: Uri.parse("https://services.cloud.mongodb.com")));
    final user = app.currentUser ?? await app.logIn(Credentials.anonymous());
    List<int> key = ProxyService.box.encryptionKey().toIntList();
    final config = Configuration.flexibleSync(
      user,
      [
        RealmITransaction.schema,
      ],
      // encryptionKey:key,
      path: await absolutePath("db_"));```
Yes I am able to see data in atlas when data are saved from desktop app.

And how does your subscriptions look?
realm.subscriptions.update((ms) {
  // what are you subscribing on?
});

.. and what do the logging tell you?

final transaction =
        realm.query<RealmITransaction>(r'branchId == $0', [branchId]);
    //https://www.mongodb.com/docs/realm/sdk/flutter/sync/manage-sync-subscriptions/
    realm.subscriptions.update((sub) {
      sub.clear();
      sub.add(transaction, name: "transactions-${branchId}", update: true);
);

FYI: I am only doing subscription to the main tread not in isolate.
Getting this in log:
I/flutter ( 5150): 2024-01-26T11:23:11.276052: [INFO] Realm: Connection[1]: Session[1]: Binding '/data/user/0/rw.flipper/app_flutter/flipper-sync/db_' to ''
I/flutter ( 5150): 2024-01-26T11:23:11.278213: [INFO] Realm: Connection[1]: Session[1]: client_reset_config = false, Realm exists = true
I/flutter ( 5150): 2024-01-26T11:23:11.279519: [INFO] Realm: Connection[1]: Connecting to 'wss://ws.ap-south-1.aws.realm.mongodb.com:443/api/client/v2.0/app/devicesync-ifwtd/realm-sync'
I/flutter ( 5150): 2024-01-26T11:23:11.626326: [INFO] Realm: Connected to endpoint '<IP_ADDRESS>:443' (from '<IP_ADDRESS>:48752')
I/flutter ( 5150): 2024-01-26T11:23:13.377412: [INFO] Realm: Connection[1]: Connected to app services with request id: "65b37a0123d017c54b8e3f4c"
I/flutter ( 5150): 2024-01-26T11:23:21.771668: [INFO] Realm: Connection[1]: Session[1]: Begin processing pending FLX bootstrap for query version 12. (changesets: 1, original total changeset size: 0)
I/flutter ( 5150): 2024-01-26T11:23:21.783177: [INFO] Realm: Connection[1]: Session[1]: Integrated 1 changesets from pending bootstrap for query version 12, producing client version 159 in 10 ms. 0 changesets remaining in bootstrap

The code bits I have seen makes me question if you are using the same realm in both the foreground and the background isolate. I see a path passed to AppConfiguration in one place, but not the other. It is important that the realm you add the subscription to, and the realm you listen for changes on, are actually backed by the same realm file..

It is okay to setup the subscription(s) in one isolate, and query/observe the realm in another, but it must be backed by the same physical realm file.
Try to print the realm.config.path in both isolates.

Sorry for late reply, I realized that not setting path in isolate was the root cause of the issue.


Workflow .github/workflows/test-action.yml needs urgent update before end of January 2025

workflow: https://github.com/asyncapi/cli/blob/master/.github/workflows/test-action.yml
reason:
GitHub
Artifact actions v3 will be closing down by January 30, 2025. You are receiving this email because you have GitHub Actions workflows using v3 of [actions/upload-artifact](https://app.github.media/e/er?s=88570519&lid=6648&elqTrackId=867ac6c9fa524bda9764f13bec8f8014&elq=49830aa2044d4416a4a36a4fbb961619&elqaid=4286&elqat=1&elqak=8AF5D94B629B77D0B2E640E5D18BB16F1795C5AC9FBBABD9859ACC84B62448387E12) or [actions/download-artifact](https://app.github.media/e/er?s=88570519&lid=6647&elqTrackId=bb54934dfd8c4c44b0e64ec2f2ba4824&elq=49830aa2044d4416a4a36a4fbb961619&elqaid=4286&elqat=1&elqak=8AF57B355AADDDE608896B61A68B985ADA84C5AC9FBBABD9859ACC84B62448387E12). After this date using v3 of these actions will result in a workflow failure. Artifacts within their retention period will remain accessible from the UI or REST API regardless of the version used to upload.

To raise awareness of the upcoming removal, we will temporarily fail jobs using v3 of actions/upload-artifact or actions/download-artifact. Builds that are scheduled to run during the brownout periods will fail. The brownouts are scheduled for the following dates and times:

January 9th, 12pm - 1pm EST
January 16th, 10am-2pm EST
January 23rd, 9am-5pm EST
What you need to do
Update workflows to begin using v4 of the artifact actions as soon as possible. While v4 improves upload and download speeds by up to 98% and includes several new features, there are [key differences](https://app.github.media/e/er?s=88570519&lid=6646&elqTrackId=e5ab3eaa455c4e8baa6af6be9fdb5c62&elq=49830aa2044d4416a4a36a4fbb961619&elqaid=4286&elqat=1&elqak=8AF58F0002C617A8A55C9135F66B95938105C5AC9FBBABD9859ACC84B62448387E12) from previous versions that may require updates to your workflows. Please see [the documentation](https://app.github.media/e/er?s=88570519&lid=6645&elqTrackId=af6fe6939f7e4dcba105e35734b7e6a0&elq=49830aa2044d4416a4a36a4fbb961619&elqaid=4286&elqat=1&elqak=8AF5780CABD5F82033875E5CE3EF4DF2FDFCC5AC9FBBABD9859ACC84B62448387E12) in the project repositories for guidance on how to migrate your workflows.

more info:

https://github.com/actions/upload-artifact?tab=readme-ov-file#breaking-changes
https://github.com/actions/upload-artifact/blob/main/docs/MIGRATION.md

@derberg can I work on this ?

I think so, probably @Shurtu-gal will need to help. I don't know, I just reported it as I got a notification from GH as org admin

I looked into the breaking changes and migration docs. We are not using any of those features, hence our bump is as simple as changing v3 -> v4
@Sahil-Shadwal if you want you can take up this issue.

[!IMPORTANT]
You have to bump both actions/upload-artifact as well as actions/download-artifact

/gfi

/gfi ci-cd

@Shurtu-gal @derberg I would like to work on this issue. Should I for the issue to get assign or start working on it?

We don't assign issues here as it can be blocking. Anybody is free to work on the issue and submit PRs

OK, I will try to submit a PR soon.

Okay, found a problem. They are not allowing multiple downloads also I suppose, although it is nowhere mentioned.
 Updated it and everything is working fine closing this issue.


Container do not start after reboot

Operating system
Ubuntu 24.04
Description
I gave the command "docker compose up -d" but the container does not initialize after the "reboot". Is there anything that can be done to make the container start automatically?
Docker compose
services:
windows:
image: dockurr/windows
container_name: windows
environment:
VERSION: "win11"
REGION: "pt-BR"
KEYBOARD: "pt-BR"
TZ: 'America/Sao_Paulo'
devices:
- /dev/kvm
volumes:
- "/var/mssql/data:/shared"
cap_add:
- NET_ADMIN
ports:
- 8006:8006
- 3389:3389/tcp
- 3389:3389/udp
stop_grace_period: 2m
Docker log
❯ Extracting Windows 11 image...
❯ Adding drivers to image...
❯ Adding win11x64.xml for automatic installation...
❯ Building Windows 11 image...
❯ Creating a 64G growable disk image in raw format...
❯ Warning: unexpected clocksource: kvm-clock
❯ Booting Windows using QEMU v8.2.4...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000007H:EDX.invtsc [bit 8]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000007H:EDX.invtsc [bit 8]
BdsDxe: failed to load Boot0002 "UEFI QEMU QEMU HARDDISK " from PciRoot(0x0)/Pci(0xA,0x0)/Scsi(0x0,0x0): Not Found
BdsDxe: loading Boot0001 "UEFI QEMU QEMU CD-ROM " from PciRoot(0x0)/Pci(0x5,0x0)/Scsi(0x0,0x0)
BdsDxe: starting Boot0001 "UEFI QEMU QEMU CD-ROM " from PciRoot(0x0)/Pci(0x5,0x0)/Scsi(0x0,0x0)
❯ Windows started succesfully, visit http://localhost:8006/ to view the screen...
BdsDxe: loading Boot0004 "Windows Boot Manager" from HD(1,GPT,4A744BA6-3210-4358-A6F5-E67E48A0CAE3,0x800,0x40000)/\EFI\Microsoft\Boot\bootmgfw.efi
BdsDxe: starting Boot0004 "Windows Boot Manager" from HD(1,GPT,4A744BA6-3210-4358-A6F5-E67E48A0CAE3,0x800,0x40000)/\EFI\Microsoft\Boot\bootmgfw.efi
BdsDxe: loading Boot0004 "Windows Boot Manager" from HD(1,GPT,4A744BA6-3210-4358-A6F5-E67E48A0CAE3,0x800,0x40000)/\EFI\Microsoft\Boot\bootmgfw.efi
BdsDxe: starting Boot0004 "Windows Boot Manager" from HD(1,GPT,4A744BA6-3210-4358-A6F5-E67E48A0CAE3,0x800,0x40000)/\EFI\Microsoft\Boot\bootmgfw.efi
❯ Received SIGTERM, sending ACPI shutdown signal...
❯ Waiting for Windows to shutdown... (1/110)
❯ Waiting for Windows to shutdown... (2/110)
❯ Waiting for Windows to shutdown... (3/110)
❯ Waiting for Windows to shutdown... (4/110)
❯ Waiting for Windows to shutdown... (5/110)
❯ Waiting for Windows to shutdown... (6/110)
❯ Waiting for Windows to shutdown... (7/110)
❯ Waiting for Windows to shutdown... (8/110)
❯ Waiting for Windows to shutdown... (9/110)
❯ Waiting for Windows to shutdown... (10/110)
❯ Waiting for Windows to shutdown... (11/110)
❯ Shutdown completed!
Screenshots (optional)

My bad. I managed to solve the problem by adding the "restart: always" command in the docker-compose.yml file.


service handlers should use more concrete signatures

Currently, our http services use always use a (w http.ResponseWriter, r *http.Request) signature. While that pattern is used for http handlers it is not used for grpc handlers which use specific request and response objects, delegating the request parsing and writing to middlewares.
AFAICT ocis-hello shows the correct pattern to use:
https://github.com/owncloud/ocis-hello/blob/edc8ef1d4b79dba29321cc70a3f6a20373d30f65/pkg/service/v0/service.go#L19-L21
Actually it should receive a Context as the first parameter, but anyway, the HTTP service is then created using chi by parsing the json and then calling the handler:
https://github.com/owncloud/ocis-hello/blob/edc8ef1d4b79dba29321cc70a3f6a20373d30f65/pkg/server/http/server.go#L92
The command first creates the handler and then wraps it with tracing, logging and metrics middlewares which can now operate directly on the request and response parameters.
https://github.com/owncloud/ocis-hello/blob/edc8ef1d4b79dba29321cc70a3f6a20373d30f65/pkg/command/server.go#L90-L107
In the chi handler for graph we can parse the odata request using godata.ParseRequest(...) and pass that to the Service interface so it can sort, filter, select and expand the object as needed.
An example signature would be:
// the old service interface
func (g Graph) GetMe(w http.ResponseWriter, r *http.Request) {}

// the new signature should be concerned with concrete types
func (g Graph) GetMe(ctx context.Context, id string) (libregraph.User, error) {}

// unfortunately, we need to pass along
// - which filelds to select and expand, GRPC uses field/update mask, see https://aip.bybutter.com/161
// - pagination, GRPC uses page_size and page_token, see https://aip.bybutter.com/158) and
// - filtering, GRPC uses an [EBNF](https://aip.bybutter.com/assets/misc/ebnf-filtering.txt) based filter, see https://aip.bybutter.com/160
func (g Graph) GetMe(ctx context.Context, id string, fieldMask string) (libregraph.User, error) {}

// for listing this becomes clearer:
func (g Graph) ListUsers(ctx context.Context, filter, fieldMask, page_token string, pageSize int) (libregraph.User, error) {}

Then the service interface from
type Service interface {
	ServeHTTP(http.ResponseWriter, *http.Request)
	GetMe(http.ResponseWriter, *http.Request)
	GetUsers(http.ResponseWriter, *http.Request)
	GetUser(http.ResponseWriter, *http.Request)
	PostUser(http.ResponseWriter, *http.Request)
	DeleteUser(http.ResponseWriter, *http.Request)
	PatchUser(http.ResponseWriter, *http.Request)
	ChangeOwnPassword(http.ResponseWriter, *http.Request)

	GetGroups(http.ResponseWriter, *http.Request)
	GetGroup(http.ResponseWriter, *http.Request)
	PostGroup(http.ResponseWriter, *http.Request)
	PatchGroup(http.ResponseWriter, *http.Request)
	DeleteGroup(http.ResponseWriter, *http.Request)
	GetGroupMembers(http.ResponseWriter, *http.Request)
	PostGroupMember(http.ResponseWriter, *http.Request)
	DeleteGroupMember(http.ResponseWriter, *http.Request)

	GetDrives(w http.ResponseWriter, r *http.Request)
	GetSingleDrive(w http.ResponseWriter, r *http.Request)
	GetAllDrives(w http.ResponseWriter, r *http.Request)
	GetRootDriveChildren(w http.ResponseWriter, r *http.Request)
	CreateDrive(w http.ResponseWriter, r *http.Request)
	UpdateDrive(w http.ResponseWriter, r *http.Request)
	DeleteDrive(w http.ResponseWriter, r *http.Request)
}

should become more meaningful...

maybe we can refactor the handlers one by one ... one challenge is that we would have to parse the query parameters in the http service but still hand them over to the business implementation. we would be implementing a protobuf like generated api and include filters and pagination. 🤔 but that would have to be able to express the odata filters.


Imputation: Last measurement carrying forward

Description of feature
The last measurement carrying forward imputation strategy can be yet another imputation strategy worth to have at hand.
A more detailed discussion on a case study available e.g. here.

duplicate of #481


Fix method call in Asset.calls property

Codecov Report

Merging #23 into master will increase coverage by 0.47%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master      #23      +/-   ##
==========================================
+ Coverage   45.94%   46.41%   +0.47%
==========================================
  Files          38       37       -1
  Lines        3104     3027      -77
==========================================
- Hits         1426     1405      -21
+ Misses       1678     1622      -56

Impacted Files
Coverage Δ

bitshares/asset.py
33.01% <0%> (ø)
:arrow_up:

bitsharesbase/bip38.py

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4018815...cea86ad. Read the comment docs.


ML4MI _2024-03-19: Advancing Foundation Models for Cellular Morphology #98

Hi @jccaicedo,
I'm working on adding a large backlog of ML4MI recordings to Nexus and was wondering if you'd be willing to create a post for your talk from March.
It would follow a format similar to Anoop’s talk, Using EHR Data to Predict Deterioration in Hospitalized Children:
https://uw-madison-datascience.github.io/ML-X-Nexus/Applications/
Here’s what we’d need:
A short paragraph summarizing your talk.
Tags that capture the relevant ML/AI categories.
An image to accompany the post (you could generate one using a tool like ChatGPT/genAI, or use a screenshot from one of your slides).
We also have a detailed guide on how to contribute. Here’s a quick summary of the process:
Typically, you’d start by posting an issue to Nexus to announce your intended contribution and get feedback. (No need this time—I’ve already created the issue).
Once feedback is received, fork the Nexus repo. (Again, you can skip this step since I’m requesting the resource).
Copy one of the template files in the templates folder (e.g., the talk template). Alternatively, you can start by copying Anoop’s talk file for reference.
Edit the template to reflect your talk.
Move the completed file to the appropriate folder (e.g., Applications/Videos for this case).
Commit and push your changes to your fork.
Start a pull request to merge your fork into the main repo.
In the future, we’d also be thrilled if you could share other AI/ML resources from your lab that might benefit AI/ML practitioners across campus. This post could serve as a great starting point for ongoing contributions.
Looking forward to your input!

@Aman13567 This looks like a repeat issue of what I've already posted. You can post an issue if there's any specific resource you'd like to add to Nexus.


Remove com from types

Expected Behavior
The types shouldn't have com
Actual Behavior
Types have com.

Actually, it's used and standard in Java libraries, so we shouldn't remove it.


PLUGIN CONFLICT: string_helper.ts

This plugin currently causes a major plugin conflict by rewriting the definition for a String and the replaceAll method (which is pretty integral) in string_helper.ts: https://github.com/sandrofigo/obsidian-pretty-bibtex/blob/master/string.extensions.ts. This should definitely not be done in this manner, I can't quite identify where this code is used to submit an FR but would it be possible to amend/remove it?
Thank you for your consideration.

Thanks for bringing this up. I will look into this as soon as possible.

@mayurankv I have removed the string extensions in release 2.0.1, after updating the plugin in your vault the conflict should be resolved.
Let me know if the problem is solved.

Works perfectly now, thank you!


Revert "Fix/logos base64 (#23)"

This reverts commit fca8bd87f11053842e00718eef2f6b3cd8ec33c8.
Description
There is an error when importing the logos.

i think is not necessary to revert, the error is for a unlocated import, in this case the import .logos, i will open a PR with that fix


[Question] VideoController.create now takes player & instead of player.handle

await VideoController.create(player); not working
please check pub.dev media_kit_video because the media_kit_video source on Github and the media_kit_video source from pub.dev are different.
[Error Log]
Error: The argument type 'Player' can't be assigned to the parameter type 'Future'.

'Player' is from 'package:media_kit/src/player.dart' ('../../.pub-cache/hosted/pub.dev/media_kit-0.0.4+1/lib/src/player.dart').
'Future' is from 'dart:async'.
controller = await VideoController.create(player);

Please don't mix the package(s) from two sources (GitHub & pub.dev).
Since the project is in very active development, there are few changes that we're making until we are 1.0.0 (#114 ).

In summary, version on GitHub (& next update) makes this change:

VideoController.create now takes player & instead of player.handle.

This is important to support Android & improve surface API. I hope that clears it.

I hope the question is answered.

Make sure you have added it to pubspec.yaml & done a hot-restart.

class HomeScreen extends StatefulWidget {
const HomeScreen({Key? key}) : super(key: key);
@override
State createState() => _HomeScreenState();
}
class _HomeScreenState extends State {
/// Create a [Player].
final Player player = Player();
/// Store reference to the [VideoController].
VideoController? controller;
@override
void initState() {
super.initState();
Future.microtask(() async {
await player.open(Media('asset:///videos/bee.mp4'));
/// Create a [VideoController] to show video output of the [Player].
controller = await VideoController.create(player.handle);
setState(() {});
});
}
@override
void dispose() {
Future.microtask(() async {
/// Release allocated resources back to the system.
await controller?.dispose();
await player.dispose();
});
super.dispose();
}
void onPlayPressed() async {
await player.play();
}
@override
Widget build(BuildContext context) {
/// Use [Video] widget to display the output.
return Stack(
children: [
GestureDetector(
onTap: onPlayPressed,
child: Video(
/// Pass the [controller].
controller: controller,
),
),
],
);
}
}
this is my code Can you tell me which part is wrong?

class HomeScreen extends StatefulWidget {
const HomeScreen({Key? key}) : super(key: key);
@override
State createState() => _HomeScreenState();
}
class _HomeScreenState extends State {
/// Create a [Player].
final Player player = Player();
/// Store reference to the [VideoController].
VideoController? controller;
@override
void initState() {
super.initState();
Future.microtask(() async {
await player.open(Media('asset:///videos/sample1.mp4'));
/// Create a [VideoController] to show video output of the [Player].
controller = await VideoController.create(player.handle);
setState(() {});
});
}
@override
void dispose() {
Future.microtask(() async {
/// Release allocated resources back to the system.
await controller?.dispose();
await player.dispose();
});
super.dispose();
}
void onPlayPressed() async {
await player.play();
}
@override
Widget build(BuildContext context) {
/// Use [Video] widget to display the output.
return Stack(
children: [
GestureDetector(
onTap: onPlayPressed,
child: Video(
/// Pass the [controller].
controller: controller,
),
),
],
);
}
}

@Bullets-jyr, you may try playing some online URL or File path.
Chances are your asset is not synced. Have you added it to pubspec.yaml?

Also, videos/bee.mp4 is different than assets/videos/sample1.mp4.
You need to add it to pubspec.yaml!!!

assets:
- assets/images/
- assets/videos/
As you said, it is added to pubspec.yaml.

await player.open(Media('assets:///videos/sample1.mp4')); I wrote the code.

https://github.com/alexmercerind/media_kit/blob/c920eba48e85e7cb49a2181783f20bb844c1d27b/media_kit_test/pubspec.yaml#L57-L62

Incorrect:
await player.open(Media('assets:///videos/sample1.mp4'));

Correct:
await player.open(Media('assets:///assets/videos/sample1.mp4'));

[Code]
class HomeScreen extends StatefulWidget {
const HomeScreen({Key? key}) : super(key: key);
@override
State createState() => _HomeScreenState();
}
class _HomeScreenState extends State {
/// Create a [Player].
final Player player = Player();
/// Store reference to the [VideoController].
VideoController? controller;
@override
void initState() {
super.initState();
Future.microtask(() async {
await player.open(Media('assets:///assets/videos/sample1.mp4'));
/// Create a [VideoController] to show video output of the [Player].
controller = await VideoController.create(player.handle);
setState(() {});
});
}
@override
void dispose() {
Future.microtask(() async {
/// Release allocated resources back to the system.
await controller?.dispose();
await player.dispose();
});
super.dispose();
}
void onPlayPressed() async {
await player.play();
print('Click onPlayPressed');
}
@override
Widget build(BuildContext context) {
/// Use [Video] widget to display the output.
return Stack(
children: [
GestureDetector(
onTap: onPlayPressed,
child: Video(
/// Pass the [controller].
controller: controller,
),
),
],
);
}
}
[pubspec.yaml]
assets:
- assets/images/
- assets/videos/sample1.mp4
Unfortunately, it doesn't play. help.

The log comes out, but the video doesn't play.

May we talk at our Discord: https://discord.gg/h7qf2R9n57 ?


Refactor dvs-agent scaffold

Agent registered in neutron-server

Please create pull-request to original repository


Wheel="false" not working

Adding the wheel="false" attribute to  seems to not do anything.

wheel="false" you pass the string "false" as the value. Try it [wheel]="false"

Oh right, it works now. Cheers.


Fix test cases in EIP-4200

"RJUMP/RJUMPI/RJUMPV as a final instruction in code section" should be removed from https://eips.ethereum.org/EIPS/eip-4200.

it's out of scope of 4200 (5450 settles what the final instruction may be)
RJUMP can be the final instruction

Fixed by https://github.com/ethereum/EIPs/pull/8595.


[BUG] an empty file \lib\swagger_dart_code_generator_temp.dart remains after successful build

Describe the bug
An empty file \lib\swagger_dart_code_generator_temp.dart is left behind after each build. I don't recall having this problem in earlier versions, but I don't know exactly when it got introduced.
To Reproduce
git clone https://github.com/epam-cross-platform-lab/swagger-dart-code-generator
cd .\swagger-dart-code-generator\example
flutter pub get
flutter packages pub run build_runner build --delete-conflicting-outputs

Expected behavior
I would expect temp files to be deleted, or be used outside of the /lib folder.
I also suggest removing this file not only in case of success, but also in case of a build failure.
Library version used (latest of everything, as of today):
OS: Windows 11
Flutter 2.10.2 • channel stable
swagger_dart_code_generator: ^2.4.2
chopper_generator: ^4.0.5
json_annotation: ^4.4.0
json_serializable: ^6.1.4

Fixed here: https://github.com/epam-cross-platform-lab/swagger-dart-code-generator/pull/353


Update failover-clustering-and-always-on-availability-groups-sql-serv…

…er.md
Corrected a minor error in an example

Learn Build status updates of commit 285b3bd:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/database-engine/availability-groups/windows/failover-clustering-and-always-on-availability-groups-sql-server.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

@MashaMSFT,
Can you review the proposed changes?
IMPORTANT: When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

awesome, thanks so much @brandonpires ! #sign-off


docker-credential-gcr not installed or not available in PATH

When starting a composer environment for the first time I got the following error:
docker.credentials.errors.InitializationError: docker-credential-gcr not installed or not available in PATH

I solved it by installing docker-credential-gcr with:
gcloud components install docker-credential-gcr

The version of the gcloud CLI I had was 416.0.0.
This should probably be added to the Prerequisites section in the README.

Hi @aarcarons ,
Is this issue still reproducible? I tried to catch this exception, but composer-dev start is working well for me when docker-credential-gcr is not installed. Perhaps, you could share some more details (image, composer-dev version, OS, etc.)?


Create tbs1.txt

Added TBS1.

@pintman While there is no problem with the tbs1.de domain, I can not find a proof of ownership for tbs1.nw.lo-net2.de. Does it belong to the same institution? Could you please help and provide such a proof? Whois database did not help in this case, but a page on the official website or a quote from a scientific paper or something similar can help. Thank you!

The domain tbs1.de is used for E-Mails from the main stuff - all teachers have E-Mails in this domain.
Our students are registered on a platform called lo-net. Therefore it would be great to have this account registered for our students as well such that they can apply for a license.
We have information for our students that link to the lo-net page. These tell how to setup their accounts (in german). https://main.tbs1.de:5443/itawiki/index.php/Lo-Net²
Could this wiki-page (which is only editable by our teachers) be a proof?

@pintman Yes, absolutely. Thank you for the clarification. Pull request merged.


feat: improve color consistency (save all labels)

SUMMARY

The difference between this PR and this one is that all label colors of the dashboard are saved to handle the native filter scenario.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

before
https://user-images.githubusercontent.com/11830681/156928847-737aceea-c80e-4f92-81bd-b3d53fbb7324.mov
after
https://user-images.githubusercontent.com/11830681/156928791-ed4ccef5-3fbc-4c42-9f68-5b61a1ec4f42.mov
TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

/testenv up

@jinghua-qa in your case it looks like they just preserved their original colors. I tested with filtring labels using different colors and it works normally. Let me know if that requires additional discussion

@jinghua-qa in your case it looks like they just preserved their original colors. I tested with filtring labels using different colors and it works normally. Let me know if that requires additional discussion

I have tried use other color theme, i will got same color for a and e, i think i am expecting different color?

I found out that the same color issue will exist when the native filter is applied:
1, select 'a', 'e' and applied native filter
2, do not remove native filter, go to dashboard edit mode and change color scheme
in this scenario, a, e always have the same color, and even clear native filter, a, e still show same color
https://user-images.githubusercontent.com/81597121/157598011-b4ab1eab-4fc5-4db7-9bb0-7d605bff8198.mov

Yes @stephenLYZ just reproduced what @jinghua-qa reported above.
Before applying filters

After applying filters

Changing color scheme and going back to the previous one

As you see now both series have the same color even if originally they didn't.

@jinghua-qa @geido This scenario is a bit tricky. Fixed here. Thanks for testing!!
https://user-images.githubusercontent.com/11830681/157720596-bd4f9b8e-e93d-4297-905b-9dd1c25d4049.mov

/testenv up

Thank you for the fix! @stephenLYZ, i verified that the native filter issue has fixed. But i found another issue that when there is color scheme or customized label applied, if user open chart in explore, user will see warning that form data is lost and the color theme or label for the chart will be lost.
https://user-images.githubusercontent.com/81597121/158010313-b9947b9d-1817-45ba-a735-27f4bd95cb23.mov

@jinghua-qa It seems to be related to the recent form_data key update,  is there any context for this? @geido @michael-s-molina

I tested this PR locally, found another issue that, when applied native filter in different series of the same color, after filtered the it will still show same color of different series, is this acceptable?
https://user-images.githubusercontent.com/81597121/158527008-4162219e-1bd1-45eb-9877-2d0924d4fcf9.mov

I tested this PR locally, found another issue that, when applied native filter in different series of the same color, after filtered the it will still show same color of different series, is this acceptable?

@jinghua-qa I think this is the expected behavior and objective of this PR. If you check before/after videos you'll see the same example.

@geido do you want to give this a final code pass? Sounds like we're about to have a beast of a problem behind us!

Spinning up another test env. The previous one does not work for me

/testenv up

Hello @stephenLYZ after testing an existing dashboard with a color scheme and no custom label colors, I found that some labels are not sharing the same colors even if they should?
https://user-images.githubusercontent.com/60598000/159033589-c75e072a-f791-4c50-80cf-edd297e241b4.mp4

@geido Nice catch!  Fixed here.

/testenv up

@stephenLYZ I am worried that something happened to the ability to bring the colors of a chart from its Dashboard scheme to Explore. As you can see the colors in Explore look different from the Dashboard.
https://user-images.githubusercontent.com/60598000/159063652-36c0b258-bc0f-4ff2-9ed8-5b06b4405a39.mp4

@geido  You can see Michael's comment, it only exists in the test environment.

@stephenLYZ I think I know what's the problem here. This https://github.com/apache/superset/pull/18976 changed the default cache type to NullCache when not in debug mode. I think this is affecting the test environments. @villebro do you know how can we change this config for test environments?

Hi, we've had some reports of unexpected colors in larger dashboards. From looking at this PR, I think this is because new colors are autogenerated once you "run out" of colors in the color scheme (correct me if I'm wrong?). I understand this is to prevent the same color being used twice, but non-unique dashboard colors are often not necessary if there are no conflicts within a single chart.
I'm instructing users to override colors with label_colors if they don't like the autogenerated "similar to color scheme" ones, but this is a pretty big hassle, and it's a bit confusing because we don't surface shared_label_colors in the UI.
Thoughts?
@zhaoyongjie @stephenLYZ

What's particularly disruptive about this change is that it seems to cause colors to get auto-picked that are outside the defined color scheme. It also seems unclear what causes one chart to get a color auto-assigned vs another that has a color from the palette.

Hi, we've had some reports of unexpected colors in larger dashboards. From looking at this PR, I think this is because new colors are autogenerated once you "run out" of colors in the color scheme (correct me if I'm wrong?). I understand this is to prevent the same color being used twice, but non-unique dashboard colors are often not necessary if there are no conflicts within a single chart.
I'm instructing users to override colors with label_colors if they don't like the autogenerated "similar to color scheme" ones, but this is a pretty big hassle, and it's a bit confusing because we don't surface shared_label_colors in the UI.
Thoughts? @zhaoyongjie @stephenLYZ

@serenajiang Hi, thanks for the feedback. This PR is not only to prevent color conflicts caused by "run out" of colors in the color-scheme, but mainly to ensure that the same dimensions of all charts in the dashboard have the same color.
The shared_label_colors is transparent to the user. It stores the colorMap({[chartId]: { [label]: color }}) of all charts in the dashboard, and if different charts have the same dimensions, these dimensions will be extracted to generate analogous colors of color scheme, which is to prevent color conflicts.
The question is why we need to extract these same dimensions and use analogous colors. In this case, if we use the remaining color of color-scheme for the same dimensions in the one chart,  we can't be sure whether the remaining color has been used in another chart, which may lead to color conflicts in another chart.
For example, one dashboard has chartA and chartB. The color scheme is ['red', 'yellow', 'blue']. The chartA has two dimensions ['a', 'b'] and chartB has three dimensions ['b', 'c', 'e'].
As before the color map is
chart A
{
  a: 'red',
  b: 'yellow',
}

chart B
{
  b: 'red',
  c: 'yellow',
  e: 'blue',
}

But we need to make sure that dimension b has the same color in the two charts. If we both use yellow as the color value for dimension b,  it will cause a color conflict for chartB. That's the problem I want to solve.

Hi @stephenLYZ, the use case you're describing makes sense - keeping chart dimension colors consistent. This was already possible via label_colors. The issue we're running into is that on a dashboard with a number of different charts, new charts suddenly get assigned random colors outside of Airbnb's color palette. This is unexpected and undesirable for our users and will ultimately hurt adoption of our dashboards.
What we're observing is also that charts that do not have dimensional cuts (i.e. simple single metric time-series charts without group-bys or multiple series), which in the past have all been assigned the first value in the color palette, now get assigned a new, non-palette, color value, despite having no dimensional cuts.
Would it be possible to make the behavior you've implemented optional at the dashboard level? E.g. "Cycle through dashboard theme colors: Yes/No" with "No" describing the behavior you've implemented?

Thanks @stephenLYZ for the thorough explanation, I see without generating colors it'd be hard to totally avoid these conflicts, and trying to fine-tune it might get really messy. However, for our deployment, I'm not sure whether we need to optimize that hard, since users can always manually override label colors. Airbnb in particular has strong loyalty to our color scheme, so unexpected colors can be a bit jarring. Making this behavior optional for each dashboard could help mitigate disruptions, or even putting it all behind a feature flag (is there a feature flag for this already?).
Re:

The shared_label_colors is transparent to the user.

Would you mind pointing me to how to view this in the UI? I couldn't view it in the json metadata when I tried to edit a dashboard, and only found it because it was in the API response.

Heya @clausherther and @serenajiang - thanks for raising your concerns. This is an intricate problem, but I'm confident we can find a sensible way forward so everyone gets what they want here. There are pros and cons to pretty much any approach, which I'll attempt to lay out here.
As a bit of a side note, we sometimes find label_colors to be an imperfect solution since (a) drill-down/up/across are coming, and across-the-board consistency will be of increasing importance, (b) we saw cases where people make a particular series a specific color from the palette, and then a neighboring line or pie slice was also assigned to be that color, and (c) as a feature, it's a little buried and cumbersome in general for wide adoption.
So... i think I see a couple paths forward, with varying levels of difficulty. Here goes...

Add an app configuration (or dare I say Feature Flag?) for USE_ANALAGOUS_COLORS_IN_DASHBOARDS which, when enabled, would act as it does today. If you turn it off, it would simply loop through the palette colors repeatedly. Default of that config is up for debate.

True would mean it works like it does today, generating colors as needed:

Upsides:

Unique color per-series, meaning there will be fewer issues disambiguating charts
Features like cross-filters (added support for this nascent feature coming soon) and drill-down/up/across will be visually coherent

Downsides:

Depending on size of dashboard and number of series in it, there may be a proliferation of colors

False would mean the dashbpard cycles repeatedly through the specified color palette.

Upsides:

Only official pallete colors get used

Downsides:

You may see the same color multiple times in a chart (depending on the number of series in the chart vs the number of colors in the palette
A color may be assigned (via label_coors or naturally) the identical color as its neighbor, causing confusion
A given series in a dashbard will have a consistent color, but not a unique color, so it may be ambiguous or misleading in some use cases

This is a less appealing idea since it has many potential issues, but we have heard of use cases for it, so here goes... we could add another configuration option for CONSISTENT_SERIES_COLORS_IN_DASHBOARDS which defaults to True to support either of the above treatments. If set to False however, the dashboard would ignore chart-to-chart consistency entirely, and just start at the beginning of the color palette in each chart. Then the first color in the palette (sometimes a brand color) is always the first color selected for a chart, and thus the dominant color in many dashboards.

Hope this helps, and hope we can steer this so that everyone wins :D


ROCm: fix rocBLAS and rocSOLVER version displays

Follow up of #3950. For some reason I didn't catch the bug that when calling cupy.show_info() CUSOLVERError was raised (perhaps my local build was dirty):
>>> import cupy as cp
>>> cp.show_config()
Traceback (most recent call last):
  File "cupy_backends/cuda/libs/cusolver.pyx", line 722, in cupy_backends.cuda.libs.cusolver.check_status
    raise CUSOLVERError(status)
cupy_backends.cuda.libs.cusolver.CUSOLVERError: rocblas_status_not_implemented
Exception ignored in: 'cupy_backends.cuda.libs.cusolver.getProperty'
Traceback (most recent call last):
  File "cupy_backends/cuda/libs/cusolver.pyx", line 722, in cupy_backends.cuda.libs.cusolver.check_status
    raise CUSOLVERError(status)
cupy_backends.cuda.libs.cusolver.CUSOLVERError: rocblas_status_not_implemented
Traceback (most recent call last):
  File "cupy_backends/cuda/libs/cusolver.pyx", line 722, in cupy_backends.cuda.libs.cusolver.check_status
    raise CUSOLVERError(status)
cupy_backends.cuda.libs.cusolver.CUSOLVERError: rocblas_status_not_implemented
Exception ignored in: 'cupy_backends.cuda.libs.cusolver.getProperty'
Traceback (most recent call last):
  File "cupy_backends/cuda/libs/cusolver.pyx", line 722, in cupy_backends.cuda.libs.cusolver.check_status
    raise CUSOLVERError(status)
cupy_backends.cuda.libs.cusolver.CUSOLVERError: rocblas_status_not_implemented
Traceback (most recent call last):
  File "cupy_backends/cuda/libs/cusolver.pyx", line 722, in cupy_backends.cuda.libs.cusolver.check_status
    raise CUSOLVERError(status)
cupy_backends.cuda.libs.cusolver.CUSOLVERError: rocblas_status_not_implemented
Exception ignored in: 'cupy_backends.cuda.libs.cusolver.getProperty'
Traceback (most recent call last):
  File "cupy_backends/cuda/libs/cusolver.pyx", line 722, in cupy_backends.cuda.libs.cusolver.check_status
    raise CUSOLVERError(status)
cupy_backends.cuda.libs.cusolver.CUSOLVERError: rocblas_status_not_implemented
CuPy Version          : 8.0.0rc1
CUDA Root             : /usr
CUDA Build Version    : 0
CUDA Driver Version   : 313700
CUDA Runtime Version  : 3137
cuBLAS Version        : CUBLASError('HIPBLAS_STATUS_NOT_SUPPORTED')
cuFFT Version         : 32673
cuRAND Version        : 201001
cuSOLVER Version      : (0, 0, 0)
cuSPARSE Version      : 0
NVRTC Version         : (9, 0)
Thrust Version        : 100902
CUB Build Version     : None
cuDNN Build Version   : None
cuDNN Version         : None
NCCL Build Version    : None
NCCL Runtime Version  : None
cuTENSOR Version      : None

It is now fixed in this PR. Also, the version number for rocBLAS is computed and printed:
>>> import cupy as cp
>>> cp.show_config()
CuPy Version          : 8.0.0rc1
CUDA Root             : /usr
CUDA Build Version    : 0
CUDA Driver Version   : 313700
CUDA Runtime Version  : 3137
cuBLAS Version        : 22200
cuFFT Version         : 0
cuRAND Version        : 201001
cuSOLVER Version      : (3, 5, 0)
cuSPARSE Version      : 0
NVRTC Version         : (9, 0)
Thrust Version        : 100902
CUB Build Version     : None
cuDNN Build Version   : None
cuDNN Version         : None
NCCL Build Version    : None
NCCL Runtime Version  : None
cuTENSOR Version      : None

Jenkins, test this please

Jenkins CI test (for commit 03330b896ac38f8a5b8337f7b32d1bba7f64fe36, target branch master) succeeded!


'StderrCatcher' object has no attribute 'isatty'

Error throw in current beta

Hmmm i can't seem to reproduce it here - there shouldn't be any references to isatty


Update README.md

Hello, thank you for your suggestions. We've made some modifications in this PR. Best wishes.


Change CI to julia 1.5

Codecov Report

Merging #78 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #78   +/-   ##
=======================================
  Coverage   93.18%   93.18%
=======================================
  Files          13       13
  Lines        1013     1013
=======================================
  Hits          944      944
  Misses         69       69

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8510bc2...c55c355. Read the comment docs.