[Question] Taxes

Hi, I have 2 question:

where can i find documentation about taxes?
Use Cart Modifiers to handle tax in documentation is empty.

It is possible to have different tax value to different product?

Taxes are handled using a CartModifier. There are two samples on how to handle taxes, one for the European model and one for the U.S. model, but you can handle them, however you want.
If you need different rates for each product, then I would add a tag to the product model and
write your own TaxCartModifier.
I admit that this should be better documented.

Thanks for the clarification.
I'm waiting for the documentation update.

just updated docs/howtos/handling-taxes.rst
please review and close

Jacob, thank you very much for completing the documentation.
I will test these methods, I will let you know how they are doing.


Updating a row with null value

I don't know if I am missing something but if I try to update a row with some columns as null, it doesn't seem to be reflected

StorIO simply passes ContentValues to SQLiteDatabase or ContentResolver methods…
We'll write test for that and see if it's real problem or not, I believe we put null in our apps without problems.

@nikitin-da can you please take this? It may affect apps you're working on :)
(I'm pretty busy unfortunately…)

@artem-zinnatullin, sure, I'll take it tomorrow


runtime: crash on linux/arm64 with "invalid p state" on 1.13

What version of Go are you using (go version)?

$ go version
go version go1.13.10 linux/arm64

Does this issue reproduce with the latest release?
Not happened with 1.14, but also in 1.13.11
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/data/choury/.cache/go-build"
GOENV="/data/choury/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/data/choury/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/data/choury/.local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/data/choury/.local/go/pkg/tool/linux_arm64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build335762780=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Run this code
package main

import (
        "io/ioutil"
        "math/rand"
        "os/exec"
        "time"
)

func main() {
        for i := 0; i < 100; i++ {
                go func() {
                        for {
                                d := time.Duration(rand.Intn(100)+1) * time.Millisecond
                                time.Sleep(d)
                                cmd := exec.Command("ls")
                                cmd.CombinedOutput()
                        }
                }()
                go func() {
                        for {
                                d := time.Duration(rand.Intn(100)+1) * time.Millisecond
                                time.Sleep(d)
                                ioutil.ReadFile("/proc/cpuinfo")
                        }
                }()
        }
        select {}
}

What did you expect to see?
No error, run forever...
What did you see instead?
crash with:
releasep: m=0x4000f82000 m->p=0x400003a500 p->m=0x0 m->mcache=0xfd4da32dac00 p->mcache=0xfd4da32dac00 p->status=0
fatal error: releasep: invalid p state
runtime: newstack at os/exec.(*Cmd).Run+0x78 sp=0x4000baef60 stack=[0x4000bae000, 0x4000baf000]
        morebuf={pc:0x88f2c sp:0x4000baef60 lr:0x0}
        sched={pc:0x883a8 sp:0x4000baef60 lr:0x88f2c ctxt:0x0}
os/exec.(*Cmd).CombinedOutput(0x4000934b00, 0x2, 0x0, 0x0, 0x0, 0x4000934b00)
        /usr/local/go/src/os/exec/exec.go:566 +0x8c fp=0x4000baef90 sp=0x4000baef60 pc=0x88f2c
main.main.func1()
        /crash/main.go:17 +0x80 fp=0x4000baefd0 sp=0x4000baef90 pc=0x8a0a0
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm64.s:1128 +0x4 fp=0x4000baefd0 sp=0x4000baefd0 pc=0x60494
created by main.main
        /crash/main.go:12 +0x3c
fatal error: runtime: stack split at bad time
signal: segmentation fault (core dumped)

If I’m understanding correctly, this reproduces reliably on 1.13, but not 1.14? Is that right?

@aclements

If I’m understanding correctly, this reproduces reliably on 1.13, but not 1.14? Is that right?

Yes

@choury I tried the case on various arm64 machines with 'go1.13.11' but it didn't crash after 10min or so, how long should I expect the error to happen? Once ran into the same error with 'go tool dist test' but never reproduced it again, so want to try with your case. Thanks.

@choury I tried the case on various arm64 machines with 'go1.13.11' but it didn't crash after 10min or so, how long should I expect the error to happen? Once ran into the same error with 'go tool dist test' but never reproduced it again, so want to try with your case. Thanks.

go tool dist test:
##### Testing packages.
ok  	archive/tar	0.189s
ok  	archive/zip	0.508s
ok  	bufio	0.632s
ok  	bytes	0.697s
ok  	compress/bzip2	0.087s
ok  	compress/flate	0.677s
ok  	compress/gzip	1.007s
ok  	compress/lzw	0.517s
ok  	compress/zlib	0.560s
ok  	container/heap	0.026s
ok  	container/list	0.507s
ok  	container/ring	0.570s
ok  	context	0.988s
ok  	crypto	0.554s
ok  	crypto/aes	0.586s
ok  	crypto/cipher	0.552s
ok  	crypto/des	0.574s
ok  	crypto/dsa	0.027s
ok  	crypto/ecdsa	0.160s
ok  	crypto/ed25519	0.067s
ok  	crypto/elliptic	0.593s
ok  	crypto/hmac	0.588s
ok  	crypto/internal/subtle	0.583s
ok  	crypto/md5	0.523s
ok  	crypto/rand	0.063s
ok  	crypto/rc4	1.188s
ok  	crypto/rsa	0.591s
ok  	crypto/sha1	0.054s
ok  	crypto/sha256	0.550s
ok  	crypto/sha512	0.470s
ok  	crypto/subtle	0.571s
ok  	crypto/tls	0.776s
ok  	crypto/x509	0.423s
ok  	database/sql	0.588s
ok  	database/sql/driver	0.577s
ok  	debug/dwarf	1.128s
ok  	debug/elf	0.149s
ok  	debug/gosym	0.507s
ok  	debug/macho	0.077s
ok  	debug/pe	0.564s
ok  	debug/plan9obj	0.016s
ok  	encoding/ascii85	0.565s
ok  	encoding/asn1	0.575s
ok  	encoding/base32	0.542s
ok  	encoding/base64	0.549s
ok  	encoding/binary	0.547s
ok  	encoding/csv	0.543s
ok  	encoding/gob	0.062s
ok  	encoding/hex	1.123s
ok  	encoding/json	0.085s
ok  	encoding/pem	0.046s
ok  	encoding/xml	0.543s
ok  	errors	0.545s
ok  	expvar	0.016s
ok  	flag	0.026s
ok  	fmt	0.570s
ok  	go/ast	0.263s
ok  	go/build	0.143s
ok  	go/constant	0.554s
ok  	go/doc	0.600s
ok  	go/format	0.020s
ok  	go/importer	0.371s
ok  	go/internal/gccgoimporter	0.038s
ok  	go/internal/gcimporter	0.606s
ok  	go/internal/srcimporter	1.178s
ok  	go/parser	0.615s
ok  	go/printer	0.856s
ok  	go/scanner	0.543s
ok  	go/token	0.233s
ok  	go/types	1.016s
ok  	hash	0.520s
ok  	hash/adler32	0.563s
ok  	hash/crc32	0.534s
ok  	hash/crc64	0.072s
ok  	hash/fnv	0.019s
ok  	html	0.051s
ok  	html/template	0.527s
ok  	image	0.112s
ok  	image/color	0.581s
ok  	image/draw	0.551s
ok  	image/gif	0.865s
ok  	image/jpeg	0.733s
ok  	image/png	0.550s
ok  	index/suffixarray	0.702s
ok  	internal/cpu	0.146s
ok  	internal/fmtsort	0.558s
ok  	internal/poll	0.077s
ok  	internal/reflectlite	0.554s
ok  	internal/singleflight	0.581s
ok  	internal/trace	0.621s
ok  	internal/xcoff	0.560s
ok  	io	0.635s
ok  	io/ioutil	0.593s
ok  	log	0.509s
ok  	log/syslog	1.251s
ok  	math	0.521s
ok  	math/big	0.840s
ok  	math/bits	0.208s
ok  	math/cmplx	0.536s
ok  	math/rand	0.568s
ok  	mime	0.515s
ok  	mime/multipart	0.194s
ok  	mime/quotedprintable	0.298s
ok  	net	2.931s
ok  	net/http	2.843s
ok  	net/http/cgi	0.253s
ok  	net/http/cookiejar	0.023s
ok  	net/http/fcgi	0.036s
ok  	net/http/httptest	0.040s
ok  	net/http/httptrace	0.016s
ok  	net/http/httputil	0.061s
ok  	net/http/internal	0.575s
ok  	net/http/pprof	2.021s
ok  	net/internal/socktest	0.591s
ok  	net/mail	0.025s
ok  	net/rpc	0.036s
ok  	net/rpc/jsonrpc	0.023s
ok  	net/smtp	0.050s
ok  	net/textproto	0.158s
ok  	net/url	0.107s
ok  	os	2.131s
ok  	os/exec	0.593s
ok  	os/signal	4.676s
ok  	os/user	0.145s
ok  	path	0.023s
ok  	path/filepath	0.526s
ok  	reflect	0.745s
ok  	regexp	0.646s
ok  	regexp/syntax	0.866s
ok  	runtime	14.929s
ok  	runtime/debug	0.071s
ok  	runtime/internal/atomic	0.530s
ok  	runtime/internal/math	0.040s
ok  	runtime/internal/sys	0.043s
ok  	runtime/pprof	5.884s
ok  	runtime/pprof/internal/profile	0.548s
ok  	runtime/trace	0.715s
ok  	sort	0.534s
ok  	strconv	1.010s
ok  	strings	0.578s
ok  	sync	0.853s
ok  	sync/atomic	0.687s
ok  	syscall	0.073s
ok  	testing	0.658s
ok  	testing/quick	0.573s
ok  	text/scanner	0.521s
ok  	text/tabwriter	0.514s
ok  	text/template	0.502s
ok  	text/template/parse	0.470s
ok  	time	3.062s
ok  	unicode	0.465s
ok  	unicode/utf16	0.524s
ok  	unicode/utf8	0.475s
ok  	cmd/addr2line	1.679s
ok  	cmd/api	0.543s
ok  	cmd/asm/internal/asm	0.690s
ok  	cmd/asm/internal/lex	0.032s
ok  	cmd/compile	0.016s
ok  	cmd/compile/internal/gc	6.612s
ok  	cmd/compile/internal/ssa	0.763s
ok  	cmd/compile/internal/syntax	0.530s
ok  	cmd/compile/internal/test	0.536s [no tests to run]
ok  	cmd/compile/internal/types	0.260s
ok  	cmd/cover	1.974s
ok  	cmd/doc	0.332s
ok  	cmd/fix	0.620s
ok  	cmd/go	33.515s
ok  	cmd/go/internal/auth	0.014s
ok  	cmd/go/internal/cache	0.623s
ok  	cmd/go/internal/dirhash	0.463s
ok  	cmd/go/internal/generate	0.045s
ok  	cmd/go/internal/get	0.035s
ok  	cmd/go/internal/imports	0.229s
ok  	cmd/go/internal/load	0.119s
ok  	cmd/go/internal/lockedfile	0.217s
ok  	cmd/go/internal/lockedfile/internal/filelock	0.311s
ok  	cmd/go/internal/modconv	0.027s
ok  	cmd/go/internal/modfetch	0.022s
ok  	cmd/go/internal/modfetch/codehost	0.243s
ok  	cmd/go/internal/modfile	0.335s
ok  	cmd/go/internal/modload	0.022s
ok  	cmd/go/internal/module	0.322s
ok  	cmd/go/internal/mvs	0.536s
ok  	cmd/go/internal/note	0.238s
ok  	cmd/go/internal/par	0.331s
ok  	cmd/go/internal/renameio	0.311s
ok  	cmd/go/internal/search	0.221s
ok  	cmd/go/internal/semver	0.428s
ok  	cmd/go/internal/sumweb	0.053s
ok  	cmd/go/internal/tlog	0.331s
ok  	cmd/go/internal/txtar	0.274s
ok  	cmd/go/internal/web	0.031s
ok  	cmd/go/internal/work	0.113s
ok  	cmd/gofmt	0.251s
ok  	cmd/internal/buildid	0.370s
ok  	cmd/internal/dwarf	0.061s
ok  	cmd/internal/edit	0.169s
ok  	cmd/internal/goobj	0.891s
ok  	cmd/internal/obj	0.028s
ok  	cmd/internal/obj/arm64	0.221s
ok  	cmd/internal/obj/x86	1.875s
ok  	cmd/internal/objabi	0.039s
ok  	cmd/internal/src	0.140s
ok  	cmd/internal/test2json	0.234s
ok  	cmd/link	6.006s
ok  	cmd/link/internal/ld	0.758s
ok  	cmd/link/internal/sym	0.091s
ok  	cmd/nm	2.166s
ok  	cmd/objdump	1.833s
ok  	cmd/pack	1.278s
ok  	cmd/trace	0.043s
ok  	cmd/vet	3.957s

##### os/user with tag osusergo
ok  	os/user	0.003s

##### GOMAXPROCS=2 runtime -cpu=1,2,4 -quick
ok  	runtime	4.667s

##### cmd/go terminal test
skipping terminal test; stdout/stderr not terminals

##### Testing without libgcc.
ok  	crypto/x509	0.325s
ok  	net	0.011s
ok  	os/user	0.005s

##### internal linking of -buildmode=pie
ok  	reflect	1.488s
ok  	os/user	0.337s

##### sync -cpu=10
ok  	sync	0.277s

##### Testing race detector
ok  	runtime/race	10.073s
ok  	flag	1.016s
ok  	net	1.082s
ok  	os	1.093s
ok  	os/exec	3.042s
ok  	encoding/gob	1.018s
ok  	flag	1.016s
ok  	os/exec	3.045s

##### ../misc/cgo/stdio
PASS

##### ../misc/cgo/life
PASS

##### ../misc/cgo/test
PASS
ok  	misc/cgo/test	3.219s
PASS
ok  	misc/cgo/test	3.107s
PASS
ok  	misc/cgo/test	3.305s
PASS
ok  	misc/cgo/testtls	0.005s
PASS
ok  	misc/cgo/testtls	0.005s
PASS
ok  	misc/cgo/testtls	0.005s
PASS
ok  	misc/cgo/nocgo	0.004s
PASS
ok  	misc/cgo/nocgo	0.005s
PASS
ok  	misc/cgo/nocgo	0.005s
PASS
ok  	misc/cgo/test	3.081s
PASS
ok  	misc/cgo/test	3.257s
PASS
ok  	misc/cgo/test	3.063s
PASS
ok  	misc/cgo/testtls	0.005s
PASS
ok  	misc/cgo/nocgo	0.006s

##### ../misc/cgo/testgodefs

##### ../misc/cgo/testso
ok  	misc/cgo/testso	0.820s

##### ../misc/cgo/testsovar
ok  	misc/cgo/testsovar	0.904s

##### ../misc/cgo/testcshared
SKIP - short mode and $GO_BUILDER_NAME not set

##### ../misc/cgo/testshared
ok  	misc/cgo/testshared	31.782s

##### ../misc/cgo/testsanitizers
PASS

##### ../misc/cgo/errors
PASS

##### ../misc/cgo/testsigfwd

##### ../test/bench/go1

##### ../test

##### API check
Go version is "go1.13.11", ignoring -next /usr/local/go/api/next.txt

ALL TESTS PASSED

It will crash within 10 min in docker golang:1.13.11 image on my machine.

@choury Thanks, I verified the case in golang:1.13.11 containers on two different models of machine, it didn't crash after dozens of minutes. Will update later if I'm able to reproduce the issue.

crash.100.goroutines.txt
crash.1000.Goroutines.txt
crash.10000.Goroutines.abridged.txt
crash.GOTRACEBACK=crash.txt
The case from @choury produced random, and different, crashes with go 1.13.12 (the # of user goroutines ranges between 1002 - 100002), enclosing several log files.
@cherrymui @ianlancetaylor @andybons @aclements  , could you please help to take a look?

wondering if it's related to https://github.com/golang/go/issues/32912, whose fix is not in 1.13.12, trying to verify.

After cherry-picking the following CLs to 1.13.12 we didn't observe any crash with multiple rounds of testing against the sample case, each running 24 hours at least.
Further verifying in production environment is ongoing and no issue was observed in the past week.
https://go-review.googlesource.com/c/go/+/192937
https://go-review.googlesource.com/c/go/+/204519
https://go-review.googlesource.com/c/go/+/189418
It's still a WA fix and the root cause is not clear to us. So far, the issue is specific to an in-house machine only, here are more details:
After applying CL #192937 and #204519, the sample case still crashed randomly but was due to nil 'm' of the current 'g' all the time,  then with CL #189418 the issue was gone.
Typical stack traces of the nil 'm' issue:

3218            msigrestore(getg().m.sigmask)
Loading Go Runtime support.
(gdb) bt
#0  0x0000000000040a34 in syscall.runtime_AfterForkInChild () at /opt/tools/installed/go/src/runtime/proc.go:3218
#1  0x0000000000070760 in syscall.forkAndExecInChild1 (argv0=0x4000ca60f0 "/usr/bin/ls", argv=<error reading variable: access outside bounds of object referenced via synthetic pointer>,
    envv=<error reading variable: access outside bounds of object referenced via synthetic pointer>, chroot=0x0, dir=0x0, attr=0x40002a4d40, sys=0x156a20 <syscall.zeroSysProcAttr>, pipe=97, r1=0, err1=0, p=..., locked=136)
    at /opt/tools/installed/go/src/syscall/exec_linux.go:225
#2  0x0000000000070390 in syscall.forkAndExecInChild (argv0=0x4000ca60f0 "/usr/bin/ls", argv= []*uint8 = {...}, envv= []*uint8 = {...}, chroot=0x0, dir=0x0, attr=0x40002a4d40, sys=0x156a20 <syscall.zeroSysProcAttr>,
    pipe=97, pid=<optimized out>, err=<optimized out>) at /opt/tools/installed/go/src/syscall/exec_linux.go:72
#3  0x00000000000722f0 in syscall.forkExec (argv0="/usr/bin/ls", argv= []string = {...}, attr=0x40002a4d40, pid=<optimized out>, err=...) at /opt/tools/installed/go/src/syscall/exec_unix.go:201
#4  0x000000000007de84 in syscall.StartProcess (argv0="/usr/bin/ls", argv= []string = {...}, pid=0, err=..., attr=<optimized out>, handle=<optimized out>) at /opt/tools/installed/go/src/syscall/exec_unix.go:248
#5  os.startProcess (name="/usr/bin/ls", argv= []string = {...}, attr=0x40002a4ee0, p=0x0, err=...) at /opt/tools/installed/go/src/os/exec_posix.go:52
#6  0x000000000007dbb4 in os.StartProcess (name="/usr/bin/ls", argv= []string = {...}, attr=0x40002a4ee0, ~r3=<optimized out>, ~r4=...) at /opt/tools/installed/go/src/os/exec.go:102
#7  0x0000000000088884 in os/exec.(*Cmd).Start (c=0x40002ac9a0, ~r0=...) at /opt/tools/installed/go/src/os/exec/exec.go:421
#8  0x0000000000088468 in os/exec.(*Cmd).Run (c=0x40002ac9a0, ~r0=...) at /opt/tools/installed/go/src/os/exec/exec.go:339
#9  0x000000000008903c in os/exec.(*Cmd).CombinedOutput (c=0x40002ac9a0, ~r0=..., ~r1=...) at /opt/tools/installed/go/src/os/exec/exec.go:566
#10 0x000000000008a1b0 in main.main.func1 () at /root/crash.go:17
#11 0x0000000000060554 in runtime.goexit () at /opt/tools/installed/go/src/runtime/asm_arm64.s:1128

Dump of assembler code for function syscall.runtime_AfterForkInChild:
   0x0000000000040a10 <+0>:     str     x30, [sp,#-64]!
   0x0000000000040a14 <+4>:     str     x29, [sp,#-8]
   0x0000000000040a18 <+8>:     sub     x29, sp, #0x8
   0x0000000000040a1c <+12>:    orr     x0, xzr, #0x1
   0x0000000000040a20 <+16>:    adrp    x27, 0x171000 <runtime.trace+62944>
   0x0000000000040a24 <+20>:    add     x27, x27, #0xb08
   0x0000000000040a28 <+24>:    strb    w0, [x27]
   0x0000000000040a2c <+28>:    bl      0x4ad60 <runtime.clearSignalHandlers>
   0x0000000000040a30 <+32>:    ldr     x0, [x28,#48]
=> 0x0000000000040a34 <+36>:    ldr     x0, [x0,#128]
   0x0000000000040a38 <+40>:    str     x0, [sp,#48]
   0x0000000000040a3c <+44>:    nop
   0x0000000000040a40 <+48>:    orr     x0, xzr, #0x2
   0x0000000000040a44 <+52>:    str     w0, [sp,#8]
   0x0000000000040a48 <+56>:    add     x0, sp, #0x30
   0x0000000000040a4c <+60>:    str     x0, [sp,#16]
   0x0000000000040a50 <+64>:    str     xzr, [sp,#24]
   0x0000000000040a54 <+68>:    orr     x0, xzr, #0x8
   0x0000000000040a58 <+72>:    str     w0, [sp,#32]
   0x0000000000040a5c <+76>:    bl      0x61110 <runtime.rtsigprocmask>
   0x0000000000040a60 <+80>:    adrp    x27, 0x171000 <runtime.trace+62944>
   0x0000000000040a64 <+84>:    add     x27, x27, #0xb08
   0x0000000000040a68 <+88>:    strb    wzr, [x27]
   0x0000000000040a6c <+92>:    ldr     x29, [sp,#-8]
   0x0000000000040a70 <+96>:    ldr     x30, [sp],#64
   0x0000000000040a74 <+100>:   ret
   0x0000000000040a78 <+104>:   .inst   0x00000000 ; undefined
   0x0000000000040a7c <+108>:   .inst   0x00000000 ; undefined
End of assembler dump.
(gdb) x $x28+48
0x40001eb530:   0x0000000000000000

"1346            gcw := &getg().m.p.ptr().gcw" of runtime.gcmarknewobject runs into the issue.
(gdb) bt
#0  runtime.systemstack () at /root/xd/go.hw/src/runtime/asm_arm64.s:193
#1  0x000000000003735c in runtime.throw (s="g.m == nil") at /root/xd/go.hw/src/runtime/panic.go:769
#2  0x000000000004b3d0 in runtime.sigtrampgo (sig=<optimized out>, info=0x4000294da0, ctx=0x4000294e20) at /root/xd/go.hw/src/runtime/signal_unix.go:332
#3  0x0000000000061288 in runtime.sigtramp () at /root/xd/go.hw/src/runtime/sys_linux_arm64.s:396
#4  <signal handler called>
#5  0x000000000002b854 in runtime.gcmarknewobject (obj=274882952224, size=96, scanSize=80) at /root/xd/go.hw/src/runtime/mgcmark.go:1346
#6  0x000000000001a804 in runtime.mallocgc (size=96, typ=0xb6de0, needzero=true, ~r3=<optimized out>) at /root/xd/go.hw/src/runtime/malloc.go:1079
#7  0x000000000001add8 in runtime.newobject (typ=0xb6de0, ~r1=<optimized out>) at /root/xd/go.hw/src/runtime/malloc.go:1151
#8  0x000000000007f06c in os.newFile (fd=273, name="/proc/cpuinfo", kind=1, ~r3=<optimized out>) at /root/xd/go.hw/src/os/file_unix.go:116
#9  0x000000000007f37c in os.openFileNolog (name="/proc/cpuinfo", flag=0, perm=0, ~r3=<optimized out>, ~r4=...) at /root/xd/go.hw/src/os/file_unix.go:222
#10 0x000000000007ef14 in os.OpenFile (name="/proc/cpuinfo", flag=0, perm=0, ~r3=<optimized out>, ~r4=...) at /root/xd/go.hw/src/os/file.go:300
#11 0x00000000000857a8 in os.Open (name="/proc/cpuinfo") at /root/xd/go.hw/src/os/file.go:280
#12 io/ioutil.ReadFile (filename="/proc/cpuinfo", ~r1=..., ~r2=...) at /root/xd/go.hw/src/io/ioutil/ioutil.go:53
#13 0x000000000008a25c in main.main.func2 () at /root/xd/0803/go.hw.2/crash.go:24
#14 0x0000000000060584 in runtime.goexit () at /root/xd/go.hw/src/runtime/asm_arm64.s:1128
(gdb) f 5
#5  0x000000000002b854 in runtime.gcmarknewobject (obj=274882952224, size=96, scanSize=80) at /root/xd/go.hw/src/runtime/mgcmark.go:1346
1346            gcw := &getg().m.p.ptr().gcw

   0x000000000002b84c <+60>:    cbnz    x27, 0x2b840 <runtime.gcmarknewobject+48>
   0x000000000002b850 <+64>:    ldr     x0, [x28,#48]
=> 0x000000000002b854 <+68>:    ldr     x0, [x0,#208]
   0x000000000002b858 <+72>:    ldrsb   x27, [x0]
   0x000000000002b85c <+76>:    ldr     x1, [x0,#4736]
   0x000000000002b860 <+80>:    ldr     x2, [sp,#64]

(gdb) p $x0
$3 = 0
(gdb) p/x $x28
$6 = 0x4000211e00
(gdb) p 'runtime.allgs'.array[196]
$7 = (runtime.g *) 0x4000211e00
(gdb) p/x *'runtime.allgs'.array[196]
$10 = {
  stack = {
    lo = 0x40009c9800,
    hi = 0x40009ca000
  },
  stackguard0 = 0x40009c9b70,
  stackguard1 = 0xffffffffffffffff,
  _panic = 0x0,
  _defer = 0x0,
  m = 0x0,
  sched = {
    sp = 0x40009c9f10,
    pc = 0x38f80,
    g = 0x4000211e00,
    ctxt = 0x0,
    ret = 0x0,
    lr = 0x0,
    bp = 0x40009c9f08
  },
  syscallsp = 0x0,
  syscallpc = 0x73e90,
  stktopsp = 0x40009c9fd0,
  param = 0x0,
  atomicstatus = 0x4,
  stackLock = 0x0,
  goid = 0xfd,
  schedlink = 0x4000157200,
  waitsince = 0x398a05ec59758,
  waitreason = 0x13,
  preempt = 0x0,
  paniconfault = 0x0,
  preemptscan = 0x0,
  gcscandone = 0x1,
  gcscanvalid = 0x1,
  throwsplit = 0x0,
  raceignore = 0x0,
  sysblocktraced = 0x1,
  sysexitticks = 0x0,
  traceseq = 0x0,
  tracelastp = 0x0,
  lockedm = 0x0,
  sig = 0x0,
  writebuf =  []uint8,
  sigcode0 = 0x0,
  sigcode1 = 0x0,
  sigpc = 0x0,
  gopc = 0x8a130,
  ancestors =  []runtime.ancestorInfo *<error reading variable: Cannot access memory at address 0x8>,
  startpc = 0x8a1f0,
  racectx = 0x0,
  waiting = 0x0,
  cgoCtxt =  []uintptr,
  labels = 0x0,
  timer = 0x400013c200,
  selectDone = 0x0,
  gcAssistBytes = 0x0
}

closing as this only appears to affect 1.13


sixel support for images with pixel resolution

addresses #213

[x] sixel support implemented
[x] check for sixel support (only ANSI terminals considered) otherwise use fallback
[x] update Render()
[x] handle partially shown images
[x] get correct dimensions from xterm
[x] fix: go-tty needed PRs: #15, #16,  #21; open, no fix yet: #20
[ ] test image widget with dynamic sizes
[ ] evaluate if external dependencies are necessary or if they could be replaced through some code
[ ] support terminal multiplexer & split view manager tmux, screen, abduco & dvtm, mtm, neercs, twin
[ ] improve rendering: don't "stack" up rendering calls if the drawing is too slow
[ ] improve rendering: draw order after windows resize is sometimes messed up and ends in an empty image widget, clicking on the UI resolves this.
[ ] fix queryTerm() function - often the terminal bugs out and needs a "stty echo". I wasn't yet able to figure it out. If someone could fix this crash demo - that would help a lot.
[ ] split image in multiple subimages if necessary in drawSixel()
[ ] test with changing image sizes and multiple images
[ ] test more terminal emulators: mlterm, xterm, tmux, screen, mintty, conhost, yaft, iTerm2, macterm
[ ] handle color support for sixel

additional drawers:

[x] iTerm2, MacTerm (untested)
[x] Kitty (remote transmission)
[x] urxvt pixbuf
[ ] conhost via GDI
[ ] X11 like w3imgdisplay perhaps with xgb(util)? (possibly remote?)

This needs the current https://github.com/mattn/go-tty!

Terminals with sixel support
mlterm, mintty (cygwin terminal), xterm with configuration, yaft
https://github.com/saitoha/libsixel/#terminal-requirements

Mintty panic

the important ANSI escape strings here are:
"\033[0c" for querying the terminal capabilities - we need a 4 for sixel.

We also need to know what the size of a character box is in pixels this differs from terminal to terminal:
"\033[14t" gives us the terminal size in pixels
"\033[18t" gives us the terminal size in cells
from that we can calculate the cell size in pixels.

We set the position with "\033[%d;%dH" before printing the sixel string. the position string is simply prepended.
The first %d is the Y Position (in cells) and the second the X Position.

xterm works now.
current state:

the rendering with images is slower and seems to "stack up"(?) - we should somehow cancel rendering (only from resize or general?) when there is already a newer change.

opened "screen" pass through issue #56063

initial tmux support - still buggy:
pass through of escape codes through tmux to the terminal works but I assume that the terminal response doesn't get back through tmux (?)

changed draw order: ASCII art first in case sixel fails

conhost issue: https://github.com/Microsoft/console/issues/120

iTerm2 sixel support: https://github.com/gnachman/iTerm2/commit/0d0003d9d20b7de3a3ffd815950acb7cc9777c38

screen can already pass escape sequences <= ~770 bytes through to the terminal. the string has to be prepended by '\033P' and appended by '\033\'.

Displaying an image through screen!
sed 's,.\{767\},&^[\\^[P,g;1s/^/^[P/;$s/$/^[\\/' snake.sixel

the ^[ are ESC chars (\033 or \x1b)
767 bytes/chars seems to be the max, perhaps it is lower.

https://github.com/tmux/tmux/issues/1391

yaft
https://github.com/uobikiemukot/yaft/issues/43
https://github.com/uobikiemukot/yaft/blob/21b69124a2907ad6ede8f45ca96c390615e3dc0c/conf.h#L26
todo: add TERM == "yaft-256color" check for sixelCapable bool

notes for other terminals

notty
escape code protocol, output/notty/mod.rs, output/notty/attachment.rs, output/mod.rs, scaffolding/tty/src/lib.rs, notty-cairo/src/image_renderer.rs

Terminology (Enlightenment)
Readme

kitty
graphics protocol, icat (python), kitty #33 graphics rendering, ranger #1077 kitty image preview, CSI 14t not supported
check for TERM == "xterm-kitty"

HTerm (archive) / hackterm
inlinedata.c, regis vector graphics

mintty sixel scrolling has changed:
https://github.com/mintty/mintty/commit/914e0a1e306a06075bf90a68d92a9b3a1ee5d5e3 (https://github.com/mintty/mintty/issues/866)

RLogin (Windows) sixel, regis

urxvt with sixel patch or from the fork (older)
How to build:
cvs -z3 -d<EMAIL_ADDRESS>co rxvt-unicode
cd rxvt-unicode/
wget https://gist.github.com/saitoha/be56f3b58c5212abe3f76a13578a548d/raw/01f7c9df2bc8d971f2b3227af89bd95419f25add/rxvt-unicode-sixel.patch
patch -p1 <rxvt-unicode-sixel.patch
sed -i '/^#include <stdio.h>/a #include <sys/stat.h>' src/screen.C
./autogen.sh
./configure --prefix=/usr/
make

ranger img_display.py Wiki Image Previews
implementations for w3mimgdisplay, urxvt pixbuf, kitty, Terminology

added support for urxvt (pixbuf backgrounds), kitty (remote transmission) and MacTerm

notes for xterm sixel implementation:
https://invisible-island.net/xterm/ctlseqs/ctlseqs.html#h2-Functions-using-CSI-_-ordered-by-the-final-character_s_
CSI ? Pi ; Pa ; Pv S
          If configured to support either Sixel Graphics or ReGIS Graph-
          ics, xterm accepts a three-parameter control sequence, where
          Pi, Pa and Pv are the item, action and value:

            Pi = 1  -> item is number of color registers.
            Pi = 2  -> item is Sixel graphics geometry (in pixels).
            Pi = 3  -> item is ReGIS graphics geometry (in pixels).

            Pa = 1  -> read
            Pa = 2  -> reset to default
            Pa = 3  -> set to value in Pv
            Pa = 4  -> read the maximum allowed value

            Pv can be omitted except when setting (Pa == 3 ).
            Pv = n <- A single integer is used for color registers.
            Pv = width ; height <- Two integers for graphics geometry.

          xterm replies with a control sequence of the same form:

               CSI ? Pi ; Ps ; Pv S

          where Ps is the status:
            Ps = 0  -> success.
            Ps = 1  -> error in Pi.
            Ps = 2  -> error in Pa.
            Ps = 3  -> failure.

          On success, Pv represents the value read or set.

          Notes:
          o   The current implementation allows reading the graphics
              sizes, but disallows modifying those sizes because that is
              done once, using resource-values.
          o   Graphics geometry is not necessarily the same as "window
              size" (see the dtterm window manipulation extensions).
              For example, xterm limits the maximum graphics geometry at
              compile time (**1000x1000** as of version 328) although the
              window size can be larger.
          o   While resizing a window will always change the current
              graphics geometry, the reverse is not true.  Setting
              graphics geometry does not affect the window size.

==> TODO:
Break image into subimages not larger than 1000x1000 for xterm.
Size should be a multiple of cell size in pixels dividable by six this is because we put the cursor position by cell not by pixel and want gaps as small as possible if any.

images in conhost:
idea from http://www.cplusplus.com/forum/beginner/223667/#msg1024626
// http://www.cplusplus.com/forum/beginner/223667/#msg1024626
// x86_64-w64-mingw32-g++-win32 conhostpixel.cc -lws2_32 -lgdi32 -user32 -static-libstdc++ -static-libgcc

#include <iostream>
#include <cstdlib>
#include <stdio.h>
#include <windows.h>

int main() {
    const auto console = ::GetConsoleWindow() ;
    const auto context = ::GetDC(console) ;
    constexpr auto red = RGB( 255, 0, 0 ) ;
    constexpr auto yellow = RGB( 255, 255, 0 ) ;

    for( int i = 2 ; i < 200 ; ++i ) for( int j = 2 ; j < 200 ; ++j )
        ::SetPixel( context, i, j, std::abs(i-j) > 25 ? red : yellow );

    Sleep(5000);
}

Go library:
https://github.com/mattn/drawcmd (image property of mattn)

I'm working on a go-w3m library here: https://gitlab.com/diamondburned/go-w3m/
Along with a port of Ueberzug in Golang (that I'm working on)

I'm working on a go-w3m library here: https://gitlab.com/diamondburned/go-w3m/
Along with a port of Ueberzug in Golang (that I'm working on)

Both are interesting libraries, thanks for creating them. I already planned implementing a drawer with xgb myself (was too much effort for me atm though learning X11...).
Would it be possible to license the w3m code under the MIT like termui? I do like copyleft-licenses like the MPL but I don't think it is wished for to include copyleft in termui. The ueberzug-go library is currently without license so not usable please add a license (if possible MIT/BSD).
The panics in the ueberzug-go should in my opinion be replaced by errors. I think an application should do the decision to panic not a library.

Latest commit changed go-w3m's LICENSE to MIT. I'm still working on ueberzug's resize feature, which sadly doesn't work at the moment.

@diamondburned I have a few problems with ueberzug-go.

I'm not sure whether it's also right for the GPLv3, but at least as far as I know it's right for the GPLv2.
If you port a program to another language you have also to license it with the GPL otherwise you're violating the GPL.
You're pretending to port ueberzug to golang (e.g. by your chosen name ueberzug-go),
but your code suggest otherwise.
The only communality is that you're also using child windows to display images.

Already stated that ueberzug-go is a work in progress, and I don't plan on mimicking any of ueberzug's features. All it is, is something that displays images the way ueberzug does it, but in Go.

Along with a port of Ueberzug in Golang

Well, that didn't sound like that.
If it isn't a port you shouldn't imply it to be one by nameing it "ueberzug-go".
If it is one you should use the GPL.
That's all.

I'm not pretending to be a port. I'll keep this as it is.

Is sixel support now working with termui?  If so, would be the only system I know besides Jexer to properly mix text and graphics.  It would be great to have more toolkits doing this.
I have found several more terminals with sixel support: https://gitlab.com/klamonte/jexer/wikis/terminals
Also, vte and xterm.js are testing sixel support now, and alacritty is implementing.  It is likely in the 3-12 month time frame that vastly more terminals will be out there that support sixel.

Just confirmed this works on foot.

Any news about it?
Will we see sixel or kitty image protocol in tmux?

@srlehn @diamondburned
This looks fantastic, awesome job!  :-)
I wanted to pass on some things I've picked up recently, on the odd chance it could help.  (Even if just by way of "go check out notcurses, it's really cool." ;) )
I added translucent windows with images under/over, thanks in large part to some clues from notcurses.  Some notes of that work are here: https://gitlab.com/klamonte/jexer/-/issues/88  .  I also wrote down more details on my particular mixed image-and-text cell model here , with some narrative (and again the notcurses shoutouts) here.
For encoding to sixel, check out notcurses' new octree implementation sometime.  It's very likely the fastest and highest-quality anywhere right now, at least for xterm-type terminals.  (libsixel was tested on real hardware, so that might give it an edge still in terms of compatibility.)
Depending on how you choose to render, you may find that 256-bit colors per sixel could still result in close to 16-bit apparent visual bit depth on the actual screen, making the drive for non-sixel support a bit less crucial.  (Note that the best bit depth for sixel is 19.92 bits: 101^3.)  The screenshot link is my very naive median-cut encoder, which is much worse than notcurses' octree encoder -- but that's still good enough that you might not be able to tell when I'm using iTerm2+PNG vs sixel.
Finally, and to my utter surprise, I may have encountered an actual xterm bug: https://gitlab.com/klamonte/jexer/-/issues/89 .  Both foot and wezterm are doing quite well on sixel, and very nice options to have for testing.

This PR transformed into this library: https://github.com/srlehn/termimg
A widget for termui can be found in the tui/termuiimg subfolder.


Running NextGen at CONUS Scale - A Step by Step Instruction

This PR provides detailed documentations on how to run ngen at CONUS scale.
Additions
NextGen_ON_CONUS.md
Removals

Changes

Testing
All Linux related commands in the documents have been tested.
mpirun jobs listed have been tested,
Visualization in brower
Screenshots
Notes

Todos
Add Topmodel
Add Routing
Checklist

[x ] PR has an informative and human-readable title
[x ] Changes are limited to a single goal (no scope creep)
[x ] Code can be automatically merged (no conflicts)
[x ] Code follows project standards (link if applicable)
[ ] Passes all existing automated tests
[ ] Any change in functionality is tested
[ ] New functions are documented (with a description, list of inputs, and expected output)
[ ] Placeholder code is flagged / future todos are captured in comments
[ ] Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
[ ] Reviewers requested with the Reviewers tool :arrow_right:

Testing checklist (automated report can be put here)

Target Environment support

[x ] Linux

Thank you Justin for the careful review. I'll implement the suggestions
tomorrow.
On Thu, Apr 18, 2024 at 9:30 AM Justin Singh-M. - NOAA <
@.***> wrote:

@.**** requested changes on this pull request.
In doc/NextGen_ON_CONUS.md
https://github.com/NOAA-OWP/ngen/pull/794#discussion_r1570847477:

+# Download the Codes

+To download the ngen source code, run the following commands:
+
+git clone https://github.com/NOAA-OWP/ngen.git https://github.com/NOAA-OWP/ngen.git
+cd ngen
+
+Then we need all the submodule codes. So run the command below:
+
+git submodule update --init --recursive
+
+# Setting up the Environment
+
+For setting up the build and computation environment, we refer the users to our documentation chapter DEPENDENCIES.md for details. Basically, you will need to have access to C/C++ compiler, MPI, Boost, NetCDF, Cmake, SQLite3. Some of them may already be on your system. Otherwise, you have to install your own version. There are also some required software packages that come with ngen as submodules, such as Udunits libraries, pybind11, and iso_c_fortran_bmi.
+
+You are most likely need to use Python. For that we recommend setting up a virtual environment. For details, see PYTHON_ROUTING.md. After setting up the Python virtual environment and activating it, you may need install additional python modules depending what ngen submodules you want to run.
⬇ Suggested change
-You are most likely need to use Python. For that we recommend setting up a virtual environment. For details, see PYTHON_ROUTING.md. After setting up the Python virtual environment and activating it, you may need install additional python modules depending what ngen submodules you want to run.
+You most likely need to use Python. For that we recommend setting up a virtual environment. For details, see PYTHON_ROUTING.md. After setting up the Python virtual environment and activating it, you may need install additional python modules depending on what ngen submodules you want to run.

In doc/NextGen_ON_CONUS.md
https://github.com/NOAA-OWP/ngen/pull/794#discussion_r1570848988:

+git clone https://github.com/NOAA-OWP/ngen.git https://github.com/NOAA-OWP/ngen.git
+cd ngen

+Then we need all the submodule codes. So run the command below:
+
+git submodule update --init --recursive
+
+# Setting up the Environment
+
+For setting up the build and computation environment, we refer the users to our documentation chapter DEPENDENCIES.md for details. Basically, you will need to have access to C/C++ compiler, MPI, Boost, NetCDF, Cmake, SQLite3. Some of them may already be on your system. Otherwise, you have to install your own version. There are also some required software packages that come with ngen as submodules, such as Udunits libraries, pybind11, and iso_c_fortran_bmi.
+
+You are most likely need to use Python. For that we recommend setting up a virtual environment. For details, see PYTHON_ROUTING.md. After setting up the Python virtual environment and activating it, you may need install additional python modules depending what ngen submodules you want to run.
+
+# Build the Executable
+
+After setting up the environment variables, we need first build the necessay dynamically linked librares. Although ngen has capability for automated building of submodule libraries, we build them explicitly so that users have a better understanding. For simplicity, we display the content a script which we name it build_libs.
⬇ Suggested change
-After setting up the environment variables, we need first build the necessay dynamically linked librares. Although ngen has capability for automated building of submodule libraries, we build them explicitly so that users have a better understanding. For simplicity, we display the content a script which we name it build_libs.
+After setting up the environment variables, we need to first build the necessary dynamically linked libraries. Although ngen has the capability for automated building of submodule libraries, we build them explicitly so that users have a better understanding. For simplicity, we display the content a script which we name it build_libs.

In doc/NextGen_ON_CONUS.md
https://github.com/NOAA-OWP/ngen/pull/794#discussion_r1570857954:

+cmake --build extern/SoilMoistureProfiles/SoilMoistureProfiles/cmake_build --target smpbmi -- -j 2 &&
+```

+Copy the content into the file named build_libs and run the command:
+
+ +source build_libs +
+
+This will build all libraries we need to run ngen at the time of this writing.
+
+Then, with the Python virtual environment activated, we can build the MPI executable using the following script:
+
+```
+cmake -S . -B cmake_build_mpi -DCMAKE_C_COMPILER=/local/lib/bin/mpicc -DCMAKE_CXX_COMPILER=/local/lib/bin/mpicxx
+-DBOOST_ROOT=/home/shengting.cui/usr/boost_1_79_0/ \
⬇ Suggested change
--DBOOST_ROOT=/home/shengting.cui/usr/boost_1_79_0/
+-DBOOST_ROOT= \

In doc/NextGen_ON_CONUS.md
https://github.com/NOAA-OWP/ngen/pull/794#discussion_r1570863244:

-DNGEN_WITH_EXTERN_PET:BOOL=OFF              \

-DNGEN_WITH_EXTERN_NOAH_OWP_MODULAR:BOOL=ON
+cmake --build cmake_build_mpi --target all -j 8
+```

+For the meaning of each option in the script, see ngen/wiki build page.
+
+Suppose the above script is named build_mpi, execute the following command to build:
+
+source build_mpi
+
+This will build an executable in the cmake_build_mpi directory named ngen and another named partitionGenerator as well as all the unit tests in the cmake_build_mpi/test.
+
+# CONUS Hydrofabric
+
+The CONUS hydrofabric is downloaded from here. The file name under the list is conus.gpkg. It is cautioned that since the data there are evolving and newer version may be available in the future. When using a newer version, be mindful that the corresponding initial configuration file generation and validation for all submodules at CONUS scale are necessary, which may be a non-trivial process due to the shear size of the spatial scale.
⬇ Suggested change
-The CONUS hydrofabric is downloaded from here. The file name under the list is conus.gpkg. It is cautioned that since the data there are evolving and newer version may be available in the future. When using a newer version, be mindful that the corresponding initial configuration file generation and validation for all submodules at CONUS scale are necessary, which may be a non-trivial process due to the shear size of the spatial scale.
+The CONUS hydrofabric is downloaded from here. The file name under the list is conus.gpkg. Note that since the data there is continually evolving, a newer version may be available in the future. When using a newer version, be mindful that the corresponding initial configuration file generation and validation for all submodules at CONUS scale is necessary, which may be a non-trivial process due to the sheer size of the spatial scale.

In doc/NextGen_ON_CONUS.md
https://github.com/NOAA-OWP/ngen/pull/794#discussion_r1570867705:

+cmake --build cmake_build_mpi --target all -j 8
+```

+For the meaning of each option in the script, see ngen/wiki build page.
+
+Suppose the above script is named build_mpi, execute the following command to build:
+
+source build_mpi
+
+This will build an executable in the cmake_build_mpi directory named ngen and another named partitionGenerator as well as all the unit tests in the cmake_build_mpi/test.
+
+# CONUS Hydrofabric
+
+The CONUS hydrofabric is downloaded from here. The file name under the list is conus.gpkg. It is cautioned that since the data there are evolving and newer version may be available in the future. When using a newer version, be mindful that the corresponding initial configuration file generation and validation for all submodules at CONUS scale are necessary, which may be a non-trivial process due to the shear size of the spatial scale.
+
+As the file is fairly large, it is worth some consideration to store it in a proper place, then simply build a symbolic link in the ngen home directory, thus named ./hydrofabric/conus.gpkg. Note the easiest way to create the symbolic link is to makedir hydrofabric and then create the full path.
⬇ Suggested change
-As the file is fairly large, it is worth some consideration to store it in a proper place, then simply build a symbolic link in the ngen home directory, thus named ./hydrofabric/conus.gpkg. Note the easiest way to create the symbolic link is to makedir hydrofabric and then create the full path.
+As the file is fairly large, it is worth some consideration to store it in a proper place, then simply build a symbolic link in the ngen home directory, thus named ./hydrofabric/conus.gpkg. Note the easiest way to create the symbolic link is to create a hydrofabric directory and then create a link to that directory.

In doc/NextGen_ON_CONUS.md
https://github.com/NOAA-OWP/ngen/pull/794#discussion_r1570870673:

+As the file is fairly large, it is worth some consideration to store it in a proper place, then simply build a symbolic link in the ngen home directory, thus named ./hydrofabric/conus.gpkg. Note the easiest way to create the symbolic link is to makedir hydrofabric and then create the full path.
+
+# Generate Partition For Parallel Computation
+
+For parallel computation using MPI on hydrofabric, a partition generate tool is used to partition the hydrofabric features ids into a number of partitions equal to the number of MPI processing CPU cores. To generate the partition file, run the following command:
+
+ +./cmake-build_mpi/partitionGenerator ./hydrofabric/conus.gpkg ./hydrofabric/conus.gpkg ./partition_config_32.json 32 '' '' +
+
+In the command above, conus.gpkg is the NextGen hydrofabric version 2.01 for CONUS, partition_config_32.json is the partition file that contains all features ids and their interconnected network information. The number 32 is intended number of processing cores for running parallel build ngen using MPI. The last two empty strings, as indicated by '', indicate there is no subsetting, i.e., we intend to run the whole CONUS hydrofabric.
+
+# Prepare the Input Data
+
+Input data include the forcing data and initial parameter data for various submodules. These depend on what best suit the user need. For our case, as of this documentation, beside forcing data, which can be accessed at ./forcing/NextGen_forcing_2016010100.nc using the symbolic link scheme, we also generated initial input data for various submodules noah-owp-modular, PET, CFE, SoilMoistureProfiles (SMP), SoilFreezeThaw (SFT). The first three are located in ./conus_config/, the SMP initial configus are located in ./conus_smp_configs/ and the SFT initial configs are located in ./conus_sft_configs/.
⬇ Suggested change
-Input data include the forcing data and initial parameter data for various submodules. These depend on what best suit the user need. For our case, as of this documentation, beside forcing data, which can be accessed at ./forcing/NextGen_forcing_2016010100.nc using the symbolic link scheme, we also generated initial input data for various submodules noah-owp-modular, PET, CFE, SoilMoistureProfiles (SMP), SoilFreezeThaw (SFT). The first three are located in ./conus_config/, the SMP initial configus are located in ./conus_smp_configs/ and the SFT initial configs are located in ./conus_sft_configs/.
+Input data includes the forcing data and initial parameter data for various submodules. These depend on what best suits the user's need. For our case, as of this documentation, beside forcing data, which can be accessed at ./forcing/NextGen_forcing_2016010100.nc using the symbolic link scheme, we also generated initial input data for various submodules noah-owp-modular, PET, CFE, SoilMoistureProfiles (SMP), SoilFreezeThaw (SFT). The first three are located in ./conus_config/, the SMP initial configs are located in ./conus_smp_configs/ and the SFT initial configs are located in ./conus_sft_configs/.

In doc/NextGen_ON_CONUS.md
https://github.com/NOAA-OWP/ngen/pull/794#discussion_r1570874067:

+./cmake-build_mpi/partitionGenerator ./hydrofabric/conus.gpkg ./hydrofabric/conus.gpkg ./partition_config_32.json 32 '' ''
+```

+In the command above, conus.gpkg is the NextGen hydrofabric version 2.01 for CONUS, partition_config_32.json is the partition file that contains all features ids and their interconnected network information. The number 32 is intended number of processing cores for running parallel build ngen using MPI. The last two empty strings, as indicated by '', indicate there is no subsetting, i.e., we intend to run the whole CONUS hydrofabric.
+
+# Prepare the Input Data
+
+Input data include the forcing data and initial parameter data for various submodules. These depend on what best suit the user need. For our case, as of this documentation, beside forcing data, which can be accessed at ./forcing/NextGen_forcing_2016010100.nc using the symbolic link scheme, we also generated initial input data for various submodules noah-owp-modular, PET, CFE, SoilMoistureProfiles (SMP), SoilFreezeThaw (SFT). The first three are located in ./conus_config/, the SMP initial configus are located in ./conus_smp_configs/ and the SFT initial configs are located in ./conus_sft_configs/.
+
+For code used to generate the initial config files for the various modules, the interested users are directed to this web location.
+
+The users are warned that since the simulated region is large, some of the initial config parameters values for some catchments may be unsuitable and cause the ngen execution to stop due to errors. Usually, in such cases, either ngen or the submodule itself may provide some hint as to the catchment ids or the location of the code that caused the error. Users may follow these hints to figure out as to which initial input parameter or parameters are initialized with inappropriate values. In the case of SFT, an initial value of smcmax=1.0 would be too large. In the case of SMP, an initial value of b=0.01 would be too small, for example.
+
+# Build the Realization Configurations
+
+The realization configuration file in Json format contains high level information to run a ngen simulation, such as interconnected submodules, paths to forcing file, shared libraries, initialization parameters, duration of simulation, I/O variables, etc. We have built the realization configurations for several commonly used submodules which are located in data/baseline/. These are built by adding one submodule at a time, test run for 10 days simulation time. The successive submodules used are:
⬇ Suggested change
-The realization configuration file in Json format contains high level information to run a ngen simulation, such as interconnected submodules, paths to forcing file, shared libraries, initialization parameters, duration of simulation, I/O variables, etc. We have built the realization configurations for several commonly used submodules which are located in data/baseline/. These are built by adding one submodule at a time, test run for 10 days simulation time. The successive submodules used are:
+The realization configuration file, in JSON format, contains high level information to run a ngen simulation, such as interconnected submodules, paths to forcing file, shared libraries, initialization parameters, duration of simulation, I/O variables, etc. We have built the realization configurations for several commonly used submodules which are located in data/baseline/. These are built by adding one submodule at a time, performing a test run for a 10 day simulation. The successive submodules used are:

In doc/NextGen_ON_CONUS.md
https://github.com/NOAA-OWP/ngen/pull/794#discussion_r1570876533:

+# Build the Realization Configurations

+The realization configuration file in Json format contains high level information to run a ngen simulation, such as interconnected submodules, paths to forcing file, shared libraries, initialization parameters, duration of simulation, I/O variables, etc. We have built the realization configurations for several commonly used submodules which are located in data/baseline/. These are built by adding one submodule at a time, test run for 10 days simulation time. The successive submodules used are:
+ +sloth (conus_bmi_multi_realization_config_w_sloth.json) +sloth+noah-owp-modular (conus_bmi_multi_realization_config_w_sloth_noah.json) +sloth+noah-owp-modular+pet (conus_bmi_multi_realization_config_w_sloth_noah_pet.json) +sloth+noah-owp-modular+pet+cfe (conus_bmi_multi_realization_config_w_sloth_noah_pet_cfe.json) +sloth+noah-owp-modular+pet+smp (conus_bmi_multi_realization_config_w_sloth_noah_pet_smp.json) +sloth+noah-owp-modular+pet+smp+sft (conus_bmi_multi_realization_config_w_sloth_noah_pet_smp_sft.json) +sloth+noah-owp-modular+pet+smp+sft+cfe (conus_bmi_multi_realization_config_w_sloth_noah_pet_smp_sft_cfe.json) +
+
+# Run Computations with Submodules
+
+With all preparation steps completed, we are now ready to run computations. We use MPI as our parallel processing application with 32 cores as an example. Users are free to choose whatever number cores they want, just make sure you will need to have the appropriate corresponding partition json file for the number of cores used. The command line for running a MPI job is as sollows:
⬇ Suggested change
-With all preparation steps completed, we are now ready to run computations. We use MPI as our parallel processing application with 32 cores as an example. Users are free to choose whatever number cores they want, just make sure you will need to have the appropriate corresponding partition json file for the number of cores used. The command line for running a MPI job is as sollows:
+With all preparation steps completed, we are now ready to run computations. We use MPI as our parallel processing application with 32 cores as an example. Users are free to choose whatever number cores they want, just make sure you will need to have the appropriate corresponding partition JSON file for the number of cores used. The command line for running a MPI job is as follows:

In doc/NextGen_ON_CONUS.md
https://github.com/NOAA-OWP/ngen/pull/794#discussion_r1570877056:

+sloth+noah-owp-modular (conus_bmi_multi_realization_config_w_sloth_noah.json)
+sloth+noah-owp-modular+pet (conus_bmi_multi_realization_config_w_sloth_noah_pet.json)
+sloth+noah-owp-modular+pet+cfe (conus_bmi_multi_realization_config_w_sloth_noah_pet_cfe.json)
+sloth+noah-owp-modular+pet+smp (conus_bmi_multi_realization_config_w_sloth_noah_pet_smp.json)
+sloth+noah-owp-modular+pet+smp+sft (conus_bmi_multi_realization_config_w_sloth_noah_pet_smp_sft.json)
+sloth+noah-owp-modular+pet+smp+sft+cfe (conus_bmi_multi_realization_config_w_sloth_noah_pet_smp_sft_cfe.json)
+```

+# Run Computations with Submodules
+
+With all preparation steps completed, we are now ready to run computations. We use MPI as our parallel processing application with 32 cores as an example. Users are free to choose whatever number cores they want, just make sure you will need to have the appropriate corresponding partition json file for the number of cores used. The command line for running a MPI job is as sollows:
+
+For a simple example run and quick turn around, you can run:
+
+```
+run -n 32 ./cmake_build_mpi/ngen ./hydrofabric/conus.gpkg '' ./hydrofabric/conus.gpkg '' data/baseline/conus_bmi_multi_realization_config_w_sloth.json conus_partition_32.json
⬇ Suggested change
-run -n 32 ./cmake_build_mpi/ngen ./hydrofabric/conus.gpkg '' ./hydrofabric/conus.gpkg '' data/baseline/conus_bmi_multi_realization_config_w_sloth.json conus_partition_32.json
+mpiexec -n 32 ./cmake_build_mpi/ngen ./hydrofabric/conus.gpkg '' ./hydrofabric/conus.gpkg '' data/baseline/conus_bmi_multi_realization_config_w_sloth.json conus_partition_32.json

In doc/NextGen_ON_CONUS.md
https://github.com/NOAA-OWP/ngen/pull/794#discussion_r1570877417:

+```

+# Run Computations with Submodules
+
+With all preparation steps completed, we are now ready to run computations. We use MPI as our parallel processing application with 32 cores as an example. Users are free to choose whatever number cores they want, just make sure you will need to have the appropriate corresponding partition json file for the number of cores used. The command line for running a MPI job is as sollows:
+
+For a simple example run and quick turn around, you can run:
+
+ +run -n 32 ./cmake_build_mpi/ngen ./hydrofabric/conus.gpkg '' ./hydrofabric/conus.gpkg '' data/baseline/conus_bmi_multi_realization_config_w_sloth.json conus_partition_32.json +
+
+For a more substantial example simulation, you can run:
+
+```
+run -n 32 ./cmake_build_mpi/ngen ./hydrofabric/conus.gpkg '' ./hydrofabric/conus.gpkg '' data/baseline/conus_bmi_multi_realization_config_w_sloth_noah.json conus_partition_32.json
⬇ Suggested change
-run -n 32 ./cmake_build_mpi/ngen ./hydrofabric/conus.gpkg '' ./hydrofabric/conus.gpkg '' data/baseline/conus_bmi_multi_realization_config_w_sloth_noah.json conus_partition_32.json
+mpiexec -n 32 ./cmake_build_mpi/ngen ./hydrofabric/conus.gpkg '' ./hydrofabric/conus.gpkg '' data/baseline/conus_bmi_multi_realization_config_w_sloth_noah.json conus_partition_32.json

In doc/NextGen_ON_CONUS.md
https://github.com/NOAA-OWP/ngen/pull/794#discussion_r1570878772:

+For a simple example run and quick turn around, you can run:
+
+ +run -n 32 ./cmake_build_mpi/ngen ./hydrofabric/conus.gpkg '' ./hydrofabric/conus.gpkg '' data/baseline/conus_bmi_multi_realization_config_w_sloth.json conus_partition_32.json +
+
+For a more substantial example simulation, you can run:
+
+ +run -n 32 ./cmake_build_mpi/ngen ./hydrofabric/conus.gpkg '' ./hydrofabric/conus.gpkg '' data/baseline/conus_bmi_multi_realization_config_w_sloth_noah.json conus_partition_32.json +
+
+For an example taken into account more realistic contributions, you can try:
+```
+run -n 32 ./cmake_build_mpi/ngen ./hydrofabric/conus.gpkg '' ./hydrofabric/conus.gpkg '' data/baseline/conus_bmi_multi_realization_config_w_sloth_noah_pet_smp_sft_cfe.json conus_partition_32.json
⬇ Suggested change
-run -n 32 ./cmake_build_mpi/ngen ./hydrofabric/conus.gpkg '' ./hydrofabric/conus.gpkg '' data/baseline/conus_bmi_multi_realization_config_w_sloth_noah_pet_smp_sft_cfe.json conus_partition_32.json
+mpiexec -n 32 ./cmake_build_mpi/ngen ./hydrofabric/conus.gpkg '' ./hydrofabric/conus.gpkg '' data/baseline/conus_bmi_multi_realization_config_w_sloth_noah_pet_smp_sft_cfe.json conus_partition_32.json

In doc/NextGen_ON_CONUS.md
https://github.com/NOAA-OWP/ngen/pull/794#discussion_r1570879246:

+With all preparation steps completed, we are now ready to run computations. We use MPI as our parallel processing application with 32 cores as an example. Users are free to choose whatever number cores they want, just make sure you will need to have the appropriate corresponding partition json file for the number of cores used. The command line for running a MPI job is as sollows:
+
+For a simple example run and quick turn around, you can run:
+
+ +run -n 32 ./cmake_build_mpi/ngen ./hydrofabric/conus.gpkg '' ./hydrofabric/conus.gpkg '' data/baseline/conus_bmi_multi_realization_config_w_sloth.json conus_partition_32.json +
+
+For a more substantial example simulation, you can run:
+
+ +run -n 32 ./cmake_build_mpi/ngen ./hydrofabric/conus.gpkg '' ./hydrofabric/conus.gpkg '' data/baseline/conus_bmi_multi_realization_config_w_sloth_noah.json conus_partition_32.json +
+
+For an example taken into account more realistic contributions, you can try:
⬇ Suggested change
-For an example taken into account more realistic contributions, you can try:
+For an example taking into account more realistic contributions, you can try:

In doc/NextGen_ON_CONUS.md
https://github.com/NOAA-OWP/ngen/pull/794#discussion_r1570881580:

+```

+For a more substantial example simulation, you can run:
+
+ +run -n 32 ./cmake_build_mpi/ngen ./hydrofabric/conus.gpkg '' ./hydrofabric/conus.gpkg '' data/baseline/conus_bmi_multi_realization_config_w_sloth_noah.json conus_partition_32.json +
+
+For an example taken into account more realistic contributions, you can try:
+ +run -n 32 ./cmake_build_mpi/ngen ./hydrofabric/conus.gpkg '' ./hydrofabric/conus.gpkg '' data/baseline/conus_bmi_multi_realization_config_w_sloth_noah_pet_smp_sft_cfe.json conus_partition_32.json +
+
+where ngen is the executable we build in the [Building the Executable](#Building the Executable) section. All other terms have been discussed above in details. With the current existing realization config files, the above jobs run 10 days simulation time on CONUS scale.
+
+Be aware that the above commands will generate over a million output files associated with catchment and nexus ids so if you issue a ls command in ngen directory, it will be significantly slower than usual to list all the file names. The exact time will depend on the computer you are working on.
You may want to note/include the use of the output_root realization
config option here, so that outputs can be stored in a separate directory
from the source tree.
In doc/NextGen_ON_CONUS.md
https://github.com/NOAA-OWP/ngen/pull/794#discussion_r1570882469:

@@ -0,0 +1,173 @@
+# NextGen on CONUS

+This documentation provides instructions on all neccessary steps and components to run NextGen jobs at CONUS scale. Considering the computations large scale, we focus only on running parallel jobs using MPI.
⬇ Suggested change
-This documentation provides instructions on all neccessary steps and components to run NextGen jobs at CONUS scale. Considering the computations large scale, we focus only on running parallel jobs using MPI.
+This documentation provides instructions on all neccessary steps and components to run NextGen jobs at CONUS scale. Considering the computation's large scale, we focus only on running parallel jobs using MPI.
—
Reply to this email directly, view it on GitHub
https://github.com/NOAA-OWP/ngen/pull/794#pullrequestreview-2009026988,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACA4SRLEKB34ATRWTBFAQ6DY57KHTAVCNFSM6AAAAABGJ6TC5CVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDAMBZGAZDMOJYHA
.
You are receiving this because you authored the thread.Message ID:
@.***>

Good catch for both. Thanks.
On Tue, Apr 23, 2024 at 10:46 AM Justin Singh-M. - NOAA <
@.***> wrote:

@.**** commented on this pull request.
In doc/NextGen_ON_CONUS.md
https://github.com/NOAA-OWP/ngen/pull/794#discussion_r1576489783:

@@ -0,0 +1,174 @@
+# NextGen on CONUS

+This documentation provides instructions on all neccessary steps and components to run NextGen jobs at CONUS scale. Considering the computation's large scale, we focus only on running parallel jobs using MPI.
+
+* Summary
+* Download the Codes
+* Setting Up the Environment
+* Build the Executable
+* CONUS Hydrofabric
This doesn't link in the rich diff, does changing the URL slightly to
#conus-hydrofabric resolve that?
⬇ Suggested change
-* CONUS Hydrofabric
+* CONUS Hydrofabric
—
Reply to this email directly, view it on GitHub
https://github.com/NOAA-OWP/ngen/pull/794#pullrequestreview-2017646668,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACA4SRLZEQL2J5S2JUR2LK3Y6Z64HAVCNFSM6AAAAABGJ6TC5CVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDAMJXGY2DMNRWHA
.
You are receiving this because you authored the thread.Message ID:
@.***>


Don't Merge! PR real time test for webhook case

Docs Build status updates of commit 6243063:
:clock10: Pending: waiting for processors (20 builds ahead of you)
:warning: Docs Build is busy, currently there are 20 builds ahead of this one, for more information you can view the Build queue graph on the Docs Portal.

Docs Build status updates of commit 6243063:
:clock10: Full build: the average full build time is 7 min(s) 29 sec(s), this is based on the last 30 build(s) for this repository.

Docs Build status updates of commit 6243063:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs-ref-conceptual/testing.md
:white_check_mark:Succeeded
View (azure-cli-latest) View (azure-cli-2017-03-09-profile) View (azure-cli-2018-03-01-hybrid)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


fix(Metaevidence): remove hardcoded block number

Superseded by this https://github.com/kleros/zk-cross-chain-realitio-proxy/pull/4


chore(deps): update typescript

This PR contains the following updates:

Package
Type
Update
Change

@types/node (source)
devDependencies
patch
20.14.9 -> 20.14.10

@types/node (source)
dependencies
patch
20.14.9 -> 20.14.10

gdata-vaas
dependencies
patch
7.1.5 -> 7.1.6

mocha (source)
devDependencies
minor
10.5.2 -> 10.6.0

typescript (source)
devDependencies
patch
5.5.2 -> 5.5.3

typescript (source)
dependencies
patch
5.5.2 -> 5.5.3

ws
dependencies
minor
8.17.1 -> 8.18.0

:warning: Warning
Some dependencies could not be looked up. Check the warning logs for more information.

Release Notes

GDATASoftwareAG/vaas (gdata-vaas)
v7.1.6
Compare Source

mochajs/mocha (mocha)
v10.6.0
Compare Source
:tada: Enhancements

#5150 feat: allow ^ versions for character encoding packages (@JoshuaKGoldberg)
#5151 feat: allow ^ versions for file matching packages (@JoshuaKGoldberg)
#5152 feat: allow ^ versions for yargs packages (@JoshuaKGoldberg)
#5153 feat: allow ^ versions for data serialization packages (@JoshuaKGoldberg)
#5154 feat: allow ^ versions for miscellaneous packages (@JoshuaKGoldberg)

Microsoft/TypeScript (typescript)
v5.5.3: TypeScript 5.5.3
Compare Source
For release notes, check out the release announcement.
For the complete list of fixed issues, check out the

fixed issues query for TypeScript v5.5.3 (Stable).
fixed issues query for TypeScript v5.5.2 (Stable).
fixed issues query for TypeScript v5.5.1 (RC).
fixed issues query for TypeScript v5.5.0 (Beta).

Downloads are available on:

npm
NuGet package

websockets/ws (ws)
v8.18.0
Compare Source
Features

Added support for Blob (#2229).

Configuration
:date: Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
:vertical_traffic_light: Automerge: Disabled by config. Please merge this manually once you are satisfied.
:recycle: Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
:ghost: Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

Edited/Blocked Notification
Renovate will not automatically rebase this PR, because it does not recognize the last commit author and assumes somebody else may have edited the PR.
You can manually request rebase by checking the rebase/retry box above.
:warning: Warning: custom changes will be lost.


Rose Testnet Bug

Got a bag when try to repay UST

Hi, @kytyi.Unfortunatly image is no longer available. If error is related to:
-32603: Internal Error - [-32000] Server error: DB Not Found Error: BLOCK HEIGHT: 86451935 Cause: Unknown. Please report a bug at https://github.com/aurora-is-near/aurora-relayer/issues, then please try again on testnet. There is a fix landed there and would be great if you could confirm/deny if it resolved this issue.

This issue should be resolved. Please reopen if it will reappear.


Outputs for provider authors

This PR provides a new type available to infer authors: infer.Output[T]. This type can be used for any field with pulumi:"{name}" in place of T. It provides secret + unknown metadata and an unknown safe infer.Apply API modeled after pulumix as implemented in the bridge.
The file example has been converted to use the new API.

Implementors Guide
We need a generic type (Output[T]) that has a couple of properties:

A bijective function between resource.PropertyValue and Output[T] (when types align). This means that Output[T] needs to understand secrets and unknown values.
An ergonomic API for creating and consuming Output[T].

For creation, users need to be able to convert from:MyField *string `pulumi:"myField"`

toMyField infer.Output[*string] `pulumi:"myField"`

This cannot have schema changes.
For consumption, we expose the same "apply" based API as pulumix, with easy access to escape.

Implementation follows the needs of the new feature closely. There are two major areas of change:

I have written a custom mapper that is able to map from resource.PropertyValue to T for any T. This new mapper will preserve information when T is Output[U]. This mapper is part of the internal ende package, but ende could conceivably be moved outside of the infer package entirely.
I have written an infer.Output[T] type that fulfills the same basic contract as pulumix.Output[T]. infer.Output[T] is different in several ways:

It deserializes correctly with ende.
It has methods to escape from it's monadic model:

Output[T].Anchor() error blocks the thread until the output has resolved, if it can.
Output[T].GetKnown() (T, error) retrieves the known value (or an error in the computation chain). It panics if the value is unknowable.
Output[T].GetMaybeUnknown() (T, error) retrieves the known value or zero value (or an error in the computation chain).

As of https://github.com/pulumi/pulumi-go-provider/pull/155/commits/ff4f1784556e6ce9931f7dbe79da3aa27b940f3f, assets, archives and resource references are not supported by the new mapper. We will need to add support for all of these to avoid a regression on merge.

Per earlier conversation, my preferred end state here would be to use the same Go SDK for provider authoring as the program authoring. This may mean waiting on Generics, or using pre-Generic pulumi.Output forms from the current Go SDK. If we are indeed headed in that direction, introducing new forms here we'll need to retract them and refactor to the desired end-state some time afterward. This is a little unfortunate.
So the question becomes in light of this direction, is this still a change worth taking now? What are the intended use cases or provider we'd like to deploy this at, that motivates doing it now vs later, would that be something that can help with Docker Image perhaps? How about external ecosystem providers depending on pulumi-go-provider do we have a sense of how many there are and how much nuisance we're imposing by introducing these types first and then asking them to take a breaking change? Is the ecosystem /uptake small enough this is not a worry at the moment?

Per earlier conversation, my preferred end state here would be to use the same Go SDK for provider authoring as the program authoring. This may mean waiting on Generics, or using pre-Generic pulumi.Output forms from the current Go SDK. If we are indeed headed in that direction, introducing new forms here we'll need to retract them and refactor to the desired end-state some time afterward. This is a little unfortunate.

My concern here is that our pulumix.Output[T] generics have extremely different semantics then infer.Output[T] vis a vis escaping from the output model. We expose infer.Output[T].Anchor() error which forces async computations to "catch up" if possible. This is possible only because we know if every accessible infer.Output[T] can be resolved on creation, which is not the case for pulumix.Output[T].
I think more work is needed to check if we can unify these definitions. If pulumix.Output[T] was an open interface, I think we could include and then extend, but it isn't an interface at all. pulumix.OutputOf[T] is a closed interface (includes a private func).

So the question becomes in light of this direction, is this still a change worth taking now? What are the intended use cases or provider we'd like to deploy this at, that motivates doing it now vs later, would that be something that can help with Docker Image perhaps? How about external ecosystem providers depending on pulumi-go-provider do we have a sense of how many there are and how much nuisance we're imposing by introducing these types first and then asking them to take a breaking change? Is the ecosystem /uptake small enough this is not a worry at the moment?

I view this change (or one like it) as necessary for shipping a high quality docker provider (that handles unknowns well). I'll defer to the @blampe on if he is interested in adopting the provider framework (with or without this change) for Docker.
This change is opt-in and non-breaking for existing providers, so I'm not concerned about a break during adoption. I think that the ecosystem that is using this framework is willing to take breaking changes to be able to write better providers. The README.md still says:

This library is in active development, and not everthing is hooked up. You should expect breaking changes as we fine tune the exposed APIs.

generics have extremely different semantics then infer.Output[T]

Can you elaborate that please?
AFAIK, every Output in a pulumi Go program is force-awaited before the program exists, so if you need Output<T> -> T that should be absolutely expressible.
Are there any other semantic differences?
BTW we do not need to wait for generic pulumi.Output[T] to land afaik as we can start using non-generic pulumi.Output perhaps as of today.

https://github.com/pulumi/pulumi/blob/master/sdk/go/internal/types.go#L380 and WorkGroup.Wait() for reference. Also https://github.com/pulumi/pulumi/blob/master/sdk/go/internal/state.go#L44

I think Go is the one language where it is guaranteed that Outputs will resolve, because there's no lifting from promises.

I view this change (or one like it) as necessary for shipping a high quality docker provider (that handles unknowns well). I'll defer to the @blampe on if he is interested in adopting the provider framework (with or without this change) for Docker.

I'm definitely interested in anything we can do to make provider development easier and safer! I'll try to grab some time with you to learn more about how this could help docker.

BTW we do not need to wait for generic pulumi.Output[T] to land afaik as we can start using non-generic pulumi.Output perhaps as of today.

I'm still gathering context, but if pulumix.OutputOf[T] is satisfied by pulumix.Output[T] as well as pulumi.Output does that mean we could use pulumix.OutputOf[T] here to get the best of both worlds? You could use new generic types, and if you need to resolve the output you can still fall back to the older types. (There are some serialization implications I don't understand yet.)

AFAIK, every Output in a pulumi Go program is force-awaited before the program exists, so if you need Output -> T that should be absolutely expressible.

I need the user to have a Output<T> -> T available. If I can do that, then I can build most of what is necessary.

Are there any other semantic differences?

infer.Output[T] tracks the fields it depends on, not the resources it depends on. It doesn't need to track resources, since it is only used in the context of custom providers (which operating within a single field).
Field dependency is injected during deserialization and then merged during applies. It is then rehydrated during serialization.

BTW we do not need to wait for generic pulumi.Output[T] to land afaik as we can start using non-generic pulumi.Output perhaps as of today.

Assuming we taught pulumi.Output to track fields, we could do this. However, the experience for nested values is terrible. If we have generics, then simple nested inputs become easy:
type Input struct {
    Field infer.Output[Nested] `pulumi:"nested"`
}

type Nested struct {
    N1 infer.Output[string] `pulumi:"n1"`
    N2 infer.Output[int] `pulumi:"n2"`
}

Expressing nested values in pulumi.Output requires defining a Nested interface with a NestedOutput type. I view that as prohibitively painful.

I think Go is the one language where it is guaranteed that Outputs will resolve, because there's no lifting from promises.

Is that exposed anywhere? I would love to use pulumix.Output if possible. We would also need to resolve the type of tracked dependency.

I'm still gathering context, but if pulumix.OutputOf[T] is satisfied by pulumix.Output[T] as well as pulumi.Output does that mean we could use pulumix.OutputOf[T] here to get the best of both worlds? You could use new generic types, and if you need to resolve the output you can still fall back to the older types. (There are some serialization implications I don't understand yet.)

pulumix.OutputOf[T] is satisfied by pulumix.Output[T] and pulumni.Output[T], but it is not possible for infer.Output[T] to satisfy pulumix.OutputOf[T].

I need the user to have a Output -> T available.

That's not safe. Providers can get called with unknown values where there is no T. Further you lose all secret and dependency tracking when you try to exit the context like that.
The Go SDK has an UnsafeAwait for Output that returns a struct of Value, Known, Secret, Deps. But working with that is a pain, and I doubt anyone uses it fully correctly.
My opinion is any design where you try to exit the Output context is wrong. If there's something unexpressible with Apply we should fix Apply.

Looks like this is still in my inbox, stalling a bit.
Throw a meeting to unblock? Or icebox for now?
Providers should have Output<T> -> T
I believe what's implied here is UnsafeAwaitOutput from the Go SDK https://github.com/pulumi/pulumi/pull/10877
I'd think it is fair game for the provider implementation framework to call that one when needed, which is not the same thing as making it necessary to call for provider authors using the framework.
Does this help untie the knot here @iwahbe ?

@blampe I believe you are the current expert consumer of this library. Do you believe that this is worth pushing through, or should we put on hold.

I believe what's implied here is UnsafeAwaitOutput from the Go SDK https://github.com/pulumi/pulumi/pull/10877

I still think that unless pulumi.OutputOf[T] is an unsealed and pluggable interface, we will need a different implementation:
In Pulumi program land, an pulumi.Output<T> is a value that may or may not resolved, and we can't know if it will. UnsafeAwaitOutput may hang during preview, because an output will never resolve. Output's track which resource they depend upon so we can build a dependency graph.
In pulumi-go-provider land, an infer.Output[T] is a value in one of three states:

resolved
unresolved (but will resolve)
won't resolve

For each infer.Output[T], we know which state it is in at any given time.
In addition, infer.Output[T] does track dependencies, but tracks dependent fields, not dependent resources.

This are not very convincing arguments to me, as forking the foundational types is still a high cost.

we don't know if pulumi.Output will resolve - every output in Pulumi afaik has to resolve, or you get | a hang semantic.

you're saying you need to query if the Output is resolved or not right now, that can possibly be arranged

possibly we need the Output a bit more extensible?  I'm not sure what's the purpose of that but if it's to track secrets we have some alternatives? Also, presumably you'd need to track normal resource dependencies sometimes when e.g. implementing remote component resources so you need everything Output offers but you might be needing a small extension?

Inclined to agree with @t0yv0 here. Outputs in Pulumi programs will resolve unless users have done very bad things (infinite loops in apply for example).
You say infer.Output needs field tracking instead of resource tracking but given that the go-provider also needs to cater to Construct calls you need normal inputs for that, and I am not against extending normal outputs to track fields (it goes well with some ideas I've been having about making previews more detailed, it would be nice to say for example a field is unknown but will resolve to the same value as field X on resource Y).

we don't know if pulumi.Output will resolve - every output in Pulumi afaik has to resolve, or you get | a hang semantic.

This is not true during preview.

you're saying you need to query if the Output is resolved or not right now, that can possibly be arranged.

I'm saying that I need to query if the output is resolvable or not, which is different than resolved. I don't know if pulumi.OutputOf[T] can be augmented to track this, but I don't believe that it does right now.

possibly we need the Output a bit more extensible?  I'm not sure what's the purpose of that but if it's to track secrets we have some alternatives? Also, presumably you'd need to track normal resource dependencies sometimes when e.g. implementing remote component resources so you need everything Output offers but you might be needing a small extension?

We do not use infer.Output for component resources, we can use pulumi.Output for that.

To be able to use pulumi.OutputOf[T], I would need 2 things:

An extension point to create field dependency info that is infectious. Something like:

v := unstable.NewOutputWithPayload[String, infer.dependency](unstable.RawOutput{
    Value: resource.PropertyOf("value"),
    Secret: true
    Computed: false,
}, infer.dependency{ fields: "valueField" })

// Give to user

i := pulumi.Apply1E(v, strconv.ParseInt)

return Args{I: I}

// Back in framework space

deps := unstable.OutputPayload[infer.dependency](result.I) // infer.dependency{ fields: "valueField" }

I need to check if an arbitrary output is resolvable.

We need the following logic to be expressible:
if o.wait() will resolve; then o.wait() && return (o.value(), o.error, true)
else return (Zero[T](), o.error, false)

I actually think that (1) and (2) are solvable if pulumi.OutputOf[T] if we can add this interface:
// unstable/output.go -- somewhere in pulumi/pulumi

func NewOutputWithPayload[T Value, P any, PG PayloadManager[P]](output RawOutput, payload P, payload) { ... }

func GetPayload[P Payload](o internal.Output) (P, bool) { ... }

type PayloadManager[P] interface {
    func Merge(ctx context.Context, outputs []struct{p P, o RawOutput}) P
    func New(ctx context.Context, output RawOutput) P
}

type RawOutput = internals.UnsafeAwaitOutputResult

Maybe I'm misunderstanding something and this is already possible.

Can you give an example of an output that's not resolvable?
Everything passed to the provider is resolvable. In preview it may resolve to an unknown.
On top of this you have provider implementation that constructs resources and generates derived Outputs. I think in provider such as TF providers all I can think of is that all such outputs will be resolvable as well. They may resolve to the unknown value.
What's an unresolvable output? That would help me understand.

Can you give an example of an output that's not resolvable?

We might be talking past each-other. I think of this as unresolvable:
UnknownValue.Apply(f)

This is unresolved, but resolvable:
KnownValue.Apply(func(value int) int { sleep(value); return value })

If we allow something to resolve to unknown, then I agree that all values are resolvable.

UnknownValue.Apply(f) === UnknownValue should hold in every Pulumi language I think. The UnknownValue output can be awaited and inspected.
In Go
result, err := UnsafeAwaitOutput(ctx, UnknownValue)
require.NoError(t, err)
require.False(t, result.Known)

Another tangent, based on your explanation of what you mean by "will resolve", this logic seems to substitute zero values for unknowns, which is a massive footgun IMHO, admittedly one that is present in our prior designs. If we're designing from first principles, why is this ever needed to inject zero values?
if o.wait() will resolve; then o.wait() && return (o.value(), o.error, true)
else return (Zero[T](), o.error, false)

Could we force the provider author to model locations that might not be known with a type that explicitly models unknowns, e.g. an Input or an Output?

If we allow something to resolve to unknown, then I agree that all values are resolvable.

We do allow that. As @t0yv0 said all SDKs will resolve unknown.apply(f) to unknown.
I massively agree with Anton about being very careful about zero values around this.

We might be talking past each-other.

It might be this is at the point where we really ought to have a meeting about it. Provider/Core ideation is coming up, maybe enough to just be a point in that.

Maybe it would help to clarify the primary problem this is aimed at? I see a few potential benefits:

Better plumbing of secrets throughout provider code (to prevent accidental logging, etc.).
Allow providers to distinguish unset versus not-yet-resolved values during previews. (I think this is only relevant for previews?)
Allow providers to schedule Apply() callbacks before things have fully resolved.
...anything else?

(1) seems like a good thing, but it also seems like we could accomplish this with existing types. For example we should be able to give the user the option to hydrate secrets as Outputs if they ask for them:
type Foo struct {
    RawSecret     string                 `pulumi:"raw,optional" provider:"secret"`
    WrappedSecret pulumix.Output[string] `pulumi:"wrapped,optional" provider:"secret"`
}

I might go further and say this deserves its own Secret type since pulumix.Output is fairly low-level and most of its functionality (e.g. Apply) isn't relevant here.
(2) I haven't run into yet, although I also haven't been focused on preview behavior. I do anticipate some gotchas around validation, and my plan is to basically special case zero-values during previews.
I do think it would be helpful to allow a user to determine if a value is unknown, although I don't think we need all the Output machinery it accomplish that. This feels similar to the "null" problem in JSON which https://github.com/aarondl/opt/ and other similar libraries help with. Similar to the example above, I could imagine an Optional helper the user could use if they need it:
type Foo struct {
    OptionalRaw    string                   `pulumi:"raw,optional"`
    Optional       Optional[string]         `pulumi:"opt,optional"`
    OptionalSecret Optional[Secret[string]] `pulumi:"secret,optional" provider:"secret"`
}

I realize you can accomplish essentially the same thing with Output but there is a lot of value in scoping the API down to something small and obvious.
(3) is a non-goal IMO. As a user I would much rather the harness handle resolving everything before invoking my code versus me needing to know or care about unresolved things (the exception being (2) above). If we do really need a micro-optimizations like this (I'm skeptical), I would prefer that to be an alternative code path instead of sacrificing simplicity in the common case.

my plan is to basically special case zero-values during previews.

:screaming-intensifies:
Optional is not the same as zero is not the same as unknown. These are three very distinct values, please for all that's holy can we design a system that understands that.
I get that we're working with Go and it's a hot mess for specifying anything but I'd hope we can do better than just sticking with Go's (frankly stupid) opinion that zero values are always good sensible defaults and can just be used for everything.

I might go further and say this deserves its own Secret type since pulumix.Output is fairly low-level and most of its functionality (e.g. Apply) isn't relevant here.

Inputs to providers are either plain (not secret, not unknown) or output'y (potentially unknown, potentially secret). I don't think there's any cases where we allow secret but don't allow unknown.

Allow providers to schedule Apply() callbacks before things have fully resolved.

We really need to define what we mean by resolved.
As far as I know all inputs to providers are always fully resolved. The engine does not send async values to providers, it waits for all inputs to resolve and then sends those resolved values. Even Call and Construct wait for resolution.
But when I say resolved I mean it has a value, but that value might be "unknown". I suspect most of you are using resolved for "not unknown"?
Either way, there's no value for callbacks in providers. The values either known or not and you know that as soon as you're called and it's not going to asynchronously change.

Haha, I start observing words like holy and since religious feelings are involved we should definitely meet before writing any more :) (oh I feel those strong feeling also).

The goal is (1) and (2) with Apply being a useful (and optional) method to operate on (1) and (2).
I have currently implemented Apply as asynchronous, but I agree that the benefits there are dubious.
I think that we have spent enough time here that this is worth a synchronous meeting. I'll write up a doc and present during IaC ideation.

my plan is to basically special case zero-values during previews.

This does convince me that a change like this is well motivated.


Roll Plugins from 2d0cec9fb4a3 to f70794df658b (1 revision)

https://github.com/flutter/plugins/compare/2d0cec9fb4a3...f70794df658b
2022-01-21<EMAIL_ADDRESS>[camera]rename dispatchQueue to captureSessionQueue (flutter/plugins#4687)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-plugins-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Commit queue failed; closing this roll.


Icons as font instead of img

Hi
Any reason you opted to include image based icons in bootstrap which are limited to the 16px dimensions?
For example http://somerandomdude.com/work/iconic/    is available as open source fonts - means you can include icons in headers, buttons of various size etc since its vector based.
Could easily be implemented with the same approach as the current icons:

i.foobar:before { content: "X" }

Yes, I second this. It would allow size and color to change (black doesn't look good on .danger, for example)

There are forks that have this implemented:
For version 1.4 have a look at https://github.com/lightglitch/bootstrap-xtra (http://lightglitch.github.com/bootstrap-xtra/)
For version 2.0 you can use https://github.com/jasny/bootstrap/tree/2.0-icons
I'm not expecting that this will end up in the main version of bootstrap.

I know this is why Im surprised bootstrap opted to lock them self in to the 16px corner which feels a little unorthodox to do in 2012.

It boils down to two things:

There isn't much out there for open source icon fonts. It's a market that I suppose is wide open, but for now extremely limited.
Aesthetically, I have a high bar and particular style I'd like to support for iconography in Bootstrap. Glyphicons gets us close to that for now.

I'm not opposed to using fonts—that'd be pretty awesome. It's just a matter of availability and quality.

I'm a fan of iconic due to their licensing and quality.

@andriijas thanks for the link, nice icon sets!

@markdotto With the vector images of Glyphicons, you can easily create a font. Unfortunately this is difficult license wise. Perhaps you can talk with the designer.

@jasny, Glyphicons in bigger sizes are sold premium :(

Using a font also brings the affordance of better color control. The pure black icons (in my opinion) don't mesh well with the off-black text.

@pokonski Well sure, I'm willing to pay the 49 bucks. That still doesn't allow me to put it in bootstrap I'm afraid, since you're not allowed to distribute them. http://glyphicons.com/glyphicons-licenses/

@andriijas No that's not the way to do it. I've also modified mixins.less.
You can do a merge (with -Xours) and rebuild bootstrap. You can also fetch my fork and do a cherrypick. The whole thing is in commit https://github.com/jasny/bootstrap/commit/54b22aa388d79a67e072ea54f8429b3b427b9d11

here are some good ones too: http://www.ffdingbatsfont.com/erler/index.html FF encourages OS devs to include it.

I'll play devils advocate and throw out the issue that using fonts/font-face for icons can and does cause some accessibility concerns. For instance if you use :before or :after to generate content to show an icon, a large majority of screen readers will read that content. (not to mention ie support). If you used the current technique of <i>x<i> some text with an icon in front of it, they would also read the "x". If you were using js to create them that may be a different, I'm not sure on that.
Maybe if it was optional, and the sprite version was still included?

I've found this Set: Web Symbols Typeface http://www.justbenicestudio.com/studio/websymbols/
Web Symbols is a set of vector html-compliant typefaces, so it might be used in any size, color and browser. License OFL

I approve of websymbols!

These might be worth a look... apparently they are supposed to be accessible and won't be read by screen readers (nice set too) http://keyamoon.com/icomoon/#toHome

actually, what would be the coolest - is if any icon guru would like to make a dedicated bootstrap icon font.

IcoMoon is looking good, sadly it's commercial :(
Most interesting thing about IcoMoon ist the way it handles Screenreaders:
http://www.youtube.com/watch?v=EWq7idrW2yU

Using @media should solve issues with screen readers
@media screen {
   [class*="icon-"] {
      ...
   }
}

http://www.w3.org/TR/css3-reader/#reader

We are doing webapp, that simplify icon fonts customization for bootstrap http://nodeca.github.com/fontomas/
Hope to fix all issues soon. Let me know, if something can be improved.

+1 for Entypo, font is in my opinion the way to go, for maximal color/size possibilities.

I've implemented the iconic icons for bootstrap 2.0. See http://jasny.github.com/bootstrap/base-css.html#iconic
You can get it by pulling the 'dev.iconic' branch. https://github.com/jasny/bootstrap/tree/dev.iconic

@jasny +1!

It seems, that iconic needs hinting.

We did script to rescale & realign Entypo https://github.com/nodeca/fontomas/tree/master/bin . Now it can be combined with iconic/websymbols without problems.

+1 for entypo

"Though I doubt that it will be included in bootstrap now that a set was included in 2.0 it has to stick for backwards incompatibility."
It shouldn't. Going forward, there ought to be a standard way to combine multiple icon sets, ie a different class name or font-face or whatever.

I've been following this thread for some time now. I've been looking around myself for an icon font that would cut it, but none were quite good enough. So I made my own. I just finished up Font Awesome, an icon font specifically designed for Twitter Bootstrap.

Replicates all 120 icons currently a part of TW Bootstrap (yet designed from scratch)
Adds 30 more
Passes the screen reader test
Completely free for commercial use

http://fortaweso.me/font-awesome/
I'd love feedback.

Dave, looks awesome!

+1 . Really awesome.

:beer:

Dave, this is awesome - nice work :+1:

:+1:
Great!

Is there still any concrete argument against the integration of fonts instead of images ? I mean now that Dave created a font-replica, I don't think there is any obstacle left, is there ? At least none that outmatches the positive sides brought by fonts in terms of flexibility : any color, any size, any effect and style.

@Anahkiasen yes, there are some cons. Vector symbols are not fine in small sizes. Care sould be done about hinting. As far as i know, the best autohinter tool  now is ttfautohint http://www.freetype.org/ttfautohint/

I've done a lot to keep the icons readable all the way down to 12px, but some browsers still aren't perfect.
@puzrin Thanks for the link to the autohinter tool. I'll definitely check it out.

:sunny: :rainbow:

@davegandy impressive! You got almost 700 watchers in 3 weeks :D

@davegandy you made my dreams come true.  Amazing work!

Hi all,
I'm the designer designer behind Entypo. I'm currently working on some new glyphs to the suite and I welcome all feedback and ideas for glyphs you feel should be in there.
Cheers

@danielbruce if you need help with scripting to build fonts - let me know.

Great to see people helping with fonts. All that hype and popularity is paying off :)

@puzrin since I hardly know how to code at all I can use all the help I can get :)


How to extract element from GList

I am confused as to how to extract the GtkWidget element held inside the GList returned by gtk_container_get_children(). Any help would be appreciated.

This isn't currently supported. Hopefully issue #20 will lay the groundwork allow to allow me to add some support for GList.


Add missing rust-version in crates

I used cargo-msrv to determine the Minimum Supported Rust Version.
I could not get ruff_macros to compile on its own so I could not determine its msrv version.

I think we should use the same rust-version value, and use workspace inheritance to manage it.

We should use the same value as in rust-toolchain. (Is it normal to have both rust-toolchain and specify the version in Cargo.toml? I assume so since rust-toolchain isn't distributed with the crates IIUC.)


new(falco): add falco_libs.snaplen option, deprecate -S / --snaplen

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind cleanup
/kind feature

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area engine

What this PR does / why we need it:
As per title. This may be the last CLI option that we couldn't replicate in the falco config file.
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

new(falco): add falco_libs.snaplen option, deprecate -S / --snaplen

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: LucaGuerra
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [LucaGuerra]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/milestone 0.40.0

@FedeDP done!

LGTM label has been added.  Git tree hash: f0a8c91d0a21ef5deeff46f7c1b4eee5bba3e327

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP, LucaGuerra
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP,LucaGuerra]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, FedeDP, LucaGuerra
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Andreagit97,FedeDP,LucaGuerra]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment