gccgo: cgo: false positive cgo argument has Go pointer to Go pointer

What version of Go are you using (go version)?
go version go1.9 gccgo (GCC) 8.0.0 20180108 (experimental) linux/amd64
Description
The following code exists in CockroachDB:
https://github.com/cockroachdb/cockroach/blob/4e6ec545cc65dc0051a797996e9642b4b5e8bdc2/pkg/storage/engine/rocksdb.go#L447-L450
https://github.com/cockroachdb/cockroach/blob/4e6ec545cc65dc0051a797996e9642b4b5e8bdc2/pkg/storage/engine/rocksdb.go#L584
https://github.com/cockroachdb/cockroach/blob/4e6ec545cc65dc0051a797996e9642b4b5e8bdc2/c-deps/libroach/db.cc#L1616
Said plainly, initialization across cgo is done by passing a Go **C.DBEngine which points to nil to a C function that allocates and sets the pointed-to *C.DBEngine. This code works perfectly when compiled with gc, but produces this error when compiled with gccgo:
panic: runtime error: cgo argument has Go pointer to Go pointer

goroutine 24 [running]:
panic
        ../../../src/libgo/go/runtime/panic.go:543
github_com_cockroachdb_cockroach_pkg_storage_engine._cgoCheckPointer
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/cgo-c-prolog-gccgo:74
engine.$nested21
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:584
github_com_cockroachdb_cockroach_pkg_storage_engine.open.pN59_github_com_cockroachdb_cockroach_pkg_storage_engine.RocksDB
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:584
engine.newMemRocksDB
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:518
github_com_cockroachdb_cockroach_pkg_storage_engine.NewInMem
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/in_mem.go:37
github_com_cockroachdb_cockroach_pkg_server.CreateEngines.pN50_github_com_cockroachdb_cockroach_pkg_server.Config
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/server/config.go:483
github_com_cockroachdb_cockroach_pkg_server.Start.pN50_github_com_cockroachdb_cockroach_pkg_server.Server
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:858
cli.$nested72
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:518
cli.$nested70
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:500
created by cli.runStart
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:474 +3255

According to https://golang.org/cmd/cgo/#hdr-Passing_pointers:

Go code may pass a Go pointer to C provided the Go memory to which it points does not contain any Go pointers.

It's not exactly clear if nil Go pointers are permitted, but the gc implementation certainly permits it. Perhaps gccgo is missing a null check?
@ianlancetaylor

I believe from there it's https://github.com/golang/gofrontend/blob/65eaa9003db4effc9c5ffe9c955e9534ba5d7d15/libgo/go/runtime/cgocall.go#L93 to https://github.com/golang/gofrontend/blob/65eaa9003db4effc9c5ffe9c955e9534ba5d7d15/libgo/go/runtime/cgocall.go#L206 to https://github.com/golang/gofrontend/blob/65eaa9003db4effc9c5ffe9c955e9534ba5d7d15/libgo/go/runtime/cgocall.go#L238, though I haven't traced through this in gdb yet.

OK, here's the actual panic backtrace:
#0  runtime.gopanic (e=...) at ../../../src/libgo/go/runtime/panic.go:498
#1  0x00007ffff6d0114b in runtime.cgoCheckUnknownPointer (msg=..., p=<optimized out>) at ../../../src/libgo/go/runtime/cgocall.go:238
#2  runtime.cgoCheckArg (t=<optimized out>, p=<optimized out>, indir=indir@entry=true, top=top@entry=true, msg=...)
    at ../../../src/libgo/go/runtime/cgocall.go:206
#3  0x00007ffff6d0124c in runtime.cgoCheckArg (t=<optimized out>, p=<optimized out>, indir=indir@entry=true, top=top@entry=true, msg=...)
    at ../../../src/libgo/go/runtime/cgocall.go:192
#4  0x00007ffff6d0124c in runtime.cgoCheckArg (
    t=t@entry=0x1e63960 <github_com_cockroachdb_cockroach_pkg_storage_engine..github_com_cockroachdb_cockroach_pkg_storage_engine._Ctype_struct___7..d>, p=<optimized out>, indir=<optimized out>, top=<optimized out>, msg=...) at ../../../src/libgo/go/runtime/cgocall.go:192
#5  0x00007ffff6d015bb in runtime.cgoCheckPointer (ptr=..., args=...) at ../../../src/libgo/go/runtime/cgocall.go:93
#6  0x0000000000f782d7 in github_com_cockroachdb_cockroach_pkg_storage_engine._cgoCheckPointer (ptr=..., args=...) at cgo-c-prolog-gccgo:74
#7  0x0000000000f3c4d4 in engine.func1 (_cgo0=0xc420a3c5f0, _cgo1=..., _cgo2=...)
    at /home/tduberstein/local/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:584
#8  0x0000000000f3c040 in github_com_cockroachdb_cockroach_pkg_storage_engine.RocksDB.open (r=0xc420a3c580)
    at /home/tduberstein/local/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:584
#9  0x0000000000f3b57b in engine.newMemRocksDB (attrs=..., param=..., MaxSizeBytes=536870912)
    at /home/tduberstein/local/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:518
#10 0x0000000000e2f30e in github_com_cockroachdb_cockroach_pkg_storage_engine.NewInMem (attrs=..., cacheSize=1048576)
    at /home/tduberstein/local/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/in_mem.go:37
#11 0x0000000000e4d84f in engine.testBatchBasics (t=0xc420b180f0, writeOnly=false,
    commit=0x1e521a0 <github_com_cockroachdb_cockroach_pkg_storage_engine.TestBatchBasics..func1..f>)
    at /home/tduberstein/local/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/batch_test.go:56
#12 0x0000000000e5000f in github_com_cockroachdb_cockroach_pkg_storage_engine.TestBatchBasics (t=0xc420b180f0)
    at /home/tduberstein/local/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/batch_test.go:132
#13 0x00007ffff6d900d0 in testing.tRunner (param=<optimized out>, fn=<optimized out>) at ../../../src/libgo/go/testing/testing.go:777
#14 0x00007ffff6e20975 in __morestack () at ../../../src/libgcc/config/i386/morestack.S:546
#15 0x00007ffff6d901cb in testing.testing..thunk24 (__go_thunk_parameter=<optimized out>) at ../../../src/libgo/go/testing/testing.go:824
#16 0x00007ffff6d18a79 in runtime.kickoff () at ../../../src/libgo/go/runtime/proc.go:1161
#17 0x00007ffff59f0d40 in ?? () from /lib64/libc.so.6

confirming that the passed pointer points to a null pointer:
(gdb) frame 8
#8  0x0000000000f3c040 in github_com_cockroachdb_cockroach_pkg_storage_engine.RocksDB.open (r=0xc420a3c580)
    at /home/tduberstein/local/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:584
584             status := C.DBOpen(&r.rdb, goToCSlice([]byte(r.cfg.Dir)),
(gdb) print r.rdb
$4 = (.github.com/cockroachdb/cockroach/pkg/storage/engine._Ctype_struct_DBEngine *) 0x0

interesting stuff starts at frame 5:
(gdb) frame 5
#5  0x00007ffff6d015bb in runtime.cgoCheckPointer (ptr=..., args=...) at ../../../src/libgo/go/runtime/cgocall.go:93
93              cgoCheckArg(t, ep.data, t.kind&kindDirectIface == 0, top, cgoCheckPointerFail)
(gdb) list
88                      default:
89                              throw("can't happen")
90                      }
91              }
92
93              cgoCheckArg(t, ep.data, t.kind&kindDirectIface == 0, top, cgoCheckPointerFail)
94      }
95
96      const cgoCheckPointerFail = "cgo argument has Go pointer to Go pointer"
97      const cgoResultFail = "cgo result has Go pointer"

things we care about are arguments to cgoCheckArg:
(gdb) print t
$9 = (.runtime._type *) 0x1e645a0 <github_com_cockroachdb_cockroach_pkg_storage_engine..github_com_cockroachdb_cockroach_pkg_storage_engine._Ctype_struct___7..d>
(gdb) print ep.data
value has been optimized out
(gdb) print t.kind&(1<<5)
$10 = 0
(gdb) print top
$11 = <optimized out>

so we don't know the value of top, but it's always true if len(args) == 0:
(gdb) print args
$12 = {__values = <optimized out>, __count = 0, __capacity = <optimized out>}

interestingly, t is different than what I mistakenly thought it was previously:
(gdb) print *t.string
$13 = 0x1e34470 "\tgithub_com_cockroachdb_cockroach_pkg_storage_engine\tengine._Ctype_struct___7"

next frame:
(gdb) frame 4
#4  0x00007ffff6d0124c in runtime.cgoCheckArg (
    t=t@entry=0x1e645a0 <github_com_cockroachdb_cockroach_pkg_storage_engine..github_com_cockroachdb_cockroach_pkg_storage_engine._Ctype_struct___7..d>, p=<optimized out>, indir=<optimized out>, top=<optimized out>, msg=...) at ../../../src/libgo/go/runtime/cgocall.go:192
192                             cgoCheckArg(f.typ, add(p, f.offset), true, top, msg)
(gdb) list
187                             }
188                             cgoCheckArg(st.fields[0].typ, p, st.fields[0].typ.kind&kindDirectIface == 0, top, msg)
189                             return
190                     }
191                     for _, f := range st.fields {
192                             cgoCheckArg(f.typ, add(p, f.offset), true, top, msg)
193                     }
194             case kindPtr, kindUnsafePointer:
195                     if indir {
196                             p = *(*unsafe.Pointer)(p)

so this type is a struct (that's where we are in the type switch). we know indir is true from the previous frame, so we go straight into this range. unfortunately, f is optimized out and the next few frames have been inlined. frame 2:
(gdb) frame 2
#2  runtime.cgoCheckArg (t=<optimized out>, p=<optimized out>, indir=indir@entry=true, top=top@entry=true, msg=...)
    at ../../../src/libgo/go/runtime/cgocall.go:206
206                     cgoCheckUnknownPointer(p, msg)
(gdb) list
201                     }
202                     if !top {
203                             panic(errorString(msg))
204                     }
205
206                     cgoCheckUnknownPointer(p, msg)
207             }
208     }
209
210     // cgoCheckUnknownPointer is called for an arbitrary pointer into Go

this time we land in the kindPtr, kindUnsafePointer case, and:
(gdb) print indir
$27 = true
(gdb) print top
$28 = true

so we execute p = *(*unsafe.Pointer)(p) (it's just above the code pasted above) and proceed into cgoCheckUnknownPointer:
(gdb) frame 1
#1  0x00007ffff6d0114b in runtime.cgoCheckUnknownPointer (msg=..., p=<optimized out>) at ../../../src/libgo/go/runtime/cgocall.go:238
238                                     panic(errorString(msg))
(gdb) list
233                             if i != 1*sys.PtrSize && !hbits.morePointers() {
234                                     // No more possible pointers.
235                                     break
236                             }
237                             if hbits.isPointer() && cgoIsGoPointer(*(*unsafe.Pointer)(unsafe.Pointer(base + i))) {
238                                     panic(errorString(msg))
239                             }
240                             hbits = hbits.next()
241                     }
242

this is what's surpring. at this point, after the dereference, I would expect to end up with the null pointer we started with, but that's obviously not the case. the actual value of p is optimized out, but obviously cgoInRange returned true for it. there's more dereferencing going on here which I don't understand. Here's what information remains in this stack frame:
(gdb) info locals
span = 0x7ffff5658c88
b =<PHONE_NUMBER>60
hbits = {bitp = 0xc41fff9fa8 "Y\360\363\322t\360\374Y\360\363", <incomplete sequence \322>, shift = 3}
sink$0 = 25
n = 112
roots = <optimized out>
base =<PHONE_NUMBER>60
i = <optimized out>

let me know if there's more detail I can provide.

I didn't notice that you weren't showing the full function call.  Looking at the source code, it's
	status := C.DBOpen(&r.rdb, goToCSlice([]byte(r.cfg.Dir)),
		C.DBOptions{
			cache:             r.cache.cache,
			block_size:        C.uint64_t(blockSize),
			wal_ttl_seconds:   C.uint64_t(walTTL),
			logging_enabled:   C.bool(log.V(3)),
			num_cpu:           C.int(runtime.NumCPU()),
			max_open_files:    C.int(maxOpenFiles),
			use_switching_env: C.bool(newVersion == versionCurrent),
			must_exist:        C.bool(r.cfg.MustExist),
			extra_options:     goToCSlice(r.cfg.ExtraOptions),
		})

The type passed to cgoCheckArg is _Ctype_struct___7.  Building the package and looking at the generated _cgo_gotypes.go file shows type _Ctype_DBOptions = _Ctype_struct___7 and
type _Ctype_struct___7 struct {
	cache			*_Ctype_struct_DBCache
	block_size		_Ctype_uint64_t
	wal_ttl_seconds		_Ctype_uint64_t
	logging_enabled		_Ctype__Bool
	_			[3]byte
	num_cpu			_Ctype_int
	max_open_files		_Ctype_int
	use_switching_env	_Ctype__Bool
	must_exist		_Ctype__Bool
	_			[2]byte
	extra_options		_Ctype_struct___0
}

So the problem is not the &r.rdb argument; it's the C.DBOptions argument.
The stack trace shows it looking at this struct in frame 4, and then looking at another struct in frame 3.  That suggests that the second struct is the extra_options field.  The type of that field is
type _Ctype_struct___0 struct {
	data	*_Ctype_char
	len	_Ctype_int
	_	[4]byte
}

In frame 2 we are looking at a pointer, which in this case must be the data field.  We're passing a struct of type C.DBOptions which has a field of type C.DBSlice.
Looking back at the Go code, the field is initialized by a call to goToCSlice.  That just passes a pointer to the start of a []byte.  Normally that should be OK, but in this case we don't know what else might be in the same memory block as that []byte.  If that []byte is allocated in the same memory region as something else that contains a Go pointer, then the runtime will report an error (this is #14210).  I don't know if that is what is happening here, but it's something to check.
Can you find out what value is in r.cfg.ExtraOptions at the point of failure?

It's an empty slice.
(gdb) frame 8
#8  0x0000000000eff807 in github_com_cockroachdb_cockroach_pkg_storage_engine.RocksDB.open (r=0xc4205bc000)
    at /home/tduberstein/local/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:585
585             status := C.DBOpen(&r.rdb, goToCSlice([]byte(r.cfg.Dir)),
(gdb) print r.cfg.ExtraOptions
$3 = {__values = 0x0, __count = 0, __capacity = 0}

Hmmm, OK, clearly there is something wrong with my analysis.  There is another pointer in the C.DBOptions value: the cache field is set to r.cache.cache, which is a *C.DBCache value.  What does that point to?
@thanm Any luck reproducing this locally?

Heh, figured you'd ask for that.
(gdb) print r.cache.cache
$4 = (.github.com/cockroachdb/cockroach/pkg/storage/engine._Ctype_struct_DBCache *) 0x7fffe7c13040
(gdb) print *r.cache.cache
$5 = {<No data fields>}

Oh, and the memory at that address is null.
(gdb) x 0x7fffe7c13040
0x7fffe7c13040: 0x00000000

Thanks.  I think we need to know whether that value (0x7fffe7c13040) is a Go pointer (whether cgoIsGoPointer would return true for it) and, if so, what allocated block it is pointing to.  If I'm reading the code correctly, it is set by a call to C.DBNewCache.

As you say - it is set by a call to C.DBNewCache, which is obviously not a
Go function (it is implemented in c-deps/libroach/cache.cc) with a call to
RocksDB's C++ API.
On Fri, Feb 2, 2018 at 3:04 PM, Ian Lance Taylor<EMAIL_ADDRESS>wrote:

Thanks. I think we need to know whether that value (0x7fffe7c13040) is a
Go pointer (whether cgoIsGoPointer would return true for it) and, if so,
what allocated block it is pointing to. If I'm reading the code correctly,
it is set by a call to C.DBNewCache.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/23391#issuecomment-362691610, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABdsPN_xOuPBW0ztfhxva1mD-xvFIfAUks5tQ2oygaJpZM4RYcDX
.

Thanks, but does cgoIsGoPointer return true for it?  And is that in fact the value that the runtime package is finding the error for?

How can I call that function? I don't see a symbol that fits the name, so I
suspect it's all been inlined. Any ideas?
On Fri, Feb 2, 2018 at 4:04 PM, Ian Lance Taylor<EMAIL_ADDRESS>wrote:

Thanks, but does cgoIsGoPointer return true for it? And is that in fact
the value that the runtime package is finding the error for?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/23391#issuecomment-362706832, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABdsPJ8119QJd7WK6Qc8wM7TbUJG7gA5ks5tQ3hvgaJpZM4RYcDX
.

Do you feel like rebuilding libgo without optimization?  You should be able to do that by running make GOCFLAGS_FOR_TARGET=-g.

Done:
nm /home/tduberstein/local/gcc/gcc/lib64/libgo.so.13 | grep cgoIsGo
00000000012df8a9 t runtime.cgoIsGoPointer

but how do I call it?
(gdb) info functions cgoIsGo
All functions matching regular expression "cgoIsGo":

File ../../../src/libgo/go/runtime/cgocall.go:
static bool runtime.cgoIsGoPointer(Pointer);
(gdb) print cgoIsGoPointer(0x7fffe4a13040)
No symbol "cgoIsGoPointer" in current context.
(gdb) print runtime.cgoIsGoPointer(0x7fffe4a13040)
No symbol "runtime" in current context.

Ah, just need to stringify it!
(gdb) print 'runtime.cgoIsGoPointer'(0x7fffe4a13040)
$2 = false

Now that nothing is inlined, this is much easier to reason about.
In frame 5 we can see the pointer in question is 0xc420083a00:
(gdb) info args
ptr = {
  __type_descriptor = 0x1e0e040 <github_com_cockroachdb_cockroach_pkg_storage_engine..github_com_cockroachdb_cockroach_pkg_storage_engine._Ctype_struct___7..d>, __object = 0xc420083a00}
args = {__values = 0x0, __count = 0, __capacity = 0}

so that's not r.cache.cache at all. In fact, I can't find this pointer anywhere. Note that C.DBOptions is being passed by value and here we are inspecting a pointer to it. This looks like miscompilation?

Here's the cgo stub (reformatted by hand):
status := func(_cgo0 **_Ctype_struct_DBEngine, _cgo1 _Ctype_struct___0, _cgo2 _Ctype_struct___7) _Ctype_struct___1 {
        _cgoCheckPointer(_cgo0, true)
        _cgoCheckPointer(_cgo2)
        return (_Cfunc_DBOpen)(_cgo0, _cgo1, _cgo2)
}(&r.rdb, goToCSlice([]byte(r.cfg.Dir)),
_Ctype_struct___7{
        cache:             r.cache.cache,
        block_size:        _Ctype_uint64_t(blockSize),
        wal_ttl_seconds:   _Ctype_uint64_t(walTTL),
        logging_enabled:   _Ctype__Bool(log.V(3)),
        num_cpu:           _Ctype_int(runtime.NumCPU()),
        max_open_files:    _Ctype_int(maxOpenFiles),
        use_switching_env: _Ctype__Bool(newVersion == versionCurrent),
        must_exist:        _Ctype__Bool(r.cfg.MustExist),
        extra_options:     goToCSlice(r.cfg.ExtraOptions),
})

so the slice isn't being checked, but the DBOptions is. Since cgoCheckPointer takes a interface{}, looks like we're allocating a Go pointer to this structure! Smells like a smoking gun to me.

Assigning DBOptions to an interface value, including allocating it, is normal behavior.  We aren't going to check that pointer itself; we're going to check the value to which it points.  Now that you have a version without optimization, can you find the pointer value that actually causes the problem?  The one passed to cgoCheckUnknownPointer when it panics?

I probably can't get back to this until Monday. Would you consider doing
this on a call or over Gitter?
On Feb 2, 2018 18:37, "Ian Lance Taylor"<EMAIL_ADDRESS>wrote:

Assigning DBOptions to an interface value, including allocating it, is
normal behavior. We aren't going to check that pointer itself; we're going
to check the value to which it points. Now that you have a version without
optimization, can you find the pointer value that actually causes the
problem? The one passed to cgoCheckUnknownPointer when it panics?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/23391#issuecomment-362741385, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABdsPAylLSSzoOzA_nQ-9LSG46w62Z4sks5tQ5xRgaJpZM4RYcDX
.

Sorry, no, I am continually interrupted.
Most effective would be some way that I can reproduce the problem myself.

(gdb) frame 5
#5  0x00007ffff6cbb3e3 in runtime.cgoCheckPointer (ptr=..., args=...)
    at /usr/local/home/tduberstein/gcc/objdir/../src/libgo/go/runtime/cgocall.go:93
93              cgoCheckArg(t, ep.data, t.kind&kindDirectIface == 0, top, cgoCheckPointerFail)
(gdb) print *ep
$17 = {
  _type = 0x1d8b5c0 <github_com_cockroachdb_cockroach_pkg_storage_engine..github_com_cockroachdb_cockroach_pkg_storage_engine._Ctype_struct___7..d>, data = 0xc420085a00}
(gdb) print args
$18 = {__values = 0x0, __count = 0, __capacity = 0}

the C.DBOptions struct is at 0x1d8b5c0.
(gdb) frame 4
#4  0x00007ffff6cbbe75 in runtime.cgoCheckArg (
    t=0x1d8b5c0 <github_com_cockroachdb_cockroach_pkg_storage_engine..github_com_cockroachdb_cockroach_pkg_storage_engine._Ctype_struct___7..d>,
    p=0xc420085a00, indir=true, top=true, msg=...) at /usr/local/home/tduberstein/gcc/objdir/../src/libgo/go/runtime/cgocall.go:192
192                             cgoCheckArg(f.typ, add(p, f.offset), true, top, msg)
(gdb) print f
$25 = {name = 0x1d8b5a0 <go..C3252>, pkgPath = 0x1d8b5b0 <go..C3253>,
  typ = 0x1d6f660 <github_com_cockroachdb_cockroach_pkg_storage_engine..github_com_cockroachdb_cockroach_pkg_storage_engine._Ctype_struct___0..d>,
  tag = 0x0, offset = 80}
(gdb) print *f.typ.string
$28 = 0x1d50400 "\tgithub_com_cockroachdb_cockroach_pkg_storage_engine\tengine._Ctype_struct___0"

we're inspecting one of the struct's fields, and it has type _Ctype_struct___0:
type _Ctype_struct___0 struct {
        data    *_Ctype_char
        len     _Ctype_int
        _       [4]byte
}

which is also the type of the second argument to C.DBOpen - it's the return value of goToCSlice: C.DBSlice. Must be extra_options, given:
typedef struct {
  DBCache* cache;
  uint64_t block_size;
  uint64_t wal_ttl_seconds;
  bool logging_enabled;
  int num_cpu;
  int max_open_files;
  bool use_switching_env;
  bool must_exist;
  bool read_only;
  DBSlice extra_options;
} DBOptions;

(gdb) frame 3
#3  0x00007ffff6cbbe75 in runtime.cgoCheckArg (
    t=0x1d6f660 <github_com_cockroachdb_cockroach_pkg_storage_engine..github_com_cockroachdb_cockroach_pkg_storage_engine._Ctype_struct___0..d>,
    p=0xc420085a50, indir=true, top=true, msg=...) at /usr/local/home/tduberstein/gcc/objdir/../src/libgo/go/runtime/cgocall.go:192
192                             cgoCheckArg(f.typ, add(p, f.offset), true, top, msg)
(gdb) print p
$40 = (Pointer) 0xc420085a50
(gdb) print f
$47 = {name = 0x1d6f310 <go..C450>, pkgPath = 0x1d6f320 <go..C451>,
  typ = 0x1d6f420 <type...1github_com_cockroachdb_cockroach_pkg_storage_engine._Ctype_char>, tag = 0x0, offset = 0}
(gdb) print *f.typ.string
$49 = 0x1d504b8 "*\tgithub_com_cockroachdb_cockroach_pkg_storage_engine\tengine._Ctype_char"

We're checking the first field, which is a *_Ctype_char, and we have a pointer to it.
(gdb) frame 2
#2  0x00007ffff6cbbf74 in runtime.cgoCheckArg (t=0x1d6f420 <type...1github_com_cockroachdb_cockroach_pkg_storage_engine._Ctype_char>,
    p=0xc4200c0af0, indir=true, top=true, msg=...) at /usr/local/home/tduberstein/gcc/objdir/../src/libgo/go/runtime/cgocall.go:206
206                     cgoCheckUnknownPointer(p, msg)
(gdb) info args
t = 0x1d6f420 <type...1github_com_cockroachdb_cockroach_pkg_storage_engine._Ctype_char>
p = 0xc4200c0af0
indir = true
top = true
msg = 0x7ffff6fbb370 "cgo argument has Go pointer to Go pointer"
(gdb) print p
$51 = (Pointer) 0xc4200c0af0

Now we've deferenced the pointer - so now our p points to the *_Ctype_char itself.
(gdb) frame 1
#1  0x00007ffff6cbc248 in runtime.cgoCheckUnknownPointer (p=0xc4200c0af0, msg=...)
    at /usr/local/home/tduberstein/gcc/objdir/../src/libgo/go/runtime/cgocall.go:238
238                                     panic(errorString(msg))
(gdb) print p
$52 = (Pointer) 0xc4200c0af0

and the next frame panics.
However, checking this pointer is only valid if the slice is not empty, right? Let's check if it is:
(gdb) frame 8
#8  0x0000000000edfc5c in github_com_cockroachdb_cockroach_pkg_storage_engine.RocksDB.open (r=0xc4204a8000)
    at /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:587
587             status := C.DBOpen(&r.rdb, goToCSlice([]byte(r.cfg.Dir)),
(gdb) print r.cfg.ExtraOptions
$53 = {__values = 0x0, __count = 0, __capacity = 0}

Seems like we're passing a nil slice into goToCSlice. What does that function do?
func goToCSlice(b []byte) C.DBSlice {
        if len(b) == 0 {
                return C.DBSlice{data: nil, len: 0}
        }
        return C.DBSlice{
                data: (*C.char)(unsafe.Pointer(&b[0])),
                len:  C.int(len(b)),
        }
}

(gdb) print 'engine.goToCSlice'(r.cfg.ExtraOptions)
$67 = {data = 0x0, len = 0, _ = "\000\000\000"}

and DBSlice is:
typedef struct {
  char* data;
  int len;
} DBSlice;

So something is going wrong here - we're passing a C.DBSlice with nil data pointer into C.DBOpen, but it's emerging on the other side with some garbage value. What did we get for the length of the slice?
(gdb) frame 3
#3  0x00007ffff6cbbe75 in runtime.cgoCheckArg (
    t=0x1d6f660 <github_com_cockroachdb_cockroach_pkg_storage_engine..github_com_cockroachdb_cockroach_pkg_storage_engine._Ctype_struct___0..d>,
    p=0xc420085a50, indir=true, top=true, msg=...) at /usr/local/home/tduberstein/gcc/objdir/../src/libgo/go/runtime/cgocall.go:192
192                             cgoCheckArg(f.typ, add(p, f.offset), true, top, msg)
(gdb) print *(st.fields.__values+1)
$64 = {name = 0x1d6f470 <go..C458>, pkgPath = 0x1d6f480 <go..C459>,
  typ = 0x1d6f560 <github_com_cockroachdb_cockroach_pkg_storage_engine..github_com_cockroachdb_cockroach_pkg_storage_engine._Ctype_int..d>,
  tag = 0x0, offset = 16}
(gdb) frame 2
#2  0x00007ffff6cbbf74 in runtime.cgoCheckArg (t=0x1d6f420 <type...1github_com_cockroachdb_cockroach_pkg_storage_engine._Ctype_char>,
    p=0xc4200c0af0, indir=true, top=true, msg=...) at /usr/local/home/tduberstein/gcc/objdir/../src/libgo/go/runtime/cgocall.go:206
206                     cgoCheckUnknownPointer(p, msg)
(gdb) print p
$65 = (Pointer) 0xc4200c0af0
(gdb) x 0xc4200c0af0+16
0xc4200c0b00:<PHONE_NUMBER>0000000000000000000001

This is looking really suspicious. We passed in nil data and a zero length but it came out the other side with length 1. @ianlancetaylor is this enough to go on?

I haven't really looked at the above yet, but I committed a patch on Saturday that could and did lead to corruption of the slice structure in very rare cases.

Could you link to the patch?

The patch is https://gcc.gnu.org/ml/gcc-patches/2018-02/msg00140.html .

Rebuilt and retested - the issue still reproduces.

x 0xc4200c0af0+16

Why add 16?  The length is at offset 8, and it's only four bytes long.
Not that that explains anything.
Can you go to your frame 7 (engine.func1) and print the value of cgo2?  It should be the DBOptions struct.  If the pointer is nil there we need to find out where it changes.

Added 16 because that's the field's offset - I printed the field out at one
point. I'll try to get you that value in an hour or so.
On Feb 5, 2018 18:28, "Ian Lance Taylor"<EMAIL_ADDRESS>wrote:
x 0xc4200c0af0+16
Why add 16? The length is at offset 8, and it's only four bytes long.
Not that that explains anything.
Can you go to your frame 7 (engine.func1) and print the value of cgo2? It
should be the DBOptions struct. If the pointer is nil there we need to find
out where it changes.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/23391#issuecomment-363256427, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABdsPO8UUHCi-oo2gn0UnhhYbMVG9oF6ks5tR46HgaJpZM4RYcDX
.

Scratch that, I won't be able to check until tomorrow.
On Feb 5, 2018 18:30, "Tamir Duberstein"<EMAIL_ADDRESS>wrote:

Added 16 because that's the field's offset - I printed the field out at
one point. I'll try to get you that value in an hour or so.
On Feb 5, 2018 18:28, "Ian Lance Taylor"<EMAIL_ADDRESS>wrote:
x 0xc4200c0af0+16
Why add 16? The length is at offset 8, and it's only four bytes long.
Not that that explains anything.
Can you go to your frame 7 (engine.func1) and print the value of cgo2? It
should be the DBOptions struct. If the pointer is nil there we need to
find out where it changes.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/23391#issuecomment-363256427, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABdsPO8UUHCi-oo2gn0UnhhYbMVG9oF6ks5tR46HgaJpZM4RYcDX
.

@ianlancetaylor cgo2 is not nil in frame 7:
(gdb) frame 7
#7  0x0000000000ee00f0 in engine.func1 (_cgo0=0xc4204a8070, _cgo1=..., _cgo2=...)
    at /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:587
587             status := C.DBOpen(&r.rdb, goToCSlice([]byte(r.cfg.Dir)),
(gdb) info args
_cgo0 = 0xc4204a8070
_cgo1 = {data = 0x0, len = 0, _ = "\000\000\000"}
_cgo2 = {cache = 0x7fffe4c13040, block_size = 32768, wal_ttl_seconds = 0, logging_enabled = false, _ = "\000\000", num_cpu = 16,
  max_open_files = 10000, use_switching_env = false, must_exist = false, read_only = false, _ = "", extra_options = {data = 0x0, len = 0,
    _ = "\000\000\000"}}

Thanks.  I didn't expect _cgo2 to be nil, since it is a struct, but it does show that the pointer in the extra_options field is nil.
And now I see the problem.  The value stored in the offset field has changed and the runtime hasn't kept up.  Argh.

Sent https://golang.org/cl/92275.  Thanks for your patience.

No problem. Thanks for your help and persistence.


wandb (Weights and Biases) logging included in the code

I have not used wandb before but it seems to be an alternative to traditional Python logging which saves logs to a website? I have not been able to run the code with the references to wandb included (it requires a login and then refuses access to this project). The code seems to work if I comment out any reference to wandb, but rather than submitting a pull request for this, I thought I would submit this Issue in case wandb has been intentionally left in? As far as I can see, there are only references to wandb in the main.py file.

Yeah, the point of wandb would be to improve reproducibility. I think for using this in practice, it's indeed better to remove it or add a fail-safe handler. I think I'd go for removing—would you be okay with opening a PR for that?


bug: IOS large title overlaps with back button

Bug Report
Ionic version:
[x] 5.0.0
Current behavior:
when navigating for the first time from one on the tabs to child page and then click the back button the problem doesn't occur. but if you after that moved to another tab and navigate from it to a child page you will see the title of the first tap overlaps with the current child back button.
Expected behavior:
first tab title should not appear in a child page of another tab
Steps to reproduce:
1 - from tab 1 click on the button which will navigate to details-page 1 then go back to tab1
2 - go to tab 2 and click on the button  which will navigate to details-page 2, after that click on back-button now you will notice that tab 1 large title overlaps the tab2 header then it will disappear
Related code:
Github(https://github.com/3adeling/ionic-5-header-title-issue)
Ionic info:
Ionic:

   Ionic CLI                     : 6.0.2 (C:\Users\adela\AppData\Roaming\npm\node_modules\@ionic\cli)
   Ionic Framework               : @ionic/angular 5.0.0
   @angular-devkit/build-angular : 0.803.25
   @angular-devkit/schematics    : 8.3.25
   @angular/cli                  : 8.3.25
   @ionic/angular-toolkit        : 2.1.2

Capacitor:

   Capacitor CLI   : 1.5.0
   @capacitor/core : 1.5.0

Utility:

   cordova-res : 0.9.0
   native-run  : not installed

System:

   NodeJS : v12.14.0 (C:\Program Files\nodejs\node.exe)
   npm    : 6.13.6
   OS     : Windows 10```

It looks like this is happening with the following route setup:
/tabs/tab-one --> /child

Leaving the tabs context causes app-tabs/ion-tabs to be considered the "leavingEl". This causes the framework to incorrectly select the leaving large title. For example when on Tab 2, it will (incorrectly) select the title for Tab 1; however, since Tab 1 is hidden, its dimensions will all be 0 so the animating large title will in the wrong location.
The following route setup does not cause the issue because only tab-one is considered the leavingEl, not the entire tabs context:
/tabs/tab-one --> /tabs/child

I will look into a fix for the issue and will post an update here when I have more to share. Thanks!

as a workaround, instead of:
    <ion-buttons slot="start">
      <ion-back-button></ion-back-button>
    </ion-buttons>

you can use:
<ion-back-button slot="start"></ion-back-button>
without <ion-buttons> wrapper. it won't do the same behavior but at least it won't show wrong title label.

Thanks for the issue. Can you try the following dev build and let me know if it resolves the issue?
npm i<EMAIL_ADDRESS>

@liamdebeasi yes it resolves the issue. Thanks for your efforts.

Glad to hear the issue is fixed. I am going to keep the issue open until I merge the fix in. Thanks!

Thanks for the issue. This has been resolved via https://github.com/ionic-team/ionic/pull/20555 and will be available in an upcoming release of Ionic Framework.


Memory leak in traffic manager

It seems traffic manager is not correctly cleaned up after actors are destroyed and world is reloaded. The issue is found on version 0.9.9 and 0.9.8. Version 0.9.6 which does not have traffic manager handling autopilot works fine.
To reproduce

run docker of (0.9.9, 0.9.8 or 0.9.6) on ubuntu 18.04. I use singularity so
singularity exec --nv carla_0.9.9.sif /bin/bash /home/carla/CarlaUE4.sh -benchmark -fps=10 -carla-server -opengl
Activate python environment with corresponding carla egg installed.
Execute the minimal example below.
Check memory usage with htop in the terminal. The same code runs on all three versions of carla. Memory usage is constant on 0.9.6 but increasing on 0.9.9/0.9.8

import carla
import numpy as np

def spawn_vehicles(world, client, num_vehicles):
    vehicles = []
    blueprints = world.get_blueprint_library().filter("vehicle.*")
    SpawnActor = carla.command.SpawnActor
    SetAutopilot = carla.command.SetAutopilot
    FutureActor = carla.command.FutureActor

    # spawn_points
    spawn_points = world.get_map().get_spawn_points()
    np.random.shuffle(spawn_points)
    spawn_points = spawn_points[0:num_vehicles]
    # batch commands
    batch = []
    for n, transform in enumerate(spawn_points):
        blueprint = np.random.choice(blueprints)
        blueprint.set_attribute('role_name', 'autopilot')
        # batch.append(SpawnActor(blueprint, transform))
        batch.append(SpawnActor(blueprint, transform).then(SetAutopilot(FutureActor, True)))

    for response in client.apply_batch_sync(batch, do_tick=True):
        if not response.error:
            vehicles.append(response.actor_id)

    print(f'Spawned {len(vehicles)} vehicles. Should Spawn {num_vehicles}')

    return vehicles

def set_sync_mode(world, sync):
    # traffic_manager.set_global_distance_to_leading_vehicle(2.0)
    # traffic_manager = client.get_trafficmanager(8000)
    # traffic_manager.set_synchronous_mode(sync)
    settings = world.get_settings()
    settings.synchronous_mode = sync
    settings.fixed_delta_seconds = 0.1

    world.apply_settings(settings)

def run():
    for i in range(100):
        print(i)
        client = carla.Client("localhost", 2000)
        client.set_timeout(5.0)
        world = client.load_world("Town01")

        set_sync_mode(world, True)
        world.tick()

        # spawn
        vehicles = spawn_vehicles(world, client, 100)
        world.tick()

        # try the best to clean world
        client.apply_batch_sync([carla.command.SetAutopilot(x, False) for x in vehicles], True)
        client.apply_batch_sync([carla.command.DestroyActor(x) for x in vehicles], True)

        client.reload_world()
        vehicles = None
        client = None
        world = None

if __name__ == '__main__':
    run()

I tried to clean up as much as possible: autopilot is turned off, actors are destroyed, client is disconnected and world is reloaded. Yet memory is not freed after each iteration.
I suspect tm is the issue because if one does not touch autopilot, the memory leak will disappear. Specifically

While spawning, do not turn on autopilot
use
batch.append(SpawnActor(blueprint, transform))
comment out
batch.append(SpawnActor(blueprint, transform).then(SetAutopilot(FutureActor, True)))
While cleaning, do not turn off autopilot
comment out
client.apply_batch_sync([carla.command.SetAutopilot(x, False) for x in vehicles], True)

Hi,
Thanks for reporting this issue! If you are worried about why TM stays alive after a map change/reload, that's an intended behavior. Since the client doesn't shut down automatically after a map change, the TM doesn't as well. Let me know if this solves your doubts!

Thanks for your answer! Why memory used by TM is not freed after the actor is destroyed? This prevents TM being used to benchmark autonomous agent or collect dataset where actors are respawned and map is reloaded in each episode. Or is there a way to shut down client from the pythonAPI?

This is indeed an issue, we are currently changing the architecture of TM for 0.9.10 to allow for better memory cleanup, among other things. If it's a benchmark you want, maybe it's better to shut down the client anyway, although I can understand the problem with it.

Thanks! I’m looking forward to the new release.

First of all, thanks @jackbart94 folks for confirming this is an issue and aiming for a fix in the next version!
I know this issue is closed, but I wanted to shed a little more light on the severity of the problem, especially for users trying to run many scenarios in a persistent world / server.
I ran the script posted by @zhejun-zhang above (with 10 iterations rather than 100) and wrapped it in a call to memory-profiler to better quantify what's happening.  The results using 0.9.6 (top plot) versus 0.9.8 (bottom plot) are shown below.

For additional context, we're developing a multiagent RLLIB gym environment for carla 0.9.8, but have been seeing very large objects appearing in RLLIB's object store that it is unable to free.  Seems that this is a likely suspect!

Hi,
Thanks for the amazing visualization! Yes this is an issue, but we've recently released T.M. 2.0 which is merged in master and included in <IP_ADDRESS>. By any chance, would you mind checking if this problem you're having is still happening?

Yes, will try asap

This is using <IP_ADDRESS> ... it looks a tiny bit better in terms of the final memory footprint, but clearly the problem persists?

Has this issue been solved?

I think the GPU memory issue still persists in <IP_ADDRESS>. Running a batch of routes and GPU usage slowly increases over time. I can get through about 20 routes before i get the "CUDA is out of memory" error... Any suggestions to minimize the impact?

I think the GPU memory issue still persists in <IP_ADDRESS>. Running a batch of routes and GPU usage slowly increases over time. I can get through about 20 routes before i get the "CUDA is out of memory" error... Any suggestions to minimize the impact?

I'm having this issue training a DQN agent on different scenarios in CARLA 0.9.12? Any suggestions?


Mind map fails at rendering mathematical elements

@lynchjames I noticed that the mind map rendering section doesn't render mathematical elements written in LaTeX.
Although this is not my area of expertise, is there anything I can do to help make this possible, if you have an idea how to go about this?

Turns out this Obsidian plugin which builds on top of the Obsidian Mind Map plugin solves this issues.
Just follow the installation instructions on the repo's README, potentially do what is recommended here if you fail to load the plugin in Obsidian, and add the following code on top of your note to be able to click on the 3 vertical dots and render the note as a mind map.
---
mindmap-plugin: basic
---

I tried this and couldn't export images :(
MindMap creates the katex html correctly, but it does not render because katex is not imported in Obsidian.
A quick fix is import katex in obsidian by downloading this plugin: https://github.com/ryanpcmcquen/obsidian-javascript-init and adding this code in the plugin:

const node1 = document.createElement('link')
node1.setAttribute("rel", "stylesheet")
node1.setAttribute("href"<EMAIL_ADDRESS>node1.setAttribute("integrity", "sha384-A3N+UgNMKg9+LRsC2HUE0ECxFY7qhztVFORxHQZEPm7lnog2poqmm7CQ91wSEnBc")
node1.setAttribute("crossorigin", "anonymous")
var address = document.getElementsByTagName('head')[0]
address.appendChild(node1)

const node2 = document.createElement('script')
node2.setAttribute("defer", "")
node2.setAttribute("src"<EMAIL_ADDRESS>node2.setAttribute("integrity", "sha384-D0gipC03W+lifpletIVjX+11DUvrzNAMhVux0JrrAMt9Ufgu9HcPdS5GxZSDmNxx")
node2.setAttribute("crossorigin", "anonymous")
var address = document.getElementsByTagName('head')[0]
address.appendChild(node2)

There should be a more elegant way to add this to obsidian, but I don't know how. My "solution" is probably a hint towards the real problem, but I couldn't find it

I have the same problem.

@lynchjames I noticed that the mind map rendering section doesn't render mathematical elements written in LaTeX.
Although this is not my area of expertise, is there anything I can do to help make this possible, if you have an idea how to go about this?


MB-5238: Pushing build failures to #prac-engineering vs. #on-call

Description
A minor revision to change the CI Build failure notifications to #prac-engineering vs. #on-call. Once this is merged, I will update the slack URL in the circleci project config.
Code Review Verification Steps

[x] Request review from a member of a different team.
[x] Have the Jira acceptance criteria been met for this change?

References

Jira story for this change

Messages

:book:
:link: MB-5238

  Generated by :no_entry_sign: dangerJS against 3f525b6fe8ef097b092327153cdbbfd9963eb5ac


Support for Analog SFCs

The meta framework for angular analog.js now supports a new experimental file format called Analog SFCs. I am not sure if supporting this format is within the scope of this project but if yes, it would be awesome seeing support for it added.

No this is something that a tree-sitter-analog parser should handle . ;)


No license

I noticed this project doesn't have a license associated with it.
By the Free Software Foundation's definition, software without a license is not free. It looks like the intent is to have this be a free software project, based on the fact that you have been accepting code from a few contributors.
I suggest you add a suitable license to this repository, to clarify that the code is free software. Choosealicense.com is a good resource for selecting a suitable license.
I personally recommend the GNU GPL v3.

Thank you very much for this hint. I thought about the same thing in the morining but was to scared to look into the licence topic.

MIT License is less limiting regarding dependencies, I don't consent to my code being GPLv3 licensed. Suggest MIT license or at least different licenses per script author

Throwing my hat in.  I agree this should be under the MIT license but won't that conflict with what the JS license is?  I know an MIT license can be in a GPL software but not vice versa.

You have a point and you won't hear any disagreements from here.  Side note:  I can't think of a use case for a GPU on these.  Perhaps a compute server?

So should we licence it to MIT?

@HermannBjorgvin please close this issue if you agree to the current license

Yes I think this is good. Issue resolved from my viewpoint at least


Ensure filterAdapters functions correclty in new ember-cli's

In older versions of ember-cli the tree passed to treeForAddon included
the string modules/ in more recent (I believe 2.13 or 2.14) this
string is not present.
This change allows for it to match both the old and new style.
Noticed this with broccoli-concat-analyzer:

Thanks!

Actually I think this would not be needed, if #164 would have been merged! This modules namespace is as far as I understand it really something internal (private) which we should not have to care about. This is just visible because _super is called before our own tree manipulation, which is reversed and thus fixed by #164...


.Renv / requirements.txt parity

In both vetiver-r and vetiver-python, the metadata of a vetiver model holds a minimal set of dependencies needed to make a prediction.
For R, the .Renv generates a lockfile of all dependencies, and then only the packages needed to make a prediction are saved in the metadata.
In Python, a requirements.txt must be generated. The big options here might be pip freeze, pip-compile, or generating this file manually from a pre-set list of requirements. The benefits of the pip options is that there would be a defined version number, to save users headache in case different metadata is used in new vetiver versions. This seems preferable to a list with unspecified version numbers. There might need to be some parsing to either pip freeze or pip-compile to specify only the packages needed for a prediction, rather than all packages a user has installed.
It seems that the "best parity" option might to be to generate a large file via pip-compile, then grep (or some other parsing) to only save the requirements specified from a set list (ie, vetiver, pins, and torch/sklearn/etc). (WIP solution)
Is there any other useful context I am missing?

I would just clarify that the order is the other way in the R side: first, the packages needed to make a prediction are saved in the vetiver_model() metadata, and then we use renv to create a scoped-down renv.lock that only tracks the packages + versions needed for prediction.

I would just clarify that the order is the other way in the R side: first, the packages needed to make a prediction are saved in the vetiver_model() metadata, and then we use renv to create a scoped-down renv.lock that only tracks the packages + versions needed for prediction.

Thank you for the clarification! I updated the text above to match your comment and avoid further confusion :)

Looking into pip-compile some more since it seems to make the most sense for this use case. We could create a temporary file from the VetiverModel metadata that contains the package names with no versions, then pip-compile tempfile --output-file=requirements.txt to generate a requirements.txt that is versioned.
I tried this out, and the generated output looks something like below. You get an ugly tempfile name (fixable, but also might want to keep it in this format so users don't try to look for a file that has been deleted?), but I think the rest of the functionality is similar to what we want. Other features include support for installing from GitHub and support for hash-checking mode, if wanted.
`#
This file is autogenerated by pip-compile with python 3.8
To update, run:

pip-compile --output-file=requirements.txt '<tempfile._TemporaryFileWrapper object at 0x107c00310>'

joblib==1.1.0
# via scikit-learn
numpy==1.22.3
# via
#   pandas
#   scikit-learn
pandas==1.4.2
# via -r <tempfile._TemporaryFileWrapper object at 0x107c00310>
...` truncated for brevity.
CC: @machow

I wonder if having concrete examples could help here? What do you think about us setting up a repo with these things?:

a folder (vetiver-r-<something>) example script + output of R vetiver's dockerfile or plumber.R generated files
a folder (vetiver-py-<something>) example script + output of py version of above (based on current draft etc..)
a folder (pins-board) with the data they are using

This way, a person can clone that repo and do something like below, and commit the results...
python
from pins import board_folder

board = board_folder("pins-board")
board.pin_read(...)
board.pin_write(...)

R
library(pins)

board <- board_folder("pins-board")

(Alternatively, could see having each repo put these somewhere?)

So something similar for the generated Dockerfile / related files might help here?

Agreed, I do think examples will help clear things up. In #8 I added a generated Dockerfile, requirements.txt, and app.py, the parallel file to plumber.R. I've seen other python packages have an examples/ directory, so thought this might be a good way to keep these files easily discoverable.
The main difference in the Python/R experience here is the existence of the requirements.txt file. It could technically be abstracted away for Docker, but it's needed for RSConnect, so it's probably useful to have the opinionated workflow the same.

For the R side, as of right now, you can look at this folder to see generated Dockerfile, renv.lock, and plumber.R files. In the R package, there are examples of these in tests as well.


Question: Post2 explanation

Hello
After some struggles to install Shapely in linux, it turns out I had to install the Post2 version. However I can't find any explanation about what the differences are between these versions.

Differences between versions are explained in https://github.com/Toblerity/Shapely/blob/master/CHANGES.txt.


New release

Please make new release, to update composer package, not using "minimum-stability": "dev".
There is a lot of good stuff since last release.

I released v4.0.


The principle of 'position' returning from function 'onCameraTransformUpdate' in

I'm trying to use Viro to implement simple indoor navigation. And I'm gonna use the 'position' returning from 'onCameraTransformUpdate' to represent the real position of the user and the destination. However, I want to know the principle of this property. I have checked the ViroARScene.js file but it does not tell me properly. I guess it might use 'slam' or sth else.

Hi, I am trying to use this function to get the return value too. But I do not know how to get the return value. So I think maybe you can help me. If you can show the code or something, about its usage, that will help me a lot!

I'm trying to use Viro to implement simple indoor navigation. And I'm gonna use the 'position' returning from 'onCameraTransformUpdate' to represent the real position of the user and the destination. However, I want to know the principle of this property. I have checked the ViroARScene.js file but it does not tell me properly. I guess it might use 'slam' or sth else.

Hi, I am trying to use this function to get the return value too. But I do not know how to get the return value. So I think maybe you can help me. If you can show the code or something, about its usage, that will help me a lot!

Hi, I am trying to use this function to get the return value too. But I do not know how to get the return value. So I think maybe you can help me. If you can show the code or something, about its usage, that will help me a lot!

Hi @Eddie-hubb , did you figure this out?


30 seconds advertisement breaks

About a month ago I noticed that my stream (playing in mpv) would randomly pause for no reason. At first I thought it's due to me not having upgraded to 7.0 (which was new at that poin) but the issue persisted after upgrading (Streamlink, the Twitch GUI as well as this plugin). The output says:
Detected advertisement break of 30 seconds
Filtering out segments and pausing stream output

And after those thirty seconds it continues where it left off.
My custom param for streamlink looks as follows:
--twitch-proxy-playlist=https://lb-eu2.cdn-perfprod.com,https://as.luminous.dev,https://eu.luminous.dev,https://eu2.luminous.dev,https://lb-eu.cdn-perfprod.com,https://eu1.jupter.ga/channel/[channel]
Is there nothing that can be done at this point or am I doing something wrong? Random channels I noticed these pauses in: quin69, shoklol and probably others but I don't know 100%.
The full log:

Plugin twitch is being overridden by C:\Users\XXXXXXXXX\AppData\Roaming\streamlink\plugins\twitch.py (sha256:59bc4b364c46eb80659df23f4706393746ea1243bf0fabb5e2760cc93baf78df)
streamlink-ttvlol 7.0.0-20241105 (7.0.0)
Please report issues to https://github.com/2bc4/streamlink-ttvlol/issues
Found matching plugin twitch for URL twitch.tv/shoklol
Using playlist proxy 'https://lb-eu2.cdn-perfprod.com'
Available streams: audio_only, 160p (worst), 360p, 480p, 720p, 720p60, 1080p60 (best)
Opening stream: 1080p60 (hls)
Starting player: d:\XXXXXXXXX\mpv-x86_64-v3-20221106-git-2590651\mpv.exe
Will skip ad segments
Low latency streaming (HLS live edge: 2)
Detected advertisement break of 30 seconds
Filtering out segments and pausing stream output
Encountered a stream discontinuity. This is unsupported and will result in incoherent output data.
Resuming stream output
Detected advertisement break of 30 seconds
Filtering out segments and pausing stream output
Encountered a stream discontinuity. This is unsupported and will result in incoherent output data.
Resuming stream output
Detected advertisement break of 30 seconds
Filtering out segments and pausing stream output
Encountered a stream discontinuity. This is unsupported and will result in incoherent output data.
Resuming stream output

Versions:
streamlink 7.0.0
streamlink-twitch-gui  v2.5.3

All you can do is try different servers. The pausing is caused by --twitch-disable-ads which just doesn't write ad segments to the player.

I see :/ I don't mind that the stream freezes due to ads, I just wish I wouldn't have any. But I guess that's just something that is dependent on the various servers :/ I think I've tried a few different ones, but I had this issue in all of them. I wonder if they don't work properly anymore. Are there any other servers I could give a shot?


feat: Time Filter shortcuts

SUMMARY
The Time Filter currently supports DSL for defining arbitrary time expressions, such as date_trunc, date_add, holidays, and more. However, users without a technical background often find it challenging to use. This pull request proposes the introduction of a new feature in the Time Filter called 'Shortcut.'
In this feature, we can easily define custom time ranges in config.py, and the arbitrary time expressions will dynamically appear in it.
The pre-defined shortcuts is above, of course, we can change it if you want
TIME_FILTER_SHORTCUTS = [
    ["Current week", "DATETRUNC(DATETIME('today'), WEEK) : today"],
    ["Current month(MTD)", "DATETRUNC(DATETIME('today'), MONTH) : today"],
    ["Current year(YTD)", "DATETRUNC(DATETIME('today'), YEAR) : today"],
]

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Shortcuts in Frame

Shortcuts

Time filter in Native Filter

TESTING INSTRUCTIONS

CI and manually
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[x] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[X] Introduces new feature or API
[ ] Removes existing feature or API

Small nitpick: Before the "(YTD)" and "(MTD)" part, there should be a space.

Good catch! will update codes and screenshots.

@zhaoyongjie weren't there some Cartel designs for more human-esque time ranges? Additionally I'm not entirely sure about the design/UX, i.e., it's not evident what represents a "shortcut" as opposed to any of the existing filters—many of which are shortcuts by themselves.

Thank you for the PR @zhaoyongjie! Given that the shortcuts don't need extra input from the user, do you think we could display them as normal select options instead of a separate section?
Using the screenshot you provided as example, we could have:
Current week
Current month (MTD)
Current year (YTD)
Last
Previous
...

If some of these become really popular, we can even make them official in the future and nothing would change for the users.

Thank you for the PR @zhaoyongjie! Given that the shortcuts don't need extra input from the user, do you think we could display them as normal select options instead of a separate section?
Using the screenshot you provided as example, we could have:
Current week
Current month (MTD)
Current year (YTD)
Last
Previous
...

If some of these become really popular, we can even make them official in the future and nothing would change for the users.

If so, should we extract all values in Last and Previous?
Current week
Current month (MTD)
Current year (YTD)
Last day
Last week
Last month
...
Previous week
...

@zhaoyongjie weren't there some Cartel designs for more human-esque time ranges? Additionally I'm not entirely sure about the design/UX, i.e., it's not evident what represents a "shortcut" as opposed to any of the existing filters—many of which are shortcuts by themselves.

The goal of this PR is providing a flexible approach that should define any time range from user config. it's not only for "Current" or "Cartel design".

If so, should we extract all values in Last and Previous?

I think this would be a good idea. If we can reduce the number of steps users need to find their desired option, it would be great. Especially because the Select allows searching so they could type "Last" and see official e shortcut options all at once. I would only present the extra panel when more input is needed from the users.
@kasiazjc @rusackas might be interested in the discussion too.
By the way, I think simplifying the user experience is really valuable so I like the intent of this PR @zhaoyongjie.

If so, should we extract all values in Last and Previous?

I think this would be a good idea. If we can reduce the number of steps users need to find their desired option, it would be great. Especially because the Select allows searching so they could type "Last" and see official e shortcut options all at once. I would only present the extra panel when more input is needed from the users.
@kasiazjc @rusackas might be interested in the discussion too.
By the way, I think simplifying the user experience is really valuable so I like the intent of this PR @zhaoyongjie.

I totally agree with you. The "Last", "Previous", "Current" and more customized item might be provide by config.py if user want to change order or to remove some item, it should be easy.

My two cents here: If we are considering a redesign, then we should make the smallest possible UI change here to support the use case for the moment.
I fear that pulling out all the day/month/etc ranges into the Select menu is... not ideal. I would just copy the pattern for "Last" and make a new option for "Current" with similar radio button options:

The customizable options via config.py are still interesting/useful, but this seems like a common use case that we ought to support as a mainstream feature. I'm also not sure what the maintenance/documentation requirements would be if people start using these custom config options frequently in installations around the world.

I have a Superset installation configured via Docker-compose. How can I activate this feature? @zhaoyongjie

The feature hasn't merged, you should cherry-pick this PR to your own codebase and build it by youself.


Read the Docs: Updating Phase Constraint

This is to get read the docs working again and to provide the most up to date example of the phase constraint documentation as an example for team members as we push documentation forward

This branch on RtD is now building successfully - it can be viewed here: https://exoctk.readthedocs.io/en/rtd_phase_constraint/

@nespinoza - This is ready now! You can review the RtDs version here: https://exoctk.readthedocs.io/en/rtd_phase_constraint/tutorials/exoctk.PhaseConstraintCalculator.html#id1

@nespinoza - Hey Nestor! Just a reminder that this is ready for review!

Oops, sorry for not coming back to this. I already did give it a read and all looks nice! Thanks for this. Approving this then!


Required .azurefunctions directory is not created when publishing in an Azure DevOps pipeline on the ubuntu-latest agent

Executing dotnet publish for a solution with an Azure Functions worker application targeting .NET 5 should generate an .azurefunctions directory in the output directory. This directory contains several assemblies, for example Microsoft.Azure.WebJobs.Extensions.FunctionMetadataLoader.dll.
Inside the publish directory there is a file named extensions.json which lists extensions' name, type, and hint path. The hint path points to the assemblies in the .azurefunctions directory, for example:
{
    "name": "Startup",
    "typeName": "Microsoft.Azure.WebJobs.Extensions.FunctionMetadataLoader.Startup, Microsoft.Azure.WebJobs.Extensions.FunctionMetadataLoader, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=551316b6919f366c",
    "hintPath": "./.azurefunctions/Microsoft.Azure.WebJobs.Extensions.FunctionMetadataLoader.dll"
}

Running dotnet publish in an Azure DevOps pipeline using the ubuntu-latest agent produces an artifact that does not contain the .azurefunctions directory. After deploying the artifact using az functionapp deployment source config-zip the application does not work (routes to HTTP-triggered functions return 404). Troubleshooting in the Azure Portal using the "Availability and Performance" diagnostics category helped me find this runtime error related to the extension mentioned above:

Unable to load startup extension 'Startup' (Type: 'Microsoft.Azure.WebJobs.Extensions.FunctionMetadataLoader.Startup, Microsoft.Azure.WebJobs.Extensions.FunctionMetadataLoader, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=551316b6919f366c'). The type does not exist. Please validate the type and assembly names.

Switching to the windows-latest agent solves the problem (see also this comment on #427).
What is missing on the Microsoft-hosted ubuntu-latest agent that causes this problem?
(Link to a list of software available on the agent)

I noticed that the Windows agent contains these 2 Visual Studio components:

Component.Microsoft.VisualStudio.Web.AzureFunctions
Microsoft.VisualStudio.ComponentGroup.AzureFunctions

Is an installation of Visual Studio required to build Azure Functions worker applications?

We shouldn't have a dependency on either of those components.
Flagging this for investigation, but would it be possible to share the following information to make sure we have a valid repro?

The csproj file
If you have the ability to share a build log file, we can provide instructions on how to share that privately
Any output from your build process in DevOps (ideally, with higher verbosity levels)

Thanks!

The builds that produced an invalid artifact were not retained unfortunately. I tried to create a small project to reproduce this, but I cannot reproduce it anymore.


A Time should be typed as XMLSchema#dateTime

Ruby does not have a class which represents a time without date. A Tod::TimeOfDay provided by the tod gem would justify XMLSchema#time, but the default Time class always includes a Date.
The mismatch between the Ruby Time and the XMLSchema#time causes information loss. For example, ActiveRecord converts a postgres datetime column to a ActiveSupport::TimeWithZone. When serializing that attribute, we lose the date part of the value.
#399 fixes this issue by using XMLSchema#dateTime. However, the current implementation is around for years, so this might break applications.
Of course, I can hack my way into a solution by prepending the Literal#new method, but I believe this PR is a logical solution since it is just a better mapping.
We could add a configuration option, so developers may choose their own implementation.

Closed via #399.


Peers and sync design

Peers
Summary
Everyone will have a list of live peers in the network. This list will never be fully in sync as peers come and go all the time. Peers will advertise themselves to network to get on the list.
Properties

speed - estimated speed metric
size - amount of data downloaded from peer, used for updating speed
trust - metric of up-to-dateness and measure of the success and error rate
last_success_time - updated to current time when

we make a successful request to peer
we learn of the peer from syncing
the peer advertises itself to us

last_fail_time - updated when we fail to make a request to the peer.

Lifetime flags

alive - there was a recent connection, i.e. last_success_time > last_fail_time
dead - last_fail_time > last_success_time. This peer is not used except when telling other peers that it's dead.
known - There was no recent update. This peer is not used but we keep the info around in case it comes back online.
delete - peer should be deleted if we don't receive any updates about it for a long time.

Definitions
FastPeer: a peer with enough trust and speed to be in the top N peers. Should sync fast.
Peer: we use random peers to be sure we get the longest chain.
RandomFastPeer: chosen randomly from the list of FastPeers that is sorted by speed.
Notes
When notified of new peers we should randomly test some.
Light nodes will only ask for 100-1000 rnadom live peers, it doesn't need to know them all.
We want to hit the speetspot between fully trusting the list of peers we receive and checking every single peer. The former may get us spammed with random IP addresses. The latter would create a lot of traffic and DDoS a peer that just connected to the network.
Sync
Summary
When starting the node, sync will start off by one-time execution of functions to get us up-to-date with blockchain state as fast as possible. First Peers, then Headers, then Blocks. In the meen time it starts routines to keep the data up-to-date. By using one-time functions specially crafted for the task we are able to tell when the node is ready.
Phases
Syntax
[f] = a blocking function
[+] = start of an ongoing task
-> = result of our action (what we have after executing the step)
Sequence

[f] fast peers - Asks random FastPeers for some RandomFastPeers -> RandomFastPeers
[f] fast headers - Asks RandomFastPeers about headers. When we don't get much new stuff we continue -> low trust (fairly up-to-date) headers
[f] fast blocks (full only) - Asks RandomFastPeers about blocks. When we have all we continue -> blocks for the low trust headers
[f] random peers - ask random Peers about random Peers -> random Peers
[+] sync peers - routine to synchronize peers. In light mode downloads more only if we don't have enough FastPeers or Peers. Downloads all peers in full mode.
[f] good headers - ask random Peers about top block and get headers -> trustworthy (most probably the longest chain of) headers
[+] sync headers (also blocks in full mode) - routine to sync headers. Asks random Peer about top headers once in while. Downloads blocks from FastPeers.
[f] ready lite - set status to ready in light mode.

Everything below is for full mode only.

[f] good blocks - ask RandomFastPeers about blocks until headers and blocks are in sync -> blocks for trustworthy headers
[+] external api - we turn on external api now. External api should be on only for peers that are fully in sync. Being out of sync and having your external api on will result in low trust.
[+] advertiser - Once in a while we advertise ourselves to random peers. We decide if we should advertise more if nodes say they don't know us.
[f] ready full - set status to ready in full mode.

Notes and issues
What if we get some peers with very high speed to mess with RandomGoodPeer??? Probably we shouldn't sync speed or do it to a limited extent.
We don't try to make sure we have all the peers, we try to have a list that's good enough. Peers should make sure they advertise themselves to be on the list (and get new blocks fast).
We use trust only for the initial download. Asking random Peers for the top block is used to build trust that's used later when we do the initial sync. It's usefull to have as good as possible initial sync so asking random Peers will result in the least new data possible.
How to store peers and how to synchronize them between two nodes? It would be good if we could work around sending whole lists.
The mentioned algorithm of spreading info about new peer in network may not be the best. There should be some existing very good solutions. (probably one's used by torrents networks?)

How is trust calculated?
How does a peer advertise itself to us?
When do we update the dead or alive flags?
What is enough the definition of GoodPeer? What is N?
How does using Peers ensure the longest chain?
In the sync sequence...

Where do we get the initial list of FastPeers to query? How many peers should we ask for?

When exactly do we continue? Why are headers low-trust and why fairly up to date? What headers do we ask for? Are these headers for the top block? How are they different from step 6 then?

Where do we get the initial list of peers?

How many peers is enough in light mode?

In the list of notes and issues...
Please clarify We use trust only for the initial download. Asking random Peers for the top block is used to build trust that's used later when we do the initial sync. It's usefull to have as good as possible initial sync so asking random Peers will result in the least new data possible.

the notion of dead or alive - is a peer forever in the dead? Is it open list or capped on some size? Probably we don't want to maintain a list of dead peers

Can you detail the force sync flow, for the freshly mined block?

When setting up some default numbers of random peers it should be relative, not hardcoded to (100-1000)

Re. point 10: you want to enable/disable API based on "state"? Lets approach it in a bit different way (or express it in different way). Node can start listening on API when its operational and it is operational after the first sync. Turning APIs on/off during regular operation is never a good idea

The API to exchange information has to be granular and extendable. You mention the exchanges of blocks and headers. Also advertising peers. Design the api in a way that it takes equivalents of maps. If we want to pull more data to sync it will be easy to add (like more transactions that are not mined)

[f] fast peers - Asks random FastPeers for some RandomFastPeers -> RandomFastPeers

I understand that when the node starts for the first time it will have one (or a bunch of) initial peers, so it shall read "Asks random FastPeers - or initial Peer specified by configuration if none known - ..."

If it "Asks ... for some RandomFastPeers", does it mean that the network call returns not only the network connection info but also the statistics e.g. trust and speed/size?

[f] fast headers - Asks RandomFastPeers about headers. When we don't get much new stuff we continue -> low trust (fairly up-to-date) headers

Is block header validation in this step?

Assuming block header validation is in this step, considering that node aims at identifying "the" longest chain, how does node acquires the headers ("-> low trust (fairly up-to-date) headers")? Does it first ask for top header hash+height from a certain number of peers, then picks any one chain at greatest height and validates it sequentially, resulting in one chain of block headers that are valid?

Assuming block header validation is in this step, considering that validation of block header depends on previous block headers for (1) validation of difficulty, (2) presence of block header identified by previous hash, (3) block header leading to genesis, does node parallelize computations?

There is always a list of known peers. We can host them and host the list on our site (like an static API)

[f] fast blocks (full only) - Asks RandomFastPeers about blocks. When we have all we continue -> blocks for the low trust headers

This step is listed as being for both full and light mode of operation. Does it mean even in light mode all blocks shall be downloaded?

Shall we create a Google Doc and paste the link here?

To everyone:

How is trust calculated?

Formula for trust has to be defined. The idea is the trust represents whether according to previous responses the node tends to return the top header for the longest chain when asked. Aside from that the trust is lowered if node gives us any untrue or malformed information.
Aside from determining the nodes we should use for sync (only high trust nodes when caching up, all nodes except those with low trust in normal operation), trust can also be used to determine nodes that are trying to attack network and to drop connections from them.

How does a peer advertise itself to us?

My idea is the information about the existence of a new peer in network will be spread in limited extend in peers sync. Node will be able to use a special "advertise_peer" request to inform other of it's existence as fully operational full node. (I suggest peer sync will be pull only)

When do we update the dead or alive flags?

the notion of dead or alive - is a peer forever in the dead? Is it open list or capped on some size? Probably we don't want to maintain a list of dead peers

I don't think, I ever used the word "flag". (proposal text was edited by someone). I don't plan to update them in any special ways. The better, but still not good anough word for those is "state". The diffrent states are just some conditions depending of which we treat the peer diffrently.
"alive" is the standard state of peer. Only "alive" peer is used for synchronization.
"dead" peer is a peer that we failed to connect to recently. We informed others about that. (this is a possible suspect to change as the peers sync systems I haven't decided upon. I'm open to some thought-out proposals)
"known" that we failed to connect so long time ago we stop to inform other we failed to connect to it, but we still keep it in memorry with all of it's statistics (speed and trustworthyness) in case it comes back online. We will be informed about that by others or by the peer itself.
"delete" state is the condition for removal from our storage all information about the peer as we think it's not going to come back online.

What is enough the definition of GoodPeer? What is N?

N has to be decided. N may be a percentage. The idea is a GoodPeer is a peer that has a fast connection and according to our experience it tends to give good (up-to-date) correct chain.
GoodPeer is used for initial sync so it's fast.

How does using Peers ensure the longest chain?

To ensure we get the longest chain we ask random known to us Peers about the block. As oposed to asking only those with good connection. Generally my idea is we use GoodPeers to achieve
fairly good sync and then we asked some random peers if they have anythning better (a longer chain). This way we get a faster sync by asking peers with good connection, but we don't lose security by relaying on peers with good connection.

Where do we get the initial list of FastPeers to query?

Where do we get the initial list of peers?

That's the point I realized someone changed GoodPeer to FastPeer in my text and I got confused. To everyone FastPeer=GoodPeer=definition in proposal Like in every (I think) P2P system initial list of peers has to be hardcoded or downloaded from somewhere. After first execution of client system will self regulate and generate new lists based on responses from network.

How many (Fast)peers should we ask for?

How many peers is enough in light mode?

The number of peers to fetch in step 1 has to be decided on. It has to be big enought. Probably something like 100 is good. The same applies for step 4 and 5.

When exactly do we continue? Why are headers low-trust and why fairly up to date? What headers do we ask for? Are these headers for the top block? How are they different from step 6 then?

By "continue" I mean: proceed to next step. In step 2 we fetch from GoodPeers/FastPeers all the headers we don't have that are on the longest chain. I call those "low-trust" or "fairly up-to-date" becouse we use peers with good network connection to get them. Peers with good network connection I don't consider random and consider them a possible source of manimulation. Some big company/wealthy person may get many very-very fast nodes and try to influence what data do other peers get, most probably by providing some short fork. Becouse of that we can only establish a trustworthy information about longest chain in slower (if used for a lot of headers) step 6.

Please clarify:

We use trust only for the initial download. Asking random Peers for the top block is used to build trust that's used later when we do the initial sync. It's usefull to have as good as possible initial sync so asking random Peers will result in the least new data possible.

Sorry, that sentence is a little bit chaotic. When we ask other peers about top block we store that information and when we establish information what's the true top block we make trustworthyness of peers that answer correctly higher and those that where significantly off lower. High trustworthyness peers are only used to get as good as possible initial sync in future. Low trustworthyness peers may be ignored as they provide false information. (Please note "may be", I'm not sure about that idea. Is seems not needed.)

Can you detail the force sync flow, for the freshly mined block?

You mean the Gossip protocol? Gossip protocol will be used for spreading freshly mined block. It's described in the docs in "Gossip" PR on aeternity/testnet. Gossip protocol will use random peers as defined in this document.

When setting up some default numbers of random peers it should be relative, not hardcoded

Hmm, I'm not sure it's important. We can assume more then some fixed percentage of network is honest. If we have 100 trully random Peers there is a (high) fixed chance we hit any of those nodes no matter the actuall size of the network. (The chance doesn't change when network grows). Could you describe why do you think a fixed number of random peers to ask (for example) for the top block isn't good enough? Is my explanation clear or maybe I should provide some numbers?

Re. point 10: you want to enable/disable API based on "state"? Lets approach it in a bit different way (or express it in different way). Node can start listening on API when its operational and it is operational after the first sync. Turning APIs on/off during regular operation is never a good idea

I think we are thinking the same.  Lite-node doesn't expose external API.
I consider the node synchronized with network and operational only after step 9.

The API to exchange information has to be granular and extendable. You mention the exchanges of blocks and headers. Also advertising peers. Design the api in a way that it takes equivalents of maps. If we want to pull more data to sync it will be easy to add (like more transactions that are not mined)

I think I'm not familiar with some concept that you are reffering to. (Maybe I don't know something) Please describe your concern.

I understand that when the node starts for the first time it will have one (or a bunch of) initial peers, so it shall read "Asks random FastPeers - or initial Peer specified by configuration if none known - ..."

You understand correctly. I consier the first sync a special situation. I will add some paragraph about diffrences in it.

If it "Asks ... for some RandomFastPeers", does it mean that the network call returns not only the network connection info but also the statistics e.g. trust and speed/size?

Yes, but probably only speed. Also as pointed out in the "notes and issues" this may be something that can be exploited and we shouldn't trust the returned information too much.

Assuming block header validation is in this (2) step, considering that node aims at identifying "the" longest chain, how does node acquires the headers ("-> low trust (fairly up-to-date) headers")? Does it first ask for top header hash+height from a certain number of peers, then picks any one chain at greatest height and validates it sequentially, resulting in one chain of block headers that are valid?

Yes, you are correct. One chain is good enough, becouse it's very-very-very expensive to create a long chain, becouse of the required proof of work, so the aquired chain shouldn't be very different from the truelly longest one.

Assuming block header validation is in this (2) step, considering that validation of block header depends on previous block headers for (1) validation of difficulty, (2) presence of block header identified by previous hash, (3) block header leading to genesis, does node parallelize computations?

Due to the nature of our blockchain node may have limmited ability to parallelize verification of headers. I didn't investigate possibilities of parallelizing verification of headers. It should be investigated, but I don't think that's important for mainnet readyness.

[f] fast blocks (full only) - Asks RandomFastPeers about blocks. When we have all we continue -> blocks for the low trust headers

This step is listed as being for both full and light mode of operation. Does it mean even in light mode all blocks shall be downloaded?

You cited a step with "(full only)" in it's description. By that I mean it will only by executed an a full-node / in full mode.

I think I answered all of the questions. If something is unclear I will provide further explanation of my ideas. I'm open to changes.

@cytadela8 Thanks for the time you put into writing this and clarifying your intent.
I'm going to close this ticket as too complex. All we need for the foreseeable future is

A list of peers where each peer is stamped with last successful network operation (last).
A scavenger process to remove peers that we haven't successfully interacted with for more than time T.

Keeping track of peer speed, for example, is an optimization that we can make at a later time.  Same for other features of this proposal. Please remember to keep it simple!

@wagerlabs I agree parts of this design aren't required for MVP, but whole thing as a principle I would keep. I think it's important to keep in mind the bigger picture when coding MVP. I consider the difference between this design and one for MVP to be:

don't implement all "fast..." steps
don't implement other things concerned with speed
don't implement trustworthyness or have minimal implementation
simpler peer lifetime as you described

Let's implement the 1 and 2 from my comments. That's all we need for now and there are other tasks to work on.


Collect consent of European users GDPR - User Messaging Platform (UMP) SDK

Describe your Issue

Important: Google is planning to launch its full support of the IAB TCFv2 for GDPR on August 15, 2020. We are releasing the Funding Choices IAB solution early to ensure you have adequate time to set up and test your consent messages on your apps prior to this launch. Prior to August 15, publishers should not publish consent messages using the IAB framework to apps with live traffic.

The Xamarin.Forms team plans to implement the User Messaging Platform SDK so that developers can comply with the new European regulation? I think this should be a priority issue on the roadmap.
Relevant information

User Messaging Platform (UMP) (Android documentation)
User Messaging Platform (UMP) (IOS documentation)

Thanks for feedback, but I need more info and first of all this is not Xamarin.Forms repo.

Hi, I leave you information about it:
EU user consent policy
The European Union in the coming days will require developers to add a banner to applications to control the following:

For Google products used on any site, app or other property that is under your control, or that of your affiliate or your client, the following duties apply for end users in the European Economic Area along with the UK.
You must obtain end users’ legally valid consent to:
the use of cookies or other local storage where legally required; and
the collection, sharing, and use of personal data for personalization of ads.

When seeking consent you must:
retain records of consent given by end users; and
provide end users with clear instructions for revocation of consent.

You must clearly identify each party that may collect, receive, or use end users’ personal data as a consequence of your use of a Google product. You must also provide end users with prominent and easily accessible information about that party’s use of end users’ personal data.

To control these cases Google has released the UMP SDK which would be great if Xamarin.Forms implemented it as a component to be able to use it.
Example:

Second question:

The Xamarin.Forms team plans to implement the User Messaging Platform SDK so that developers can comply with the new European regulation?

Is this a statement or question?

It is a question since I think it would be interesting to implement this.

UMP is completely new artifact groupId namely com.google.android.ump from:
implementation 'com.google.android.ump:user-messaging-platform:1.0.0'

And this repo binds only Android artifacts from following groups:
*. com.google.android.gms
*. com.google.firebase

Ok, I wasn't quite sure where I should post this request, thanks for transferring the thread to XamarinComponents, you're right UMP being a new artifact makes more sense than this in this repository.
I think that with the links that I have left to the UMP SDK documentation and the EU user consent policy there is a lot of information about it, if you need more information, please leave a comment and I will try to provide as much as possible . Thank you.

https://www.nuget.org/packages/Xamarin.Google.UserlMessagingPlatform/1.0.0-preview01
https://github.com/xamarin/XamarinComponents/pull/935

Published
https://www.nuget.org/packages/Xamarin.Google.UserMessagingPlatform/
Note:
There was typo in initial bindings (link above)
https://www.nuget.org/packages/Xamarin.Google.UserlMessagingPlatform/1.0.0-preview01
https://www.nuget.org/packages/Xamarin.Google.UserlMessagingPlatform/

Is there any example of how to use it? The one from Google seems in its barebones and that something is missing.

Is there any example of how to use it? The one from Google seems in its barebones and that something is missing.

Here is a repository that shows how to use this library: https://github.com/GalaxiaGuy/GoogleUserMessagingPlatform

I created a full tutorial also:
https://supernovaic.blogspot.com/2022/02/guide-for-using-xamaringoogleusermessag.html

I was able to implement AdMob GDPR Message on Xamarin Forms apps using Xamarin.Google.UserMessagingPlatform NuGet. But I have problems with porting the same to iOS. I was able to add Xamarin.Google.iOS.UserMessagingPlatform NuGet, but then I can't figure out what Namespaces should I use in C# code. If I use the same namespace as for Android (Xamarin.Google.UserMesssagingPlatform), then I see all classes available but missing references to Mono.Android, which should not be required for iOS Project obviously. My guess that I should use a different namespace, but I really can't locate it anywhere. What namespace/classes is declared in Xamarin.Google.iOS.UserMessagingPlatform NuGet, there's no debug information anywhere, or instructions/examples. Could you help?

I created a full tutorial also:
https://supernovaic.blogspot.com/2022/02/guide-for-using-xamaringoogleusermessag.html

This is a great tutorial, but this is for Android. For iOS I can't even understand which namespace to use and where all the classes declared. I imported Xamarin.Google.iOS.UserMessagingPlatform nuget, but that's it, can't go any further.

I created a full tutorial also:
https://supernovaic.blogspot.com/2022/02/guide-for-using-xamaringoogleusermessag.html

This is a great tutorial, but this is for Android. For iOS I can't even understand which namespace to use and where all the classes declared. I imported Xamarin.Google.iOS.UserMessagingPlatform nuget, but that's it, can't go any further.

Have you created a new ticket for iOS? I think it will be easier to get their attention. The iOS version is old (10/20/2020) while the Android one is just two months old:
https://www.nuget.org/packages/Xamarin.Google.iOS.UserMessagingPlatform
Any thoughts @moljac where this ticket can be raised? It is an official Microsoft/Xamarin Package:

Bing Chat provided me this answer:
Thank you for clarifying your question. UserMessagingPlatform is a library that helps you display GDPR consent messages to your app users in the European Economic Area (EEA) or the UK¹. It is part of the Google Mobile Ads SDK and works with AdMob and other Google advertising products².
To use UserMessagingPlatform in your iOS app, you need to follow these steps:

Install the Xamarin.Google.iOS.UserMessagingPlatform NuGet package in your app project³.
Add the User Messaging Platform framework to your Xcode project⁴.
Initialize the Google Mobile Ads SDK in your app delegate's FinishedLaunching method.
Request consent information from the User Messaging Platform API using the ConsentInformation class.
Check if the user is located in the EEA or the UK using the IsRequestLocationInEeaOrUnknown property.
If yes, load and display a consent message using the ConsentForm class.
Handle the user's consent choice using the ConsentStatus enum.

Here is an example of how to use UserMessagingPlatform in your iOS app:
using Google.MobileAds;
using Google.MobileAds.Consent;
using UIKit;

namespace MyApp
{
    public class AppDelegate : UIApplicationDelegate
    {
        public override bool FinishedLaunching(UIApplication application, NSDictionary launchOptions)
        {
            // Initialize Google Mobile Ads SDK
            MobileAds.SharedInstance.Start(null);

            // Request consent information
            ConsentInformation.SharedInstance.RequestConsentInfoUpdate(new string[] { "pub-0[1](https://cookieinformation.com/resources/blog/is-your-mobile-app-gdpr-compliant/)[2](https://www.appknox.com/resources/guides/make-your-mobile-app-compliant-to-gdpr)[3](https://support.google.com/admob/answer/10113207?hl=en)[4](https://stackoverflow.com/questions/50486708/how-to-add-gdpr-consent-dialog-in-app-and-gdpr-related-confusions)56789012345" }, (error) =>
            {
                if (error == null)
                {
                    // Check user location
                    if (ConsentInformation.SharedInstance.IsRequestLocationInEeaOrUnknown)
                    {
                        // Load consent message
                        ConsentForm consentForm = new ConsentForm();
                        consentForm.Load((loadError) =>
                        {
                            if (loadError == null)
                            {
                                // Show consent message
                                consentForm.Show((dismissError, userPrefersAdFree) =>
                                {
                                    if (dismissError == null)
                                    {
                                        // Handle user choice
                                        switch (ConsentInformation.SharedInstance.ConsentStatus)
                                        {
                                            case ConsentStatus.NonPersonalized:
                                                // Show non-personalized ads
                                                break;
                                            case ConsentStatus.Personalized:
                                                // Show personalized ads
                                                break;
                                            default:
                                                // No consent given, do not show ads
                                                break;
                                        }
                                    }
                                });
                            }
                        });
                    }
                }
            });

            return true;
        }
    }
}

I hope this helps you with your app development. If you have any other questions, please feel free to ask. 😊
Source: Conversation with Bing, 9/18/2023
(1) GDPR compliance for mobile apps: A checklist in 7 steps. https://cookieinformation.com/resources/blog/is-your-mobile-app-gdpr-compliant/.
(2) Guide: Make Your Mobile App Compliant to GDPR - Appknox. https://www.appknox.com/resources/guides/make-your-mobile-app-compliant-to-gdpr.
(3) Create GDPR messages for apps - Google AdMob Help. https://support.google.com/admob/answer/10113207?hl=en.
(4) How to add GDPR consent dialog in app and GDPR related confusions. https://stackoverflow.com/questions/50486708/how-to-add-gdpr-consent-dialog-in-app-and-gdpr-related-confusions.
But I'm not so sure if it will work as expected, you can try it, @mikeks, but most likely you will face another issue since the Official Microsoft lib is outdated.

@moljac do you know if there is any plan to implement an iOS version which supports .NET MAUI applications (I'm using .NET 7)? Whether that be an update to Xamarin.Google.UserMessagingPlatform to include support for iOS or an update to Xamarin.Google.iOS.UserMessagingPlatform. I've tried to use Xamarin.Google.iOS.UserMessagingPlatform, but it doesn't seem to contain the namespaces that are required.

Is there any update on implementing an iOS version for .NET MAUI applications?
This is very important for developers that need to use Admob on iOS using MAUI since the new RGPD requirements for user concent. Is a blocking issue since January 16 2024, and I can't find any solution for iOS yet.

Bing Chat provided me this answer of how use it, but I'm not sure it's workable. @mikeks would be a better tester than me.

The code from Bing Chat is outdated, but I was able to compile it with some changes. I didn't test it yet, I will update this when I test the code.
Corrected code:
			var requestParameters = new RequestParameters();
			requestParameters.TagForUnderAgeOfConsent = false;

#if DEBUG
			requestParameters.DebugSettings = new DebugSettings
			{
				Geography = DebugGeography.Eea
			};
#endif

			ConsentInformation.SharedInstance.RequestConsentInfoUpdate(requestParameters, (error) =>
			{
				if (error == null)
				{
					consentStatus = ConsentInformation.SharedInstance.ConsentStatus;
					// Check user location
					if (consentStatus == ConsentStatus.Unknown
					|| consentStatus == ConsentStatus.Required) // IsRequestLocationInEeaOrUnknown
					{
						// Load consent message
                                                ConsentForm.Load((consentForm, loadError) =>
                                                {
	                                                if (loadError == null)
	                                                {
		                                                // Show consent message
		                                                consentForm.Present(viewController);
	                                                }
                                                });
					}
				}
			});


Migration to API v9

Hey👋
I prepared the migration from the v8 to the v9 tracking API. Unfortunately, there are breaking changes in the package API because it was impossible to work without a workspace ID in cases where it was required before. I did a bit of refactoring to improve the package API consistency. Migration to Report API v3 is not part of this change.
I left comments about the not-so-obvious changes in the PR to give you some context.

Hey @jason0x43, all comments were addressed. There is one more change in StartTimeEntryForProject, I switched to billable *bool, which makes more sense. I only found it once I started migrating the alfred-toggl.


Assert the explicit admin in subteam create

There was no test for this before, and since things might change in a variety of places, it would be nice to get this into master before that happens.

nevermind...will fix for real.


Fix DtxTrackingRecord table name

There is a hardcode table name in DtxTrackingRecord. This pr corrects it.

LGTM.