gollvm: error: no matching function for call to llvm::LoadInst::LoadInst(llvm::Type*&, llvm::Value*&, std::string&)

What version of Go are you using (go version)?

$ go version
go version go1.12.10 linux/amd64

Does this issue reproduce with the latest release?
Yes.
I cloned the lastest version from
https://go.googlesource.com/gollvm/
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ns1server/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/ns1server/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/go-1.12"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.12/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build459238245=/tmp/go-build -gno-record-gcc-switches"

What did you do?
I fixed one of the issues with MaybeAlign(). I changed it to Align() from llvm:Align().
I am trying a ninja build for gollvm using cmake & ninja-build.

I am getting the error as shown below.
/home/ns1server/workarea/llvm-project/llvm/tools/gollvm/bridge/go-llvm.cpp:616:74: error: no matching function for call to ‘llvm::LoadInst::LoadInst(llvm::Type*&, llvm::Value*&, std::string&)’
  616 |     llvm::Instruction *loadInst = new llvm::LoadInst(vt, spaceVal, ldname);
      |                                                                          ^
In file included from /home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/IRBuilder.h:33,
                 from /home/ns1server/workarea/llvm-project/llvm/tools/gollvm/bridge/go-llvm-irbuilders.h:18,
                 from /home/ns1server/workarea/llvm-project/llvm/tools/gollvm/bridge/go-llvm-builtins.h:18,
                 from /home/ns1server/workarea/llvm-project/llvm/tools/gollvm/bridge/go-llvm.cpp:15:
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:198:3: note: candidate: ‘llvm::LoadInst::LoadInst(llvm::Type*, llvm::Value*, const llvm::Twine&, bool, llvm::Align, llvm::AtomicOrdering, llvm::SyncScope::ID, llvm::BasicBlock*)’
  198 |   LoadInst(Type *Ty, Value *Ptr, const Twine &NameStr, bool isVolatile,
      |   ^~~~~~~~
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:198:3: note:   candidate expects 8 arguments, 3 provided
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:194:3: note: candidate: ‘llvm::LoadInst::LoadInst(llvm::Type*, llvm::Value*, const llvm::Twine&, bool, llvm::Align, llvm::AtomicOrdering, llvm::SyncScope::ID, llvm::Instruction*)’
  194 |   LoadInst(Type *Ty, Value *Ptr, const Twine &NameStr, bool isVolatile,
      |   ^~~~~~~~
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:194:3: note:   candidate expects 8 arguments, 3 provided
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:192:3: note: candidate: ‘llvm::LoadInst::LoadInst(llvm::Type*, llvm::Value*, const llvm::Twine&, bool, llvm::Align, llvm::BasicBlock*)’
  192 |   LoadInst(Type *Ty, Value *Ptr, const Twine &NameStr, bool isVolatile,
      |   ^~~~~~~~
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:192:3: note:   candidate expects 6 arguments, 3 provided
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:190:3: note: candidate: ‘llvm::LoadInst::LoadInst(llvm::Type*, llvm::Value*, const llvm::Twine&, bool, llvm::Align, llvm::Instruction*)’
  190 |   LoadInst(Type *Ty, Value *Ptr, const Twine &NameStr, bool isVolatile,
      |   ^~~~~~~~
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:190:3: note:   candidate expects 6 arguments, 3 provided
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:188:3: note: candidate: ‘llvm::LoadInst::LoadInst(llvm::Type*, llvm::Value*, const llvm::Twine&, bool, llvm::BasicBlock*)’
  188 |   LoadInst(Type *Ty, Value *Ptr, const Twine &NameStr, bool isVolatile,
      |   ^~~~~~~~
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:188:3: note:   candidate expects 5 arguments, 3 provided
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:186:3: note: candidate: ‘llvm::LoadInst::LoadInst(llvm::Type*, llvm::Value*, const llvm::Twine&, bool, llvm::Instruction*)’
  186 |   LoadInst(Type *Ty, Value *Ptr, const Twine &NameStr, bool isVolatile,
      |   ^~~~~~~~
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:186:3: note:   candidate expects 5 arguments, 3 provided
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:185:3: note: candidate: ‘llvm::LoadInst::LoadInst(llvm::Type*, llvm::Value*, const llvm::Twine&, llvm::BasicBlock*)’
  185 |   LoadInst(Type *Ty, Value *Ptr, const Twine &NameStr, BasicBlock *InsertAtEnd);
      |   ^~~~~~~~
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:185:3: note:   candidate expects 4 arguments, 3 provided
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:183:3: note: candidate: ‘llvm::LoadInst::LoadInst(llvm::Type*, llvm::Value*, const llvm::Twine&, llvm::Instruction*)’
  183 |   LoadInst(Type *Ty, Value *Ptr, const Twine &NameStr,
      |   ^~~~~~~~
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:183:3: note:   candidate expects 4 arguments, 3 provided

What did you expect to see?
A proper build success result.
What did you see instead?
An Error.
/home/ns1server/workarea/llvm-project/llvm/tools/gollvm/bridge/go-llvm.cpp:616:74: error: no matching function for call to ‘llvm::LoadInst::LoadInst(llvm::Type*&, llvm::Value*&, std::string&)’
  616 |     llvm::Instruction *loadInst = new llvm::LoadInst(vt, spaceVal, ldname);
      |                                                                          ^
In file included from /home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/IRBuilder.h:33,
                 from /home/ns1server/workarea/llvm-project/llvm/tools/gollvm/bridge/go-llvm-irbuilders.h:18,
                 from /home/ns1server/workarea/llvm-project/llvm/tools/gollvm/bridge/go-llvm-builtins.h:18,
                 from /home/ns1server/workarea/llvm-project/llvm/tools/gollvm/bridge/go-llvm.cpp:15:
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:198:3: note: candidate: ‘llvm::LoadInst::LoadInst(llvm::Type*, llvm::Value*, const llvm::Twine&, bool, llvm::Align, llvm::AtomicOrdering, llvm::SyncScope::ID, llvm::BasicBlock*)’
  198 |   LoadInst(Type *Ty, Value *Ptr, const Twine &NameStr, bool isVolatile,
      |   ^~~~~~~~
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:198:3: note:   candidate expects 8 arguments, 3 provided
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:194:3: note: candidate: ‘llvm::LoadInst::LoadInst(llvm::Type*, llvm::Value*, const llvm::Twine&, bool, llvm::Align, llvm::AtomicOrdering, llvm::SyncScope::ID, llvm::Instruction*)’
  194 |   LoadInst(Type *Ty, Value *Ptr, const Twine &NameStr, bool isVolatile,
      |   ^~~~~~~~
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:194:3: note:   candidate expects 8 arguments, 3 provided
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:192:3: note: candidate: ‘llvm::LoadInst::LoadInst(llvm::Type*, llvm::Value*, const llvm::Twine&, bool, llvm::Align, llvm::BasicBlock*)’
  192 |   LoadInst(Type *Ty, Value *Ptr, const Twine &NameStr, bool isVolatile,
      |   ^~~~~~~~
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:192:3: note:   candidate expects 6 arguments, 3 provided
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:190:3: note: candidate: ‘llvm::LoadInst::LoadInst(llvm::Type*, llvm::Value*, const llvm::Twine&, bool, llvm::Align, llvm::Instruction*)’
  190 |   LoadInst(Type *Ty, Value *Ptr, const Twine &NameStr, bool isVolatile,
      |   ^~~~~~~~
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:190:3: note:   candidate expects 6 arguments, 3 provided
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:188:3: note: candidate: ‘llvm::LoadInst::LoadInst(llvm::Type*, llvm::Value*, const llvm::Twine&, bool, llvm::BasicBlock*)’
  188 |   LoadInst(Type *Ty, Value *Ptr, const Twine &NameStr, bool isVolatile,
      |   ^~~~~~~~
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:188:3: note:   candidate expects 5 arguments, 3 provided
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:186:3: note: candidate: ‘llvm::LoadInst::LoadInst(llvm::Type*, llvm::Value*, const llvm::Twine&, bool, llvm::Instruction*)’
  186 |   LoadInst(Type *Ty, Value *Ptr, const Twine &NameStr, bool isVolatile,
      |   ^~~~~~~~
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:186:3: note:   candidate expects 5 arguments, 3 provided
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:185:3: note: candidate: ‘llvm::LoadInst::LoadInst(llvm::Type*, llvm::Value*, const llvm::Twine&, llvm::BasicBlock*)’
  185 |   LoadInst(Type *Ty, Value *Ptr, const Twine &NameStr, BasicBlock *InsertAtEnd);
      |   ^~~~~~~~
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:185:3: note:   candidate expects 4 arguments, 3 provided
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:183:3: note: candidate: ‘llvm::LoadInst::LoadInst(llvm::Type*, llvm::Value*, const llvm::Twine&, llvm::Instruction*)’
  183 |   LoadInst(Type *Ty, Value *Ptr, const Twine &NameStr,
      |   ^~~~~~~~
/home/ns1server/workarea/llvm-project/llvm/include/llvm/IR/Instructions.h:183:3: note:   candidate expects 4 arguments, 3 provided
```.

cc @thanm

Any way to fix it? I see that the llvm trunk has a different signature for LoadInst() function.

There has been a lot of churn on LLVM trunk lately relating to alignment -- this is another example. I'll send a CL on Monday to fix it up. In the mean time, try rolling back your LLVM client to e53d4869a0d, that's the most recent version I've verified.

Thanks for this. It works but gold fails to build & link some go packages in libffi .
due to stack split error. I guess it is  a problem with gold and not with gollvm itself.
https://github.com/libffi/libffi.git

Thanks for this. It works but gold fails to build & link some go packages in libffi .
due to stack split error. I guess it is a problem with gold and not with gollvm itself.

I tried building just now with tip libffi and I don't see any issues. Could you please attach the build log that shows the problem?
It would be useful also to know what version of linux you are running. Wondering if this new issue might relate to -fcf-protection=branch (see https://github.com/golang/go/issues/38728).

@codersguild I have submitted https://golang.org/cl/234337 -- let me know if you are seeing any other problems, so I can close out this bug. Thanks.

Thanks.
Not facing any other issue except for libffi.

Thanks. Still happy to look at the libffi / gold issues if you want to post the build log that shows the error.

I will definitely do that. Thanks

Sorry for the delay.
This is the error I am getting when I am using ninja to build gollvm.

I will definitely do that. Thanks.
I will close this issue.

/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-assert.c.o: failed to match split-stack sequence at section 5 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-caller.c.o: failed to match split-stack sequence at section 11 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-caller.c.o: failed to match split-stack sequence at section 13 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-caller.c.o: failed to match split-stack sequence at section 15 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-caller.c.o: failed to match split-stack sequence at section 17 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-cdiv.c.o: failed to match split-stack sequence at section 4 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-cdiv.c.o: failed to match split-stack sequence at section 6 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-cgo.c.o: failed to match split-stack sequence at section 4 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-cgo.c.o: failed to match split-stack sequence at section 6 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-fieldtrack.c.o: failed to match split-stack sequence at section 7 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-nosys.c.o: failed to match split-stack sequence at section 4 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-nosys.c.o: failed to match split-stack sequence at section 6 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-nosys.c.o: failed to match split-stack sequence at section 8 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-nosys.c.o: failed to match split-stack sequence at section 10 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-nosys.c.o: failed to match split-stack sequence at section 12 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-nosys.c.o: failed to match split-stack sequence at section 14 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-nosys.c.o: failed to match split-stack sequence at section 16 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-nosys.c.o: failed to match split-stack sequence at section 18 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-nosys.c.o: failed to match split-stack sequence at section 20 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-nosys.c.o: failed to match split-stack sequence at section 22 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-nosys.c.o: failed to match split-stack sequence at section 24 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-nosys.c.o: failed to match split-stack sequence at section 26 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-nosys.c.o: failed to match split-stack sequence at section 28 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-nosys.c.o: failed to match split-stack sequence at section 30 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-reflect-call.c.o: failed to match split-stack sequence at section 5 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-setenv.c.o: failed to match split-stack sequence at section 5 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-unsetenv.c.o: failed to match split-stack sequence at section 5 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-varargs.c.o: failed to match split-stack sequence at section 4 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-varargs.c.o: failed to match split-stack sequence at section 6 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-varargs.c.o: failed to match split-stack sequence at section 8 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-varargs.c.o: failed to match split-stack sequence at section 10 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-varargs.c.o: failed to match split-stack sequence at section 12 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-varargs.c.o: failed to match split-stack sequence at section 14 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-varargs.c.o: failed to match split-stack sequence at section 16 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-varargs.c.o: failed to match split-stack sequence at section 18 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/go-varargs.c.o: failed to match split-stack sequence at section 20 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/env_posix.c.o: failed to match split-stack sequence at section 4 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/panic.c.o: failed to match split-stack sequence at section 9 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/print.c.o: failed to match split-stack sequence at section 8 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/runtime/print.c.o: failed to match split-stack sequence at section 10 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/go/log/syslog/syslog_c.c.o: failed to match split-stack sequence at section 5 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/go/reflect/makefunc_ffi_c.c.o: failed to match split-stack sequence at section 4 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/go/syscall/errno.c.o: failed to match split-stack sequence at section 4 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/go/syscall/errno.c.o: failed to match split-stack sequence at section 6 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libgo_c_piclib.dir/__/gofrontend/libgo/go/syscall/signame.c.o: failed to match split-stack sequence at section 5 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/backtrace.c.o: failed to match split-stack sequence at section 4 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/backtrace.c.o: failed to match split-stack sequence at section 6 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/dwarf.c.o: failed to match split-stack sequence at section 18 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/dwarf.c.o: failed to match split-stack sequence at section 79 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/dwarf.c.o: failed to match split-stack sequence at section 82 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/elf.c.o: failed to match split-stack sequence at section 31 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/elf.c.o: failed to match split-stack sequence at section 35 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/fileline.c.o: failed to match split-stack sequence at section 6 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/fileline.c.o: failed to match split-stack sequence at section 10 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/mmapio.c.o: failed to match split-stack sequence at section 5 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/mmapio.c.o: failed to match split-stack sequence at section 8 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/mmap.c.o: failed to match split-stack sequence at section 4 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/mmap.c.o: failed to match split-stack sequence at section 7 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/mmap.c.o: failed to match split-stack sequence at section 9 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/posix.c.o: failed to match split-stack sequence at section 4 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/posix.c.o: failed to match split-stack sequence at section 7 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/print.c.o: failed to match split-stack sequence at section 5 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/print.c.o: failed to match split-stack sequence at section 8 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/print.c.o: failed to match split-stack sequence at section 10 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/simple.c.o: failed to match split-stack sequence at section 4 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/simple.c.o: failed to match split-stack sequence at section 6 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libbacktrace_piclib.dir/libbacktrace/state.c.o: failed to match split-stack sequence at section 4 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libffi_piclib.dir/libffi/src/closures.c.o: failed to match split-stack sequence at section 4 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libffi_piclib.dir/libffi/src/closures.c.o: failed to match split-stack sequence at section 6 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libffi_piclib.dir/libffi/src/closures.c.o: failed to match split-stack sequence at section 14 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libffi_piclib.dir/libffi/src/closures.c.o: failed to match split-stack sequence at section 22 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libffi_piclib.dir/libffi/src/closures.c.o: failed to match split-stack sequence at section 26 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libffi_piclib.dir/libffi/src/raw_api.c.o: failed to match split-stack sequence at section 8 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libffi_piclib.dir/libffi/src/raw_api.c.o: failed to match split-stack sequence at section 12 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libffi_piclib.dir/libffi/src/raw_api.c.o: failed to match split-stack sequence at section 14 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libffi_piclib.dir/libffi/src/java_raw_api.c.o: failed to match split-stack sequence at section 4 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libffi_piclib.dir/libffi/src/java_raw_api.c.o: failed to match split-stack sequence at section 6 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libffi_piclib.dir/libffi/src/java_raw_api.c.o: failed to match split-stack sequence at section 12 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libffi_piclib.dir/libffi/src/java_raw_api.c.o: failed to match split-stack sequence at section 14 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libffi_piclib.dir/libffi/src/x86/ffi64.c.o: failed to match split-stack sequence at section 12 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libffi_piclib.dir/libffi/src/x86/ffi64.c.o: failed to match split-stack sequence at section 20 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libffi_piclib.dir/libffi/src/x86/ffi64.c.o: failed to match split-stack sequence at section 22 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libffi_piclib.dir/libffi/src/x86/ffi64.c.o: failed to match split-stack sequence at section 26 offset 0
/usr/bin/ld.gold: error: tools/gollvm/libgo/CMakeFiles/libffi_piclib.dir/libffi/src/x86/ffiw64.c.o: failed to match split-stack sequence at section 16 offset 0
collect2: error: ld returned 1 exit status
[3485/3544] Linking C static library tools/gollvm/libgo/libgo.a
ninja: build stopped: subcommand failed.

The ld.gold error message looks a lot like one reported in a previous bug, https://github.com/golang/go/issues/38728.
Can you please check to make sure that your client is sync'd past the CLs that fix this issue, e.g. 0edc44a02852dd0d56cac35163a45634b497d6cd.
You may also need to delete your CMakeCache.txt file and re-run cmake.

I built gollvm in a docker and it seems to work there quite well. I am not facing any issues and the build is passing with all tests OK.

Just wanted to update on one thing. The gold offset error occurs only with focal release of Ubuntu. On bionic it works fine and the build succeeds.


Certificates prompt on the browser even though the require_client_certificate is specified to be false

Title: When accessing the UI of my server, the browser will prompt the certificate selection although the require_client_certificate is specified to be false
Description:

I have made the DownstreamTlsContext to not require a client certificate but seems like it's not working. Because I'm still running into the certificate prompt when accessing my server via the browser.
Another potential bug: when I specify the include_peer_certificate to be true in the ext auth service, if and only if the validation_context is setup properly, the client certificate will be included in the CheckRequest.

Repro steps:

For the certificate prompt

please use the following envoy yaml config file to start a proxy
access this server via the browser

For the potential bug I mentioned above

please use the following envoy yaml config file to start a proxy
on the server internally, try this curl API call: curl -k --key "the location of your private key" --cert "the location of your public certificate" https://localhost/ANY_API_SUPPORTED and you are supposed to see the client cert in the CheckRequest
remove the config "validation_context" in the yaml config file and restart the proxy
try that above curl API call again and you are supposed to not see the client cert (which confused me a lot.)

Runtime Envoy Config:
Here is the listener and cluster config dump for the following envoy yaml config file:
(some ports used are modified on purpose out of privacy and may not match that in the real envoy yaml config file)
https://paste.ubuntu.com/p/kHbyG7qwpn/
Logs:
Here is the envoy log:
(some useless logs are ignored and the most significant part is the client cert shown in the CheckRequest)
https://paste.ubuntu.com/p/s62F24YzvR/
Envoy yaml:
static_resources:
  listeners:
  - address:
      socket_address:
        address: <IP_ADDRESS>
        port_value: 443
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: AUTO
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: upstream
              domains:
              - "*"
              routes:
              - match:
                  prefix: "/api"
                route:
                  cluster: test-service
              - match:
                  prefix: "/"
                route:
                  cluster: reverse-proxy
          http_filters:
          - name: envoy.filters.http.ext_authz
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
              grpc_service:
                envoy_grpc:
                  cluster_name: ext-authz
                timeout: 0.250s
              transport_api_version: V3
              include_peer_certificate: true
          - name: envoy.filters.http.router
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
          common_tls_context:
            tls_params: {"cipherSuites": ["ECDHE-ECDSA-AES128-GCM-SHA256", "ECDHE-RSA-AES128-GCM-SHA256", "ECDHE-ECDSA-AES128-SHA", "ECDHE-RSA-AES128-SHA", "AES128-GCM-SHA256", "AES128-SHA", "ECDHE-ECDSA-AES256-GCM-SHA384", "ECDHE-RSA-AES256-GCM-SHA384", "ECDHE-ECDSA-AES256-SHA", "ECDHE-RSA-AES256-SHA", "AES256-GCM-SHA384", "AES256-SHA"],"ecdhCurves": ["P-256"]}
            tls_certificates:
            - certificate_chain:
                filename: ".tomcat_cert.pem"
              private_key:
                filename: ".tomcat_key.pem"
            validation_context:
              trust_chain_verification: "ACCEPT_UNTRUSTED"
          require_client_certificate: false

  clusters:
  - name: test-service
    connect_timeout: 0.25s
    type: static
    lb_policy: round_robin
    load_assignment:
      cluster_name: test-service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: <IP_ADDRESS>
                port_value: 6502
  - name: ext-authz
    connect_timeout: 0.25s
    type: static
    lb_policy: round_robin
    http2_protocol_options: {}
    load_assignment:
      cluster_name: ext-authz
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: <IP_ADDRESS>
                port_value: 6501
  - name: reverse-proxy
    connect_timeout: 0.25s
    type: static
    lb_policy: round_robin
    load_assignment:
      cluster_name: reverse-proxy
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: <IP_ADDRESS>
                port_value: 6500

admin:
  access_log_path: "/dev/null"
  admin_allowed: ["/", "/certs", "/clusters", "/config_dump", "/init_dump", "/listeners", "/logging", "/stats"]
  address:
    socket_address:
      address: <IP_ADDRESS>
      port_value: 8082

If you specify ACCEPT_UNTRUSTED, the verify_mode will be set to SSL_VERIFY_PEER. https://github.com/envoyproxy/envoy/blob/main/source/extensions/transport_sockets/tls/cert_validator/default_validator.cc#L58
Which will still request a client certificate even if you set require_client_certificate to false. https://github.com/envoyproxy/envoy/blob/main/source/extensions/transport_sockets/tls/cert_validator/default_validator.cc#L441

I think this flag is more for debugging, because if you wanna use mTLS and by providing related configuration, it should always ask for the client certificate. "Asking" and "Verifying" are two different concepts here. We can still ask but not verifying it.
I guess it makes sense to have the flag for debugging purpose, which is something that need to be fixed. The flag should unset all the verification mode.

I think this flag is more for debugging, because if you wanna use mTLS and by providing related configuration, it should always ask for the client certificate. "Asking" and "Verifying" are two different concepts here. We can still ask but not verifying it.
I guess it makes sense to have the flag for debugging purpose, which is something that need to be fixed. The flag should unset all the verification mode.

Get it. Thanks for providing these details.
BTW do you happen to know that there’s some way to let Envoy forward the client cert(if presented in the incoming request) to the ext authz service with only one way ssl enabled?
Because my ultimate aim is to give the access of the client cert to the ext authz service. And why the above doesn’t work for me? It’s because if I enable mTLS(the two way ssl) in Envoy, the browser will always ask for a client cert which is a little bit annoying(but the tomcat won’t do this when mTLS is enabled).

Are you saying you want to forward the client cert from Envoy to your backend? This might help - https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-forwarded-client-cert

My answers are not official btw, ill let the envoy team to confirm them :)

Are you saying you want to forward the client cert from Envoy to your backend? This might help - https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-forwarded-client-cert

This ext authz service is a backend service but it's used differently compared to other backend service.
Actually it's used as an auth filter to authN the incoming request and IFF it passes this filter, then it can be forwarded to the real backend service(upstream service).
Thanks anyway :)

Are you saying you want to forward the client cert from Envoy to your backend? This might help - https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-forwarded-client-cert

This XFCC header works better for the mTLS connections. Seems like not a working option in my case per the experiment I did.

Found out the rootcause of this issue: https://github.com/envoyproxy/envoy/issues/17348


PR checks not always running

I've noticed recently that tests aren't getting run automatically for new pull requests. Today there was also an issue with the CLA check (it didn't get re-run as expected on PR #709, but I've confirmed on our side that pollyMath did sign it). I don't know of any recent changes we've made that might have caused this, but GH's and Travis CI's status pages both look good, so it's probably us.

We can close this issue right? I see that the patch is already merged.


Extension

How do i use this as a chrome extension,if possible?

This unfortunately is not possible, however, you could give Sapiengraph a try which pulls a lot of data from LinkedIn straight into a Google Sheet. Pretty convenient.


v1 doesn't support send to multiple tokens anymore ( any workarounds beside pub/sub ? )

I am currently migrating from legacy http to v1 http, but I encounter this problem where v1 looks like it doesn't accept array of tokens to send to.
similar to this person
https://groups.google.com/g/firebase-talk/c/4GuTzvRT_10?pli=1
after researching a bit it seems like the better practice is to do a pub/sub
where I need to make each device subscribe to a specific topic,
and then just broadcast a single message that will target those who are subscribe to the topic.
But I believe this will change our system a bit because we now want to be able to store the data of users which are currently
subscribed to a specific topic. ( We could maybe do a dirty implementation of temporarily subscribing tokens -> send message -> then unsubscribe all of them,,, but I believe this is not what firebase want us to do... or am I wrong)

https://github.com/decision-labs/fcm/issues/122

firebase suggest the following
for 1 million recipients -> it is better to fanout
for less than 1 million recipients or whatever is advantageous to migrate to fanout for you-> it is better to send request


[SPARK-29425][SQL] The ownership of a database should be respected

What changes were proposed in this pull request?

Keep the owner of a database when executing alter database commands
Why are the changes needed?

Spark will inadvertently delete the owner of a database for executing databases ddls
Does this PR introduce any user-facing change?

NO
How was this patch tested?

add and modify uts

cc @cloud-fan @gatorsmile @wangyum

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16918/
Test PASSed.

retest this please

Test build #112267 has started for PR 26080 at commit 9f8ce9f.

Merged build finished. Test PASSed.

Test build #112267 has finished for PR 26080 at commit 9f8ce9f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112267/
Test FAILed.

Test build #112277 has started for PR 26080 at commit 3077f5b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17260/
Test PASSed.

Test build #112280 has started for PR 26080 at commit 4ccec2a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17263/
Test PASSed.

Test build #112277 has finished for PR 26080 at commit 3077f5b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112277/
Test FAILed.

Test build #112280 has finished for PR 26080 at commit 4ccec2a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112280/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17268/
Test PASSed.

Test build #112285 has started for PR 26080 at commit f463f48.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17269/
Test PASSed.

Test build #112285 has finished for PR 26080 at commit f463f48.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112285/
Test FAILed.

Test build #112310 has started for PR 26080 at commit 9efc8cf.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17293/
Test PASSed.

Test build #112310 has finished for PR 26080 at commit 9efc8cf.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112310/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17295/
Test PASSed.

Test build #112315 has started for PR 26080 at commit 9efc8cf.

Test build #112315 has finished for PR 26080 at commit 9efc8cf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112315/
Test FAILed.

Test build #112354 has started for PR 26080 at commit 892813e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17333/
Test PASSed.

Test build #112354 has finished for PR 26080 at commit 892813e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112354/
Test FAILed.

Test build #112381 has started for PR 26080 at commit f86d84d.

Merged build finished. Test PASSed.

Test build #112381 has finished for PR 26080 at commit f86d84d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112381/
Test FAILed.

Test build #112423 has started for PR 26080 at commit f86d84d.

The ut failed seems not related to this change https://issues.apache.org/jira/projects/SPARK/issues/SPARK-29538

Test build #112423 has finished for PR 26080 at commit f86d84d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112423/
Test PASSed.

gentle ping @cloud-fan

Test build #112602 has started for PR 26080 at commit 72f8c7a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17566/
Test PASSed.

Test build #112602 has finished for PR 26080 at commit 72f8c7a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112602/
Test PASSed.

Hi @cloud-fan, could we take this forward? Although Spark does not have an ACL management implementation of its own, but it does not mean that we can not use spark' extensions to realize it. I have some oss projects of my own - https://github.com/yaooqinn/spark-authorizer (V1) https://github.com/yaooqinn/spark-ranger (V2) - which can make spark achieve Column/Row-level fine-grained authorization support.  We always use Spark/Hive, maybe impala together towards the same hive metastore with all components secured, which is a common big data warehouse scenario, where data and meta security matters.
If Spark always change databases or tables' ownership unexpectedly, it will become hard to use in a secured data warehouse.
Thanks.

Test build #113798 has started for PR 26080 at commit 16f64f1.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18670/
Test PASSed.

Test build #113798 has finished for PR 26080 at commit 16f64f1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113798/
Test PASSed.

what's the meaning of owner type? I can't find it in CatalogTable. Shall we make them consistent?

what's the meaning of owner type? I can't find it in CatalogTable. Shall we make them consistent?

The Owner Type a.k.a PrincipalType can be USER GROUP and ROLE, meanings are simple as their names. If we will support ACL management in the future or just keep consistency with hive, I suggest we make them consistent in CatalogTable.

Test build #114105 has started for PR 26080 at commit 34d84fa.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18963/
Test PASSed.

Test build #114105 has finished for PR 26080 at commit 34d84fa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114105/
Test FAILed.

Test build #114131 has started for PR 26080 at commit fc3698e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18989/
Test PASSed.

Test build #114131 has finished for PR 26080 at commit fc3698e.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114131/
Test FAILed.

retest this please

Test build #114148 has started for PR 26080 at commit fc3698e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19007/
Test PASSed.

Test build #114160 has started for PR 26080 at commit 5467ede.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19015/
Test PASSed.

Test build #114148 has finished for PR 26080 at commit fc3698e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114148/
Test PASSed.

Test build #114160 has finished for PR 26080 at commit 5467ede.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114160/
Test PASSed.

Test build #114250 has started for PR 26080 at commit 035fc9c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19099/
Test PASSed.

Test build #114250 has finished for PR 26080 at commit 035fc9c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114250/
Test PASSed.

Can we create an umbrella JIRA for the ownership stuff? IIUC, what we need to do

add owner type to table
add owner name/type to database (this PR)
add SQL command to set owner name/type
support v2 catalog API, like https://github.com/apache/spark/pull/26617

Can we create an umbrella JIRA for the ownership stuff? IIUC, what we need to do

add owner type to table
add owner name/type to database (this PR)
add SQL command to set owner name/type
support v2 catalog API, like #26617

SGTM, do I have the privilege to create an umbrella JIRA? If I do, I'd love to.

Test build #114721 has started for PR 26080 at commit 9ffbd64.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19545/
Test PASSed.

Test build #114721 has finished for PR 26080 at commit 9ffbd64.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114721/
Test FAILed.

Test build #114730 has started for PR 26080 at commit 0762f84.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19554/
Test PASSed.

Test build #114730 has finished for PR 26080 at commit 0762f84.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114730/
Test PASSed.

Test build #114765 has started for PR 26080 at commit e5c162e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19587/
Test PASSed.

Test build #114778 has started for PR 26080 at commit 87c1f28.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19599/
Test PASSed.

Test build #114765 has finished for PR 26080 at commit e5c162e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114765/
Test PASSed.

Test build #114782 has started for PR 26080 at commit 4aeeeda.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19603/
Test PASSed.

Test build #114778 has finished for PR 26080 at commit 87c1f28.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114778/
Test FAILed.

Test build #114782 has finished for PR 26080 at commit 4aeeeda.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114782/
Test FAILed.

Test build #114796 has started for PR 26080 at commit 89050ec.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19618/
Test PASSed.

Test build #114796 has finished for PR 26080 at commit 89050ec.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114796/
Test PASSed.

Test build #114833 has started for PR 26080 at commit f11e7c8.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19656/
Test PASSed.

Test build #114833 has finished for PR 26080 at commit f11e7c8.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114833/
Test FAILed.

LGTM except https://github.com/apache/spark/pull/26080/files#r353595224

Test build #114844 has started for PR 26080 at commit c3da686.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19667/
Test PASSed.

Test build #114844 has finished for PR 26080 at commit c3da686.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114844/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19677/
Test PASSed.

Test build #114854 has started for PR 26080 at commit 364d120.

Test build #114861 has started for PR 26080 at commit 8ad8645.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19684/
Test PASSed.

Test build #114864 has started for PR 26080 at commit 372923a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19687/
Test PASSed.

Test build #114861 has finished for PR 26080 at commit 8ad8645.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #114866 has started for PR 26080 at commit 11f4e4e.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114861/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19689/
Test PASSed.

Test build #114868 has started for PR 26080 at commit e92c93f.

Test build #114854 has finished for PR 26080 at commit 364d120.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19691/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114854/
Test PASSed.

Test build #114864 has finished for PR 26080 at commit 372923a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114864/
Test FAILed.

Test build #114866 has finished for PR 26080 at commit 11f4e4e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114866/
Test FAILed.

Test build #114868 has finished for PR 26080 at commit e92c93f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114868/
Test PASSed.

thanks, merging to master!

thanks for merging and your help


Validation of dx-date-box with native Angular Forms

In our application we need a form which allows a user to input a time range, which allows him to delete a reservation or a part of it. Because of that, the time input should be limited by the start time and end time of the corresponding reservation. Additionally the user is only allowed to choose values according to a grid (e.g. 10:00, 10:15, 10:30, 10:45, etc.).
Unfortunately we encountered several problems while implementing this use case with devextreme components and native Angular form (template-driven and reactive).
I tried to write a plnkr showing the problems we encountered: http://plnkr.co/edit/P7ZkWk
Validation with template-driven form
We decided to use the validation capabilities from Angular and the devextreme components to show the validation messages by using the properties isValid and validationError. Furthermore we subscribed on the forms valueChanges and created the corresponding objects for validationError.
 ngAfterViewChecked() {
    this.templateFormChanged();

  }

  templateFormChanged() {
    if (this.currentForm === this.templateForm) { return; }
    this.templateForm = this.currentForm;
    if (this.templateForm) {
      this.templateForm.valueChanges
        .subscribe(data => this.onTemplateValueChanged(data));
    }
  }

  onTemplateValueChanged(data?: any) {
    if (!this.templateForm) { return; }
    const form = this.templateForm.form;

    for (const field in this.templateFormErrors) {
      var message = '';
      const control = form.get(field);

      if (control && control.dirty && !control.valid) {
        for (const key in control.errors) {
          message += key + ' ';
        }
        this.templateFormErrors[field] = { message: message };
      } else {
        this.templateFormErrors[field] = '';
      }
    }

This doesn't work for template-driven forms, because the control is immediately "dirty" after the page load and therefore the user sees a error before even entering a single character.
(plnkr: complete template-driven section)
Validation of dx-date-box in time mode with min and max
As explained above, we want to limit the dx-date-box and only accept values which are within a given range. The definition of the dx-date-box is as following example, with minTimeStart and maxTimeStart as the valid time range.
<dx-date-box formControlName="shiftStart" name="shiftStart" type="time" [min]="minTimeStart" [max]="maxTimeStart" [interval]="30" required [isValid]="reactiveForm.controls.shiftStart.valid || reactiveForm.controls.shiftStart.pristine" [validationError]="reactiveFormErrors.shiftStart"></dx-date-box>

If we do it like that, the dx-date-box shows the error message "value is out of range" after the user selects an value from the drop-down-list, despite the control itself is valid and reactiveForm.controls.shiftStart.valid || reactiveForm.controls.shiftStart.pristine evaluates to true. This is most likely because our value for min has a date which is not the same as the date of the internal value and this causes the internal validation to fail.
(plnkr: template-driven section (both dx-date-box) and reactive section (start of shift))
After some research we concluded, that the isValid and validationError settings were overwritten internally and we found a workaround using a listener for the onValueChanged event (and a custom validator to guarantee the validity of the values).
  onOptionChanged(e) {
    console.log(e);
    if (e.name == 'isvalid' || e.name == 'validationError') {
      e.component._options.isValid = this.reactiveForm.get('shiftEnd').valid && !this.reactiveForm.errors;
      e.component._options.validationError = this.reactiveFormErrors['shiftEnd'];
      e.component.repaint();
    }
  }

Sadly, this workaround does NOT work on mobile devices and the native time input control (tested on Nexus 5X), because after the input is accepted by the user, the control does the internal validation, and does not accept the value and the repaint of the workaround restores the previous value.
(plnkr: reactive section (end of shift))
To help us process the issue more efficiently, please provide the following information:

Specify the version of the devextreme-angular and devextreme packages you are using.
devexteme version: 17.1.4
devextreme-angular version: 17.1.4

Specify the type of the issue (check one with "x"):
[x] bug Report
[ ] feature request
[ ] support inquiry

"value is out of range" bug on dx-date-box with min and max values is fixed with #494


Sound: FC upon changing the Headphone Gain (mako kernel with faux123 sound control)

When I don't use the option "Independent Head Phone Gain" and then change the Headphone Gain, the app has a FC. See logcat: https://gist.github.com/spezi77/f561ce97741e4d43a2b1
The other options work fine. Also when using "Independent Head Phone Gain" I can change Headphone Gain Left/Right without any issue.

This should fix it.
https://github.com/yoinx/kernel_adiutor/commit/921cc3c28e570e431f1ea8f7d1dd3f8848fb568b
I assume that you're building your own apk? If so, could you test it with that change and make sure that it works?

Thanks very much. I will add your commit and test it by tomorrow with a new build. :)

https://github.com/yoinx/kernel_adiutor/commit/a7eed03b2012398860537d9d6dcb9b15d05e8b55
Tested this one on a different kernel, it fixes the issue.

Thanks, that has fixed it.


CT_CellFormula.Write: fix writing the si attribute

CT_CellFormula.Write method ensure the siField is written into fField's attributes on Write even when it has a default value for cases when the type of the formula is shared formula.
It seems that Excel expects this atribute to be written into xml even when it is of the default value for shared formulas. If not present - the xlsx file is flagged by it as broken and is repaired.
Not test is added because it is hard to test for Excel not trying to repair the file.

LGTM


update to pull from the 8.4 nightlies

@james-antill @siteshwar Is this all that we need to update to pull content from a different minor-nightly?

Yeh

It might be worth adding a command line option for that.

Nevermind. LGTM.


Ispn 5943 Access log of cache operations

https://issues.jboss.org/browse/ISPN-5943
Added rest and made logging configuration more consistent

Rebased and fixed trace level.

Squashed and pushed. Thanks


package.json: order dev dependencies alphabetically

npm re-orders these every time a dev dependency is added or removed.  Making the change permanently would save a bit of fiddling when changing dependencies.

i've resisted doing this myself since it'll nuke the blame but i don't feel strongly


Question: why LoRa_APRS_Wx_Station features isn't implement in LoRa_APRS_iGate LoRa_APRS_iGate?

As in topic: why LoRa_APRS_Wx_Station features isn't implement in LoRa_APRS_iGate LoRa_APRS_iGate?
e.g. as a additional section to activate this functionality (sensors)
I just wanted to ask you this when I get back from vacation.

Hi,
as the first iteration of this new firmware there is not plan to add this soon to igate because of the needed timing for sensors to read the data and also to avoid loosing data when Rx, or digirepeating and so on
in a close future timing I will try to though

I would also be appreciated if the weather funtions (esp. Rain data) will be implemented into IGate as an additional section. Do you have an idea when this will be implemented?

I would also be appreciated if the weather funtions (esp. Rain data) will be implemented into IGate as an additional section. Do you have an idea when this will be implemented?

i have many things to add to tracker (webconfig and web installer) and also a few new things to the igate.
adding this sensors could be done but I need time (so maybe a few weeks until all the other requests are done)


Improve Memory usage with large numbers of groups

Is your feature request related to a problem or challenge?
When running a query with "high cardinality" grouping in DataFusion, the memory usage increases linearly both with the number of groups (expected) but also with the number of cores.
Is the root cause of @ychen7's observation that ClickBench q32 fails As https://github.com/apache/arrow-datafusion/issues/5276#issuecomment-1504574207
To reproduce, get the ClickBench data https://github.com/ClickHouse/ClickBench/tree/main#data-loading and run this:
CREATE EXTERNAL TABLE hits
STORED AS PARQUET
LOCATION 'hits.parquet';

set datafusion.execution.target_partitions = 1;
SELECT "WatchID", "ClientIP", COUNT(*) AS c, SUM("IsRefresh"), AVG("ResolutionWidth") FROM hits GROUP BY "WatchID", "ClientIP" ORDER BY c DESC LIMIT 10;

set datafusion.execution.target_partitions = 4;
SELECT "WatchID", "ClientIP", COUNT(*) AS c, SUM("IsRefresh"), AVG("ResolutionWidth") FROM hits GROUP BY "WatchID", "ClientIP" ORDER BY c DESC LIMIT 10;

This is what the memory usage looks like:

The reason for this behavior can be found in the plan and the multi-stage hash grouping that is done:
explain SELECT "WatchID", "ClientIP", COUNT(*) AS c, SUM("IsRefresh"), AVG("ResolutionWidth") FROM hits GROUP BY "WatchID", "ClientIP" ORDER BY c DESC LIMIT 10;

| physical_plan | GlobalLimitExec: skip=0, fetch=10                                                                                |
|               |   SortPreservingMergeExec: [c@2 DESC]                                                                                |
|               |     SortExec: fetch=10, expr=[c@2 DESC]                                                                                |
|               |       ProjectionExec: expr=[WatchID@0 as WatchID, ClientIP@1 as ClientIP, COUNT(UInt8(1))@2 as c, SUM(hits.IsRefresh)@3 as SUM(hits.IsRefresh), AVG(hits.ResolutionWidth)@4 as AVG(hits.ResolutionWidth)]                                                                                |
|               |         AggregateExec: mode=FinalPartitioned, gby=[WatchID@0 as WatchID, ClientIP@1 as ClientIP], aggr=[COUNT(UInt8(1)), SUM(hits.IsRefresh), AVG(hits.ResolutionWidth)]                                                                                |
|               |           CoalesceBatchesExec: target_batch_size=8192                                                                                |
|               |             RepartitionExec: partitioning=Hash([WatchID@0, ClientIP@1], 16), input_partitions=16                                                                                |
|               |               AggregateExec: mode=Partial, gby=[WatchID@0 as WatchID, ClientIP@1 as ClientIP], aggr=[COUNT(UInt8(1)), SUM(hits.IsRefresh), AVG(hits.ResolutionWidth)]                                                                                |
|               |                 ParquetExec: file_groups={16 groups: [[Users/alamb/Software/clickbench_hits_compatible/hits.parquet:0..923748528], [Users/alamb/Software/clickbench_hits_compatible/hits.parquet:923748528..1847497056], [Users/alamb/Software/clickbench_hits_compatible/hits.parquet:1847497056..2771245584], [Users/alamb/Software/clickbench_hits_compatible/hits.parquet:2771245584..3694994112], [Users/alamb/Software/clickbench_hits_compatible/hits.parquet:3694994112..4618742640], ...]}, projection=[WatchID, ClientIP, IsRefresh, ResolutionWidth] |
|               |                                                                                |
+---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Specifically since the groups are arbitrarily distributed in the files, the first AggregateExec: mode=Partial has to build a hash table that has entries for all groups. As the number of target partitions goes up, the number of  AggregateExec: mode=Partial goes up to and thus so does the number of copies of the data
The AggregateExec: mode=FinalPartitioned only see a distinct subset of the keys and thus as the number of target partitions goes up  there are more AggregateExec: mode=FinalPartitioned  s each sees a smaller and smaller subset of the group keys
In pictures:
               ▲                          ▲
               │                          │
               │                          │
               │                          │
               │                          │
               │                          │
   ┌───────────────────────┐  ┌───────────────────────┐       4. Each AggregateMode::Final
   │GroupBy                │  │GroupBy                │       GroupBy has an entry for its
   │(AggregateMode::Final) │  │(AggregateMode::Final) │       subset of groups (in this case
   │                       │  │                       │       that means half the entries)
   └───────────────────────┘  └───────────────────────┘
               ▲                          ▲
               │                          │
               └─────────────┬────────────┘
                             │
                             │
                             │
                ┌─────────────────────────┐                   3. Repartitioning by hash(group
                │       Repartition       │                   keys) ensures that each distinct
                │         HASH(x)         │                   group key now appears in exactly
                └─────────────────────────┘                   one partition
                             ▲
                             │
             ┌───────────────┴─────────────┐
             │                             │
             │                             │
┌─────────────────────────┐  ┌──────────────────────────┐     2. Each AggregateMode::Partial
│        GroubyBy         │  │         GroubyBy         │     GroupBy has an entry for *all*
│(AggregateMode::Partial) │  │ (AggregateMode::Partial) │     the groups
└─────────────────────────┘  └──────────────────────────┘
             ▲                             ▲
             │                            ┌┘
             │                            │
        .─────────.                  .─────────.
     ,─'           '─.            ,─'           '─.
    ;      Input      :          ;      Input      :          1. Since input data is
    :   Partition 0   ;          :   Partition 1   ;          arbitrarily or RoundRobin
     ╲               ╱            ╲               ╱           distributed, each partition
      '─.         ,─'              '─.         ,─'            likely has all distinct
         `───────'                    `───────'

Some example data:
              ┌─────┐                ┌─────┐
              │  1  │                │  3  │
              ├─────┤                ├─────┤
              │  2  │                │  4  │                After repartitioning by
              └─────┘                └─────┘                hash(group keys), each distinct
              ┌─────┐                ┌─────┐                group key now appears in exactly
              │  1  │                │  3  │                one partition
              ├─────┤                ├─────┤
              │  2  │                │  4  │
              └─────┘                └─────┘

─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─

              ┌─────┐                ┌─────┐
              │  2  │                │  2  │
              ├─────┤                ├─────┤
              │  1  │                │  2  │
              ├─────┤                ├─────┤
              │  3  │                │  3  │
              ├─────┤                ├─────┤
              │  4  │                │  1  │
              └─────┘                └─────┘                Input data is arbitrarily or
                ...                    ...                  RoundRobin distributed, each
              ┌─────┐                ┌─────┐                partition likely has all
              │  1  │                │  4  │                distinct group keys
              ├─────┤                ├─────┤
              │  4  │                │  3  │
              ├─────┤                ├─────┤
              │  1  │                │  1  │
              ├─────┤                ├─────┤
              │  4  │                │  3  │
              └─────┘                └─────┘

          group values           group values
          in partition 0         in partition 1

Describe the solution you'd like
TLDR is I would like to propose updating the AggregateExec: mode=Partial to emit their hash tables if they see more than some fixed size number of groups (I think @mingmwang  said DuckDB uses a value of 10,000 for this)
This approach bounds the memory usage (to some fixed constant * the target partitioning) and also should perform quite well
In the literature I think this approach could be called "dynamic partitioning" as it switches approaches based on the actual cardinality of the groups in the dataset
Describe alternatives you've considered
One potential thing that might be suggested is simply to repartition the input to AggregateExec: mode=Partial
This approach would definitely reduce the memory requirements, but it would mean that we would have to hash repartition all the input rows so the number of input values that need to be hashed / copied would likely be  much higher (at least as long as the group hasher and hash repartitioner can't share the hashes, which is the case today)
The current strategy actually works very well for low cardinality group bys because the AggregateExec: mode=Partial can reduce the size of the intermediate result that needs to be hash repartitioned to a very small size
Additional context
We saw this in IOx while working on some tracing queries that look very similar to the ClickBench query, something like the following to get the top ten traces
SELECT trace_id, max(time)
FROM traces
GROUP BY trace_id
ORDER BY max(time)
LIMIT 10;

I did some tests just based on a heuristic (e.g. number of columns in input / group by) in https://github.com/apache/arrow-datafusion/pull/6938 but saw both perf. improvements (likely the high cardinality queries) and degradations.
Also for distributed systems like Ballista, the partial / final approach probably works better in most cases (even for higher cardinality ones), so I think we would have to make this behaviour configurable.

it also seems hash partitioning is not really fast ATM)

Yes -- here are some ideas to improve things:

Reuse the hash values (already computed for the Partial group bys)
Avoid the extra copy with CoalesceBatches -- today we "take" in in repartiton (which copies things) and then CoealsceBatches concat's them again with another copy)

DuckDB's partitioned hash table only take place with its pipeline execution model.

I think we can skip the partial aggregation adaptively if the partial aggregation can not reduce the row count.

Yes I think this is a good strategy 👍

Updates from @yahoNanJing
I am running some tests of the various approaches on

fixed width row format, inlined into hash table: https://github.com/yahoNanJing/arrow-datafusion/tree/EBAY-KYLIN-4003-5
unsafe row access https://github.com/apache/arrow-rs/pull/4539

I think this PR contains some nice ideas
https://github.com/duckdb/duckdb/pull/8475

I did some experimentation with emitting of rows once every 10K is hit (but keep grouping) in this branch:
https://github.com/Dandandan/arrow-datafusion/tree/emit_rows_aggregate
Doing that reduces the memory usage, but often with higher cost, which can be seen in the benchmark:
Comparing main and emit_rows_aggregate
--------------------
Benchmark tpch_mem.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃     main ┃ emit_rows_aggregate ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │ 294.54ms │            293.83ms │     no change │
│ QQuery 2     │  38.74ms │             37.99ms │     no change │
│ QQuery 3     │  48.79ms │             48.48ms │     no change │
│ QQuery 4     │  40.19ms │             39.64ms │     no change │
│ QQuery 5     │  96.84ms │             95.31ms │     no change │
│ QQuery 6     │  10.81ms │             10.73ms │     no change │
│ QQuery 7     │ 197.87ms │            198.20ms │     no change │
│ QQuery 8     │  73.50ms │             76.23ms │     no change │
│ QQuery 9     │ 145.04ms │            144.91ms │     no change │
│ QQuery 10    │  94.58ms │             93.05ms │     no change │
│ QQuery 11    │  41.47ms │             42.64ms │     no change │
│ QQuery 12    │  68.41ms │             68.07ms │     no change │
│ QQuery 13    │ 103.76ms │            104.87ms │     no change │
│ QQuery 14    │  13.44ms │             13.35ms │     no change │
│ QQuery 15    │  15.91ms │             15.26ms │     no change │
│ QQuery 16    │  41.70ms │             39.29ms │ +1.06x faster │
│ QQuery 17    │ 110.51ms │            191.93ms │  1.74x slower │
│ QQuery 18    │ 291.45ms │            314.17ms │  1.08x slower │
│ QQuery 19    │  58.09ms │             59.28ms │     no change │
│ QQuery 20    │  81.24ms │             72.96ms │ +1.11x faster │
│ QQuery 21    │ 267.58ms │            262.45ms │     no change │
│ QQuery 22    │  29.58ms │             29.95ms │     no change │
└──────────────┴──────────┴─────────────────────┴───────────────┘

Doing that reduces the memory usage, but often with higher cost, which can be seen in the benchmark:

Maybe we can get the performance back somehow (like make the output creation faster somehow) 🤔
Alternately, we could consider making a single group operator that does the two phase grouping within itself
so instead of
group by (final)
  repartition
    group by (initil)

We would have
group by

And do the repartitioning within the operator itself (and thus if the first phase isn't helping, we can switch to the second phase)
This might impact downstream projects like ballista that want to distribute the first phase, however 🤔


About QNX Network

Hi, xtang,
Appreciate for the article you shared.  Will there be an introduction about the network ?

Anything particular about the network? :-)


[test] Remove lc_ctrl fatal alerts from chip_sw_alert_handler_lpg_sleep_…

…mode_alerts
lc_ctrl fatal alerts prevent SW from continuing and require the
alert_handler to issue a reset. Add those to the list of exceptions for
the vseq.
Fixes #18053

Thanks for the help with the alert_handler @matutem !

Oops, this apparently runs counter to the test plan, which explicitly says the alert_handler should not be triggered to reset. Maybe need a different plan (i.e. we have a missing exception?)

Change to omit lc_ctrl fatal alerts from consideration by the vseq.

Alex, can I ask you to edit https://cs.opensource.google/opentitan/opentitan/+/master:hw/top_earlgrey/data/chip_testplan.hjson;drc=45d97270936056f063a97ebbc53b740ba4ea5716;l=1510 indicating flash_ctrl, sram_ctrl, and lc_ctrl are excepted because alerts cause them to disable the CPU, and need a reset for the system to continue to function.

Alex, can I ask you to edit https://cs.opensource.google/opentitan/opentitan/+/master:hw/top_earlgrey/data/chip_testplan.hjson;drc=45d97270936056f063a97ebbc53b740ba4ea5716;l=1510 indicating flash_ctrl, sram_ctrl, and lc_ctrl are excepted because alerts cause them to disable the CPU, and need a reset for the system to continue to function.

Absolutely, will do


(OpenCriticMetadata) Playnite to access OpenCritic Scores for Metadata

Extension name
OpenCriticMetadata
Feature Description
I see that you seem to have already programmed something that allows Playnite to access OpenCritic Scores as metadata source -- I just thought maybe you might please find time to compile it and create a release package of the extension  --- or better yet, perhaps even add it to the addon browser inside of Playnite --- many thanks in advance! : https://github.com/darklinkpower/PlayniteExtensionsCollection/tree/master/source/Metadata/OpenCriticMetadata
Screenshots
No response

Hi, I don't plan to release this extension publicly but please send me a private message on reddit to talk about this.


HADAR细节询问

Hi,Dr.Bao:Thank you for your contributions in the infrared field, I am very interested in your work, I tried to reproduce, but vmap has been poor performance, so I would like to know how your vmap is calculated? Thank you very much!

Hello,
Thanks for your interest in our work.
Please check Sec.SIII of the supple. info. You can find more details on how V and X are solved out there.


Plotting color instead of by var in table and alpha not respected

In the aggregation notebook in spatialdata-notebooks we color shapes at some point by aggregated points of gene b. However, it seems b is interpreted as color blue and the values in the table are ignored for the fill color:

Also one can see that the alpha is ignored. This is also the case currently in the notebook version in the scverse docs.

Of note, I am testing the multi table branch of spatialdata but this did not have an effect on what the table looks like as far as I can tell.

Particularly this part of the code in render_shapes is the problem:
        elif colors.is_color_like(color):
            logger.info("Value for parameter 'color' appears to be a color, using it as such.")
            color = color
            col_for_color = None

Ultimately it would only be the case if genes were called like this so I think we should not fix it here, but rather change the names of genes in the example.


docs: announcing v0.2

Blog post for v0.2. Hoping to release very soon!

https://deploy-preview-85--anywidget.netlify.app/blog/anywidget-02/


Hebrew

Hello,
How can I make this library work with Hebrew ?
(I would like to match a sound from Hebrew alphabet to Russian/French/English alphabet)
Thank you

Something along a transliteration algorithm, for example?


[API integration] Add Jamendo provider API script

Fixes #51
This PR adds the script to pull the audio data from Jamendo API. This is the first audio provider integration to Openverse.
There are two 'audio_url' fields returned by Jamendo API: one for streaming and one for downloading.
"audio": "https://mp3d.jamendo.com/?trackid=168&format=mp32&from=37t85X61BkFJQKB4JnaEcg%3D%3D%7CjxH6jKzRNIYQdqbHcm2mpg%3D%3D",
"audiodownload": "https://mp3d.jamendo.com/download/track/168/mp32/",

When the download for audio is allowed("audiodownload_allowed": true), the audiodownload field is saved as the audio url.

Can't wait to run this locally and test


Generate a random tree

Build a random tree based on the passed data:

ensure the tree does not exceed max_depth
generate decision nodes and leaves

Implemented in 517a4c9d1b6915815f3c453270e60f92dbafab69.


Migrate article: Log integration with Fluentd

Migrate the content from this article to fit the documentation portal format https://help.aiven.io/en/articles/3660090
(Assign this issue to yourself before you start, and remember to mention the issue number when you open your pull request)

Hi @fmanimali are you planning to work on this? This issue is part of the high-priority articles Epic that we're aiming to complete by August. Thanks.


Lemur not adding OCSP and CRL url's to issued certificates

Hi guys.
Does anyone have an issue with this? Plugin used is CFSSL. Everything else is fine. Intermediate certificate does have OCSP and CRL url's in attributes.
It seems like lemur is not sending correct profile to cfssl. If i am correct, it should send profile selected  in gui.
config.json is:
{
  "signing": {
    "default": {
         "ocsp_url": "http://ocsp.domain.com:8889",
         "crl_url": "https://crl.ddomain.com/crl",
         "expiry": "8760h",
         "usages": [
             "signing",
             "key encipherment",
             "client auth"
         ]
    },
    "profiles": {
      "intermediate": {
        "usages": [
            "signing",
            "digital signature",
            "key encipherment",
            "cert sign",
            "crl sign",
            "server auth",
            "client auth"
        ],
        "expiry": "8760h",
        "ca_constraint": {
            "is_ca": true,
            "max_path_len": 0,
            "max_path_len_zero": true
        }
      },
      "peer": {
        "usages": [
            "signing",
            "digital signature",
            "key encipherment",
            "client auth",
            "server auth"
        ],
        "expiry": "8760h"
      },
      "ocsp": {
        "usages": [
          "digital signature",
          "ocsp signing"
        ],
        "expiry": "8760h"
      },
      "server": {
        "usages": [
          "signing",
          "digital signature",
          "key encipherment",
          "server auth"
        ],
        "expiry": "8760h"
      },
      "client": {
        "usages": [
          "signing",
          "digital signature",
          "key encipherment",
          "client auth"
        ],
        "expiry": "8760h"
     }
    }
  }
}

Hello people of github :) anybody home? :))))))

Hi @netcom-rs,
CFSSL plugin is a community contribution, which I am not familiar with.
Glancing at the code, looks like the certificate request data mostly consists of the CSR. You might need to add additional metadata to indicate OCSP or CRL, or the algorithms to be used, similar to how the DigiCert request is being constructed.
Also often, you can configure your profile with the CA, and just send the profile_name in your request, similar to here
Happy to review a PR, however, we don't have the option to test it.

@hosseinsh I agree with what you said.
Csr is constructed in lemur by selecting either predefined profile or manually ticking x509 extensions. After that, csr is sent to cfssl for signing. The certificate then comes back from cfssl into lemur database. Either the csr is not constructed well on lemur side(with wrong x509 attributes) or the cfssl is not properly signing.
The question is, how cfssl knows which profile to select? Or does it use default signing profile?

The default is currently used, but there's a feature request out for supporting multiple profiles. Closing this as a duplicate of https://github.com/Netflix/lemur/issues/3950


localForage.hasItem

Can we have a hasItem function to check whether a key is present?
getItem(...) === undefined doesn’t work if the key is present but the value is undefined

Hey there!
You can use the .key() method.
As a sidenote, the resolved value of a .getItem() promise can't be undefined, but returns null instead.


sh: yarn command not found

Hi guys,
Got the following error when tried to install moment-range when no yarn installed:

Maybe use npm in package.json?

hi @killmenot, what command did you run to get that output?

@gf3 just npm install (yarn is not installed)

@killmenot i've got a PR up to fix the issue—should be taken care of soon! thank you

@gf3 thx!


Move authentication configuration to authselect in place of authconfig

Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1707537
In RHEL8/CentOS8 authconfig is being replaced by authselect starting.
This PR updates the miqldap_to_sssd config tool to use authselect when available an to revert back to authconfig otherwise. This approach enables the miqldap_to_sssd tool to be agnostic to the OS upgrade to RHEL8/CentOS8 and will do the correct thing regaring authconfig/authselect on RHEL8/CentOS8 and earlier.
The approach was to:
rename the AuthConfig class to AuthEstablish
adding logic to AuthEstablish to invoke authselect when available and authconfig otherwise.
Add class AuthTemplateFiles to be shared by ConfigureApache and SssdConf as both now deal with configuration files and .erbs
Add SssdConf#create method to create the sssd.conf file as seeded by the sssd.conf.erb made available in a separate PR (see below)
Links [Optional]
PR to add TEMPLATE/x/sssd.conf.erb
Steps for Testing/QA [Optional]
Configure two appliances. One built on RHEL8 and the other on RHEL7
Configure each for Authentication mode LDAP/Get Groups From LDAP.
Run the miqldap_to_sssd conversion tool
Run the following command on the RHEL8 based test VM

grep authselect /var/www/miq/vmdb/log/miqldap_to_sssd.log

The output should look similar to:

DEBUG: Ran command: authselect select sssd --force

Run the following command on the RHEL7 based test VM

grep authconfig /var/www/miq/vmdb/log/miqldap_to_sssd.log

The output should look similar to:

DEBUG: Ran command: authconfig <params... bla bla bla>

@miq-bot add_label enhancement
@miq-bot add_label bug
@miq-bot add_label authentication
@miq-bot add_label hammer/no
@miq-bot assign @abellotti

This looks good @jvlcek, thanks for fixing this 👍


PhysicalBone and BoneAttachement churn tscn file when saving with animation

Godot version
3.4.4.stable.arch_linux
System information
Linux 5.16.16-arch1-1
Issue description
Similar to #53407, saving a scene with a PhysicalBone or BoneAttachment parented to an animated Skeleton will produce a diff in the .tscn every time, which creates unnecessary churn for version control.
Steps to reproduce

Open the attached project
Run git status, verify there are no changes
Save the Mage.tscn scene without changing anything
Run git status, see the scene changed

Minimal reproduction project
example.zip

I just tried this in 4.0, and even with the animation disabled, the scene is dirtied by tiny floating point changes each save, e.g.
 [node name="Physical Bone UpperLegL" type="PhysicalBone3D" parent="Armature/Skeleton3D" index="16"]
-transform = Transform3D(0.730141, 0.567688, 0.382148, -0.0225362, -0.540021, 0.842251, 0.682897, -0.624777, -0.38079, -0.0418361, 0.85965, 0.246105)
+transform = Transform3D(0.730141, 0.567757, 0.382148, -0.0225362, -0.540051, 0.842251, 0.682897, -0.624844, -0.38079, -0.0418361, 0.85965, 0.246105)
 joint_type = 1
-joint_offset = Transform3D(1, 0, 0, 0, 1, 0, 0, 0, 1, 1.87078e-07, 4.91102e-06, 0.252345)
-body_offset = Transform3D(0.999981, 5.81145e-06, -3.57628e-07, -5.96046e-07, -1.75834e-05, -0.999973, -1.84774e-06, 1.00005, -5.66244e-07, -9.68575e-08, 0.252338, -4.76837e-06)
+joint_offset = Transform3D(1, 0, 0, 0, 1, 0, 0, 0, 1, 1.87051e-07, 4.91055e-06, 0.252345)
+body_offset = Transform3D(0.999981, 1.13547e-05, -3.57628e-07, -5.96046e-07, -4.41521e-05, -0.999973, -1.84774e-06, 1.00015, -5.66244e-07, -9.68575e-08, 0.252338, -4.76837e-06)

I can confirm this bug is still present in 4.0.2
It does not seem related to animations though.
This are my steps to reproduce:

Imported a model with skeleton and no animations (characterMedium.fbx) and created a physical skeleton
Created an inherited scene (character_medium_1.tscn) and saved it
Created a duplicate of previous scene (character_medium_2.tscn)
Open and save character_medium_1.tscn without any further edit

character_medium_1.tscn is now different from character_medium_2.tscn (all physical bones transforms and offset are different)
godot-physbone-churn.zip


add max token length validator for Cohere; add truncate to default_options

add max token length validator for Cohere
add truncate to default_options

During the test, Cohere tokenize API  returns an input with token length 252. The model command max token length is 4096, so I set the max_tokens to 4096 - 252 = 3844. But the generate's return has 3846 token length, which exceeds the max allowed token length and throws an exception. So, I add the default truncate option to START.

hey @jinshen1983, thanks for putting this together. Could you also inherit from BaseValidator and re-use the logic implemented there.

@jinshen1983 I went ahead and bumped up the cohere gem for you with your fix: https://github.com/andreibondarev/langchainrb/pull/201


Leak checks

Perform leak checks of the new code with CodeGuard. At lease, in the volume of the common code use.

Several leaks were sealed


Upgrade node-fetch to version 3

node-fetch has the new major version 3: https://www.npmjs.com/package/node-fetch

node-fetch v3 drops support for ESM, see https://github.com/node-fetch/node-fetch/blob/main/docs/v3-UPGRADE-GUIDE.md#converted-to-es-module
That means we have to stay with v2 for the moment.