Blas dependency (updated)

supersedes https://github.com/Homebrew/linuxbrew/pull/511
I think we need some testing on linux before putting it to Homebrew.
@Rombur @dpo @tjhei @jppelteret could you please test it on your environment (perhaps on a fresh linuxbew)?
Below is the detailed instruction to make it work with CentOS cluster and native openmpi, gcc, cmake, git, MKL.
1.  For openblas on Linuxbrew one has to install it first before Step 9. (last one) and then add HOMEBREW_BLASLAPACK_XXX with correct values.
2. Get linuxbrew from this PR
git clone https://github.com/davydden/linuxbrew.git $HOMEBREW_PREFIX
cd $HOMEBREW_PREFIX
git checkout blas-dependency

3. Compile ruby if needed
Mine was too old, thus:
# compile ruby
wget https://cache.ruby-lang.org/pub/ruby/2.2/ruby-2.2.3.tar.gz
tar xvzf ruby-2.2.3.tar.gz ruby-2.2.3
./configure --prefix=$HOMEBREW_PREFIX
make
make install

4. Make simlinks for GCC to make sure linuxbrew picks it up
ln -s `which gcc` $HOMEBREW_PREFIX/bin/gcc-`gcc -dumpversion |cut -d. -f1,2`
ln -s `which g++` $HOMEBREW_PREFIX/bin/g++-`g++ -dumpversion |cut -d. -f1,2`
ln -s `which gfortran` $HOMEBREW_PREFIX/bin/gfortran-`gfortran -dumpversion |cut -d. -f1,2`

5. Make simlinks for Cmake to avoid its installation in Linuxbrew
mkdir -p $HOMEBREW_PREFIX/Cellar/cmake/HEAD/bin
ln -s `which cmake` $HOMEBREW_PREFIX/Cellar/cmake/HEAD/bin/
brew link cmake

6. Wrap mpirun and mpiexec if needed
On a cluster frontend i was not able to execute mpirun without extra flags, thus
## make MPI runnable on frontend with shared memory
cd $HOMEBREW_PREFIX
mkdir bin_temp
cd bin_temp
touch mpiexec

# put the following to the file (adjust path to mpiexec as needed):

#!/bin/bash
exec /apps/OpenMPI/1.8.3-gcc/bin/mpiexec --mca btl self,sm "$@"

# make the file executable:
chmod +x mpiexec

# do the same steps for mpirun

7. Get openssl
brew install openssl
brew postinstall openssl

8. Get modified homebrew-science
i updated formulae used below for the new :blas dependency
brew tap davydden/science
cd $HOMEBREW_PREFIX/Library/Taps/davydden/homebrew-science
# branch name is misleading, it's not MKL only! At least it should not be :-)
git checkout mkl_dealii

9. Install minimal deal.II suite
# to make metis tests run:
export LD_LIBRARY_PATH_OLD=$LD_LIBRARY_PATH
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$HOMEBREW_PREFIX/lib"

# in case you wrap mpirun and mpiexec
export PATH_OLD=$PATH
export PATH="$HOMEBREW_PREFIX/bin_temp:$PATH"

# finally install
brew install hypre --with-mpi
brew install metis
brew test metis
brew install parmetis
brew test parmetis
brew install superlu_dist
brew test superlu_dist

# i installed without check because it tries to run mpiexec directly,
# no the wrapper i created => failures on front end of the cluster
brew install scalapack --without-check
brew install mumps
brew test mumps

brew install petsc --without-fftw --without-hdf5 --without-hwloc --without-netcdf --without-suite-sparse --without-sundials --without-superlu
brew install slepc --without-arpack
brew install p4est
brew install trilinos --without-x11 --without-adol-c --without-cppunit --without-doxygen --without-hwloc --without-libmatio --without-mumps --without-netcdf --without-scalapack --without-scotch --without-suite-sparse --without-eigen --without-glpk --without-hdf5 --without-tbb --without-glm --without-yaml-cpp

# don't know why but we need to remove Homebrew from LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH_OLD

# install deal.II
brew install dealii --HEAD --without-arpack --without-hdf5 --without-muparser --without-netcdf --without-opencascade --without-suite-sparse --without-tbb

Thinking about this a little, I think the environment variable is an implementation detail that the formula shouldn't have to rely on. Encapsulating that logic in a new Homebrew::BlasLapack class somewhere is probably better engineering.
I like the simpler configuration of BLASLAPACK_LDFLAGS and BLASLAPACK_CFLAGS; we could add helper methods that extract the library names and paths for the picky formulas that need them, maybe?

I would prefer to stick with the current input (names/lib/inc), and provide extra methods to combine those into linker flags and compiler flags.
The reason is one coud follow the MKL Linker Advisor and write something like '-L/path/to/mkl -lmkl_core -lmkl_sequential -lpthread -lm' but this does not mean that 'libpthread' and 'libm' exist in '/path/to/mkl'. Asa matter of fact they dont. So there is more confusion possible.
Doing 3-argument-input makes it more rigorous, so advanced user should make sure those libraries exist in directory 'lib'.
@tdsmith could you briefly advice on how to introduce those helper functions?

Ps. For convinience of discussion I will give links to usage in formulae tomorrow, when i browse from my mac (currently on a phone).

but this does not mean that 'libpthread' and 'libm' exist in '/path/to/mkl'.
Does it matter?

I'd have to experiment; I don't have good advice about an implementation. I'm at a conference for the rest of the week so I may not have time to play with this. @xu-cheng is a much better programmer than I am and might have some thoughts if he thinks this is interesting.

Thanks for pushing this forward; I appreciate it.

Sorry, I'm kinda losing context here. What's the problem here?
I think the right course is putting the logic to alter *flags in env block. You can see JavaRequirement for example. However, there is a problem regarding Linuxbrew which is running under Stdenv rather than Superenv. As result all of env blocks will never be called.

but this does not mean that 'libpthread' and 'libm' exist in '/path/to/mkl'.
Does it matter?

it does for cases when we need library names or full path (see summary of :blas usage in the first post).

@xu-cheng Please, have a quick look at the usage section in the OP. The goal would be to get helper methods/functions (how/where?) which use NAMES, LIB to create

ldflags
full path to libraries (with needed separation: " " or ";").

I already do that in each single formula i modded, but what @tdsmith was suggesting, as far as I understand, is to create some helper functions to do the job. And then use them in formulae to make things prettier / more neat.

i added a check for single precision Fortran blas in order to encapsulate veclibfort inside :blas => :fortran. On linux works as expected:
without BLAS:
$ unset HOMEBREW_BLASLAPACK_NAMES
$ unset HOMEBREW_BLASLAPACK_LIB
$ unset HOMEBREW_BLASLAPACK_INC
$ brew install blastest_single
Warning: BLAS not configured
Falling back to brewed openblas. If you prefer to use a system BLAS, please set
  HOMEBREW_BLASLAPACK_NAMES (e.g. "mkl_intel_lp64;mkl_sequential;mkl_core")
  HOMEBREW_BLASLAPACK_LIB   (e.g. "${MKLROOT}/lib/intel64")
  HOMEBREW_BLASLAPACK_INC   (e.g. "${MKLROOT}/include")
to correct values.
==> Installing dependencies for blastest_single: openblas
==> Installing blastest_single dependency: openblas
==> Building with an alternative Fortran compiler
This is unsupported.
Warning: No Fortran optimization information was provided.  You may want to consider
setting FCFLAGS and FFLAGS or pass the `--default-fortran-flags` option to
`brew install` if your compiler is compatible with GCC.

If you like the default optimization level of your compiler, ignore this
warning.
==> Downloading https://github.com/xianyi/OpenBLAS/archive/v0.2.14.tar.gz
==> Downloading from https://codeload.github.com/xianyi/OpenBLAS/tar.gz/v0.2.14
######################################################################## 100.0%
==> make FC=mpif90 libs netlib shared
==> make FC=mpif90 tests
==> make PREFIX=/home/woody/iwtm/iwtm108/.linuxbrew/Cellar/openblas/0.2.14_1 install
/home/woody/iwtm/iwtm108/.linuxbrew/Cellar/openblas/0.2.14_1: 14 files, 51M, built in 64 seconds
==> Installing blastest_single
Warning: BLAS not configured
Falling back to brewed openblas. If you prefer to use a system BLAS, please set
  HOMEBREW_BLASLAPACK_NAMES (e.g. "mkl_intel_lp64;mkl_sequential;mkl_core")
  HOMEBREW_BLASLAPACK_LIB   (e.g. "${MKLROOT}/lib/intel64")
  HOMEBREW_BLASLAPACK_INC   (e.g. "${MKLROOT}/include")
to correct values.
==> Downloading https://gist.githubusercontent.com/davydden/1f9ebf3692beca2438f8/raw/0ba39c8775db3bb21c09c3440bf78ff5f778277d/blas.f90
Already downloaded: /home/woody/iwtm/iwtm108/Cache/HOMEBREW/blastest_single-1.0.f90
==> mpif90 blas.f90 -I/home/woody/iwtm/iwtm108/.linuxbrew/opt/openblas/include -o blastest -L/home/woody/iwtm/iwtm108/.linuxbrew/opt/openblas/lib -lopenblas -pthread -lm
/home/woody/iwtm/iwtm108/.linuxbrew/Cellar/blastest_single/1.0: 2 files, 16K, built in 2 seconds
$ brew test blastest_single
Testing blastest_single
==> /home/woody/iwtm/iwtm108/.linuxbrew/Cellar/blastest_single/1.0/bin/blastest

with MKL:
$ brew install blastest_single
==> Downloading https://gist.githubusercontent.com/davydden/1f9ebf3692beca2438f8/raw/0ba39c8775db3bb21c09c3440bf78ff5f778277d/blas.f90
Already downloaded: /home/woody/iwtm/iwtm108/Cache/HOMEBREW/blastest_single-1.0.f90
==> mpif90 blas.f90 -I/apps/intel/ComposerXE2013/composer_xe_2013.5.192/mkl/include -o blastest -L/apps/intel/ComposerXE2013/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -pthread -lm
/home/woody/iwtm/iwtm108/.linuxbrew/Cellar/blastest_single/1.0: 2 files, 16K, built in 2 seconds
$ brew test blastest_single
Testing blastest_single
==> /home/woody/iwtm/iwtm108/.linuxbrew/Cellar/blastest_single/1.0/bin/blastest

test on mac
$ brew install blastest
==> Downloading https://gist.githubusercontent.com/tdsmith/345b08bb656ad54c1701/raw/c0d4e2b1362d43dad0760bcc1ec24e4b881683ef/blastest.c
######################################################################## 100.0%
==> clang blastest.c  -o blastest -lblas -llapack -pthread -lm
🍺  /usr/local/Cellar/blastest/1.0: 2 files, 20K, built in 2 seconds
h4-dyn-05:Downloads davydden$ brew test blastest
Testing blastest
==> /usr/local/Cellar/blastest/1.0/bin/blastest

for fortran part it something seems to be strange on OS-X
$ brew install blastest_single -v
Warning: BLAS not configured
Falling back to brewed openblas. If you prefer to use a system BLAS, please set
  HOMEBREW_BLASLAPACK_NAMES (e.g. "mkl_intel_lp64;mkl_sequential;mkl_core")
  HOMEBREW_BLASLAPACK_LIB   (e.g. "${MKLROOT}/lib/intel64")
  HOMEBREW_BLASLAPACK_INC   (e.g. "${MKLROOT}/include")
to correct values.
==> Installing dependencies for blastest_single: veclibfort
rm /usr/local/lib/libvecLibFort.a
rm /usr/local/lib/libvecLibFort.dylib
rm /usr/local/lib/libvecLibFortI.dylib
==> Installing blastest_single dependency: veclibfort
==> Using Homebrew-provided fortran compiler.
This may be changed by setting the FC environment variable.
==> Downloading https://github.com/mcg1969/vecLibFort/archive/0.4.2.tar.gz
Already downloaded: /Library/Caches/Homebrew/veclibfort-0.4.2.tar.gz
==> Verifying veclibfort-0.4.2.tar.gz checksum
tar xf /Library/Caches/Homebrew/veclibfort-0.4.2.tar.gz
==> make all
clang -O   -c -o vecLibFort.o vecLibFort.c
clang -shared -O -DVECLIBFORT_INTERPOSE -o libvecLibFortI.dylib -O vecLibFort.c \
		-Wl,-reexport_framework -Wl,Accelerate \
		-install_name /usr/local/lib/libvecLibFortI.dylib
ar -cru libvecLibFort.a vecLibFort.o
clang -shared -o libvecLibFort.dylib vecLibFort.o \
		-Wl,-reexport_framework -Wl,Accelerate \
		-install_name /usr/local/lib/libvecLibFort.dylib
ranlib libvecLibFort.a
==> make check
gfortran -o tester -O tester.f90 vecLibFort.o -framework Accelerate
./tester
 If the return value interface is fixed, none of these values will
 be zero, nor will they be nonsensically large or small. On the
 other hand, if the translation is incorrect, it is more likely
 that this program will carsh.

   62.0000000       64.0000000       8.00000000       24.0000000
 (  34.0000000    ,  170.000000    ) (  154.000000    , -70.0000000    )
   14.2828569       36.0000000
 ( -12.000000000000000     ,  92.000000000000000     )
 (  68.000000000000000     , -36.000000000000000     )
   13.0384054       20.1990089       9.43398094       10.5830050

 These are the machine constants generated by SLAMCH. We expect
 some of them to be small (E-08,E-38).

   5.96046448E-08   1.17549435E-38   2.00000000
   1.19209290E-07   1.00000000      -125.000000
   1.17549435E-38   128.000000       3.40282347E+38
==> make PREFIX=/usr/local/Cellar/veclibfort/0.4.2_2 install
mkdir -p /usr/local/Cellar/veclibfort/0.4.2_2/lib
cp -f libvecLibFort.a /usr/local/Cellar/veclibfort/0.4.2_2/lib
cp -f libvecLibFort.dylib /usr/local/Cellar/veclibfort/0.4.2_2/lib
cp -f libvecLibFortI.dylib /usr/local/Cellar/veclibfort/0.4.2_2/lib
==> Cleaning
==> Finishing up
ln -s ../Cellar/veclibfort/0.4.2_2/lib/libvecLibFort.a libvecLibFort.a
ln -s ../Cellar/veclibfort/0.4.2_2/lib/libvecLibFort.dylib libvecLibFort.dylib
ln -s ../Cellar/veclibfort/0.4.2_2/lib/libvecLibFortI.dylib libvecLibFortI.dylib
==> Caveats
Installs the following files:
  * libvecLibFort.a: static library; link with -framework vecLib
  * libvecLibFort.dylib: dynamic library; *replaces* -framework vecLib
  * libvecLibFortI.dylib: preload (interpose) library.
Please see the home page for usage details.
==> Summary
🍺  /usr/local/Cellar/veclibfort/0.4.2_2: 6 files, 100K, built in 2 seconds
==> Installing blastest_single
Warning: BLAS not configured
Falling back to brewed openblas. If you prefer to use a system BLAS, please set
  HOMEBREW_BLASLAPACK_NAMES (e.g. "mkl_intel_lp64;mkl_sequential;mkl_core")
  HOMEBREW_BLASLAPACK_LIB   (e.g. "${MKLROOT}/lib/intel64")
  HOMEBREW_BLASLAPACK_INC   (e.g. "${MKLROOT}/include")
to correct values.
==> Using Homebrew-provided fortran compiler.
This may be changed by setting the FC environment variable.
==> Downloading https://gist.githubusercontent.com/davydden/1f9ebf3692beca2438f8/raw/0ba39c8775db3bb21c09c3440bf78ff5f778277d/blas.f90
Already downloaded: /Library/Caches/Homebrew/blastest_single-1.0.f90
==> Verifying blastest_single-1.0.f90 checksum
==> /usr/local/bin/gfortran blas.f90 -I/usr/local/opt/veclibfort/include -o blastest -L/usr/local/opt/veclibfort/lib -lveclibfort -pthread -lm
f951: Warning: Nonexistent include directory '/usr/local/opt/veclibfort/include' [-Wmissing-include-dirs]
==> Cleaning
==> Finishing up
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/blastest
Target /usr/local/bin/blastest
is a symlink belonging to blastest. You can unlink it:
  brew unlink blastest

To force the link and overwrite all conflicting files:
  brew link --overwrite blastest_single

To list all files that would be deleted:
  brew link --overwrite --dry-run blastest_single

Possible conflicting files are:
/usr/local/bin/blastest -> /usr/local/Cellar/blastest/1.0/bin/blastest
==> Summary
🍺  /usr/local/Cellar/blastest_single/1.0: 2 files, 24K, built in 2 seconds
h4-dyn-05:Downloads davydden$ brew test blastest_single
Testing blastest_single
==> /usr/local/Cellar/blastest_single/1.0/bin/blastest

i would speculate that #{ENV["FC"]} is not set inside blas_requirement.rb. Can one somehow require :fortran inside?

i delete two previous messages as i fixed the issue (and update the last commit). So here's the output on the mac:
with openblas
$ export HOMEBREW_BLASLAPACK_NAMES="openblas"
$ export HOMEBREW_BLASLAPACK_LIB="/usr/local/opt/openblas/lib/"
$ export HOMEBREW_BLASLAPACK_INC="/usr/local/opt/openblas/include/"
$ $ brew install blastest_single
==> Using Homebrew-provided fortran compiler.
This may be changed by setting the FC environment variable.
==> Downloading https://gist.githubusercontent.com/davydden/1f9ebf3692beca2438f8/raw/0ba39c8775db3bb21c09c3440bf78ff5f778277d/blas.f90
Already downloaded: /Library/Caches/Homebrew/blastest_single-1.0.f90
==> /usr/local/bin/gfortran blas.f90 -I/usr/local/opt/openblas/include/ -o blastest_single -L/usr/local/opt/openblas/lib/ -lopenblas -pthread -lm
🍺  /usr/local/Cellar/blastest_single/1.0: 2 files, 20K, built in 2 seconds
$ brew test blastest_single
Testing blastest_single
==> /usr/local/Cellar/blastest_single/1.0/bin/blastest_single

with -framework Accelerate (=> need veclibfort)
$ unset HOMEBREW_BLASLAPACK_NAMES
$ unset HOMEBREW_BLASLAPACK_LIB
$ unset HOMEBREW_BLASLAPACK_INC
$ brew install blastest_single
Warning: BLAS not configured
Falling back to brewed openblas. If you prefer to use a system BLAS, please set
  HOMEBREW_BLASLAPACK_NAMES (e.g. "mkl_intel_lp64;mkl_sequential;mkl_core")
  HOMEBREW_BLASLAPACK_LIB   (e.g. "${MKLROOT}/lib/intel64")
  HOMEBREW_BLASLAPACK_INC   (e.g. "${MKLROOT}/include")
to correct values.
==> Installing dependencies for blastest_single: veclibfort
==> Installing blastest_single dependency: veclibfort
==> Using Homebrew-provided fortran compiler.
This may be changed by setting the FC environment variable.
==> Downloading https://github.com/mcg1969/vecLibFort/archive/0.4.2.tar.gz
Already downloaded: /Library/Caches/Homebrew/veclibfort-0.4.2.tar.gz
==> make all
==> make check
==> make PREFIX=/usr/local/Cellar/veclibfort/0.4.2_2 install
==> Caveats
Installs the following files:
  * libvecLibFort.a: static library; link with -framework vecLib
  * libvecLibFort.dylib: dynamic library; *replaces* -framework vecLib
  * libvecLibFortI.dylib: preload (interpose) library.
Please see the home page for usage details.
==> Summary
🍺  /usr/local/Cellar/veclibfort/0.4.2_2: 6 files, 100K, built in 2 seconds
==> Installing blastest_single
Warning: BLAS not configured
Falling back to brewed openblas. If you prefer to use a system BLAS, please set
  HOMEBREW_BLASLAPACK_NAMES (e.g. "mkl_intel_lp64;mkl_sequential;mkl_core")
  HOMEBREW_BLASLAPACK_LIB   (e.g. "${MKLROOT}/lib/intel64")
  HOMEBREW_BLASLAPACK_INC   (e.g. "${MKLROOT}/include")
to correct values.
==> Using Homebrew-provided fortran compiler.
This may be changed by setting the FC environment variable.
==> Downloading https://gist.githubusercontent.com/davydden/1f9ebf3692beca2438f8/raw/0ba39c8775db3bb21c09c3440bf78ff5f778277d/blas.f90
Already downloaded: /Library/Caches/Homebrew/blastest_single-1.0.f90
==> /usr/local/bin/gfortran blas.f90 -I/usr/local/opt/veclibfort/include -o blastest_single -L/usr/local/opt/veclibfort/lib -lveclibfort -pthread -lm
🍺  /usr/local/Cellar/blastest_single/1.0: 2 files, 24K, built in 2 seconds
$ brew test blastest_single
Testing blastest_single
==> /usr/local/Cellar/blastest_single/1.0/bin/blastest_single

i added static functions to the class (have not updated homebrew-science yet, but the usage is clear from blastest_single and blastest). As far as I can tell, it works for me as intended both on OSX and Linux.
@tdsmith @dpo @xu-cheng  further wishes?

i updated the first post to include hdf5, seems to be working for me. Will update it further to install trilinos --with-hdf5 after i test it.

i updated homebrew-science branch to use static functions. Looks much cleaner now! Will be submitting a Homebrew PR today (will cherry-pick commits), if nobody has any further comments to this PR.

@davydden I tested this according to your instructions using Ubuntu 14.04 (LTS) in a virtual machine. I configured to separate installations, one using the system BLAS/LAPACK and one using OpenBLAS from linuxbrew. Apart from there being a couple of redundant steps (3,4,6) and it requiring a upgrade of gcc to 5.x in order to install parmetis, everything appears to have worked as expected!

@jppelteret Great! Thanks a lot, Jean-Paul!


Dependencies not works in langchain:sample

I tried to run container langchain:samples with
./run.sh $(./autotag langchain:samples)
As the description in Readme, the langchain:sample is based on:

Dependencies
build-essential cuda cudnn python tensorrt numpy cmake onnx pytorch huggingface_hub llama_cpp langchain rust jupyterlab

So there should be pytorch and llama_cpp and langchain in the container,
However...
in  langchain:sample
python3 -c 'import torch'                                                                                Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'

python3 -c 'import langchain'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'langchain'

but it works in langchain:r35.4.1
python3 -c 'import torch; print(f"PyTorch version: {torch.__version__}");'
PyTorch version: 2.0.0+nv23.05
python3 -c 'import langchain;print(f"LC version: {langchain.__version__}");'
LC version: 0.0.350

But langchain:sample is based on  langchain:r35.4.1, it's weird that  langchain:sample don't have pytorch and langchain package, not mention to llama_cpp package.
I cannot even successfully execute the default demo LangChain_Local-LLMs.ipynb
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[1], line 1
----> 1 from langchain.llms import LlamaCpp
      2 from langchain.callbacks.manager import CallbackManager
      3 from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler

ModuleNotFoundError: No module named 'langchain'

Thanks for pointing this out @smithlai, it should be fixed in https://github.com/dusty-nv/jetson-containers/commit/a637fc9eb8e169454890a0874aca617a81edbadc

OK, these are rebuilding now across all JP5/JP6 container versions and are being pushed to dockerhub 👍

Thank you dusty,
And there is one more issue there.
I tried to rebuild my container with ./build -n my_contatiner langchain or ./build -n my_contatiner llama_cpp
but always failed here
Step 17/17 : RUN pip3 show llama-cpp-python | grep llama &&     python3 -c 'import llama_cpp' &&     python3 -m llama_cpp.server --help
 ---> Running in 9e276c3c1db9
Name: llama_cpp_python
Summary: Python bindings for the llama.cpp library
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/llama_cpp/llama_cpp.py", line 74, in _load_shared_library
    return ctypes.CDLL(str(_lib_path), **cdll_args)
  File "/usr/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcuda.so.1: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/llama_cpp/__init__.py", line 1, in <module>
    from .llama_cpp import *
  File "/usr/local/lib/python3.8/dist-packages/llama_cpp/llama_cpp.py", line 87, in <module>
    _lib = _load_shared_library(_lib_base_name)
  File "/usr/local/lib/python3.8/dist-packages/llama_cpp/llama_cpp.py", line 76, in _load_shared_library
    raise RuntimeError(f"Failed to load shared library '{_lib_path}': {e}")
RuntimeError: Failed to load shared library '/usr/local/lib/python3.8/dist-packages/llama_cpp/libllama.so': libcuda.so.1: cannot open shared object file: No such file or directory

This platform is Jetson AGX Orin

Hi @smithlai, did you set your default docker runtime to nvidia?   This enables CUDA to be used during the builds.
https://github.com/dusty-nv/jetson-containers/blob/master/docs/setup.md#docker-default-runtime

Hi @smithlai, did you set your default docker runtime to nvidia? This enables CUDA to be used during the builds.
https://github.com/dusty-nv/jetson-containers/blob/master/docs/setup.md#docker-default-runtime

My apologies!
I did't noticed that link.
Now everything goes better now.


Count Aggregator Empty Cells

First off, this package is great!  I showed it to my boss yesterday, and he loved it.
My question is, is there a way to make the built-in Count Aggregator assign 0 to empty cells rather than just leaving them empty?  This behavior makes for some weird looking results in the C3 line chart, where lines start, then disappear, then start again.  I am guessing I could accomplish this with a custom aggregator, but I wonder if there's a built-in way to do this?
Thanks!

There's no built-in way to do this, but a small modification on the count aggregator should be easy. I'll think a bit about what the default should be...

Hello,
that's what i have

There's simple survey - one question, two answer options and objects. First two objects contains only one answer option, rest of them got both. When data pass through aggregator there is only 10 records, and i want to count it as 12. So, is there any way aggregator could resolve missing records as 0? Not just render in cells, but count them for totals? Thanks!

There's no way to count one row twice... values can only be scalars, not objects :)


ssh connection shut down prematurely

Here's my deployment log:
✔ Executing task deploy:prepare
✔ Executing task deploy:release
✔ Executing task deploy:update_code
✔ Executing task deploy:symlink
Executing task cleanup
Notice: Unable to open channel in phar:///usr/local/bin/dep/vendor/phpseclib/phpseclib/phpseclib/Net/SSH2.php on line 3255
✔otice: Connection closed prematurely in phar:///usr/local/bin/dep/vendor/phpseclib/phpseclib/phpseclib/Net/SSH2.php on line 3396
➤ Executing task success
This seems to be happening after several successful deployments. When I remove the releases dir on the server, it solves the problem, but as I do more and more deployments this problem always comes back.

Please show your version of Deployer, PHP and log from deploy command with debug option -vvv


Update about-client-settings.md

#MVPHackadoc2019
Added Allow Network Wake-up section under Power Management

@SweJorgen : Thanks for your contribution! The author, @aczechowski, has been notified to review your proposed change.

Thanks @SweJorgen for the contribution. I'm going to approve this change. I know @mestew has some updates to make for this feature, so she can adjust this as needed. But you can at least get your contribution included. 😃
#sign-off
Once merged, this should be live with the next daily publishing process. Right now that should be tomorrow (Tuesday, 26 March) around 11:00 Pacific time. No further action needed from you.

@ktoliver : Thanks for your contribution! The author, @aczechowski, has been notified to review your proposed change.


Possibility to mutate string nodes in JSX/TSX

Is your feature request related to a problem? Please describe.
I'd like Stryker to be able to test simple string nodes in JSX/TSX:

Hello there

This is a basic component

Here's the simple repo demonstrating that Stryker isn't covering the This is a basic component part: https://github.com/limonte/stryker-sandbox
The test only covers
expect(screen.getByText('Hello there')).toBeInTheDocument();

and it doesn't cover This is a basic component.
Despite that both jest and Stryker report 100% coverage while it's not true as This is a basic component isn't verified.

The expected on the screenshot above would be mutating the This is a basic component string and reporting that it's not covered with tests.

I think you are onto something here @limonte!
I agree that it seems like we are not fully asserting that all the code is as it should be. I think a lot of the javascript mutators could be applied in a similar way here.
Here are just a few potential jsx mutators I can think of:

delete each element

delete attribute of each element

replace attribute value of each element with different value

(for elements like  that take inner html), change inner html of each element

Also, I think these same mutators could be applied not just to jsx / tsx but more generally to mutation testing of any markup language (like Angular or regular html files).

Hi @limonte, thanks for opening an issue. Sorry for the late response, because of the summer holiday we haven't really been active with developing Stryker. Currently we support mutating Javascript statements inside JSX. For now we don't have plans to implement mutating HTML elements inside JSX in the near future as this is Stryker for Javascript and not Stryker for HTML. Though in the future we would like to add the possibility for StrykerJS users to create their own mutators just like in Stryker.NET. I Hope this answers your question

Hey @odinvanderlinden and thank you for the reply!
I'm not sure how familiar you are with React, but JSX/TSX isn't HTML. It's JS/TS that looks like HTML.
Here's a nice article that'll explain what I mean: https://legacy.reactjs.org/docs/jsx-in-depth.html

This is a basic component
 is JSX/TSX is actually JS that looks like:
React.createElement(
  'p',
  {},
  'This is a basic component'
)

It's a simple JS with 3rd parameter standing for children. It can be mutated.

Thanks for the fast reply, i'm familiar with JSX/TSX so i know how it works. If we are going to mutate strings inside babel we are basically testing if babel works correctly wich is something we shouldn't do i think. Also the strings we write inside these HTML like JSX elements are constants because they can't change. Except ofcourse if we write it in a javascript statement and we already mutate those. So in my eyes in the end we are still mutating something that is ment to be HTML.


Monoidal instead of Applicative

The http://github.com/non/cats project has been exploring favoring the just as powerful (Monoidal)[https://wiki.haskell.org/Typeclassopedia#Alternative_formulation] typeclass over Applicative.
Applicative can be cumbersome in languages that don't default to partial function application and I also think it has a more intuitive definition than Applicative since it's symmetrical as well as having much nicer laws (naturality, left and right identity, and associativity).
You can also get apply builders almost for free since map2, map3, map4 just become map on tuple2, tuple3, tuple4.

This is awesome and I would love to see it happen, but alongside our current Applicative.  Swift does not support tuple extensions, so any generality we could extract would have to be by writing 7 billion overloads again.

Yeah, there would probably have to be a ProductN boilerplate type like Zip2Sequence.
What I like about the ProductN route, though, is that it would only have to be used internally and would be transparent to the client. No longer will the client be required to define a curried initializer create.
Instead of Foo.create <^> fa <*> fb <*> fc we would have map(fa, fb, fc) { Foo($0, $1, $2) }, which looks more natural in Swift land IMO.

So, how much boilerplate is enough boilerplate then?

Which is to say, I definitely agree that that looks more natural, but am I gonna have to maintain 15 more structs in the near future?

Scala seemed to have made the decision for everyone that Tuple22 was enough 😃

Then to 22 we shall go!  Would you like to draw up a pull request, or shall I?

I might have time tonight to get something going. Would you want Applicative to be defined in terms of Monoidal, vice versa, or independent of one another?

Do whatever is most natural (which probably means separate at first).  We'll worry about the nitty-gritty in code review.

I've got Tuple to Tuple5 if that helps. Couldn't give up on shrinking in SwiftCheck...

Unrelated, but have you considered opening a gitter channel for this project? I think I remember a talk where @non mentioned that it contributed Cats growing community but maybe I made that part up.

I promoted the creation of a slack channel... but it's pretty lonely their at the mo. Will look into gitter.

What's nice about gitter is that it's discoverable. And you don't need a slack account.

right. slack is imperfect for unpaid use.

Having read through this three, I've just created https://gitter.im/typelift and invite @CodaFi.  Anyone else want in?

I'd love to give gitter a go.

Is it not going to be a public channel?

I guess we can make it public, but I don't know how to do that!

Isn't it just "Create Room" -> Repository -> Swiftz?

Maybe?  I'm on mobile right now, so it'll have to wait for a few hours.

Okay, https://gitter.im/typelift/general has just been created, and it's public!

Hurrah!
On Tue, 27 Oct 2015 at 00:46, Alexander Altman<EMAIL_ADDRESS>wrote:

Okay, https://gitter.im/typelift/general has just been created, and it's
public!
—
Reply to this email directly or view it on GitHub
https://github.com/typelift/Swiftz/issues/260#issuecomment-151327332.

Btw, regarding the need for the boilerplate curried constructor, in Swift 2 this now works:
struct Foo { let a: Int, let b: String }
curry(Foo.init) <^> 1 <*> "hello"

Of course you'll end up hitting the "expression too complex" pretty quickly...

Whatever happened to this proposal?  Should I draw up the pull request for this myself?


[recommend] GPT-Runner: Conversations with your files! Manage and run your AI presets!

Github: https://github.com/nicepkg/gpt-runner

Hi @2214962083! Thank you for your information proposal. I will add this information to the list next update. Also, I would like to add you as a contributor. Could you please send a pull request, using the next pull request as a reference? Please add the next line to the category you wish to add and send a pull request.
 * [Ultimate-ChatGPT](https://github.com/imabutahersiddik/Ultimate-ChatGPT) - The Ultimate ChatGPT is an AI-powered virtual assistant designed to provide an enhanced writing and problem-solving experience.

Reference:
https://github.com/taishi-i/awesome-ChatGPT-repositories/pull/20/files
If there's no need to add you as a contributor, I will add your repository, so no worries in that case. Thank you in advance.

@taishi-i
Of course no problem, I will submit a pr to add gpt-runner for each language, since my gpt-runner has management prompt and cli and web and vscode extension at the same time, I request to put it in the prompt, so that More people know about it, hope you agree. Thank you !

The pr is created, thanks to review it: https://github.com/taishi-i/awesome-ChatGPT-repositories/pull/24

Thank you for your merge ! I think this can be closed


Deogiri Institute of Engineering and Management Studies

@omcoolkarni22 I have to close this request without merging, because you requested to add a non-existent domain to this repository: Host dietms.asia not found: 3(NXDOMAIN) Please ensure that you are familiar with this repository rules located at the bottom of this page and submit a new request with the correct information. Thank you for your understanding!


Fix zoom snapping issue

Also just clean up some of the rendering stuff related to it from clipping changes

Fixed here: 40e50b443a7cd71ecd485309cb6aae22a609a056


UX: Inconsistent FAQ lists and hard-to-find answers

Where to find the issue
FAQ lists at:

https://www.coronawarn.app/de/faq/
https://www.bundesregierung.de/breg-de/themen/corona-warn-app/corona-warn-app-faq-1758392
https://www.bundesregierung.de/resource/blob/1753814/1760022/0c64afa282398b1b6255d43c7c3c4950/2020-06-12-faq-langfassung-de-data.pdf?download=1

(Linking to the German versions here.)
Describe the issue
Three different lists of frequently asked questions exist and differ in content. The CWA website contains one, the app (Android) links to a different one with fewer questions at bundesregierung.de, and a link points from there to a longer third one in a PDF document at bundesregierung.de. While there is some overlap between the three, they do not seem to constitute different representations or subsets of the same main document.
This makes it difficult to find answers contained in some of the FAQs but not others, to remember where to find certain information, or to find from the app all information a user may need.
Example: A new user asking themselves whether they may close the app after activation would most likely turn to the FAQ prominently advertised by the app (no. 2) but not find their question answered. They may or may not find from there the longer PDF document (no. 3), which also fails to answer their question. Only if they happen to check the FAQ at coronawarn.app (no. 1) they will find the answer to their question.
Suggested change
Maintain a single, consistent FAQ or include all questions of immediate relevance for using the app in whichever FAQ version the app links to.

Internal Tracking ID: EXPOSUREAPP-1877

Thanks for bringing that up, @sventuerpe. This is indeed an issue and we are actually working on a consolidation. Originally, it was planned to use the website and FAQ on https://www.coronawarn.app/en/faq/ as the version for the more tech-savvy folks and the one on bundesregierung.de as the one for the endusers, but well, plans need to be adopted to how things go in reality, right?
Thank you for your understanding that this might take a while as we are busy handling the incoming requests, but we will of course keep this issue open until we have a proper solution.
Mit freundlichen Grüßen/Best regards,
SW
Corona Warn-App Open Source Team

I wrote this in #303 - it got closed as a duplicate to this issue (which I don't think it is) but since I don't have the power to reopen #303 I will copy my reasoning/request from there in here:

Currently, the target of the FAQ link in the apps is the Bundesregierung's FAQs at: https://www.bundesregierung.de/breg-de/themen/corona-warn-app/corona-warn-app-faq-1758392
I consider this subideal, because this page is:

noisy
not very well designed
not under direct project control
doesn't include the detail of the project FAQs
doesn't answer many important questions

It would be better, if the FAQ link in the apps targeted the project website FAQs at coronawarn.app/en/faq/ (and German equivalent). These FAQs are cleaner, more up to date, under project control etc.

btw: The terms of use also includes a detailed FAQ (that imho does a better job in explaining technical requirements of the app)

Hello @sventuerpe ,
since it has been a while, I will double-check internally and we will get back to you and the community with an update.
Thanks,
LMM
Corona-Warn-App Open Source Team

There are still multiple FAQs for the Corona-Warn-App. This makes finding information for the user more complicated. Maintenance of the FAQs is done by different organisations and only the Open Source team's FAQ provides a proven mechanism for feedback from users.

Site Maintainer
German URL
English URL

Die Bundesregierung
https://www.bundesregierung.de/corona-warn-app-faq
https://www.bundesregierung.de/corona-warn-app-faq-englisch

Open Source Team (SAP)
https://www.coronawarn.app/de/faq/
https://www.coronawarn.app/en/faq/

Robert Koch Institute (RKI)
https://www.rki.de/DE/Content/InfAZ/N/Neuartiges_Coronavirus/WarnApp/Warn_App.html
https://www.rki.de/EN/Content/infections/epidemiology/outbreaks/COVID-19/CWA/CWA.html

https://www.coronawarn.app/en/faq/#who_can_help also provides links to other related pages of RKI and the Federal Government.
Especially the English language FAQ pages of the RKI and Federal Government currently have content issues. There are consistency issues between the German languages sites as well:

The minimum version of iOS is listed inconsistently (see https://github.com/corona-warn-app/cwa-website/issues/737#issuecomment-751769948)
Various issues with https://www.bundesregierung.de/corona-warn-app-faq-englisch (see https://github.com/corona-warn-app/cwa-website/issues/711) such as missing links to Blog and Open Source FAQ.
The RKI English FAQ page is out-dated and refers to CWA 1.7 as the latest version, whereas 1.9 has been released and is currently being replaced by the rollout to 1.10.

If the continued maintenance of three different FAQ sites in two different languages (in total 6 sites) is unavoidable, then a formal mechanism for users to report issues on the FAQ pages of the RKI and Federal Government sites should be documented.

Tickets updated.

The minimum version of iOS is listed inconsistently (see corona-warn-app/cwa-website#737 (comment))
==> EXPOSUREAPP-4481
==> EXPOSUREAPP-4479

Various issues with https://www.bundesregierung.de/corona-warn-app-faq-englisch (see corona-warn-app/cwa-website#711) such as missing links to Blog and Open Source FAQ.
==> EXPOSUREAPP-4481

The RKI English FAQ page is out-dated and refers to CWA 1.7 as the latest version, whereas 1.9 has been released and is currently being replaced by the rollout to 1.10.
==> EXPOSUREAPP-4479

I want to stress this point (and #303) again.
If it takes too long to get all the sites combined into one, please consider to link to https://www.coronawarn.app/en/faq/ from inside the app. Here's why:
The bundesregierung.de page answers questions like:
Why do we need a Corona-Warn-App?,
How does the Corona-Warn-App work?,
Am I obliged to use the Corona-Warn-App?,
and best of all: Why should I use the Corona-Warn-App?
I think I don't have to say much more, for somebody who has the app installed on his phone the question "Why should I use the Corona-Warn-App" is just irrelevant. "How does the Corona-Warn-App work?" is answered during the app's onboarding, "Why do we need a Corona-Warn-App?" is a question which is asked by people who don't think that the app helps something (which are not the users of the app obviously).
So please consider to change the link from inside the app, or best, like proposed in the issue here, combine all these FAQ pages into one.

I want to stress this point (and #303) again.
If it takes too long to get all the sites combined into one, please consider to link to https://www.coronawarn.app/en/faq/ from inside the app. Here's why:
The bundesregierung.de page answers questions like:
Why do we need a Corona-Warn-App?,
How does the Corona-Warn-App work?,
Am I obliged to use the Corona-Warn-App?,
and best of all: Why should I use the Corona-Warn-App?
I think I don't have to say much more, for somebody who has the app installed on his phone the question "Why should I use the Corona-Warn-App" is just irrelevant. "How does the Corona-Warn-App work?" is answered during the app's onboarding, "Why do we need a Corona-Warn-App?" is a question which is asked by people who don't think that the app helps something (which are not the users of the app obviously).
So please consider to change the link from inside the app, or best, like proposed in the issue here, combine all these FAQ pages into one.

Update:
Should either be closed or moved to the Wishlist since there will not be one combined FAQ entry:
https://github.com/corona-warn-app/cwa-website/issues/769#issuecomment-770253203

Update:
Should either be closed or moved to the Wishlist since there will not be one combined FAQ entry:
https://github.com/corona-warn-app/cwa-website/issues/769#issuecomment-770253203


Update Veganify Link

VeganCheck had to change its name due to legal reasons. This should also be reflected in all linking pages/repos.
You can link to us now as Veganify @ veganify.app
Thank you!
Insert URLs to the project here:
https://veganify.app

[x] The projects is active, documented, open source licensed, shows usage from external parties and is directly targeting environmental sustainability. Find more details in the Contribution Guide.

All issues labeled as 'Good First Issue' of the project listed on OpenSustain.tech will be visible on ClimateTriage.com. This is a great way to welcome new community members to your project.

Thank you for creating this great open source project and for helping to keep OpenSustain.tech up to date.


Question :How are the word embeddings encoded?

The paper attention is all you need explains that the source sentence is position encoded and fed into the 1st layer which generates the key-value pairs. But what sort of encoding is given as input to position encoder. How are the word embeddings of the source sentence encoded without RNN ? Please help .

@kaushalshetty  Is this issue solved now? Otherwise I guess we could close that issue :)

@stefan-it Yes I will close it. Thanks.


[docs] Migrate Globals demo to TypeScript

[x] I have followed (at least) the PR section of the contributing guide.

This PR Contains

Migration of Global Demos (GlobalCss.js and DefaultProps.js) to Typescripts

Test Passed
yarn docs:typescript:check

@limatgans Much appreciated, thanks!

@eps1lon My Pleasure! :) :)


SELECT with JOIN incompatible with standard LIMIT clause

I am attempting to use the following script, but run into issues with the LIMIT clause.
.echo on
.load sqlite-vec/dist/vec0.so

CREATE VIRTUAL TABLE vectors using vec0(
    id              INTEGER PRIMARY KEY NOT NULL,
    vector          FLOAT[4]
);

INSERT INTO vectors (id, vector) VALUES (1, '[1, 2, 3, 4]');
INSERT INTO vectors (id, vector) VALUES (2, '[-1, -2, -3, -4]');
INSERT INTO vectors (id, vector) VALUES (3, '[1, 5, 2, 7]');

CREATE TABLE vectors_metadata(
    id              INTEGER PRIMARY KEY NOT NULL,
    vector_id       INTEGER UNIQUE NOT NULL,
    other_metadata  INT
);

INSERT INTO vectors_metadata (id, vector_id, other_metadata) VALUES (1, 1, 55);
INSERT INTO vectors_metadata (id, vector_id, other_metadata) VALUES (2, 2, 22);
INSERT INTO vectors_metadata (id, vector_id, other_metadata) VALUES (3, 3, -5);

SELECT vectors_metadata.other_metadata
    FROM vectors_metadata
    INNER JOIN vectors ON vectors_metadata.vector_id = vectors.id
    WHERE vectors.vector MATCH '[1, 2, 5, 4]'
    LIMIT 2;

Upon running this with
sqlite3 :memory: < script.sqlite

I receive the following error:
Parse error near line 23: A LIMIT or 'k = ?' constraint is required on vec0 knn queries.

I see another similar issue was opened in #41, which suggests upgrading to a sqlite version >=3.42. I am running a version higher than this, so I suspect this is a separate issue.
$ sqlite3 --version
3.46.1 2024-08-13 09:16:08 c9c2ab54ba1f5f46360f1b4f35d849cd3f080e6fc2b6c60e91b16c63f69aalt1 (64-bit)

I have tested this with the following tagged releases, (and the head of main at the time of posting this issue), and receive the same error with all commits.

v0.1.1
v0.1.2-alpha.2
v0.1.2-alpha.7
d529eb09d16bafc612e9e5fa0de6d7924dea4e5e

Thanks for the detailed report. A workaround is to use k = 2 in this specific query, like so:
SELECT vectors_metadata.other_metadata
FROM vectors_metadata
INNER JOIN vectors ON vectors_metadata.vector_id = vectors.id
WHERE vectors.vector MATCH '[1, 2, 5, 4]'
  AND k = 2;

Or perform the KNN query in a separate CTE step:
with knn_matches as (
    select id
    from vectors
    where vector match '[1, 2, 5, 4]'
      and k = 2
)
SELECT vectors_metadata.other_metadata
FROM knn_matches
INNER JOIN vectors_metadata ON vectors_metadata.vector_id = knn_matches.id;

In the second query, the k = 2 can be substituted with limit 2 in at least SQLite 3.46, maybe as early as 3.42.
I'm not 100% sure why the single query join doesn't work with the LIMIT clause. I fear this may be an edge case where the SQLite optimizer doesn't properly pass over the LIMIT constraint to the virtual table because of the JOIN.
Since the CTE works as expected, I'm not sure how big of a bug this is. In general I'm trying to only use the k = 999 syntax over the LIMIT 999 syntax because there are weird subtle errors with LIMIT on virtual tables, like this odd bug. If I track down the root cause I can file a bug with the SQLite team, but it might be hard to track down .Personally I like the CTE approach anyway since it clearly separate the KNN search from the metadata JOINs, but that's just personal preference

Much appreciated for the information. In my particular case, the k = <n> approach doesn't quite do what I want when I add further filters, for instance:
SELECT vectors_metadata.other_metadata
FROM vectors_metadata
INNER JOIN vectors ON vectors_metadata.vector_id = vectors.id
WHERE vectors.vector MATCH '[1, 2, 5, 4]'
  AND k = 2
  AND vectors_metadata.other_metadata > 0;

In this case, the k=2 filter appears to be applied first (getting the two closest matches), and only then are further filters applied. This means that when k=2 is used, for instance, the query may return <2 rows even when two or more valid rows exist, if the two closest vectors (or joined rows) fail to pass the additional conditions. For a more concrete example, with the same database setup as above, I run the following query:
SELECT vectors_metadata.other_metadata
FROM vectors_metadata
INNER JOIN vectors ON vectors_metadata.vector_id = vectors.id
WHERE vectors.vector MATCH '[1, 5, 2, 7]'
    AND k = 2
    AND vectors_metadata.other_metadata > 0;

I expect this query to return the values [55, 22], however the only row returned is [55].
Is there a way to get the behavior I'm looking for in this case with the k = <n> syntax? This may answer the question posed here as well.
Cheers :smile:

It seems this example SQL can support this use case.
SELECT id, distance FROM vectors v
WHERE vector MATCH '[1, 2, 5, 4]' AND id in (SELECT id FROM vectors_metadata o WHERE other_metadata > 0)
LIMIT 2

I added sqlite-vec as an ANN to txtai here - https://github.com/neuml/txtai/issues/780
A future task will integrate this even further into a single SQLite data file with content and vectors.

I added sqlite-vec as an ANN to txtai here - https://github.com/neuml/txtai/issues/780
A future task will integrate this even further into a single SQLite data file with content and vectors.

It seems this example SQL can support this use case.
SELECT id, distance FROM vectors v
WHERE vector MATCH '[1, 2, 5, 4]' AND id in (SELECT id FROM vectors_metadata o WHERE other_metadata > 0)
LIMIT 2

It will work but has 2 problem:
1, what if the ids filtered by the second select are too many?
2, there is no way to get the values in other columns in a single query, we have to query again in another table then combine the results.


new macro: safe_add()

I'm a little surprised we don't have something for this yet! It's a simple but neat use case. Here's some code that I recently saw in a PR:
coalesce(orders.total_gross_revenue, 0) +
            coalesce(refunds.total_refund_amount, 0) +
            coalesce(orders.total_discounts, 0) +
            coalesce(refunds.total_refund_shipping_amount, 0) +
            coalesce(orders.total_shipping_cost, 0) +
            coalesce(refunds.total_refund_tax_amount, 0) +
            coalesce(orders.total_tax, 0)
            as total_sales

Adding up the values for a bunch of different fields happens frequently in an ecommerce context (and likely in other contexts as well, although this is where I see it most). This could be solved by a function very similar to surrogate_key()--not fancy, but useful.

Love this idea!!

Really like this. Reminds me of a related concept: safe_greatest, safe_least.
nullif(greatest(
    coalesce(source_one.updated_at,'0000-01-01'),
    coalesce(source_two.updated_at,'0000-01-01'),
    coalesce(source_three.updated_at,0000-01-01')
),'0000-01-01')

nullif(least(
    coalesce(source_one.updated_at,'9999-01-01'),
    coalesce(source_two.updated_at,'9999-01-01'),
    coalesce(source_three.updated_at,9999-01-01')
),'9999-01-01')


Does it support hexadecimal large number calculation

If Inscribe

Regarding hexadecimal, the calculated BigDecimal can be converted to BigInteger, which is then converted to hexadecimal.

@qq1176914912, Were you asking if it is possible to use two strings in an operation like BigDecimal.Multiply(x,y)?
Where x and y are both hexadecimal strings?
And/or the resulting BigDecimal can also be shown as a hexadecimal string?

BigInteger

My idea at that time was to input the hexadecimal string directly to compute and output the hexadecimal result, and then I thought that I could convert to BigInteger and then convert to hexadecimal.

BigDecimal can be converted to BigInteger, which is then converted to hexadecimal.

Yeah, but then you are losing the decimal part.
No it doesn't support hexadecimal representation yet, but just like how Int32 and other numeric types support a "X" or "Xn" format specifier, I think we should too (Note: In "Xn", the n specifies the minimum number of digits desired, e.g."X2" or "X8").
See Microsoft - Hexadecimal format specifier for the specifics.
Note here that as far as format specifier strings go, all numeric types support the Binary ("B"), Currency ("C"") and Exponential (scientific) ("E") formats too. I think we already support the exponential specifier. Should we also support the Binary and Currency too? Currency is simple. Binary would be the same level of effort as the hexadecimal. Actually, we might as well support arbitrary base at that point. Write it once for arbitrary base. Funnily enough, this could all be done by easily by taking a dependency on my Polynomial library, but we wont do that. But the code there does show an example of how it could be done.
Implementation note: Microsoft says that the max value it supports for the precision identifier is Int32.MaxValue (or 999,999,999 for .NET 7, which is smaller for some reason). While it's probably not necessary to support beyond Int32.MaxValue, if out whole schtick is arbitrary precision, we might as well parse the precision identifier as a BigInteger and give the user enough rope to hang themselves, or give them the ammo they want for their foot-gun, or substitute in your own silly idiom here if needed.
This would be a useful feature. Not sure how much it would be used though.
Note to contributors who watch this library: I'm not asking anyone else to develop this, I don't mind. I've already written this type of code before.
QUESTION: If we support arbitrary base, how should that be specified? ToString("BaseN"), where N is a number?

BigDecimal can be converted to BigInteger, which is then converted to hexadecimal.

Yeah, but then you are losing the decimal part.
No it doesn't support hexadecimal representation yet, but just like how Int32 and other numeric types support a "X" or "Xn" format specifier, I think we should too (Note: In "Xn", the n specifies the minimum number of digits desired, e.g."X2" or "X8"). See Microsoft - Hexadecimal format specifier for the specifics.
Note here that as far as format specifier strings go, all numeric types support the Binary ("B"), Currency ("C"") and Exponential (scientific) ("E") formats too. I think we already support the exponential specifier. Should we also support the Binary and Currency too? Currency is simple. Binary would be the same level of effort as the hexadecimal. Actually, we might as well support arbitrary base at that point. Write it once for arbitrary base. Funnily enough, this could all be done by easily by taking a dependency on my Polynomial library, but we wont do that. But the code there does show an example of how it could be done.
Implementation note: Microsoft says that the max value it supports for the precision identifier is Int32.MaxValue (or 999,999,999 for .NET 7, which is smaller for some reason). While it's probably not necessary to support beyond Int32.MaxValue, if our whole schtick is arbitrary precision, we might as well parse the precision identifier as a BigInteger and give the user enough rope to hang themselves, or give them the ammo they want for their foot-gun, or substitute in your own silly idiom here if needed.
This would be a useful feature. Not sure how much it would be used though.
Note to contributors who watch this library: I'm not asking anyone else to develop this, I don't mind. I've already written this type of code before.
QUESTION: If we support arbitrary base, how should that be specified? ToString("BaseN"), where N is a number?

Thank you for taking the time to reply. I think your idea is very good and can be easily understood by using "ToString("BaseN").


Cycle 3 Funding: Develop a Jupyter widget-based app for reducing and performing stellar photometry on optical images

This issue captures execution of the Cycle 3 project “Develop a Jupyter widget-based app for reducing and performing stellar photomtery on optical images” (#279). The full text of the proposal is available here.
This project was funded to the level of $32,000, funded from the Moore grant (please submit invoices here).

Hey @mwcraig — now that the funding for this project has been approved, it would be a good idea to start drawing up a bit more concrete plan for how you'd like to disburse the funds. I understand from the proposal that you will lead, so we probably want to start with deciding how much of the budget you are going to need personally and then I can start the ball rolling with a scope of work etc.

Would it be ok to start working on this next week (after Nov 7) -- I'm heading to a conference in a few days.

I think I fixed it. Photometry , right?

Yes, typo is fixed. Thanks!

I hate to be picky, but the approved proposal is for an app to perform photomtery. If we get some photometry tool out of it, I'm not sure it really does the job...

Would it be ok to start working on this next week (after Nov 7) -- I'm heading to a conference in a few days.

No problem. Please post on this issue whenever you're ready.

Hey @mwcraig — just a ping. No need to rush, but it would be nice to get started on this when you're ready.

@jdswinbank -- I wasn't sure whether you wanted what I had in mind here as a comment or not, so I put it into a gist for now: https://gist.github.com/mwcraig/4f42b512997333e79948dd5cae5e50b8

Hi @mwcraig — thanks for that; a very nice summary.
I had previously assumed that we'd press ahead with a contract/subaward for you personally, since the original proposal lists you as leading this project. However, I understand from the disbursement proposal that the first step is actually to hire a third party to start work. That's not a problem; I'll put thoughts of a contract for you on hold for now.
After we have all the contracts for existing project members sorted out (and we're nearly done now) the Finance Committee will put our heads together and come up with some guidance for hiring third parties. We'll get back to you on this shortly (I'm guessing that'll be early next year).

This is an updated request in response to the call for additional funding opportunities. As reminder, the amount of the original request was $40,000, and the amount approved was $32,000.
The updated request is for $58,025, as detailed below. Part of the money would be used to hire me or people I've identified directly as independent contractors, and part would be a subaward for MSUM (my institution) to be spent Fall 2023.
Since this is such a substantial increase I have specified a priority for funding for each of the items below.
Area 1: Code development and guide writing
Priority: High
Amount: $36,000 (360 develop hours at $100/hour)
Description
As explained in this gist, the two main goals of this project are to develop a photometry guide and to develop a general-purpose photometry tool for stellar astronomy, particularly aimed at relatively novice users.
The initial phase of this work is to restructure the package stellarphot, which I've used for the last couple of years in an introductory observational astronomy course, to have a simpler and better documented and tested API, In parallel with that, work will begin on the photometry guide.
I am proposing that we hire, as independent contractors over the summer, at the rate of $100/hour:

Person
Hours
Justificaton

Matt Craig
160
Write the photomtry guide, oversee the restructuring of stellarphot, and help with that restructuring.

Contractor[1]
160
Propose new structure for  stellarphot and assist in much of the code re-write.

Contractor[2]
40
Act as beta tester for revamped stellarphot and photometry guide.

[1] There is a colleague of mine with a background in computer science and astronomy that I have identified as an excellent fit for this role.
[2] Another colleague with background in astronomy and some familiarity with Python has been identified as an excellent fit for this role.
In addition, undergraduate students paid with separate funds will participate in testing over the summer and are learning how to make code contributions.
Area 2: Compensation for undergraduates at MSUM in Fall 2023 for ongoing testing and bug fixes
Priority: Medium
Amount[3]: $10,125 ($15/hour for 600 hours, plus 12.5% overhead and 7.65% for fringe benefits)
Description
The photometry guide and stellarphot will be used in an observational astronomy course this fall. I anticipate usage will uncover bugs in stellarphot and areas of the guide that are unclear or incorrect.
The students hired with these funds will fix those bugs and help with updating the guide.
[3]: The overhead and fringe rates are approximate and will need to be confirmed before the subaward is made. The total budget for this area will not exceed $10,125 and the pay rate will not be less than $15/hour. The number of hours will be slightly adjusted, if necessary.
Area 3: Teaching release for Matt Craig
Priority: Low
Amount[4]: $11,900 (2 credits of teaching release at $3872/credit, with 12.5% overhead and 40% for fringe benefits, and 1% pay increase)
Description
Though part of my teaching load in Fall 2023 will be observational astronomy, I will also be teaching several other courses. Two credits of teaching release will provide an additional 6 hours per week to work on fixing bugs and updating the photometry guide as the semester progresses.
[4]: The overhead and fringe rates are approximate and will need to be confirmed before the subaward is made. In addition, there will likely be a pay increase when our new contract is negotiated this summer. The total budget for this area will not exceed $11,900. The number of credits of teaching release will be adjusted, or the money will be used to pay for additional duty days during the Fall semester.

To be clear, the request is for an additional $26,025, correct. Bringing the total budget from $32k to $58k?

Yes, though the highest priority is an increase of $4k to $36k for the summer pay.
Some money for undergrads would be helpful.
Whether I get reassigned time is really a very low priority.

Hey @mwcraig — the Finance Committee has approved an increase to the allocated funding of $26,025 for this project for a total of $58,025. Please submit invoices to the Moore grant as described above. We encourage you to invoice for all this work, including the extension, by the end of December if possible, and certainly not later than 31 March 2024.

@jdswinbank -- wonderful, thanks for the news!

I am working with my campus to get our internal paperwork ready for Areas 2 and 3 of the revised proposal. What additional steps will need to be taken to get that subaward set up?
The money for Area 1 is meant to be paid directly to the folks doing the work. I can ask them to set up accounts in OpenCollective. What additional steps beyond that need to be taken before submitting invoices?

As I understand it, the plan is that:

Each of the individuals working on Area 1 will receive a separate contract with NumFOCUS.
MSUM will receive a subaward from NumFOCUS covering all the work to be performed in Areas 2 & 3.

The process for both the contracts and the subaward looks similar from our point of view.

We — you and I — will agree on brief (couple of paragraph) statements of work describing what the award/contract will cover. The text you've provided above should pretty much cover this; I will do some editing and propose some text for you to  approve. We should also agree start/end dates for the contract.
We will pass these to NumFOCUS and ask them to begin the contracting process.
NumFOCUS will take it from there, reaching out to the funded entities with a contract for their signature.
Everybody will need to be registered not just with OpenCollective, but also with Rocket (https://numfocus.typeform.com/to/umyusM). This is pretty straightforward, and we/NumFOCUS can provide help as necessary.

I hope that covers everything — let me know if you have questions, and please ping me if you've not received draft SOWs within the next few days.

Just a note that I've also asked MSUM about administering Area 1, in the event they are able to do that with minimal deductions to minimize the number of contracts.

Here's a draft of the SoW. I've written this on the assumption that all of this work takes place through an award to MSUM; we can split off work package 1 (your area 1, above) into separate contracts if necessary.
Craig _ MSUM SOW.pdf
I think this is all pretty straightforward — it's basically just reiterating what you've said above — but please yell if anything looks incorrect.

@jdswinbank -- this looks good.
A couple of updates from our end:

The fall money that was to be used for teaching release for me will instead be used to augment my pay in the fall (it was too late to hire a replacement). My load includes an observational astronomy course in which we will use the material developed under the contract.
Once we moved all of the summer salary into the university award, the costs increased somewhat. I am negotiating with the university to reduce those costs, but said i would ask about a couple of possibilities. "No" is a perfectly fine answer to both of these.

Would Astropy consider increasing the grant by an additional $8,000, to $66,000?
Would Astropy consider increasing the grant by an additional $1,000, to approximately $59,000?

What is the motivation for the increases? I'm guessing maybe fringe or overhead rate change?

Sort of; by running the pay through the university more of it is subject to overhead.

Can you provide us with a break down of the budget indicating direct and indirect/overhead costs?

@kelle and the rest of the Finance Committee -- some good news. MSUM has decided to use its overhead to partially defray the cost of the fringe benefits in the restructured grant.
As result, the amount of the increase I am requesting is $698.17, bringing the total to $58,723.
A detailed accounting of the cost is in this google sheet.
The work hours dedicated to the grant activities are still roughly 6.5% higher than in the original proposal, as explained below. With my administration's commitment to dedicate the overhead to paying fringe benefits to cost to Astropy for this increase in hours would be the amount of the increase I am requesting, $698.17.

Including fringe benefits in the Astropy funding means that the folks who have been doing the summer work do not have to pay self-employment tax. They have agreed to work more hours at a reduced rate to recognize that increase in their take home pay. The increase in hours is about 6.5%.

I'm giving this a friendly bump -- I'm wondering if a decision has been made about the additional increase.

Hi Matt — sorry for the delay on this one. We approve this increase of $698.17, for a total budget of $58,723, and apolgise for how long it took us to get here!

@jdswinbank -- we should have invoices for some of this working on their way soon. I wanted to ask one question: What is the last day that work for this can be invoiced?

The official end date of the Moore grant, which is funding this work, is 2024-04-01, so that's an absolute hard deadline. In order to avoid a last minute panic, though, if possible I'd encourage you to submit well in advance of that. We have to report to Moore on 1 March, and we'd love to have as much as possible finalized by then.

I wanted to provide a brief update on the status of this work prior to submitting a final report at the end of March.

The package stellarphot is close to an alpha release of version 2.0. We are testing internally and I anticipate the alpha will be out early the week of Mar 11. A few highlights:

Once settings have for your camera, observatory, etc., doing photometry on an image ro folder of images is a couple of lines of code.
Rewriting the code to work with pydantic 2, which was released in early Fall 2023 took much more effort than anticipated. It turns out to be much more flexible for adding adding pydantic behavior to non-pydantic classes like astropy's Time or SkyCoord or Quantity. The upside is that we've worked out how to use pydantic to serialize those objects, generate json files of settings for use in stellarphot, and automatically generate widget-based forms for entering values.
The AAVSO has expressed interest in using stellarphot to perform photometry on images gather by amateurs using highly automated telescopes like those made by Unistellar and ZWO's SeeStar.
I have names/emails of roughly 20 AAVSO members from the Astropy workshop presented at the AAVSO meeting last November who are willing to try out the code. That will start this month and continue well beyond that, I think.

The photometry guide is (finally) coming along nicely. Look for a large-ish PR the week of Mar 11, with updates to an open PR prior to that.

Though the jupyter-book project on which the guide builds has recently released the first stable version of the package, I plan to wait to update the guide to that version until early April so as to ensure content gets in before then.

I am overdue for an update.

The photometry guide is still coming along. I made a new branch in the repository if you would like to look at the current state. I ened up updating the jupyter-book version, in part to add a feature that comments may be made on any of the pages. Those comments are turned into github issues. Though this isn't ready for broad distribution and feedback yet, you can look at the photomery sections here: https://www.astropy.org/ccd-reduction-and-photometry-guide/v/pdev

The list of issues remaining in to be resolved in stellarphot is down to a handful, listed here. Almost all of them are re-writes of the notebook interface, which are straightforward now that the underlying infrastructure is complete.


[FEAT]

Will you share an update on .aab files?

I don't have an update. I don't believe Apktool has anything to do with aab.
We are discussing this in many areas:

https://github.com/iBotPeaches/Apktool/discussions/2651
https://github.com/iBotPeaches/Apktool/discussions/2572


phoneNumber and deviceId alternative to email / password authentication

Twilio or similar SMS service required to authenticate the phone (analogous to email verification links or codes). But much faster route than email for auth on mobile apps.
How apps used on multiple devices would have to adapt is up for debate, and thoughts are welcome.
Could be the start of a 2FA scheme as well, with verified emails able to regain access to new devices. Again, there are a couple of ways that could work and some thought should be given as to which method would be best?

I love the idea of using this type of Authentication. Facebook AccountKit is an alternative that takes away the need for an external SMS provider, keeping track of phone numbers, etc. It might also make it easier to use across projects. However, the one thing missing from it is that it is not linked to a device. This is a big plus in this example.

I'm using React Native with Expo, which isn't really compatible with AccountKit at the moment (at least not seamlessly). And ironically for a Facebook stack adopter, Facebook's Graph-related tools feel just a bit creepy (my app would just add to their personal data gathering empire). Keeping an eye on how that develops, though.

I like the example, thanks! I am not quite sure how Twilio ties into this. Also, where is the devide id coming from?
What would be the possible additions to make this example more distinct from the email/pw example?

Sure! Actually, phone number verification could be open-ended similar to email verification is right now. Twilio (or other similar service like Amazon SNS) can send a text message with a verification code to a phone, which a user can then input and get confirmed. Whether or not a developer would want to integrate the full Authy or similar token-dispensing service would also be up to them; either way, a confirmation via text would be followed by a login request to Graph.cool with a phone number and a device ID.
The device ID would have to be figured out based on device type and platform, and could really be any available unique identifier. In my case, with React Native on Expo (mobile only, no tablets), it's a super simple call.
Again, very open to modifications. It feels like a bit of a hack, and yet it would be so similar to the standard email methods that I can't see why not?

My 2 cents: there's account registration and verification.
If you register using a phone number, you need a way to verify the phone number. This can be done using any call/sms provider, like Twilio.
If you register using a deviceId, the way to verify this would be using a push notification to the device.
If you register using an email address, you need an email verification.
Then there's login and login verification. Login can be using any authentication method available. If you need additional verification, you need some form on 2FA. This can be SMS/phone call/Authenticator/email with one time code/etc.
As far as I can see, this example focuses on user registration using deviceId/phone number, and verification of that phone number using an SMS provider.


Make TPC-H data publicly available

Hi, I am trying to access the TPC-H benchmarking data but running into some permissioning issues:

Am I accessing the right data, or could the data be made public please?

Ah, I thought that this was in coiled-datasets-rp, the requester pays bucket.
@fjetter who I think manages a lot of this is on PTO this week.  @hendrikmakait or @phofl thoughts?  (also ok to wait until Florian gets back)


⚠ BCR Bot has degraded performance

In 071c58d, BCR Bot (https://bcr.alternativasinteligentes.com/BCRApi/web/chatbot) experienced degraded performance:

HTTP code: 200
Response time: 1116 ms

Resolved: BCR Bot performance has improved in ddb86b7.


Enhance documentation with a "read-write" operation

I followed the readme and was able to get a hand on a Service object (read-only / get operations), but I could not find the logic needed to trigger an upgrade (e.g. trying to change the docker image associated to the service). How can this be done ? I guess such an example in the README.md would be very appreciated.
thanks !

I'm also looking to do this. Any updates here?

Looks like the actions you can perform are done off the main static Service methods.
ServiceService serviceService = rancher.type(io.rancher.service.ServiceService.class);
// get a single service
Service service = ..
// upgrade
serviceService.upgrade(service.getId(), ...);


Can't restart a stopped contiainer

When restart a stopped wandb/local container, the container exited instantly.

Hey @imyhxy to restart the container you should run:
docker stop wandb-local, then run wandb local or manually start the container again with docker.

@vanpelt When restart the container with docker start wandb-local, the container existed instantly.

@imyhxy calling docker start wandb-local will not work if you don't have a container named wandb-local.  Calling docker restart wandb-local does not work either.  If you need to start a wandb local container, you should run wandb local.


Fix broken links to non-existant "user_guide/"

user_guides is no longer valid as we have restructured our documentation. However, some of the pages have references to "user_guide/". These need to be fixed by replacing references to /docs/about/user_guide/ with /docs/guides/.
> cd content
> find . -name "*" | xargs grep "user_guide" 2>&1 | grep -v grep

./docs/started/getting-started-minikube.md:177:If the page doesn't refresh, please see [troubleshooting](/docs/about/user_guide/#problems-spawning-jupyter-pods).
./docs/started/getting-started-minikube.md:189:Refer to the [user guide](/docs/about/user_guide/)
./docs/guides/troubleshooting.md:145:Kubeflow requires ksonnet version 0.9.2 or later [see here](/docs/about/user_guide/#requirements). If you run `ks apply` with an older version of ksonnet you will likely get the error `Unknown variable: env` as illustrated below:
./docs/guides/troubleshooting.md:160:If your ksonnet version is lower than v0.9.2, please upgrade it and follow the [user_guide](/docs/about/user_guide/) to recreate the app.

/priority p1
/release 0.3.0

/release 0.3.0

/close


fix: Read terraformVersion from ComponentCommon.Common

This seems to fix the incorrect terraform version being detected per component

it didn't fix it :( https://github.com/HandshakesByDC/terraform.handshakes.liftandshift/pull/53#discussion_r1009061222


CoA pack add article description

Status
Ready
Related Issues
fixes: link to the issue
Description
Add article description so it shows up in the index
Screenshots
Paste here any images that will help the reviewer

@michalgold @mayagoldb please review


Discussion: Is it possible to add a GODEBUG option to disable the "cooperative goroutine preemption"

Discussion: Is it possible for golang runtime to add a GODEBUG option to disable the "cooperative goroutine preemption"
Allow me to introduce our app logic code organization method:
We use a single-threaded multi-process approach, similar to nodejs and nginx, the logic code run in a single-threaded environment, and we deploy multiple instances to ultilize the multi-cores of the server.
The advantage of running logic code in a single-threaded environment is that we do not need mutex to protect most public variables, only need to handle a few cases caused by async IO (such as access to redis).
However, due to the "cooperative goroutine preemption" introduced by golang 1.1x: when a goroutine runs for more than 10ms, it will be set to preemptable by sysmon, and it will be preempted when it enters the runtime.newstack function(caused by any function call).
Therefore, we now need mutex to protect all public variables access (e.g.: a map), but this does increase our development burden, because we need to consider where to lock or unlock.
Therefore, I hope to have a GODEBUG option, similar to the "asyncpreemptoff" option of "signal-based asynchronous goroutine preemption", thus we can disable the "cooperative goroutine preemption" when we need to.
Anticipate your suggestions, thanks so much.
runtime code relative:
https://github.com/golang/go/blob/11b3730a02c93fd5745bfd977156541a9033759b/src/runtime/proc.go#L4726

I don't understand the problem you're trying to solve.
You're writing a single-threaded program. Don't use the go statement. Then you won't need locks on your global variables. It's irrelevant whether, or how, the runtime interrupts your goroutine occasionally to get its own work done.
In any case, this issue tracker isn't the right place for this kind of discussion. Please see https://golang.org/wiki/Questions for good places to ask.

@randall77 Thanks for your reply, and sorry for that I did not make my problem clear.
For example, I have a simple http server, this is my main function:
func main() {
    runtime.MAXGOPROC(1); // use only one thread to run all goroutines
    ...... // start http server
}

And, each client request will call into this function:
func simpleHandler(r*Request) {
    ...... // do long time cost job, consume almost 10ms time
   aMap[key] = value // access to a global map instance named aMap
}

as you sugguest, I do not use "go" to create new goroutine, but the HTTP runtime does: every new client request correspond to a new goroutine.
because long time cost job has cost almost 10ms,  and when it run into "aMap[key]=value", the goroutine will have great chance to be preemted, and then another goroutine read the map, it will panic:
https://github.com/golang/go/blob/f1f8f9af9a55d73dfc6603a93bee0559fdc9024d/src/runtime/map.go#L411
If I can disable the "cooperative goroutine preemption", then everything goes right.

Regardless of the GOMAXPROC setting your program has a logical data race, and is considered by the go specification to be undefined. We're not going to change the language to support this use case but maybe you could explain the underlying problem that has lead you to use a shared map, there may be another way to solve your underlying problem without changing the language.

@davecheney Thanks for your reply.
I recognize that I can use mutex, but the most important thing to me, is how can we get rid of the missing lock/unlock of mutex.? Maybe someday, somebody goes wrong, and then a thread safety(or concurrent safety)  bug come, unfortunately, this type of bug show up randomly, and is difficult to reproduce and fix.
If golang has the facility just like rust does, utilize the compiler to force things to be thread safety, then mutex is free to use(without the consideraton of performance). But for now, golang need the programer to ensure thread safety, that make my hair curl.
So, if we can disable the '10ms coorperative preemption', then we need not to use mutex most of time.

@linguohua Have you seen https://blog.golang.org/race-detector and https://golang.org/doc/articles/race_detector.html ?  The race detector can help detect a missing lock/unlock of a mutex.

Thank you for explaining your use case.
Generally the practice to ensure lock invariants are followed is to place the type under the mutex and the mutex itself into a struct and drive access to those values through methods on that struct.
Alternatively, grant the ownership of the thing under the lock to a single goroutine and mediate access to that state via messages over a channel.
Lastly, pervasive use of the race detector in test and canary deploys is recommended to local data races.

@ianlancetaylor @davecheney Thank you both.
I will follow your advices to refactor our code.


[ffi]Is it necessary to manually free the memory?

class RawImage extends Struct {
  @Int32()
  external int length;
  external Pointer<Uint8> bytes;
}

Pointer<RawImage> mallocRawImage(Uint8List bytes) {
  final bytesPointer = bytes.allocatePointer();
  final refPointer = malloc<RawImage>();
  refPointer.ref
    ..length = bytes.length
    ..bytes = bytesPointer;
  return refPointer;
}
extension Uint8ListBlobConversion on Uint8List {
  Pointer<Uint8> allocatePointer() {
    final blob = malloc<Uint8>(length);
    final blobBytes = blob.asTypedList(length);
    blobBytes.setAll(0, this);
    return blob;
  }
}

I manually malloc memory for Struct to pass it to native, should I manually free the Pointer<RawImage> and the Pointer<Uint8>? And how about the Struct returned from native?

FFI is a Dart feature, not part of Flutter itself, so questions and issues related to it should go to the Dart project rather than the Flutter project.


Guilford County, NC, USA

The city fields look strange.  For example:
$ head -10 guilford.csv
LON,LAT,NUMBER,STREET,UNIT,CITY,DISTRICT,REGION,POSTCODE,ID,HASH
-79.7640512,36.1418192,4677,CHAPEL RIDGE DR,,PCTYGR,,NC,27405,,64fcba3319afc028
-80.0110425,36.0171039,1737,MIRABEAU CT,,PCTYHP,,NC,27265,,32d8fff9e4185de5
-79.5487278,36.0279245,1987,WHEELER BRIDGE RD,,PCTYBU,,NC,27215,,2d6893c49023f3c0
-79.9186727,36.019066,3066,RENAISSANCE PKWY,,PCTYJA,,NC,27282,,822cae49d577c4af
-79.9180216,36.0200071,3055,RENAISSANCE PKWY,,PCTYJA,,NC,27282,,e79ab6866913b38f
-79.9183187,36.0265167,803,JARMAN DR,,PCTYJA,,NC,27282,,342d0cf07d63bf13
-79.9187974,36.0189528,3066,RENAISSANCE PKWY,,PCTYJA,,NC,27282,,7009d78d667819c4
-79.9187782,36.0188594,3072,RENAISSANCE PKWY,,PCTYJA,,NC,27282,,2f0b0f567cf982e4
-79.555987,35.9495979,6410,KIMESVILLE RD,,PCTYLI,,NC,27298,,e7941340591149f6
Should these be Greensboro?

@nigelhorne is this still an issue with the current data? Thanks!


comparison in the number accidents by/days/ month/ years in a single segment of the road

this will be done as part of use case investigations


Google Search bridge: don't return already found URLs

The Google Search bridge is awesome, but I often get results that I've already seen. It would be great if there would be an option which keeps a list of already returned URLs so they won't be shown a second time.

For some reason im only getting 3 results.

This could likely be solved by setting the URL as the uid. Then, feed clients can keep track of them internally (as many do).

I think the "problem" here is google and it's AMP service. This is the result from the current bridge in rss:
	<item>
		<title>rss-bridge - bytemeta</title>
		<link>https://bytemeta.vip/repo/RSS-Bridge/rss-bridge/issues?after=Y3Vyc29yOnYyOpK5MjAyMi0wMi0xM1QwMTo1MDozNCswODowMM5DnPsU&before=&page=11</link>
		<guid isPermaLink="true">https://bytemeta.vip/repo/RSS-Bridge/rss-bridge/issues?after=Y3Vyc29yOnYyOpK5MjAyMi0wMi0xM1QwMTo1MDozNCswODowMM5DnPsU&before=&page=11</guid>
		<pubDate>Mon, 09 May 2022 00:00:00 +0000</pubDate>
		<description> rss-bridge repo issues. ... Bridge request for CBC Editor's Blog. doowruc. doowruc CLOSED · Updated 2 months ago · NASA APOD Bridge failed with error 0. </description>

	</item>

As you can see, the GUID already is the link, which should cause exactly what we want.
Other entries look fine:
	<item>
		<title>faq | bubbletea.dev</title>
		<link>https://bubbletea.dev/faq/</link>
		<guid isPermaLink="true">https://bubbletea.dev/faq/</guid>
		<pubDate>Wed, 09 Mar 2022 00:00:00 +0000</pubDate>
		<description> archivebox; booru; cannery; gitea; ipfs; irc; mealie; minecraft; misskey; mumble; neko; nextcloud; peertube; revolt; rss; rss-bridge; searx; start; wakapi ... </description>

	</item>

This should only come up once as the GUID shouldn't change.
So I think this is already implemented, it just sometimes doesnt work because the google link changes. I dont think there is an easy solution for this.
I would close this, do you agree @siccovansas ?

You can see a query parameter in my first example. Thats the first item when you search for rss-bridge.
I would close this as the root cause itself is basically already implemented. If sometimes the links change because of how google does things, I dont think it would make sense to try to fight that neverending battle just for a single "mark as read" click every once in a while :)


Questions about Tapuino functionality

Hello.
First of all, thank you for this great little device. I bought mine from Edu Arana, some time ago, but tested it just few days ago.
https://github.com/arananet/Tapuino-Reloaded
I have some questions that others might find interesting as well:

Is there a PDF manual which describes all Tapuino functions and button combinations?
Is there a way to play T64 files directly from SD card on Tapuino?
If I use some save state functionality by Replay cart and dump state of C64 on floppy to restore it later, can I manually set position on Tapuino at exact percent where it was before I dumped state? For example 75% of loaded TAP file (FW, RW I suppose)?

Sorry if this was asked before. Thank you.

Hi there,
Thanks for your interest, to briefly answer your questions:

Unfortunately not. If anyone feels like writing something up in Markdown and sending it to me or submitting a pull request, that would be great.
T64 is not supported
Not in this version

The good news is that I have released the next version of the hardware / firmware which supports (3) and have plans to document it better. I'll possibly look into T64. The Atmega / Arduino doesn't have sufficient flash and ram for new features and enhancements so I am moving to the ESP32 to provide this.
Check the project out here: https://github.com/sweetlilmre/TapuinoNext

@sweetlilmre Are you sure you cannot add support for FW-RW to tap file?
Something like, now there are Play, Record, and you add "Play from", and when user selects that he must using forward and back keys find exact position in percent, and confirm it with "play" or "wait" option?
No need to provide save state functionality on the Tapuino device itself.

@sweetlilmre OK, I tested this using Tapuino reloaded, EasyFlash3 with Retro Replay cart and Las ninja game.
When game loaded it showed 50% on tap file. I saved the game/backed up  RAM to floppy.
Turned on machine and loaded save state from floppy, and pressed Play on tapuino on the same file. Till restore was loaded tapuino came to 10% of tap file, and when game loaded from restore, tapuino stopped oading procedure.
I played first level and when I finished it, tapuino continued to increase percent. I had to wait till it reached 50% and then I heard second level loading music, few percent later. Second level loaded just fine.
So basically, there is no extra need for some hacks, id just Forward/Rewind function could be added Tapuino qould have one extra functionality.
I do not know how hard is to do it, and do we need some special programmer to load new firmware, or it can be done from SD card. This tapuino reloaded version does not have and microUSB slot that I can see.

RAM and flash limits are almost at the max, it might be possible to squeeze this function (and accompanying UI) into the code, but it would be tight. At this point in time I don't have the bandwidth to look into this as I am concentrating on the Next, the code is open though so if someone wants to send in a pull request I'd look at merging it (the functionality can be found in the TapuinoNext source code).