grpcio fails to install on Apple Silicon

What version of gRPC and what language are you using?
grpcio-1.34.0
What operating system (Linux, Windows,...) and version?
macOS big Sur 11.0.1, running on M1 Apple Silicon
What runtime / compiler are you using (e.g. python version or version of gcc)
Python 3.8.7 (default, Dec 30 2020, 02:09:32)
[Clang 12.0.0 (clang-1<IP_ADDRESS>)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

$ clang --version
Apple clang version 12.0.0 (clang-1<IP_ADDRESS>)
Target: arm64-apple-darwin20.1.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

What did you do?
Please provide either 1) A unit test for reproducing the bug or 2) Specific steps for us to follow to reproduce the bug. If there’s not enough information to debug the problem, gRPC team may close the issue at their discretion. You’re welcome to re-open the issue once you have a reproduction.
pip install grpcio==1.34.0
What did you expect to see?
That grpcio will be successfully installed
What did you see instead?
Collecting grpcio
  Using cached grpcio-1.34.0.tar.gz (21.0 MB)
    ERROR: Command errored out with exit status 1:
     command: /Users/yoav/Library/Caches/pypoetry/virtualenvs/orca-g6p4a6cZ-py3.8/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-install-n05vw0u0/grpcio_103d947f24954cde8b8a019b4c9f88fe/setup.py'"'"'; __file__='"'"'/private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-install-n05vw0u0/grpcio_103d947f24954cde8b8a019b4c9f88fe/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-pip-egg-info-z41yqiw6
         cwd: /private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-install-n05vw0u0/grpcio_103d947f24954cde8b8a019b4c9f88fe/
    Complete output (10 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/jj/r3j0p91s2qgdpcpf5ptqr9qm0000gn/T/pip-install-n05vw0u0/grpcio_103d947f24954cde8b8a019b4c9f88fe/setup.py", line 359, in <module>
        if mac_target and (pkg_resources.parse_version(mac_target) <
      File "/Users/yoav/Library/Caches/pypoetry/virtualenvs/orca-g6p4a6cZ-py3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 113, in parse_version
        return packaging.version.Version(v)
      File "/Users/yoav/Library/Caches/pypoetry/virtualenvs/orca-g6p4a6cZ-py3.8/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/version.py", line 275, in __init__
        match = self._regex.search(version)
    TypeError: expected string or bytes-like object
    ASM Builds for BoringSSL currently not supported on: macosx-11-arm64
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).
See TROUBLESHOOTING.md for how to diagnose problems better.
Anything else we should know about your project / environment?

I managed to get the grpcio Python package installed on Python 3.9.1 installed via Pyenv on a M1 macOS:
Python 3.9.1 (default, Jan  5 2021, 10:39:36)
[Clang 12.0.0 (clang-1<IP_ADDRESS>)] on darwin

Native arm64:
$ lipo -archs $(which python)
arm64

I had to set a few environment variables to get it to work:
GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

to make the installation process use the libraries I've already got on the system, and then the following to point it at OpenSSL installed via Homebrew:
CFLAGS="-I/opt/homebrew/opt/openssl/include"
LDFLAGS="-L/opt/homebrew/opt/openssl/lib"

I managed to get the grpcio Python package installed on Python 3.9.1 installed via Pyenv on a M1 macOS:
Python 3.9.1 (default, Jan  5 2021, 10:39:36)
[Clang 12.0.0 (clang-1<IP_ADDRESS>)] on darwin

Native arm64:
$ lipo -archs $(which python)
arm64

I had to set a few environment variables to get it to work:
GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

to make the installation process use the libraries I've already got on the system, and then the following to point it at OpenSSL installed via Homebrew:
CFLAGS="-I/opt/homebrew/opt/openssl/include"
LDFLAGS="-L/opt/homebrew/opt/openssl/lib"

you can use system python3 (/usr/bin/python3)
mkdir grpctest
cd grpctest
/usr/bin/python3 -m pip install virtualenv
/usr/bin/python3 -m virtualenv venv
source venv/bin/activate __
arch -x86_64 pip install --upgrade pip setuptools
arch -x86_64 pip install grpcio
arch -x86_64 pip install grpcio-tools
touch test.proto
...
arch -x86_64 python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. test.proto
touch main.py
arch -x86_64 python main.py

attention
arch -x86_64 should be used before the command

you can use system python3 (/usr/bin/python3)
mkdir grpctest
cd grpctest
/usr/bin/python3 -m pip install virtualenv
/usr/bin/python3 -m virtualenv venv
source venv/bin/activate __
arch -x86_64 pip install --upgrade pip setuptools
arch -x86_64 pip install grpcio
arch -x86_64 pip install grpcio-tools
touch test.proto
...
arch -x86_64 python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. test.proto
touch main.py
arch -x86_64 python main.py

attention
arch -x86_64 should be used before the command

@yoav-orca We currently doesn't have ARM64 machines to produce binary wheels, but we are working on a plan.
Back to the build, the failure is not caused by the ASM optimization. Can you check the result of sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET') on your Mac? It seems returning a number or some sort, which supposed to be a string (or bytes) of targeted macOS version.

I just tried to install the library from the master branch with the fix included, but it still fails out of the box.
Is the version of BoringSSL and zlib referenced in this repository supposed to be working on ARM macOS? If not it might be better to try and detect when running on a ARM Mac (platform.machine() can do this) and set the equivalent of the following environment variables automatically:
GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

Does that sound reasonable? If so, I don't mind trying to put a PR together to do that.

I just tried to install the library from the master branch with the fix included, but it still fails out of the box.
Is the version of BoringSSL and zlib referenced in this repository supposed to be working on ARM macOS? If not it might be better to try and detect when running on a ARM Mac (platform.machine() can do this) and set the equivalent of the following environment variables automatically:
GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

Does that sound reasonable? If so, I don't mind trying to put a PR together to do that.

@Tenzer That would be great if you can put up a PR, and you can ensure it works on the target machine (I don't have an Apple M1).

@Tenzer That would be great if you can put up a PR, and you can ensure it works on the target machine (I don't have an Apple M1).

I got a successful build using @Tenzer 's solution, with addition of CFLAGS and LDFLAGS using brew --prefix protobuf, brew --prefix openssl, brew --prefix zlib, and setting the GRPC_PYTHON_BUILD_SYSTEM_* environment variables, with arm64 homebrew and Python 3.9.1 built via pyenv's latest master.

I got a successful build using @Tenzer 's solution, with addition of CFLAGS and LDFLAGS using brew --prefix protobuf, brew --prefix openssl, brew --prefix zlib, and setting the GRPC_PYTHON_BUILD_SYSTEM_* environment variables, with arm64 homebrew and Python 3.9.1 built via pyenv's latest master.

Thanks @Tenzer and @achimnol! This worked for me as well.
Note if the following issue occurs after those recommendations:
File "____/venv/lib/python3.9/site-packages/pkg_resources/_vendor/packaging/version.py", line 275, in __init__
    match = self._regex.search(version)
TypeError: expected string or bytes-like object

Edit version.py line 275 on your system to:
match = self._regex.search(str(version))

Thanks @Tenzer and @achimnol! This worked for me as well.
Note if the following issue occurs after those recommendations:
File "____/venv/lib/python3.9/site-packages/pkg_resources/_vendor/packaging/version.py", line 275, in __init__
    match = self._regex.search(version)
TypeError: expected string or bytes-like object

Edit version.py line 275 on your system to:
match = self._regex.search(str(version))

Setting the environment variables as recommended by @Tenzer works perfectly for me, too! Thank you! I used it to install stable_baselines3 on my Apple Silicon M1 🙂
I simply set the environment variables first:
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

Then installed the grpcio package with:
pip install grpcio

At the time of writing this installs and works with grpcio-1.35.0
@lidizheng @Tenzer Since it seems like this problem is not yet resolved I would over help creating a PR, however I am completely new to this kind of stuff (pip install, setuptools etc.) and don't really know where I would have to put the code. This is what I think would have to go somewhere according to your contributions:
import platform
import os
from distutils import util

if 'arm64' in platform.machine() and 'mac' in util.get_platform():
    os.environ["GRPC_PYTHON_BUILD_SYSTEM_OPENSSL"] = "1"
    os.environ["GRPC_PYTHON_BUILD_SYSTEM_ZLIB"] = "1"
else:
    pass

Maybe in  setup.py around line 82? Of course the required import statements are there already.
My Setup
Device: Apple M1 MacBook Pro running macOS Big Sur 11.2
Python:
Python 3.9.1 | packaged by conda-forge | (default, Jan 26 2021, 01:30:54)
[Clang 11.0.1 ] on darwin

Let me know if there is any more info that could be helpful!
I also added the wheel generated by pip if that is helpful to anyone.
grpcio-1.35.0-cp39-cp39-macosx_11_0_arm64.whl.zip

I tried doing many of the above, and unfortunately all failed for me. Ended up getting it to work by following the steps here (but for grpcio) https://stackoverflow.com/questions/65564673/how-do-i-import-pandas-on-apple-m1-chip. Posting in case it's helpful to anyone in the future.

Still having this problem after having tried the above. Also, the listed solutions are pretty involved workarounds. Is someone working on a more permanent fix?

Yes, someone please respond ... I've already burned over 2 days on this error!!
I've tried Py3.8, Py3.9, Conda-miniforge
I've tried bash arm64, bash x86, zsh arm64
How do I install and run this library??  The failure is below:
import google.cloud.ndb as ndb
File "/opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.8/site-packages/google/cloud/ndb/init.py", line 28, in
from google.cloud.ndb.client import Client
File "/opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.8/site-packages/google/cloud/ndb/client.py", line 18, in
import grpc
File "/opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.8/site-packages/grpc/init.py", line 23, in
from grpc._cython import cygrpc as _cygrpc
ImportError: dlopen(/opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so, 2): Symbol not found: _CFRelease
Referenced from: /opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so
Expected in: flat namespace
in /opt/homebrew/Caskroom/miniforge/base/envs/env/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-darwin.so

@dgaedcke I don't have a M1 machine. The answer is based on what I read.
Have you try to build your environment with emulated x86?
arch -x86_64 /bin/bash

Then you should be able to use x86 Python, and x86 grpcio binary wheel, no compilation needed.

Thanks for your response.
Your suggestion is a really good one!!!
And I already got this fixed by using the Conda miniforge build of grpcio
I suspect this will be fixed in this repo when they release 1.39.1 but I'm not sure!!
really appreciate your help!

arch -x86_64

Thank you so much for this. Worked like a charm in my Macbook Pro M1 with Big Sur (11.4)

My related post on stack overflow. I found a different solution. https://stackoverflow.com/questions/68181171/install-of-python-firebase-admin-on-apple-silicon-m1-fails-grpico-fails

incase anyone still having these issue, i was success runing grpc and google-protobuf in ruby 2.5.7 by installing all required lib such as rbenv, ruby-build by using prefix arch -x86_64 $(command) :
e.g. arch -x86_63 bundle install
notes :
make sure u have 2 versions of homebrew installed, and use rbenv from -x86 directory instead.
and this is partial versions of my zshrc :
###### Multiple Homebrew
alias abrew="/opt/homebrew/bin/brew"
alias i="arch -x86_64"
alias ibrew="arch -x86_64 /usr/local/bin/brew"
_ARCH=$(arch)
PROMPT="$_ARCH $PROMPT"
# Requires iterm2
if [[ "$_ARCH" == "i386" ]]; then
 #usr/local is X_86
 alias bundle="arch -x86_64 bundle"
 export PATH="/usr/local/bin:$PATH"
 export PATH="/usr/local/opt:$PATH"
 export PATH="/usr/local/opt/mysql@5.7/bin:$PATH"
fi
if [[ "$_ARCH" == "arm64" ]]; then
 #usr/local is X_86
 # export GEM_HOME=$HOME/.gems
 export PATH=$HOME/.gems/bin:$PATH
 export PATH="/opt/homebrew/bin:$PATH"
 export PATH="/opt/homebrew/opt:$PATH"
 export PATH="/opt/homebrew/opt/curl/bin:$PATH"
 export PATH="/opt/homebrew/opt/imagemagick@6/bin:$PATH"
 export PATH="/opt/homebrew/opt/mysql@5.7/bin:$PATH"
 export LDFLAGS="-L/opt/homebrew/opt/libffi/lib"
 export CPPFLAGS="-I/opt/homebrew/opt/libffi/include"
 export PATH="$(brew --prefix openssl@1.1)/bin:$PATH"
 export PATH="$(brew --prefix postgresql)/bin:$PATH"
 export RUBY_CONFIGURE_OPTS="--with-openssl-dir=$(abrew --prefix openssl@1.1)"
 export PKG_CONFIG_PATH="/opt/homebrew/opt/libffi/lib/pkgconfig"
fi

######

eval "$(rbenv init -)"

PS:
i was following this tutorial, but instead using abrew, i'm using ibrew and all x86 command instead.
https://udnpico.medium.com/install-ruby-di-macbook-pro-m1-a109be60b8aa (its indonesian, but the commands are pretty clear tho)
hope this helps.

Running Python 3.8.8 within a miniforge conda environment:
Setting the environment variables didn't resolve the problem for me - each time pip install grpcio failed:
conda install grpcio directly worked

God bless you @Tenzer - it works now! I've wasted so much time googling the solution. Thank you so much. <3

I have these set:
GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1
GRPC_BUILD_WITH_BORING_SSL_ASM=
Still getting ERROR: Failed building wheel for grpcio
Grateful for any pointers!!

@rimager You would have to provide more information than that. The message you included just tells us the installation failed but no information about why. Could you take the entire output of the command you ran and put in a gist and post the link here? That would make it easier for other people to help you out.

It was a long long error about the wheel. I resolved it by removing this flag:
GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
For some reason it wanted to use another ssl library.

grpcio 1.38.1 is used to have the issue but it seems that it's finally fixed in 1.39.0 \m/

Just simply export and install through pip package installer
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1
pip install grpcio

I got a successful build using @Tenzer 's solution, with addition of CFLAGS and LDFLAGS using brew --prefix protobuf, brew --prefix openssl, brew --prefix zlib, and setting the GRPC_PYTHON_BUILD_SYSTEM_* environment variables, with arm64 homebrew and Python 3.9.1 built via pyenv's latest master.

Can you be more explicit with exactly which variables you set, and with which values?

See the first comment on this issue: https://github.com/grpc/grpc/issues/25082#issuecomment-754565768.

I have tried with this (in fish shell)
> set -x LDFLAGS "-L/opt/homebrew/opt/openssl/lib -L/opt/homebrew/opt/zlib/lib -L/opt/homebrew/opt/protobuf/lib"
> set -x CFLAGS "-I/opt/homebrew/opt/openssl/include -I/opt/homebrew/opt/zlib/include -I/opt/homebrew/opt/protobuf/include"
> set -x GRPC_PYTHON_BUILD_SYSTEM_OPENSSL 1
> set -x GRPC_PYTHON_BUILD_SYSTEM_ZLIB 1

But I still cannot install grpcio 1.41.0 (as a dep  firebase-admin)
Python 3.9.7

I was able to resolve this error via
> sudo rm -rf /Library/Developer/CommandLineTools
> sudo xcode-select -s /Applications/Xcode.app

Add
GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

worked for me, native compiled, no need to add arch -x86_64 before pip command

This works for me (M1 Air)
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

And change the default CC to clang inorder to fix gcc: error: unrecognized command-line option '-stdlib=libc++' error
export CC=/usr/bin/clang
export CXX=/usr/bin/clang++

If anyone still encounters compilation errors when trying to pip install grpcio on a M1 Mac, and none of the above suggestions work, try this:
sudo xcode-select -s /Library/Developer/CommandLineTools
This solved the problem in my case.
Also, another piece of information that might be helpful in solving this: I was able to successfully install grpcio before I had installed xcode. Seems to me like there is a mismatch in the binary tools used for the compilation; the ones that come in /Library/Developer/CommandLineTools work file, while those that come bundled with xcode do not.

Hey guys, I am appreciative of all of the advice here but unfortunately I'm having the same issue again, I don't know whether to open a new story or keep commenting on this one.
OSX 12.2.1
M1 Chip
PyEnv install of Python 3.9.5
All dev tools up to date with the latest versions (Xcode command line tools/XCode)
Brew installed latest OpenSSL, re2 etc
Installing grpcio==1.44.0
Env vars set to the following:
export GRPC_BUILD_WITH_BORING_SSL_ASM=""
export GRPC_PYTHON_BUILD_SYSTEM_RE2="1"
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL="1"
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB="1"
export<EMAIL_ADDRESS>export<EMAIL_ADDRESS>
I've spent several hours reading and trying things to no avail.
I'm trying to install grpcio=1.44.0 and it is failing with the following:

      Traceback (most recent call last):
        File "/private/var/folders/r8/xs35x2dx48b88y7msj4nybvw0000gp/T/pip-install-85s2slvh/grpcio_c2ecc2b58cd44f59b3dd132d0e055106/src/python/grpcio/commands.py", line 285, in build_extensions
          build_ext.build_ext.build_extensions(self)
        File "/Users/hunter/.pyenv/versions/3.9.5/lib/python3.9/distutils/command/build_ext.py", line 449, in build_extensions
          self._build_extensions_serial()
        File "/Users/hunter/.pyenv/versions/3.9.5/lib/python3.9/distutils/command/build_ext.py", line 474, in _build_extensions_serial
          self.build_extension(ext)
        File "/Users/hunter/.pyenv/versions/motoapi/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 202, in build_extension
          _build_ext.build_extension(self, ext)
        File "/Users/hunter/.pyenv/versions/3.9.5/lib/python3.9/distutils/command/build_ext.py", line 529, in build_extension
          objects = self.compiler.compile(sources,
        File "/private/var/folders/r8/xs35x2dx48b88y7msj4nybvw0000gp/T/pip-install-85s2slvh/grpcio_c2ecc2b58cd44f59b3dd132d0e055106/src/python/grpcio/_parallel_compile_patch.py", line 58, in _parallel_compile
          multiprocessing.pool.ThreadPool(BUILD_EXT_COMPILER_JOBS).map(
        File "/Users/hunter/.pyenv/versions/3.9.5/lib/python3.9/multiprocessing/pool.py", line 364, in map
          return self._map_async(func, iterable, mapstar, chunksize).get()
        File "/Users/hunter/.pyenv/versions/3.9.5/lib/python3.9/multiprocessing/pool.py", line 771, in get
          raise self._value
        File "/Users/hunter/.pyenv/versions/3.9.5/lib/python3.9/multiprocessing/pool.py", line 125, in worker
          result = (True, func(*args, **kwds))
        File "/Users/hunter/.pyenv/versions/3.9.5/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
          return list(map(*args))
        File "/private/var/folders/r8/xs35x2dx48b88y7msj4nybvw0000gp/T/pip-install-85s2slvh/grpcio_c2ecc2b58cd44f59b3dd132d0e055106/src/python/grpcio/_parallel_compile_patch.py", line 54, in _compile_single_file
          self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
        File "/private/var/folders/r8/xs35x2dx48b88y7msj4nybvw0000gp/T/pip-install-85s2slvh/grpcio_c2ecc2b58cd44f59b3dd132d0e055106/src/python/grpcio/commands.py", line 268, in new_compile
          return old_compile(obj, src, ext, cc_args, extra_postargs,
        File "/Users/hunter/.pyenv/versions/3.9.5/lib/python3.9/distutils/unixccompiler.py", line 120, in _compile
          raise CompileError(msg)
      distutils.errors.CompileError: command '/Library/Developer/CommandLineTools/usr/bin/clang' failed with exit code 1

      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> grpcio

@hunt3r  Same issue, M1 Monterey, Python 3.9.10
Fixed it
Inside virtualenv:
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

and then
pip install grpcio
It works

Also works with python 3.8.11 in a conda environment if followed the above steps!

This doesn't work for me

Yeah I had to do it with Conda, a pyenv environment did not work for me. Feels weird but I got past the issue for now.

Resolved install python 3.9.8 or superior :D 🚀

Here's what worked on my setup.
My configuration:

MacBook Pro (16-inch, 2021), with M1 Pro
macOS 12.3
Homebrew installed into /opt/homebrew

What worked was a combination of @matanby's comment and some CFLAGS / LDFLAGS options that worked in the past from my colleagues.
I verified and both are needed: removing either makes the compilation fail.
Other flags and compilation options weren't successful.
With this combination of things, grpcio correctly compiles on Python 3.8, 3.9, and 3.10 (installed via pyenv).
# Xcode tools made compilation crash
sudo xcode-select -s /Library/Developer/CommandLineTools
brew install pyenv  # pyenv shell setup omitted...

# Python must be compiled with these options!
CFLAGS="-I$(brew --prefix xz)/include" LDFLAGS="-L$(brew --prefix xz)/lib" pyenv install 3.10.3
pyenv global 3.10.3

# Install grpcio from source tarball
pip install -v https://files.pythonhosted.org/packages/65/75/8b706e1170e2c7b6242b1675259e47986bb4fc490f29387989a965972e6e/grpcio-1.44.0.tar.gz

@yoav-orca We currently doesn't have ARM64 machines to produce binary wheels, but we are working on a plan.

@lidizheng it is not necessary to have an ARM64 machine to compile grpcio wheels for Apple Silicon: see this solution for a way to cross-compile from Intel. 🙂

none of the solutions above worked for me. It ended up being setuptools version. grpcio installation would fail with setuptools==60+ after changiing the env setuptools version to 57.5.0, grpcio installs fine.

Adding onto @pietrodn's solution above: when installing grcpio from the source tarball, make sure to pip uninstall any existing grcpio versions first, otherwise pip may skip the build stage.
Your output for pip install should include:
  running build_py
  running build_project_metadata
  creating python_build
  creating python_build/lib.macosx-11.3-arm64-3.10
  creating python_build/lib.macosx-11.3-arm64-3.10/grpc
  copying src/python/grpcio/grpc/_compression.py -> python_build/lib.macosx-11.3-arm64-3.10/grpc
  copying src/python/grpcio/grpc/_channel.py -> python_build/lib.macosx-11.3-arm64-3.10/grpc
  copying src/python/grpcio/grpc/_common.py -> python_build/lib.macosx-11.3-arm64-3.10/grpc
  (.....)

+1 with this issue on Macbook Pro 12.2.1 version. Tried every possible "fix" suggested here, the only thing that finally worked was installing it via conda rather than pypy

I was also not able to fix this problem. The installation would be successful but importing grpc would fail (shown below)
Python 3.9.12 | packaged by conda-forge | (main, Mar 24 2022, 23:25:14)
[Clang 12.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import grpc
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/private/var/tmp/_bazel_serhat/27cfb29c86a835e3df3ccb992f3eed84/execroot/core/bazel-out/darwin_arm64-fastbuild/bin/src/python/core/main/shell.runfiles/pypi__39__grpcio_1_41_0/grpc/__init__.py", line 22, in <module>
    from grpc import _compression
  File "/private/var/tmp/_bazel_serhat/27cfb29c86a835e3df3ccb992f3eed84/execroot/intenseye/bazel-out/darwin_arm64-fastbuild/bin/src/python/core/main/shell.runfiles/pypi__39__grpcio_1_41_0/grpc/_compression.py", line 15, in <module>
    from grpc._cython import cygrpc
ImportError: dlopen(/private/var/tmp/_bazel_serhat/27cfb29c86a835e3df3ccb992f3eed84/execroot/intenseye/bazel-out/darwin_arm64-fastbuild/bin/src/python/core/main/shell.runfiles/pypi__39__grpcio_1_41_0/grpc/_cython/cygrpc.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_CFRelease'

However, installing from conda would be just fine, so I checked their instructions. I was missing some build flags via imports. The following export statements worked for me, and hope they would be helpful for you as well. Taken from here.
export GRPC_BUILD_WITH_BORING_SSL_ASM=""
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB="True"
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL="True"
export GRPC_PYTHON_BUILD_SYSTEM_CARES="False"
export GRPC_PYTHON_USE_PREBUILT_GRPC_CORE=""
export GRPC_PYTHON_BUILD_WITH_CYTHON="True"
export GRPC_PYTHON_LDFLAGS=" -framework CoreFoundation"

I lost a full day on this. Here's how I managed to make it work.
If you install python while running in arm mode it won't work. I've tried many Python versions with conda, pipenv, venv, Poetry.
You need to compile Python in x86_64 mode.
arch -x86_64 zsh
asdf install python 3.10.4

I use asdf but it should work with pyenv and similar tools.
I hope this helps.

Just for reference...
If you need to install grpcio through poetry you need to force the installation of the source package. And to make the compilation work you need to set up the following envvars before poetry install:
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1

or directly with poetry install:
GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 poetry install

The content of pyproject.toml will need the following session:
[tool.poetry.dependencies.grpcio]
python = "^3.10.0"
markers = "platform_machine == 'arm64'"
url = "{url-for-the-source-package}"  # eg. https://files.pythonhosted.org/packages/[...]/grpcio-1.46.1.tar.gz

Same issue here

I ran into this (again) today so I came back to check out the bug.
@gnossen can you provide guidance on when a fix will make it into a release so pip users will no longer encounter this?

If you have any trouble about installing python 3.x under 3.9 version, please check this article
https://chrisjune-13837.medium.com/how-to-install-python-3-x-on-apple-m1-9e77ff94266a

I was also not able to fix this problem. The installation would be successful but importing grpc would fail (shown below)
Python 3.9.12 | packaged by conda-forge | (main, Mar 24 2022, 23:25:14)
[Clang 12.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import grpc
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/private/var/tmp/_bazel_serhat/27cfb29c86a835e3df3ccb992f3eed84/execroot/core/bazel-out/darwin_arm64-fastbuild/bin/src/python/core/main/shell.runfiles/pypi__39__grpcio_1_41_0/grpc/__init__.py", line 22, in <module>
    from grpc import _compression
  File "/private/var/tmp/_bazel_serhat/27cfb29c86a835e3df3ccb992f3eed84/execroot/intenseye/bazel-out/darwin_arm64-fastbuild/bin/src/python/core/main/shell.runfiles/pypi__39__grpcio_1_41_0/grpc/_compression.py", line 15, in <module>
    from grpc._cython import cygrpc
ImportError: dlopen(/private/var/tmp/_bazel_serhat/27cfb29c86a835e3df3ccb992f3eed84/execroot/core/bazel-out/darwin_arm64-fastbuild/bin/src/python/core/main/shell.runfiles/pypi__39__grpcio_1_41_0/grpc/_cython/cygrpc.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_CFRelease'

However, installing from conda would be just fine, so I checked their instructions. I was missing some build flags via imports. The following export statements worked for me, and hope they would be helpful for you as well. Taken from here.
export GRPC_BUILD_WITH_BORING_SSL_ASM=""
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB="True"
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL="True"
export GRPC_PYTHON_BUILD_SYSTEM_CARES="False"
export GRPC_PYTHON_USE_PREBUILT_GRPC_CORE=""
export GRPC_PYTHON_BUILD_WITH_CYTHON="True"
export GRPC_PYTHON_LDFLAGS=" -framework CoreFoundation"

It works for me.

On my case I only be able to so using poetry. (for docker)

Looks like it was fixed in release v1.47.0 thanks to https://github.com/grpc/grpc/pull/30026 (and issue https://github.com/grpc/grpc/issues/28387 )
The "universal" darwin wheel is now labeled x86_64, so arm64 mac users will automatically download and compile grpcio from source

@ElliottWilson This is likely a build failure while installing from source. First, make sure that a build toolchain is installed (i.e. XCode). It seems there is currently an issue building from source when gRPC is also install via homebrew, so make sure that that is not the case either.
If the install still fails, please open a new issue with full logs included.

@gnossen I and my team has experienced gprcio source build failures many times and discovered that depending on the development stack configurations the required build flags are vastly different laptop by laptop.  Just installing XCode with command line tools and latest homebrew did not solve the problem and sometimes we could not figure out what installation/configuration triggers such difference.
It would be nice if we have a concrete guide to how to set up the grpcio build environment on macOS and/or the pre-built binary wheel distribution for Apple Silicon.

It's not quite a guide, but after lots of trial and error the invocation that worked for me was:
CC=/usr/bin/clang GRPC_BUILD_WITH_BORING_SSL_ASM="" GRPC_PYTHON_BUILD_SYSTEM_RE2=true GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=true GRPC_PYTHON_BUILD_SYSTEM_ZLIB=true pip install grpcio==1.47.0

YMMV!

I am still having this issue :(

This did not work for me either for grpcio==1.47.0, but it does for 1.44.0

@reetika-roy  answer is the only thing that saved me in tox. We have grpcio transitively as a dependency of firebase and I tried nearly everything, including env variables - but alas pinning grpcio==1.44.0 in our setup.py made it work, thanks much!

# Install grpcio from source tarball
pip install -v https://files.pythonhosted.org/packages/65/75/8b706e1170e2c7b6242b1675259e47986bb4fc490f29387989a965972e6e/grpcio-1.44.0.tar.gz

It does not work otherwise. what is the difference from pip install grpcio that fails to find wheel and compiles? @pietrodn

@pmontu in theory nothing should change in your case. If pip fails to find a compatible binary wheel, it uses the source tarball as last resort.
However keep in mind that that link is of an older version.
What are you experiencing exactly?
pip install grpcio==1.44.0 fails and pip install https://files.pythonhosted.org/packages/65/75/8b706e1170e2c7b6242b1675259e47986bb4fc490f29387989a965972e6e/grpcio-1.44.0.tar.gz works?

I had a similar issue with installing grpcio=1.47.0 but I used pipenv and Pipfile.lock
What helped me - do uninstall pipenv previously installed with pip and install it again but with homebrew:
brew install pipenv
Then, pipenv could install grpcio without any problems

@lidizheng so we are still facing this issue with grpc 1.48.1 on macOS 12 on M1.
The installation via pip install grpcio works, but after successful installation the resulting shared object cannot be loaded:
ImportError: dlopen(~/grpc/_cython/cygrpc.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '_CFRelease'

The wheel is not used during the installation, so the artefact is built on the local machine, but the results are busted.
If we export magic variables from the conda recipe, keep conda libraries installed and install Cython, then pip install grpcio produces a working artefact:
export GRPC_BUILD_WITH_BORING_SSL_ASM=""
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB="True"
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL="True"
export GRPC_PYTHON_BUILD_SYSTEM_CARES="True"
export GRPC_PYTHON_USE_PREBUILT_GRPC_CORE=""
export GRPC_PYTHON_BUILD_WITH_CYTHON="True"
export GRPC_PYTHON_LDFLAGS=" -framework CoreFoundation"

With system libraries without conda provided libraries the installation fails.
We would really appreciate if you guys can manage to upload arm64 wheels and/or at least fix your installation process to produce working artefacts on M1 by default without dark conda recipe environment variable magic.
/cc @mm-ivanovig @marns93

Binary wheels are available here: https://github.com/pietrodn/grpcio-mac-arm-build thanks to #29262 /cc @mm-ivanovig @marns93

Folks, I'm still having the issue and I believe this should be open until:

everyone has a working release, OR
there's an easy local fix.

Folks, I'm still having the issue and I believe this should be open until:

everyone has a working release, OR
there's an easy local fix.

Yes, it's still need to set env vars explicitly to install grpcio for me:
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

python version: 3.9.13
grpcio version: 1.49.0

I managed to get the grpcio Python package installed on Python 3.9.1 installed via Pyenv on a M1 macOS:
Python 3.9.1 (default, Jan  5 2021, 10:39:36)
[Clang 12.0.0 (clang-1<IP_ADDRESS>)] on darwin

Native arm64:
$ lipo -archs $(which python)
arm64

I had to set a few environment variables to get it to work:
GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

to make the installation process use the libraries I've already got on the system, and then the following to point it at OpenSSL installed via Homebrew:
CFLAGS="-I/opt/homebrew/opt/openssl/include"
LDFLAGS="-L/opt/homebrew/opt/openssl/lib"

How can I buy you a cup of coffee? you really saved me

I ran into the same problem today with the following specs:
python version: 3.10.6
grpcio version: 1.49.1
os: mac os monterey 12.6 (M1 PRO)
Solved it by uninstalling, cleaning the conda and pip cache and then reinstalling grpcio again:
pip uninstall grpcio
conda clean --all
pip cache purge
pip install grpcio

I couldn't get anything working, with the environment variables still leaving me with
Traceback (most recent call last):
  File "/Users/tom/code_tmp/clone/grpc/examples/python/helloworld/./greeter_server.py", line 19, in <module>
    import grpc
  File "/Users/tom/opt/anaconda3/envs/basemod/lib/python3.9/site-packages/grpc/__init__.py", line 22, in <module>
    from grpc import _compression
  File "/Users/tom/opt/anaconda3/envs/basemod/lib/python3.9/site-packages/grpc/_compression.py", line 15, in <module>
    from grpc._cython import cygrpc
ImportError: dlopen(/Users/tom/opt/anaconda3/envs/basemod/lib/python3.9/site-packages/grpc/_cython/cygrpc.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace (_CFRelease)

when trying to import grpc.
conda install -c conda-forge grpc gave me
(basemod) tom@MacBook-Pro helloworld % conda install grpcio -c conda-forge
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/common/logic.py", line 125, in _convert
        return self.names[name]
    KeyError: 'conda-forge/osx-arm64::certifi-2022.6.15-py38h10201cd_0'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/exceptions.py", line 1114, in __call__
        return func(*args, **kwargs)
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/cli/main.py", line 86, in main_subshell
        exit_code = do_call(args, p)
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/cli/conda_argparse.py", line 90, in do_call
        return getattr(module, func_name)(args, parser)
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/cli/main_install.py", line 20, in execute
        install(args, parser, 'install')
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/cli/install.py", line 259, in install
        unlink_link_transaction = solver.solve_for_transaction(
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/core/solve.py", line 152, in solve_for_transaction
        unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/core/solve.py", line 195, in solve_for_diff
        final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/core/solve.py", line 313, in solve_final_state
        ssc = self._add_specs(ssc)
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/core/solve.py", line 605, in _add_specs
        conflict_specs = ssc.r.get_conflicting_specs(tuple(concatv(
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/resolve.py", line 1109, in get_conflicting_specs
        C = r2.gen_clauses()
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/common/io.py", line 86, in decorated
        return f(*args, **kwds)
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/resolve.py", line 911, in gen_clauses
        nkey = C.Not(self.to_sat_name(prec))
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/common/logic.py", line 144, in Not
        return self._eval(self._clauses.Not, (x,), (), polarity, name)
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/common/logic.py", line 130, in _eval
        args = self._convert(args)
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/common/logic.py", line 120, in _convert
        return type(x)(map(self._convert, x))
      File "/Users/tom/opt/anaconda3/lib/python3.9/site-packages/conda/common/logic.py", line 127, in _convert
        raise ValueError("Unregistered SAT variable name: {}".format(name))
    ValueError: Unregistered SAT variable name: conda-forge/osx-arm64::certifi-2022.6.15-py38h10201cd_0

`$ /Users/tom/opt/anaconda3/bin/conda install grpcio -c conda-forge`

  environment variables:
                 CIO_TEST=<not set>
        CONDA_DEFAULT_ENV=basemod
                CONDA_EXE=/Users/tom/opt/anaconda3/bin/conda
             CONDA_PREFIX=/Users/tom/opt/anaconda3/envs/basemod
    CONDA_PROMPT_MODIFIER=(basemod)
         CONDA_PYTHON_EXE=/Users/tom/opt/anaconda3/bin/python
               CONDA_ROOT=/Users/tom/opt/anaconda3
              CONDA_SHLVL=1
           CURL_CA_BUNDLE=<not set>
                 INFOPATH=/opt/homebrew/share/info:
                  MANPATH=/Users/tom/.nvm/versions/node/v17.6.0/share/man:/Users/tom/software/gi
                          t-subrepo/man:/opt/homebrew/share/man::
                     PATH=/Users/tom/opt/anaconda3/bin:/Users/tom/opt/anaconda3/envs/basemod/bin
                          :/Users/tom/opt/anaconda3/condabin:/Users/tom/.nvm/versions/node/v17.6
                          .0/bin:/Users/tom/.poetry/bin:/opt/homebrew/opt/openssl@1.1/bin:/Users
                          /tom/software/git-subrepo/lib:/Users/tom/Library/Jupyter/kernels:/User
                          s/tom/Documents/code/mon/scripts:/Users/tom/.nimble/bin:/Users/tom/ins
                          tall/nim/bin:/Users/tom/Applications/Sublime Text.app/Contents/SharedS
                          upport/bin:/Users/tom/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/lo
                          cal/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion Tech
                          Preview.app/Contents/Public:/usr/local/zfs/bin
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>

     active environment : basemod
    active env location : /Users/tom/opt/anaconda3/envs/basemod
            shell level : 1
       user config file : /Users/tom/.condarc
 populated config files : /Users/tom/.condarc
          conda version : 4.13.0
    conda-build version : 3.21.9
         python version : 3.9.12.final.0
       virtual packages : __osx=12.5.1=0
                          __unix=0=0
                          __archspec=1=arm64
       base environment : /Users/tom/opt/anaconda3  (writable)
      conda av data dir : /Users/tom/opt/anaconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/anaconda/osx-arm64
                          https://conda.anaconda.org/anaconda/noarch
          package cache : /Users/tom/opt/anaconda3/pkgs
                          /Users/tom/.conda/pkgs
       envs directories : /Users/tom/opt/anaconda3/envs
                          /Users/tom/.conda/envs
               platform : osx-arm64
             user-agent : conda/4.13.0 requests/2.27.1 CPython/3.9.12 Darwin/21.6.0 OSX/12.5.1
                UID:GID : 501:20
             netrc file : None
           offline mode : False

Installing the binary wheel from the latest artifact at https://github.com/pietrodn/grpcio-mac-arm-build/actions by downloading it and
pip uninstall grpcio
cd ~/Downloads/artifacts/
pip install ./grpcio-1.48.1-cp39-cp39-macosx_11_0_arm64.whl

worked a treat. Thank you so much!! <3

I ran into the same problem today with the following specs:
python version: 3.10.6 grpcio version: 1.49.1 os: mac os monterey 12.6 (M1 PRO)
Solved it by uninstalling, cleaning the conda and pip cache and then reinstalling grpcio again:
pip uninstall grpcio
conda clean --all
pip cache purge
pip install grpcio

Confirming python 3.10.6 works on Monterey 12.5 with no env vars set. 3.9 did not seem to work for me with any of the incantations above.

Would it be fair to say that this issue is not resolved? Should there be another one opened, or this one perhaps? We've got OSS users that encounter this problem somewhat regularly

Of course the issue should be open, it is not fixed. I don't understand the trend on GitHub to close issues before a fix has been created, it is not like the issue goes away just by marking the issue closed:( Now it is almost 2 years since the issue was created and nobody cares enough about this to actually do the work needed to fix it. It is really frustrating that an important package like this can not be installed by pip without problems.
On my Mac m1 12.6 python 3.10.6 I was not able to install 1.50.0 I simply gave up trying get it compiled :( and installed 1.46.3 with the following env vars set:
GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1
GRPC_BUILD_WITH_BORING_SSL_ASM=
GRPC_PYTHON_BUILD_SYSTEM_CARES=True
GRPC_PYTHON_USE_PREBUILT_GRPC_CORE=
GRPC_PYTHON_BUILD_WITH_CYTHON=True
GRPC_PYTHON_LDFLAGS= -framework CoreFoundation
Not sure if the env vars matter though

Just to follow up on my previous comment, it turned out I could not use version 1.46.3. Anyway I managed to install version grpcio version 1.50.0. My pyenv was pulling in libraries for macOS 12.4 and after I updated pyenv and reinstalled python versions 3.10.6 (which pyenv builds from source) I could install grpcio version 1.50.0.
Here are the commands I used:
brew upgrade pyenv
pyenv uninstall 3.10.6
pyenv install 3.10.6
pyenv virtualenv 3.10.6 google
pyenv activate google
pip install grpcio
...
Installing collected packages: six, grpcio
Running setup.py install for grpcio ... done
Successfully installed grpcio-1.50.0 six-1.16.0

Would it be fair to say that this issue is not resolved? Should there be another one opened, or this one perhaps? We've got OSS users that encounter this problem somewhat regularly

Yeah it is absolutely true, I spent like 2 hours trying to install it without venv, nothing really helps...

Hello, since Google is doing nothing about this, I decided to set up a periodic GitHub Actions workflow to provide daily builds of Apple Silicon wheels for Mac. It will automatically cut a new release the day a new grpcio version is released on PyPI.
In the releases page of this repo you can find and download the wheels for Python 3.8, 3.9, 3.10, 3.11.
(Use at your own risk, these are not tested.)
https://github.com/pietrodn/grpcio-mac-arm-build

I found out that setuptools-65.5.0 seems not able to install the latest grpcio-2.10.0, so I choose to install grpcio2.9.0 and then it works.

@pietrodn Thanks 👍👍

@pietrodn the issues appear to also affect the grpcio-tools package can you add that to your build example (and the repo where you've got working builds)

@techdragon ✅  Done! https://github.com/pietrodn/grpcio-mac-arm-build/releases

Official builds seem to land in GRPC 1.52
https://github.com/grpc/grpc/pull/31747

@levrik Yes, to confirm, 1.52.0 includes prebuilt universal2 artifacts which work on both x86-64 and arm64 macs. The release candidate is already uploaded and the official release is scheduled for later this week.

1.52.0 has been released with proper universal2 artifacts which support both x86 macs and M1/2 macs.

1.52.0 has been released with proper universal2 artifacts which support both x86 macs and M1/2 macs.

Gracie 1.52.0 is Yanked

Can we re-open?

I can see that there is a new release https://github.com/grpc/grpc/releases/tag/v1.52.1 and that was announced for today.
https://github.com/grpc/grpc/issues/32306#issuecomment-1431780658
But it is not published on PyPI....

1.51.2 is an unrelated C++-only patch release for a security vulnerability affecting only the C++ API. The release process for 1.51.3, which backports universal2 artifacts, is in progress and is still estimated to be complete today.

Due to an issue with the CI build, the estimate is now Tuesday, February 28th.

1.51.3 has now been released with universal2 artifacts. I'll leave this issue open for a couple more days in case there are further issues.


ACCUMULO-4769 Sanity check for valid CryptoModule and KeyEncryptionStrategy in config

Added a check to the ConfigSanityCheck to validate the cryptomodule and the keyencryptionstrategy
Added unit tests to make sure the checks work
Removed silent failure in the CryptoModuleFactory since we have already checked for these failures when these values were set.

Can one of the admins verify this patch?

@PircDef sorry this sat so long.  I can make the asSubclass() changes if needed.  After those changes are made I can merge.

Merged in 416b743.  Used asSubclass in dce8c359056905405abaea9b99efc21dadbc5920


Model 'object_detection' is not loaded.

Hi
I retrain with MobileNet SSD V1 with 256x256 DM=0.125，but I got an issue as follw
Could anybody solve the issue ?
pi@raspberrypi:~/AIY-voice-kit-python/src/examples/vision $ ./object_detection.py --input glassskew_Resized.png --output glassskew_detected.png
Traceback (most recent call last):
File "/opt/aiy/projects-python/src/aiy/_drivers/_spicomm.py", line 128, in transact
fcntl.ioctl(self._dev, SPICOMM_IOCTL_TRANSACT, self._tbuf)
OSError: [Errno 14] Bad address
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./object_detection.py", line 39, in main
result = inference.run(image_center)
File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 100, in run
return self._engine.image_inference(self._key, image, params)
File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 327, in image_inference
return self._communicate(request).inference_result
File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 191, in _communicate
response.ParseFromString(self._transport.send(request.SerializeToString()))
File "/opt/aiy/projects-python/src/aiy/_drivers/_transport.py", line 33, in send
return self._spicomm.transact(request)
File "/opt/aiy/projects-python/src/aiy/_drivers/_spicomm.py", line 143, in transact
raise SpicommInternalError
aiy._drivers._spicomm.SpicommInternalError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./object_detection.py", line 50, in
main()
File "./object_detection.py", line 46, in main
image.save(args.output)
File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 110, in exit
self.close()
File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 103, in close
self._engine.unload_model(self._key)
File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 251, in unload_model
self._communicate(request)
File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 194, in _communicate
raise InferenceException(response.status.message)
aiy.vision.inference.InferenceException: UnloadModel: Model 'object_detection' is not loaded.

I have had this issue before, something is wrong with your model.
The chip is not able to handle. Based on my web searching, nobody knows what is wrong. It simply fails during execution.
Solution - use a model that works to start with and retrain that one.

Could you please provide the check point file & Mobile SSD V1 config file to me, then I could train it myself. thanks.

You will find what you are looking for here:
https://github.com/google/aiyprojects-raspbian/issues/402
and
https://github.com/google/aiyprojects-raspbian/issues/314

Today i retrain Embedded MobileNet V1 SSD in the tensorflow 1.10，compiled successful. But i got same problem - Model 'object_detection' is not loaded.
How to get rid of it ?

my transfer command was as follow :
./bonnet_model_compiler.par --frozen_graph_path=./Glass_skewEmbeded_MobileNet_SSD_Res256_DM_P125/Glassskew_Embeded_Model_256_256_DM_P125_20180914_1857.pb --output_graph_path=./Glass_skewEmbeded_MobileNet_SSD_Res256_DM_P125/Glassskew_Embeded_Model_256_256_DM_P125_20180914_1857.binaryproto --input_tensor_name="Preprocessor/sub" --output_tensor_names="concat,concat_1" --input_tensor_size=256 --debug

@RickZLLiao @kissste
VisionKit has very limited HW resources and it can only support particular architecture with specific configuration.
When you configure the training pipeline, start with embedded_ssd_mobilenet_v1_coco.config,
https://github.com/tensorflow/models/blob/master/research/object_detection/samples/configs/embedded_ssd_mobilenet_v1_coco.config
That one is VisionKit friendly.
We provided more details about your use case, search
How to train and deploy a customized object detection model at
https://aiyprojects.withgoogle.com/vision/#makers-guide
zhoujustin@ provided a pretrained checkpoint in issue 314
Let me know if this solves the issue or not. Thanks!

@sainttelant
Thanks for your help! Would I have to reflesh Image first ?

@sainttelant
I test 4 case，
(1)python3 test_run_model_on_bonnet.py --model_path /opt/aiy/models/mobilenet_ssd_256res_0.125_person_cat_dog.binaryproto --input_height 256 --input_width 256  ==>Default Model，It's ok.
(2)python3 test_run_model_on_bonnet.py --model_path /opt/aiy/models/Glassskew_Embeded_Model_256_256_DM_P125_20180914_1857.binaryproto ==> It's failed. (My training model - Embeded Mobile Model，256x256, depth Multiplier: 0.125 )，Error Messafe as follow...
(3)python3 test_run_model_on_bonnet.py --model_path /opt/aiy/models/pikachu_detector.binaryproto --input_height 256 --input_width 256 ==> It's failed. (.binaryproto was from https://github.com/giacomobartoli/vision-kit/tree/master/pikachu-detector)
(4)python3 test_run_model_on_bonnet.py --model_path /opt/aiy/models/cat_detector_cwh_180602.binaryproto --input_height 256 --input_width 256 ==>It's failed.(.binaryproto was from chadwallacehart's github ，https://github.com/chadwallacehart/aiy_custom_cat_detector)
Case (2)~(4)，has same error , as follow
Traceback (most recent call last):
File "/opt/aiy/projects-python/src/aiy/_drivers/_spicomm.py", line 128, in transact
fcntl.ioctl(self._dev, SPICOMM_IOCTL_TRANSACT, self._tbuf)
OSError: [Errno 14] Bad address
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test_run_model_on_bonnet.py", line 66, in main
for i, result in enumerate(camera_inference.run()):
File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 76, in run
yield self._engine.camera_inference()
File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 267, in camera_inference
return self._communicate(request).inference_result
File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 191, in _communicate
response.ParseFromString(self._transport.send(request.SerializeToString()))
File "/opt/aiy/projects-python/src/aiy/_drivers/_transport.py", line 33, in send
return self._spicomm.transact(request)
File "/opt/aiy/projects-python/src/aiy/_drivers/_spicomm.py", line 143, in transact
raise SpicommInternalError
aiy._drivers._spicomm.SpicommInternalError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test_run_model_on_bonnet.py", line 77, in main
(i, fps, ','.join(output_tensor_str)))
File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 87, in exit
self.close()
File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 79, in close
self._engine.stop_camera_inference()
File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 273, in stop_camera_inference
self._communicate(request)
File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 194, in _communicate
raise InferenceException(response.status.message)
aiy.vision.inference.InferenceException: StopCameraInference: Camera inference is not running.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test_run_model_on_bonnet.py", line 81, in
main()
File "test_run_model_on_bonnet.py", line 77, in main
(i, fps, ','.join(output_tensor_str)))
File "/usr/lib/python3/dist-packages/picamera/camera.py", line 758, in exit
self.close()
File "/usr/lib/python3/dist-packages/picamera/camera.py", line 752, in close
raise exc
File "/usr/lib/python3/dist-packages/picamera/camera.py", line 1995, in _control_callback
"No data recevied from sensor. Check all connections, "
picamera.exc.PiCameraRuntimeError: No data recevied from sensor. Check all connections, including the SUNNY chip on the camera board

@sainttelant & @weiranzhao，I need your help.

@RickZLLiao Quick thing to check, are you on the latest SD card image?

@weiranzhao，I'm not update to latestest SD card image. I'll try today, and let you know the result, thanks.

@weiranzhao，After i update  the latestest SD card image (aiyprojects-2018-08-03.img)，i got errors as follow : (No such file or directory: '/dev/vision_spicomm')
pi@raspberrypi:~/AIY-projects-python/src/examples/vision $ python3 any_model_camera.py --model_path /opt/aiy/models/Glassskew_Embeded_Model_256_256_DM_P125_20180914_1857.binaryproto --input_height 256 --input_width 256
Traceback (most recent call last):
File "any_model_camera.py", line 63, in
main()
File "any_model_camera.py", line 56, in main
with CameraInference(model) as inference:
File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 94, in init
self._engine = self._stack.enter_context(InferenceEngine())
File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 227, in init
self._transport = make_transport()
File "/opt/aiy/projects-python/src/aiy/_drivers/_transport.py", line 89, in make_transport
return _SpiTransport()
File "/opt/aiy/projects-python/src/aiy/_drivers/_transport.py", line 29, in init
self._spicomm = _spicomm.Spicomm()
File "/opt/aiy/projects-python/src/aiy/_drivers/_spicomm.py", line 289, in init
self._dev = os.open(SPICOMM_DEV, os.O_RDWR)
FileNotFoundError: [Errno 2] No such file or directory: '/dev/vision_spicomm'

@weiranzhao，I try commands as follw, but i got same error
==> sudo apt-get update; sudo apt-get upgrade; sudo reboot
==> dpkg -l aiy*
from : https://aiyprojects.withgoogle.com/help/#updating-your-sd-card

@RickZLLiao
That error seems like system issue. Could you try to reflash the sd card with latest image?

@dmitriykovalev
Any idea on the error message after reflashing?

@weiranzhao，After i update the latestest SD card image (aiyprojects-2018-08-03.img) & finishing update，I also found
message : ' Failed to start load kernel modules ' , when rebooting in the first line.

@manoj7410，thanks your responding, i'll try it. And let you know the result.

@weiranzhao
@manoj7410
@dmitriykovalev
After download the sdcard image from:  https://github.com/google/aiyprojects-raspbian/releases/tag/v20180803，i could excute my embedded_mobilenet_ssd model successfully。
I excute cat detection and pikachu detection also successfully.
Thanks all your help.


Removing sqs:SendMessage permission to Dead Letter Queue for lambda-sqs-worker

Description

In the lambda-sqs-worker template,
There is no longer a need to specifically grant sqs:SendMessage* for your lambda to your DeadLetterQueue
This is now automatically done by AWS under the hood
Therefore, we can remove this from the provider.iam yaml config
As long as the RedrivePolicy is configured in the SQS Message Queue

Validation

The apply-services team has a new worker that has this removed.
And DLQ messages are still being sent to the DLQ.
Link to serverless: https://github.com/SEEK-Jobs/apply-pipeline-job-enricher/blob/629a1e338edef91bbbbab552285c21308b347503/serverless.yml#L25
Another example of the virus-scanner-service-submitter worker also having this removed:
(https://github.com/SEEK-Jobs/virus-scanner-service/blob/379ba1fdbba4db3ddc6d8f72cd59d54014ac9c83/serverless.yml#L27)

While the bundled Lambda templates don't demonstrate this, it's relatively common for productionised Lambda functions to re-queue messages to the input and dead letter queues directly in code rather than leaving it up to invocation failures and redrive policies. In particular, AWS will throttle a failing function which can lead to further degradation.

Tech strategy on the above https://github.com/SEEK-Jobs/tech-strategy/pull/178/files#diff-3170c20a3b7f0649b0889fc63f22634e8c363dde973931d43d3638363425a2ceR196

Tech strategy on the above
https://github.com/SEEK-Jobs/tech-strategy/blob/master/docs/v1/technology.md#performance-degradation-on-lambda-function-errors
A Lambda function with high throughput requirements should handle and retry errors in code rather than bubbling them up to the Lambda service as function errors.

Thank you for the clarification @72636c and @tobyhei !
Will close this PR 👍


Why are groups' roles defined on each reference?

I'm confused, it seems wherever groups are used, roles are assigned to them.  For example:
Roles.addUsersToRoles([user1, user2], ['glorious-admin', 'perform-action'], 'example.org')

To my understanding this line is adding user1 and user2 to the 'example.org' group, which encompasses the roles 'glorious-admin' and 'perform-action'.
I would expect only the group to be assigned to the users, and the roles assigned to groups elsewhere.  That way if we decide the example.org group are no longer glorious-admins we don't have to change every line example.org is used, only the one assigning glorious-admin to example.org.
Also what happens if this line is executed right after?
Roles.addUsersToRoles([user3, user4], ['perform-action'], 'example.org')

Is example.org still glorious-admin?  What happens to user1 and users2's roles?
Can you elaborate on the semantics here? Thanks!

Hi @jdmswong, I updated the readme with a section about groups.  Does this make it more clear?

I understand now, groups don't represent collections of permissions.  You think of them as separate entities a user has specific permissions for, independent of other groups.


tensorflow_addons.losses.metric_learning.pairwise_distance. Cannot convert a symbolic Tensor (2nd_target:0) to a numpy array

Whent I try to use the pairwise_distance function is the the following error "Cannot convert a symbolic Tensor (2nd_target:0) to a numpy array".
I found that is because line 67 in the pairwise_distance function is written as:
tf.ones([num_data])
Instead of:
tf.ones((num_data))
Please check if this is correct and fix it.
The function works perfectly after I changed that line of code.

@spaghettix ,
Can you please share the link where requested change is to be made.Thanks!

Hi, do you mean this?
https://github.com/tensorflow/addons/blob/master/tensorflow_addons/losses/metric_learning.py#:~:text=tf.linalg.diag(-,tf.ones([num_data],-)


OrderedProfileService

Hello, I am a huge fan and an avid user of your ProfileService in my game Slither.io Simulator
I propose some interesting ideas concerning applying the concepts of ProfileService to a façade around OrderedDataStore.
When players exit the arena in Slither, I write their current Kill count to an OrderedDataStore
Sometimes if they frequently join and leave it is causing too many writes and I get messages in the log like "DataStore request was added to queue"
I thought it would be an interesting exercise to translate some of the features of this fabulous ProfileService lib, to similar concepts against OrderedDataStore, which has some different and unique challenges of it's own (most importantly the ability to store only a single number on each row).  The graceful way that ProfileService handles reads and writes, so as "not to go Brrr" is what I propose could also be applied to OrderedDataStore.
Would you consider creating a copy of ProfileService which is re-tooled for that purpose?  You, better than anyone, would be most knowledgeable to understand the scope and complexity involved in that endeavor.
Another proposal would be a way of tying together the OrderedDataStore with the ProfileService.  Because I guess every dev who wants to implement it is going to need both of these tools, and what we all need is a way to correlate the Profile into one or more OrderedDataStores, with all that extra sugar that we love about using ProfileService.

I agree with this, hoping to see OrderedDataStore implemented in ProfileService.

I would love to have this! I have this issue also.. Can't really find a way to mitigate the queue request.

Thank you for sharing your ideas.
ProfileService is a highly specialised project meant to make session locking on DataStores easy and, for simplicity sake, that should remain the only thing ProfileService does.


Implement custom interval between probes

Important
Please provide the requested parameters below to help us review your pull request.
Pull requests without a body or explanation will be rejected.
Describe your changes
I implemented the functionality for custon interval between probes.
Issue ticket number and link
Closes #111
Checklist before requesting a review

[ ] I have performed a self-review of my code
[ ] If it is a core feature, I have added tests.
[ ] I have run make check and there are no failures.

Type of change
Please delete options that are not relevant.

[ ] New feature (non-breaking change which adds functionality)

I don't know why I couldn't reopen #125, so here I am.

Hi @luca-patrignani,
Thanks for reopening this PR.
Overall looks very good.
I ran the program and got this output:

Please note the:

total uptime
TCPing ended at
duration (HH:MM:SS)

One of the culprits is this
https://github.com/pouriyajamshidi/tcping/blob/d9b7d6f39ff399cfbe83f851dc0597bb7b4f57eb/tcping.go#L194-L195
I am not sure how much work and fix this is just off the top of my head. Let me know your thoughts

About this, an awful workaround can be to just print the current time, isn't it?

About this, an awful workaround can be to just print the current time, isn't it?

hmm, I think that would still cause some corner cases.
One option is to get rid of this logic https://github.com/pouriyajamshidi/tcping/blob/d9b7d6f39ff399cfbe83f851dc0597bb7b4f57eb/tcping.go#L194
To calculate the total uptime, given that the time report remains as accurate as it is now (without the -i flag ofc)
That should take care of start, end and duration values.

Then remains the total uptime value which is coming from here https://github.com/pouriyajamshidi/tcping/blob/d9b7d6f39ff399cfbe83f851dc0597bb7b4f57eb/tcping.go#L745
and might be a bit more tricky to address.
I do not have a solution for it off the top of my head. If you are not feeling comfortable to address it in this PR, it is fine. I will try to allocate some time in the coming weeks to do some cleanup.
Let me know.

I think these are good solutions, I'll work on it by next week, maybe something in this weekend too.

No worries mate. Thank you.


Login page height jumping

Describe the bug
Browser: Chrome.
Device: Desktop.
Page: Login.
When you navigate to the login page (New or Old user), the height of the page is rapidly changing. This causes

the vertical scrollbar to appear and disappear every second.
The input fields shrink and expand by a little with the same frequency.

To Reproduce
Steps to reproduce the behavior:

Go to https://flextype.org/en/downloads
Download the archive version 0.9.6
Install flextype locally
Go to Admin page
Observe the jumping

The same goes for the /admin/login page.
This behavior doesn't happen on mobile (checked in the Chrome emulator).
Expected behavior
Steady page.
Screenshots

Desktop (please complete the following information):

OS: Windows 7
Browser: Chrome.
Version: 79.0.3945.88 (Official Build) (64-bit)

Smartphone (please complete the following information):
not tested
Additional context
n/a

can't reproduce in 0.9.7.


ARROW-9967: [Python] Add compute module documentation

#8163 exposes pyarrow.compute kernels and generates their docstrings. This PR adds documentation for the module in the User Guide and the Python API reference.

@arw2019 thanks a lot for working on this! (and sorry for the overlap with https://github.com/apache/arrow/pull/8163)

Question: in 952649d I'm attempting to expose PartitionNthOptions but it errors at compilation. What is (are) my mistake(s)?
This is needed to make the partition_nth_indices Python wrapper usable.

Cython generally doesn't convert implicitly between arbitrary Python objects and C/C++ types, you have to type your code explicitly. Also, it is better to use __cinit__ here so that the constructor never gets skipped, so for example (untested):
    def __cinit__(self, int64_t pivot):
        self.partition_nth_options.reset(new CPartitionNthOptions(pivot))

Cython generally doesn't convert implicitly between arbitrary Python objects and C/C++ types, you have to type your code explicitly.

Thanks @pitrou! That was the problem

This is ready for re-review.
I believe that I've addressed the feedback from previous reviews. I've also now exposed all the option classes so that all the kernels listed in the docs are usable from Python.

@github-actions crossbow submit test-ubuntu-18.04-docs

CI failures are unrelated.

Thanks @pitrou @jorisvandenbossche for reviewing!


Thread 1: signal SIGABRT

0x1052f8578 <+8>: b.lo 0x10528598； <+40>   -》〉》Thread 1: signal SIGABRT

I can see in the GitHub notification I was forwarded (altho not in what appears above) that this is a result of not finding the MarkupEditor.framework file. Obviously from the next issue you filed, you managed to fix or work around the issue. Do you know what caused it and then fixed it? If it's something I did on packaging, I'd like to know. Thx.


[TLS] Add a uText to our TLS

ICU's uText APIs offer a cross-encoding means of interacting with
grapheme breaking. Storing one on our TLS allows us to explore mised
encodings, and even for UTF-16 might be profitable if setText
mallocs/frees an internal uText (which seems to be the case). This
isn't hooked up to anything yet, but its highly likely we will be
using it soon and adding it now enables more rapid development of the
UTF-8 prototype, due to it's tie-in with the runtime.

@swift-ci please smoke benchmark

@swift-ci please test

@weissi, once this lands and gets incorporated into a toolchain, prototype development with non-fast-pathed-graphemes can continue.

ArrayOfPOD
783
858
+9.6%
0.91x

How to read the data
The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.
If you see any unexpected regressions, you should consider fixing the regressions before you merge the PR.
Noise: Sometimes the performance results (not code size!) contain false alarms. Unexpected regressions which are marked with '(?)' are probably noise. If you see regressions which you cannot explain you can try to run the benchmarks again. If regressions still show up, please consult with the performance team (@eeckstein).

Hardware Overview
  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB


[FT] Open ai endpoint

Evaluation Short Description
I am attempting to evaluate a model similar to ChatGPT using the OpenAI interface. Is there any way to accomplish this?
Why is this evaluation interesting?
This evaluation is interesting because it would allow for the assessment of models like ChatGPT or custom models using the OpenAI interface.
How used is it in the community?
I believe many people will find this useful, as it can facilitate the evaluation of various models within the OpenAI ecosystem.
Evaluation Metadata
Provide all available resources:

OpenAI Python

Hi! We now support Open AI models here!


BUG: Issue with chat ui because of gradio when using apache vhost as reverse proxy

Describe the bug
We encountered a bug when serving Xinference (since 0.8.5 & through docker) through an apache vhost with reverse proxy configuration. Issue is the dynamic resolution of paths (through gradio>=4.0.0) in the chat ui. URLs pointing to http://$HOST/$MODEL_ID/info and http://$HOST/$MODEL_ID/theme.css will be instead resolved as http://$HOST/info and http://$HOST/theme.css. This makes the chat ui unusable when served through an apache vhost.
The issue was introduced with commit #983, removing the hard requirement of gradio<4.0.0.
To Reproduce

Launch latest (0.10.0) Xinference docker image.
Install apache vhost and configure reverse proxy:

<VirtualHost *:80>
     ServerName $server_name

     #default proxy via internal port
     ProxyPreserveHost On

     RewriteEngine on
     RewriteCond %{HTTP:Upgrade} =websocket [NC]
     RewriteCond %{REQUEST_URI} ^/
     RewriteRule /(.*) ws://localhost:9997/$1 [P,L]

     RewriteRule ^/(.*)$ http://<IP_ADDRESS>:9997/$1 [P]

     Header always set Content-Security-Policy: upgrade-insecure-requests

     HostnameLookups Off

     UseCanonicalName Off

     ServerSignature On

   <Directory />
     AllowOverride None
     Options -Indexes
   </Directory>
     TraceEnable off

     Errorlog /usr/share/apache2/log/llm/llm-error.log
     CustomLog /usr/share/apache2/log/llm/llm-access.log combined

</VirtualHost>

Launch a model in the UI
Open chat UI

To Fix
I created a derived version of the official Xinference docker image as follows:
FROM xprobe/xinference:latest
RUN pip install gradio==3.50.2

Repeating step 1.-4. for this docker image will result in a working chat UI.

Therefore I propose to fix the gradio dependency to 3.50.2.

@ChengjieLi28 Do you think it's reasonable to fix the gradio version for docker build?


DOC: error in TransferFunctionDiscrete example

Omitting the dt=, passes "0.5" as part of the *system. which is
passed to normalise() which complains it gets 3 arguments instead of
2.

Please try to skip some jobs in the CI for next PRs.

Oh yes, sorry I forgot about this...


[SPARK-9013][SQL] generate MutableProjection directly instead of return a function

Why did we do this at the very beginning?

I think MutableProject will create a MutableRow inside it, it's not thread-safe, can not be used by multiple tasks. cc @marmbrus,  Is it correct?

Merged build triggered.

Merged build started.

Test build #37136 has started for   PR 7373 at commit 4e7372c.

@davies that is correct.  The goal was to be able to amortize the cost of getting the code and then create a new copy with its own row multiple times.  Though looking at the change, it doesn't seem like we actually use that functionality in practice. (and given the use of references, I'm not sure that more than one copy can actually be used in a thread safe way anymore anyway).

Test build #37136 has finished for   PR 7373 at commit 4e7372c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Merged build triggered.

I think we will do codegen at executor side for every partition, so multiple tasks won't share one codegened MutableProject, is that correct?

@cloud-fan How about local mode? We don't always create the projection inside mapPartition

@davies  which "local" do you mean? If you mean LocalBackend(--master "local[*]"), we will still serialize and deserialize task sets, if you mean "running locally", currently we only allow one-partition job to run locally.
cc @rxin, an unrelate question here, I saw we set up task context even for local execution, and set partition id to 0, is there really a case that we don't have task context?

A new question here, now we support non-deterministic/stateful expressions in codegen, so all generated class is not thread safe, not only MutableProjection.
If generated class is shared among multiply tasks, we should handle all generated classes in the way like MutableProjection did.

Merged build triggered.

Merged build started.

Test build #41924 has started for   PR 7373 at commit 62233a9.

cc @yhuai

Test build #41924 has finished for   PR 7373 at commit 62233a9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41924/
Test PASSed.

@cloud-fan can you rebase?

Test build #56328 has started for PR 7373 at commit 19f2d81.

LGTM

Test build #56328 has finished for PR 7373 at commit 19f2d81.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56328/
Test PASSed.

Merging in master. Thanks.


Join cardinality computation for cost-based nested join optimizations

Which issue does this PR close?

This PR is an initial step towards #128, although it does not close it (more info below).
Rationale for this change

DataFusion's statistics can be helpful when making cost-based optimizations (e.g. the probe ordering for hash join uses the statistics to determine which side is more 'heavy'). However there are a few operations that don't produce any statistics since it is hard to know beforehand and we don't want to await the full results before we can make the optimizations (the physical optimizer pass is for example not adaptive, but rather just done at once when planning the query with the stats we can derive statically). There have been extensive academical and practical work done on this estimation, and there are both very complex algorithms and more simpler ones. This PR is just an introduction towards implementing the basic principles of join cardinality computation from Spark's Catalyst optimizer (which AFAIK is heavily inspired from Hive's Optiq CBO) which is very simple and heavily battle-tested (initially shared by @Dandandan & @alamb on #128).
What changes are included in this PR?

The catalyst optimizer is very extensive in terms of cost calculation, so this PR is just the initial steps towards having something similar. It implements cardinality estimation (without filter selectivity, which would probably be the next follow up) for inner join and other join types which we can estimate through deriving the result from the inner join cardinality. More description on the algorithms are available in the code, as well as the blog post Spark/DataBricks team has shared.
Are there any user-facing changes?

Yes, this brings more statistics and more potential for optimization.

I think keeping the model as simple as possible is likely the most robust and lead to the least surprising plans. This PR seems to have a reasonably straightforward model so 👍
In general, there are many known limitations to a cost based optimizer (because cost models are always estimates and thus can "get it wrong" sometimes, and typically it is hard to predict / debug when this happens (because there are correlations or skew in the data, for example)).
I would personally love to see DataFusion head more towards the "dynamically optimize at runtime" approach for joins (like have the join operators themselves scan the inputs to see if they can determine which is smaller, etc).
That being said, the reason that CBO is so prevalent is that it is relatively simple to implement and well understood, so I don't have any objections to pursuing a more sophisticated cost model for DataFusion
Thank you for this @isidentical

I will try and find some time tomorrow to review this in detail. It is hard to find the time to review all the PRs!

I will try and find some time tomorrow to review this in detail. It is hard to find the time to review all the PRs!

Thanks so much for all the reviews today ❤ Let me know if there is anything that I can do to shed more light on (in terms of implementation), would be happy to assist in the review process as much as I can.

I left some comment / style suggestions but nothing required -- @isidentical  please let me know if you would like to merge this PR as is or make changes.

@alamb thanks so much for all the reviews, I addressed all the stylistic points. Regarding the logical part (exponential join_selectivity, filter selectivity, cartesian product at the worst case scenario), I've tried to explain the current reasoning and potential for future work, but I can also start including some in this PR as well (depending on what is easier for you folks to review).

Thanks again @isidentical  -- I think this is a great step forward

I think the stats estimation should be applied to LogicalPlans instead of PhysicalPlans

I think the stats estimation should be applied to LogicalPlans instead of PhysicalPlans

Could you explain why? The estimation moved to physical plans at some point, as at the moment of a logical plan there are often no statistics available.

By the way, should we try to keep the discussion in a single place for making it easier to follow/see by everyone? (we already have an open thread in the design doc, so maybe we should either move it to a new issue to discuss further (on top of the existing discussionS in #962) or keep it in the document)

Sure, let's discuss the topic in the google doc.


feat: validate input media type for oras.PackManifest

Resolves: #571

Will split this into 2 PRs. Closing this for now.


Simplify synchronization logic of session switching and initialization

Right now RSession deals with 3 different locks which cause different types of synchronization issues.
This needs to be simplified.

We should do this soon rather than later. Related: #2599, #2504

Already working on it, though looks like #2599 is fixed.


Collect multiple text messages and sent in single mail

collect text messages for a configurable time until a mail is sent by debug bot that summarizes all messages
see #886

Cleaning up old issues


ClassCastException mapd 1.0.8

In mapdb 1.0.7 this works fine
private Var<byte[]> lastBlockVar;
private byte[] lastBlockSignature;
this.lastBlockVar = database.getAtomicVar("lastBlock");
this.lastBlockSignature = lastBlockVar.get();
(here null is returned, which is OK)
In mapdb 1.0.8 an empty String is returned but should be null (starting from scratch).
java.lang.ClassCastException: java.lang.String cannot be cast to [B
at database.BlockMap.(BlockMap.java:46)
at database.DBSet.(DBSet.java:95)

https://github.com/Qoracoin/Qora/blob/0.22.0/Qora/src/database/BlockMap.java

There were some problems with persisting nulls. So I changed default value for Atomic.Var from null to "". Solution for this is to supply your own default value as second parameter: DB.createAtomicVar("lastBlock",new byte[0]);
Or use 2.0 which has this problem solved and allows null.s


Feature request: Support read-only mode for CodeEditor

Tell us how you think we can improve Sandpack
In some cases it would be useful to control the editability of the code editor.
As of now (0.12.0), we can use either a CodeEditor or a CodeViewer.
Feature request #97 is somewhat related to this (file-based editability), but this feature is easier to implement because the underlying CodeMirror already supports a "ReadOnly" property.
Packages affected

[ ] sandpack-client
[x] sandpack-react

How would your idea work?
Add a new readOnly CodeEditor prop (default: false) which is then passed to the underlying CodeMirror which already supports this property.
When/If #97 is implemened, it would just mean that the editability of every bundled file will default to the current CodeEditor editability.

If it helps, I am ready to implement it myself.

Hey @andreifloricel! I would totally encourage you to implement it by yourself, and I would be more than happy to guide you through it. But unfortunately, Sandpack is in the middle of a relicensing process, and we can't accept core contributions for the time being.
So, I'll try to spend some time on this, maybe during this week. The CodeMirror component already accepts a read-only mode, so it should not be hard to implement it. This is what I have in mind:

SandpackCodeEditor accepts read-only mode;
Be able to set read-only by file #97;
Figure out a visual element saying that a file is read-only;

Please, let me know if you have suggestions or requests related to it.

Hi @danilowoz,
I agree with the first 2 items, not sure I understand the 3. one (...Figure out a visual element saying that a file is read-only...).
As you said, theoretically it shouldn't be hard to implement it both cases as:

CodeMirror already accepts a read-only
SandpackCodeEditor could just pass a global read-only prop
file-based read-only should be easy to achieve by checking the activePath against the bundled files configuration (the SandpackBundlerFile would need a new readOnly property)


Rake task for testing if documentation complete

I'd find it awefully helpful to have a Rake task ready for checking if 100% of my project is documented. This task could also be executed by some CI server like Jenkins or Bamboo. Having a complete documentation is in my eyes a requirement for a successful build.
My Rake task doing this currently looks like this:
begin
  require 'yard'
  desc 'Run yard documentation checks'
  task :yard do
    stdout_str, _stderr_str, status = Open3.capture3("yard stats")
    fail 'Generating docs failed' unless status == 0
    # Match if >0 undocumented or <100% documented is found in output
    if stdout_str.match(/[1-9]+ undocumented/) || stdout_str.match(/^\s*[0-9]{1,2}(\.[0-9]{1,2})*% documented/)
      puts stdout_str
      fail "The project is not fully documented. Check with 'yard stats'"
    end
  end
rescue LoadError
  puts '>>>>> Yard gem not loaded, omitting tasks' unless ENV['CI']
end

With this .yardopts file
--no-output
--no-cache
--list-undoc
--exclude ./test/*
--exclude ./spec/*
--exclude ./*_spec.rb
--exclude ./*matchers.rb
./**/*.rb

The output of yard stats --list-undoc will list undocumented methods, you can check for empty output there.
You can also do this without a subprocess:
require 'yard'

# ideally you've already generated the database to .load it
# if not, have this task depend on the docs task.
YARD::Registry.load
objs = YARD::Registry.select do |o|
  # your rules for what "undocumented" means here,
  # possibly just:
  o.docstring.blank?
end
raise "undocumented methods" unless objs.empty?

I will also add that, in practice, breaking your build on a non-functional check incentivizes you to find the quickest solution to solve the problem, and the quickest solution is rarely "correct documentation". More realistically, it will involve tricking your "blind" coverage tool into thinking everything is fine, which usually means adding things like # TODO document this later, or even the more insidious # split will split a string..
If it's failing at CI, it's too late. It means you failed to incorporate documentation in your development cycle from the start. This stuff should be in your unit tests if you care that much about it, IMO.

Marking as WontFix. YARD is not a coverage tool, but you can certainly write YARD plugins that are!


bug:  collapse arrow  in RTL mode transition issue

What version of daisyUI are you using?
V.2.49.0
Describe your issue
The icon transition is not corrected by a collapse arrow in RTL mode.
https://user-images.githubusercontent.com/18399665/215311980-f24861d7-50e3-4019-891e-a87b1fcf335c.mov
What browsers are you seeing the problem on?
Chrome, Safari, Firefox, Other
Reproduction URL (optional)
https://play.tailwindcss.com/F6gJTHZyTb

Fixed in the latest version 2.50.0: https://play.tailwindcss.com/qLSKfCDk6k
But if your page is RTL, you should have dir=rtl in your root element of course.


Alignment of attributed string

I noticed attributed string does not take alignment into account. I put together a quick patch to add alignment to attributed string only if paragraph style is not present in the attributed string. Although I could see why this may be not a good idea and may not work in some cases...
Let me know if you'd be interested in this patch and I'll prepare a pull request.

+1


Fix ftxvalidator test.

This is a version of the ftxvalidator test that I believe works much better. I'll put some comments for clarification into the diffs.

Cool! Thanks :-D

@graphicore be sure to check this page a while after pushing a new commit:
https://travis-ci.org/googlefonts/fontbakery/requests

ok, good. But ftx_data should be aligned to 'kATS[...]' otherwise we get a new flake8 error:

Oh, that should get fixed in flake8 ;-) I consider this bad style!

Ok, thanks! I was just inspecting the flake8 thing though. I get some interesting errors here on my machine, especially:
fontbakery-check-ttf.py:1751:42: E901 SyntaxError: invalid syntax

for the line:
    except subprocess.CalledProcessError, e:

I think the accepted syntax is:
    except subprocess.CalledProcessError as e:

I called it like this flake8  fontbakery-check-ttf.py --ignore=E111,E114,E701,E121,E266,E221,E226,E126 so maybe it's a version difference, I'm using my distros flake8.


Added Search Bar for searching components

Fixes Issue🛠

Closes #174
Description👨💻
Added search bar.

Type of change📄

[X] New feature (non-breaking change which adds functionality)

How this has been tested✅
tested locally on my machine.

Checklist✅

[X] My code follows the style guidelines of this project
[X] I have performed a self-review of my own code
[X] I have commented my code, particularly in hard-to-understand areas
[X] I am an Open Source Contributor

The search bar should not come in this home page, it should come only after clicking Get Components and that too in the components page. Keep your code after this line
Make sure to not close the PR again and again else you will be unassigned for spamming the PRs @pranjal12-web

@Rakesh9100
Actual there were conflicts in previous branch , which coudn't be resolved so I closed the PR.

Merged 🎉 @pranjal12-web


Tests not successful on Travis

I have mostly fixed the tests on Travis CI.
However, there still seems to be an issue with the version of sqlite3, maybe that can also be updated on Travis, or use postgres instead.

fixed in commits:
6c94dd18a99f331a56b28095fe1499618e75a2c1
8b0123f4ecbee7a13ba3d01a47f5df6eb56a08d0


Update w3id redirects & webhooks

Leave one vocab pointing to test.skohub.io for testing purposes.


Adding solution for 20

Fixes #20

My code sample is in :

[x] CPP
[ ] Java

Screenshot of working code sample  :

Checklist

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] My code sample is formatted and works correctly

needs formatting