query for libaio package using known package managers

This depends on PR https://github.com/microsoft/DeepSpeed/pull/1247
It extends that PR to add package manager detection and corresponding error messages on which package to install.  This is related to https://github.com/microsoft/DeepSpeed/issues/1126#issuecomment-854101593, though it has modified things to only print messages based on the package manager.

@stas00 , I have these two PRs for the libaio search.  This is a bit different from the original plan we discussed.
The distutils.ccompiler test seemed to work well for both my conda install and a system install where libaio is available in the default search paths, which should be the case for most package managers.  So I opted to use that as the first search method.
This adds the package manager query step, though now it's used to help print a message to the user which may help them resolve the issue rather than a check as to whether the build will actually work.
Please let me know if you still think we should flip this.

The ./install.sh build also works in my environment, so I'm not able to reproduce the same failure mode.  This has been reworked to include try/catch statements using different ccompiler methods.  I've also added some print statements to perhaps shed more light on things.
@tjruwase , when you get a chance, would you please try your build again this lastest version?

when it works for Tunji, please ping me and I will run a test on my box too before merging.

@adammoody, I am so sorry! I previously shared a number of incorrect feedback on this PR as I totally misread my logs. Apologies for the unnecessary alarm and confusion from me and the extra work on you. Let me try to set the record straight.

What self.has_function() was reporting in the build log are compiler warnings, not compiler errors. I recognize that there is a big difference between this two. Moreover, given that the purpose of self.has_function(), as you explained, is simply to use linker errors to detect the presence/absence of a function symbol, then I think it is okay to ignore preceding compiler warnings since they don't prevent the linker step from succeeding.
These compiler warnings did not fail the entire build, as I previously claimed. I simply did not read the previous log carefully.  This line from the previous log showed that the async_io is OKAY, i.e. deemed compatible by the build process.

worker-0: async_io ............... [NO] ....... [OKAY]

Still on the previous build, the libaio extension (i.e., deepspeed.ops.aio) was not disabled and is infact usable. I verified this by successfully running the associated unit test as follows:

pytest tests/unit/test_aio.py
======================================================= test session starts ========================================================
platform linux -- Python 3.6.10, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir:
plugins: forked-1.3.0, pythonpath-0.7.3, cov-2.10.1, hypothesis-4.50.8
collected 28 items

tests/unit/test_aio.py ............................                                                                          [100%]

================================================== 28 passed in 361.58s (0:06:01) ==================================================

With the latest updates to this branch, the main difference I see are the extra log messages showing the entire failing compiler command line.

worker-0: Compiler default opts: ['gcc', '-pthread', '-B', '/opt/conda/compiler_compat', '-Wl,--sysroot=/', '-Wsign-compare', '-DNDEBUG', '-g', '-fwrapv', '-O3', '-Wall', '-Wstrict-prototypes']
worker-0: Compiler (so) default opts: ['gcc', '-pthread', '-B', '/opt/conda/compiler_compat', '-Wl,--sysroot=/', '-Wsign-compare', '-DNDEBUG', '-g', '-fwrapv', '-O3', '-Wall', '-Wstrict-prototypes', '-fPIC']
worker-0: Linker default opts: ['gcc', '-pthread', '-B', '/opt/conda/compiler_compat', '-Wl,--sysroot=/']
worker-0: Linker (so) default opts: ['gcc', '-pthread', '-shared', '-B', '/opt/conda/compiler_compat', '-L/opt/conda/lib', '-Wl,-rpath=/opt/conda/lib', '-Wl,--no-as-needed', '-Wl,--sysroot=/']
worker-0: Tempfile: /tmp/tmpbs6r3xbl/test.c
worker-0: Test program: int main(int argc, char** argv) { io_submit(); return 0; }
worker-0: CFLAGS: []
worker-0: creating tmp/tmpbs6r3xbl
worker-0: gcc -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -c /tmp/tmpbs6r3xbl/test.c -o tmp/tmpbs6r3xbl/test.o
worker-0: /tmp/tmpbs6r3xbl/test.c: In function 'main':
worker-0: /tmp/tmpbs6r3xbl/test.c:1:35: warning: implicit declaration of function 'io_submit' [-Wimplicit-function-declaration]
worker-0:  int main(int argc, char** argv) { io_submit(); return 0; }
worker-0:                                    ^~~~~~~~~
worker-0: LDFLAGS: []
worker-0: gcc -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ tmp/tmpbs6r3xbl/test.o -laio -o /tmp/tmpbs6r3xbl/a.out
worker-0: Cleaning up temp directory for the test C program

@adammoody, once again I do apologize for the unnecessary alarm and extra work.
@stas00, please go ahead with testing on your side.
Going forward, I wonder if we should suppress the expected compiler warnings (i.e., -Wimplicit-int, -Wimplicit-function-declaration) in the build process since they are expected behavior of self.has_function().  What do you guys think?

So I went ahead to test my proposal of suppressing these specific compiler warnings using the following change to this branch:
diff --git a/op_builder/builder.py b/op_builder/builder.py
index 033a3b6..db0cfef 100644
--- a/op_builder/builder.py
+++ b/op_builder/builder.py
@@ -204,7 +204,7 @@ class OpBuilder(ABC):
             #os.dup2(filestderr.fileno(), sys.stderr.fileno())

             # Attempt to compile the C program into an object file.
-            cflags = []
+            cflags = ['-Wno-implicit-int', '-Wno-implicit-function-declaration']
             if 'CFLAGS' in os.environ:
                 cflags = os.environ['CFLAGS'].split()
             print('CFLAGS:', cflags)

And now the corresponding build log snippet looks like the following. What do you think?
worker-0: Testing for io_submit
worker-0: Compiler default opts: ['gcc', '-pthread', '-B', '/opt/conda/compiler_compat', '-Wl,--sysroot=/', '-Wsign-compare', '-DNDEBUG', '-g', '-fwrapv', '-O3', '-Wall', '-Wstrict-prototypes']
worker-0: Compiler (so) default opts: ['gcc', '-pthread', '-B', '/opt/conda/compiler_compat', '-Wl,--sysroot=/', '-Wsign-compare', '-DNDEBUG', '-g', '-fwrapv', '-O3', '-Wall', '-Wstrict-prototypes', '-fPIC']
worker-0: Linker default opts: ['gcc', '-pthread', '-B', '/opt/conda/compiler_compat', '-Wl,--sysroot=/']
worker-0: Linker (so) default opts: ['gcc', '-pthread', '-shared', '-B', '/opt/conda/compiler_compat', '-L/opt/conda/lib', '-Wl,-rpath=/opt/conda/lib', '-Wl,--no-as-needed', '-Wl,--sysroot=/']
worker-0: Tempfile: /tmp/tmp871com4z/test.c
worker-0: Test program: int main(int argc, char** argv) { io_submit(); return 0; }
worker-0: CFLAGS: ['-Wno-implicit-int', '-Wno-implicit-function-declaration']
worker-0: creating tmp/tmp871com4z
worker-0: gcc -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Wno-implicit-int -Wno-implicit-function-declaration -c /tmp/tmp871com4z/test.c -o tmp/tmp871com4z/test.o
worker-0: LDFLAGS: []
worker-0: gcc -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ tmp/tmp871com4z/test.o -laio -o /tmp/tmp871com4z/a.out
worker-0: Cleaning up temp directory for the test C program
worker-0: async_io ............... [NO] ....... [OKAY]

with libaio preinstalled I get:
    Running setup.py (path:/mnt/nvme1/code/github/00optimize/deepspeed/setup.py) egg_info for package from file:///mnt/nvme1/code/github/00optimize/deepspeed
    Created temporary directory: /tmp/pip-pip-egg-info-xsatlyf7
    Running command python setup.py egg_info
    DS_BUILD_OPS=0
    Testing for io_submit
    Compiler default opts: ['gcc', '-pthread', '-B', '/home/stas/anaconda3/envs/py38-pt19/compiler_compat', '-Wl,--sysroot=/', '-Wsign-compare', '-DNDEBUG', '-g', '-fwrapv', '-O3', '-Wall', '-Wstrict-prototypes']
    Compiler (so) default opts: ['gcc', '-pthread', '-B', '/home/stas/anaconda3/envs/py38-pt19/compiler_compat', '-Wl,--sysroot=/', '-Wsign-compare', '-DNDEBUG', '-g', '-fwrapv', '-O3', '-Wall', '-Wstrict-prototypes', '-fPIC']
    Linker default opts: ['gcc', '-pthread', '-B', '/home/stas/anaconda3/envs/py38-pt19/compiler_compat', '-Wl,--sysroot=/']
    Linker (so) default opts: ['gcc', '-pthread', '-shared', '-B', '/home/stas/anaconda3/envs/py38-pt19/compiler_compat', '-L/home/stas/anaconda3/envs/py38-pt19/lib', '-Wl,-rpath=/home/stas/anaconda3/envs/py38-pt19/lib', '-Wl,--no-as-needed', '-Wl,--sysroot=/']
    Tempfile: /tmp/tmp7b212fef/test.c
    Test program: int main(int argc, char** argv) { io_submit(); return 0; }
    CFLAGS: []
    creating tmp
    creating tmp/tmp7b212fef
    gcc -pthread -B /home/stas/anaconda3/envs/py38-pt19/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -c /tmp/tmp7b212fef/test.c -o tmp/tmp7b212fef/test.o
    /tmp/tmp7b212fef/test.c: In function ‘main’:
    /tmp/tmp7b212fef/test.c:1:35: warning: implicit declaration of function ‘io_submit’ [-Wimplicit-function-declaration]
        1 | int main(int argc, char** argv) { io_submit(); return 0; }
          |                                   ^~~~~~~~~
    LDFLAGS: []
    gcc -pthread -B /home/stas/anaconda3/envs/py38-pt19/compiler_compat -Wl,--sysroot=/ tmp/tmp7b212fef/test.o -laio -o /tmp/tmp7b212fef/a.out
    Cleaning up temp directory for the test C program
    Testing for io_submit
    Compiler default opts: ['gcc', '-pthread', '-B', '/home/stas/anaconda3/envs/py38-pt19/compiler_compat', '-Wl,--sysroot=/', '-Wsign-compare', '-DNDEBUG', '-g', '-fwrapv', '-O3', '-Wall', '-Wstrict-prototypes']
    Compiler (so) default opts: ['gcc', '-pthread', '-B', '/home/stas/anaconda3/envs/py38-pt19/compiler_compat', '-Wl,--sysroot=/', '-Wsign-compare', '-DNDEBUG', '-g', '-fwrapv', '-O3', '-Wall', '-Wstrict-prototypes', '-fPIC']
    Linker default opts: ['gcc', '-pthread', '-B', '/home/stas/anaconda3/envs/py38-pt19/compiler_compat', '-Wl,--sysroot=/']
    Linker (so) default opts: ['gcc', '-pthread', '-shared', '-B', '/home/stas/anaconda3/envs/py38-pt19/compiler_compat', '-L/home/stas/anaconda3/envs/py38-pt19/lib', '-Wl,-rpath=/home/stas/anaconda3/envs/py38-pt19/lib', '-Wl,--no-as-needed', '-Wl,--sysroot=/']
    Tempfile: /tmp/tmp24l1d9ti/test.c
    Test program: int main(int argc, char** argv) { io_submit(); return 0; }
    CFLAGS: []
    creating tmp/tmp24l1d9ti
    gcc -pthread -B /home/stas/anaconda3/envs/py38-pt19/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -c /tmp/tmp24l1d9ti/test.c -o tmp/tmp24l1d9ti/test.o
    /tmp/tmp24l1d9ti/test.c: In function ‘main’:
    /tmp/tmp24l1d9ti/test.c:1:35: warning: implicit declaration of function ‘io_submit’ [-Wimplicit-function-declaration]
        1 | int main(int argc, char** argv) { io_submit(); return 0; }
          |                                   ^~~~~~~~~
    LDFLAGS: []
    gcc -pthread -B /home/stas/anaconda3/envs/py38-pt19/compiler_compat -Wl,--sysroot=/ tmp/tmp24l1d9ti/test.o -laio -o /tmp/tmp24l1d9ti/a.out
    Cleaning up temp directory for the test C program

If I uninstall libaio-dev:
    Running setup.py (path:/mnt/nvme1/code/github/00optimize/deepspeed/setup.py) egg_info for package from file:///mnt/nvme1/code/github/00optimize/deepspeed
    Created temporary directory: /tmp/pip-pip-egg-info-femeyzk_
    Running command python setup.py egg_info
    DS_BUILD_OPS=0
    Testing for io_submit
    Compiler default opts: ['gcc', '-pthread', '-B', '/home/stas/anaconda3/envs/py38-pt19/compiler_compat', '-Wl,--sysroot=/', '-Wsign-compare', '-DNDEBUG', '-g', '-fwrapv', '-O3', '-Wall', '-Wstrict-prototypes']
    Compiler (so) default opts: ['gcc', '-pthread', '-B', '/home/stas/anaconda3/envs/py38-pt19/compiler_compat', '-Wl,--sysroot=/', '-Wsign-compare', '-DNDEBUG', '-g', '-fwrapv', '-O3', '-Wall', '-Wstrict-prototypes', '-fPIC']
    Linker default opts: ['gcc', '-pthread', '-B', '/home/stas/anaconda3/envs/py38-pt19/compiler_compat', '-Wl,--sysroot=/']
    Linker (so) default opts: ['gcc', '-pthread', '-shared', '-B', '/home/stas/anaconda3/envs/py38-pt19/compiler_compat', '-L/home/stas/anaconda3/envs/py38-pt19/lib', '-Wl,-rpath=/home/stas/anaconda3/envs/py38-pt19/lib', '-Wl,--no-as-needed', '-Wl,--sysroot=/']
    Tempfile: /tmp/tmpkmtnddn5/test.c
    Test program: int main(int argc, char** argv) { io_submit(); return 0; }
    CFLAGS: []
    creating tmp/tmpkmtnddn5
    gcc -pthread -B /home/stas/anaconda3/envs/py38-pt19/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -c /tmp/tmpkmtnddn5/test.c -o tmp/tmpkmtnddn5/test.o
    /tmp/tmpkmtnddn5/test.c: In function ‘main’:
    /tmp/tmpkmtnddn5/test.c:1:35: warning: implicit declaration of function ‘io_submit’ [-Wimplicit-function-declaration]
        1 | int main(int argc, char** argv) { io_submit(); return 0; }
          |                                   ^~~~~~~~~
    LDFLAGS: []
    gcc -pthread -B /home/stas/anaconda3/envs/py38-pt19/compiler_compat -Wl,--sysroot=/ tmp/tmpkmtnddn5/test.o -laio -o /tmp/tmpkmtnddn5/a.out
    /home/stas/anaconda3/envs/py38-pt19/compiler_compat/ld: cannot find -laio
    collect2: error: ld returned 1 exit status
    Failed to link a test C program
    Cleaning up temp directory for the test C program
     [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
     [WARNING]  async_io: please install the libaio-dev package with dpkg
     [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
    Testing for io_submit
    Compiler default opts: ['gcc', '-pthread', '-B', '/home/stas/anaconda3/envs/py38-pt19/compiler_compat', '-Wl,--sysroot=/', '-Wsign-compare', '-DNDEBUG', '-g', '-fwrapv', '-O3', '-Wall', '-Wstrict-prototypes']
    Compiler (so) default opts: ['gcc', '-pthread', '-B', '/home/stas/anaconda3/envs/py38-pt19/compiler_compat', '-Wl,--sysroot=/', '-Wsign-compare', '-DNDEBUG', '-g', '-fwrapv', '-O3', '-Wall', '-Wstrict-prototypes', '-fPIC']
    Linker default opts: ['gcc', '-pthread', '-B', '/home/stas/anaconda3/envs/py38-pt19/compiler_compat', '-Wl,--sysroot=/']
    Linker (so) default opts: ['gcc', '-pthread', '-shared', '-B', '/home/stas/anaconda3/envs/py38-pt19/compiler_compat', '-L/home/stas/anaconda3/envs/py38-pt19/lib', '-Wl,-rpath=/home/stas/anaconda3/envs/py38-pt19/lib', '-Wl,--no-as-needed', '-Wl,--sysroot=/']
    Tempfile: /tmp/tmpbjngta83/test.c
    Test program: int main(int argc, char** argv) { io_submit(); return 0; }
    CFLAGS: []
    creating tmp/tmpbjngta83
    gcc -pthread -B /home/stas/anaconda3/envs/py38-pt19/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -c /tmp/tmpbjngta83/test.c -o tmp/tmpbjngta83/test.o
    /tmp/tmpbjngta83/test.c: In function ‘main’:
    /tmp/tmpbjngta83/test.c:1:35: warning: implicit declaration of function ‘io_submit’ [-Wimplicit-function-declaration]
        1 | int main(int argc, char** argv) { io_submit(); return 0; }
          |                                   ^~~~~~~~~
    LDFLAGS: []
    gcc -pthread -B /home/stas/anaconda3/envs/py38-pt19/compiler_compat -Wl,--sysroot=/ tmp/tmpbjngta83/test.o -laio -o /tmp/tmpbjngta83/a.out
    /home/stas/anaconda3/envs/py38-pt19/compiler_compat/ld: cannot find -laio
    collect2: error: ld returned 1 exit status
    Failed to link a test C program
    Cleaning up temp directory for the test C program
     [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
     [WARNING]  async_io: please install the libaio-dev package with dpkg
     [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.

So the only nuance from my side is that many Ubuntu users

may not know what dpkg is, as apt is the package manager
dpgk doesn't fetch packages, apt does

Could we expand that lookup table to have an entry for the program that we tell users to use? e.g. on Ubuntu that would be:
 [WARNING]  async_io: please install the libaio-dev package with apt

alternatively just dropping the package manager name?
 [WARNING]  async_io: please install the libaio-dev system package

note I changed the wording to system package, so users won't try to do pip install

But while we are at it, as I raised this issue in the past. Why are we issuing a warning and not an exception? We know DS_BUILD_AIO=1 will fail to build, and the warning is much harder to see.
Any reason why not to make it an exception during the build?
this shouldn't succeed w/o libaio, yet it does:
DS_BUILD_AIO=1 pip install -e . --global-option="build_ext" --global-option="-j8" --no-cache -v --disable-pip-version-check

and then will fail at runtime.

@stas00, so should this case fail because we are building only one extension? I believe by default, we test compatibility of all extensions even though user might not need all of them.

Yes, this is what I was suggesting. So if DS_BUILD_AIO=1 (or BUILD_ALL) and the setup is incompatible it should assert.
So this is an additional suggestion.
And if it's just checking  for compatibility then warning is right.

Yes, this is what I was suggesting. So if DS_BUILD_AIO=1 (or BUILD_ALL) and the setup is incompatible it should assert.
So this is an additional suggestion.
And if it's just checking for compatibility then warning is right.

Agreed. @adammoody, is this something that is easy to add to this PR or do you think it should be a separate PR? Thanks!

Thanks for running those tests, @tjruwase and @stas00 .
I've disabled the prints and also enabled the mechanism that redirects stderr to a file, which is a bigger hammer to silence compile and linker output.
I added a verbose=False argument to the has_function interface that lets one turn on some messages.  In particular, I think this should print the actual compile and link lines that are used, and it does not redirect stderr.  So that should help in debugging, though there is no easy way to enable it as a user just yet.
@stas00 , you can see there are also except statements that can catch compile and link errors separately, in case we want to add more specific messages there.

Another potential problem with my existing code is that if someone happens to include an argument with an embedded space in CFLAGS or LDFLAGS, like CFLAGS=-DVAL='HELLO WORLD', then the string split() in there will not do the right thing.  I found I had to manually add those to pass them through to the compile and link steps.  In which case, the test might report a failure when the library does actually exist.
I know Python and/or PyTorch must be doing something similar, because I know I had to set those variables to point to my libaio.  With some digging, we could likely figure that out.

@stas00 , I'm happy to change the missing package messages about installing libaio-dev.  Do you have a preference between the two you suggested?
I can also change things to throw an exception if the aio op was requested.  Though I might need some more direction to do that correctly.

As you commented elsewhere, here is the solution: https://stackoverflow.com/a/79985/9201239
import shlex
shlex.split(os.environ['CFLAGS'])

you can also recode the above multline into:
cflags = shlex.split(os.environ.get('CFLAGS', ""))

no need for if

@stas00 , I'm happy to change the missing package messages about installing libaio-dev. Do you have a preference between the two you suggested?

I think the first approach would be the least misleading, the 2nd is just faster to implement but it's not as clear.

may I suggest to use a normal push and not force push? This way all the previous experiments get preserved and can be re-used / rolled back to, or seen what has been tried and abandoned. At the end the PR gets squashed into one commit anyway, so there is no harm in having a 100 commits while working on the PR.
Also I'm not sure it preserves the code comments if the code gets wiped out via force-push, if I'm not mistaken those get lost.
Just a suggestion of course, it's your PR :)

may I suggest to use a normal push and not force push? This way all the previous experiments get preserved and can be re-used / rolled back to, or seen what has been tried and abandoned. At the end the PR gets squashed into one commit anyway, so there is no harm in having a 100 commits while working on the PR.

Oh, sure.  No problem.  I was worried about all of those intermediate commits getting merged in, not knowing they'd be squashed.  Good to know I can keep those in.

@stas00 , I'm happy to change the missing package messages about installing libaio-dev. Do you have a preference between the two you suggested?

I think the first approach would be the most directive, the 2nd is just faster to implement but it's not as clear.
Thank you.

Added this naming apt, pacman, and yum.  Is pacman the right tool to name for archlinux?

re: archlinux - it looks like pacman through and through: https://wiki.archlinux.org/title/pacman#Installing_packages
re: redhat, is yum the standard package manager these days? I haven't used redhat in decades, but from what I remember rpm was a standalone package manager.
This is different with apt, where dpkg is low-level tool and apt is the package manager.

I can also change things to throw an exception if the aio op was requested. Though I might need some more direction to do that correctly.

@adammoody, thanks for offering to help with this. On second thought, I think it might make sense as a separate PR as we need that support for all deepspeed ops. Also, I feel this PR is probably heavy enough as is.
grep -r DS_BUILD_ op_builder/*.py
op_builder/async_io.py:    BUILD_VAR = "DS_BUILD_AIO"
op_builder/cpu_adam.py:    BUILD_VAR = "DS_BUILD_CPU_ADAM"
op_builder/fused_adam.py:    BUILD_VAR = "DS_BUILD_FUSED_ADAM"
op_builder/fused_lamb.py:    BUILD_VAR = 'DS_BUILD_FUSED_LAMB'
op_builder/quantizer.py:    BUILD_VAR = "DS_BUILD_QUANTIZER"
op_builder/sparse_attn.py:    BUILD_VAR = "DS_BUILD_SPARSE_ATTN"
op_builder/stochastic_transformer.py:    BUILD_VAR = "DS_BUILD_STOCHASTIC_TRANSFORMER"
op_builder/transformer.py:    BUILD_VAR = "DS_BUILD_TRANSFORMER"
op_builder/transformer_inference.py:    BUILD_VAR = "DS_BUILD_TRANSFORMER_INFERENCE"
op_builder/utils.py:    BUILD_VAR = "DS_BUILD_UTILS"

What do you think?

@tjruwase , I just added a change to setup.py that might do the trick before I saw your note.  If we think it'll take some iterations, we can do that in a different PR.

re: redhat, is yum the standard package manager these days? I haven't used redhat in decades, but from what I remember rpm was a standalone package manager.
This is different with apt, where dpkg is low-level tool and apt is the package manager.

I believe we use yum to install rpm packages on our systems.  It plays a role like apt where it can fetch packages from remote repositories and then install them via rpm.

@tjruwase , I just added a change to setup.py that might do the trick before I saw your note.
7bf5c0f
If we think it'll take some iterations, we can do that in a different PR.

Awesome. This worked for me in that without libaio-dev this now breaks, sudo  DS_BUILD_AIO=1 pip install -e . --global-option="build_ext" --global-option="-j8" --no-cache -v --disable-pip-version-check
      File "/data/users/olruwase/deepseed/public/findlibaio2/setup.py", line 122, in <module>
        assert False, f"Unable to pre-compile {op_name}, perhaps disable with {env_var}=0"
    AssertionError: Unable to pre-compile async_io, perhaps disable with DS_BUILD_AIO=0
    DS_BUILD_OPS=0
     [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
     [WARNING]  async_io: please install the libaio-dev package with apt
     [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 188, in _main
    status = self.run(options, args)
  File "/opt/conda/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 185, in wrapper
    return func(self, options, args)
  File "/opt/conda/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 333, in run
    reqs, check_supported_wheels=not options.target_dir
  File "/opt/conda/lib/python3.6/site-packages/pip/_internal/resolution/legacy/resolver.py", line 179, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))

@adammoody, this looks good to merge.
@stas00, what say ye?

@tjruwase , squeezed in one more commit that I think should just run each is_compatible test once.  I'll put down the pencil now.

Let me re-test

@tjruwase , squeezed in one more commit that I think should just run each is_compatible test once. I'll put down the pencil now.

And all the surviving DeepSpeed bugs let out a collective sigh of relief 🥇

with the lib it's good, w/o it, I don't think the user can make any sense out of it:
DS_BUILD_AIO=1 pip install -e . --global-option="build_ext" --global-option="-j8" --no-cache -v --disable-pip-version-check

    Running setup.py (path:/mnt/nvme1/code/github/00optimize/deepspeed/setup.py) egg_info for package from file:///mnt/nvme1/code/github/00optimize/deepspeed
    Created temporary directory: /tmp/pip-pip-egg-info-nx64tvmv
    Running command python setup.py egg_info
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/mnt/nvme1/code/github/00optimize/deepspeed/setup.py", line 124, in <module>
        assert False, f"Unable to pre-compile {op_name}, perhaps disable with {env_var}=0"
    AssertionError: Unable to pre-compile async_io, perhaps disable with DS_BUILD_AIO=0
    DS_BUILD_OPS=0
     [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
     [WARNING]  async_io: please install the libaio-dev package with apt
     [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
WARNING: Discarding file:///mnt/nvme1/code/github/00optimize/deepspeed. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 180, in _main
    status = self.run(options, args)
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 318, in run
    requirement_set = resolver.resolve(
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 103, in resolve
    r = self.factory.make_requirement_from_install_req(
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 444, in make_requirement_from_install_req
    raise self._build_failures[ireq.link]
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 185, in _make_candidate_from_link
    self._editable_candidate_cache[link] = EditableCandidate(
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 334, in __init__
    super().__init__(
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 151, in __init__
    self.dist = self._prepare()
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 234, in _prepare
    dist = self._prepare_distribution()
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 345, in _prepare_distribution
    return self._factory.preparer.prepare_editable_requirement(self._ireq)
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 622, in prepare_editable_requirement
    dist = _get_prepared_distribution(
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 60, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 36, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 549, in prepare_metadata
    self.metadata_directory = self._generate_metadata()
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 524, in _generate_metadata
    return generate_metadata_legacy(
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/pip/_internal/operations/build/metadata_legacy.py", line 67, in generate_metadata
    call_subprocess(
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/pip/_internal/utils/subprocess.py", line 244, in call_subprocess
    raise InstallationSubprocessError(proc.returncode, command_desc)
pip._internal.exceptions.InstallationSubprocessError: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Removed file:///mnt/nvme1/code/github/00optimize/deepspeed from build tracker '/tmp/pip-req-tracker-3wthsjb4'
Removed build tracker: '/tmp/pip-req-tracker-3wthsjb4'

After a very close study I finally found the assert message way before the normal traceback.
   AssertionError: Unable to pre-compile async_io, perhaps disable with DS_BUILD_AIO=0
       DS_BUILD_OPS=0

and then warnings follow, which is very odd.
My vote is at least to kill "perhaps disable with DS_BUILD_AIO=0 DS_BUILD_OPS=0" - as this doesn't suggest anything useful. I enabled these manually because I needed those. And definitely not DS_BUILD_OPS=0 because it defeats the purpose of pre-buillding completely.
What this needs is a rethinking of why warnings are printed separately from the error and after it. Since it's illogical to the user.
But perhaps some other time.
For now, perhaps just changing the assert message to:
AssertionError: Unable to pre-compile async_io - can't find libaio .so and/or header files. See other messages for more details.

will be a good enough first step.

Hmm, yeah, the messages are being displayed in a different order than how they are printed, probably due to a mixing of stdout and stderr streams.  Let me take another look.  One of those is coming from assert and the others are self.warnings().  Let me take another stab at it.

@stas00 , how about this change?  I modified setup.py to only print the suggestion to disable an op if it sees that the environment variable for that op is not already defined.  The assert message just says that it failed to build, but offers no suggestion.
The order of those messages might still come out mixed (stderr vs stdout), though maybe it's less confusing.
To fix the order, we probably should have builder.warning() write to stderr or define a builder.error() function that prints to stdout like builder.warning() does.

how about this change? I modified setup.py to only print the suggestion to disable an op if it sees that the environment variable for that op is not already defined. The assert message just says that it failed to build, but offers no suggestion.

Looks good. thank you!
Trying to sync std streams can be a madness, when I mentioned re-think I was thinking that different components do their work, and print nothing and return success/failure and any messages instead to the caller. Then the manager that runs these components has the freedom to print any messages it collected from the components in the order it needs.
But again, it's probably a job for another time.

It looks like setup.py only uses assert to bail out when it detects a fatal error.  I'm guessing that assert is what generates the (noisy) python stack trace.  We could avoid that if there is a better way to exit, but I don't know the convention for bailing with an error from a setup script offhand.

adding a few empty lines before and after the assert?
e.g. pytorch recently started doing logs with banners that stand out from the other noise, examples:
******************************************
  /home/ubuntu/tmp/test_script.py FAILED
==========================================
Root Cause:
[0]:
  time: 2021-07-23_02:22:07
  rank: 0 (local_rank: 0)
  exitcode: 1 (pid: 43258)
  error_file: <N/A>
  msg: "Process failed with exitcode 1"
==========================================
Other Failures:
[1]:
  time: 2021-07-23_02:22:07
  rank: 1 (local_rank: 1)
  exitcode: 1 (pid: 43259)
  error_file: <N/A>
  msg: "Process failed with exitcode 1"
[2]:
  time: 2021-07-23_02:22:07
  rank: 2 (local_rank: 2)
  exitcode: 1 (pid: 43260)
  error_file: <N/A>
  msg: "Process failed with exitcode 1"
[3]:
  time: 2021-07-23_02:22:07
  rank: 3 (local_rank: 3)
  exitcode: 1 (pid: 43261)
  error_file: <N/A>
  msg: "Process failed with exitcode 1"
******************************************

**********************************************************************
               CHILD PROCESS FAILED WITH NO ERROR_FILE
**********************************************************************
CHILD PROCESS FAILED WITH NO ERROR_FILE
Child process 3094556 (local_rank 0) FAILED (exitcode 1)
Error msg: Process failed with exitcode 1
Without writing an error file to <N/A>.
While this DOES NOT affect the correctness of your application,
no trace information about the error will be available for inspection.
Consider decorating your top level entrypoint function with
torch.distributed.elastic.multiprocessing.errors.record. Example:

  from torch.distributed.elastic.multiprocessing.errors import record

  @record
  def trainer_main(args):
     # do train
**********************************************************************

I can revert that with a new commit if needed, but this last commit adds a abort function that prints and error message in red and then calls sys.exit(1) rather than assert.

I'm not sure if all terminals support color codes - typically I don't see those being used other than in some small projects. It's up to @tjruwase to approve or not.

sys.exit(1) can break the "super"-system that calls setup.py - e.g. if installing some other package via pip and deepspeed is just a dependency of some package, I think this would be a very unwelcome behavior.

Ah, I see.  Great points, @stas00 .
A call to sys.exit(1) definitely seems like a bad idea.
I borrowed the color code idea from the builder.warning() code:
https://github.com/microsoft/DeepSpeed/blob/97f7ed9e9836f2894387c3a7e38e7db2a8e5dcfc/op_builder/builder.py#L13-L15
I will revert this and go back to the assert.

I am fine with color code actually. Yes, sys.exit() seems an overkill.

From the previous commit https://github.com/microsoft/DeepSpeed/pull/1250/commits/0e4cde6b0a650591c3fafface7e27b4efd9aad4f, we could replace the sys.exit(1) with an assert False, msg.

Ok, I'll stop now, for real this time :-)
Thanks for all of your help with this, @tjruwase and @stas00 .

I think your PR got hit by never starting CI - happened a few times to my PRs, you probably need to push an empty commit.
git commit --allow-empty -m "Trigger CI"
git push

I just started the CI.


Request

Hi, I love your plugin.
I wanted to ask if it's possible to detect and give money ONLY when 1 player is on the block?

Good idea, I’ll try to implement this.

Thanks, that will add some challenge :D


Use *.puml as default extension for generated PlantUML files

Currently, when generating the PlantUML diagrams with:
        new Documenter(modules)
                .writeModulesAsPlantUml()
                .writeIndividualModulesAsPlantUml()
        ;

It generates PlantUML files with uml as extension. According to https://plantuml.com/sources PlantUML files should use pu or puml as extension.
Using uml does, for example, not allow the IntelliJ PlantUML plugin to view the files, but renaming them to puml does.
Modulith can be configured to use puml like this:
        new Documenter(modules)
                .writeModulesAsPlantUml(Documenter.DiagramOptions.defaults().withTargetFileName("components.puml"))
                .writeIndividualModulesAsPlantUml(Documenter.DiagramOptions.defaults().withTargetFileName("module-%s.puml"))
        ;

But I think this should be the default.

I could take this issue if that's okay.

You can take it then, I don't have enough time today :)

That's in place now.

That was fast. Thanks!


Updated "tested up to" tags and changelogs

Synchronizing with svn repo state.

made compatible with GF 2.7 and WP 6.1.1


Feature/#349 keep sidebar menu open on more urls

Fixes #349

Coverage decreased (-0.1%) to 73.77% when pulling db343af8cdabe7f9bfefdabd42bce5900d39c10e on feature/#349-keep-sidebar-menu-open-on-more-urls into 001a3d00a7c0a576b27a4db48e83b788296bfe59 on develop.


JENKINS-38093 - Don't provision checkbox

added Don't provision checkbox to vagrant up

@sparkoo I wouldn't want to move the destroy to the post-build, if at all, will clone it there for backward compatibility and not to break functionality for users already haing this

@ShimiTaNaka I guess the comment should be to PR #6

@ShimiTaNaka can you please check again this one?


is it possible to integrate glslify in glsl-canvas?

@shengsword it's possible, could you please provide a possible use case?

@actarian The use case would be to preview glsl shaders that are written using glslify imports. Many people write glsl shaders using syntax like:
#pragma glslify: noise = require('glsl-noise/simplex/3d')

rather than pasting the noise function at the beginning of the file, but that means those glsl files can no longer be previewed using glslCanvas.


Unable to retrieve typed objects after changing their class' property type from int to decimal

Description
Unable to retrieve typed object after changing some of it's class' property type from int to decimal.
Repro steps:

Create a model:

public class Customer
{
    public int Id { get; set; }
    public int? Sum { get; set; }
}

Create an object:

var customer = new Customer() { Sum = 100 };

Persist the typed document:

liteDatabase.GetCollection<Customer>("Customer").Insert(customer);

Change type of Sum property from int to decimal.

public class Customer
{
    public int Id { get; set; }
    public decimal? Sum { get; set; }
}

Retrieve the typed document

liteDatabase.GetCollection<Customer>("Customer").FindById(customerId);

Expected Result
The document successfully retrieved.
Actual Result
System.InvalidCastException: 'Unable to cast object of type 'System.Int32' to type 'System.Decimal'.' on retrieval.
Tests
Repository with the tests. More specifically:

TypedCollectionModelPropertyTypeChangeTests

The same works fine for a schemeless collection:

SchemelessCollectionModelPropertyTypeChangeTests

Hi! With the objective of organizing our issues, we are closing old unsolved issues. Please check the latest version of LiteDB and open a new issue if your problem/question/suggestion still applies. Thanks!


[FLINK-13690]Fix JDBC LookupFunction getFieldFromResultSet BUG

What is the purpose of the change
fix JDBCLookupFunction getFieldFromResultSet bug, if one column was NULL(SQL) then the next column will be NULL even though it is not null
Brief change log

fix JDBCLookupFunction getFieldFromResultSet bug

Verifying this change

This change is a trivial rework / code cleanup without any test coverage

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no

Hi @molsionmo , thank you to fix it. But there was some fix in https://github.com/apache/flink/pull/9236/files#diff-d862d64d640a9de117328b800fb3fb1dL155
Can you have a check and review? Is that cover your fix?

Hi @molsionmo , thank you to fix it. But there was some fix in https://github.com/apache/flink/pull/9236/files#diff-d862d64d640a9de117328b800fb3fb1dL155
Can you have a check and review? Is that cover your fix?

Yes , it can cover this fix. I have checked and reviewed it.

Will close this because #9236 has covered this.


"Schiau" is listed in the drop-down menu

Description:
"Schiau" is listed in the drop-down menu alongside its corresponding county and commune, according to the law
Precondition:
The website is up an running.

Step 1
Type "Schiau" in the search bar.

Expected results
The location is listed in the drop-down menu as "Schiau, Argeș (Bascov)".


Does the ES256 have a plan to support the S-256 curve?

Does the ES256 have a plan to support the S-256 curve?

ES256 uses P-256 curve. I can't manage to find information about the S-256 curve.
Closing, in any case JWA specification and https://github.com/brianloveswords/node-jwa would need to defined and implement respectively support for any new curve/algorithm.


AdvancedDataGrid focus issue (selectedIndex should be last selectedIndex after saving)

When user select second last record and press save button the detail of entry get disappear.

System behave incorrectly on resaving. System should remain on the selected entry which has been resaved.

Go to Purchase Module
Setups - > Shipment Modes
file name dbzShipmentModeSetup

The dataProvider is being changed which causes the renderers to be refreshed.  I was trying to debug into the code to see what caused the dataProvider to be changed, but then someone logged into the VDI and kicked me out, so I'm going to stop doing any VDI work since it is near the end of my work day anyway.
Ideally, the dataProvider would not be changed.  I can see that it contains the same data, but I think it is a new instance.  But also, the DataGrid code doesn't currently check to see if any of the renderers are selected.  If it did, you probably wouldn't notice.  So if there is a good reason to change the dataProvider, then I recommend changing AdvancedDataGridItemRenderer to check the selectedIndex before choosing a background color.

Pashmina has logged into vdi for 5 mins. We will take care next time.we will not access the vdi again. I think on reassigning new values to  the dataprovider advancedDataGrid isn't updating the grid values.

When saving, there is a new server request.  It probably gets the updated data from the server and resets the data provider.  That seems reasonable, so I will update AdvancedDataGridItemRenderer to check for selectedIndex

@aharui Thanks for the fix


Fix typo in AUTHORS.rst

I left out a bracket.

Why did you close?

Closed inadvertently in the course of forgetting to merge from upstream.

👍 I was going to squash merge, but thanks for preemptively fixing.


gnome3 test: increase timeout

The gnome3 test has been failing recently (1 2); this is due to exit code 124 which is the command timing out.
This increases the timeout to 900, to align with the similar timeout in https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/test-driver/Machine.pm#L222

I hate those high timeout but let's see :) thanks.


Updated the Pipelines Quickstart screenshots and related text for Kubeflow 0.5

Part of issue https://github.com/kubeflow/website/issues/505

This change is

Fixes https://github.com/kubeflow/website/issues/587

Preview:

https://deploy-preview-585--competent-brattain-de2d6d.netlify.com/docs/pipelines/pipelines-quickstart/
https://deploy-preview-585--competent-brattain-de2d6d.netlify.com/docs/pipelines/pipelines-overview/
https://deploy-preview-585--competent-brattain-de2d6d.netlify.com/docs/pipelines/pipelines-concepts/

/assign @abhi-g

/lgtm
/approve
Thanks Sarah


[html]  element should be formatted like  in editor, not like !

VSCode Version: 1.7.1
OS Version:  Windows 7 Ulimate SP1

Steps to Reproduce:

Create a <pre> element with multiline content
Stock VSC HTML formatter will remove line breakes rendering <pre> useless. This behavious is not seen in <code> element. Obviously a <pre> must be auto formatted like <code> element. Fix this thing please.

The list of elements that are not formatted are controlled by the setting:
"html.format.unformatted". I added pre to the defaults.
In the meantime you can add it to the user settings.


Cannot get the install manifest from v2.12.12

I am getting a 404 error when going to this link.. is anyone getting the same issue?
https://raw.githubusercontent.com/argoproj/argo/v2.12.12/manifests/install.yaml

This is not the place to get this for v2.12.2. Let me find the right place...

https://raw.githubusercontent.com/argoproj/argo-workflows/v2.12.12/manifests/install.yaml

@alexec
Thanks! If you look in the latest releases section this is the link posted there. Also I was unable to curl and gunzip the argo cli tool, you might have a typo in the releases section..


Allow @index on __typename

Is this feature request related to a new or existing Amplify category?
api
Is this related to another service?
No response
Describe the feature you'd like to request
Creating an index on __typename should be allowed for use cases such as querying in date ranges without expensive and inneficient scan filtering. This is even recommended by the amplify docs https://docs.amplify.aws/guides/api-graphql/query-with-sorting/q/platform/js/#implementation. It seems silly to create a dedicated "type" field when the schema already impllicitly adds the __typename field which has the exact same value in every case.
With the transformer v2, one can explicitly specify the __typename field with the desired index but codegen will throw the error Name "__typename" must not begin with "__", which is reserved by GraphQL introspection. and also required explicit __typename input which would then be redundant as well.
Please let me know if there is currently a proper way to do this that I am missing.
Describe the solution you'd like
Should be able to specify __typename: String @index(name: "...", sortKeyFields: ["createdAt"], queryField: "...") and typename would be implicitly added as usual (along with createdAt in this case).
Describe alternatives you've considered
Using redundant typename without the leading underscores...
Additional context
No response
Is this something that you'd be interested in working on?

[ ] 👋 I may be able to implement this feature request

Would this feature include a breaking change?

[ ] ⚠ This feature might incur a breaking change

Hey @iyz91 :wave: thanks for raising this! While we evaluate this request with the team, I wanted to callout the note in the docs you linked where this @index is placed on a property on the model that is not a GraphQL meta field

When created a Todo, you must now populate the type field for this to work properly.

Could this perhaps be accomplished by manually adding an additional type property with a default value and index?
type Todo @model {
	type: String @default(value: "Todo") @index(name: "...", sortKeyFields: ["createdAt"], queryField: "...")
}

@josefaidt Thanks for the response. The @default is a good idea, I'll add it to the redundant type/typename field mentioned above for the time being.


attach visual stdio code optimize

Issue Type: Feature Request
when attach visual stdio code,I need to another operation,
why dirct attach this by click
VS Code version: Code 1.65.1 (8908a9ca0f221f36507231afb39d2d8d1e182702, 2022-03-08T02:06:27.846Z)
OS version: Windows_NT x64 10.0.19044
Restricted Mode: No
Remote OS version: Linux x64 5.10.0-11-amd64
Remote OS version: Linux x64 5.10.0-11-amd64

/needsMoreInfo
您当前的表述令人费解。如果您不擅长英语，请寻找一位熟悉技术的人类来协助翻译。

/needsMoreInfo

sorry,I am poor in english,so I describe it by picture
------------------ 原始邮件 ------------------
发件人:                                                                                "microsoft/vscode"                                                                                @.>;
发送时间: 2022年3月10日(星期四) 下午3:57
@.>;
<EMAIL_ADDRESS>主题: Re: [microsoft/vscode] attach visual stdio code optimize (Issue #144813)
/needsMoreInfo
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.Message ID: @.***>

Please upload your image using GitHub website or mobile apps. Images in Email reply won't be attached.

Hey @weinand, this issue might need further attention.
@dian-yu-luo, you can help us out by closing this issue if the problem no longer exists, or adding more information.


feat: allow Resource to be created with some attributes resolving asynchronously

… promise

Which problem is this PR solving?
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue)
Short description of the changes
Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[ ] Test A

Checklist:

[ ] Followed the style guidelines of this project
[ ] Unit tests have been added
[ ] Documentation has been updated

I really like this proposal - to propagate the async task down to resource attribute consumers.
This is a backward-compatible (I think) and very clean way to address the issue.
The promise means that every time we need to access this resource attributes, the caller will have to await to get the full attributes values.
I wonder if that is a constraint that is easy to fulfill.
For example, in aspecto's distro we read the resource attributes in a span processor and use the values to apply filtering rules on span attributes. This is done in the onEnd invocation which is a sync function of course.
I guess we could wrap it with some mechanism that aggregates the processor spans until the resource is resolved, but that is not trivial.
Another possible issue is that we cannot have any guarantee on how long it will take for the promise to resolve. Until it resolves, spans might be piling up in the exporter export invocations, all waiting for the resource to be resolved so they can be sent on the wire.
Really interesting approach, looking forward to discussing this more.

According to spec resource is immutable. I think delaying the read until first use in e.g. exporter sounds to me like it's still spec conform.
I don't think the await can be avoided by any wrapping/API we add in common code. A consumer awaits on first use and may store the result for future use (maybe already serialized/compressed/... to avoid to redo this on every export).

I wonder if top level await would be able to achieve this as well? @aabmass / @Flarna , do you know?

Played around with it a bit. For ESM, adding an loader module would work and is able to asyncronously wait for stuff to resolve. The thing is, it's only for ESM.

That's quite cool. Is the limitation of only a single loader module removed yet?

I think top Level await can be used but as far as I know it works only in ECMAScript module mode not for commonJs. I requires node.js 14.8 as far as I know. Not sure about browsers.
Once we are in module mode we have to use loaders which are still experimental.
The limitation to have only single loader is gone on node master (see here), backport to 18.x has not yet happened as some other breaking changes in loaders are planned and target is to combine all breaking changes in a single 18.x release.

I guess now span processors are expected to handle these async promises? We already have some users which have implemented custom span processors. I guess this is considered breaking for them? What does the spec say about breaking changes for sdk interface implementers?

I would consider this also as a breaking change so this would be semver major.

This PR actually introduces another problem from the specification:

OnEnd(Span)
OnEnd is called after a span is ended (i.e., the end timestamp is already set). This method MUST be called synchronously within the Span.End() API, therefore it should not block or throw an exception.

https://github.com/open-telemetry/opentelemetry-specification/blob/9abbdd39d0b35f635f833f072013431da419894e/specification/trace/sdk.md#onendspan

This PR actually introduces another problem from the specification:

OnEnd(Span)
OnEnd is called after a span is ended (i.e., the end timestamp is already set). This method MUST be called synchronously within the Span.End() API, therefore it should not block or throw an exception.

<EMAIL_ADDRESS>
I actually realized this isn't a problem. If it was, then the batch span processor would be not compliant.


The server_credentials should be on the server port and not on the server itself.

A grpc server should be able to run multiple ports with different types of credentials associated with it.

IUC, this has been fixed and should be closed.  If there is still outstanding work, feel free to reopen it.


Handle forward compatibility & JVM target version

Concerns:

[x] JVM target-version seems to be not pegged properly
[x] All of the modules are compiled with Kotlin 1.9.0 & making them not forward-compatible with, let say ~1.6.0
[x] The Kotlin-std-lib is getting bundled in into the artifacts

Binaries are now compatible with Kotlin-1.5.0

Turns out, just setting kotlinOptions.apiVersion & kotlinOptions.languageVersion to version-x will not make it (forward) compatible with version-x. And also Kotlin makes no such guarantees about forward-compatibility too.
(Taken from https://kotlinlang.org/docs/kotlin-evolution.html#evolving-the-binary-format)

Preferably (but we can't guarantee it), the binary format is mostly forwards compatible with the next feature release, but not later ones (in the cases when new features are not used, e.g. 1.3 can understand most binaries from 1.4, but not 1.5).

So, even tho if you set kotlinOptions.apiVersion & kotlinOptions.languageVersion to an appropriate version and don't use any new features added to Kotlin after that version, there is no guarantee that it would be compatible with that version (while compiling with a newer version).

With the current setup (compiling with Kotlin 1.9.0, and languageVersion set to 1.5.0), artifacts are compatible upto Kotlin 1.7.0, (came to this conclusion by testing my self, didn't find any source on this).
Which I feel is pretty decent for now.

With the current situation, we can also relax kotlinOptions.apiVersion & kotlinOptions.languageVersion to Kotlin 1.7.0.


PHP: return channel arg type

LGTM. Note that this supersedes #4865, but that one has unresolved comments and no CLA, so that's not a bad thing.


Exception: Undefined constant GLib::Bytes

Using this code:
require "gtk4"

file = Gio::File.parse_name("/some/image/path.png")
file.load_bytes(Gio::Cancellable.new)

Results in this exception:
Showing last frame. Use --error-trace for full trace.

In lib/gtk4/src/gio-2.0/file.cr:747:55

 747 | def load_bytes(cancellable : Gio::Cancellable?) : GLib::Bytes
                                                         ^----------
Error: undefined constant GLib::Bytes

I removed Bytes from the bindings
https://github.com/hugopl/gi-crystal/blob/master/g_lib.yml#L212
my initial thought was that you would not need it since there's a Crystal equivalent, however a lot of GTK4 API use the GFile object that have Bytes as return value of some methods.... so I think I was wrong.

I'm thinking if would be possible to remove GFile from API... :thinking: , in the cases that C API requires a GFile as parameter we can do a transparent conversion from Path to GFile.
The first problem I see is if some API have a GFile parameter expecting the file to be open and another API has a GFile parameter that requires the file to be closed... and things like that.

I don't know of any API that requires the GFile to be open.. but there are a lot of methods in GFile that could be quite useful and hard to accomplish with crystal.
For example I wouldn't know how to implement g_file_trash or g_file_mount_mountable in crystal (or how to asynchronously do basic io operations using the crystal methods)

Yeah, so I think GFile and GBytes must be back.
:thinking: ... for other datatypes like GDateTime maybe would be useful to also not remove them but instead accept the pure-Crystal type or the G* type, i.e. a function that accept a GDateTime as parameter would now accept GLib::DateTime | ::Time.
I need to do some experiments with this to see if it works.

I was taking a look on this... Gio::File was never removed from APIs, GLib::Bytes was.
By what I saw Bytes binding need to be handmade since it is ref-counted with its own set of functions and I don't know if GIR give enough info to generate automatic bindings for this... even if it does, Bytes class seems to be an exception and have not much methods, so a handmade first implementation still make sense.

Bytes is back and working, but there are still other bugs that cause Gio::File.parse_name to not work yet.

Despite of the missing cross project reference in the commit message of https://github.com/hugopl/gi-crystal/commit/c583d0db4c44900cdab6e3976e471a0bdd92f900 if fixed this issue :tada:
The fix was delayed because I was thinking about to refactor the code that deal with the parameters in the generator that become a bit messy over time, however my first refactor attempt failed, so it will still messy for some more time until a find time and a design to unmessify it.
BTW, now I just remove the optional parameters, for sure this doesn't match all use cases... so I'm still thinking a way to let these functions with return values + optional out parameters to be accessible on Crystal API without affect the use case where people don't care about the optional out parameter.... :thinking:


[Monitoring] Alert on non-Bisonai node down

This can be closed since we have peer count check and health check
if peer count decrease and healthcheck doesn't say anything, that'd mean non-bisonai node is down
ex)
non-bisonai node down
number of peers in orakl offchain cluster has decreased from 2 to 1

bisonai node down
orakl-node is down


When the answers are tapped quickly the app crash.

When one of the answers are tapped quickly are counted each time as an option.
It is necessary to disable the screen once the user selects an option until the next question is displayed, or the level is won or lost.

agregada validacion de que si ya se respondió no hace nada


Make shop window (mostly) editable

Addressed in v1.6
https://github.com/garylzimmer/Gnome-Depot/releases/tag/v1.6


chore: automate releases

It's now sufficient to just push a tag to the repository, this will create a release using GitHub actions with reasonable release notes, and it also uses a release artifact since GitHub doesn't guarantee stability of their archives.
This is taken from the https://github.com/bazel-contrib/rules-template which is maintained by experts on the Rules Authors SIG to make rules easy to maintain and also more consistent.

Do we need to start using https://www.conventionalcommits.org/ for this to work better? I've never been a fan of it as you can see from the git history, but can adapt if this is preferred for canonical rulesets.

Once this is merged, I can cut the initial v0.1.0 release and we take it from there, right?

I do think some semantic commit messages help explain the history and in particular whether the next release is a patch or minor. Also makes nicer changelog. I don't think it has to be a prerequisite as long as you're okay with a generated changelog.
Yeah I would suggest to tag a v0.1.0 after merge

I do think some semantic commit messages help explain the history and in particular whether the next release is a patch or minor. Also makes nicer changelog. I don't think it has to be a prerequisite as long as you're okay with a generated changelog.

I guess we can agree on a single format then to avoid different styles in Git history. Let's stick to conventional commits, I will try to follow its format.


two_passes example breaking on Windows

Bevy version
Main (f514d5c)
Relevant system information
AdapterInfo { name: "AMD Radeon RX 6750 XT", vendor: 4098, device: 29663, device_type: DiscreteGpu, driver: "AMD proprietary driver", driver_info: "24.1.1 (AMD proprietary shader compiler)", backend: Vulkan }
SystemInfo { os: "Windows 11 Pro", kernel: "22631", cpu: "", core_count: "6", memory: "31.9 GiB" }

What you did
cargo run --example two_passes
What went wrong
Sometimes it crashes (in 5 runs, crashes 2 times). The problem is only on startup, after that, seems to be working fine.
thread '<unnamed>' panicked at crates\bevy_core_pipeline\src\msaa_writeback.rs:83:18:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Encountered a panic in exclusive system `bevy_render::renderer::render_system`!
thread 'Compute Task Pool (5)' panicked at crates\bevy_render\src\pipelined_rendering.rs:145:45:
called `Result::unwrap()` on an `Err` value: RecvError
error: process didn't exit successfully: `target\debug\examples\two_passes.exe` (exit code: 101)

backtrace.log
Additional information
Seems to be related to #10812, since it works fine on commit 7705c1d but breaks on 9f7e61b

This is not specific to windows, I have reproduced the issue on Linux.

I can't reproduce on macOS
This should be investigated for the 0.13 so we have an idea of the underlying issue

Msaa_writeback is the only file that miss the checking of the result when getting the pipeline.
Doing a PR to fix that


Add verbose messages

It will be nice to have simple convergence messages, like in scipy.optimize.

@FiksII , thanks for this. This algorithm is now part or SciPy (called dual_annealing) and is aligned with scipy optimize interface.
You can now use the callback mechanism which means each time the algorithm finds a better value, the callback function is called with the x components and the current objective value. This maybe what you need.

@FiksII , you can refer to:
https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.dual_annealing.html#scipy.optimize.dual_annealing

Yes, I have already found it. Thanks.


[FR] Be able to configure version_requirement_updates differently for production and development dependencies

Hello there 👋
First of all thanks for this amazing tool, we've been using it on https://github.com/usabilla, inside our private repositories, for a while now, since #1230 👌
There's one feature that I'd like to share an idea with you about it: give the users the ability to configure the version_requirement_updates option differently for the two kinds of dependencies: production and development.
Something like this:
version: 1 # We are using `dependabot-preview` because of Private Repositories needs (see #2778 and https://github.com/dependabot/dependabot-core/issues/2297#issuecomment-655380840)
update_configs:
  - package_manager: "php:composer"
    directory: "/"
    update_schedule: "daily"
    default_labels:
      - "Dependencies"
    # version_requirement_updates: "auto" // This is how it's today
    version_requirement_updates:
      - match:
          dependency_type: "development"
          requirement_type: "increase_versions"
      - match:
          dependency_type: "production"
          requirement_type: "widen_ranges"

What do you think? And, can you point me to the code where I should do the change to propose a PR? :)

Thanks for the proposal @carusogabriel, I think your suggestion makes total sense. Currently we don't really have the bandwidth to tackle these sort of improvements, but we hope to have more time in a few months to make these kinds of improvements.
I appreciate your offer to tackle this in a PR, unfortunately the code that powers the config file etc is part of the private services we use to run Dependabot at GitHub.


Performance Rails Cop proposal: Replace Pluck Include? with Where Exists?

pluck(:status).include?('element') can be refactored to where(status: 'element').exists?
There is no need to export a lot of data, creating an array and then running include?, when activerecord can check directly if the element exists on the database.

Additionally, where(status: 'element').exists? can be written exists?(status: 'element'). 😉
But that's another cop altogether. 🙂

@Drenmi I would prefer using where(status: 'element').exists? over exists?(status: 'element'). Starting around Rails 4.2, Rails has been deprecating or removing the finder ability of most of the ActiveRecord query methods. Even though I have not seen anything about exists? with a condition being deprecated, I wouldn't count on that functionality being supported for much longer.

We're in the process of moving all Rails-related functionality to a standalone library (gem) named rubocop-rails. Please, migrate this issue to its issue tracker https://github.com/rubocop-hq/rubocop-rails/issues/


Labware v2 defs: move loadName to top level

Right now loadName is under parameters, instead it should move to top level of the JSON.
This shouldn't require anything across our stack to meaningfully change, it's just a refactor chore across everything that uses loadName.

Too late


v0.7.1 conflicts with html5 canvas drawn objects, v0.4.1 doesn't

I've tested  with color-0.4.1.min.js and this works nicely with html5 canvas objects (context.rect(x, y, w, h)).
When replacing with color-0.7.1.min.js, objects on the html5 canvas get distorted. So the color script somehow conflicts with the drawing of objects on the html5 canvas.

Can you provide some code or an living example in order to debug this plz?

No feedback in 1 month, closing.


Embedding videos using Polymer element

Could you add option for embedding videos using Polymer element?
https://github.com/GoogleWebComponents/google-youtube

I don't think I'm going to manage to get to this, but hopefully by now someone has made a separate plugin to do this.


In online planning like name nodes we should strip the identity keyed producers

We've had strange error cases around them not being expected. It would be good to just strip the graph to just operationally require components

This is pretty easy to fix using the DagOptimizer. Should be an easy subset of #634


Fix requierments.txt file

Hi All,
Does someone has a fix file of requierments.txt which works?
BR

I switched requests to 2.25.0 and urllib3 to 1.25.6

Hi @GastroGee
Thank's but pyjq is still erroring...
https://github.com/duo-labs/cloudmapper/issues/888#issuecomment-900388571

Ah i see, so i was building a docker image and here is some adjustment with my dockerfile
Note, i installed pyjq with my dockerfile ...
# Dockerize https://github.com/duo-labs/cloudmapper
# This project pulls Cloudmapper and initializes it in a docker container.

FROM python:3.7-slim-buster

EXPOSE 8000
WORKDIR /opt/cloudmapper

## basic installs
RUN apt-get update
RUN apt-get install autoconf automake build-essential libtool python3.7-dev python3-tk jq awscli -y
RUN apt-get install git jq -y

## Clone cloudmapper from duo release page
RUN git clone https://github.com/duo-labs/cloudmapper.git .

## The default requirements file has lots of dependency issues, removing it.
RUN rm requirements.txt

## Passing in our own tested requirement file that works
COPY requirements.txt ./requirements.txt

## Install pip and others
RUN pip install --upgrade pip setuptools wheel
RUN pip install PEP517
RUN pip install pipenv && pipenv clean && pipenv lock --clear && pipenv sync && pipenv install
RUN pip install --upgrade awscli
RUN pip install pyjq

VOLUME /opt/cloudmapper/web
VOLUME /opt/cloudmapper/account-data

COPY entrypoint.sh ./entrypoint.sh

# COPY .env ./.env

COPY config.json ./config.json

CMD ["bash","entrypoint.sh"]

and then my requirements file is this
astroid==2.4.1
autoflake==1.3.1
autopep8==1.5.2
boto3==1.14.6
botocore==1.17.6
certifi==2<IP_ADDRESS>
chardet==3.0.4
coverage==5.1
docutils==0.15.2
idna==2.8
isort==4.3.21
Jinja2==2.11.3
jmespath==0.9.5
lazy-object-proxy==1.4.3
MarkupSafe==1.1.1
matplotlib==3.2.2
mccabe==0.6.1
mock==4.0.2
netaddr==0.7.19
nose==1.3.7
pandas==1.1.3
parliament==0.5.0
policyuniverse==<IP_ADDRESS>01103
pycodestyle==2.5.0
pyflakes==2.2.0
pyjq==2.3.1
pylint==2.5.2
python-dateutil==2.8.1
PyYAML==4.2b4
requests==2.25.0
s3transfer==0.3.0
seaborn==0.10.1
six==1.14.0
toml==0.10.0
typed-ast==1.4.1
urllib3==1.25.6
wrapt==1.12.1

hope that helps

Thank you @OlaSegha , can you open a PR with that fix so that we can all take a look?
Cheers!

Hi @OlaSegha,
Thank's!
I tried to reinstalled it on my local Linux VM after I updated the requierments.txt file with you code and run the
pip install pipenv && pipenv clean && pipenv lock --clear && pipenv sync && pipenv install
l still got the pyjq error:
An error occurred while installing pyjq==2.3.1 --hash=sha256:5f3cd45b856ae4b42a1058d2e91e13d84df5ea188aedc3ae397dc0e01c270db9 --hash=sha256:45f7688d8191a4043e7c4839d6cabff8f16abc98fe48c3a4a0cc2da05c62096d! Will try again.
Is it necessary to install it on a docker and how would you suggest to do it the quickest way?
Any Idea why?
@w0rmr1d3r  did it worked for you? Is it install locally on your Linux?
BR

Hello @cgaudit !
I'm not running a Linux 😅  , however, since this project gave me a lot of issues installing it and the maintainers aren't reviewing the PRs with the fixes proposed, I switched to another tool which works.
This one -> https://github.com/toniblyx/prowler
See if it helps with what you want to scan or do.
Cheers!

Hi All,
I'm sorry for the similar question.
The requirements.txt is not correct, could you please tell me the correct file or how to fix it?
I have run this command.
pip install -r requirements.txt

However, I encountered the following error.
ERROR: botocore 1.17.6 has requirement urllib3<1.26,>=1.20; python_version != "3.4", but you'll have urllib3 1.26.5 which is incompatible.
ERROR: requests 2.22.0 has requirement urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1, but you'll have urllib3 1.26.5 which is incompatible.

I followed @OlaSegha's advice to check requirements.txt but it did not lead me to a solution😢
Please help me from this error.

@oicho014 @cgaudit
So i just attempted this on a linux VM, my virtual env is python3.7
First the requirements file is wrong, i removed pyjq 2.1.0 from the list .... and updated urllib3 to 1.25.6.
astroid==2.4.1
autoflake==1.3.1
autopep8==1.5.2
boto3==1.14.6
botocore==1.17.6
certifi==2<IP_ADDRESS>
chardet==3.0.4
coverage==5.1
docutils==0.15.2
idna==2.8
isort==4.3.21
Jinja2==2.11.3
jmespath==0.9.5
lazy-object-proxy==1.4.3
MarkupSafe==1.1.1
matplotlib==3.2.2
mccabe==0.6.1
mock==4.0.2
netaddr==0.7.19
nose==1.3.7
pandas==1.1.3
parliament==0.5.0
policyuniverse==<IP_ADDRESS>01103
pycodestyle==2.5.0
pyflakes==2.2.0
pyjq==2.3.1
pylint==2.5.2
python-dateutil==2.8.1
PyYAML==4.2b4
requests==2.25.0
s3transfer==0.3.0
seaborn==0.10.1
six==1.14.0
toml==0.10.0
typed-ast==1.4.1
urllib3==1.25.6
wrapt==1.12.1

after which i just ran
pip install pyjq==2.1.0

I tried what you said, and the build went through.
Now I can sleep soundly!
Thanks, @OlaSegha!

Hi
@w0rmr1d3r  sorry for the delay... I'm running on Linux

Hi
I run it using ubuntu with 3.7 on the demo and it succeeded, thank's!
Is it nessecery to add external CIDR names to the config.json file or it can be ignored? If not how can I get it?
BR

It is not necessary

Thank's I will delete this part..
Anyway:

I don't have MFA
Got error when running :
python3 cloudmapper.py prepare --config config.json.xxx --account xxx
Traceback (most recent call last): File "cloudmapper.py", line 72, in <module> main() File "cloudmapper.py", line 66, in main commands[command].run(arguments) File "/home/hunter/cloudmapper-main/commands/prepare.py", line 833, in run prepare(account, config, outputfilter) File "/home/hunter/cloudmapper-main/commands/prepare.py", line 662, in prepare cytoscape_json = build_data_structure(account, config, outputfilter) File "/home/hunter/cloudmapper-main/commands/prepare.py", line 420, in build_data_structure for region_json in get_regions(account, outputfilter): File "/home/hunter/cloudmapper-main/shared/common.py", line 161, in get_regions regions = pyjq.all(".Regions[]{}".format(region_filter), region_data) File "/home/hunter/cloudmapper-main/venv/lib/python3.8/site-packages/pyjq.py", line 49, in all return compile(script, vars, library_paths).all(_get_value(value, url, opener)) File "_pyjq.pyx", line 209, in _pyjq.Script.all (_pyjq.c:2561) _pyjq.ScriptRuntimeError: Cannot iterate over null (null)


Improve Docs for Gated Models

With more and more powerful models being released in the field of "generative AI", gating models will become more and more important IMO.
The current state of the model docs can be improved IMO.

"Gated models" don't show up here: https://huggingface.co/docs/hub/index , but are IMO very important docs. Let's add them there as well no?
The are put under "model cards" here, which is def not the place where I'd look for them. IMO they have much more functionality than model cards, e.g. allow model authors to download stats etc... -> or is this just a bug?
Not a huge fan of the docs: https://huggingface.co/docs/hub/models-gated . Think they mix information for "model authors" and "model users" too much. I'd write the docs a bit differently here:

Why gated models?
For authors: How to set-up gated models?
For users: How to use gated models

=> Happy to open a PR to potentially improve the docs there.
wdyt @osanseviero @julien-c ?

Yes, let's add them to the index!
They are not a sub-section under model cards. They are a top section under models and the next one to model cards/
Sounds good, yes!

👍
if you rewrite gated models, please also rewrite gated datasets, as they should be consistent
also cc @SBrandeis who wrote the initial doc

We now have gated models and datasets docs. I agree with

Not a huge fan of the docs: https://huggingface.co/docs/hub/models-gated . Think they mix information for "model authors" and "model users" too much. I'd write the docs a bit differently here:
Why gated models?
For authors: How to set-up gated models?
For users: How to use gated models

would be cool to rewrite some of this as gated repos are being used more and more. We also have no docs on the settings tab and button to download the info. There is also @Wauplin's issue with examples on using it programmatically. I like the idea of rewriting with proposed structure

Intro - why gated models
How to use gated models (just clicking things through form, logging in, etc)
How to manage gated models

WDYT? cc @apolinario as well

(nit) for me would be more logical in that order:

Intro - why gated models
How to manage gated models
How to use gated models (just clicking things through form, logging in, etc)

agree with everything else!

Opened a PR for it: https://github.com/huggingface/hub-docs/pull/1128.


fix: avoid validation error of minimist-options

ref. https://github.com/vadimdemedes/minimist-options/issues/12

👍 Thank you!!!


Add Zydis library for disassembling

Will it work on windows????

Pull Request Test Coverage Report for Build 339392258

13 of 13 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.01%) to 81.921%

Totals

Change from base Build 339375511:
0.01%

Covered Lines:
15488

Relevant Lines:
18906

💛  - Coveralls

That was too easy


[RFC] Uniform Entity Refs

Background
A catalog entity is uniquely identified by the triplet kind, metadata.name and metadata.namespace, where the namespace is optional and implicitly "default" when not given.
The need to address a certain entity comes up in several places, without a clear standard on how it's done:

In Backstage urls: These are currently on the form /catalog/:kind/:optionalNamespaceAndName/ as a bespoke way of permitting the above semantics

In TechDocs: For constructing storage paths

In the Catalog backend API: These are currently on the form /entities/by-name/:kind/:namespace/:name where namespace is mandatory so that the caller has to be able to understand how to generate the default value

Inside entities themselves: There are more and more use cases for entities referencing other entities - for example a Component pointing to an API that it implements, or (soon) a Component to the System that it belongs to.

In particular for references, it is even not entirely clear what the semantics should be when the namespace is omitted. It could either mean "the same namespace as myself", or "the default namespace".
There seems to exist a need to be able to clearly define a simple single-string representation of an Entity by-name identifier. Note that this is different to the metadata.uid field of entities, which also is a valid means of addressing entities in the catalog backend API - but that's an opaque random string that is not human friendly, and changes if the entity is unregistered/re-registered, and should not appear in interfaces that people interact directly with.
The identifier will be typed a lot by hand in yaml files, and therefore humans must be able to construct it easily.
The identifier will appear in human visible URLs, and should be readable with a minimum of fuss.
Proposal
We suggest that this shall be the canonical form for a reference by-name to an entity:

The string is a colon separated triplet on the form <kind>:<namespace>:<name>.

The three strings are used verbatim without encoding.

The namespace may be empty, and is then considered as unspecified (rather than an empty string). The string is then <kind>::<name>; the colon may not be omitted.

The kind is case insensitive.

The end result is that a Backstage URL within the catalog may be for example /catalog/component::my-frontend/docs.
We also suggest that there may be places where nature of the reference makes the kind implicit, and where the namespace can be assumed to be the same as that of the source of the reference. One such example is a Component that wants to point to the System it belongs to, or a Group declaring its parent group. For those cases, the schema may define that only a single string with only the metadata.name of the target entity shall be used - and may not even support the full ref form.
Implementation
Parsing and construction of these identifiers will be added to the package @backstage/catalog-model.
Existing APIs and routes will be changed in a backwards-incompatible way to accommodate the new representation.
Drawbacks

If it is very common that the namespace is omitted, it will be a nuisance (and source of mistakes?) to always have the two colons. Another candidate would be to instead use <kind>:<name>[:[<namespace>]] where the last colon and/or part can be omitted entirely, but the other variant has already seen some use, and if there were ever a place where people might want to put a colon - it would be in the metadata.name.

Since the three parts are unencoded, they (or at least not the first two) must not contain colons. This is the case in the default setup of the catalog, but it's a big assumption. However having to enter parts on an encoded form would be prohibitively frustrating.

While being easy to type out by hand, the key can't easily be extended with more parts, filters, specifiers, variants, or versioning. This is a risk in the face of possibly wanting to extend or tweak it in the future.

Risks

There may be confusion on the distinction of by-name identifiers as per this proposal, vs. by-uid identifiers. We do not foresee that they will occur directly next to each other in writing or in interfaces, and people will generally only be aware of the by-name IDs unless they program directly against the catalog API.

The choice of making the kind case insensitive carries risk of introducing bugs in both frontend and backend code that makes string comparisons or selects data in persistent storage. We consider that the risk / inconvenience of typing out these IDs wrong may be more important.

There are ongoing discussions about whether there will be future need for further grouping/discrimination of entities, for example in a multi-tenancy situation to allow different organisations to collect entities under their own org. It is currently assumed that this will not have an effect the keys described in this RFC. Those groupings tend to cut along auth/access control boundaries, and would therefore be limited by other mechanisms. As a purely hypothetical future example, orgA may have a kind+name+namespace collision with orgB but would still only see their own entity based on an auth token passed to the catalog request.

Haven't thought this through a lot, but another possible delimiter to use is ., since it can be used as is in URLs. Not sure how smart it is tho xP

Inside entities themselves: There are more and more use cases for entities referencing other entities - for example a Component pointing to an API that it implements, or (soon) a Component to the System that it belongs to.

In regard of a field such as implementApis, do you think that the contents would always be referenced in the full <kind>:<namespace>:<name> model although this field only allows kind=API? We are also thinking about the system model and the not yet existing Domains and Systems, and there will often be the case that target relationships are limited to certain kinds.

@dhenneke There's an upcoming RFC where we can discuss that, just haven't gotten around to writing it yet :grin:
Hopefully shows up today or tomorrow.

That's one place where I'm pretty sure it can be just the name. But it's open for discussion! If we ended up having to reference APIs in a different namespace, it wouldn't suffice.

@Rugvip yeah colon is "reserved" although not declared unsafe. One safe char is tilde ~ - a bit odd tho

Just leaving these here to see what they feel like :grin:
.
https://backstage.spotify.net/entity/Component.backstage.frontend/builds

metadata:
  owner: User..patriko

spec:
  dependencies:
    - ..database
    - .playlists.playlist-service

:
https://backstage.spotify.net/entity/Component%3Abackstage%3Afrontend/builds

metadata:
  owner: User::patriko

spec:
  dependencies:
    - ::database
    - :playlists:playlist-service

An idea for dealing with double delimiters would be to assume that they are supposed to be filled in first in the middle, and then prefixed. So a:b would expand to a::b, while :b and b would expand to ::b.
With that in mind:
.
metadata:
  owner: User.patriko

spec:
  dependencies:
    - database
    - .playlists.playlist-service

:
metadata:
  owner: User:patriko

spec:
  dependencies:
    - database
    - :playlists:playlist-service

It's less to type but feeling a bit too magic

Just also dropping in here that k8s forces you to specify pluralized kinds too, so that they can form urls such as /users/freben (which doesn't address the namespace in and of itself). That's something we could conceivably do too, at a bit of a greater effort.

Looking at internal usage, we have for example tons of pipelines and similar that have dots in their names. Probably no colons though.

Looking at them as paths:
freben > implicitly same as ./freben and /users/default/freben (which incidentally would be the frontend URL as well)
Even though they are pathlike, they are treated as plain strings if entered like that in yaml
owner: /users/default/freben

Hmm.

@freben yeah .'s look a bit shite anyway, happy to not move forward with that idea 😁
Yeah paths could be a way to go, bit tricky to make namespace optional then though?
Using the previous examples:
https://backstage.spotify.net/entity/components/backstage/frontend/builds

metadata:
  owner: /users/patriko

spec:
  dependencies:
    - database
    - playlists/playlist-service

Gonna suggest that we settle on the following:
# absolute ref
<kind>:<namespace>:<name>

# namespace relative
<kind>:<name>

# kind and namespace relative
<name>

Parsing would look like this (not tested :grin:):
let kind, namespace, name;

const split = ref.split(':');
if (split.some((part) => part === '')) {
  throw new Error('no empty!');
}
if (split.length === 1) {
  [name] = split;
} else if (split.length === 2) {
  [kind, name] = split;
} else if (split.length === 3) {
  [kind, namespace, name] = split;
} else {
  throw new TypeError('meh');
}

return { kind, namespace, name };

Or described with words:

Entity references contain 1-3 components separated by :, where none of the components are allowed to be empty. The last component is always the name. If there is more than one component, the first one is the kind. If there are three components, the middle one is the namespace.

The reason for skipping the <kind>::<name> format is that we're not 100% certain we'll actually want to keep namespaces long-term. It also removes a bit of clutter from the most commonly used relative references, which I'm assuming to be <name> and <kind>:<name>.
The downside is that there is no kind relative reference, i.e. referencing the same kind in a different namespace. It is possible to open up for format if we feel a need for that though, by allowing :<namespace>:<name>. The downside is that it may not be intuitive what the difference between <x>:<y> and :<x>:<y> is, which is why I think we should just stick to absolute refs instead of kind-relative ones.
Another downside of this format is that it requires slightly more parsing logic than [kind, namespace, name] = split(':'), but in reality I think we will at least also want to be checking whether name is non-empty, at which point it's
worth extracting the parsing into a separate function exported from @backstage/catalog-model anyway.

aaaaaand revised version after some discussion with @freben:
# absolute ref
<namespace>/<kind>/<name>

# namespace relative
<kind>/<name>

# kind and namespace relative
<name>

Parsing would look like this (not tested :grin:):
const split = ref.split('/');
if (split.some((part) => part === '') || split.length === 0 || split.length > 3) {
  throw new Error('bad!');
}

const [name, kind, namespace] = split.reverse()
return { kind, namespace, name };

Difference is that we use / as a delimited to make it more URL-like, which is actually what kubectl does too, e.g. pod/proxy-7c5958c8fb-2ttdt.
We also flip the namespace around to be first, which simplifies the parsing logic and makes the relative forms a bit more intuitive.
We'd also switch to using the absolute format in the catalog URLs, i.e. /catalog/default/component/backstage. Remains to be decided what we do with the namespace there though, as it's wouldn't be as straightforward to migrate away from namespaces.

Is it really desirable to make it URL like? IMHO it makes it harder to see what part in a URL refers to the component and what's just part of the URL structure. It would also make it harder to abbreviate it if using default namespace.

@hooloovooo  I was thinking we'd be comparing namespace%3Akind%3Aname and namespace%2Fkind%2Fname in that case, both of which are a bit shite. Found https://stackoverflow.com/questions/2053132/is-a-colon-safe-for-friendly-url-use#answer-14269897 though, which suggests that : is probably completely fine to use in URLs, even though they're intended to be encoded.
With that in mind I'd be very open for using <namespace>:<kind>:<name>, even as in URLs, with the namespace being optional.
Ending up with something like https://backstage.spotify.net/catalog/component:backstage-frontend/builds, or https://backstage.spotify.net/catalog/backstage:component:frontend/builds with a namespace.
If we feel moving the namespace to the front is too weird I'd happily go with https://github.com/spotify/backstage/issues/1947#issuecomment-684891760 as well.
If we're happy to use unencoded colons in the URL I think that has a pretty big benefit over splitting the entity ref into 3 separate path components in the URL.

Ah, https://stackoverflow.com/questions/1737575/are-colons-allowed-in-urls#answer-43283492 describes it well, so a relative link from /catalog to component:backstage-frontend would be broken, but it's fixed easily by using ./component:backstage-frontend. That might actually be worth the hassle?

If that's the only downside it seems like a nice solution. It would probably be quite easy to get a lint rule in place to check it as well.

Possible caveat that one doesn't always control the url encoding. If you pass the raw identifier to any kind of URL builder, you can be sure that it will uriencode it and that will encode the colons whether you want it or not :)

I just don't want to end up in a situation where it's unclear if we risk ending up with double encoding or double decoding.

With the current character set I don't think we're really gonna get in that situation right? Since we're not allowing % we can essentially decode until it's a noop. Or in a more sane way, always decode, and if it wasn't encoded that's no biggie.

decode until it's a noop

Believe me, that way lies madness :)
@hooloovooo for context, I am more and more leaning towards the idea that forming URLs, and forming identifiers to be typed by humans, are two fundamentally different problem spaces and should not be conflated at all.
I believe that we have two choices.
Either we pick a nice hand written format and use that verbatim in URLs too, and accept that this WILL mean they will be ugly-encoded, no two ways about it.
Or, we shape URLs to be nice for routing and human understanding (encoding the three parts as usual where necessary), and separately handle how to express identifiers in yaml.
As far as I understand it, k8s themselves chose the latter. They even pick pluralized versions of your kind, so your route is /jobs/.... And I think they made that decision for good reason.
So I voted that our frontend paths be made /catalog/default/component/my-id and no magic about it. And that the hand written refs could, by symmetry, be the same, if we decide that slashes will never, ever, for any reason, be allowed in neither namespaces, kinds nor names :)

For example:
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#list-statefulset-v1-apps
Here's an example of how resource definitions work
https://docs.okd.io/3.10/admin_guide/custom_resource_definitions.html

decode until it's a noop

Believe me, that way lies madness :)

I know, that's why I also proposed a sane way :p

Attempting to nail down the two remaining options, as I understand them.
In both options, we retain the notion that strings are case insensitive from a consumer's point of view; for example, you may specify "component" instead of "Component".
Option 1 - Separate URL logic, use slashes
Frontend and backend service URLs that want to reference an entity by name will ALWAYS contain a triplet of namespace, kind, and name, in that order. If the source entity has no namespace, the equivalent string "default" is used. There are no special encoding rules, apart from the normal ones for URLs - each path segment is encoded according to web standards. Example:
# /catalog/<namespace>/<kind>/<name>/builds
/catalog/default/component/my-service/builds

YAML files that want to reference an entity by name will use a slash separated sequence of optional namespace, optional kind, and required name. If the namespace is left out, it is implied to be the same namespace as that of the referencing entity. If the kind is left out, it is implied to be contextual (e.g. if it is known from the context of the value that it points to an API kind entity, then it has the value "api"). Partial example:
metadata:
  name: my-service
spec:
  # All explicit
  owner: infra/group/my-team
  implementsApis:
    # Same as default/api/external-customer-api since this entity had no namespace given
    - external-customer-api

Values in YAML files are not URL encoded.
This assumes that namespaces, kinds and names never will contain slashes.
Option 2 - Unified logic, use colons
Frontend and backend service URLs that want to reference an entity by name will use a colon separated triplet of optional namespace, required kind, and required name. If the namespace is left out, it is implied to be default. The path segment as a whole MAY be encoded, so a reader will have to try to decode it once. Example:
# /catalog/[<namespace>:]<kind>:<name>/builds
/catalog/component:my-service/builds
# - or -
/catalog/component%3Amy-service/builds

YAML files that want to reference an entity by name will use a similar scheme, with the addition that the kind can be left out as well (but only when the namespace is left out), implying that it is contextual. Example:
metadata:
  name: my-service
spec:
  # All explicit
  owner: infra:group:my-team
  implementsApis:
    # Same as default:api:external-customer-api since this entity had no namespace given
    - external-customer-api

Values and colons in YAML files are not URL encoded.
This assumes that namespaces, kinds and names never will contain slashes.

Just because this popped into my mind I'm throwing in
Option 3 - Explicit delimiters
Use different separators for kind and namespace, making it possible to emit either in a clear way, as required by different contexts.
For frontend and backend service URLs, the encoding is similar to option 1 and 2. Example:
# /catalog/<kind>/[<namespace>:]<name>/builds
/catalog/component/my-service/builds
# - or -
/catalog/component/default:my-service/builds

YAML files that want to reference an entity by name will use the same schema, with the addition that the kind can be left out as well. Example:
metadata:
  name: my-service
spec:
  # All explicit
  owner: group/infra:my-team
  implementsApis:
    # Same as api/default:external-customer-api since this entity had no namespace given
    - external-customer-api
    # - or -
    - default:external-customer-api
    # - or -
    - api/external-customer-api
    # - or -
    - api/default:external-customer-api

Values and colons in YAML files are not URL encoded.
This assumes that namespaces, kinds and names never will contain colons or slashes.

For option 3 my brain really wanted it to be api:default/external-customer-api because the delimiters match how I model them in my brain :) even though it may seem a bit more odd in a URL - but again, I argue that it's worthwhile to separate out how URLs are formed from how hand written refs are written, since they have conflicting goals.

For the YAML ref field type, I could even picture it being effectively
string | { namespace?: string; kind?: string; name: string }

Just to have this here then:
metadata:
  name: my-service
spec:
  owner:
    kind: group
    namespace: infra
    name: my-team
  implementsApis:
    - external-customer-api
    # - or -
    - namespace: default
      name: external-customer-api
    # - or -
    - kind: api
      name: external-customer-api
    # - or -
    - kind: api
      namespace: default
      name: external-customer-api

Meh, it is pretty verbose for something that might have to actually be typed out and read quite a bit. Ofc way more explicit though. Possibly a starting point where we can wrap things up in an API that is able to read that type, just so we kinda get that plumbing in place? Then in the future we might end up adding some short form if we feel it's needed.
Regarding URLs it looks like we might end up in a situation where they might be a bit easier to change down the line, so not too opinionated about that. The effort of this decision vs the possible value differences between the different outcomes is pretty high at this point xP - i.e. fine with just going for x/y/z

But I meant that the string form would be the compound option 1 or 3 :)
So I would actually start with the short form, but yes, implementing that function in catalog-model which eats a Json value and spits out a namespace, kind, name triplet object is step uno.

https://github.com/spotify/backstage/pull/2532

Closed by #2532


[JS] [KeyVault] Correct LICENSE files

Fixes #3842

The current (upcoming) recommendation is:
LICENSE. A copy of the MIT license text in a file called "LICENSE.TXT." Include your project name above the copyright statement.
The link to the text for the file is at: https://docs.opensource.microsoft.com/releasing/overview.html under License Files. (apologies for those outside to MS, may be internal link)


SPIKE: Add font selection

Complex font support was added to tCCreate.  It needs to be added as a menu item in gatewayEdit. Note that font sizing is done on a card by card basis.
Can fonts be changed for all the helps cards at the same time?
Can the font be changed at the app level and then customized for scripture cards? Note: The scripture pane functions differently than the individual helps cards.
Is there a way we can do both app level and card level fonts? For instance when they choose their language.

-[ ] tying the fontSize feature in-app level, Currently functions on individual help cards


[Model] Add gemma model support.

TODO list:

[x] [xDNN] add gelu ACT support.
[x] [mlp] add gelu ACT support.
[x] [mlp] fix catup/gate gemm bug.

Add support in example.cpp please.

Add support in example.cpp please.

the Gemma support for example.cpp will be updated in a new PR, not adding to the new content of this PR.

Wait for checking Silu API.