running R2D2 without Docker

I'm trying to run seed rl (R2D2) without Docker on Ubuntu 18.04. I've tried to decouple the files as much as I can from docker. When I try to run r2d2_main.py in leaner mode in the terminal,
 python atari/r2d2_main.py --run_mode=learner --logtostderr --pdb_post_mortem --num_actors=2,
I get this error:
Traceback (most recent call last):
  File "atari/r2d2_main.py", line 27, in <module>
    from seed_rl.agents.r2d2 import learner
  File "/home/dave/Documents/AI/2020_seed_rl/seed_rl/agents/r2d2/learner.py", line 38, in <module>
    from seed_rl import grpc
  File "/home/dave/Documents/AI/2020_seed_rl/seed_rl/grpc/__init__.py", line 21, in <module>
    from seed_rl.grpc.python.ops import *
  File "/home/dave/Documents/AI/2020_seed_rl/seed_rl/grpc/python/ops.py", line 25, in <module>
    from seed_rl.grpc.python.ops_wrapper import gen_grpc_ops
  File "/home/dave/Documents/AI/2020_seed_rl/seed_rl/grpc/python/ops_wrapper.py", line 25, in <module>
    gen_grpc_ops = tf.load_op_library(os.path.join(tf.compat.v1.resource_loader.get_data_files_path(), '../grpc_cc.so'))
  File "/home/dave/anaconda3/envs/dave/lib/python3.7/site-packages/tensorflow_core/python/framework/load_library.py", line 57, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /home/dave/Documents/AI/2020_seed_rl/seed_rl/grpc/python/../grpc_cc.so: undefined symbol: _ZN10tensorflow14DataTypeStringENS_8DataTypeE

The only change I made to r2d2_main.py is add
import sys
sys.path.insert(1, '/home/dave/Documents/AI/2020_seed_rl/')

for path purposes.

Do you build the grpc library in docker? If no, try and do that.
If you do already: I have seen this error before with specific versions of TF, so make sure you use exactly the same.

@lespeholt
My TF version is 2.1.0 which is the same.
I think my issue is with the grpc library as when I ran this simple example I got the same error.
However, as I am quite new to Docker and grpc, I am not quite sure how to 'build' grpc from docker (even after reading up on grpc and docker).
The file structure of grpc in seed_rl seems quite different from those examples given by the tutorial and example repository. The following gave me a lot of confusion:

there is no _pb2_grpc.py generated file as specified in the tutorials.
there seems to be a grpc.cc file and grpc_cc.so file, which is not common if using grpc python.
i am looking for a way to run r2d2 without docker, so it seems that building grpc from docker would not be ideal.

Above all, do we really need the c++ and .so files to use grpc? Is there a way to do it like the examples in the tutorial (using grpc with python) without those files and without docker?
thanks for your patients and I really appreciate all the help I can get thanks ! (:

We are using C++ grpc, not Python grpc.
Do you right now use the prebuild .so file (i.e. you don't try to build the grpc library?

@lespeholt

Do you right now use the prebuild .so file (i.e. you don't try to build the grpc library?

yes im running this
python atari/r2d2_main.py --run_mode=learner --logtostderr --pdb_post_mortem --num_actors=2
on the existing .so file from the repository.

We are using C++ grpc, not Python grpc.

Is there any advantages running c++ instead of python?

I'm not sure what goes wrong for you, the following in Docker works:
FROM ubuntu:18.04
RUN apt-get update && apt-get install -y tmux libsm6 libxext6 libxrender-dev python3-pip
RUN pip3 install --upgrade pip
RUN pip3 install tensorflow==2.1.0
this should be fairly close to what you're doing.

ok @lespeholt I will look into it again on running without docker.
Also, is it possible to use grpc python instead of grpc c++? do you know if there would be any slowly down in speed/bandwidth or if there are any features which cant be implemented in grpc Python?

Using Python grpc would be significantly slower than C++ and the custom batching.

Reviving this. I'm trying to do the same since apparently profiling with nvprof is problematic inside the docker; I'm getting segmentation faults.
I'm getting a very similar error: tensorflow.python.framework.errors_impl.NotFoundError: /home/nvidia/PycharmProjects/seed_rl/grpc/python/../grpc_cc.so: undefined symbol: _ZN10tensorflow8OpKernel11TraceStringEPNS_15OpKernelContextEb
Is there a solution proposed here? I'm not sure I understood. The suggestion to compile grpc within the docker is not relevant, right? since I'm not using docker...

you can still compile grpc inside docker, copy the file and then not use docker at all when you run the training.

How can I run this code without docker?
I did it successfully, here I want to share my experience.
1. First, you can create a virtual environment using conda or virtualenv (my python version is python3.6.7 ), installing the following packages:
absl-py==0.9.0
appdirs==1.4.4
asn1crypto==0.24.0
astunparse==1.6.3
atari-py==0.2.6
cachetools==4.1.1
certifi==2020.6.20
cffi==1.14.0
chardet==3.0.4
cloudpickle==1.3.0
cryptography==2.1.4
decorator==4.4.2
distlib==0.3.1
filelock==3.0.12
future==0.18.2
gast==0.3.3
google-auth==1.18.0
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
grpcio==1.30.0
gym==0.17.2
h5py==2.10.0
idna==2.10
importlib-metadata==1.7.0
importlib-resources==3.0.0
Keras-Preprocessing==1.1.2
keyring==10.6.0
keyrings.alt==3.0
Markdown==3.2.2
numpy==1.19.0
oauthlib==3.1.0
opencv-python==<IP_ADDRESS>
opt-einsum==3.2.1
Pillow==7.2.0
protobuf==3.12.2
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycairo==1.19.1
pycparser==2.20
pyglet==1.5.0
PyGObject==3.36.1
requests==2.24.0
requests-oauthlib==1.3.0
rsa==4.6
scipy==1.4.1
SecretStorage==2.3.1
six==1.15.0
tensorboard==2.2.2
tensorboard-plugin-wit==1.7.0
tensorflow-estimator==2.2.0
tensorflow-gpu==2.2.0
tensorflow-probability==0.9.0
termcolor==1.1.0
urllib3==1.25.9
Werkzeug==1.0.1
wrapt==1.12.1
zipp==3.1.0

2. Second, we need to configure the cuda_10 environment
conda install cudatoolkit==10.1
conda install cudnn -c https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64/

3. Third, we add the seed_rl path to the python path, the pwd is under the seed_rl folder
export PYTHONPATH=$(dirname "$PWD"):$PYTHONPATH

4. Forth, we create 5 tmux windows named learner, actor0, actor1, actor2, actor3
Run the following commands in these 5 windows respectively：
python3 atari/r2d2_main.py --run_mode=learner --logtostderr --pdb_post_mortem  --num_actors=4
CUDA_VISIBLE_DEVICES='' python3 atari/r2d2_main.py --run_mode=actor --logtostderr --pdb_post_mortem  --num_actors=4 --task=0
CUDA_VISIBLE_DEVICES='' python3 atari/r2d2_main.py --run_mode=actor --logtostderr --pdb_post_mortem  --num_actors=4 --task=1
CUDA_VISIBLE_DEVICES='' python3 atari/r2d2_main.py --run_mode=actor --logtostderr --pdb_post_mortem  --num_actors=4 --task=2
CUDA_VISIBLE_DEVICES='' python3 atari/r2d2_main.py --run_mode=actor --logtostderr --pdb_post_mortem  --num_actors=4 --task=3

Note: you should make sure that you have grpc_cc.so file (9.4M) under the grpc folder.
That's all, hope you can succeed！

@zhuliwen thanks!

Excellent, I'll give it a try. Thanks a lot!

FYI, the appropriate version should be used as this repository is updated.
The version is written in docker files.
https://github.com/google-research/seed_rl/blob/master/docker/Dockerfile.grpc
https://github.com/google-research/seed_rl/blob/master/docker/Dockerfile.atari
Now, you need to use
tensorflow-gpu==2.4.1
tensorflow-probability==0.11.0


Unified Utility Script for Groovy Wrapper

Adds a unified script for all utility commands for the Groovy Wrapper.
General Syntax:
groovyw util (type) (sub-command)
Check more information using groovyw util usage

Can one of the admins please verify this patch?

Looking around some - it would be useful in general if you could elaborate a bit on the changes made and any non-obvious design decisions made :-)
From what I can tell initially it looks pretty good, nicely done. The typed-item approach should work. Some notes:

The handling of the GitHub home seems a little suspect. Have you tested that it respects an alternative org home set in gradle.properties ? What about meta modules since they read a different property?
There are some code style issues, mostly minor like whitespace. Missing spaces, sometimes indentation is 2 spaces sometimes 4, etc. It might be possible to scan it with Checkstyle with a bit of tweaking
There are some javadoc/comment bits that could still be updated some, like recurse only working for modules (only indicated in some places at present)
The create item isn't handling the types quite right, but is close. The modules have module.txt and build.gradle at present, but the meta modules doesn't have any Gradle at all and the facades should get the template file for facades, not modules
updateItem seems to have been renamed away from module, but its content still seems to be specific to modules? At least in naming/logging
The Groovy fall through warning override isn't in place for the case switch, so there's a warning about a missing break (which is intentional and why there was an override there)
One of the getUserString calls uses String and the other GString (" vs ') - that should mean the $displayName variable won't expand in one case. Did you test for that?
Could displayName be renamed to something clearer?
I think the module update-all could use a URL check - I managed to get it to break as I had a module where the remote was no longer valid
Trying to test meta leads to trouble as I guess it requires the rename of the "meta" to "metas" - right? That's what I think we should do. If you could confirm with the rename and add the rename to the PR I'd appreciate it :-)

Superseded / merged via #3263 - thanks! :-)


[AIRFLOW-3322] Update qubole_hook to fetch command args dynamically from qds_sdk

Jira

[x] My PR addresses the following jira

https://issues.apache.org/jira/browse/AIRFLOW-3322

Description

[x] Updating Qubole Hook to fetch command args dynamically from qds_sdk

Tests

[x] The existing testcases should be green

Commits

[ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[x] No new functionality for end users, so no doc changes needed

Code Quality

[x] Passes flake8

Codecov Report

Merging #4165 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #4165   +/-   ##
=======================================
  Coverage   77.68%   77.68%
=======================================
  Files         199      199
  Lines       16273    16273
=======================================
  Hits        12641    12641
  Misses       3632     3632

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 52ee34c...7786cfe. Read the comment docs.

@Joylal4896 can you also bump up the min qds_sdk version required. Also, the commit subject should be as per the guidelines.


Add support to load library externally

As pyzbar is a very usable library to handle images, some applications wants to handle images in a very scalable context like serverless platforms (aws lambda, azure function and so on). These platforms run workloads on top of default environments that is not easy to customize to add a shared lib as pyzbar requires.
So, my suggestions is to add a new try during the load library function in pyzbar/zbar_library.py. The code below is showing only trying get the library in PATH, but a good try is find the library in a specific env var like ZBAR_PATH.
else:
        # Assume a shared library on the path
        path = find_library('zbar')
        if not path:
            path = os.environ.get('ZBAR_PATH')
            if not path:
                raise ImportError('Unable to find zbar shared library')
        libzbar = cdll.LoadLibrary(path)
        dependencies = []

Having the same issue with my app deployed on Heroku. Any word on the status of the PR?

I have the same issues with AWS ec2. Don't know how to assign a path

Apologies for this problem (and for my rather late response).
Thanks very much for the PR. I have merged to feature/40-path-to-zbar, where I will add unit tests and make other changes. I have no experience of AWS nor or Heroku.
Where does libzbar.so get installed to on these platforms?
The behaviour of find_library (the Python standard library function used to find libzbar.so) changed in Python 3.6 - "...the value of the environment variable LD_LIBRARY_PATH is used when searching for libraries, if a library cannot be found by any other means". Do you see the behaviour when using Python 3.6 or 3.7?

Hi,
In case of AWS ec2, the library is installed by instance owner, in other hand in heroku and others managed application server, the library get installed in a given docker image or can be load from disk, thus in second case you can embed the lib in application code and load it at run time.
The PR I had pushed, add the option to load the library from any path to handle cases where the installation is not a responsibility of the developer.

OK thanks. It would be good to use the standard mechanisms for finding the library unless there really is no other option.
Do these platforms really not make the shared lib location available on PATH or LD_LIBRARY_PATH? If not, can you not append the correct location to PATH or LD_LIBRARY_PATH, as part of a setup or provisioning step?

Does anyone have an example of how they got this to work with a Lambda function in AWS? I am installing pyzbar in the packaged deploy of my lambda but still getting issues about the zbar shared library.

Can't speak for AWS lambda, but deploying to an Amazon Linus 2.8:
Adding the zbar package via an .ebextensions declaration works:
packages:
  yum:
    zbar: []

This is the equivalent of yum install zbar
Deployment was successful after this and desired functionality via pyzbar was working.

If anyone still has problems using the library to work with heroku, take a look at this. I described my example setup heroku app.

I'm gonna leave this for posterity. In alpine it seems the libzbar gets installed as usr/lib/linzbar.so.0 so is not found by the python module. We had to create a link with the name /usr/lib/libzbar.so.
Here's the complete line for docker alpine
RUN apk --no-cache add libzbar --repository=http://dl-cdn.alpinelinux.org/alpine/edge/community && \
    ln -s /usr/lib/libzbar.so.0 /usr/lib/libzbar.so

@sebastiandev hey there! Looks very interesting. Any idea how to make an AWS lambda layer? The one you use is based on alpine, so it's a bit different for AWS.
I did some research and tried the following:
Creating a file Docerfile:
FROM amazonlinux

RUN yum update -y
RUN yum install -y gcc gcc-c++ make
RUN yum install -y wget
RUN yum install -y tar
RUN yum install -y bzip2
RUN mkdir /lambda-DecodeQR
RUN wget https://jaist.dl.sourceforge.net/project/zbar/zbar/0.10/zbar-0.10.tar.bz2
RUN tar -jxvf zbar-0.10.tar.bz2

RUN cd zbar-0.10 && ./configure --build=x86_64 --prefix=/lambda-DecodeQR CPPFLAGS=-I/usr/include --with-libiconv-prefix=/usr/include --with-python=no --with-libiconv-prefix=/usr/include --without-imagemagick$
RUN ln -s /usr/lib/libzbar.so.0 /usr/lib/libzbar.so
RUN yum install -y zip
RUN yum -y clean all
RUN zip -gr DecodeQR.zip share lib64 lib include bin

Then running the following commands:
docker build -t amzlinuxpy .
docker run -it amzlinuxpy bash # let this run in another tab
docker ps # change the XXX_ID_XXX below and match it with the container ID you see with this command
docker cp XXX_ID_XXX:/DecodeQR.zip DecodeQR.zip

Not proud on the Dockerfile, but i's something.
This zip contains the libraries of zbar, that should work on AWS Lambda inside a Lambda layer. I have no clue what to with this now, because I just learned about AWS Lambda today and I don't understand docker layers completely. I think https://hackernoon.com/how-to-deploy-aws-lambda-with-docker-containers-e51j3141 might be an interesting article to look at for continuing this journey.
Any feedback is welcome, if this is not relevant anymore for you, feel free to yeet it with a 🚀  emoji I guess :)
Might also be interesting for @novomotus, @cmendesce, @3Mcolab because they reference AWS too.

In part to answer the question from @vincentcox, but also to re-open this thread, getting pyzbar under AWS Lambda to load a library provided by the user is not easy, specifically because this proposed patch did not get used.
The problem is that the way that ctypes.utils.find_library() implements searching LD_LIBRARY_PATH relies on calling the ld program, and that is not present on AWS Lambda Python instances. As a result pyzbar can not find libraries using LD_LIBRARY_PATH when running in a modern Lambda instance. Having support for explicitly setting the location of libzbar.so would therefore still be very helpful.
There is an ugly, sub-optimal work-around to this problem that @vincentcox is facing which is to include code in your Lambda handler that explicitly creates a link from /lib64/libzbar.so to the library in your bundle and then also manually rebuild the library cache using os.system("/sbin/ldconfig") or similar, and do this before attempting to import pyzbar. This works because the implementation of find_library() tries printing our the ldconfig cache as one of its ways to try to find a matching library. It should be noted that as well as this technique relying on the fact that Lambda processes run as root within their container, this also has the further disadvantage of adding to the startup time of your Lambda function, since you have to wait for the library cache to be rebuilt.
All in all it would be hugely better if we could just explicitly provide the file path.
As a final note, I believe that @vincentcox 's script above for building the zbar library may not work as required because the current version on pyzbar seems not to work with the v0.10 zbar source. I initially tried pretty much the same method as proposed above and the resulting library wouldn't find QR codes in my images. I have created a gist for building an up-to-date libzbar.so from the more recent (and still supported) fork of zbar on GitHub which should help.

Thank you @nickovs for providing the script for building the Zbar library for AWS Lambda. I followed your gist and generated three libraries libzbar.so (0kB) , libzbar.so.0 (0kB) and libzbar.so.0.3.0 (1002kB). Is the file size correct? How do I continue from here? Do I have to package a .zip together or do I have to use a docker image to upload to AWS Lambda? Sorry - i am fairly new to building my own layers for AWS Lambda and I need to use pyzbar for QR Code detection. Thank you for your help.

@felixb-dev Yes, those sizes are what I get. Note that the first two are just soft links; the third file is the real library.
As for how to use the library, right now you need to apply the patch in #41 to your own copy of pyzbar and include the patched version in the code you deploy in Lambda. Hopefully the patch will get merged onto the master branch and released to PyPI at some point soon, but we need some contributor like @quicklizard99 to do that.

@nickovs Thanks a lot for all your research around this. I would also love to have this PR merged into master. Would help a lot. Or have something like pdf2image package has where you can set the poppler_path in the function.
I'm still pretty fresh on both python and lambda. I have run the script from your gist and added the .so files as a layer. I have forked the repo and added your #41 patch and pulled that repo into my function and set the ZBAR_PATH env variable to the correct in my layer. How can I now import this package? The file structure is:
app.py
--- pyzbar
          [git files and other from the pyzbar repo]
          --- pyzbar
                    --- pyzbar.py

I have tried in my app.py with from pyzbar.pyzbar import decode but get an error that it could not find the module.
Thanks a lot and really hope that @quicklizard99 or some other contributor look at this again. Would be very helpful to be able to set the zbar path somehow.

@MarGul What I did was copied the inner pyzbar directory (i.e. the one that directly contains the Python files) into the base directory that I was packaging up for Lambda. I copied the libzbar.so.0.3.0 file into the same base directory and renamed it libzbar.so, so the files I package up look like this:
lambda_bundle
  - myscript.py
  - libzbar.so
  - pyzbar
    - __init__.py
    - ...

If you do this then you should be able to just use from pyzbar import decode from your Lambda script.

Thanks a lot @nickovs . I had to update the pyzbar/__init__.py to include this code
from .pyzbar import (
    decode
)

All is working now. I have the libzbar.so as a Layer instead. Thanks a lot for your help. Hope this issue get's resolved soon.

I've tried following everything from above and I'm still not getting the zbar library found.
Was a layer's path ever worked out?  If not, a bundled .zip file?
I've tried both.  I've tried patching the zbar_library.py based on #41 and I've compiled and put the libzbar.so in a bundle, in a layer, etc......
I'm working perfectly in windows or ubuntu.
But I cant get past here on aws lambda.
import json
from pdf2image import convert_from_path
from pyzbar import decode

@andrologixx I managed to take a bit of everyone's help on this issue, and created the following repository to create your own Lambda Function working with pyzbar.
I still have not managed to include the shared libraries as a layer, but included them in the bundle .zip and it seems to work that way, if someone manages to make it work as part of a layer happy to modify it!

I'm using AWS SAM to create my function and resources. This is in my template.yaml
ZbarBinary:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: zbar
      Description: The zbar package to be able to use pyzbar
      ContentUri: src/layers/zbar-binary
      CompatibleRuntimes:
        - python3.9
      RetentionPolicy: Retain
    Metadata:
      BuildMethod: makefile

The important part here is the BuildMethod: makerfile. This file basically just copy all the .so files into the layer. You can read more about this here. This is what the Makefile looks like:
build-ZbarBinary:
	mkdir -p "$(ARTIFACTS_DIR)/lib/zbar_binary"
	cp -a * "$(ARTIFACTS_DIR)/lib/zbar_binary"

My ZBAR_PATH environment variable is now  /opt/lib/zbar_binary/libzbar.so.
Hope this helps.

@MarGul What I did was copied the inner pyzbar directory (i.e. the one that directly contains the Python files) into the base directory that I was packaging up for Lambda. I copied the libzbar.so.0.3.0 file into the same base directory and renamed it libzbar.so, so the files I package up look like this:
lambda_bundle
  - myscript.py
  - libzbar.so
  - pyzbar
    - __init__.py
    - ...

If you do this then you should be able to just use from pyzbar import decode from your Lambda script.

Hi @nickovs, How do I get a libzbar.so file for AMI-2 (CentOS 7 based image)? It looks like you're using ubuntu based .so file.

@souravjamwal77 The gist script I posted builds the library for running on AWS Lambda instances, which run a Red Hat derived Linux, but if you need a copy for Centos7 you can just replace the Docker image name amazonlinux:2 with centos:7 and it seems to build just fine.

I used the following solution for AWS, it's using a layer approach, which is a little different from @MarGul :

create the zip file for a lambda layer with the zbar libraries. You can follow @DanyStinson repo. This will give you the zip file for zbar in your s3 bucket
use the zip file to create a lambda layer on AWS and save the ARN adress
Adjust your SAM local template. It should look something like the following:

Resources:   OCR:     Type: AWS::Serverless::Function      Properties:       PackageType: Image       Layers:         - arn:...:layer:pyzbar:1

You should be able to build your serverless architecture now with sam build and then invoke it with sam invoke

For anyone attempting to get a standalone AWS layer, I was able to do so by using the following Dockerfile while follows the same approach from @DanyStinson above. Uploading pyzbar_layer.zip directly to an AWS layer worked correctly for me.
# use latest https://gallery.ecr.aws/sam/build-python3.12
FROM public.ecr.aws/sam/build-python3.12:1.121.0-20240730174605

WORKDIR /var/task

RUN dnf update -y && \
    dnf install -y autoconf gettext-devel automake pkgconfig libtool git make gcc-c++ zip

RUN pip install pyzbar -t python/lib/python3.12/site-packages/

RUN git clone https://github.com/mchehab/zbar.git && \
    cd zbar && \
    autoreconf -vfi || true && \
    ./configure && \
    make

RUN cp zbar/zbar/.libs/libzbar.so.0.3.0 python/lib/python3.12/site-packages/pyzbar/libzbar.so

RUN sed -i "s/find_library('zbar')/('\/opt\/python\/lib\/python3.12\/site-packages\/pyzbar\/libzbar.so')/g" python/lib/python3.12/site-packages/pyzbar/zbar_library.py

RUN zip -r pyzbar_layer.zip python


Unable to access Jfrog UI

The antifactory images has been successfully created with artifactory-oss.yml file. Due to permission issue of artifactory user, I have added the     user: "${UID}:${GID}" in compose file and able to run the container.
But after creating, not able to access the Jfrog UI. I tried to curl http://localhost:8080/artifactory inside the contaier and getting connection refused.
Could you please advise anyone how can I access the UI? I have used the same yml file in the code

Which version of artifactory are you using?


Realm upgrade is crashing the app

How frequently does the bug occur?
All the time
Description
Old app Realm version 4.4.1
New version of app[upgrade from old] with Realm 5.5.2
Try to create a new entry in to the DB crashes the app.
The app is crashing with the following exception:
Terminating app due to uncaught exception 'RLMException', reason: 'Key already used'
Stacktrace & log output
Terminating app due to uncaught exception 'RLMException', reason: 'Key already used'

Can you reproduce the bug?
Yes, always
Reproduction Steps
Try to create a new entry in to the DB crashes the app.
Version
5.5.2
What SDK flavour are you using?
Local Database only
Are you using encryption?
No, not using encryption
Platform OS and version(s)
13.7 onwards
Build environment
Xcode version: ...
Dependency manager and version: ...

Hi @vaibhavlaghane any reason you are not upgrading to our latest version v10.15.1?

Hello,
Appreciate your response.
Looking to work with minimal version change as dependent libraries in the app use 4.4.1 or 5.5.2 .
Testing and releasing them will be huge task.
Is there a file format change between the versions?
Any specific script to upgrade?
Is there a patch fix for 5.5.2 ?
Thanks


Move web-console dependency declaration from druid-server to druid-distribution

Description
This is a follow up of #12486
#12486 removes the web-console dependency from druid-server module because the dependency check reports that it's not used in druid-server module.
Because druid-server module is referenced by many other modules, the existence of web-console dependency in druid-server module makes that module to be compiled after the web-console module. However, building of web-console module usually takes a long time. If the whole module is built in parallel with -T mvn parameter, this dependency makes the building slower.
After #12486 is merged, @vogievetsky found that this dependency must be there or the web-console module won't be shipped in the distribution package(https://github.com/apache/druid/pull/12486#discussion_r863492827)
The druid distribution package is packaged by the mvn-assembly-plugin referenced in the druid-distribution module.
https://github.com/apache/druid/blob/0206a2da5c6211caf451751876d847f40c2e7755/distribution/pom.xml#L256-L258
This plugin is also configured to copy all dependencies declared in the pom of druid-distribution to the lib directory as below.
https://github.com/apache/druid/blob/0206a2da5c6211caf451751876d847f40c2e7755/distribution/src/assembly/assembly.xml#L126-L129
So, we can move the dependency declaration from the druid-server module to the druid-distribution module so that we still gain the  building speed up as well as the package integrity.
This PR has:

[X] been self-reviewed.

[ ] using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)

[ ] added documentation for new or modified features or behaviors.
[ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
[ ] added or updated version, license, or notice information in licenses.yaml
[ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
[ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
[ ] added integration tests.
[X] been tested in a test Druid cluster.

@FrankChen021, thanks for making this fix: the revised dependency looks right: it should be distribution that depends on the console, not some module which distribution depends on. That is, make the dependency direct, not transitive.
Adding an IT is usually a Good Thing. There is, however, a big cost to adding a test group in the current setup: we build the entire Druid stack for each test group. I've been chipping away at First cut at restructuring the integration tests to make the ITs simpler and faster: your new test is a good candidate to port.
In the mean while, can the IT be added to an existing group? It can just piggy-back on any one of the existing clusters, can't it?

@paul-rogers Thanks for the review.
I need to make some changes to the CI. Since the web-console is moved to distribution, building of IT module does not involve packaging the web-console module, the newly added test can not pass. To make the test in a separated group can let it run independently in a separated CI job, in that way, other jobs don't need to packaging the web-console module which can save time.

Since #12776 has moved web-console e2e test to phase 1, there's no need to an IT test anymore, revert these changes.

Hello @clintropolis @vogievetsky Could you review this PR?

By this adjustment, the building speed with -T 1C parameter on mvn command is improved from about 2:50min to about 2min on my Mac book.

Hello @clintropolis Any update on this PR?

Thank you @clintropolis

hmm, I think this might have broken how I was debugging druid services in intellij using run configurations (similar to https://github.com/apache/druid/blob/master/dev/intellij-setup.md#coordinatorxml), since the web-console is no longer working for me. I haven't verified that it is this change yet, but it seems pretty likely given that the above setup is not using the distribution packaging at all.

I haven't verified that it is this change yet, but it seems pretty likely given that the above setup is not using the distribution packaging at all.

confirmed it is related to this change, I'll see if I can find a solution to my debugger problem, since I use that method of debugging fairly frequently...

Hello @clintropolis
Sorry to hear that. I didn't use debug from Intelli to lanch process, instead of I usually use 'attach' to an existing running process.
Since web-console is served byt router, do you mean you debug router in this way?

I'm wondering if we can use 'profile' to include the web-console in the server/pom.xml and active such profile from intellij when debugging?

I'm wondering if we can use 'profile' to include the web-console in the server/pom.xml and active such profile from intellij when debugging?

I think it's possible. Intellij provides a profile panel that allows us to active specific profiles.

We can keep that newly added profile selected from Intellij. @clintropolis Do you have any other ideas?

I think I've got it working well enough using -Xbootclasspath to customize what the router has access to when it starts up. Yeah, I have a handful of xml files that I use to launch druid processes directly within intellij. Extensions (and now the web-console) are a bit funny in this model as you have to externally build them and refer to their location, but works pretty well otherwise.
Here is my Router.xml for example (after fixing):
<component name="ProjectRunConfigurationManager">
  <configuration default="false" name="Router" type="Application" factoryName="Application">
    <option name="ALTERNATIVE_JRE_PATH" value="1.8" />
    <option name="MAIN_CLASS_NAME" value="org.apache.druid.cli.Main" />
    <module name="druid-services" />
    <option name="PROGRAM_PARAMETERS" value="server router" />
    <option name="VM_PARAMETERS" value="-server -Xbootclasspath/a:$PROJECT_DIR$/conf/router/:$PROJECT_DIR$/web-console/target/web-console-25.0.0-SNAPSHOT.jar -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Xmx256M -Xmx256M -XX:+UseG1GC -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dorg.jboss.logging.provider=slf4j -Dlog4j.configurationFile=$PROJECT_DIR$/core/src/main/resources/log4j2.debug.xml -Ddruid.host=localhost -Ddruid.zk.service.host=localhost -Ddruid.emitter=logging -Ddruid.router.managementProxy.enabled=true -Ddruid.extensions.directory=$PROJECT_DIR$/distribution/target/extensions/ -Ddruid.extensions.loadList=[\"druid-histogram\",\"druid-bloom-filter\",\"druid-datasketches\",\"druid-lookups-cached-global\",\"druid-basic-security\"] -Ddruid.server.http.numThreads=50  -Ddruid.generic.useDefaultValueForNull=false " />
  </configuration>
</component>

If you put these xml files in .idea/runConfigurations/ then you have all of these options to run various services in the run/debug dropdown menu:

I do this because in my experience its got a faster turn-around than building a quickstart and attaching, can modify most stuff on the fly and live reload, etc.

Glad to hear it works. Sorry for the inconvenience caused by this change. @clintropolis


Fix: Ensure Unique AdmnMessageID() Generation

Updated the AlphaSerialNumber function to guarantee unique message ID generation even when multiple IDs are created simultaneously. By eliminating the dependency on the current time, this fix ensures that message IDs remain distinct regardless of generation timing. This adjustment mitigates potential conflicts in message identification. This function is called inside AdmnMessageID(). (The issue will occur when you have two sets of queues and you are trying to echo at the same time in both queues. Generated message IDs will be exactly the same if they depend on time.)

I like this, but we knew that time based sequences will conflict. Random numbers will still conflict and will produce out-of-sequence values, which may be unexpected. Perhaps we offer a version of these ID functions which take a generator function.

We have tried using this library and faced that challenge. Hence, proceeding with this approach seems appropriate.(as of now)

Thanks for opening these PR's. I've opened https://github.com/moov-io/rtp20022/pull/61 which is a bit more generic and applies to every serial number as they all need this fix. I've included a serial generator using crypto/rand as well.


I find a bug when picking up coordinates on 3dtiles

When I pick up coordinates at the edge of the 3D tiles model, I often make an error (as shown in the picture), which causes the program to crash.
Browser: Google Chrome  73.0.3683.86
Operating System: Win10

@githubhanjunjun can you put together a Sandcastle example that reproduces the crash?  That will help us track down where the problem is. Thanks!

Closing due to inactivity


RFC: dts: silabs: use 'aliases' to remove dts_fixup defines

Following PR is a request for comments. The change is implemented for efr32fg1p SoC only.
By adding 'aliases' node in SoC .dtsi file it is possible to generate DT_ defines which specify a logical name rather than relay on a module location on APB bus. E.g. DT_SILABS_GECKO_USART_40010000_LABEL becomes DT_SILABS_GECKO_USART_USART_0_LABEL. Thus it is possible to remove dts_fixup.h defines. There is no limit on adding aliases for a given node and user is free to add a different one. Nordic family SoCs use the same approach.
Another possibility to avoid adding dts_fixup entries is to modify the script which generates DT_ defines. A DTS node is defined as
[label:] node-name[@unit-address] {
	[properties definitions]
	[child nodes]
}

e.g.
usart0: usart@40010000 { /* USART0 */
		[...]
};

where usart0 is a node label. If a script used a node label rather than a unit address the resulting defines would relay on logical names and not on part number dependent, fixed addresses. However, since this approach was not implemented by maintainers of DTS scripts there may be a good reason to avoid it.

@ekarlso, @diegosueiro, @mtuxpe I've prepared another PR as an RFC with a goal to remove reliance on dts_fixup.h file. The change was done for efr32fg1p series only. If this approach is accepted remaining SiLabs series would also need to be updated. Your comments are welcome.

Codecov Report

Merging #11707 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #11707   +/-   ##
=======================================
  Coverage   48.21%   48.21%
=======================================
  Files         279      279
  Lines       43308    43308
  Branches    10370    10370
=======================================
  Hits        20880    20880
  Misses      18282    18282
  Partials     4146     4146

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c43a116...cd9c388. Read the comment docs.

I've pushed a new version that implements required changes for all SiLabs SoCs and boards. This PR is now suitable for merging.
I confirmed that all SiLabs boards compile with no error. However, some more testing would definitely be welcomed. Especially verifying that i2c, usart, uart, leuart modules work in real life on the board.


Question about using ordinary (digital) optimizers on analog layers

Hi!
It seems my models also train when using Adam or SGD, instead of the AnalogSGD provided in the package. Am I safe to assume that the analog trainers are meant for on-device training, whereas hardware-aware training can (and should) be optimally performed using regular trainers?
Or are there any important differences that I should be aware of when aiming for optimal hardware-aware training?
Thanks in advance!

Hi @Winfriedvdd:
that is correct, the hardware-aware training can be done with any optimizer from pytorch (or with AnalogSGD which then defaults to standard SGD). The analog-aware optimizers are only necessary for analog training.


Could not build uvloop: uvloop/loop.c no such file or directory

Hi,
I tried to build the project to run the test on my machine (Ubuntu 14.04, 64b).
First I had to install automake and libtool (you may want to add this step on the readme), then running make I got:
    building 'uvloop.loop' extension
    creating build
    creating build/temp.linux-x86_64-3.5
    creating build/temp.linux-x86_64-3.5/uvloop
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -I/home/user/.local/share/virtualenvs/ace813a9968fde5/include/python3.5m -Ivendor/libuv/include -c uvloop/loop.c -o build/temp.linux-x86_64-3.5/uvloop/loop.o
    x86_64-linux-gnu-gcc: error: uvloop/loop.c: No such file or directory
    x86_64-linux-gnu-gcc: fatal error: no input files
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 4
    make: *** [compile] Erreur 1

The project isn't even in its "alpha" state right now ;)
To build you need cython, and to clone the repo with '--recursive'.  Then simply run "make"

I just wanted to test it here to report failing test, hoping it would help you since I like the project. If you think it's too early for that, I can do that when the project is more mature. I don't want to add noise.
After installing cython, I could run the test and good news:
$ make test
PYTHONASYNCIODEBUG=1 python3 -m unittest discover -s tests
.................................................
----------------------------------------------------------------------
Ran 49 tests in 13.877s

OK
python3 -m unittest discover -s tests
.................................................
----------------------------------------------------------------------
Ran 49 tests in 1.992s

OK

Let me know if you need me test more stuff :)

I'm working on the project right now.    I think we'll have an alpha pretty soon.

Just pinging to get some news on the project. Again, if you need me to run tests on some machines, let me know.

I'm a bit distracted by a few other projects, but I'm going to find some time to push uvloop to alpha soon. I need it for our core product in my firm, so be sure -- the project isn't dead ;)

Closing this one.


Background removal

I would like to add a project for background removal from a image.
Example: a image of person with garden in the background, after implementation of this feature we will have only the person parts of the garden would be removed.

@sudipg4112001, Your issue is not in this project domain. For now I am closing this issue.


Updated Peoria, IL (/peoria)

closes #1600

It looks like there are conflicts -- also why the extra line space? @rpalakkal

I normally (for new emails prs) add one line of space between paragraphs for readability


Support for calling custom SQL Server functions that return a scalar value?

I'm using node-mssql that relies on your module to call a SQL Server function, and getting the error:
TypeError: Cannot set property 'value' of undefined
This appears to be occurring because there is no "output" variable name, that is to say, if I'm supposed to set one, I don't know what to use.
This is unlike calling a Stored Procedure where you need to declare INPUT and OUTPUT parameters/variables.  Though I am declaring the INPUT parameters for my function call, I'm not declaring what the output is...but it does seem reasonable that I might need to for this library.
Any help would be appreciated.

You're going to need to provide a complete example which demonstrates your issue, and the complete error message, including the stack trace, if you're hoping for someone to help you.

I can appreciate that, but I was given a workaround, using a SELECT statement instead of referencing the function as a Stored Procedure, which is working fine for the moment as I'm just trying things out.
http://stackoverflow.com/questions/32428025/how-does-one-call-a-sql-server-scalar-function-using-the-mssql-package-for-nodej

If you believe that using the Stored Procedure approach will result in a speed improvement (I'm not implying it's slow now, but I'm OK with faster :) ), then I'd be happy to provide more info.

Either providing an output parameter or using a select statement seems like the right way to me. Sounds like you got it worked out.

As a custom SQL Server function doesn't have a named output parameter (as
far as I know), what name would I use?
The error was occurring in the tedious code where the value of the named
property was being set, but as no output parameter was defined/named, the
code is trying to set value on a null object.
On Mon, Sep 7, 2015 at 2:35 PM Bret Copeland<EMAIL_ADDRESS>wrote:

Either providing an output parameter or using a select statement seems
like the right way to me. Sounds like you got it worked out.
—
Reply to this email directly or view it on GitHub
https://github.com/pekim/tedious/issues/311#issuecomment-138365863.

A stored procedure is not the same thing as a function. If it's a function, not a stored procedure, then calling it as a stored procedure isn't going to work. If it's a function, then using SELECT is the right way to get its return value. There is no other way that I'm aware of.
For a stored procedure you have the option of calling it directly and providing output parameters, or you can work it into a larger SQL statement and still use SELECT on the values you want.

But even when calling an UDF in an incorrect way, tedious probably should not throw TypeErrors. @cmeans Could you please provide a very simple example to reproduce this issue?

Sure.
Use this as replacement code for the nodejs code in the stackoverflow.com question above.
function performPointInPoly(
    coordinates,
    cb)
{
    var request =
        new sql.Request(
            connection);

    request.input(
        'F1',
        sql.Float,
        coordinates.latitude);
    request.input(
        'F2',
        sql.Float,
        coordinates.longitude);

    request.execute(
        "[dbo].[udfTestScalar]",
        function(err, recordsets, returnedValue)
        {
            if (err)
            {
                cb(err);
            }
            else
            {
                cb(
                    null,
                    returnedValue);
            }
        });
}


Multiroot workspace: AL extension tries to compile other projects in the workspace.

I have created a VS Code workspace that includes an AL project, and a related Azure Function that is used by the AL extension. When I try to debug the Azure extension I get the following error:
error AL1001: Source file '/Path/to/AzureAppFolder/app.json' could not be found
Error: The package could not be created.
It looks like the AL compiler is trying to also compile the AzureApp, and that causes the Azure App compilation to fail.
(If I open the AzureApp folder on its own, it compiles and debugs without problem).
To reproduce it:

Create an Azure App project.
Create an AL project.
Create a VS Code workspace that includes both projects/folders.
Select the Azure App debug configuration and press F5.
Instead of compiling and running the Azure App, it tries to compile the Azure App using the AL compiler and fails.

Tested on macOS with version 2.0.56865 of the AL extension.

We do not support mixing/debugging different project types in a multi-root workspace for now.

Fine, I will mark that as an enhancement.

@kalberes
In case you manage different extensions as different projects, it seems challenging to apply the same company rules / settings accross these extensions / projects, re-use the company snippets, ... Therefore, we've experienced using VSCode workspaces and have included a 'shared' folder containing our company snippets / templates / code samples / guidelines / wiki / ... This 'shared' folder does not contain an app.json, and thus results in the app.json error as mentioned here.
However, on https://docs.microsoft.com/en-us/dynamics365/business-central/dev-itpro/developer/devenv-multiroot-workspaces it is stated:

Grouping a set of disparate project folders into one workspace
It is not mandatory to use only AL-based roots. Different kinds of projects can be mixed ...

Am I misunderstanding the above, telling me that 'mixed' projects should be supported?

Yes, I think this scenario should be supported. I will mark it as a bug

Any progress on this @kalberes?
We're having our pipeline yaml files in a root folder in our workspace, giving us this issue.
Just to get rid of the error I've added an app.json to our Pipelines root folder:
{
    "name": "Pipeline - not an app, only to workaround error from the AL Language Extension",
    "publisher": "NoOne",
    "id": "00000000-0000-0000-0000-000000000000",
    "version": "<IP_ADDRESS>"
}

I haven't seen any undesired side effect of that yet...

Not yet, but now that I have accepted it as in progress I will get the bug. If it is not something that would require major redesign of our vscode integration i should fix it for 15.1.

Great, thanks for the update @kalberes!

Has a design decision been made on the fix?
app.json can't be encapsulated on a multiroot workspace nor in a subdirectory of a repo.
We can suppress errors with @jwikman 's fix but e.g. relative paths in control-addins also seem to just check the root of the current project, instead of the nearest app.json (you get an error if you open the project from another folder level), so is there a way to be more direct on the AL projects' roots?
Hopefully we're not supposed to develop extensions and control-addins in multiple instances of vscode, since it 1. requires you to manually open them (instead of just defining a .code-workspace file) and 2. it's not as compact for related but encapsulated projects, you might want to separate your other build tools and dependencies from the AL compiler, while still having them open in vscode.
Will it be possible to specify the AL project's root path?
Or the other way araound - specify paths for the compiler to ignore (include/exclude paths)?

Somewhat related: Is there any more information available on how to handle multiple AL projects in one VS Code workspace? One of our colleagues told us that at Directions a feature had been demoed showing updates on one app where immediately reflected on dependend apps.
Usecase we've in mind is to create our own 'ISV System Module' project and 'ISV Base Application' project and build other dependend extensions on top of these. Having one VS Code workspace where all extensions reside would allow us to do modications (or at a later stage moving objects) accross these different extensions (system > base > dependend) and package / build / publish all at once from within the same workspace.

@fvet Just including both projects in the workspace is enough. The AL extension detects that one project depends on another project within the workspace, and when you add a field to a table (for example) in one project, the one that depends on it detects the change immediately, instead of having to publish, download symbols, etc. as before. Furthermore, when you publish the extension that depends on the other, it will publish both to the server if the dependency has changed since the last installed version.
I was in one of the sessions at Directions where they mentioned it, and it worked in the few projects where I've tried it 😄
P.S. That being said, please keep the thread on-topic. I'd like to be able to use workspaces when an AL extension depends on an Azure Function...

I hope some of you saw Vjeko's presentation on NAV TechDays today, Control AddIns Supercharged, where he shared his frustration over this issue.
The AL extension should not pollute sibling workspaces (which can be non-AL related) with rad.json, complain over missing app.json, etc.

This issue has been fixed as far as I know. It should already be available on the internal daily builds. And it will also be available on the next 4.X update of the Al extension package.

The fix for this issue has been checked in to the master branch. It will be available in the bcinsider.azurecr.io/bcsandbox-master Docker image starting from platform build number 38063.
If you don’t have access to these images you need to become part of the Ready2Go program: aka.ms/readytogo
For more details on code branches and docker images please read:
https://blogs.msdn.microsoft.com/nav/2018/05/03/al-developer-previews-multiple-releases-and-github/
https://blogs.msdn.microsoft.com/freddyk/2018/04/16/which-docker-image-is-the-right-for-you/


Segfault in delete_wal_older_than

Describe the bug
A segfault occurs when pgmoneta attempts to delete historic WALs
Dec 13 01:25:00 server02.example.com kernel: pgmoneta[3527739]: segfault at 5565a6579 ip 00007f125d869e41 sp 00007ffdaab1de90 error 4 in libc-2.28.so[7f125d7ce000+1bb000]
Dec 13 01:25:00 server02.example.com kernel: Code: 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 53 48 83 ec 10 48 8b 05 c8 30 32 00 48 8b 00 48 85 c0 0f 85 84 00 00 00 48 85 ff 74 6f <48> 8b 47 f8 48 8d 77 f0 a8 02 75 >
Dec 13 01:25:00 server02.example.com systemd[1]: Starting Check pmlogger instances are running...
Dec 13 01:25:00 server02.example.com systemd[1]: Started Process Core Dump (PID 3527741/UID 0).
Dec 13 01:25:00 server02.example.com systemd[1]: Started Check pmlogger instances are running.
Dec 13 01:25:00 server02.example.com systemd-coredump[3527743]: Process 3527739 (pgmoneta) of user 1001 dumped core.

                                                                                Stack trace of thread 3527739:
                                                                                #0  0x00007f125d869e41 __libc_free (libc.so.6)
                                                                                #1  0x00007f125dba817a delete_wal_older_than (libpgmoneta.so.0)
                                                                                #2  0x00007f125dba83cb pgmoneta_delete_wal (libpgmoneta.so.0)
                                                                                #3  0x00007f125dbcd0c4 retain_execute (libpgmoneta.so.0)
                                                                                #4  0x00007f125dbbae7d pgmoneta_retention (libpgmoneta.so.0)
                                                                                #5  0x00007f125f477f63 ev_invoke_pending (libev.so.4)
                                                                                #6  0x00007f125f47bff8 ev_run (libev.so.4)
                                                                                #7  0x00005565a601f55e ev_loop (pgmoneta)
                                                                                #8  0x00007f125d808d85 __libc_start_main (libc.so.6)
                                                                                #9  0x00005565a601ff6e _start (pgmoneta)

To Reproduce
Steps to reproduce the behavior.
Version
$ pgmoneta -V
pgmoneta 0.8.0

libc
glibc-2.28-225.el8_8.6.x86_64
OpenSSL
What is the version of OpenSSL ?
OS
Red Hat Enterprise Linux release 8.8 (Ootpa)
$ uname -a Linux server02.example.com 4.18.0-477.27.1.el8_8.x86_64 #1 SMP Thu Aug 31 10:29:22 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
Configuration
Can you provide the configuration pgmoneta ?
[pgmoneta]
host = *
metrics = 5001

base_dir = /home/postgres/cluster/Backup/

compression = zstd

retention = 7

log_type = file
log_level = debug1
log_path = /home/postgres/cluster/Logs/pgmoneta.log
log_mode = create

unix_socket_dir = /tmp/

[primary]
host = localhost
port = 6000
user = repl
wal_slot = backup
#wal_shipping = /mnt/backup/cluster/

Debug logs
Can you provide any debug logs (log_level = debug5) of the issue ?
Tip
Use ``` before and after the text to keep the output as is.

Oops, I'll look into this.

coredumpctl debug
addr2line -e ./libpgmoneta.so.0.8.0 -fp 0x1926a
delete_wal_older_than at /home/jews/PostgreSQL/pgmoneta/official/src/libpgmoneta/delete.c:266 (discriminator 3)

I think it's fixed. It entered error section by mistake. So sorry for the trouble, this part is quite hard to test out.

And thanks a lot for the core dump. That saves me a lot of time.

Thanks!


RuntimeError: PytorchStreamReader failed reading file data/0: invalid header or archive is corrupted

Please check that this issue hasn't been reported before.

[X] I searched previous Bug Reports didn't find any similar reports.

Expected Behavior
Generate the correct Lora after training is completed
Current behaviour
Using the following command to merge models, there is an error message：
python3 -m axolotl.cli.merge_lora sft_34b.yml --lora_model_dir="/workspace/axolotl/output/Yi-34B/ljf-yi-34b-lora" --output_dir=/data1/ljf2/data-check-test

Steps to reproduce
The meaning of this parameter is not effective
save_safetensors: true
Actually generated
adapter_model.bin
Config yaml
No response
Possible solution
No response
Which Operating Systems are you using?

[X] Linux
[ ] macOS
[ ] Windows

Python Version
3.10
axolotl branch-commit
main
Acknowledgements

[X] My issue title is concise, descriptive, and in title casing.
[X] I have searched the existing issues to make sure this bug has not been reported yet.
[X] I am using the latest version of axolotl.
[X] I have provided enough information for the maintainers to reproduce and diagnose the issue.

is the issue that the merge doesn't work? or that specifying save_safetensors produces a Pytorch_model.bin? or both?

Using the following command to merge models, there is an error message：

Hey, seems like the PeftModel loading failed. Can you check the files in lora_model_dir are valid (aren't a few KBs)?
Did you run out of space during training?
Since the training had a few checkpoints, could you try pointing the model_dir to one of those and see what happens?

The meaning of this parameter is not effective
save_safetensors: true

I'm a bit confused at this as the you said the merge failed.

Now I am continuing SFT training from checkpoint and reporting this error again
I have configured this parameter：
use_reentrant: true
resume_from_checkpoint: /workspace/axolotl-main/checkpoint-5865

resuming from a "peft checkpoint" is not the same as resuming from a regular checkpoint. You'll want to set lora_model_dir to point to the checkpoint directory iirc. @NanoCode012 does that sound right?


scipy.sparse.csgraph.reverse_cuthill_mckee : response function of the size of the matrix

<<Please describe the issue in detail here, and for bug reports fill in the fields below.>>
For a small matrix (NxN) with N=20, the result is correct but for larger matrix the result is flipped like with numpy.flip !
Reproducing code example:
<<A short code example that reproduces the problem. It should be self-contained, i.e., possible to run as-is via 'python myproblem.py'>>

Error message:
<<Full error message, if any (starting from line Traceback: ...)>>

Scipy/Numpy/Python version information:
<<Output from 'import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)'>>

Please provide explicit example.

no response, closing


chore(release): 2.37.0

See CHANGELOG

AWS CodeBuild CI Report

CodeBuild project: githubautobuild-for-cdk-v2
Commit ID: f6c7dc92e7234da87935a96a87bc0f934c19ab9d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: githubautobuild-for-cdk-v2
Commit ID: f6c7dc92e7234da87935a96a87bc0f934c19ab9d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: githubautobuild-for-cdk-v2
Commit ID: 221523f26d569d371ef66970d1dffc1c9659999f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository


QueenASTSemanticValidationVisitor.java: Continue implementing AST nod...

The puzzle<PHONE_NUMBER> originating from #33 has to be resolved:
https://github.com/jvmqueen/queen-of-java/blob/4bef7722a573b6eeb27f61e640a49ec049b9df43/src/main/java/org/queenlang/transpiler/QueenASTSemanticValidationVisitor.java#L41-L41
"Continue implementing AST nodes such as Statements and expressions.".
The puzzle was created by amihaiemil at 2022-12-26 17:49:10 +0200.
Estimation is 60 minutes.
If you have any technical questions, don't ask me, I won't be able to help. Open new issues instead.

@amihaiemil this is your task now, please go ahead. Deadline (when this ticket should be closed) is 2023-01-05T15:50:31.075602.
Estimation here is 60 minutes, that's how much you will be paid.
Remember, you don't have to solve everything in this ticket - you can solve it partially and leave todo markers in the code, which will become future tasks.
If you have any questions don't ask me, I'm not a technical person. Open new tickets instead.
If you don't have time or simply don't want to work on this, you can always resign.

@amihaiemil Don't forget to close this ticket before the deadline (2023-01-05T15:50:31). You are past the first half of the allowed period.

@amihaiemil Looks like you've missed the task deadline (2023-01-05T15:50:31). You are now resigned from this task.
Please stop working on it, you will not be paid. I will assign it to someone else soon.

@amihaiemil I couldn't find any assignee for this task. This is either because there are no contributors with role DEV available or because the project does not have enough funds.
Please, make sure there is at least one available contributor with the required role and the project can afford to pay them.

Puzzle disappeared from the code, that's why I closed this ticket.


Drop support for Python 2.4

This is partly because Tox dropped support for Python 2.4 in v1.4.

I have forked this library and dropped python2 support entirely here – https://github.com/PennyDreadfulMTG/pystache
The main reason for doing this is that the current build process no longer works because setuptools has dropped support for the use_2to3 option.


add information about kubernetes integration issue in 6.8.0 to the changelog

Codecov Report

Merging #2790 into master will increase coverage by 0.67%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2790      +/-   ##
==========================================
+ Coverage   57.09%   57.76%   +0.67%
==========================================
  Files         383      354      -29
  Lines       24556    19862    -4694
==========================================
- Hits        14021    11474    -2547
+ Misses       9612     7562    -2050
+ Partials      923      826      -97

Impacted Files
Coverage Δ

pkg/collector/runner/runner.go
53.15% <0%> (-13.87%)
:arrow_down:

pkg/metadata/resources/resources.go
75% <0%> (-8.34%)
:arrow_down:

pkg/quantile/key.go
46.15% <0%> (-7.18%)
:arrow_down:

pkg/util/kubernetes/clustername/clustername.go
57.89% <0%> (-5.75%)
:arrow_down:

pkg/procmatch/signature.go
66.66% <0%> (-4.77%)
:arrow_down:

pkg/metrics/event.go
67.1% <0%> (-4.64%)
:arrow_down:

pkg/metrics/service_check.go
56.71% <0%> (-4.33%)
:arrow_down:

pkg/collector/corechecks/embed/jmx/runner.go
54.54% <0%> (-4.08%)
:arrow_down:

pkg/collector/py/py.go
76.59% <0%> (-3.67%)
:arrow_down:

pkg/logs/service/service.go
71.42% <0%> (-3.58%)
:arrow_down:

... and 237 more


0.001 for Mean test fail is too sensitive. Change to 0.1

Is it possible to configure this per test?  My guess would be that some tests have different tolerances.

@viathor  I'm not sure, but https://travis-ci.com/quantumlib/Cirq/jobs/207496954 failed after the merge
Performance has regressed:
	test_example_runs_hello_line_perf (0001_dde1a77) - Field 'mean' has failed DifferenceRegressionCheck: 0.005218593 > 0.001000000
	test_example_runs_bcs_mean_field_perf (0001_dde1a77) - Field 'mean' has failed DifferenceRegressionCheck: 0.002496049 > 0.001000000
--------------------------------------------------------------------------------
ERROR: Performance has regressed.

Maybe 0.01 will do. It's seems like guesswork.

I'm going to leave this as is even though it seems conservative because other people's builds will fail with what is currently in the mainline.

If we cannot specify this per-test as @dabacon suggested, maybe we can use percentages instead of seconds?
Also, a more rigorous way to arrive at the threshold would be to run the tests many times (20-100) and compute standard deviation of their runtimes. Then you can use 68-95-99.7 rule or something similar to decide on the threshold. WDYT?

I'm going to merge this so the test always passes. We can take this conversation to https://github.com/quantumlib/Cirq/issues/1124.


[BUG] 403 Forbidden error on /api/saved_objects/_find?fields=title&per_page=10000&type=index-pattern

What is the bug?
The Discover page is blank in OpenSearch Dashboards with a 403 Forbidden error on /api/saved_objects/_find?fields=title&per_page=10000&type=index-pattern with the following message:
no permissions for [indices:data/read/search] and User [name=test, backend_roles=[], requestedTenant=null]: security_exception: [security_exception] Reason: no permissions for [indices:data/read/search] and User [name=test, backend_roles=[], requestedTenant=null]

How can one reproduce the bug?
Follow all the steps of the following Example: https://opensearch.org/docs/2.16/security/access-control/users-roles/#example
Even after adding indices:data/read/search + indices:data/read/search* + indices:data/read/search/template in both Cluster and Index permissions, the error occurs.
What is the expected behavior?
The Discover page should show the data.
What is your host/environment?

OS: Ubuntu 22.04
OpenSearch: 2.16.0

[Triage] @sebastienbonami can you share a little bit more about the full role definition of the user making the call.
The saved_objects API queries the .kibana* indices. This is also what the tenant_permissions section of a security role is for if you have a cluster with multi-tenancy.

Here's the result of the query on _opendistro/_security/api/roles/bulk_access:
{
    "bulk_access": {
        "reserved": false,
        "hidden": false,
        "cluster_permissions": [
            "cluster_composite_ops",
            "test"
        ],
        "index_permissions": [
            {
                "index_patterns": [
                    "test-data*"
                ],
                "dls": "",
                "fls": [],
                "masked_fields": [],
                "allowed_actions": [
                    "write",
                    "indices:data/read/search",
                    "indices:data/read/search*",
                    "indices:data/read/search/template"
                ]
            }
        ],
        "tenant_permissions": [],
        "static": false
    }
}

The action group called test under cluster_permissions contains:

indices:data/read/search
indices:data/read/search*
indices:data/read/search/template

About multi-tenancy, I'm don't have custom tenants, here's the result of the query on _opendistro/_security/api/tenants:
{
    "global_tenant": {
        "reserved": true,
        "hidden": false,
        "description": "Global tenant",
        "static": true
    }
}

Thanks for your help.


Update vep struct to csq to allow omitting sift and polyphen

This allows sift and polyphen to be omitted from the vep csq annotation

Sorry about that!


Fix typo in connected_components example

Hi guys,
Congratulations on the public release!
Here's a very quick PR with a typo fix in connected_components. I have another PR to update the docs on the project website; I will cross-link the PRs momentarily.
Best wishes,
Matt
Signed-off-by: Matthew Forshaw<EMAIL_ADDRESS>

Thanks! Our first external contribution :)


Applying Aggregation to run.py

Thanks for the PR -- we're not considering major architectural changes currently, but we appreciate the contribution to open-source software!


fix typo in docs/API.md

Coverage remained the same at 87.282% when pulling 0225ffaf44d29c67213d60695b27474a89d11331 on yunyu950908:fix/typo into 3b8c87c25cf6acf44f0ec619f21705358e4fec7e on dvajs:master.


Remove Hardcoded domain name

https://github.com/sestus/godyndns/blob/2afd8da227c05fa4b5e1ed5f9ee8cf2313ac57f6/godyndns.go#L59
Cool project.  I would suggest removing the hard coded domain name, and instead pulling it off of the hostname.

Hey, thanks for raising this!
Indeed, I intented to remove my own domain from the code but it turns out I never did it. Will remove that and release a new version.

Hey, thanks for raising this!
Indeed, I intented to remove my own domain from the code but it turns out I never did it. Will remove that and release a new version.

Hey @jcekstrom , I have now removed the hardcoded url as discussed. Also added some extra checks on the subdomain given by the user. Feel free to have a look at the linked PR should you wish to do so.

Looks good.


Update sbt to 1.4.9

Updates org.scala-sbt:sbt from 1.4.8 to 1.4.9.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.scala-sbt", artifactId = "sbt" } ]

labels: library-update, semver-patch

obsolete.


KARAF-5292 : removed uneeded dependency to dbcp in eclipselink feature

R: @jbonofre

retest this please

(last comment is just to trigger a new build on Jenkins)


[DO NOT MERGE] Update main_test.go

test pull request

/hold
Do not merge, test pull request

/close


rate() incompatibility with Prometheus

We have a grafana dashboard measuring rate on a metric. With Prometheus, it works, but with Google Managed Prometheus we get rate() can only be used with counter series, got GAUGE.
Reading the prometheus docs for the rate() function, it notes "rate should only be used with counters. It is best suited for alerting, and for graphing of slow-moving counters." But in practice, it allows gauges.
Our application is a metric exporter. It is querying another application and reporting the metrics. That backend application increments values like a counter. But, since it can restart, thus resetting the counter, our exporter application uses a prometheus GAUGE to set the prometheus metric to match the values from the backend application.
This functionality works for Prometheus, but not Google Managed Prometheus.

Our backend system (Monarch) is strongly typed, while Prometheus is not. We send in TYPE information on time series as our system requires it. In practice this leads to situations exactly like you discovered, where the loosely-typed Prometheus will cast gauges as counters and run rates, even though you shouldn't be able to do so.
It sounds like you're converting this metric to a gauge unnecessarily. Prometheus counters handle these resets correctly without any user intervention, and you shouldn't need to convert them to gauges (Basically Prometheus treats counter value drops as a reset and rate functions will implicitly add the fresher, lower value to the previous value to create a standard increasing counter time series when you run the query).
Can you try this and make sure it works? The other thing you can do is send it to us without a TYPE, but when you do that we actually store it both as a counter and a gauge, costing you 2x in ingested samples but it'll work.
Closing this as we can't un-type our system to support this.

It makes sense that you are limited since it's probably unreasonable to loosen Monarch's typing in this situation. And I see what you are saying about converting the metric to a gauge unnecessarily. That's true in an abstract sense, but the Golang library makes it difficult for us. As you can see in https://pkg.go.dev/github.com/prometheus/client_golang/prometheus#Counter, the interface does not support a Set(). We only get that with a gauge type. So I see your point that from a wire format standpoint, we should just let the metric value reset, tagged as a counter TYPE, and let rate() handle it at query time. But with the current Go library, we cannot do that.
That said, this is probably an edge case and I suspect I can find a way to hack around this.

Ack - thanks for the context. An easy hack would be to remove the TYPE field at the source, but you'd be paying twice in that case (once to ingest metric as a gauge, once to ingest as a counter). May be just fine if the volume is relatively low.


Unit of task results

Hi,
When I run Python and complete this process, I get the task results, but I don't know the units of the parameters, such as some large circuit, its units seem to get larger, resulting in small data，such as total_logic_block_area =1

Best regards

@TOM-once The unit for

time is second
critical path delay is second
area is computed in terms of Minimum Width Transistor Area (MWTA).

For larger circuit, if turn on auto sizing in VPR, the array size will be increased to fit your design. That is why you see area is increased.


Add a CONTRIBUTING.md File for Contribution Guidelines

To streamline the contribution process, I suggest adding a CONTRIBUTING.md file. This would provide clear guidelines for new contributors, including steps for setting up the project locally, coding standards, and how to raise issues or submit pull requests. This document can improve the onboarding process for open-source contributors, making it easier to contribute effectively.
Could you please assign this to me?

Hey @Varsha567,
Thank you for the suggestion! We appreciate your initiative. We're assigning this issue to you—looking forward to seeing your contribution!

Well done, @Varsha567 , on completing the issue! 🎉
Your efforts are truly valued, and the updates you implemented look excellent. Keep up the great work, and continue contributing your talents!

Hii @Anuj3553  I noticed that my contributions are not reflected on the leaderboard, and I would like to kindly request your assistance in rectifying this issue. It’s important for me to have my contributions acknowledged, and I believe they have met the required criteria.
Thank you for your attention to this matter. I appreciate your support, and I look forward to your response.

Hyy @Varsha567, Can you do star mark this repo?

Hyy @Varsha567, Can you do star mark this repo?

Already done!


Getting another error uploading an api.json

Created by: https://gist.github.com/infomofo/566f3ddb05d61acd46c9

We were missing validation of default values when used in operation parameters. This has been fixed on the 222-union-types branch where we had overhauled validation. Will go live once that branch is released

Released


Support regex in feature tag

modified testInfo parser
added new tests for feature tag

run tkg-test

grinders: Grinder/34931, Grinder/34933, Grinder/34934, Grinder/34935


Add kapp getting started

Assistance needed:

Please review (it has previously been reviewed by Paul and a couple Carvel folks and I fixed some edits).
Help making the #'d bullets work
Help making kapp show up in the list
kapp technically is lowercase...should it be upper to match the rest of the headers on the side?

Thanks
@bryanfriedman @BrianMMcClain

Alright, gave it a look, and other than the things you mentioned in your comment, this is great! Wasn't sure exactly what you had in mind for the numbered bullets but take a look at the changes I suggested, because I might be misunderstanding. I also think that if the official capitalization of the name is "kapp", let's keep it how you have it.


bump prometheus nats exporter to v0.14.0

Description
bump prometheus nats exporter to v0.14.0 https://github.com/nats-io/prometheus-nats-exporter/releases/tag/v0.14.0

Reasons
security and performance
Attachments

released with https://github.com/kyma-project/nats-manager/releases/tag/1.1.0


Albert/senior page

Summary
Implement a simple page for listing Seniors Name
Detail

Ordered lexicographically ascending

Pictures / Gif

Checks

[ ] Simon
[ ] Bernard

@albertsutz Can you also make some filter to find the entry
Other comments:

The boxes are too near, maybe adjust the spacing?


partial def for leaflet module

Partial definition for http://leafletjs.com/reference-1.2.0.html

Looking to merge as partial. I covered what we needed and want to share that for people to build on.
Anything you'd like to see in the code as caveat? (btw, feel free to edit in place.)

@turadg Did you see this one?
https://github.com/flowtype/flow-typed/pull/1074

Shoot, nope! I searched Google and for files in master.
Why not merge that one? Seems like it would be valuable even as is. At the least to unify efforts to improve leaflet typing. :)
Incidentally, it looks like https://github.com/DefinitelyTyped/DefinitelyTyped/blob/master/types/leaflet/index.d.ts is very complete for v1.2. I tried https://github.com/joarwilk/flowgen but it has some bugs. The output can be manually fixed, but then there's the dependency on geojson which also is in DefinitelyTyped but not flow-typed.

If you'd like to help with that PR, ask Michael. I'm sure he will agree to give an access to his fork.

Yes, I will


Allow Symfony 7

Symfony 7 will be released soon and it would be great for Port to work with it.

Port doesn't use any deprecated functionality in Symfony 6 and there are no concerning-looking breaks in the 7.0 changelog.
Tests pass when run against Symfony 7.0.0-RC1.

Cool, thanks!


Fix: Spacing of inputs and outputs in transaction section

Description of change

Added left border for all the sections of inputs and outputs.
Fix spacing between different sections.

Resolves https://github.com/iotaledger/explorer/issues/576 https://github.com/iotaledger/explorer/issues/581
Type of change

Bug fix (a non-breaking change which fixes an issue)

Transaction payload now broken on mainnet...

 Transaction payload now broken on mainnet...

Fixed.


Resolve installation into Blender

This issue is to track building OTIO for installation into Blender, and also the creation of wheels that Blender can consume.

From my understanding the reason this did not work for Blender was that its Python development artifacts (the include and lib) dir were not installed in a standard location.

Blender on Windows (I wont talk about Linux since I don't know the situation there) doesn't ship with the Python headers. See https://developer.blender.org/T74470 and https://blender.community/c/rightclickselect/gtfbbc/ . So that is one problem on his own.

@rogernelson I looked more deeply at Pybind11 cmake files and PYTHON_EXECUTABLE and PYBIND11_FINDPYTHON  are not mutually exclusive.
If PYBIND11_FINDPYTHON  it will call pybind11NewTools.cmake (see https://github.com/pybind/pybind11/blob/e08a58111dbea38d667b209f7543864d51a3b185/tools/pybind11Common.cmake#L195-L200), then if PYTHON_EXECUTABLE is not defined, it does some logic for venv, conda and GitHub Actions. But if PYTHON_EXECUTABLE is present, then it uses pre cmake 3.12 FindPythonInterp (see https://github.com/pybind/pybind11/blob/e08a58111dbea38d667b209f7543864d51a3b185/tools/FindPythonLibsNew.cmake#L71,L98).
Anyway, here it doesn't matter much if it's mutually exclusive or not I think. What matter is controlling which python version is used. On that front, I noticed that CMake's own FindPython doesn't understand PYTHON_EXECUTABLE. It only seems to understand Python_EXECUTABLE  while pybind11 uses PYTHON_EXECUTABLE, which leads to confusion (notice the different in cap letters)
So, if we use want to support all cases actually supported by CMake's FindPython (while using CMake 3.20+), we can use find_package(Python REQUIRED COMPONENTS Interpreter Development.Module) if we replace PYTHON_EXECUTABLE by Python_EXECUTABLE in our setup.py file. We won't need to bring back PYBIND11_FINDPYTHON. And this will rely on the fact that if pybind11 sees that Python_FOUND is set, then it won't try to find python and will simply reuse the existing python targets (https://github.com/pybind/pybind11/blob/e08a58111dbea38d667b209f7543864d51a3b185/tools/pybind11Common.cmake#L196, https://pybind11.readthedocs.io/en/stable/compiling.html#findpython-mode).
 Cases to be tested:

manylinux docker images. (Linux only)
Executing /path/to/python3.7 setup.py bdist_wheel, make sure it's not in PATH and also make sure a more recent version of python (for example 3.9) is present in PATH. (All platforms)
Executing /path/to/python3.7 -m pip install . --user, make sure it's not in PATH and also make sure a more recent version of python (for example 3.9) is present in PATH. (All platforms)
Executing /path/to/python3.7 -m pip wheel ., make sure it's not in PATH and also make sure a more recent version of python (for example 3.9) is present in PATH. (All platforms)
Execute python setup.py bdist_wheel from inside a virtualenv. (all platforms)
Execute cmake while making sure Python is in PATH (it's great when we can just execute cmake and things just work). See how it reacts if a newer version of python is present in PATH before or after the Python version we want.
Try with Blender on Windows and Linux (just to confirm that my previous tests still work)
Test with RV's Python (I don't have access to RV right now, so once I've tested all cases, I could push my changes in my fork and @Roger Nelson you could test it?)

Note that I persist to try to use find_package because if it works, then we can basically sit on the shoulders of a CMake to make sure we support future cases and fixes. It also handles quite a lot more cases than what Pybind11 can handle. Technically we could just use PYBIND11_FINDPYTHON which basically calls find_package(Python REQUIRED COMPONENTS Interpreter Development) but the problem is that they request the Development component, which implies that they require the Python library (.so, dll, etc) to be present, which is not present case for the manylinux docker container. So we have to call find_package(Python REQUIRED COMPONENTS Interpreter Development.Module (Development.Module instead of Development).
I hope what I just wrote makes sense. Let me know if it's not clear or have any questions or concerns.

Regarding RV's sysconfig wrong behavior @rogernelson , that should get fixed in RV :wink: . It always annoyed me. sysconfig should always return paths that exist and are valid. When sysconfig returns wrong values, a bunch a tools in the Python ecosystem just don't work.

I don't fully understand the context here. But for the large majority of Blender users, compiling Python modules from source is not something they are expected to be doing. The bundled Python version is not intended to be used for that, the recommendation is to use a standard Python install.
For add-ons it's most user-friendly to bundle binaries. But it would be great if OpenTimelineIO could provide wheels for Python 3.7 and 3.9 too. I think it's only 3.7 now, but Blender 2.93 and the next VFS reference platform use 3.9. As far as I know, building the wheels does not require any changes to make them compatible with Blender.

The context is that we've been trying to build otio against Blender's bundled Python, with setup.py placing the build artifacts into the blender installation in the correct place.
The meta-context is that we want to be able to use otio within Blender.
The meta-meta-context is that the purpose of this thread is discover the right way to do this, make any changes necessary to our build, and then document how any developer can use otio in blender to create their own tools or add ons. Our initial understanding was that we need to build against Blender itself, but that's not the real goal of course.
Ultimately, I think Blender's timeline should natively deal with otio, but it's up to us to lay the groundwork for that to be possible. That part of the problem is out of scope for this thread.

But for the large majority of Blender users, compiling Python modules from source is not something they are expected to be doing.

To add to that, a large majority of the Python community is not expected to compile Python extensions (an extension being anything that is not pure python) from source.
Also this issue is for Blender, but I'm kind of "hijacking" it to make sure we support a bunch of other cases correctly as can be seen from the matrix of scenarios to be tested from my proposed change.

The meta-meta-context is that the purpose of this thread is discover the right way to do this, make any changes necessary to our build, and then document how any developer can use otio in blender to create their own tools or add ons.

And bonus if we are able to support other "exotic" setups/environments I think without any additional effort from the OTIO side.

@rogernelson You can try to build using https://github.com/JeanChristopheMorinPerso/OpenTimelineIO/tree/python_build_findpython. Let me know how it goes!

@JeanChristopheMorinPerso
I tried your branch, and it seems to do the right thing for me.
RV Python 2.7
-- Found Python: /Users/nelsonr/git/shotgun-rv/rv-cxx14-debug-qt5-python2.7x3.7/bin/py-interp (found version "2.7.17") found components: Interpreter Development.Module
-- Python_INCLUDE_DIRS: /Users/nelsonr/git/shotgun-rv/rv-cxx14-debug-qt5-python2.7x3.7/python2/include/python2.7
-- Python_VERSION: 2.7.17

I also tried with RV and Python 3.7.6
-- Found Python: /Users/nelsonr/git/shotgun-rv/rv-cxx14-debug-qt5-python2.7x3.7/bin/py-interp (found version "3.7.6") found components: Interpreter Development.Module
-- Python_INCLUDE_DIRS: /Users/nelsonr/git/shotgun-rv/rv-cxx14-debug-qt5-python2.7x3.7/python3/include/python3.7m
-- Python_VERSION: 3.7.6

If this same setup works for Blender, then this is great.

Regarding RV's sysconfig wrong behavior @rogernelson , that should get fixed in RV 😉 . It always annoyed me. sysconfig should always return paths that exist and are valid. When sysconfig returns wrong values, a bunch a tools in the Python ecosystem just don't work.

I work primarily with C++, so I'm not an expert on all this Python stuff, but I agree the RV values are wrong.  Since I'm working on integrating OTIO into RV's Python, I'll see if I can tackle this too while I'm in there.

From my understanding the reason this did not work for Blender was that its Python development artifacts (the include and lib) dir were not installed in a standard location.

Blender on Windows (I wont talk about Linux since I don't know the situation there) doesn't ship with the Python headers. See https://developer.blender.org/T74470 and https://blender.community/c/rightclickselect/gtfbbc/ . So that is one problem on his own.

If adding the python headers to Blender, you should say it out loud here: https://devtalk.blender.org/t/opentimelineio-devs-needs-bf-devs-to-solve-python-related-installation-problems/19192/14


css style property receiving 'NaN' in enzyme environment ( when using virtual reference)

im trying to test a click outside event that's suppose to change the state of my Hint component, i'm using react popper and a virtual reference that calculate style from a given reference

the problem is that when i want to test an event
for example:

i got this:

anyone have an idea on how to solve this ...

Unfortunately Popper.js doesn't work with JSdom, you need to mock Popper.js using the instructions provided in the README


Isn't this just RxSwift?

Isn't this just RxSwift?

No, but is solves same problem in similar manner.


Provisioning: Add headless mode for Synful provisioning script

Currently, the Synful Provisionin Script runs as an interactive script.
It should have a headless mode that takes command line parameters for configuration as well.

This has been completed.
# Usage
# ./synful [version|stable|dev-unstable] [mysql-password] [y/n]
# The first parameter is the version to install,
# The second parameter is the MySql password to use,
# The third parameter is either y/n for allowing Sql over network.


from functools import partial

partial() is called on lines 1039 and 1064.
This pull request addresses the problems described at issue #1547

Thanks!


Update image ghcr.io/zoriya/kyoo_front to 40255cf

This PR contains the following updates:

Package
Update
Change

ghcr.io/zoriya/kyoo_front
digest
ce3ffc5 -> 40255cf

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

Edited/Blocked Notification
Renovate will not automatically rebase this PR, because it does not recognize the last commit author and assumes somebody else may have edited the PR.
You can manually request rebase by checking the rebase/retry box above.
⚠ Warning: custom changes will be lost.


test-watch fails without running tests if lint fails

~/code/private/redux-actions  (feat/create-actions|%=) $ make test-watch
node_modules/.bin/eslint src

/Users/VictorAlvarez/code/private/redux-actions/src/__tests__/createActions-test.js
  36:7  error  Missing semicolon  semi

✖ 1 problem (1 error, 0 warnings)

make: *** [lint] Error 1

Is this the right behavior? I feel like linting in test-watch should lint once and not fail, or lint continuously.

Afaik eslint doesn't have any watching capabilities.

Right. We could combine this with gulp.watch to achieve the continuous linting behavior.
https://github.com/yangmillstheory/redux-standard-actions/blob/master/gulpfile.babel.js#L83

I would like to add this feature without implementing a build system.
How about one of these?

https://www.npmjs.com/package/eslint-watch
https://www.npmjs.com/package/watch

This issue will be closed by https://github.com/acdlite/redux-actions/pull/107.


validate-jwt not validating AAD token

I'm having trouble getting my APIM policy to validate JWT tokens from my AAD OAuth 2.0 login..
I have followed the instructions in the documentation, but I'm always getting a 401 error in return regarding of what I put into the Authorization header. Is the documentation outdated?
My current APIM policy:
<policies>
    <inbound>
        <validate-jwt header-name="Authorization" failed-validation-httpcode="401" failed-validation-error-message="Unauthorized. Access token is missing or invalid.">
            <openid-config url="https://login.microsoftonline.com/<tenant-id>/.well-known/openid-configuration" />
        </validate-jwt>
    </inbound>
    <backend>
        <base />
    </backend>
    <outbound>
        <base />
    </outbound>
    <on-error>
        <base />
    </on-error>
</policies>

I have also tried specifying the audiences, removing the openid-config, with the 2.0 well-known url etc, but nothing validates..
The bearer token is created in a react app using msal-browser (MSAL 2.0), but I have also tested with the token from MS Graph explorer, just to make sure the token is fine...
Does some of you see any errors in my policy config, or am I missing something different here...? Is the documentation outdated?
Thanks in advance!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 248a0753-0fa7-1cc0-d13a-d378ab182e95
Version Independent ID: 7efa247a-7184-6e7f-c1f5-3ff2531df357
Content: Azure API Management access restriction policies
Content Source: articles/api-management/api-management-access-restriction-policies.md
Service: api-management
GitHub Login: @vladvino
Microsoft Alias: apimpm

@MatsAnd Thank you for your feedback! We will review and update as appropriate.

@MatsAnd I tried to follow the same documentation and unable to reproduce the issue. I would request you to validate the token (https://jwt.io/) and confirm the tenant name in the policy once again. If you still encounter the issue then it must be application specific and needs dedicated troubleshooting. Hence I would request you to open a support ticket with Microsoft Support if you have a subscription which allows you to do so. Otherwise , you can send an email with subject line “Attn:Jaya” to AzCommunity[at]Microsoft[dot]com referencing this thread along with the subscription id.

@MatsAnd Since we have not heard back from you we will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly reopen the issue and continue the discussion.


Custom local fonts do not load.

🐛 Bug Report
Custom (Locally stored) fonts do not load. I have seen that there is documentation for importing Google Fonts, but have not yet seen anything that can help with fonts that are stored locally.
I am probably calling the fonts incorrectly, but due to no documentation being present for fonts that are stored locally, and no errors showing up I have no clue why my fonts will not load.
Have you read the Contributing Guidelines on issues?
Yes
To Reproduce

Clone git@github.com:emilnewel/docusaurus-issue.git
Install with yarn on npm.
Run project with npm start.
See that the font does not change to the desired font. (Setimo in my case)

Expected behavior
The big red text on the front page to be in Setimo font.
Actual Behavior
Font is just some basic system font.

Your Environment

Docusaurus version used: 2.0.0-alpha.68
Environment name and version (e.g. Chrome 78.0.3904.108, Node.js 10.17.0): Chrome 86.0.4240.198 and Node v14.5.0
Operating system and version (desktop or mobile): Mac OS 11.0.1 Big Sur.
Fonts are stored in static/ folder
Extended webpack is stored under plugins/my-loaders/ and is also set in docusaurus.config.js

Reproducible Demo
https://github.com/emilnewel/docusaurus-issue

You can load fonts by dropping the .woff into static, and adding this to your custom.css
@font-face {
  font-family: "JetBrains Mono Regular";
  src: url('/static/JetBrainsMono-Regular.woff') format('woff');
}

:root {
  --ifm-font-family-monospace: "JetBrains Mono Regular";
  --ifm-font-family-base: 'Noto Sans JP';
}

@andrewtruong, this does nto seem to work for me.

As you can see in the image, according to chrome the font is "Setimo" but this is not the correct font.
This is my current folder structure :
website
│
└───src
│   │   pages
│   │   ...
│   │
│   └───css
│       │  custom.scss
│
└───static
    │   Setimo_Rg.ttf

Inside of custom.scss I have:
@font-face {
  font-family: "Setimo";
  src: url("/static/Setimo_Rg.ttf") format("ttf");
  font-weight: 400;
  font-style: normal;
}

:root {
  --ifm-font-family-base: "Setimo";
}

Any ideas? 🤔

I figured out that by changing the format to format("truetype") the font would load locally. But when I tried to build and serve the fonts went missing again. :/

Strange, but your font works for me (see https://epic-pike-f3b627.netlify.app/).

Can I take a peek at your code and compare to my own?

Here it is, based on the comments in this issue:
@font-face {
    font-family: Setimo;
    src: url(/fonts/Setimo_Rg.ttf) format("truetype");
    font-weight: 400;
    font-style: normal;
    font-display: swap
}

@font-face {
    font-family: Setimo;
    src: url(/fonts/Setimo_Bd.ttf) format("truetype");
    font-weight: 700;
    font-style: normal;
    font-display: swap
}

@font-face {
    font-family: Setimo;
    src: url(/fonts/Setimo_It.ttf) format("truetype");
    font-weight: 400;
    font-style: normal;
    font-display: swap
}

@font-face {
    font-family: Setimo;
    src: url(/fonts/Setimo_BdIt.ttf) format("truetype");
    font-weight: 700;
    font-style: normal;
    font-display: swap
}

@font-face {
    font-family: Setimo;
    src: url(/fonts/Setimo_Lt.ttf) format("truetype");
    font-weight: 400;
    font-style: normal;
    font-display: swap
}

@font-face {
    font-family: Setimo;
    src: url(/fonts/Setimo_LtIt.ttf) format("truetype");
    font-weight: 400;
    font-style: normal;
    font-display: swap
}

:root {
    --ifm-font-family-base: "Setimo";
}

Welp, this is super weird.
I removed my css and pasted yours. Didn't work.
I removed node_modules, build/ and package-lock.json and did a fresh npm install. Didn't work.
When I wake up tomorrow I'm gonna try scraping the whole project and adding it back together piece by piece. Maybe I'll  find something that is breaking things.

Try running yarn clear (docusaurus clear), it might help. Look at the Network tab, there should be requests that load fonts:

That's exactly my issue, the fonts just will not load when I do npm run build and npm run serve.
I tried doing running yarn clear but it didn't work.

@emilnewel can you share an updated repro/branch with the advice above applied?
Can you host the build somewhere so that we can inspect it easily?

No need, I got it to work by adding font-family:  'Setimo' to the top of :root in custom.scss. For some reason just having --ifm-font-family-base: 'Setimo' was not enough.

@emilnewel can you share an updated repro/branch with the advice above applied?
Can you also host the build somewhere so that we can inspect it easily?

The problem I'm having is that the console can't see the request, which is strange, but the font seems to be working


Update dataset.py to be compatible with TF 2.x

The functions in the dataset.py are outdated and deprecated in Tensorflow 2.x. This commit updates the code to be compatible with the latest TF versions by replacing the deprecated functions with their TensorFlow 2.x equivalents.
Thanks.

looks good to me


docs: Move "API Documentation" sections in package READMEs from the autogenerated "footer" section to the "header"

The general idea here is that the "header" describes mechanical aspects of installing / using the library. Where the "footer" enumerates relevant Fluid Framework policies. The API docs belong in the former category.

🔗 Found some broken links! 💔
Run a link check locally to find them. See https://github.com/microsoft/FluidFramework/wiki/Checking-for-broken-links-in-the-documentation for more information.

I've noticed this error comment on a few PRs. What's the issue here? Is this a built-in GitHub check that fails because of how we're using Hugo or something? Is there a way to fix it or else disable this error if it doesn't matter?

🔗 Found some broken links! 💔
Run a link check locally to find them. See https://github.com/microsoft/FluidFramework/wiki/Checking-for-broken-links-in-the-documentation for more information.

I've noticed this error comment on a few PRs. What's the issue here? Is this a built-in GitHub check that fails because of how we're using Hugo or something? Is there a way to fix it or else disable this error if it doesn't matter?

Our redirect logic for API docs doesn't play nicely with the link checker. We have some work filed to fix this. I believe scheduled this sprint on Team Sasha.


How to enable pretty print JSON for swagger 0.9.5 , springmvc

Hi,
I have to enable pretty-print JSON , I am using CustomJavaPluginConfig for generating the API-DOCS.
Please suggest how can i do that.
Thanks
Regards

You'd need to use jacksons object mapper for that. See this stackoverflow answer for an example

I don't think the original poster got this working by using that stackoverflow answer. I have tried the same by setting objectMapper.enable(SerializationFeature.INDENT_OUTPUT); with no effect. One solution is to extend ObjectMapper with a CustomObjectMapper and override registerModule(). Then you can check for the "SwaggerJacksonModule" and then creating this module yourself by copying the code in SwaggerJacksonProvider. This is super-dirty, but it works :)