"Custom role definition artifact" - documentation unclear

I have a question about the page "Custom role definition artifact in Azure Managed Applications".
On this page it says: "It's used to determine what permissions the managed application needs to perform its functions."
What does this mean? Does this mean this file will determine which permissions the managed identity of the managed application will have on the managed resource group?
I've tried setting this file (customRoleDefinition.json) to:
{
  "contentVersion": "<IP_ADDRESS>",
  "roles": [
      {
          "properties": {
              "roleName": "Owner"
          }
      }
  ]
}

But it seems nothing changed. The managed identity still has the 'Contributor' role on the managed resource group.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: da38a6af-2be2-e520-a68f-f46d4f1ffd05
Version Independent ID: c441a69b-e874-6735-f57f-1dd7dfe53be2
Content: Overview of custom role definitions - Azure Managed Applications
Content Source: articles/azure-resource-manager/managed-applications/concepts-custom-role-definition.md
Service: managed-applications
GitHub Login: @jjbfour
Microsoft Alias: jobreen

Hi, we are investigating your issue and will update you shortly.
Best,
James

@jjbfour  Can you please provide your inputs on this issue?

#reassign:davidsmatlak

@Freia3 Thanks for your feedback about the documentation. Unfortunately, your issue wasn't reviewed in a timely manner. Your issue remains open and will be reviewed, and the timeline for resolution may vary based on resourcing. There was a recent change to the content's author and I'm working through the open GitHub issues and updating documentation. I'll post a comment when I'm working on your issue.

@davidsmatlak, is there any update on this? I am wondering about the same.

@davidsmatlak: I have the same question as @Freia3
Any update on this issue?

@johannbol I don't have an update at this time but will investigate. I'll post an update when more information is available.

@johannbol @Freia3 Can you provide more details about how you're trying to use the customRoleDefinition.json file in your managed application and the expected result? To clarify, your version of customRoleDefinition.json looks like the sample in this issue?
GitHub issues are public, so don't post any confidential information.

This article is being removed because this is a deprecated feature. I'm closing this issue, but if you have feedback for the question, post a comment.

#please-close


[GLUTEN-7116] [CH] support outer explode

What changes were proposed in this pull request?
Support lateral view outer explode
(Fixes: #7116)
How was this patch tested?
This patch was tested by unit tests

wait review velox code

@marin-ma Can you help to review it?


Compatible gemspec

The old gemspec was failing here because of FileList, I mean it's not on core Ruby.
I replaced with default Dir.glob and array operations, so it's more compatible.

Oops, I missed that. Thank you.


Grow private input 'leftwards'

https://github.com/scroll-tech/ceno/blob/matthias/ceno_host/ceno_host/src/lib.rs test_rkyv_padding has some example code to show that we can left-pad rkyv serialised data without any problems, but right padding doesn't work, at least not naively.
Thus we should fix the last byte of the input to always be in the same VM address, and make the start move further and further left, if we add more data.
See also https://github.com/scroll-tech/ceno/discussions/607 and https://github.com/scroll-tech/ceno/pull/622

Have a look at the linked code for an example.

How does one use rkyv with the usual rightwards growth, like Vec or "write" APIs?

That would be interesting to look up!  But luckily it's irrelevant here.

And what's the difference between left and right?

Have a look at the linked code.  Basically in rkyv: chain(random garbage, your data) is deserialisable, but chain(your data, random garbage) ain't.  You can replace 'random garbage' with 'memory addresses that throw an error when you try to access them'.

I understand the difference between headers and footers in a streameable format, but I don't see how that applies here.

Luckily, we don't need to understand anything here.

To make your life easier, let me copy and paste the relevant link from the docs in the code that I mentioned: https://rkyv.org/format.html

As long as your buffer ends right at the end of your root object

Ok sure that just means something else does the framing, i.e. keep track of the entry point. Just store the length in the first word then. Maybe the library actually has this feature already. That is when the data has to have a dynamic layout in the first place.

As long as your buffer ends right at the end of your root object

Ok sure that just means something else does the framing (e.g. Rust slice length). Just store the length or entry point in the first word then. Maybe the library actually has this feature already. That is when the data has to have a dynamic layout in the first place.

No, that's not how it works.  Why do you think the start is more special than the end?
The library does not store the length or entry point at the start of the slice. What makes you think so?
(The experiments I linked you to wouldn't work, if it did.)

To spell it out:
Rust slices have all the information necessary to determine eg their end, start, midpoint, length etc.
Rkyv's derserialisation expects to find a specific 'root' at a fixed offset from the end of the slice you pass it.
It then re-assembles the whole dag of your data structure from that root, by following pointers.
If you have some rkyv serialised data in a slice of absolute memory addresses of start..end, you can extend it to earlier_start..end and still get exactly the same data out, as long as earlier_start <= start.
Specifically in our case, we can fill the available hint input to always end at end when we prepare the VM's memory. Then we can ask rkyv to deserialise from hint_io_start..end, even if hint_io_start is much smaller than the actual start of the data in memory we prepared.
Does this make sense?

This sounds like you have misinterpreted my last message which was admittedly compact, and assumed I didn’t know what I was talking about. I will try to expand some other time.

Ok, thanks.

@naure Please expand when you have time.

Somebody has explained that solution: https://github.com/scroll-tech/ceno/pull/631#discussion_r1858437482


Find center Line?

Hi there,
is there  an easy way to get the center line from the edges?
so for example the red line here:

Any idea would be great! Thanks a lot

Good question. I think what you can do to achieve this is the following:

Initialize a list of line segments.
For each polygon in the straight skeleton, iterate over its edges and add them to the list with the exception of these 3 edges: the edge that connectes the first node and the last node, and two edges adjacent to it. So an n-gon would contribute n - 3 edges to the list.
Process the list: remove duplicates, merge edges into a graph, etc.

For a more complex polygon the result should look like this:

Let me know if you have any further questions.

I'm probably really late to the party but I'm trying to do the same thing. I ended up doing something very similar but instead of excluding the edges from the polygons I only add the edges if both nodes have a z-value (the last value of the vertex) greater than zero and if their winding number is positive.


Import error when installing from the latest binaries on Ubuntu14, Python3.4

I wanted to updat tensorflow to the newest release and now cannot import it (I am not in a source tree).
Environment info
Operating System:
Ubuntu 14
If installed from binary pip package, provide:

Which pip package you installed.
Latest 3.4 cpu Ubuntu
sudo pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.7.1-cp27-none-linux_x86_64.whl

The output from python -c "import tensorflow; print(tensorflow.version)".

If installed from sources, provide the commit hash:
What have you tried?
 python3 -c "import tensorflow; print(tensorflow.__version__)".
Logs or other output that would be helpful
dima@dima-TP-T410s:~/data/repos$ python3 -c "import tensorflow; print(tensorflow.__version__)"
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/__init__.py", line 35, in <module>
    from tensorflow.core.framework.graph_pb2 import *
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/core/framework/graph_pb2.py", line 6, in <module>
    from google.protobuf import descriptor as _descriptor
ImportError: cannot import name 'descriptor'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/__init__.py", line 23, in <module>
    from tensorflow.python import *
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/__init__.py", line 41, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/__init__.py", line 35, in <module>
    from tensorflow.core.framework.graph_pb2 import *
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/core/framework/graph_pb2.py", line 6, in <module>
    from google.protobuf import descriptor as _descriptor
ImportError: cannot import name 'descriptor'

Error importing tensorflow.  Unless you are using bazel,
you should not try to import tensorflow from its source directory;
please exit the tensorflow source tree, and relaunch your python interpreter
from there.

Can you try pip uninstalling tensorflow and protobuf, then reinstalling tensorflow?

this solved the issue

envy@ub1404:~/os_pri/github$
envy@ub1404:~/os_pri/github$ pip show protobuf
Name: protobuf
Version: 2.6.1
Location: /home/envy/.local/lib/python2.7/site-packages
Requires: setuptools
envy@ub1404:~/os_pri/github$ pip show tensorflow
Name: tensorflow
Version: 0.7.1
Location: /home/envy/.local/lib/python2.7/site-packages
Requires: six, protobuf, wheel, numpy
envy@ub1404:~/os_pri/github$
i got this error yet!

Thanks guys that worked. When i installed tensorflow it uninstalled protobuf 2.6.1 with the latest 3.0 version required for tensorflow..works now


Custom preloaded images in the BTRFS partition

Connects to #103
Builds will fail as it need changes in board specific repositories too (rename of resin-supervisor-disk recipe).
@floion @telphan @petrosagg @Page-

Other than the 2 comments above, SGTM

LGTM


[FEAT] Tangle Leaderboard Functionalities

Is your feature request related to a problem? Please describe.
The Tangle Leaderboard UI task is ready here. Need to add functionalities to it, including the clock and table data.
Also the links for the buttons and FAQ items also need to be update.
Describe the solution you'd like
The following categories should be done:

[ ] Button links
[ ] FAQ
[ ] Clock

Post-launch
An easy tutorial for manually adding data.

[ ] Ranking Table

Additional context
Refer back to the UI issue here

CC @thomivy to discuss the ranking table. What we plan to be manual, what we plan to automate? Should we use a Notion table and keep it all manual, etc.

To clarify we need automated infra for the following, which can arrive 2-3 weeks after testnet launch at the latest, earlier the better. We have a manual submission process setup.
https://www.notion.so/hicommonwealth/Public-Facing-Terms-and-Promo-Doc-68b4b70d37ab4e4c87100e74c31a9a22?pvs=4


 Two of the widely adopted are x86(32 bits)/x86_64(64 bits) from Intel and ARM. Seldom people are still using x86(32 bit) system so DJL only supports x86_64. ARM based CPU which dominates the edge device world also have a family. We will only discuss two well-known ISA in the rest of the doc. Armv7 is 32 bit only and still exists on lots of edge devices. Armv8 on the other hand, starts to support 64 bit and remains backward compatible with 32 bit system. To distinguish between 32 bit and 64 bit, people also call armv8-64 bit aarch64 or arm64.
Introduction
ARM based CPU chip has been dominating dominating the edge devices like Android, Raspberry and Nvidia Jetson. This year Apple also launch arm-based new Macbook pro & Mac Mini. C++ unlike Java “write once, run everywhere” attribute is required to recompile on the target platform. We are going to describe what DJL is missing for three engines and proposed several approaches to fill up the gap in this doc. There are many arm-based edge device. We’ll start with two widely-used platforms: Raspberry Pi and Nvidia Jetson with three major engines MXNet, PyTorch and TensorFlow.
We can then further look into the arm architecture of two platforms we focus on. Let’s look at the Raspberry Pi first. There are Raspberry Pi 3, 4 which use armv8-64 bit and older Pi 2 which is embedded with armv7-32 bit. Both of them have corresponding official docker image support. By adding armv7 and armv8 to our native package, we should be able to cover majority of Raspberry Pi and devices without GPU. Notably, ARM based GPU MALI support is out of scope as none of three engines can run against it. The only way that is closest to work is to compile the model with neo-tvm or SageMaker Neo and run against DLR engine. Customers are responsible for building the MALI-enabled DLR before we decide to offer prebuilt DLR with MALI support.
Nvidia Jetson modules is equipped with Ubuntu Aarch64 which is part of armv8. There are Jetson Nano(Maxwell), TX1(Maxwell), TX2(Pascal), Xavier NX(Volta) and AGX Xavier(Volta). All devices in the Jetson family share the same CPU ISA. The only difference is GPU architecture. Once you install Jetpack, you can run with TensorFlow, PyTorch or TensorRT
Google Coral is another popular edge device that comes with edge TPU. Although Coral CPU is Cortex-A53 which implements armv8, TPU as of today, can only be run with TFLite and PyTorch. Most people that uses Coral heavily relies on TPU than its CPU. It could be part of stretch goal and work on it when there is customer request.
Besides edge devices, AWS A1 instance is powered by Arm Neoverse and provides cost-effective performance. The chip implements armv8 which means we could build one armv8 native library that runs on both Raspberry Pi 4 and AWS A1.
In particular, we are only discussing the linux-based OS as this is the only platform customer is asking for. New ARM-based macos and window are out of scope and should be added as per customer request.
Scope of work
The tasks are divided into three pieces

Native Library and Java C++ bridge library(JNI, JavaCpp)
LibUtils and new package publish
Test & Test pipeline

1. Native Library/ (Shared Object)
For native libraries, I propose we add following new classifiers. It aligns with docker image supportand allows us to build the binaries on a powerful machine with docker container rather than leveraging edge device that has limited CPU power and memory.
For CPU

arm32v7
arm64v8

and two CUDA versions support

arm64v8-cu102
arm64v8-cu110

MXNet
MXNet officially support cross-platform compilation for Raspberry Pi and Nvidia Jetson. In fact, MXNet CI has already built and tested those two platforms on the Jenkins that is triggered daily. We should be able to work with MXNet team to download the pre-built MXNet arm libraries directly. In terms of bridge code JNA, I can find the arm, aarch64 jar on their jar list and no extra step like rebuild is needed.
PyTorch
Today, if you want to use PyTorch on Raspberry Pi, you will need to download a pytorch arm pip wheel from unofficial website or build from source. To name a few, pytorch-arm-builds, pytorch-aarch64. It is risky to rely on them because those are not well-tested and hard to find the compile flags which might have significant impact on performance. Such a build requires machine and pipeline. There are several options. It could be as small as DJL team own the pipeline or as big as AWS PyTorch team official joint effort with Facebook. Once we have native libraries ready, we can take advantage of the same pipeline to rebuild the JNI on top of new native libraries.
TensorFlow
TensorFlow officially supports Raspberry Pi pip wheel. Unfortunately, we consume TensorFlow with JavaCpp bindings which is compiled along with TensorFlow. JavaCpp itself supports arm. As a result, we will have similar stories as PyTorch. We can own the pipeline and machine privately or work with sig-jvm to publish arm-based TensorFlow Java package.
2. LibUtils
Now we get the easier part. This section only involves LibUtils.java and build.gradle change. We are able to detect the right classifier by removing the hardcoded x86_64 here. Most of the code are already considered for future support of arm classifier. Additionally, we’ll update the corresponding Github action or CodeBuild for native package build automation.
3. Test/ Test pipeline
In terms of verification, there are two steps. The first step is sanity test. At least ResNet50 & one LSTM model should be tested with new binary on new platform. As a stretch goal, all existing models in model zoo should be tested and marked if error happen. We can leverage docker image to simulate edge device but we might consider to buy the edge device to run a real test. The second steps is to capture performance benchmark data point in comparison with Python baseline. If a significant performance degradation is observed, we should file an issue and work with other team to resolve it. Next is test automation.
We can reuse the build pipeline for testing. As long as the pipeline support docker, we can do cross-platform compilation and then do testing. There are several options. It could be Github action contributors outside of Amazon can easily access. Alternatively, we can build the pipeline with CodeBuild. It have better CloudWatch integration but less accessible for 3rd contributors.
Q & A

Do we still want to support 32 bit armv7?
Yes. Arm claimed 64-bit only devices coming in 2023. All of Nvidia Jetson family are 64 bit. Now leave us only old model of Raspberry Pi. If we only decide to only support armv8, we are dropping Pi 2(armv7,released on Feb 2015) and several Zero (PCB v1.2, PCB v1.3, W). It is safe to treat those the minority. We can support armv8 first and add armv7 later if there is customer request.
Where is the infrastructure to test Raspberry Pi and Nvidia Jetson?
We can leverage QEMU to emulate arm-based cpu through command-line which is able to make the whole testing pipeline automated. The pipeline can be used as part of integration test or canary test as well while not worry about the device setup. Here is a reference of how to run raspberry pi with [QEMU])https://azeria-labs.com/emulate-raspberry-pi-with-qemu/)
How TFLite process input without operator support?
TfLite creates ImageProcessor for image use case in tflite-support package. As claimed by the official TensorFlow, the ultimate goal is to have a complete operator support as tf.image package. The package now is limited to android device and is working in process. The source code can be found: https://github.com/tensorflow/tflite-support/tree/master/tensorflow_lite_support/java. We can see in the future the lib will extend to IOS platform and more CV and NLP use cases.

native mxnet have arm build ready https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fedge/detail/PR-19564/4/pipeline/48/

Does DJL have a timeline for supporting osx-aarch64? I see you are publishing binaries for osx-x86_64 and was curious about the priority of ARM and specifically other OS X targets in the near future. Thanks!

@breandan
We are planning to add PyTorch linux-aarch64 in Q1 2022, osx-aarch64 has not planned yet.
For MXNet, if you have osx-aarch64 binary installed in the system, you should be able to use it by setting environment variable like:
export MXNET_LIBRARY_PATH=YOUR_PATH

@frankfliu Do you have a suggestion for obtaining the osx-aarch64 binary?
I tried compiling the MxNet 1.8 branch from source on my M1 Mac, although there appears to be a number of issues making the build broken there.  The MxNet master branch does build, but it crashes at runtime if I try to use it (which seems reasonable since DJL is using MxNet 1.8 but the master branch is for v2.0)

@ccjernigan
For M1 Mac, I recommend you try pytorch 1.11.0, what you need to do is:

build libtorch from source, manually create libtorch folder in pytorch-native folder, and copy required .so file and hader files into libtorch folder
build pytorh jni:

./gradlew :engines:pytorch:pytorch-native:compileJNI

Thank you—I'll try that and report back how that goes!

@frankfliu I made some progress, but I think I'm missing a step somewhere as I get a build failure with djl.
Here's what I did:

Checked out the PyTorch repo, switched to the 1.11.0 tag, and updated the submodules
conda install astunparse numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing_extensions future six requests dataclasses pkg-config libuv
conda activate
4.export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
USE_DISTRIBUTED=1 CMAKE_OSX_ARCHITECTURES=arm64 MACOSX_DEPLOYMENT_TARGET=11.0 USE_MKLDNN=OFF USE_NNPACK=OFF USE_QNNPACK=OFF BUILD_TEST=OFF python setup.py bdist_wheel copied from https://github.com/pytorch/pytorch/blob/cd6363de3687a33f25848341bb4e594f7c9f99c3/.jenkins/pytorch/macos-build.sh#L36
Copied the outputs of pytorch/build to djl/engines/pytorch/pytorch-native/libtorch
Modified the build.sh to skip download
Ran ./gradlew :engines:pytorch:pytorch-native:compileJNI` but I get the error at the below

Besides the error message, I'll note that my build output copied in step 6 contains a somewhat different set of files compared to what comes from a fresh download of the libtorch 1.11.0 library.  I included the lists of files below as well.
Is there a step that I'm missing or something that I'm doing differently from you?
> Task :engines:pytorch:pytorch-native:compileJNI FAILED
+++ dirname build.sh
++ cd .
++ pwd
+ WORK_DIR=djl/engines/pytorch/pytorch-native
+ export WORK_DIR
+ NUM_PROC=1
++ command -v nproc
+ [[ -n '' ]]
++ command -v sysctl
+ [[ -n /usr/sbin/sysctl ]]
++ sysctl -n hw.ncpu
+ NUM_PROC=20
++ uname
++ tr '[:upper:]' '[:lower:]'
+ PLATFORM=darwin
+ VERSION=1.11.0
+ FLAVOR=cpu
+ AARCH64_CXX11ABI=-cxx11
+ CXX11ABI=-cxx11-abi
+ [[ cxx11 == \p\r\e\c\x\x\1\1 ]]
+ ARCH=aarch64
+ [[ ! -d libtorch ]]
+ [[ 1.11.0 =~ ^1\.10\..*|^1\.9\..* ]]
+ pushd .
~/NoSync/Development/djl/engines/pytorch/pytorch-native ~/NoSync/Development/djl/engines/pytorch/pytorch-native
+ rm -rf build
+ mkdir build
+ cd build
+ mkdir classes
+ javac -sourcepath ../../pytorch-engine/src/main/java/ ../../pytorch-engine/src/main/java/ai/djl/pytorch/jni/PyTorchLibrary.java -h include -d classes
+ cmake -DCMAKE_PREFIX_PATH=libtorch -DPT_OLD_VERSION= ..
-- The C compiler identification is AppleClang <IP_ADDRESS>60021
-- The CXX compiler identification is AppleClang <IP_ADDRESS>60021
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Building djl_torch with the host...
CMake Error at libtorch/Caffe2Config.cmake:14 (include):
  include could not find requested file:

    djl/engines/pytorch/pytorch-native/libtorch/public/utils.cmake
Call Stack (most recent call first):
  libtorch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:13 (find_package)

CMake Error at libtorch/Caffe2Config.cmake:17 (include):
  include could not find requested file:

    djl/engines/pytorch/pytorch-native/libtorch/public/threads.cmake
Call Stack (most recent call first):
  libtorch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:13 (find_package)

CMake Error at libtorch/Caffe2Config.cmake:109 (include):
  include could not find requested file:

    djl/engines/pytorch/pytorch-native/libtorch/public/mkl.cmake
Call Stack (most recent call first):
  libtorch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:13 (find_package)

CMake Error at libtorch/Caffe2Config.cmake:116 (include):
  include could not find requested file:

    djl/engines/pytorch/pytorch-native/libtorch/Caffe2Targets.cmake
Call Stack (most recent call first):
  libtorch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:13 (find_package)

CMake Error at libtorch/TorchConfig.cmake:186 (set_target_properties):
  set_target_properties Can not find target to add properties to: torch
Call Stack (most recent call first):
  CMakeLists.txt:13 (find_package)

-- Found Torch: djl/engines/pytorch/pytorch-native/libtorch/lib/libtorch.dylib
-- Found JNI: /Users/carter/Library/Java/JavaVirtualMachines/azul-17.0.3/Contents/Home/lib/libjawt.dylib
-- Configuring incomplete, errors occurred!
See also "djl/engines/pytorch/pytorch-native/build/CMakeFiles/CMakeOutput.log".
See also "djl/engines/pytorch/pytorch-native/build/CMakeFiles/CMakeError.log".

List of files from building pytorch from source
aten
bdist.macosx-10.9-x86_64
bin
build.ninja
c10
caffe2
Caffe2Config.cmake
Caffe2ConfigVersion.cmake
cmake_install.cmake
cmake_uninstall.cmake
CMakeCache.txt
CMakeFiles
compile_commands.json
confu-deps
CTestTestfile.cmake
empty.cpp
FXdiv
include
indented.txt
install_manifest.txt
lib
lib.macosx-10.9-x86_64-3.9
sleef
temp.macosx-10.9-x86_64-3.9
third_party
TorchConfig.cmake
TorchConfigVersion.cmake

List of files in a fresh download of pytorch binaries
bin
build-hash
build-version
include
lib
share

I have one small update: I had better luck downloading a precompiled version of Pytorch from here https://anaconda.org/pytorch/pytorch/files?version=1.11.0.arm64
With that, I got the Gradle scripts to run in the djl project.  I think I need to make some additional changes before this fully works, but it seems like progress.

@ccjernigan
I made a PR for M1 for macOS: https://github.com/deepjavalibrary/djl/pull/1656

@frankfliu I tried building from your PR commit.  When I run the task publishToMavenLocal, I don't see pytorch-native appearing in my ~/.m2 repository.  Is that expected?
I do see artifacts with version 1.11.0-0.18.0-SNAPSHOT for:

pytorch-model-zoo
pytorch-engine
pytorch-jni

If I unzip the pytorch-jni JAR file in m2, I do see osx-aarch64 bundled inside of it as expected.

@ccjernigan
pytorch-native-cpu package is option, is used for offline distribution. Without pytorch-native-cpu, DJL will download osx-aarch64 native library from DJL's server. I assume macOS is just for developers and always has network access, I didn't create pytorch-native-cpu for macOS M1 yet.

Now we have aarch64 support for PyTorch and OnnxRuntime.


ESLint not working in some components in React

Hi
I've been having issues with ESLint. It seemed to work fine in an older project but i think it's even broken in that now. I've managed to get it working in at least 1 component in a React project, but it doesn't on another in the same directory heirarchy (at least not with the rules i've added - the default create-react-app linting works).
I made the project with create-react-app. In my package.json I have:
"eslintConfig": {
  "extends": [
    "react-app",
    "react-app/jest",
    "shared-config"
  ]
},

and "lint:fix": "eslint src/**/*.js --fix" under scripts.
In my eslintrc.json I have:
"plugins": [
  "react"
],
"settings": {
  "react": {
    "version": "detect"
        }
    },
"parserOptions": {
  "ecmaVersion": 6,
  "sourceType": "module",
    "ecmaFeatures": {
      "jsx": true
        }
    },
"extends": [
  "plugin:react/recommended"
],
"env": {
  "browser": true,
  "node": true,
  "commonjs": true,
  "es6": true
}

and then all my custom rules after.
It is the default formatter in my settings and i have the VSCode plugin installed. Rules in my settings.json are:
"eslint.alwaysShowStatus": true,
"eslint.format.enable": true,
"eslint.runtime": "",
"editor.defaultFormatter": "dbaeumer.vscode-eslint",
"eslint.codeAction.showDocumentation": {
  "enable": true
},
"eslint.packageManager": "yarn",
"eslint.workingDirectories": [
        {"pattern": "./*/"}
   ],

I added this too as it fixed issues for someone else but it didn't work for me:
"eslint.validate": [
  "vue",
  "html",
  "javascript",
  "typescript",
  "javascriptreact",
  "typescriptreact"
]
```,

I went through every rule to get it how i like it because i use it as my code formatter too as i don't like opinionated ones like prettier.

Every other code formatter i've installed since i've had these problems (as far as i can remember) also doesn't work. The component that it doesn't seem to be working in is showing 1 error which i think is coming from the original settings from create-react-app.

Can anyone see any obvious issues? I'd be incredibly grateful if anyone can help as this has been driving me crazy for a while!

Thanks

@oguru does the validation happen correctly when you use eslint in the terminal. If this is the case can you please provide me with a GitHub repository I can clone that demos what you are experiencing.

@dbaeumer I'd not used the terminal for ESLint before but it was working so after some further investigation I found that I was extending things that didn't exist which seemed to have been causing the issue, so it's all good now. Sorry for messing you around!


Carpenters freezes when I try to save a project without files attached to the items.

When trying to save a "standard project" with "dummy items" in it, I added the number of items and saved the project without attaching any files.  After clicking save, the wheel  in the bottom right corner spins and doesn't quit spinning. I try to exit the program by clicking the red X box at the top right of the window, but it won't let me.  However, if I force quit the application through the task manager and re-open the project file, everything seems to have been saved.

I have been able to reproduce this issue, too.


Remote help doesn't default to Codespaces option when in a codespace

You just need VS Code Insiders and the latest Codespaces release to verify this issue. Then open a codespace in VS Code desktop.


Unable to write new partitions with type timestamp on tables created with delta-rs 0.10.0

Environment
Delta-rs version:
Binding: 0.18.0
Environment:

OS: Ubuntu 22.04 LTS

Bug
What happened:
I have a table written with delta-rs 0.10.0. The schema is:
{
   "type":"struct",
   "fields":[
      {
         "name":"Date",
         "type":"date",
         "nullable":false,
         "metadata":{}
      },
      {
         "name":"Timestamp",
         "type":"timestamp",
         "nullable":false,
         "metadata":{}
      }
   ]
}

I'm triyng to write a new partition on that table with the following schema:
pa.schema(
    [
        ("Day", pa.date32(), False),
        ("Timestamp", pa.timestamp("us"), False),
    ]
)

But i get: DeltaError: Generic DeltaTable error: Writer features must be specified for writerversion >= 7, please specify: TimestampWithoutTimezone.
With deltalake 0.16.2 worked fine. Now i dumped to 0.18.0 and i get this error with tables created with an old delta-rs client.
If i the table is created with newer delta-rs client, this doesn't happen.
How to reproduce it:

Create a table with deltalake==0.10.0

import deltalake as dl
import pyarrow as pa

dl.__version__   # 0.10.0

ta = pa.Table.from_pydict(
    {
        "Date": ["2023-01-01", "2023-01-02"],
        "Timestamp": ["2023-01-01T14:37:35.386235", "2023-01-01T14:37:35.386235", "2023-01-01T14:37:35.386235"]
    }
)

ta = ta.cast(
    pa.schema(
        [
            ("Date", pa.date32(), False),
            ("Timestamp", pa.timestamp("us"), False),
        ]
    )
)
dl.write_deltalake(
    table_or_uri="tmp/table",
    mode="overwrite",
    data=ta,
)

Write a new partition with deltalake==0.18.0

import deltalake as dl
import pyarrow as pa

dl.__version__   # 0.18.0

ta = pa.Table.from_pydict(
    {
        "Date": ["2024-06-28"],
        "Timestamp": ["2024-06-28T14:37:35.386235"]
    }
)

ta = ta.cast(
    pa.schema(
        [
            ("Date", pa.date32(), False),
            ("Timestamp", pa.timestamp("us"), False),
        ]
    )
)
dl.write_deltalake(
    table_or_uri="tmp/table",
    mode="overwrite",
    data=ta,
    pertition_filters=[("Date", "=", "2024-06-28"]
)

More details:

Debugging the code (at least from python) i noticed that the table created with delta 0.10 has "timestamp" as a primary type, while new tables now have "timestamp_ntz"
If i add the timezone (for example UTC), even if the table is created with delta 0.10, the write is successful

I am additionally have the same issue when writing new partitions with type timestamp in 0.18.1, when time stamps are structured as for example: '2015-10-30T06:40:15.000Z', year-month-dayThour-min-month.000Z

We fixed a longstanding bug where timestamps where incorrect, the pyarrow engine however incorrectly writes UTC timestamps as Z

Is there a way to migrate the schema from "timestamp" to "timestamp_ntz" without recreating the table? (and load all the historical data)
I tried with schema_mode: overwrite but i get the same error. It seems that delta is unable to write the new "timestamp_ntz" type over the legacy "timestamp".

@emanueledomingo easiest is to recreate the table at the moment

Closing this, since this change was backwards incompatible to fix a long standing issue


Add ts-ignore to fix build

Description
Add ts-ignore to stop failing build caused by #2061

Note that build still fails due to @ts-ignore not working until with Revert the reverted ts upgrade


[App] Line colours have identical default values for different lines

Describe the bug
Many times, the line plot gives almost identical default colors to different plots. This makes it very hard to read the plots without manually readjusting the colors. It would be great to easily read the line plots, without having to manually choose different colors.

Proposed solution
Any of the two following solutions would be satisfactory:
(Preferred) Wandb automatically chooses different colors when the line plot is created
(Satisfactory) A "refresh with different colors" button inside the plot, picks different colors for the line plots, if the default ones are not satisfactory for the user.

Hey Andrei, thank you for your suggestion. I'll file a ticket so that the team can consider adding this feature.

Another kind of related issue is that when hovering over a label, the corresponding line is highlighted. However, since the highlighting is done by making that line thicker, it can blend in with other lines of similar colours. This particular problem could be solved by giving the currently highlighted line a black or white outline so that it could be more easily distinguished from the rest.


Create a Portal Publication model

As with Portal Study and Portal Dataset model, though this is more similar to Portal Study, which existed before its formalization here, while Portal Dataset was completely new.

New publication model should take into account issue #479

PR:


installed-react-vite

Pull Request for react-ionic-paystack
Installing React Vite First
Description:

What changes did imake?

I have started this process by installing Vite and React for faster response
I have installed Vite using the following command in this pull request.
npm create vite
select react
cd ionic-react-paystack
npm install
npm run dev

created react template with vite


Name collision: wasm.memory() function and wasm.memory array

Declaring #[wasm_bindgen] memory() function inside Rust code breaks the whole module. Maybe there should be less generic name for the wasm.memory array or at least an error on memory() function declaration.

Thanks for the report, and I definitely agree that this is something we can improve! It may be a somewhat large-ish breaking change to rename the export of memory itself (as it's often directly imported), but having an error or warning for a function called memory seems reasonable to me!


Sort systems in stdout

The script displays 'Impacted packages' for each system.
This ensures that the systems are enumerated in the canonical order.

@mergify queue

@mergify rebase

@mergify dequeue

@mergify rebase

@mergify queue


Line above and below photo ready to type in it when adding photo

Version
suneditor - 2.42.0
suneditor-react - 3.4.0
Additional context
When adding an image into the editor it has these buttons on top and bottom available once you click on the image to add a line above or below, and the button is hidden if there is a line already.

Is it possible (recommended?) to trigger this behavior of adding a line above and below (if there's none yet) right away when the image is being added? If so, how can be implemented?
Note: I'm using handleImageUploadBefore in order to save image in DB and get corresponding url instead of having base64 in editor's content

editorInstance.onImageUpload = function (targetElement, index, state, info, uploadedFileLength, core) {
    if (!targetElement) return;

    const component = core.getFileComponent(targetElement).component;
    const p = core.util.createElement('p');
    p.innerHTML = '
';

    if (!component.nextElementSibling) {
        core.context.element.wysiwyg.appendChild(p.cloneNode(true))
    }

    if (!component.previousElementSibling) {
        core.context.element.wysiwyg.insertBefore(p.cloneNode(true), component);
    }
}

It works! Thanks!

Hi @JiHong88
could you please let me know why the same code doesn't work for onVideoUpload?
  editorInstance.onVideoUpload = function (targetElement, index, state, info, remainingFilesCount, core) {
      if (!targetElement) return;

      const component = core.getFileComponent(targetElement).component;
      const p = core.util.createElement('p');
      p.innerHTML = '
';

      if (!component.nextElementSibling) {
          core.context.element.wysiwyg.appendChild(p.cloneNode(true))
      }

      if (!component.previousElementSibling) {
          core.context.element.wysiwyg.insertBefore(p.cloneNode(true), component);
      }
  }


Implement camunda:FormData#businessKey for Start Events

When at least one camunda:FormField with a camunda:id is configured a camunda:businessKey can be set.

Updating camunda:FormData is handled in https://github.com/camunda/camunda-bpmn-js/pull/47.
Closes #110

Is this ready for review?

Is this ready for review?

Yes! 👍🏻

UX feedback:

I'm not sure why the business key deserves its own section. I would have expected it to be part of the form section, below the form key. Is it technically easier to control its visibility in its own section or is there another reason for the separation?

I would like to change the order of the sections. The basic rule for the order is: from global to specific; or a logical order (b depends on a). In that sense I would prefer this order:

Form
Form fields
Form business key (in case we keep this section)

As listed in the example under point 2, I would also like to see the togetherness of these sections reflected in the naming. In the naming that is shown in Philipp's screenshot, it is hard to understand that these properties belong together. And the confusing order strengthens that issue.

Screenshot that showcases the mentioned expectations:

UX feedback:

I'm not sure why the business key deserves its own section. I would have expected it to be part of the form section, below the form key. Is it technically easier to control its visibility in its own section or is there another reason for the separation?

=> The FormBusinessKey is only relevant for generatedFormData. Hence putting it in the Forms section makes it even more semantically confusing IMO. Ideally we would put it in the GeneratedForm section, but that does not work since it is a list section (we would need to go with nested lists then, and disable the businessKey field if no formField was configured, which is also not nice). Therefore IMO this is okay.

I would like to change the order of the sections. The basic rule for the order is: from global to specific; or a logical order (b depends on a). In that sense I would prefer this order:

Form
Form fields
Form business key (in case we keep this section)

=> Agreed

As listed in the example under point 2, I would also like to see the togetherness of these sections reflected in the naming. In the naming that is shown in Philipp's screenshot, it is hard to understand that these properties belong together. And the confusing order strengthens that issue.

=> Agreed

@andreasgeier @MaxTru Reordering the groups makes sense. As Max mentioned, the business key has to go in its own group since it can't be put in the Generated task form.

I've reordered the groups as suggested.


Using HTTPS results in TLS warning

TLS Cert is incorrect.
In Chrome:
Your connection is not private

Attackers might be trying to steal your information from choosealicense.com (for example, passwords, messages, or credit cards). NET::ERR_CERT_COMMON_NAME_INVALID
  Automatically report details of possible security incidents to Google. Privacy policy
Back to safetyHide advanced
This server could not prove that it is choosealicense.com; its security certificate is from www.github.com. This may be caused by a misconfiguration or an attacker intercepting your connection.

Proceed to choosealicense.com (unsafe)

and www is the same

Your connection is not private

Attackers might be trying to steal your information from www.choosealicense.com (for example, passwords, messages, or credit cards). NET::ERR_CERT_COMMON_NAME_INVALID
  Automatically report details of possible security incidents to Google. Privacy policy
Back to safetyHide advanced
This server could not prove that it is www.choosealicense.com; its security certificate is from www.github.com. This may be caused by a misconfiguration or an attacker intercepting your connection.

Proceed to www.choosealicense.com (unsafe)

@client9 thanks for filing this issue, but I don't think it'll be fixed soon. choosealicense.com is hosted on GitHub Pages. This is a known problem with GitHub Pages and GitHub hasn't seemed interested in fixing it. (Also, technically it's somewhat difficult.)
tl;dr: they know, and don't care. :(

@strugee would it be possible for choosealicense.com to use Cloudflare as a proxy between their universal SSL and Github's SSL?
https://blog.keanulee.com/2014/10/11/setting-up-ssl-on-github-pages.html
https://sheharyar.me/blog/free-ssl-for-github-pages-with-custom-domains/

I haven't looked at the links, but I know what you're talking about.
Since GitHub already uses Cloudflare for GitHub Pages I doubt they'll throw another Cloudflare instance in front of it, which would be a hacky solution. Instead I think they'll probably just file an internal ticket about it, which is nice (because then eventually it's fixed for everyone) but also sucks (because then we have to wait).
Maybe. I'm just guessing; I don't really know and I don't speak for GitHub.

Instead I think they'll probably just file an internal ticket about it

This is that ticket. :smile:

This is a known problem with GitHub Pages and GitHub hasn't seemed interested in fixing it.

GitHub Pages (which hosts this site) does not currently support HTTPS. It's one of our most consistent feature requests, but as you noted, it's a non-negligible technical challenge, especially around custom domains.

@benbalter out of curiosity, what's preventing you guys from doing this for non-custom domains? CloudFlare already supports TLS termination, so why can't you just deploy TLS on your origin servers and issue redirects? Obviously custom domains are way harder - but why is *.github.io an issue? Scalability? Mixed content?
Just wondering. Thanks for being responsive :)

what's preventing you guys from doing this for non-custom domains?

Making it a great experience for users with all levels of technical and security knowledge. Mixed content is a big part. There's also a question of where the encryptions is stripped (FWIW, we don't use CloudFlare). And that's just for github.io. When you start thinking about custom domains and cert material, things get complicated fast. We want to do it right.

@benbalter oh, I thought you used CloudFlare as your CDN. Guess you have your own then (nice!). Thanks for the response!


Combine two fields in one

Combine field "Movements" and field "Trainings" to one field (движения и тренировки)

Combined and moved to another table (Activity)

Still the data from LifeStyle needs to be copied, after that i will create a migration for removing fields(Movement (Lifestyle tbl), Training (Lifestyle tbl) and DailyMealfs (PatientHealth tbl))


MobiFlight Connector can start with MSFS2020

Is your feature request related to a problem? Please describe.
MobiFlight shall run together with MSFS2020. It is possible to manullay add the start command to the exe.xml file of MSFS2020. But that is errorprone, and complicated for many users.
Describe the solution you'd like
Let MobiFlight auto-register itself in the xml file for correct auto-start.
A user can enable and disable this feature in the settings.
Describe alternatives you've considered
Do it manually.
Additional context
FSUIPC and other tools leverage the same mechanism.

Additionally close if MSFS2020 closes the application


MIFARE Classic: Writing in sectors other than 4, 5 or 6 appears to literally break the NFC tag's memory

Literally what I said in the title, after writing to those sectors on my MIFARE Classic 1K I can't do anything else with my NFC tags. What's happening and how can I fix that?
EDIT: Authentication straight up fails with error 0x6300

Hi @nSneerfulBike,
You are probably writing application data into so-called sector trailers.
Take a look into MIFARE Classic 1K data sheet. The Memory organization is described in Section 8.6:

... memory is organized in 16 sectors of 4 blocks. One block
contains 16 bytes.

What's important, it's that the last block in each sector is the "sector trailer" (see Section 8.6.3), which contains access conditions for the other 3 data blocks of the sector.
In other words, sector trailers are not for application data. If you write some random data to it, you will change access conditions and authentication keys A and B for the corresponding sector in an unexpected way.
When accessing these blocks with this library, blocks are numbered as follows:

sector 0:

block 0 – manufacturer data (read only)
block 1 – data block
block 2 – data block
block 3 – sector trailer

sector 1:

block 4 – data block
block 5 – data block
block 6 – data block
block 7 – sector trailer

sector 2:

block 8 – data block
block 9 – data block
block 10 – data block
block 11 – sector trailer

and so on

I hope it explains and solves your problem. 🙂 I've closed the issue, but feel free to comment on it or re-open if you have further questions or face any problems.

PS Don't forget to star ⭐ my library if you find it useful. 😃 Thanks.

I get how it works, I solved the problem now after making a lot of attempts and destroying so many sectors. I'd really consider this a bug, since if I have let's say a 64-bytes long Buffer and I want to write it, I'd assume that this module takes care of not destroying the sectors, but it doesn't. It just writes everything and doesn't care about the sector trailers.


Canonical examples from Brochu 2010

Would love to have everything to reproduce fig 7 from http://arxiv.org/pdf/1012.2599v1.pdf (and some of the other figures?)
This would also serve as a way to check the correctness of our implementation (for which I currently have doubts regarding EI, as reported in #17)

I'll try work on that this week. I'll backport the example I did for https://github.com/glouppe/talk-bayesian-optimisation

Relevant for #101

Closing, this was somehow addressed with #101


update

Codecov Report

Merging #2 into master will decrease coverage by 3.87%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master       #2      +/-   ##
==========================================
- Coverage   61.63%   57.75%   -3.88%
==========================================
  Files         282      282
  Lines       52722    52110     -612
==========================================
- Hits        32494    30098    -2396
- Misses      20228    22012    +1784

Impacted Files
Coverage Δ

scripts/imagetransfer.py
18.69% <ø> (ø)
:arrow_up:

scripts/welcome.py
18.59% <ø> (+0.04%)
:arrow_up:

pywikibot/login.py
53.63% <ø> (-1.02%)
:arrow_down:

scripts/noreferences.py
34.07% <ø> (-1.12%)
:arrow_down:

scripts/add_text.py
35.33% <ø> (ø)
:arrow_up:

scripts/delete.py
20.89% <0%> (-0.91%)
:arrow_down:

setup.py
0% <0%> (-84.13%)
:arrow_down:

tests/disambredir_tests.py
35.59% <0%> (-64.41%)
:arrow_down:

tests/interwiki_graph_tests.py
37.5% <0%> (-62.5%)
:arrow_down:

tests/patrolbot_tests.py
36.36% <0%> (-60.61%)
:arrow_down:

... and 124 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2ccc5d3...65115a8. Read the comment docs.


Cannot open .sdt files anymore

This seems to be already fixed in the current development version on main


os: Expand eats $$s in the middle of the string (but not at the end)

What version of Go are you using (go version)?
go version go1.10 linux/amd64
Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/opennota/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/opennota/gocode"
GORACE=""
GOROOT="/home/opennota/go"
GOTMPDIR=""
GOTOOLDIR="/home/opennota/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build224817525=/tmp/go-build -gno-record-gcc-switches"

What did you do?
https://play.golang.org/p/ulxC4AgmVNF
What did you expect to see?
The program outputs abc$|def$
What did you see instead?
The program outputs abc|def$
This is unfortunate, because go generate uses os.Expand to expand environment variables in the command line and if one needs to pass a regexp to the executed command, the regexp gets garbled.

Hello
This seems an side effect os this:
References to undefined variables are replaced by the empty string.

https://golang.org/pkg/os/#ExpandEnv
The example of this command shows what happens when both variables are not defined:
“ lives in .”
https://play.golang.org/p/SgYOD-pASWl
My guess is: $var works for alphanum and ${var} works for all chars. Since we have variables like $? $$ etc you must do ${?} to consider this a variable to expand. So $? Seems a corner case where you have an “empty” variable $ follow “?” - and doing abc${|}edf$ may work

To be consistent, I think the trailing $ needs to be removed too. The first $ is correctly gobbled because | is neither a shell special variable nor an alphanumeric character.  The expected output, IMO, should be abc|def.
Moreover abc$|def${} is expanded to abc|def, whereas abc$|def$ is expanded to abc|def$.
It's a relatively small change. /cc @ianlancetaylor for decision.

@agnivade
Correct me if I'm wrong, but with your proposed change nobody would be able to use a literal $ in the arguments to a go:generate command, like
//go:generate esc -o static.go -ignore "\.map$" build/

I believe that the correct way to fix it is to leave $ alone in case it can't be recognized as a variable, as bash does:
$ echo -e "$$\n$?\n$+\n$-\n$|\n$"
4292
0
$+
himBH
$|
$

My proposal does not change that behavior. It is already the existing behavior. My proposal was to make it consistent. Theoretically, I agree with your proposal, but I am not sure how much of a breaking change is this considered to be.
I will leave it to others to make the call.

I believe that the correct way to fix it is to leave $ alone in case it can't be recognized as a variable, as bash does:

That's sound's incorrect.
The bug is that $ doesn't consider the a zero length string/eof to be a valid follow token, but it works when ${} is at the end of the string. This logic is on the domain of the function during the lexing stage. You're suggesting instead, that the function should look at the range after the evaluation of the mapping function has happened, and treating its output (the empty string) as a special case.
The documentation says that os.Expand replaces the ${var} or $var in the string based on the mapping function. It doesn't matter if the mapping function returns an empty string, your bug happens because of the lexical scanning never even makes it call the mapping function in the first place.

@as
You're speaking in terms of implementation simplicity. I speak in terms of the expected behavior.

@opennota I don't expect this behavior either. I expect all tokens denoting a potential variable name to be excised after one pass of os.Expand
Besides, your desired behavior is trivial to implement: just make your own func (s string) string that returns a literal $ on empty input to it.

For the reference, in both Python 2 and 3 os.path.expandvars doesn't even touch variables with non-alphanumeric names:
>>> import os
>>> os.path.expandvars('$$ $? $+ $- $| $')
'$$ $? $+ $- $| $'

That's the most sane behaviour, IMO.

This issue seems to be about two different things.
The first is that os.Expand ignores a $ at the end of the input string.  That does seem peculiar though it's also true that there is no obvious way to handle it.  It's not obvious to that we should just remove it.  It seems more consistent to me to call the function with the empty string, but that could break existing code.  Perhaps we should just document the behavior and carry on.  Most people who call os.Expand have some control over the input string, and so need not worry about a trailing $.
The second problem in this issue is that go generate uses os.Expand, and that makes it hard to pass a regular expression as an argument in a go generate command.  That is true.  I don't see any way backward compatible way to fix that.  That problem doesn't really have anything to do with the way that os.Expand handles a trailing $, it's a general problem with any use of $.  After all, a regular expression can contain $ at places other than at the end, such as within [].  Actually, I take it back: we could address this with only a minor break in compatibility by expanding $$ to $; currently I believe that $$ always expands to the empty string.

How about allow escape $ as literal $ ?
If it is too bad ( backward compatibility, etc ), we can add an option/flag to escape or add an new explicit method for this (maybe in go 2)

@peczenyj First let me repeat that I think there there are two separate problems here.  Which one are you trying to address?
That said, I don't agree with adding backslash quoting to either case.  It's not necessary, because we can use $$.  It introduces more complexity, since now we need to use backslash quoting for backslash itself.  It naturally leads to people wondering about using backslash quoting for other characters, such as \n.  Is that two characters \ n, or one character n, or one character ASCII LF?

@ianlancetaylor
How about ignoring $ altogether unless it is followed by NAME or {NAME}? I fail to see how the existing code might reasonably depend on os.Expand removing single $s, and it's not that there were no breaking changes in the past (the cgo pointer passing rules introduced in Go 1.6 come to mind).

The cgo pointer passing rules were different, as use of cgo is not covered by the Go 1 compatibility guarantee.
The current os.Expand implementation has special code to support $*, etc., and it doesn't seem right to drop all of that.

The current os.Expand implementation has special code to support $*, etc., and it doesn't seem right to drop all of that.

Why not? It was never officially supported (Expand replaces ${var} or $var, unquote), and if someone uses this (is there anyone who does? Go itself doesn't.), they do it on their own risk.
Alternatively, allow only a limited subset of characters to be treated as a variable, otherwise leave $ alone.

Let's not change existing behavior unless we are solving a clear problem.
We already check for a list of special characters; is isShellSpecialVar in os/expand.go.  But changing os.Expand so that $| is left as $| but $? disappears seems to me to increase confusion, not decrease it.

We should be consistent. It seems that keeping the $ in both cases is ideal rather than eliminating it only at the end.

@spf13 - So just to be clear, you want $ to remain in the output in case the following character neither a shell special variable nor alphanumeric . So, abc$|def$ would result in this abc$|def$.

Yes.

Cool. Will send a CL.


Consonant ordering

Just curious - any special reason for this altered consonant ordering about த, ந? A bit disorienting :smile:
https://github.com/oligoglot/mayal/blob/e9b5f83787dc0e786466836388193a2a1519e35b/mayal.py#L16

Nice to see your comment on this, @rselvaraj. I wanted to discuss this paper with you and Vel after completing our work. The change in order was to group vallinam and mellinam together to show the neatness of pattern as in: https://github.com/oligoglot/mayal/blob/main/out/ettuthokai-consolidated.png @rpdwarf suggested this specific order to progress the place of articulation from the back to the front, I believe.

Comment addressed in the previous response


Select Option: Observable doesn't work when created in ngOnInit()

Description
I set my select's options to use a local observable that has not been initialized when my component is initially constructed, instead I am creating the observable in ngOnInit when the component has been initialized. However, the formly select does not get updated to use the new Observable with it's latest values.
Instead, if I set the select options to use a Subject instead, I can get this to work, but only if I put the .next() call on the subject inside of a setTimeout call (to push the call to the end of the event stack). Trying to set the Observable inside of the setTimeout doesn't work.
I figured this was a life-cycle issue, so I put it into ngAfterViewInit(), but then I get the "view changed after checked" error, and it still only works with my subject.
Do I have a logic issue here? I expected to be able to give a reference to an observable to the select option and then when I create the observable later, the async pipe inside of the select component should pick this up, but it doesn't. I figured using a Subject would be more concise (it is an observable at the end of the day anyways), but at my example shows, unless I use the setTimeout hack or put it later in the component lifecycle, the select component doesn't get the latest value from my subject.
Is there an issue with change detection in the select component?
Minimal Reproduction
I've created several different select fields here showing the issue at different life-cycle stages. It appears to be affecting all UI types (here's bootstrap and material):
https://stackblitz.com/edit/ngx-formly-ui-bootstrap-4akfpl?file=src/app/app.component.ts
https://stackblitz.com/edit/ngx-formly-ui-material-9wkakg?file=src%2Fapp%2Fapp.component.html
Your Environment

Angular version: ~8.2.14 & ^9.0.0
Formly version: ~5.5.11

Additional context
None

Closing, it appears you can work around this by using the hooks and onInit property in templateOptions and then setting the select's options to the observable there. This adds a bit of boilerplate to selects and is a little bit counter intuitive, but I don't see a better way around it since the observable might not be created yet until you "init" / "open" the select...


Hgraph dill patch

This PR is a patch for HPU Graph issue.

Fixes https://github.com/huggingface/tgi-gaudi/issues/130
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[x] Did you read the contributor guideline,
Pull Request section?
[ ] Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
[ ] Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
[ ] Did you write any new necessary tests?

Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@regisss

LGTM!
@kdamaszk WDYT?


Feature Request: Add Conditional (query-based) Upsert feature

Please add upset feature that either updates an existing Thing or insert a new Thing depending on a query.
As a reference, MongoDB this feature https://docs.mongodb.com/manual/reference/method/db.collection.update/
Ideally there'll be option to update multiple Things, and option to balance speed vs consistency.
The Mongo version supports those using the multiple and writeConcern flags.
This is the Slack discussion on the topic
https://weaviate.slack.com/archives/C017EG2SL3H/p1617887739020800

@AnielP didn't you have a similar request?

@mingfang would your expectation be that if the query matches, would the document be replaced, or merged (or both depending on a flag).
e.g.  (in pseudo-code)
update: {
  foo: "bar",
}
Query: where country=='Canada'

with existing object
{ id: 1, country: "Canada", unrelatedProp: "some value" }

would the object be replaced, e.g.
{ id:1,  foo: "bar"}

or would it be merged, i.e.
{ id1:1, country: "Canada", unrelatedProp: "someValue", foo: "bar"}

I like the way ArangoDB handles upserts https://www.arangodb.com/docs/stable/aql/operations-upsert.html
Using your example, it would be great to be able to something like this
insert: {
  country: "Canada",
  count: 1
},
update: {
  foo: "bar",
  count: OLD.count + 1
}
Query: where country=='Canada'

Note the OLD references the existing Thing so that I can make relative updates.

@AnielP didn't you have a similar request?

That is correct.
In my case it was necessary for update scenarios to define the uniqueness of an Object in Weaviate  based on key:value pairs in the source data. This was accomplished by generating UUID3 type uuid's from the relevant key:value pairs.
However, this implies that a UUID3 generated check must be performed for each update.
With the feature requested by @mingfang , I only need to "upsert" by the associated key:value pairs.

👋

👋🤖


Docker Image from Main branch

Have you considered to provide a Docker image that's nightly builded from the Main branch? You could name it opensearch:unstable or something like that.
We are using massively Docker in our company and to get a preview of what's going into the next version and to test the features would be great.

Tagging @bbarani for thoughts on this.

[Triage] The distribution builds are meant to be of a higher quality since much effort needs to be built on top of them, we will be closing out this issue, but if you have a compelling use case please reopen.


Use PennyLane measurement processes instead of observables

Hello from PennyLane! I'm here to request a migration from using the Observable type to using the MeasurementProcess type in this plugin, because Observable.return_type is being deprecated in the upcoming release of PennyLane. As this CI run shows, the PL-Braket plugin will need some updating to be compatible with the latest PennyLane.
Request for this plugin
There are various functions that use/accept observables, but some of them actually need measurement processes. For example, BraketQubitDevice.statistics expects a list of Observables, then checks the return_type of the observables. This is a good indicator that we actually want the measurement processes. The proposed fix would be to call statistics with circuit.measurements instead of circuit.observables, probably updating variable names and type-hints along the way.
On the other hand, sometimes we intend on looking at the Observable and not the MeasurementProcess. From what I see, we only want to use Observables when we're about to call _translate_observable. Assuming you rename the observable argument for translate_result_type to something like measurement, you could add something like observable = measurement.obs (perhaps gracefully handling the case where it's None) before translating any Observables, and that should do it!
Background
In PennyLane, measurement processes would previously be described using the Observable class. However, we've been moving towards using the MeasurementProcess class instead, as this encapsulates more accurate/general information about circuit measurements. As an example, consider the qml.probs() measurement - there is no PennyLane Observable being measured per se, but there is a measurement to be processed. This issue has been masked for some time by being able to sneak the type of measurement process onto an Observable (on the return_type property), but it is now deprecated. Comparing QuantumTape.observables to QuantumTape.measurements best highlights this distinction, so I'll show an example here:
>>> tape = qml.tape.QuantumTape(measurements=[qml.expval(qml.PauliZ(0)), qml.state()])
>>> statemp = tape.observables[1]  # note that this is not an Observable
>>> isinstance(statemp, qml.operation.Observable), isinstance(statemp, qml.measurements.MeasurementProcess)
(False, True)
>>> all(isinstance(mp, qml.measurements.MeasurementProcess) for mp in tape.measurements)
True
>>> [mp.obs for mp in tape.measurements]
[PauliZ(wires=[0]), None]

tape.observables will quietly return something that isn't an Observable, which can lead to unexpected behaviour. On the other hand, tape.measurements will always return MeasurementProcess instances. The standard way to check if a process is measuring an observable is mp.obs is not None.
tl;dr, tape.observables returns a mish-mash of Observable and MeasurementProcess instances, whereas tape.measurements always returns a list of MeasurementProcess instances, with the mp.obs property either being an Observable, or None. So, if you need information about a measurement process, you should use tape.measurements.
Word of warning
The Tensor class actually has a property called obs, so there exists a funny case in which you might accidentally refer to the wrong attribute. For example:
>>> tape = qml.tape.QuantumTape(measurements=[qml.expval(qml.PauliZ(0) @ qml.PauliZ(1))])
>>> tape.measurements[0].obs, tape.observables[0].obs
(PauliZ(wires=[0]) @ PauliZ(wires=[1]), [PauliZ(wires=[0]), PauliZ(wires=[1])])

Hello and thank you for opening this! Braket is looking at making the changes to prepare for this. I shall assign the issue to myself to ensure this is followed up with.

Merged in https://github.com/amazon-braket/amazon-braket-pennylane-plugin-python/pull/222 to unblock this issue.

I shall resolve this issue once the braket-stable-latest tests start to pass. The rest of the builds are passing within the test matrix.
Currently, the Braket stable test is using 1.23.0. The changes were introduced in 1.24.0. The expectation is that once this version is being used for the test, we shall see it begin to pass.

All checks are passing.


Two team member

Design work for two team member:
Test URLs:

Before: https://main--investsec-revego--hlxsites.hlx.page/
After: https://two-team-member--investsec-revego--hlxsites.hlx.page/

Before we can merge this we will need a higher resolution picture of Ziyaad Sarang.


Cleanup configuration of pilot-agent

This tracks an effort to clean up the configuration of pilot agent. The root issue here is that we need to construct the meshconfig/proxy config in pilot-agent. To do this, we pass a variety of flags and environment variables to the agent, and reconstruct the config from these sources. This makes our injection templates and gateway deployments quite complicated and inconsistent.
The goal here is to make this fairly minimal - at best, we could have the deployment simply mount mesh config and run with no arguments and everything would work.
Action plan:

[x] Add mesh config as an env var in the injection template, which is read by pilot agent: #20920. This allows us to remove extra fields like --drainDuration="{{ formatDuration .ProxyConfig.DrainDuration }}", which is done in https://github.com/istio/istio/pull/21162.
[ ] Read labels and annotations from downward api, to remove helm hacks required to get these to agent: https://github.com/istio/istio/pull/21184
[ ] Allow annotations to override mesh config fields, to remove fields like --discoveryAddress={{ annotation .ObjectMeta sidecar.istio.io/discoveryAddress .ProxyConfig.DiscoveryAddress }} https://github.com/istio/istio/pull/21206
[ ] Make sure all of these are supported in Gateway as well. This will likely require the gateway to mount mesh config, since we cannot inject it as an environment variable.
[ ] Move fields that should be in mesh config to mesh config. This will allow us to remove fields like --dnsRefreshRate={{ valueOrDefault .Values.global.proxy.dnsRefreshRate "300s" }}.
[ ] Remove all replaced env vars and flags in pilot-agent, only support reading from mesh config
[ ] Consider doing the same for istio-init. This may be overkill though.
[ ] Investigate all other usages of .Values. Can these be moved to mesh config? If so, move them and remove the need to pass values.yaml in the config map entirely
[ ] (maybe) Remove support for control plane bootstrap configuration, provided it is no longer needed for legacy mixer/galley

Things to move to mesh config:

logLevel
componentLogLevel (we can merge this into one field if we are ok with some post-processing, such that something like info is passed to envoy as -l but http:info as componentLogLevel)
NO: cluster domain - I think this is the same trust domain
statusPort
stsPort

Things to remove:

--serviceCluster can be determined in agent code from labels/pod name variables
--domain can be determined from pod namespace env and trust domain from mesh config (todo verify this)
dnsRefreshRate: https://github.com/istio/istio/pull/21283
--trust-domain we already have it in mesh config...

This is 95% done. The rest is cosmetic mostly. Bumping to nebulous future P1


Issue #3747

Should suppress the warnings about the naming of these two functions. Added TODO

ping

closes #3747

Superseded by #3950


issues with propagation of the data model

The gcxml tool used to generate version specific tsv grammar files incorrectly supports:
fn:IsRequired(fn:SinceVersion(2.0,(fn:RectWidth(Rect)>0) || (fn:RectHeight(Rect)>0)))

This expression is replaced by true in the tsv file for version 2.0. The issue seem to be related with the code of
https://github.com/pdf-association/arlington-pdf-model/blob/80ada1cc457ff2c6d5ed0ccc2955b710a001f51e/gcxml/src/main/java/gcxml/TSVHandler.java#L596

This and many other problems!
Rather than constantly trying to keep multiple implementations (C++, Java, Python) in sync, I propose to move to an ANTLR definition for be official definition of the Arlington predicate grammar: https://github.com/pdf-association/arlington-pdf-model/discussions/63. Do you like this idea or not?


Update README.md

Updated commas and typos. Sir it's general grammar please.

Updated commas and typos. Sir it's general grammar please.
Clearly describe the problem and solution
Relevant issue number(s) if applicable
Did you run npm run lint from the viewer or parliament directory (whichever you are making changes to) and correct any errors
Did you Ensure that all tests still pass by navigating to the tests directory and running ./tests.pl --viewer
License
I confirm that this contribution is made under an Apache 2.0 license and that I have the authority necessary to make this contribution on behalf of its copyright owner.

I guess we need to decide if we are oxford comma or serial comma. In either case a PR with just a single comma change feels like something strange is going on. Is there a number of PR contest or something?


Proposal: Add "runtime" labels support when building

Background
External tools, like docker/compose are currently using name convention to identify image they build as part as their workflow, in order to find them back and manage them. As proposed in this issue #1403, it would be easier and more flexible to use labels for this use cases.
The current way to add labels to images are, for now, to add LABEL lines in Dockerfile. This proposal aims to add a flag(s) to the build command to allow settings labels to images when building them and support this in the API as well (so external tools can work with it).
Although the main use case that comes in mind for me is the use in external tools like compose, I think it's good to be abel to add image label as part of the building, that weren't defined in the Dockerfile.
Syntax
To be coherent, the syntax should act the same as when using the run command (--label and --label-files).
$ docker build --label key=value -t myimage .
$ docker build --label key1=value1 --label key2=value2 -t myimage .
$ docker build --label-file=mybuildinglabels -t myimage .

Let me now how you feel about that. I can work on it if this is something the community finds it useful.

I believe we talked about this a bit during the initial label implementation and I agree it is a very useful feature.  The only way to add Labels to an image today is through a Dockerfile or a commit.  I've never tried the commit and the the commit CLI doesn't support it, but theoretically through the remote API it works.
I agree with the design of this proposal, as it matches run. :+1:

Thanks, and I'm +1
We will need to decide whether the command-line labels will be merged or will replace the existing LABEL definitions in the Dockerfile.
For the compose use-case, the labels should be merged (compose should not replace user defined labels with the compose labels). (Also, obviously, compose uses the Remote API, not the CLI)

@ibuildthecloud I'm not sure it's possible using the API for the commit, or at least it's not documented in https://docs.docker.com/reference/api/docker_remote_api_v1.18/.
@thaJeztah Yes, we need to discuss how /label conflict/ would be handle. If a --label from docker build takes over the LABEL definition or not — and if not, how should docker react (error or /warn and ignore/). We also need to discuss if it should be configurable or not (I would say no, adds complexity for nothing).
I think most of the time, LABEL defined in the Dockerfile should not be overridden by the ones defined when building. Note that, currently, the docker run --label overrides the value specified in the image — not sure it's important for this though.

+1, I have a couple of internal images that combine to repositories. One contains the application code, the other the infrastructure code. I would like to embed the git sha1's as labels in the resulting images

What I concern about this is: we use a Dockerfile to distribute our application and we should make sure everyone use the same Dockerfile should build the completely same image, but if we can use a --lable  on build, different user can build an image with different label

@coolljt0725 Yes, that's why the idea would be to be able to add lables, and not override them (the merged vs replaced). That way you can just add metadata for a given image not removing or replacing some.

#dibs on this
I'll work on that this week and submit a PR to discuss on the possible(s) implementation(s) there.

@vdemeester thanks! make sure to mention the "merge" vs "override" issue when creating the PR so that that is not overlooked during the design review :)

I'm sorry but I'm going to be -1 on this one for a single reason: if we can have a command-line flag to basically provide additional instructions, why not ENV? EXPOSE? VOLUME? I don't want to open this door.

Closing this one for the same reason as th PR :wink:

just curious if this is in 1.8.1?

@mrichard no, it looks like this isn't going to happen. This feature has been rejected by @icecrime IMHO without taking into the consideration the use cases. It can be worked around by using a Makefile to generate the Dockerfile and building that.
Now, maybe he has a point and rather than adding random labels we just want docker build to have a --version option to allow you to add versioning information to the image. But that would mean a whole lot more work than what this feature request required (for example, you'd want to be able to filter on it).

@kleptog docker 1.9 will allow you to set build-time args, so this will
be possible:
FROM ubuntu:14.04

# Define the "foo" arg and set a default value
ARG foo=defaultfoo

# Create a label "mylabel" and set it to the value of $foo
LABEL mylabel=$foo

Building without passing a build-time arg:
docker build -t foo .
docker inspect foo

produces:
...
"Labels": {
    "mylabel": "defaultfoo"
},
...

And specifying a custom value:
docker build -t foo --build-arg foo=commit123 .
docker inspect foo

produces:
...
"Labels": {
    "mylabel": "commit123"
},
...

Oh nice, that's much better and solves the problem nicely. and I see that 1.9 is scheduled for end of this month. This release is full of nice new features!

Excellent! Thanks much for your consideration; this will fit the bill nicely (and is more generally useful than a --label flag).

For what it's worth, this solution does not fit my organization's needs because it requires knowledge of the labels inside the Dockerfile.
Our use case is that we have a generic docker-image-build Jenkins job that builds any repo with ./Dockerfile in it, and pushes it to our registry. We want these images labeled with the repository, commit, and build URL.
If labels are really supposed to be metadata, it doesn't make sense to me to define them inside the Dockerfile, where they really become part of the artifact itself, not metadata about it.

@jantman I just want to echo your comments. We build Docker images as part of our CI process. We would like them labeled with say the SHA1 hash of the git repo (or other meta data) to make debugging, etc. a bit easier. I would think that this is a pretty common use case, but can't seem to find any good ways to do it. What would be nice if one could do something like:
LABEL com.mycompany.myproduct.sha1=$(git rev-parse HEAD)

Or somthing like that.

@hansenms that can be done, but it's a bit of a pain as it needs to be declared in the Dockerfile, i.e.:
ARG git_commit
ARG git_url
ARG build_url

# regular Dockerfile stuff here

LABEL com.myorg.commit=$git_commit com.myorg.repo=$git_url com.myorg.build_url=$build_url

and then build with a command like docker build --build-arg git_commit=$COMMIT --build-arg git_url=$URL --build-arg build_url=$BUILD_URL
What I'd really like is for docker build (and the relevant API) to accept arguments for labels, so that I could just docker build --label com.myorg.commit=$COMMIT --label com.myorg.repo=$URL --label com.myorg.build_url=$BUILD_URL and not have to put anything label-related in the Dockerfile; that should be handled internally to Docker. I really feel that calling this "metadata" is inaccurate; I want to be able to add metadata to a Docker build, not need to have it be an explicit part of the Dockerfile. The way it stands now, if we want to add a new standard label to our Docker containers, we need dozens of teams to update tens - soon to be hundreds - of Dockerfiles to accept the new arg. Or else we need to programmatically edit Dockerfiles in the build process, which means we're not actually building the source we say we are...

@jantman thanks. I was just looking at that. I had docker 1.8 installed, and the arguments were introduced in 1.9 (I think), so I has missed that. Still, it is not quite what I wan't either. Say for instance that you have multiple git repos that are cloned and compiled for a stack that you are putting together in an image. Now you need to first pull those git repos locally before building the docker image so that you can pass the arguments to the build command.  Better would be to let the build process pull various labels and version information to label the images with those values.
I will probably pursue something with the ARGs, but still, I think it is a bit ugly. In any case, thanks for your comments. It was helpful.

Sure! And yes, the build args were introduced in 1.9. We were still running 1.6 on the Docker build slaves for our CI system, and had to upgrade for this.

@jantman I actually already discovered a problem with using the ARGs for the situation above. Scenario is this: during build, my image clones a few gitrepos and in order to LABEL the image with the version or SHA1 hash of those repos, I pull them locally and then pass the SHA1 hash in with an ARG. However, in one case, one of the repos changed between when I did the local pull and when I did the docker build, so now the LABEL doesn't match the actual code that was built in the image.
What I really need is a way to add metadata (LABELs or something else) to reflect specific build events appropriately. Passing it in on the command line is really not the best option no matter how it is done.

@hansenms bit tangential to this issue, but I have some advice about your race condition w/ git repositories.
As you're probably aware, cloning private git repositories inside an image build is inadvisable for security reasons; in addition, if your image's dependencies must be fetched both before and during the image build, one encounters the race condition that you have described.
For both of these reasons, I have adopted the Golden Rule of building private Docker images: fetch your dependencies before of the image build; COPY them and install them during the build.
For instance: Ruby on Rails projects could do a bundle package before invoking docker build, allowing for a much wider range of SSH keys or other Git options to be used. Node projects would perform npm install before building and copy node_modules into the build, etc. Whatever package manager you're working with, likely has a way to pre-fetch dependencies.
Needless to say, applying the Golden Rule requires you to run some shell commands on the host before building the image; it isn't suitable for DockerHub Automated Builds, but it works great with CI environments.
In your case, fetching the dependencies once would also ensure that the metadata you pass to docker build is consistent with what gets built into your image.

@xeger thanks for the comments. With our private repos, I do exactly as you say for reasons of security and other stuff. In general for our own main projects, I do exactly as you say since I may want to build a docker image with a specific version rather than the latest one.
But I also have other dependencies that may not be the focus of that particular build and the whole purpose of the build may be so see if our stuff is breaking with the very latest versions of certain packages. It is actually a handy way of keeping an eye on whether some breaks are about to happen given what is at the head of some other related projects. In that case, it makes a lot of sense to just clone those packages during the build and then LABEL the image with the versions so that when a given image breaks, it is easy to inspect them and see what versions it was running and what changed since last passing build.
Also as you say for environments such as DockerHub, it is also problematic.
Moreover, it is hard for me to understand why one would not allow the build process to set metadata LABELs. Maybe I am missing something.
I think what I may do is just build a text log inside the images with all the labels and then simply create a container from the image and cat the file.

@hansenms I was going to say the same thing, but @xeger beat me to it.
The way we accomplish this is by using common tooling (a Jenkins job) to build Docker images. Each image has its own git repository, with ./Dockerfile in it. However, our build job also looks for ./prebuild.sh, ./postbuild.sh and ./postpush.sh and executes them if present; that's where we do things like cloning repos, and anything else that has to be done on the host. It would be much easier if Dockerfiles had a LOCAL_RUN option that would execute commands on the build host at the appropriate point in the build, but that's not the case. On the whole, we've had to put a lot of hacks in place to work around limitations in Dockerfiles.

@jantman, @xeger
Thanks for the comments and suggestions. I am experimenting with various strategies. I am concerned that the need to clone lots of repos locally is going to prevent us from making some of these images on external build systems. So what I am doing right now is something like this.  After building, you can simply look at those tags:
docker run --rm my_docker_image cat /opt/code/myrepo_sha1.txt

One could of course maintain all of these hashes and versions in a single file (json or something) in the image. It would be like a build manifest and then one could simply get all the metadata in one go and do whatever searching and processing is needed.
I think both strategies have advantages and disadvantages for sure and they are all just workarounds to overcome limitations in the Dockerfiles but if it gets the job done, then OK.
Thoughts?

Flat-file metadata bundled into the image is certainly a valid solution; it's what we did before build-time labeling came along. Each approach has its pros and cons; if it works for you, then use it with impunity!

@xeger What do you mean with "before build-time labeling came along". Isn't that the whole point, there is no way to create LABELs during a docker build. Or is that a new feature?

@hansenms one can use --build-arg to provide variable inputs to the Dockerfile, so that LABEL and other instructions can consume information from the build environment. For instance:
# in Dockerfile
ARG gitref unknown
LABEL org.git.ref ${gitref}

# on CLI
docker build --build-arg gitref=`git rev-parse --verify HEAD`

This is build-time labeling in the sense that I can influence the built image without changing the Dockerfile.
You have a chicken-and-egg problem that makes it hard to provide accurate build args, but a --label flag to docker build would have the same problem .. the best solution for your issue, it seems, would be to allow the Dockerfile to "capture" build state (e.g. "which version of this dependency did I end up installing during my last RUN instruction?) and then apply that build state to future instructions.
The Docker team has been careful about determinism during image builds: images built repeatedly from a given build context should ideally be identical. Among other benefits, determinism allows for layer caching, which speeds up image builds. Allowing build state to be captured and used as build args, would promote non-deterministic builds and make layer caching complex or impossible.

@xeger, understood.
I guess my view on this is that as long as you can do git clone, wget, apt-get update, etc. etc. within a Dockerfile, images built repeatedly from the same context can and will vary. The fact that there is no way to capture what those variations may be really doesn't change that. It just makes it opaque.
The flat file metadata is really the only way to capture that right now. It is fine, but it would be more elegant and standardized if there was a way to capture such metadata. The command line build-args are also problematic for some automated build systems, e.g., it is hard to do what you propose on Docker Hub.
In any case, I understand the position of the Docker team and I respect that my "important" features may in fact be distractions that could create unintended consequences. But in case these sorts of features were being discussed for the roadmap, I wanted to register a vote in favor.

@xeger @jantman
If it is of interest, I wrote up how we handle it here:
http://hansenms.github.io/devops/tips/2016/02/26/docker-build-manifest.html


Refactored inline style to address eslint warning.

Summary:
Seemed a shame to have validate:eslint marred by a warning, so threw the inline style in to StyleSheet.create.
Test Plan:
npx yarn test in example directory passes.
Verified example app visually on iOS and Android emulators.

Hi @michalchudziak and @wojteg1337. Tagging you as this isn't attached to any issue. Sing out if there is anything you need to make this an actionable PR.


Cell numbers in labels.npz not same as cell_barcode_count.csv

Hi,
I want to investigate if STARmap data can be analyzed with out spatial methods.
These work by relating gene expression to spatial coordinates.
In all the data available at https://www.starmapresources.com/data/ there are more regions defined in the labels.npz file compared to the number of cells in the cell_barcode_count.csv file.
Here is a quick check I did:
for fname in glob('*/*/cell_barcode_count.csv'):
    counts = pd.read_csv(fname, header=-1)
    cells, genes = counts.shape

    labels_fname = fname.replace('cell_barcode_count.csv', 'labels.npz')
    labels = np.load(labels_fname)['labels']
    regions = len(regionprops(labels))

    rootname = fname.split('/')[1]

    print(f'{rootname}\t- Regions: {regions}\tCells: {cells}\tGenes: {genes}')

>
20171110_BF2_dark	- Regions: 1173	Cells: 1167	Genes: 160
20170915_BD2_dark	- Regions: 963	Cells: 941	Genes: 160
20171115_BF5_light	- Regions: 976	Cells: 975	Genes: 160
20171120_BF4_light	- Regions: 932	Cells: 931	Genes: 160
20180421_BZ19_cocaine	- Regions: 1327	Cells: 1320	Genes: 166
20180424_BZ14_control	- Regions: 1222	Cells: 1215	Genes: 166
20180419_BZ9_control	- Regions: 1385	Cells: 1380	Genes: 166
20180417_BZ5_control	- Regions: 1130	Cells: 1127	Genes: 166
20180505_BY3_1kgenes	- Regions: 1550	Cells: 1549	Genes: 1020
20180410-BY3_1kgenes	- Regions: 974	Cells: 973	Genes: 1020

It seems some of the regions should be discarded, but I have no way of knowing which ones?
Looking through your code here, there doesn't seem to be any code that e.g. plots a given genes expression at spatial coordinates?
Best
/Valentine

Hi Valentine,
Good catch -- yes, I applied some size filtering of the cells (keeping only cells with < 100,000 and >10,000 pixels) in reads_to_cells.py. (https://github.com/weallen/STARmap/blob/master/python/reads_to_cells.py). I've updated the function GetQHulls in viz.py (https://github.com/weallen/STARmap/blob/master/python/viz.py) to do this; the previous version had the wrong threshold.
There is also code in that file to plot expression overlaid on the cell segmentation. In general, I only used QHulls to plot the cell segmentation, as it's much faster than working with the raw pixel mask.
E.g. to plot expression of gene N, try:
import matplotlib.pyplot as plt
qhulls,coords = GetQHulls(labels)
n=10
plot_poly_cells_expression(labels, qhulls, counts.iloc[:,n], plt.cm.jet)

I don't have any code in Python that plots the individual dots for a gene -- if you write one, I can  incorporate it into the package.

Hi Valentine,
Yes, I think this is due to a difference between how regionprops works, and how I had been selecting cells using indexing into the array. All of my plotting in the paper was using QHulls selected using indexing directly into the array, and I hadn't properly tested the regionprops code -- my apologies, I'll update the starmapresources website to explain this. I suspect there is an off-by-one error in how regionprops deals with labels==0.
For now, I would just use the GetQHulls function, and get centroids using the code:
all_centroids  = np.vstack([c.mean(0) for c in coords])
E.g. to plot PCP4 (gene index = 102) in the dark 160 gene experiment:
plt.figure(figsize=(10, 4))
plt.axes().set_aspect('equal', 'datalim');
plt.scatter(-all_centroids[:, 1], -all_centroids[:, 0], c=np.log1p(counts.iloc[:,102]));
plt.axis('off')
`

This worked for me on both the 1000 gene data and 160 gene dark dataset.

Will

Thanks Will!
Now I got it working, the GetQHulls function takes a while, but I can of course save the result and re-use.
By the way, I was also wondering something about the large 30K cell dataset: How come the gene counts are so much higher per cell compared to the other datasets?
For example Pcp4 has an average expression of ~10 counts/cell in the 160 gene dataset, but an average expression of ~6,000 counts/cell in the large dataset.
Is the sequential barcoding version of STARmap much more sensitive than the combinatorial barcoding version?
/Valentine

Hi Valentine,
I think this was explained in the paper, but the measurements in the sequential mode are not counts but relative intensity measurements within some volume around the cell nucleus. The scaling is somewhat arbitrary. We imaged at low magnification to capture the large volume within a reasonable amount of time, so individual dots were not resolvable.
Will

Hi Will,
Oh I see, I didn't catch that when I read it.
That makes sense, those expression levels don't really behave the way counts do. It's cool that it still seems pretty quantitative!
Thanks a lot for your help with these things.
Best
/Valentine


✨ add progressbar component

@rishichawda dummy comment to verify git hook.

Codecov Report

Merging #125 into development will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff              @@
##           development     #125   +/-   ##
============================================
  Coverage        86.92%   86.92%
============================================
  Files               20       20
  Lines              497      497
============================================
  Hits               432      432
  Misses              65       65

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ba80249...9fb3ea4. Read the comment docs.

@rishichawda Looks fine. The number looks small. But for now will merge it. Will update the design when I get time


cant build properly

Hugo Version:  0.68.3
Build Command: hugo
Hello, I have been trying to use your theme, I added it to my theme  folder and when i launch the server everything works perfectly... but when building it with the command hugo, i click on the public/index.html and the page is just white with text https://i.imgur.com/u5ygLRV.png

Because got an error:
GET file:///css/core.min.dd5205527998168e53b52fa8c89c1c62d4ab7c724ca28242df50a57a7b69e918e6465a463d96ffd00ef82c1affc0b44b.css net::ERR_FILE_NOT_FOUND

Recommend debug theme with hugo server mode.


Remove project coverage report

Any objections?

No strong objections, since the patch coverage is arguably more import for PRs, just curious what the rationale is for disabling the project coverage report :slightly_smiling_face:


Add --skip-simulation and --gas-estimate-multiplier cli options to forge script command

Motivation
In some edge cases the on-chain simulation currently blocks users from running certain types of transactions, even though they would work fine when submitted on chain.
See #2522
For cases like these it'd be great if the user could bypass on-chain simulation.
Solution
Add command line parameter --skip-simulation that the user can set to disable on chain simulation.
When skipping simulation the gas costs will be estimated through the rpc-provider individually before each transaction submission.
When testing that new functionality the transactions ran out of gas and failed so I added the parameter --gas-estimate-multiplier to manually adjust the factor by which gas estimates are inflated (or reduced).
Default value for this is 130 (meaning gas estimates will be multiplied by 1.3) which was the value previously hardcoded for estimations coming from the on-chain simulation.
For consistency this default will also be applied to the provider estimate in the case the respective chain has a different gas calculation, which was previously used as returned by the provider (see this change), this should be the only change to the existing behaviour (when the user uses none of the new arguments).

@onbjerg: Any feedback / review on this would be greatly appreciated 🙏


Fix hidden buttons option

Thanks for the PR! I didn't notice the error, yet there's a small typo in it. It's hiddenButtons instead of hiddendButtons. If you could fix that, I will gladly merge it.

Ops, sorry. Fixed ;)

Awesome! Thank you.


Rename Fut to ResponseFuture

Description
Resolves https://github.com/rustasync/http-service/issues/24
Motivation and Context
Currently, the associated params of HttpService are ConnectionFuture which corresponds to connect, and the Fut which corresponds to respond.
I think it'd be better to call this ResponseFuture such that it maintains parity.
connect -> ConnectionFuture
respond -> ResponseFuture
rather than the Fut that isn't really informative or intuitive.
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.

You'll need to change the associated type in http-service-mock to satisfy CI, but other than that this change gets a :+1: from me. (It's a tiny modification with potentially huge positive impact.)

Yup. Was just working on it. This should do it


Capture images using Camera

To capture the image using webcam using Python Assistant, ecapture module of python is implemented

Please assign this issue to me @Umesh-01

"Hi, I am Raj Kushwaha, a contributor to girls script foundation 2022. I would like to work on this issue.Can you allow me to add my skills to this project.

Fixed in #16


'Bottom' position rendering over field

Using 0.5.1, when setting position to 'bottom' ('top' works fine), I'm seeing the container overlap the input field, so the user can't see what's being typed. Tried to correct by adding some css, but no luck. Can the position of the container be adjusted easily?

Also experiencing this

I'm also seeing this issue.
I'm using Blaze with Meteor 1.5.x, and the Materialize library for styling.