unable to run tflite_runtime with libedgetpu on raspberry pi zero wh

Hi,
Having some issues getting the edgetpu runtime to work with the tflite_runtime package on my raspberry pi zero (with the USB accelerator) - I'm new to all this and I've tried everything I can think of, can anyone help? Findings are below, many many thanks in advance.
dpkg -l | grep edgetpu
ii  libedgetpu1-std:armhf                 14.1                                   armhf        Support library for Edge TPU
ii  python3-edgetpu                       14.1                                   armhf        Edge TPU Python API

Attempting using my own compiled tflite_runtime package following the steps specified here: https://github.com/google-coral/edgetpu/issues/198 results in: Illegal instruction:
python3 -c 'print(__import__("tflite_runtime").__version__)'
2.1.0
LD_PRELOAD=/usr/lib/arm-linux-gnueabihf/libatomic.so.1.2.0 python3 index.python
Illegal instruction

and then I've tried using some prebuilt wheels that other people have posted on the various issues reported, the one below is tflite_runtime 2.2.0 (not sure if the rc3 makes any significant differences here)
python3 -c 'print(__import__("tflite_runtime").__version__)'
2.2.0rc3
LD_PRELOAD=/usr/lib/arm-linux-gnueabihf/libatomic.so.1.2.0 python3 index.python
Traceback (most recent call last):
  File "index.python", line 94, in <module>
    interpreter.allocate_tensors()
  File "/home/pi/Documents/Python 3/Object Detection/tflite1-env/lib/python3.7/site-packages/tflite_runtime/interpreter.py", line 242, in allocate_tensors
    return self._interpreter.AllocateTensors()
  File "/home/pi/Documents/Python 3/Object Detection/tflite1-env/lib/python3.7/site-packages/tflite_runtime/interpreter_wrapper.py", line 115, in AllocateTensors
    return _interpreter_wrapper.InterpreterWrapper_AllocateTensors(self)
RuntimeError: Internal: Unsupported data type in custom op handler: 0Node number 2 (EdgeTpuDelegateForCustomOp) failed to prepare.

Then the last one is tflite_runtime 2.3.1:
python3 -c 'print(__import__("tflite_runtime").__version__)'
2.3.1
LD_PRELOAD=/usr/lib/arm-linux-gnueabihf/libatomic.so.1.2.0 python3 index.python
Traceback (most recent call last):
  File "index.python", line 94, in <module>
    interpreter.allocate_tensors()
  File "/home/pi/Documents/Python 3/Object Detection/tflite1-env/lib/python3.7/site-packages/tflite_runtime/interpreter.py", line 243, in allocate_tensors
    return self._interpreter.AllocateTensors()
RuntimeError: Internal: Unsupported data type in custom op handler: 0Node number 2 (EdgeTpuDelegateForCustomOp) failed to prepare.

From what I can tell the last 2 issues (tflite_runtime_2.2.0rc3 & tflite_runtimme_2.3.1) are closer to success than the first, the issues reported before mine indicate some sort of mismatch between the commit of tensorflow that the libedgetpu runtime has been built with and the version used by the tflite_runtime, however my compiled version of tflite_runtime_2.1.0 should have fixed this shouldn't it?
(All of these have been ran making sure I have the correct libedgetpu runtime for the architecture of the RPI0 (armv6) by doing the below commands from the official edgetpu git repository (which I believe is 14.1?):
git clone https://github.com/google-coral/edgetpu.git
cd edgetpu/libedgetpu/throttled/armv6
sudo cp libedgetpu.so.1 /usr/lib/arm-linux-gnueabihf

but the above is just me guessing that this needs to be done really.
Once again, thanks in advance!

@rhadnum hello, couple you share the outputs of this:
$ uname -a

hi @Namburger, here you go:
uname -a
Linux raspberrypi 5.4.51+ #1333 Mon Aug 10 16:38:02 BST 2020 armv6l GNU/Linux

The issue is that we no longer release binaries for armv6 as per requirements. So when trying to use the tflite_runtime package, it is throwing some illegal instruction because it wasn't built for that.
Just to ensure that you've built the right commit, could you also share this outputs?
python3 -c 'print(__import__("tflite_runtime").__git_version__)'

Ah right, that makes sense.
Weirdly it returns the below:
python3 -c 'print(__import__("tflite_runtime").__git_version__)'
0.6.0-76902-gd855adfc5a

although I definitely remember running git checkout d855adfc5a0195788bf5f92c3c7352e638aa1109 in the tensorflow repo route, am I doing something wrong here?
Regarding the LD_PRELOAD - if I don't specify that then it returns the below so not sure if I've compiled something wrong there:
python3 index.python
Traceback (most recent call last):
  File "/home/pi/Documents/Python 3/Object Detection/tflite1-env/lib/python3.7/site-packages/tflite_runtime/interpreter_wrapper.py", line 14, in swig_import_helper
    return importlib.import_module(mname)
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 670, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 583, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1043, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /home/pi/Documents/Python 3/Object Detection/tflite1-env/lib/python3.7/site-packages/tflite_runtime/_interpreter_wrapper.so: undefined symbol: __atomic_compare_exchange_8

Hope I'm not spamming or throwing too much info, but here is my suggestion:

As above, get your tflite_runtime package commit version (use one that doesn't cause your Illegal instruction issue, since that's an external dependencies that we can't control):

python3 -c 'print(__import__("tflite_runtime").__git_version__)'

Build your own libedgetpu.so from this repo: https://github.com/google-coral/libedgetpu/
Make sure to change the commit here to be the same commit with your tflite_runtime package and then build using (you'll need docker installed and do this on an x86_64 machine):

DOCKER_CPUS="armv6" DOCKER_TARGETS=libedgetpu make docker-build

Then you can copy the resulting libedgetpu.so to your rpi0 and mv it to /usr/lib/arm-linux-gnueabihf.
I do want to make a note that we stopped supporting armv6 and don't have any tests coverage for this architect, but I'd love to see some success stories! Please let me know if this works,
Cheers

Not spam at all, the more I can try the better! 😄
Brilliant yeah I never thought of building the edgetpu runtime myself, completely overlooked that.
This is more of a hobby project for me, curious to see what the performance improvements look like with the usb accelerator as without it's 0.07 fps (which is understandable considering the memory limitations of the RPI0).
Once again, many thanks for the quick responses and the suggestions :)

Managed to get it working on my raspberry pi zero now following the steps mentioned by @Namburger. Performance is still subpar which is to be expected but a definite increase in fps at around 0.7 / 0.8 (roughly 10x with the usb accelerator!)

Awesome!

@rhadnum hey I was wondering if you also built your own tflite_runtime package for 2.3.1? I have another user who is trying to work with the pi0 and this may helps

Hi @Namburger - yeah I did manage to build it but it took a while to do (and I don't quite remember how I did it 😄 )
All the .whl files that I tried can be found here (only the 2.3.1 one worked in the end for me): https://drive.google.com/drive/folders/1VtkKbpKnv9vHylQ62ekuGvyco2Jg9pQe?usp=sharing
To build the libedgetpu runtime I:

got the commit hash for this specific build by running:

python3 -c 'print(__import__("tflite_runtime").__git_version__)'

Get the full commit hash by running:

git rev-parse {short-hash-here}

Then just build the libedgetpu runtime by using your instructions above and replacing the hash in this file with the commit hash returned from the above command.

Hope this helps!

@rhadnum thanks for the package, you're amazing!

Hi @rhadnum
Just wondering how you got the full hash of the tflite_runtime from the SHA? (perhaps I'm missing what git directory I'm supposed to me in)

@sim-kelly I think after having the SHORT_HASH, you can use git rev-parse SHORT_HASH

OK - thanks so much for the help with this!!
So I think I'm just missing something in following this.
I have installed the specific .whl that @rhadnum linked (I now have 2.3.1 tflite_runtime).
When I get the short hash, following the instructions, I do not follow which .git directory (if any, and on what machine (the armv6 or x86_64) I should be in to run the following command
git rev-parse SHORT_HASH

Oh, so after installing tflite_runtime 2.3.1, you need to run this:
$ python3 -c 'print(__import__("tflite_runtime").__git_version__)'

That give you the SHORT_HASH, then you can run git rev-parse to get the full hash

Yes - I have the short hash, but I cannot run the git rev-parse from the root directory?? (and I dont know which directory I should be in as I installed the .whl without touching git)

@sim-kelly so git rev-parse should works anywhere that you have git installed :)

I also changed line 41 (TENSORFLOW_SHA256) to match the commit, and things compiled!

Right, I forgot you need to change that too.
Now that you have it compiled, you can move the new libedgetpu.so to /usr/lib/arm-linux-gnueabihf. If the commits I sent is correct then everything should works!

These are the steps that worked for me:
Download tflite_runtime-2.3.1-cp37-cp37m-linux_armv6l.whl to the RPI Zero. Install it using:
$ sudo pip3 install tflite_runtime-2.3.1-cp37-cp37m-linux_armv6l.whl
Get the SHORT_HASH:
$ python3 -c 'print(__import__("tflite_runtime").__git_version__)'
It was 0.6.0-88810-g5681c179ef in this situation.
Clone the tensorflow repo and run:
$ git rev-parse 0.6.0-88810-g5681c179ef
It returned 5681c179eff80bce00e526303950b67b23cad14c.
Generate the SHA256 (the filename is the value returned by git rev-parse with tar.gz added):
wget https://github.com/tensorflow/tensorflow/archive/5681c179eff80bce00e526303950b67b23cad14c.tar.gz
shasum -a 5681c179eff80bce00e526303950b67b23cad14c.tar.gz
It returned 18c5798270ea27c6991ee9bbf1357884d8f91589554799f4b4e7abff0ab75278
Clone the libedgetpu repo, open the file workspace.bzl, find the lines:
TENSORFLOW_COMMIT = ...
TENSORFLOW_SHA256 = ...

and replace with
TENSORFLOW_COMMIT = "5681c179eff80bce00e526303950b67b23cad14c"
TENSORFLOW_SHA256 = "18c5798270ea27c6991ee9bbf1357884d8f91589554799f4b4e7abff0ab75278"

Now, still inside libedgetpu repo, run:
DOCKER_CPUS="armv6" DOCKER_TARGETS=libedgetpu make docker-build
It takes a while, but at the end it generates the out/direct/armv6/libedgetpu.so.1. Copy it to the RPI Zero:
scp out/direct/armv6/libedgetpu.so.1 pi@<RPI_IP>:/home/pi/.
Log into the RPI Zero (ssh pi@<RPI_IP>). Move the libedgetpu.so.1:
sudo mv libedgetpu.so.1 /usr/lib/arm-linux-gnueabihf/.
It was necessary to add udev rules:
sudo vi /etc/udev/rules.d/99-edgetpu-accelerator.rules
SUBSYSTEM=="usb",ATTRS{idVendor}=="1a6e",GROUP="plugdev"
SUBSYSTEM=="usb",ATTRS{idVendor}=="18d1",GROUP="plugdev"

And add the user to plugdev group:
sudo usermod -aG plugdev $USER
Connect the Coral TPU USB to a micro USB to USB OTG adapter (no external power)...
...after all that, it worked!
I tested it according to the Coral docs and I found it quite fast. On my laptop it was not that much faster.
On the RPI Zero W:
----INFERENCE TIME----
Note: The first inference on Edge TPU is slow because it includes loading the model into Edge TPU memory.
117.7ms
11.7ms
13.4ms
11.8ms
13.4ms
-------RESULTS--------
Ara macao (Scarlet Macaw): 0.77734

On my laptop (USB3, biffy CPU...):
----INFERENCE TIME----
Note: The first inference on Edge TPU is slow because it includes loading the model into Edge TPU memory.
12.7ms
4.4ms
4.3ms
4.3ms
4.1ms
-------RESULTS--------
Ara macao (Scarlet Macaw): 0.77734

However, when I try to generate my own pip package (commit 2be88a7f1a42ca8ff623e6953d7439ec284f67b8, generating tflite_runtime-2.5.0-cp37-cp37m-linux_armv6l.whl), it needs the LD_PRELOAD=/usr/lib/arm-linux-gnueabihf/libatomic.so.1.2.0 and it always gives the Illegal instruction after import tflite_runtime.interpreter as tflite.
Here is my build_pip_package_with_bazel.sh in case someone can point me the way:
#!/usr/bin/env bash
# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
set -ex

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PYTHON="${CI_BUILD_PYTHON:-python3}"
VERSION_SUFFIX=${VERSION_SUFFIX:-}
export TENSORFLOW_DIR="${SCRIPT_DIR}/../../../.."
TENSORFLOW_LITE_DIR="${TENSORFLOW_DIR}/tensorflow/lite"
TENSORFLOW_VERSION=$(grep "_VERSION = " "${TENSORFLOW_DIR}/tensorflow/tools/pip_package/setup.py" | cut -d= -f2 | sed "s/[ '-]//g")
export PACKAGE_VERSION="${TENSORFLOW_VERSION}${VERSION_SUFFIX}"
BUILD_DIR="${SCRIPT_DIR}/gen/tflite_pip/${PYTHON}"
TENSORFLOW_TARGET=$1

# Fix container image for cross build.
if [ ! -z "${CI_BUILD_HOME}" ] && [ `pwd` = "/workspace" ]; then
  # Fix for curl build problem in 32-bit, see https://stackoverflow.com/questions/35181744/size-of-array-curl-rule-01-is-negative
  if [ "${TENSORFLOW_TARGET}" = "armhf" ]; then
    sudo sed -i 's/define CURL_SIZEOF_LONG 8/define CURL_SIZEOF_LONG 4/g' /usr/include/curl/curlbuild.h
    sudo sed -i 's/define CURL_SIZEOF_CURL_OFF_T 8/define CURL_SIZEOF_CURL_OFF_T 4/g' /usr/include/curl/curlbuild.h
  fi

  # The system-installed OpenSSL headers get pulled in by the latest BoringSSL
  # release on this configuration, so move them before we build:
  if [ -d /usr/include/openssl ]; then
    sudo mv /usr/include/openssl /usr/include/openssl.original
  fi
fi

# Build source tree.
rm -rf "${BUILD_DIR}" && mkdir -p "${BUILD_DIR}/tflite_runtime"
cp -r "${TENSORFLOW_LITE_DIR}/tools/pip_package/debian" \
      "${TENSORFLOW_LITE_DIR}/tools/pip_package/setup_with_bazel.py" \
      "${TENSORFLOW_LITE_DIR}/tools/pip_package/MANIFEST.in" \
      "${TENSORFLOW_LITE_DIR}/python/interpreter_wrapper" \
      "${BUILD_DIR}"
cp "${TENSORFLOW_LITE_DIR}/python/interpreter.py" \
   "${BUILD_DIR}/tflite_runtime"
echo "__version__ = '${PACKAGE_VERSION}'" >> "${BUILD_DIR}/tflite_runtime/__init__.py"
echo "__git_version__ = '$(git -C "${TENSORFLOW_DIR}" describe)'" >> "${BUILD_DIR}/tflite_runtime/__init__.py"

# Build python interpreter_wrapper.
cd "${BUILD_DIR}"
case "${TENSORFLOW_TARGET}" in
  armhf)
    BAZEL_FLAGS="--config=elinux_armhf
      --copt=-march=armv6
      --copt=-O3 --copt=-fno-tree-pre --copt=-fpermissive
      --define tensorflow_mkldnn_contraction_kernel=0
      --define=raspberry_pi_with_neon=false"
    ;;
  aarch64)
    BAZEL_FLAGS="--config=elinux_aarch64
      --define tensorflow_mkldnn_contraction_kernel=0
      --copt=-O3"
    ;;
  native)
    BAZEL_FLAGS="--copt=-O3 --copt=-march=native"
    ;;
  *)
    BAZEL_FLAGS="--copt=-O3"
    ;;
esac

# We need to pass down the environment variable with a possible alternate Python
# include path for Python 3.x builds to work.
export CROSSTOOL_PYTHON_INCLUDE_PATH

case "${TENSORFLOW_TARGET}" in
  windows)
    LIBRARY_EXTENSION=".pyd"
    ;;
  *)
    LIBRARY_EXTENSION=".so"
    ;;
esac

bazel build -c opt -s --config=monolithic --config=noaws --config=nogcp --config=nohdfs --config=nonccl \
  ${BAZEL_FLAGS} ${CUSTOM_BAZEL_FLAGS} //tensorflow/lite/python/interpreter_wrapper:_pywrap_tensorflow_interpreter_wrapper
cp "${TENSORFLOW_DIR}/bazel-bin/tensorflow/lite/python/interpreter_wrapper/_pywrap_tensorflow_interpreter_wrapper${LIBRARY_EXTENSION}" \
   "${BUILD_DIR}/tflite_runtime"
# Bazel generates the wrapper library with r-x permissions for user.
# At least on Windows, we need write permissions to delete the file.
# Without this, setuptools fails to clean the build directory.
chmod u+w "${BUILD_DIR}/tflite_runtime/_pywrap_tensorflow_interpreter_wrapper${LIBRARY_EXTENSION}"

# Build python wheel.
cd "${BUILD_DIR}"
case "${TENSORFLOW_TARGET}" in
  armhf)
    ${PYTHON} setup_with_bazel.py bdist --plat-name=linux-armv6l \
                       bdist_wheel --plat-name=linux-armv6l
    ;;
  aarch64)
    ${PYTHON} setup_with_bazel.py bdist --plat-name=linux-aarch64 \
                       bdist_wheel --plat-name=linux-aarch64
    ;;
  *)
    if [[ -n "${TENSORFLOW_TARGET}" ]] && [[ -n "${TENSORFLOW_TARGET_ARCH}" ]]; then
      ${PYTHON} setup_with_bazel.py bdist --plat-name=${TENSORFLOW_TARGET}-${TENSORFLOW_TARGET_ARCH} \
                         bdist_wheel --plat-name=${TENSORFLOW_TARGET}-${TENSORFLOW_TARGET_ARCH}
    else
      ${PYTHON} setup_with_bazel.py bdist bdist_wheel
    fi
    ;;
esac

echo "Output can be found here:"
find "${BUILD_DIR}"

# Build debian package.
if [[ "${BUILD_DEB}" != "y" ]]; then
  exit 0
fi

PYTHON_VERSION=$(${PYTHON} -c "import sys;print(sys.version_info.major)")
if [[ ${PYTHON_VERSION} != 3 ]]; then
  echo "Debian package can only be generated for python3." >&2
  exit 1
fi

DEB_VERSION=$(dpkg-parsechangelog --show-field Version | cut -d- -f1)
if [[ "${DEB_VERSION}" != "${PACKAGE_VERSION}" ]]; then
  cat << EOF > "${BUILD_DIR}/debian/changelog"
tflite-runtime (${PACKAGE_VERSION}-1) unstable; urgency=low

  * Bump version to ${PACKAGE_VERSION}.

 -- TensorFlow team<EMAIL_ADDRESS>$(date -R)

$(<"${BUILD_DIR}/debian/changelog")
EOF
fi

case "${TENSORFLOW_TARGET}" in
  armhf)
    dpkg-buildpackage -b -rfakeroot -us -uc -tc -d -a armhf
    ;;
  aarch64)
    dpkg-buildpackage -b -rfakeroot -us -uc -tc -d -a arm64
    ;;
  *)
    dpkg-buildpackage -b -rfakeroot -us -uc -tc -d
    ;;
esac

cat "${BUILD_DIR}/debian/changelog"

I changed the lines:
--copt=-march=armv7-a --copt=-mfpu=neon-vfpv4

to
--copt=-march=armv6

...and
--define=raspberry_pi_with_neon=true"

to
--define=raspberry_pi_with_neon=false"

...and finally
    ${PYTHON} setup_with_bazel.py bdist --plat-name=linux-armv7l \
                       bdist_wheel --plat-name=linux-armv7l

to
    ${PYTHON} setup_with_bazel.py bdist --plat-name=linux-armv6l \
                       bdist_wheel --plat-name=linux-armv6l

It goes until the end and generates, among other things, tensorflow/lite/tools/pip_package/gen/tflite_pip/python3.7/dist/tflite_runtime-2.5.0-cp37-cp37m-linux_armv6l.whl, but as I said before, it doesn't work.

@ricardodeazambuja wow this is actually very nice, many many thanks!


Implement AKS Security Profile

Hello!

Vote on this issue by adding a 👍 reaction
If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details
After enabling azure defender on an AKS cluster that has been deployed with pulumi, a high priority recommendation will be created saying that: Azure Kubernetes Service clusters should have Defender profile enabled. The Microsoft docs specify that this feature can be enabled by including a securityProfile in your cluster definition. Currently pulumi does not support a securityProfile on AKS clusters, meaning we cannot resolve this security concern without making custom pulumi resource definitions.
Affected area/feature

I am not sure which feature would be affected.

Are there any plans to add a securityProfile property to ManagedClusterArgs

this is implemented as microsoftDefender on the resource so going to close this out.


Having problems using with React.js

How do we integrate this with react.js?

I haven't tried yet, but I figure you'd do it like you would any other jQuery component, just instantiate it on your element in componentDidMount()
class MyForm extends Component {
  componentDidMount() {
    React.findDOMNode(this).validator()
  }

  componentWillUnmount() {
    React.findDOMNode(this).validator('destroy')
  }

  render() {
    return (
      <form>

          Name
          <input type="text" className="control-label" required>

      </form>
    )
  }
}

hi @1000hz  thanks alot for your reply!

:+1:


Considerations for modeling age

Consider if there is anything special we want to do with mother's age, since the relationship between age and fertility is nonlinear.
Consider adding a "psuedo-feature" which is the age-specific fertility rate for that person's age.

Interact age with sex for both self and partner

I generated a feature on partner's age by "coalescing" data on partner's birthyear across waves. The partner's birthyear is only recorded at the first wave when that partner is present. For example, if someone got together with a new partner in 2015, the partner's birthyear is recorded that year, but is not recorded again in future years even if they are still together. Therefore, I created a feature that contains the most recently reported partner birth year.
Otherwise, we did not do anything special with the age features. XGboost can already handle interactions and nonlinearities.


Unresponsive PC when a new layout is added, followed by exceptionless crash

Let me start by saying, workspacer is fantastic. I've been search and trying to find exactly this, but all other projects fall short. On to the issue -
I've written a new layout engine, compiled the app with dotnet 3.1, instantiated that engine in the config, but once I launch the built workspacer.exe, the PC becomes sluggish to the point of unresponsiveness (for 4-5 seconds) and then crashes without an exception message. I've tried to copy/paste one of the included layouts while changing the class names to make it unique, and the same thing happens
Removing it from the config makes workspacer work as expected
Thanks!

Hi @doctorshiban , I could think of two reasons why this might happen based of the fact that you testing method of copy-pasting an existing LayoutEngine didn't work.

There is an error in your config file
it should look something like this (I'm using sticky workspaces but that shouldn't matter):

var sticky = new StickyWorkspaceContainer(context, StickyWorkspaceIndexMode.Local);
context.WorkspaceContainer = sticky;

sticky.CreateWorkspace("Vertical Layout", new VertLayoutEngine());

You ran the "wrong" workspacer.exe
If you have installed the release version and also compiled one yourself make sure that you run the right workspacer.exe file (depending on how you compiled it would be in
Path to the source files\src\workspacer\bin\Debug\netcoreapp3.1\win-x64
or
Path to the sourcefiles\src\workspacer\bin\Release\netcoreapp3.1\win-x64

The LayoutEngines  are stored in the workspacer.shared.dll file, so if you compile it from source that creates  a new .dll , but the release version still uses the "old" file. So if you have both a source-compiled and the github release version make sure you execute the right one. You could also just copy the newly created .dll to the existing installation folder (by default in C:\Program Files\workspacer) and overwrite the original.
I hope this helps, if it's neither of the two then a bit more digging might be needed though

This seems to have been the issue (thanks for the patience)!


uReplicator vs. Kafka MirrorMaker w/New Consumer (not high-level consumer)

Hi.
The documentation on the web for uReplicator seems to compare uReplicator to Kafka MirrorMaker before the arrival of "new consumer" support for it.
https://docs.confluent.io/current/clients/consumer.html
Does this upstream change obsolete uReplicator in any way?  There are features of uReplicator that I enjoy like the REST API -- but are the high-level consumer concerns about MirrorMaker on the wiki and blog pages still relevant with the new consumer?  Thanks!

No. MirrorMaker worker doesn't use those features in high-level consumer like auto partition assignment/balance. It depends on Controller to assign partitions to worker, in which the consumer simply consume the partition it got assigned. So it doesn't make much different to use SimpleConsumer or NewConsumer in Worker. Besides the REST API, the biggest benefit of uReplicator Controller is that it can assign partitions in a smart way: based on number of partitions and traffic, and can move lagging partitions to a dedicated worker, which can significantly improve the replication performance.


HDDS-8261. Fail integration tests if RocksDB objects are leaked

What changes were proposed in this pull request?
To catch RocksDB object leaks earlier, we should fail integration tests proactively, if leak is detected.

Added assertion about ManagedRocksObjectsMetrics in MiniOzoneClusterImpl#shutdown.
Check test output for warnings emitted by gRPC and managed RocksDB objects.  (Minor fix: avoid duplicates due to some tests matching multiple conditions.)
Fix some tests that still had leaks.  Production code leaks were fixed separately for easier backports.

https://issues.apache.org/jira/browse/HDDS-8261
How was this patch tested?
CI:
https://github.com/adoroszlai/hadoop-ozone/actions/runs/4733776484

Thanks @duongkame, @Xushaohong for the review.


Feat: Add GSoC Program Projects for Multiple Years

Describe the feature
We need to extend the existing structure for GSoC projects to include data for multiple years, including but not limited to 2024, 2023, 2022, 2021, and 2020.
Current Structure:
A folder named assets/projects/gsoc_projects has been created, containing a file named gsoc24projects.json for GSoC 2024 program projects.
Required Changes:
Add JSON files for GSoC projects from previous years (e.g., gsoc23projects.json, gsoc22projects.json, etc.) in the assets/projects/gsoc_projects folder.
Ensure each file follows the same format as gsoc24projects.json.
Update any relevant code or documentation to accommodate the new files.
This will help in maintaining a comprehensive list of GSoC projects across different years, making it easier to reference past projects.
Add ScreenShots
na
Record

[X] I agree to follow this project's Code of Conduct
[ ] I'm a GSSOC'24 contributor
[ ] I want to work on this issue

/assign

please assign me this @andoriyaprashant

Can I work on this issue?


clean output directory "build" before building

Everytime a build is created, the output directory "build" should be cleaned.
This prevents from uncesessary/deleted files from last build and also helpful during development.

@blackmad  Please have a look on this PR.
First PR :)


E2e test for max-pods test case and resource limit checks

What this PR does / why we need it:
Adding additional E2e test cases

/lgtm
/approve


current version of the linter doesn't allow map(..).flatten() to be called on an Option

Hi there,
As the tile says, the current version of the linter doesn't allow map(..).flatten() to be called on an Option. This currently breaks the CI and any new PR will fail. See for example (https://github.com/georust/gdal/runs/5443188765?check_suite_focus=true)
The solution suggested by the compiler seems to work all the tests still pass. This PR fixes the issue.

unfortunately two tests (unrelated?) are now failing.
    vector::vector_tests::tests::test_field_in_layer
    vector::vector_tests::tests::test_owned_layers

will see if I can fix them

not accurate description of the problem and messy solution in the PR. Will open new issue and PR. Sorry for the noise


Conversion b/w Set and List of same/mapped object types

Hi,
The generated code is complaining about conversion between a Set and List of same object type. I suggest it should be supported OOTB.
I also tried to define 2 methods to convert, with generic types (since its just a conversion of collection type, not contents), but Selma is not able to pickup those methods and still complains.
Am i doing anything wrong?
Thanks.

Also, to convert b/w Set and List of mapped objects (conversion methods have been provided to map/convert individual objects), still its asking for methods to convert Set/Lists of those objects again!!
for ex:
Class A{
Set set1;        // note the Set here doesn't indicate the type of objects its holding, but are of type 'C'.
}
Class B {
List list1;
}
mapper method in an interface:
D c2d(C in, D out);
and vice-verse.
with mapping fields definition:   @Field({"B.list1","A.set1"})
Type of 'set1' in class A can't be indicated as its generated by another library.
Selma is now asking all combinations of methods to properly convert A to/from B in above scenario.
Please suggest, what is recommended here.
Thanks.

Hi,
the thing here again is that Lists and Sets do not share the same contract and you can loose passing from one to another. So I think this kind of conversion should be decided by the dev and not simply automate but that can be subject to discussion.
Now, please note that Selma is statically typed mapping which means it can not infer type at runtime. So for a custom mapper to work you should specify it's complete exact type signature.
So you can specify a custom mapper :
public Set mapListToSet (List in) {
// ...
}

and inside this one infer the generic type of the contained objects to call the mapping method. This can be easily done in an abstract class mapper.
I am not sure this answer to your question, if not, can you give a real sample code ?

Sorry, didn't see this in time. I did use custom mappers only.
Thank you.


G-Hub support

Is there a possibility to add G-Hub support? Currently it's only working with the normal Gaming Software but it is not shown in G-Hub.
Might this feature be added to the plugin?

Unfortunately Logitech has not released a G-Key SDK for G-Hub. You can see an overview of their SDKs here:
https://www.logitechg.com/en-us/innovation/developer-lab.html


LinkedIn changed selector

Hi, I was getting no jobs found for common authenticated searches like "MBA". It looks like LinkedIn changed the name of the container that the scraper uses to validate that the page has loaded. The old and new selector seem to be:
#old search results container
'.jobs-search-two-pane__container'

#new search results container
'.scaffold-layout__list-container'

I'm not able to make a PR, but I was able to successfully scrape after I changed line 23 of authenticated-strategy.py to
container = '.scaffold-layout__list-container'

It looks like the other important selectors are still working, it's just the initial validation one that changed.
Thanks for making this very useful tool 😄

Thank you sir!
I've just updated the container selector, let me know if it works!


[Feature]: Support for multiple column sorting in table widget

Is there an existing issue for this?

[X] I have searched the existing issues

Summary
In the table widget, we can sort using a single column, by enabling the sort for multiple column can be a good feature when we have lots of records in a table and pagination is disabled.
Why should this be worked on?
This will improve the visibility for larger dataset and ease the pain to work with these datasets. Today we autopaginate the table after some records but this will definitely be helpful once we have scrollable table widget.
Ref image from user:

i think that this is very useful.
If i can help you on this let me now!

Thanks @billyrudi
@riodeuno something we may consider after adding infinite scrolling table, @somangshu can we discuss this once for the current paginated version?

@dilippitchika  Sounds good. We need to consider this when designing the infinite scrolling table, and we can implement this in a later phase.
I am wondering if, the order in which the columns are sorted matters here? Seems like it does.

@riodeuno yes, the ideal implementation will need an order to sort. Tagging below an example of how notion does it.

t

it seem good: this will be not at designe time right? i think that the right location is where we have the filter settings.

Hey @billyrudi yes, this not at the design time but in the filter settings.


Problem when turn on TV

Describe the bug
TV won't trun off
Expected behavior
Toggle ON Tv
Screenshots
If applicable, add screenshots to help explain your problem.
Environment details:

Environment (HASSIO, Raspbian, etc): HASSio
Home Assistant version installed: 2021.9.7
Component version installed: 0.4.0
Last know working version: NB
TV model: UE43NU7025

Output of HA logs
Paste the relavant output of the HA log here.
Logger: homeassistant.components.websocket_api.http.connection
Source: custom_components/samsungtv_smart/media_player.py:967
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 19:10:47 (7 occurrences)
Last logged: 19:20:07

[140036236903424] None is not a valid PowerOnMethod
[140036236905824] None is not a valid PowerOnMethod
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 185, in handle_call_service
    await hass.services.async_call(
  File "/usr/src/homeassistant/homeassistant/core.py", line 1492, in async_call
    task.result()
  File "/usr/src/homeassistant/homeassistant/core.py", line 1527, in _execute_service
    await handler.job.target(service_call)
  File "/usr/src/homeassistant/homeassistant/components/homeassistant/__init__.py", line 121, in async_handle_turn_service
    await asyncio.gather(*tasks)
  File "/usr/src/homeassistant/homeassistant/core.py", line 1492, in async_call
    task.result()
  File "/usr/src/homeassistant/homeassistant/core.py", line 1527, in _execute_service
    await handler.job.target(service_call)
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 213, in handle_service
    await self.hass.helpers.service.entity_service_call(
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 658, in entity_service_call
    future.result()  # pop exception if have
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 843, in async_request_call
    await coro
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 695, in _handle_entity_call
    await result
  File "/config/custom_components/samsungtv_smart/media_player.py", line 1004, in async_turn_on
    await self._async_turn_on()
  File "/config/custom_components/samsungtv_smart/media_player.py", line 988, in _async_turn_on
    if not await self._async_power_on(set_art_mode):
  File "/config/custom_components/samsungtv_smart/media_player.py", line 967, in _async_power_on
    turn_on_method = PowerOnMethod(
  File "/usr/local/lib/python3.9/enum.py", line 384, in __call__
    return cls.__new__(cls, value)
  File "/usr/local/lib/python3.9/enum.py", line 702, in __new__
    raise ve_exc
ValueError: None is not a valid PowerOnMethod

I think that I found the issue, it will be fixed with next release

Ok that's great! Thank you.
I will try your fix by copy the file that you've changed in custom folder and try this.
I will tell you if fix it tomorrow.

Fix ok work great. You can push this to the next release


Don't reach into BIGNUM internals.

Instead, call BN_is_negative.

You should add a commit to add yourself as appropriate to the AUTHORS/CONTRIBUTORS files.
I'll merge this PR once this is done.

@pphaneuf Done.


Provide option for Local Server Address to bind to all local interfaces (select by default)

Do not require user to select specific local interface
Default to bind to <IP_ADDRESS> (ie. all interfaces) allowing underlying OS to handle routing

Version 17.2.1

model/project.py default_local_server changed IP from <IP_ADDRESS> to <IP_ADDRESS>
view/settings/dicom_node_dialog ensure "<IP_ADDRESS>" is provided to IP address dropdown


enhancement: missing filter for selection of CSV files

in the section to import participants click on the button "Search" to select a CSV file. Unfortunately this dialog misses a file type filter. Currently it is ".":

but it should be ".csv; *.txt" since only CSV files (or textfiles in general) are allowed. This eases to find the CSV file in folders with many files.

Hm. This dialog is created by your browser. Does anyone know how to tell the broweser that it should filter the files?

Use the accept attribute for the input field:

The value must be a comma-separated list of unique content type specifiers: A file extension starting with the STOP character (U+002E). (E.g.: ".jpg,.png,.doc")
A valid MIME type with no extensions
audio/* representing sound files HTML5
video/* representing video files HTML5
image/* representing image files HTML5

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input

@tsiegleauq Can you try this issue?

Works fine now. Many thanks!


Subset matching broken on objects implementing to_hash and include?

Subject of the issue
#1073 seems to break subset hash matching for objects implementing #to_hash and #include?. Previously these objects were converted using #to_hash on actual. Due to not converting to a Hash the check in comparing_hash_to_a_subset? will always be false and subset check won't be performed anymore (unless the objects #include? implements subset matching).
We had lots of breaking tests after the rspec-expectations patch update due to having specs on e.g. ActionDispatch::Response::Header like this:
expect(response.headers).to include 'Location' => 'http://test.host/...'

These expectations produced a nice diff when failing showing all send headers. We currently fixed it by explicitly calling #to_hash on these objects.
It would be nice if such use case could still be supported.
Your environment

Ruby version: ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
rspec-expectations version: 3.8.2

Steps to reproduce
Run the following test case:
begin
  require 'bundler/inline'
rescue LoadError => e
  $stderr.puts 'Bundler version 1.10 or later is required. Please update your Bundler'
  raise e
end

gemfile(true) do
  source 'https://rubygems.org'
  gem 'rspec-expectations', '3.8.2'
  gem 'minitest'
end

require 'rspec/matchers'

require 'minitest/autorun'

class CollectionLikeObject
  def to_hash
    {'Header' => 'Value'}
  end

  def include?(key)
    to_hash.include?(key)
  end
end

class IncludeTest < Minitest::Test
  def test_subset_match
    matcher = RSpec::Matchers::BuiltIn::Include.new('Header' => 'Value')
    actual  = CollectionLikeObject.new

    assert matcher.matches?(actual)
  end
end

Running the test case with 3.8.1 passes.
Expected behavior
I'd expect the expectations to still pass after a patch level upgrade.
Actual behavior
All expectations failed while showing a nice diff showing no difference.

I'm not sure whats the correct behaviour here, if headers were a hash, it should fail:
 { 'Location' => 'xyz'}.include?('Location' => 'xyz')
 => false

So was headers previously behaving like an array? Or is it the individual header object not working... This feels like it's a bug that should be addressed either by Rails, or by rspec-rails providing a matcher that produces the diff, not here..

Subset matching on hashes is well documented too.

Yes I'm aware, the issue is that response.headers is not a hash, its a hash like object. Previously we relied on an undocumented call to to_hash but documented that we were using include? this broke for objects that were legitimately responding to include? in a hash like way (which was documented as working) but didn't implement to_hash in a similar fashion. I'm open to a fix for your issue, provided it doesn't regress the fix for the correct behaviour.

@jgraichen Do you think you could have time to try to submit a patch?

I'm still thinking about a solution. I do understand to use #include? if available and how it has been documented.
Given the requirement to not change the behavior of using include? if available (and unless actual is Hash) there is no solution to our problem. Even using #to_hash only when expected is a Hash would break that behavior (because one might test that there is a specific hash included).
Apparently we kindly profited from that bug and the bug fix suddenly broke our workflow. ;-)
I do not see any easy patch not breaking the requirement of using include? if available. We will probably change our specs to explicitly calling #to_hash upfront or build another matcher. Personally I would call objects described by @JonRowe as breaking the contract of #to_hash, because #to_hash, like #to_str, means the object behaves like a Hash (in difference to #to_h). But that is a matter of perspective to duck typing (and documentation of the include matcher). Luckily Hashes are still handled special.
Thanks for your work!

Thanks @jgraichen for this detailed answer. I'm closing the issue for now because as you mention:

Apparently we kindly profited from that bug and the bug fix suddenly broke our workflow. ;-)

I will let @JonRowe comment if wanted.

I'm still interested in doing something with this to better support hash shaped objects

I'm pretty sure this change broke my test which uses ActionController::Parameters
expect(policy.permitted(params)).to include('foo' => 'bar')

I fixed it for now by adding a .to_h in the expect, but this worked before updating from 3.8.1 to 3.8.3.


Add support for python 3.11

This installs fine with Python 3.11 on Debian 12 here, but I'd like to see a newer requests dependency; there is no reason to explicitly pin it on the ancient 2.28.1 - this hampers the rest of the project code that we use PYHANDLE in.
Can this be changed to 'requests>=2.28.1; python_version>="3.6.0"' ? That works for us.
Thanks in advance, and nice work!

In addition to requests 2.28.1 just being old, all versions between 2.3.0 and 2.30.0 have a security vulnerability: https://requests.readthedocs.io/en/latest/community/updates/#release-history

hello @aquatix ,
I think Python 3.6 can only handle requests 2.27.1.
All the other Python versions have gotten the latest requests version 2.31.0.
Cheers,
Angelos

Hey @agelostsal those changes look great, thanks! Also thanks for not pinning the requests version to 2.31.0 but including newer ones too.

I'm also running into issues with the requests being pinned at 2.28.1 in the current version of pyhandle (1.2.1).
Happy to see that there is already work being done to make it 3.11 compatible including updating the requests requirement.
Any idea when this will be merged and a new version of pyhandle will be released?


Removing debug events

Codecov Report

Merging #127 into master will decrease coverage by 0.21%.
The diff coverage is 77.77%.

@@            Coverage Diff             @@
##           master     #127      +/-   ##
==========================================
- Coverage   95.11%   94.89%   -0.22%
==========================================
  Files          49       49
  Lines        1576     1587      +11
==========================================
+ Hits         1499     1506       +7
- Misses         77       81       +4

Impacted Files
Coverage Δ

cli-commands/commands.js
95.55% <77.77%> (-4.45%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d321fea...51db9a3. Read the comment docs.


[REQUEST]Need I load data to memory at all stages when I use pipiline engine ?

why not only read data at the node of the pipeline's first stage?

Hi there, do you mean the incoming batch data? The first and last pipeline stages require data loaders. The last stage uses it to supply labels for the loss calculation.
Totally agreed with you that it would be nicer if only the first pipeline stage needs the input data! I wanted to implement this initially and just have the first pipeline stage communicate the batch for the loss calculation.
There's no fundamental reason we can't only load on the first stage. The tricky bit is just that the shapes/formats of the incoming data are only known on the first stage and requires some additional coordination and/or restrictions on the batch format. We actually already have some utilities that could enable those communications in runtime/pipe/p2p.py, but they should be rewritten as they're slow and also use pickle which is insecure.

Hi there, do you mean the incoming batch data? The first and last pipeline stages require data loaders. The last stage uses it to supply labels for the loss calculation.
Totally agreed with you that it would be nicer if only the first pipeline stage needs the input data! I wanted to implement this initially and just have the first pipeline stage communicate the batch for the loss calculation.
There's no fundamental reason we can't only load on the first stage. The tricky bit is just that the shapes/formats of the incoming data are only known on the first stage and requires some additional coordination and/or restrictions on the batch format. We actually already have some utilities that could enable those communications in runtime/pipe/p2p.py, but they should be rewritten as they're slow and also use pickle which is insecure.

Additional,can we distribute the pipeline engine's first and last stage at different nodes，my model requires memory fewer but data needs much more

If you're training with multiple nodes, the first and last pipeline stages will be placed on different nodes.
We don't have any mechanisms for sharing memory allocations across pipeline stages. There's a TiedLayer utility for keeping model parameters in sync across stages (e.g., an embedding layer at the start/end), but both stages will ultimately have to allocate that memory. There are other pipeline parallelism approaches that instead map the reused layer to the same stage.

If you're training with multiple nodes, the first and last pipeline stages will be placed on different nodes.
We don't have any mechanisms for sharing memory allocations across pipeline stages. There's a TiedLayer utility for keeping model parameters in sync across stages (e.g., an embedding layer at the start/end), but both stages will ultimately have to allocate that memory. There are other pipeline parallelism approaches that instead map the reused layer to the same stage.

Up to now, the first and last pipeline stages are placed on same nodes.
I have 8 nodes with 64 gpus, my topo strategy is :
topo = PipeModelDataParallelTopology(
        num_pp = 8,
        num_mp = 1,
        num_dp = 8)

Hm, with that topology each pipeline stage should get its own node so that the data-parallel communications can stay intra-node. Can you share the log with the mappings? Perhaps there is something strange happening.

It distributes stages according to the topology

Hm, with that topology each pipeline stage should get its own node so that the data-parallel communications can stay intra-node. Can you share the log with the mappings? Perhaps there is something strange happening.

Firstly, global_ rank=local_ rank+8 * node_ id.
Then, it distributes the stage based on the topology , prioritizing the data dimension. so global_ rank=data_ id+8 * pipe_ id, which means that the first stage (pipe_id=0) are all on a node whose global_ rank is from 0 to 7.
I load my data to the memory with local_rank 0, but it still costs 8x memory of the data during trainning:
def read_data(local_rank,path):
    torch.distributed.barrier()
    if local_rank !=0:
        torch.distributed.barrier()
    data = pyarrow.parquet.read_table(path)
    if local_rank ==0:
         torch.distributed.barrier()
    return data

Firstly, global_ rank=local_ rank+8 * node_ id. PipelineEngine distributes stages according to the topology , while the PipeModelDataParallelTopology's axis is [pipe,model,data]. It prioritizes the data dimension. so global_ rank=data_ id+8 * pipe_ id, which means that the first stage (pipe_id=0) are all on the first node whose global_ rank is from 0 to 7.
Now I load my data to the memory by parquet, it costs 8x memory of the data on all nodes during trainning:
data = pyarrow.parquet.read_table(path)

Maybe I should pass aixs=[data,model,pipe] to ProcessTopology, and use some fake dataset in the ranks who is neither the first stage nor the last stage to reduce the usage of memory to 2x .

Firstly, global_ rank=local_ rank+8 * node_ id.
PipelineEngine distributes stages according to the topology , while the PipeModelDataParallelTopology's axis is [pipe,model,data]. It prioritizes the data dimension. so global_ rank=data_ id+8 * pipe_ id, which means that the first stage (pipe_id=0) are all on the first node whose global_ rank is from 0 to 7.
Now I load my data to the memory by parquet, it costs 8x memory of the data on all nodes during trainning:
data = pyarrow.parquet.read_table(path)
Maybe I should pass aixs=[data,model,pipe] to ProcessTopology, and use some fake dataset in the ranks who is neither the first stage nor the last stage to reduce the usage of memory to 2x .

Hm, with that topology each pipeline stage should get its own node so that the data-parallel communications can stay intra-node. Can you share the log with the mappings? Perhaps there is something strange happening.

Firstly, global_ rank=local_ rank+8 * node_ id.
PipelineEngine distributes stages according to the topology , while the PipeModelDataParallelTopology's axis is [pipe,model,data]. It prioritizes the data dimension. so global_ rank=data_ id+8 * pipe_ id, which means that the first stage (pipe_id=0) are all on the first node whose global_ rank is from 0 to 7.
Now I load my data to the memory by parquet, it costs 8x memory of the data on all nodes during trainning:
data = pyarrow.parquet.read_table(path)

Maybe I should pass aixs=[data,model,pipe] to ProcessTopology, and use some fake dataset in the ranks who is neither the first stage nor the last stage to reduce the usage of memory to 2x .

Hm, with that topology each pipeline stage should get its own node so that the data-parallel communications can stay intra-node. Can you share the log with the mappings? Perhaps there is something strange happening.

Can I pass label from first stage to last stage which may reduce the usage of memory to 1x .

Hm, with that topology each pipeline stage should get its own node so that the data-parallel communications can stay intra-node. Can you share the log with the mappings? Perhaps there is something strange happening.

Can I pass label from first stage to last stage which may reduce the usage of memory to 1x ?

@janelu9 , not out of the box but that should be possible with some modifications to the pipeline schedule! I think it would make a great improvement.


Add IsSuccessStatusCode & EnsureSuccessStatusCode to ContainerResponse/DatabaseResponse

Is your feature request related to a problem? Please describe.
We need to implement our own method to check if the StatusCode is valid or to raise an Exception when required.
Describe the solution you'd like
Provide the same implementation as ResponseMessage which provides access to :
public virtual bool IsSuccessStatusCode { get; }
public virtual ResponseMessage EnsureSuccessStatusCode();

Describe alternatives you've considered
I create my own methods to manage these behaviors because the following methods are not public :
internal static bool IsSuccess(this HttpStatusCode httpStatusCode)
{
        return ((int)httpStatusCode >= 200) && ((int)httpStatusCode <= 299);
}

throw new CosmosException(
        response.ErrorMessage,
        response.StatusCode,
        (int)response.StatusCode,
        response.Headers.ActivityId,
        response.Headers.RequestCharge);

@YohanSciubukgian I think there is a misunderstanding on the exception logic. For all typed responses like ContainerResponse/DatabaseResponse/ItemResponse<T> the SDK will throw a CosmosException if the status code is a failure. The status code on those objects are used for where you need to tell if an item was created or read when calling something like CreateContainerIfNotExists.
APIs that return a ResponseMessage like CreateItemStreamAsync will not throw service exceptions like 404 Not found. It can still throw argument exceptions or task cancelled exceptions. The ResponseMessage contains the EnsureSuccessStatusCode method.

Thanks for details, I didn't know that the Exception logic was binded to specific types (ContainerResponse/DatabaseResponse/ItemResponse<T>). I though it was only related to HttpClient calls that could be done by the SDK.
Is there any documentation (MS doc or on the repository) that could give me more details about which kind of Exception can be raised for each method ?
eg: the HttpClient has code documentation like this :
// Exceptions:
//   T:System.ArgumentNullException:
//     The requestUri is null.
//
//   T:System.Net.Http.HttpRequestException:
//     The request failed due to an underlying issue such as network connectivity, DNS
//     failure, server certificate validation or timeout.

Some methods doesn't have any override which implement ResponseMessage return type like this CreateContainerIfNotExistsAsync method.

The SDK should always throw a CosmosException for service issues, and should be documented on the methods. It's still possible to get an OperationCancelledException or ArgumentException.

The reason behind the difference is performance. The stream APIs are designed for advanced scenarios where performance is critical. Throwing exceptions causes a lot of overhead. To avoid that overhead it return a response instead of throwing an exception. This goes against dot net guidelines and what most users except. That is why the typed APIs still throw exceptions to make it more inline what developers are expecting.

I like the way it's currently implemented, the StreamAPIs let me customize usages for performance reasons & typed API are used for most common scenarios with Exception handling.
That makes sense to consider different implementation for Management methods.
However, as CreateDatabaseStreamAsync & CreateContainerStreamAsync exist, could it have values to implement Stream APIs for CreateDatabaseIfNotExistsAsync & CreateContainerIfNotExistsAsync ?

The CreateDatabaseIfNotExistsAsync and CreateContainerIfNotExistsAsync methods do use the stream APIs to handle the logic.

Thanks you for all those details.
I have checked the source code & I now have a better understanding on how you are handling all these cases.


Feature:  Want to add toast notification for any event

Is your feature request related to a problem? Please describe.
I want to add a toast notification for any event like when user likes the profile so we can show toast or saves the blog
https://github.com/user-attachments/assets/a4087f4f-2a90-415b-892f-f9bd63a06c71

cany see your video

https://github.com/user-attachments/assets/8df159f9-5d67-4099-a3fb-8c0310a44490
@sanjay-kv sir this type of toast

video not available

@sanjay-kv sir their may be any problem but i am talking about this type of toast

You can do it

Ok sir


Remove custom CI jobs in faviour of Unified CI

Remove test, test with race, generate checker and lint in favour of standard unified CI jobs to only contain fuzzing.

OK i think we are missing this

I have manually enabled merge_group trigger until it is set at upstream workflows.


Luminosity ratio for "Reset view" link text is 3.7:1 which is less than 4.5:1

A team member will update this issue with more details soon.

This is your friendly Microsoft Issue Bot. I created this issue automatically as requested by a team member.


Support multiple return values

Support for functions returning multiple values has many implications in terms of potential language features:

Being able to index intro lists using lists (list[1 .. 5], where .. returns the range from 1 to 5)
Comprehensions

Optionals (in V, in Rust, etc.)

Tentatively closing, as this behavior is largely covered by the new tuple types. We can probably reopen discussion for something like a splat operator at a later date.

I think splat operator could be "easily faked" with macros and a compile-time list or key-value map. So I think it's very low priority.


chore: updates for converting the api command

🎉 Thanks for submitting a pull request! 🎉
Summary
This adds some changes to converting the api command into using the new NetlifyLog. It is part of a bigger piece of work that can be seen in https://github.com/netlify/cli/pull/6293. This is also the reason why this PR is merged into the branch CP-101/design-system-clack-implementation. It's a way in which I'm trying to keep possible failing tests, but also the review work to a minimum amount of work for those who are reviewing.

Before

After

For us to review and ship your PR efficiently, please perform the following steps:

[ ] Open a bug/issue before writing your code 🧑💻. This ensures we can discuss the changes and get feedback from everyone that should be involved.
[ ] Read the contribution guidelines 📖. This ensures your code follows our style guide and
passes our tests.
[ ] Update or add tests (if any source code was changed or added) 🧪
[ ] Update or add documentation (if features were changed or added) 📝
[ ] Make sure the status checks below are successful ✅

A picture of a cute animal (not mandatory, but encouraged)

@khendrikse I understand. When there isn't a message will it end with the last line of the table above it? I think that is fine.

@khendrikse I understand. When there isn't a message will it end with the last line of the table above it? I think that is fine.

Yeah that's the problem @samkap, right now we don't have easy logic that will allow that.
We could default to 'Done' or 'Finished' if someone does not pass a message?


[SPARK-24762][SQL] Enable Option of Product encoders

What changes were proposed in this pull request?
Encoders has a limitation that we can't construct encoders for Option of Product at top-level, because in SparkSQL entire row can't be null.
However for some use cases such as Aggregator, it should be able to construct encoders for Option of Product at non top-level.
For top-level Option of Product, we can encode it as a single struct column.
How was this patch tested?
Added test.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/759/
Test PASSed.

Merged build finished. Test PASSed.

Test build #92729 has started for PR 21732 at commit e1b5dee.

Test build #92729 has finished for PR 21732 at commit e1b5dee.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92729/
Test FAILed.

retest this please.

Test build #92739 has started for PR 21732 at commit e1b5dee.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/765/
Test PASSed.

Test build #92739 has finished for PR 21732 at commit e1b5dee.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92739/
Test FAILed.

retest this please.

Test build #92748 has started for PR 21732 at commit e1b5dee.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/772/
Test PASSed.

Test build #92748 has finished for PR 21732 at commit e1b5dee.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92748/
Test PASSed.

I'm wondering should we add encoders of Option of Product into object Encoders.

cc @cloud-fan @hvanhovell

how about we treat the top level Option[Product] as a single struct column? then we can git rid of this limitation entirely.

It sounds like much more behavior changing?

yes it is, but it makes the encoder framework more consistent. And making a failure case into runnable is a safe behavior change.

Non top-level and top-level encoders for Option[Product] have a little difference.
As you said, top-level Option[Product] can only be encoded as a single struct column.
For non top-level one, we can't apply the same change because it is already a struct column. We don't want to change current behavior of it from a struct column to a struct column of a struct column.
This means that we can remove the limitation of top-level Option[Product], but it doesn't help too much Aggregator issue here. We still need to use non top-level Option[Product] encoders for Aggregator case.
@cloud-fan Do you want to incorporate top-level Option[Product] encoders into this PR too? Or let's create another Jira & PR for it? Thanks.

ping @cloud-fan

At the end of encoder creation? You mean at the end of calling ExpressionEncoder.apply()? But it is used both for top-level encoder e.g., Dataset[Option[Product]] and  non top-level encoder e.g., Aggregator's encoder. If we flatten it, doesn't it mean for top-level, it is encoded as a row, not a struct column?

ping @cloud-fan @hvanhovell

Again, can we always support Option[Product] with some special handling for top-level encoder expression?

This PR is just a special handling for Option[Product] in Aggregator, I think we don't need it when we have the more general solution, right?

Test build #93762 has started for PR 21732 at commit 6e32abe.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1468/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1469/
Test PASSed.

Test build #93763 has started for PR 21732 at commit 2b73b33.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1470/
Test PASSed.

Test build #93764 has started for PR 21732 at commit b6c6e9f.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1471/
Test PASSed.

Merged build finished. Test PASSed.

Test build #93765 has started for PR 21732 at commit 4f5628d.

Test build #93763 has finished for PR 21732 at commit 2b73b33.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  * For example, we build an encoder forcase class Data(a: Int, b: String) and the real type

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93763/
Test FAILed.

Test build #93762 has finished for PR 21732 at commit 6e32abe.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  * For example, we build an encoder forcase class Data(a: Int, b: String) and the real type

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93762/
Test FAILed.

Test build #93764 has finished for PR 21732 at commit b6c6e9f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93764/
Test FAILed.

Test build #93765 has finished for PR 21732 at commit 4f5628d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  * For example, we build an encoder forcase class Data(a: Int, b: String) and the real type

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93765/
Test FAILed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1473/
Test PASSed.

Merged build finished. Test PASSed.

Test build #93768 has started for PR 21732 at commit 50f6430.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1475/
Test PASSed.

Test build #93770 has started for PR 21732 at commit 80506f4.

Test build #93770 has finished for PR 21732 at commit 80506f4.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  * For example, we build an encoder forcase class Data(a: Int, b: String) and the real type

Test build #93768 has finished for PR 21732 at commit 50f6430.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  * For example, we build an encoder forcase class Data(a: Int, b: String) and the real type

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93770/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93768/
Test FAILed.

retest this please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1489/
Test PASSed.

Test build #93788 has started for PR 21732 at commit 80506f4.

Test build #93788 has finished for PR 21732 at commit 80506f4.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  * For example, we build an encoder forcase class Data(a: Int, b: String) and the real type

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93788/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2018/
Test PASSed.

Merged build finished. Test PASSed.

Test build #94527 has finished for PR 21732 at commit 80506f4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  * For example, we build an encoder forcase class Data(a: Int, b: String) and the real type

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94527/
Test FAILed.

retest this please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2030/
Test PASSed.

Test build #94542 has started for PR 21732 at commit 80506f4.

Test build #94542 has finished for PR 21732 at commit 80506f4.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  * For example, we build an encoder forcase class Data(a: Int, b: String) and the real type

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94542/
Test FAILed.

retest this please

Test build #94554 has started for PR 21732 at commit 80506f4.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2039/
Test PASSed.

Test build #94554 has finished for PR 21732 at commit 80506f4.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  * For example, we build an encoder forcase class Data(a: Int, b: String) and the real type

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94554/
Test FAILed.

retest this please.

retest this please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2050/
Test PASSed.

Test build #94573 has started for PR 21732 at commit 80506f4.

Test build #94573 has finished for PR 21732 at commit 80506f4.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  * For example, we build an encoder forcase class Data(a: Int, b: String) and the real type

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94573/
Test FAILed.

retest this please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2067/
Test PASSed.

Test build #94596 has started for PR 21732 at commit 80506f4.

Test build #94596 has finished for PR 21732 at commit 80506f4.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  * For example, we build an encoder forcase class Data(a: Int, b: String) and the real type

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94596/
Test FAILed.

retest this please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2080/
Test PASSed.

Test build #94617 has started for PR 21732 at commit 80506f4.

Test build #94617 has finished for PR 21732 at commit 80506f4.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  * For example, we build an encoder forcase class Data(a: Int, b: String) and the real type

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94617/
Test PASSed.

Can you give a description of the new encoding rules regarding Option?
e.g.
Option[Int] in normal encoder -> a nullable int column
Option[Int] in agg encoder -> a nullable int column
Option[Int] in tupled encoder -> a nullable int column

Option[Product] in normal encoder => a nullable struct column
Option[Product] in agg encoder => ?
...

Yeah. The encoding rules regarding Option are:

For non Product types, no change is made.

Option[Int] in normal encoder -> a nullable int column
Option[Int] in agg encoder -> a nullable int column
Option[Int] in tupled encoder -> a nullable int column

E.g.,
val df = Seq(Some(1), Some(2), None).toDF()
df.printSchema

root
 |-- value: integer (nullable = true)

val df2 = Seq((1, Some(1)), (2, Some(2)), (3, None)).toDF()
df2.printSchema

root
 |-- _1: integer (nullable = false)
 |-- _2: integer (nullable = true)

For Product types, we don't support it at top level and aggregator for now. After this change, we can support it in normal encoder and aggregator.

Option[Product] in normal encoder => a nullable struct column

E.g.,
val df3 = Seq(Some(1 -> "a"), Some(2 -> "b"), None).toDF()
df3.printSchema

root
 |-- value: struct (nullable = true)
 |    |-- _1: integer (nullable = false)
 |    |-- _2: string (nullable = true)

Option[Product] in agg encoder => a nullable struct column // topLevel = false
Option[Product] in agg encoder => a nullable struct column nested in a struct column // topLevel = true

val df4 = Seq(
    OptionBooleanIntData("bob", Some((true, 1))),
    OptionBooleanIntData("bob", Some((false, 2))),
    OptionBooleanIntData("bob", None)).toDF()

val group = df4
    .groupBy("name")
    .agg(OptionBooleanIntAggregator("isGood").toColumn.alias("isGood"))
group.printSchema

// topLevel = false
root
 |-- name: string (nullable = true)
 |-- isGood: struct (nullable = true)
 |    |-- _1: boolean (nullable = false)
 |    |-- _2: integer (nullable = false)

// topLevel = true
root
 |-- name: string (nullable = true)
 |-- isGood: struct (nullable = true)
 |    |-- value: struct (nullable = true)
 |    |    |-- _1: boolean (nullable = false)
 |    |    |-- _2: integer (nullable = false)

For Option[Product] in tupled encoder, no change is made too:

val df5 = Seq((1, Some(1 -> "a")), (2, Some(2 -> "b")), (3, None)).toDF()
df5.printSchema

root
 |-- _1: integer (nullable = false)
 |-- _2: struct (nullable = true)
 |    |-- _1: integer (nullable = false)
 |    |-- _2: string (nullable = true)

From the above, I think the aggregator encoder for Option[Product] might be a bit tricky to use for users, since they might need to know the difference between topLevel = true and topLevel = false. But I don't have good idea for now on this. @cloud-fan WDYT?

if we can totally hide the topLevel parameter from users, it should be fine. Generally I think the behavior is consistent, now Option[Product] is always a struct type column. But we need to improve the implementation to hide the topLevel.

The only tricky thing is, Product is handled specially in the top level, being flattened into multiple columns.

if we can totally hide the topLevel parameter from users, it should be fine. Generally I think the behavior is consistent, now Option[Product] is always a struct type column. But we need to improve the implementation to hide the topLevel.

Sounds good. I'm wondering if there is good way to hide topLevel.

Test build #95191 has started for PR 21732 at commit ed3d5cb.

@cloud-fan I made attempt to remove topLevel parameter. The approach is to flatten serializers and deserialzer at TypedAggregateExpression. So users are not aware of difference when using expression encoder.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2516/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2517/
Test PASSed.

Test build #95192 has started for PR 21732 at commit 9fc3f61.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2518/
Test PASSed.

Test build #95193 has started for PR 21732 at commit 5f95bd0.

Test build #95191 has finished for PR 21732 at commit ed3d5cb.

This patch fails due to an unknown error code, -9.
This patch does not merge cleanly.
This patch adds no public classes.

Test build #95192 has finished for PR 21732 at commit 9fc3f61.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #95193 has finished for PR 21732 at commit 5f95bd0.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95191/
Test FAILed.

Merged build finished. Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95192/
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95193/
Test FAILed.

retest this please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2520/
Test PASSed.

Test build #95198 has started for PR 21732 at commit 5f95bd0.

Test build #95198 has finished for PR 21732 at commit 5f95bd0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95198/
Test PASSed.

Test build #95212 has started for PR 21732 at commit a4f0405.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2531/
Test PASSed.

Test build #95212 has finished for PR 21732 at commit a4f0405.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95212/
Test FAILed.

retest this please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2540/
Test PASSed.

Test build #95226 has started for PR 21732 at commit a4f0405.

Test build #95226 has finished for PR 21732 at commit a4f0405.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95226/
Test PASSed.

Merged build finished. Test PASSed.

Test build #95240 has started for PR 21732 at commit c1f798f.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2548/
Test PASSed.

Test build #95240 has finished for PR 21732 at commit c1f798f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95240/
Test PASSed.

retest this please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2665/
Test PASSed.

Test build #95413 has started for PR 21732 at commit c1f798f.

Test build #95413 has finished for PR 21732 at commit c1f798f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95413/
Test PASSed.

The only tricky thing is, Product is handled specially in the top level, being flattened into multiple columns.

@cloud-fan Compared with Option of Product which is not supported before, the encoding of Product is current behavior. I think we don't need to change it so far. WDYT?

Test build #97027 has started for PR 21732 at commit 80e11d2.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3730/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3731/
Test PASSed.

Test build #97029 has started for PR 21732 at commit 23be39a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3732/
Test PASSed.

Test build #97030 has started for PR 21732 at commit 0f029b0.

Test build #97027 has finished for PR 21732 at commit 80e11d2.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #97029 has finished for PR 21732 at commit 23be39a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #97030 has finished for PR 21732 at commit 0f029b0.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97029/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97030/
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97027/
Test FAILed.

Merged build finished. Test FAILed.

retest this please

Test build #97038 has started for PR 21732 at commit 0f029b0.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3737/
Test PASSed.

Test build #97038 has finished for PR 21732 at commit 0f029b0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97038/
Test FAILed.

retest this please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3740/
Test PASSed.

Test build #97041 has started for PR 21732 at commit 0f029b0.

Test build #97041 has finished for PR 21732 at commit 0f029b0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97041/
Test PASSed.

Since we have to handle some special cases, can you list them in the PR description?
For Product, normally it should be a struct type column, the special case I can think of is: 1) at root level, flatten it into multiple columns, and the Product can't be null.
For Option[Product], normally it should also be a struct type column, seems there is no special case.

For Product, normally it should be a struct type column, the special case I can think of is: 1) at root level, flatten it into multiple columns, and the Product can't be null.
For Option[Product], normally it should also be a struct type column, seems there is no special case.

This doesn't make any change to Product.
For Product, 1. at root level, the schema is all fields are flatten it into multiple columns. The Product can't be null.
val df = Seq((1 -> "a"), (2 -> "b")).toDF()
df.printSchema()

root
 |-- _1: integer (nullable = false)
 |-- _2: string (nullable = true)

At non-root level, Product is a struct type column.

val df = Seq((1, (1 -> "a")), (2, (2 -> "b")), (3, null)).toDF()
df.printSchema()

root
 |-- _1: integer (nullable = false)
 |-- _2: struct (nullable = true)
 |    |-- _1: integer (nullable = false)
 |    |-- _2: string (nullable = true)

For Option[Product], I've posted some description in the PR description.

This doesn't make any change to Product.

yea I know, but we should put things together to evaluate a design.
After reading your description, Option[Product] is the same as Product except at top level?

Option[Product] is the same as Product except at top level?

Yeah, at top level, Option[Product] was not supported. After this change, it is a struct type column.

Thanks. I've updated the PR description to summarize encoding of Product and Option[Product].

Rebased this after refactoring ExpressionEncoder.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4557/
Test PASSed.

Merged build finished. Test PASSed.

Test build #98131 has started for PR 21732 at commit 79d10c1.

Test build #98131 has finished for PR 21732 at commit 79d10c1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98131/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4561/
Test PASSed.

Test build #98146 has started for PR 21732 at commit fec1cac.

Test build #98146 has finished for PR 21732 at commit fec1cac.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98146/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4747/
Test PASSed.

Test build #98440 has started for PR 21732 at commit 3956cdd.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4751/
Test PASSed.

Test build #98445 has started for PR 21732 at commit 8304de8.

Test build #98445 has finished for PR 21732 at commit 8304de8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

I think this is close, can you answer https://github.com/apache/spark/pull/21732/files#r228782670 ?

Test build #98652 has started for PR 21732 at commit 0d79fc4.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4888/
Test PASSed.

Test build #98652 has finished for PR 21732 at commit 0d79fc4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98652/
Test PASSed.

Test build #98672 has finished for PR 21732 at commit 2d2057b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98672/
Test PASSed.

LGTM except a few comments, good job!

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5185/
Test PASSed.

Test build #99063 has started for PR 21732 at commit 4e4718a.

Test build #99063 has finished for PR 21732 at commit 4e4718a.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99063/
Test FAILed.

retest this please.

Test build #99090 has started for PR 21732 at commit 4e4718a.


[RSDK-5872] MQL Query BSON

Convert MQL queries to BSON rather than strings

@dmhilly I think we rip the bandaid and make this a breaking change directly -- the usage is probably minimal at this point since we only released it yesterday


Documentation

It would be great to have a minimal set of docs. I think we can get a lot of inspiration from the PyIceberg project where we use mkdocs: https://github.com/apache/iceberg-python/tree/main/mkdocs
We can copy the structure and the CI to deploy it.
With the CNAME file we can tell the ASF to set up a DNS entry to the docs. We could do https://rs.iceberg.apache.org/ or https://rust.iceberg.apache.org/, but I'll leave that up to the project.

The name https://rust.iceberg.apache.org/ seems appropriate given that our repository is named iceberg-rust.

cc @Xuanwo

Just curious, what would be the relationship with docs of crates on docs.rs?

From the ASF perspective, anything outside of apache.org does not exist. We should host our guide and API documentation on rust.iceberg.apache.org as the single source of truth. The documents on docs.rs are merely for the convenience of Rust users.

I took a look at py.iceerg.apache.org,  it seems that the API and Code Reference sections should leave to docs.rs, while other parts should remain in website.

I took a look at py.iceerg.apache.org, it seems that the API and Code Reference sections should leave to docs.rs, while other parts should remain in website.

I'm not super familiar with the Rust ecosystem, but I don't see any issues there. For examples, how to contribute, how to release etc it would be best to keep that on the Apache side. It would also be great if we use mkdocs to make it easier to potentially merge the languages later on.

I'm not super familiar with the Rust ecosystem, but I don't see any issues with having that on docs.rs.

Rust ecosystem has centralized online docs, here is just an example:
https://docs.rs/opendal/latest/opendal/

I'm fine with not keeping a main branch for the API documentation. We can begin with the simplest approach and expand it later.

cc @Xuanwo Are you working on this? I think we are ok to ship the first release now, all blockers has been resolved.

cc @Xuanwo Are you working on this? I think we are ok to ship the first release now, all blockers has been resolved.

I expect to finish it in next week.


install rtmp-module with nginx installed

i use centos. I install nginx earlier by the command yum install nginx. So how to add modules RTMP-nginx

install  by sources.

i have install by sources nginx download but no success. Not found director
rtmp
On Sat, Nov 7, 2015 at 8:56 AM, 曹超<EMAIL_ADDRESS>wrote:

install by sources.
—
Reply to this email directly or view it on GitHub
https://github.com/arut/nginx-rtmp-module/issues/720#issuecomment-154597287
.

let me see your configure command and error output.

Here
[root@ongmap ~]# cd /usr/build
[root@ongmap build]# ls
nginx-1.9.6  nginx-1.9.6.zip  nginx-rtmp-module
[root@ongmap build]# cd nginx-1.9.6
[root@ongmap nginx-1.9.6]# /configure
--add-module=/usr/build/nginx-rtmp-module --with-http_ssl_module
-bash: /configure: No such file or directory
[root@ongmap nginx-1.9.6]# make
make: *** No targets specified and no makefile found.  Stop.
[root@ongmap nginx-1.9.6]# make install
On Sat, Nov 7, 2015 at 9:33 AM, 曹超<EMAIL_ADDRESS>wrote:

let me see your configure command and error output.
—
Reply to this email directly or view it on GitHub
https://github.com/arut/nginx-rtmp-module/issues/720#issuecomment-154601005
.

./configure , not /configure

@minhtamwebmaster  the tips have giving the reason of failure

nginx: [emerg] unknown directive "rtmp" in /etc/nginx/nginx.conf:35
nginx: configuration file /etc/nginx/nginx.conf test failed
nginx.conf
user  nginx;
worker_processes  1;
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
events {
worker_connections  1024;
}
http {
include       /etc/nginx/mime.types;
default_type  application/octet-stream;
log_format  main  '$remote_a

On Mon, Nov 9, 2015 at 10:42 AM, joke<EMAIL_ADDRESS>wrote:

@minhtamwebmaster https://github.com/minhtamwebmaster the tips have
giving the reason of failure
—
Reply to this email directly or view it on GitHub
https://github.com/arut/nginx-rtmp-module/issues/720#issuecomment-154910896
.

i use nginx version 1.9.6 error and i use nginx 1.9.2 not error


Improve KeyWallet collateral selection and setting logic

An attempt to improve collateral selection logic for KeyWallet. The idea is to utilize the new Babbage collateral return feature, which will allow us to 1: consider utxos with non-ADA assets as collateral input and 2: not to spend the entire collateral value in case of script failure.
Currently it's not possible to test the functionality introduced in this PR, since we need to update the Haskell Server to support the new Babbage fields when estimating fees (blocked by https://github.com/Plutonomicon/cardano-transaction-lib/issues/645). @ngua
Update:
I implemented an algorithm that attempts to select the most optimal combination of utxos to use as collateral by constructing all possible combinations of utxos and then applying the following considerations:

There is a protocol parameter for the maximum number of collateral inputs (maxCollateralInputs).
We would like collateral inputs to have a total of at least 5 Ada (minRequiredCollateral).
We prefer the utxo combinations that require the lowest utxo min ada value for the corresponding collateral output. Thus, we try to keep totalCollateral high, given that it is calculated as sum_value(collateralInputs) - value(collateralOutput).
Also, if two utxo combinations correspond to return outputs with the same utxo min ada value, we prefer the one with fewer inputs.

https://github.com/Plutonomicon/cardano-transaction-lib/blob/0947f662e61d18ebd2213cedf9b15d75c2de394b/src/BalanceTx/Collateral.purs#L175-L192
Pre-review checklist

[x] All code has been formatted using our config (make format for Purescript, nixpkgs-fmt for Nix)
[x] All Purescript imports are explicit, including constructors
[x] All existing examples have been run locally against a fully-synced testnet node
[x] The integration and unit tests have been run (npm run test) locally
[x] The changelog has been updated under the ## Unreleased header, using the appropriate sub-headings (### Added, ### Removed, ### Fixed)

@klntsky https://github.com/Plutonomicon/cardano-transaction-lib/pull/845