Add labels to chat list icons

cc @stitesExpensify
Details

Added accessibility labels for the icons in the LHN. Note: I only was able to test this on web because testing screenreader stuff on emulators is incredibly difficult. Any suggestions would be appreciated!
Fixed Issues

$ https://github.com/Expensify/App/issues/5671
Tests

I ran through the page using the tab key with VoiceOver on on web and desktop, and it correctly calls out the icons with no other changes to the interface.
I have not been able to test screen readers on mobile devices as this seems to be close to impossible without physical devices. These may have to wait until production for QA with actual devices.

[x] Verify that no errors appear in the JS console

PR Review Checklist

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] iOS / native
[x] Android / native
[x] iOS / Safari
[x] Android / Chrome
[x] MacOS / Chrome
[x] MacOS / Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product was added in all src/languages/* files
[x] I verified any copy / text that was added to the app is correct English and approved by marketing by tagging the marketing team on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

PR Reviewer Checklist
The reviewer will copy/paste it into a new comment and complete it after the author checklist is completed

[ ] I have verified the author checklist is complete (all boxes are checked off).
[ ] I verified the correct issue is linked in the ### Fixed Issues section above
[ ] I verified testing steps are clear and they cover the changes made in this PR

[ ] I verified the steps for local testing are in the Tests section
[ ] I verified the steps for Staging and/or Production testing are in the QA steps section
[ ] I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[ ] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[ ] I checked that screenshots or videos are included for tests on all platforms
[ ] I verified tests pass on all platforms & I tested again on:

[ ] iOS / native
[ ] Android / native
[ ] iOS / Safari
[ ] Android / Chrome
[ ] MacOS / Chrome
[ ] MacOS / Desktop

[ ] If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
[ ] I verified proper code patterns were followed (see Reviewing the code)

[ ] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
[ ] I verified that comments were added to code that is not self explanatory
[ ] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[ ] I verified any copy / text shown in the product was added in all src/languages/* files
[ ] I verified any copy / text that was added to the app is correct English and approved by marketing by tagging the marketing team on the original GH to get the correct copy.
[ ] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[ ] I verified the JSDocs style guidelines (in STYLE.md) were followed

[ ] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[ ] I verified that this PR follows the guidelines as stated in the Review Guidelines
[ ] I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
[ ] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[ ] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[ ] If a new component is created I verified that:

[ ] A similar component doesn't exist in the codebase
[ ] All props are defined accurately and each prop has a /** comment above it */
[ ] The file is named correctly
[ ] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[ ] The only data being stored in the state is data necessary for rendering and nothing else
[ ] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[ ] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[ ] All JSX used for rendering exists in the render method
[ ] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[ ] If a new CSS style is added I verified that:

[ ] A similar style doesn't already exist
[ ] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[ ] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[ ] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[ ] I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

QA Steps

Pin a chat to the top of the list
Add a draft message in that chat, and switch to another chat
Confirm there are no visible changes from current behavior
Turn on a screen reader and navigate through the page (using the tab key on desktop)
Confirm that "Has draft message" and "Chat pinned" are read to the user.

[ ] Verify that no errors appear in the JS console

Screenshots

Should be no visible changes (although in Desktop and Web I included the VoiceOver window showing the updated reading).
Web

Mobile Web - Chrome

Mobile Web - Safari

Desktop

iOS

Android

I have read the CLA Document and I hereby sign the CLA

@rushatgabhane Oh amazing, thank you! It's not quite ready - waiting for final copy - but I will let you know as soon as it's good to go.

Thanks for your patience! This is ready to go now

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

🚀 Deployed to staging by @grgia in version: 1.2.19-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by @chiragsalian in version: 1.2.19-2 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


kube-aws: no kms service in China region

The kms servcie is not available in AWS China region.  Can I skip the KMS key creation step?

KMS is used to encrypt/decrypt all TLS assets so they can go into the userdata securely, I would not recommend you to remove this.
As a workaround, you may set an KMS key in another region and replace {{.Region}} [1][2] with the region name where the KMS key was created.
[1] https://github.com/coreos/coreos-kubernetes/blob/7075232e47cc4475bb4f0370c5ccd8c5e36f793e/multi-node/aws/pkg/config/templates/cloud-config-controller#L179
[2] https://github.com/coreos/coreos-kubernetes/blob/7075232e47cc4475bb4f0370c5ccd8c5e36f793e/multi-node/aws/pkg/config/templates/cloud-config-worker#L111

Thanks for the workaround. I will try it today. I need to verify is whether it is possible to call KMS in another region from China region, since the accounts are also separated.

This does work if I provide AWS credentials for another region where KMS key was created. This means I have to live with embed AWS credential in userdata if there is no better solutions.


[Docker] PHP GRPC, Protobuf and Msgpack extensions should be available in base image by default

These are common extensions required by Temporal, KV, and GRPC plugins. We should enable them by default.

@rauanmayemir It's a very popular serialization format. Like protobuf or JSON. You can take a look at: https://msgpack.org/index.html

@rauanmayemir It's a very popular serialization format. Like protobuf or JSON. You can take a look at: https://msgpack.org/index.html

It’s alternative format for new RPC codes. It needed to properly pass binary data. It’s optional but provide performance benefits.

It’s alternative format for new RPC codes. It needed to properly pass binary data. It’s optional but provide performance benefits.


excellent tutorial

Thanks for creating the python tutorial

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 2103a566-905f-f3a1-08b6-a07d076398ad
Version Independent ID: 926343f9-0e59-1991-473c-f604ab369eb5
Content: Python in Visual Studio tutorial step 6, work with Git
Content Source: docs/python/tutorial-working-with-python-in-visual-studio-step-06-working-with-git.md
Product: visual-studio-windows
Technology: vs-python
GitHub Login: @cwebster-99
Microsoft Alias: cowebster

@PaulArvada -- Paul, you're welcome!


Wrong tip orientation when using put_start_and_end_on on Arrow in a ThreeDScene

Description of bug / unexpected behavior
When using the put_start_and_end_on on a Arrow object in a ThreeDScene, the arrow tip does not rotate correctly.
How to reproduce the issue
class BadArrowOrientation(ThreeDScene):
    def construct(self):
        self.set_camera_orientation(
            phi=70 * DEGREES,
            theta=50 * DEGREES
        )
        axes = ThreeDAxes()

        vec = Arrow(
            start=(0, 2, 3),
            end=(0, 5, 1.5),
            max_tip_length_to_length_ratio=0.5,
            buff=0
        )

        self.add(axes, vec)
        self.wait()

        self.play(
            vec.animate.put_start_and_end_on(
                (0, 3, 2.5),
                (0, -2, 2.5)
            )
        )
        self.wait()

Additional media files

Logs
Terminal output

Manim Community v0.10.0

[09/28/21 20:03:27] DEBUG    Animation with empty mobject      animation.py:149
                    INFO     Caching disabled.            cairo_renderer.py:100
                    DEBUG    List of the first few        cairo_renderer.py:115
                             animation hashes of the
                             scene: ['uncached_00000']
[09/28/21 20:03:29] INFO     Animation 0 : Partial     scene_file_writer.py:422
                             movie file written in 'fakepath
                             \1080p60\partial_movie_files\Test\uncac
                             hed_00000.mp4'
                    INFO     Caching disabled.            cairo_renderer.py:100
                    DEBUG    List of the first few        cairo_renderer.py:115
                             animation hashes of the
                             scene: ['uncached_00000',
                             'uncached_00001']
[09/28/21 20:03:30] INFO     Animation 1 : Partial     scene_file_writer.py:422
                             movie file written in 'fakepath
                             \1080p60\partial_movie_files\Test\uncac
                             hed_00001.mp4'
                    DEBUG    Animation with empty mobject      animation.py:149
                    INFO     Caching disabled.            cairo_renderer.py:100
                    DEBUG    List of the first few        cairo_renderer.py:115
                             animation hashes of the
                             scene: ['uncached_00000',
                             'uncached_00001',
                             'uncached_00002']
[09/28/21 20:03:32] INFO     Animation 2 : Partial     scene_file_writer.py:422
                             movie file written in 'fakepath
                             \1080p60\partial_movie_files\Test\uncac
                             hed_00002.mp4'
                    DEBUG    Partial movie files to    scene_file_writer.py:471
                             combine (3 files): ['fakepath
                             \\1080p60\\p
                             artial_movie_files\\Test\
                             \uncached_00000.mp4', 'fakepath
                             \\1080p60\
                             \partial_movie_files\\Tes
                             t\\uncached_00001.mp4', '
                             fakepath\\1080p6
                             0\\partial_movie_files\\T
                             est\\uncached_00002.mp4']
[09/28/21 20:03:33] INFO                               scene_file_writer.py:606
                             File ready at 'fakepath\1080p60\Test.mp4'

                    INFO     Rendered Test                         scene.py:233
                             Played 3 animations
                    INFO     Previewed File at: 'fakepath\      file_ops.py:193
                             1080p60\T
                             est.mp4'

System specifications
System Details

OS (with version, e.g Windows 10 v2004 or macOS 10.15 (Catalina)): Windows 10 21H1
RAM: 8 GB
Python version (python/py/python3 --version): 3.8.5
Installed modules (provide output from pip list): a lot

LaTeX details

LaTeX distribution (e.g. TeX Live 2020): MiKTeX
Installed LaTeX packages:

FFMPEG
Output of ffmpeg -version:
ffmpeg version 4.4-full_build-www.gyan.dev Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 10.2.0 (Rev6, Built by MSYS2 project)
configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libdav1d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libglslang --enable-vulkan --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
libavutil      56. 70.100 / 56. 70.100
libavcodec     58.134.100 / 58.134.100
libavformat    58. 76.100 / 58. 76.100
libavdevice    58. 13.100 / 58. 13.100
libavfilter     7.110.100 /  7.110.100
libswscale      5.  9.100 /  5.  9.100
libswresample   3.  9.100 /  3.  9.100
libpostproc    55.  9.100 / 55.  9.100

Additional comments
Maybe using position_tip could help. This bug doesn't appear when the arrow is in the x and y plane.

Are you using v.0.10.0 ? I thought this was fixed in #1924

Are you using v.0.10.0 ? I thought this was fixed in #1924

I am indeed. The issue you are referencing is only fixing tip orientation when adding object on scene. It is indeed working in v0.10.0 when adding an arrow, but the bug still appears when using put_start_and_end_on on it.

@Viicos I opened a PR as an attempt to solve this issue before I noticed your in-depth run down of the problem. The way I solved it was to simply dodge the positioning of the tip wrt the vertical angle, because the way put_start_and_end_on currently works is by removing the tip, scaling and rotating the existing line, then adding the tip again (as you have explained). The process indeed does not account for any rotation along the vertical.
Could you take a look if the PR works as you intended?

@icedcoffeeee It seems to be working perfectly this way. I've made a few tests and the tip is oriented correctly. Will take a look at the PR.


Loading a trained model, popping the last two layers, and then saving it

So I'm working with this architecture for a facial point network:
n_kpts = 68 # number of keypoints

input_shape = (1,100,100)

input_1 = Input(shape=input_shape)

conv_1 = Conv2D(34, kernel_size=(9,9),
				activation='tanh',
				input_shape=input_shape,
				padding='same',
				data_format='channels_first')(input_1)

conv_2 = Conv2D(34, kernel_size=(9,9),
				activation='tanh',
				padding='same',
				data_format='channels_first')(conv_1)

conv_3 = Conv2D(34, kernel_size=(9,9),
				activation='tanh',
				padding='same',
				data_format='channels_first')(conv_2)

conv_4 = Conv2D(34, kernel_size=(9,9),
				activation='tanh',
				padding='same',
				data_format='channels_first')(conv_3)

conv_5 = Conv2D(34, kernel_size=(9,9),
				activation='tanh',
				padding='same',
				data_format='channels_first')(conv_4)

softargmax = spatial_softArgmax(68)(conv_4)

reshape = Reshape((68,2))(softargmax)

model = Model(inputs=input_1, outputs=reshape)

I need to get rid of the reshape and softargmax (it's a custom layer) - and just save the model as the input and conv_1 - conv_5; I want the output to just be the output of that last convolutional layer. I have a model that's trained as an h5 with all of these layers, but i run into some trouble when trying to pop and resave - here's the script I wrote for that:
def get_weights_without_softargmax(fname):
	model = load_model(fname, custom_objects={'spatial_softArgmax':spatial_softArgmax})

	model.summary()
	model.layers.pop() # reshape layer
	model.layers.pop() # spatial softargmax

	model.summary()

	model.save("no_softargmax_" + str(fname))

The first model.summary returns a summary of the whole network;
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         (None, 1, 100, 100)       0
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 34, 100, 100)      2788
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 34, 100, 100)      93670
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 34, 100, 100)      93670
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 34, 100, 100)      93670
_________________________________________________________________
spatial_soft_argmax_1 (spati (None, 68)                0
_________________________________________________________________
reshape_1 (Reshape)          (None, 34, 2)             0
=================================================================
Total params: 283,798
Trainable params: 283,798
Non-trainable params: 0

and the second one returns the now popped network
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         (None, 1, 100, 100)       0
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 34, 100, 100)      2788
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 34, 100, 100)      93670
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 34, 100, 100)      93670
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 34, 100, 100)      93670
=================================================================
Total params: 283,798
Trainable params: 283,798
Non-trainable params: 0

but when I try to do model.save() - i get this error:
Traceback (most recent call last):
  File "weight_chopper.py", line 18, in <module>
    get_weights_without_softargmax("34pts_94percent.h5")
  File "weight_chopper.py", line 16, in get_weights_without_softargmax
    model.save("no_softargmax_" + str(fname))
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 2553, in save
    save_model(self, filepath, overwrite, include_optimizer)
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 107, in save_model
    'config': model.get_config()
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 2390, in get_config
    new_node_index = node_conversion_map[node_key]
KeyError: u'reshape_1_ib-0'

Note how it's referencing the old reshape layer? When I defined the model; I said model = Model(inputs=input_1, outputs=reshape) - so does it still think that the model has that reshape output? How can I convince it otherwise? I've tried doing another Model(inputs=..., outputs=...) type command; but there aren't any appropriate values to plug in for the inputs and outputs!
How can I get the model to save (preferably as a compiled model) with just the convolutional layers?

Build your model before calling save: model.build(). I think that should solve it.

@fchollet thanks so much for the quick reply; what arguments do i feed model.build(), though?
It's saying  TypeError: build() takes exactly 2 arguments (1 given) - and I couldn't find anything on this after a scan of the documentation.

model.build(None). It normally takes an input shape, but in the case of a Sequential model that's not relevant.

Wait, I got confused. I thought you were using the pop method of a Sequential model, but that's not what you are doing. Please post your full code.
Note that pop is not possible with the functional API, it's only implemented for Sequential. If you want to drop some layers in the functional API, you'd do:
new_model = Model(inputs=input_1, outputs=conv_5)

In you case.

@fchollet I guess I was mistakenly using the pop method; wasn't aware that it only worked on Sequential models - in this case, I can't do the new_model = Model(inputs=input_1, outputs=conv_5) - because I have already trained and saved the weights of the full model - When loading that model; there aren't any values i can give for inputs= and outputs= because I haven't defined the model in that script...
As for my full code; here is where i define, train, and save my model: https://gist.github.com/robbiebarrat/5bbd42c9f485198b0f37f6e36ca81a14
here is the attempted "layer removal" script where I load that model, remove the last two layers, and save it again:
https://gist.github.com/robbiebarrat/161d88ff6e2548d87d98a70129e56b95

Try something like this:
import keras
from keras.models import Model, load_model
from keras.layers import Input, Dense
from keras.optimizers import RMSprop
import numpy as np

# Create original model and save it
inputs = Input((1,))
dense_1 = Dense(10, activation='relu')(inputs)
dense_2 = Dense(10, activation='relu')(dense_1)
dense_3 = Dense(10, activation='relu')(dense_2)
outputs = Dense(10)(dense_3)

model = Model(inputs=inputs, outputs=outputs)
model.compile(optimizer=RMSprop(), loss='mse')
model.save('test.h5')

# Load the model and make modifications to it
loaded_model = load_model('test.h5')
loaded_model.layers.pop()
loaded_model.layers.pop()

# Create your new model with the two layers removed and transfer weights
new_model = Model(inputs=inputs, outputs=dense_1)
new_model.compile(optimizer=RMSprop(), loss='mse')
new_model.set_weights(loaded_model.get_weights())

new_model.summary()
new_model.save('test_complete.h5')

there aren't any values i can give for inputs= and outputs= because I haven't defined the model in that script...

You can retrieve these from your model:
new_model = Model(model.inputs, model.layers[-3].output)  # assuming you want the 3rd layer from the last

@fchollet oh sick - i think that worked; got no errors and the model summary looks good.
Here's the final code I used:
def get_weights_without_softargmax(fname):
	model = load_model(fname, custom_objects={'spatial_softArgmax':spatial_softArgmax})

	model.summary()

	new_model = Model(model.inputs, model.layers[-3].output)

	new_model.summary()

	new_model.set_weights(model.get_weights())

	new_model.save("no_softargmax_" + str(fname))

get_weights_without_softargmax("34pts_94percent.h5")

Only concern is that the new model is 1.2 MB, and the old one was 2.3 MB - the file size shouldn't have decreased so much; as the last two layers didn't have any trainable parameters?

You can use the following function to cut a model and obtain a new model with the original input and given layer output.
`def build_bottleneck_model(model, layer_name):
for layer in model.layers:
if layer.name == layer_name:
output = layer.output
bottleneck_model = Model(model.input, output)
bottleneck_model.compile(loss='categorical_crossentropy',
                         optimizer='adam',
                         metrics=['accuracy'])

return bottleneck_model`

Closing as this is resolved

@fchollet oh sick - i think that worked; got no errors and the model summary looks good.
Here's the final code I used:
def get_weights_without_softargmax(fname):
	model = load_model(fname, custom_objects={'spatial_softArgmax':spatial_softArgmax})

	model.summary()

	new_model = Model(model.inputs, model.layers[-3].output)

	new_model.summary()

	new_model.set_weights(model.get_weights())

	new_model.save("no_softargmax_" + str(fname))

get_weights_without_softargmax("34pts_94percent.h5")

Actually you don't need to call set_weights() on new model, its weights are copied from your old one.

You can use the following function to cut a model and obtain a new model with the original input and given layer output.
def build_bottleneck_model(model, layer_name):
    for layer in model.layers:
        if layer.name == layer_name:
            output = layer.output
    bottleneck_model = Model(model.input, output)
    bottleneck_model.compile(loss='categorical_crossentropy',
                             optimizer='adam',
                             metrics=['accuracy'])
    return bottleneck_model```

What if i wanna cut the model as two parts and then obtian the first part as model1 and the second part as model2? I'm doing this because tflite conversion cannot support Lambda Layer or customized layer.


Snowtrace will be discontinued

I saw Snowtrace is going to be discontinued on Nov 30th, so AVAX providers will be affected, I think it would be great to elaborate an action plan to be ready for this change

Yes, it really is something we have to change as soon as possible, do you want to take care of this issue ?

For sure, I can take care of it  🦾


Support version output of OCCM

What this PR does / why we need it:
Which issue this PR fixes(if applicable):
fixes #
Special notes for reviewers:

Release note:

NONE

/approve
/lgtm

/lgtm


Detect available channels

Allow prospero to detect channels that the server can subscribe to.
Requirements:

The user should be able to execute prospero channel search. The result would be a list of "known" channels including the channel GA, their manifest GA(V)/URL, repositories and (optionally) name/description.

Displaying name/description would require resolving all channels - if that's a performance issue, this can be an option in this channel search operation or a new operation channel describe --ga

The source of the channels is a channel itself - a manifest deployed to a maven repository

The definition of that channel-of-channels (repository and manifest GA) has to be customizable at build time (it is a new channels.yaml file)
Each stream in the channel-of-channels corresponds to a -channel artifact GA
The version of the channel is an open pattern - newest version of channel should always be used
Each listed channel file should contain only a single channel

Otherwise it should be ignored and warning should be printed

Already subscribed channels are marked as such and/or listed separately

Nice-to-have:

Make it possible to subscribe to channels using the channel GA
make the source of channel detection customisable by the user.

the definition of channel-of-channels would have to be writable at runtime (e.g. in .installation of Prospero)
all the above logic would have to work with a list of channels
optional: new operations to manage the list of channels-of-channels (otherwise - use vi)

detect channels that constitute different versions of subscribed channels (e.g. wildfly-27/wildfly-28)

Is anyone currently working on this issue?


New remove resource groups automation pack

Thanks for putting this together!  I added a comment to this review in case you want to make the update, but this looks good.

I'd like to keep the links as-is so if we update the underlying script / template it will be automatically reflected vs tying to a specific commit.

Wasn't sure what to call it 101/102 etc with the naming here, but if it works for you - then all good :)

One note - you might want to update the API version for some of your other templates using the 2015-01-01-preview versus the newer 2015-10-31 version.

Thanks David!  I’d been meaning to update the schema version, but just hadn’t gotten around to it yet.  ☺
From: David Tesar<EMAIL_ADDRESS>Sent: Tuesday, August 9, 2016 11:03 AM
To: azureautomation/automation-packs<EMAIL_ADDRESS>Cc: Beth Cooper<EMAIL_ADDRESS>State change<EMAIL_ADDRESS>Subject: Re: [azureautomation/automation-packs] New remove resource groups automation pack (#15)
One note - you might want to update the API version for some of your other templates using the 2015-01-01-preview versus the newer 2015-10-31 version.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHubhttps://github.com/azureautomation/automation-packs/pull/15#issuecomment-238638165, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKUUYgUmiicp_iS0vbsmTjbyqRyjoR1lks5qeMDbgaJpZM4JfbiY.


Documentation fixes

This commit fixes multiple documentation formatting issues in the Traffic Control documentation.

Which Traffic Control components are affected by this PR?

Documentation

What is the best way to verify this PR?

N/A -- Will edit once I have more guidance.
PR submission checklist
This is documentation. I do not believe that it fits the criteria of having a checklist at this time. (Though I could be wrong)

Please review changes. Thank you!


accept more postal codes

Can I still use staging db for test purposes?

@maxammann https://tiles.staging.ehrenamtskarte.app/map.html gives a 502 Bad Gateway... Is the staging server still running?

It wasnt running anymore, I started it again.

Should we close this or do we still want to do changes here?

reopen if still relevant


linediff number-prefixed diff format

Hi there,
Would a new diff format be of interest to the project?
I wanted to write a simple format that would allow me to simplify the line-matching logic. It took me a while to get it to work reliably on gpt-4o-mini (it's what I could extensively test with). More advanced models should be able to comply without such a huge system prompt.
Pros:

Simple rules by leveraging LLMs recitation capabilities.
Client-side: Simpler line matching logic.

Cons:

More verbose, although a lot of that is the big system reminder I had to add so that 4o-mini followed the specification.

You can also have a look at https://github.com/dceluis/ln-diff for a deeper look into the decisions behind the format.
Benchmark for context:

I understand the implementation is nowhere near mergeable but I figured I might show it anyways for easier consideration.
Cheers!

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


blackdoc fails to format docstrings in docstrings

Example code:
def myfunc():
    """Docstring for the function myfunc.

    Here are some docstrings.

    Examples
    --------

    >>> def myfunc2(arg1, arg2):
    ...     '''Docstring for function myfunc2 in docstring
    ...
    ...     More description of the function.
    ...     '''
    ...     pass
    """
    pass

Error message:
blackdoc test.py
error: cannot format test.py: Cannot parse: 10:4: EOF in multi-line string: def myfunc2(arg1, arg2):
Oh no! 💥 💔 💥
1 file fails to reformat.

thanks for the report, @seisman.
There are actually two bugs triggered by your example:

the continuation prompt was not detected for the empty line in the doctest function's docstring (so checking for ...  failed), resulting in the error you saw
black replaces ''' with """, which would result in broken code. Fortunately, this is currently hidden by the first bug.

I'm working on a fix for both.

I can confirm that it works. Thanks for your quick fix!


Two inherited bugs in lwip tcp

First bug:  When attempting to use tcp_write() large data without making copies of the data:  apiflags=0, and TCP_WRITE_FLAG_COPY configuration flag set to 0, the system crashes.  The problem is that tcp_write() creates two linked pbufs, one for the header, and one with pointer to the data.  Downstream in esp_interface low_level_output()  contains a loop that writes the linked pbufs  using sdk_ieee80211_output_pbuf() firmware code.  sdk_ieee80211_output_pbuf() trashes the data in the 2nd pbuf.   Evidence below shows pbufs before and after the call.  Note sdk_ieee80211_output_pbuf() calls back to pbuf_ref() to update the reference count and the data in the 2nd pbuf has been corrupted.  I also note that the low_level_output() loop in the official FREERTOS code looks suspicious (and does not work).
Pbuf:   p: (0x3fff3e50) [00] next(0x3fff3ec8) payload(0x3fff3e8a) tlen: 0b7 len: 036 type:00 flags: 00 ref:01 eb(0xa5a5a5a5)
Pbuf:  p->next (0x3fff3ec8) [00] next(0) payload(0x3fff3ee0) tlen: 081 len: 081 type:01 flags: 00 ref:01 eb(0xa5a5a5a5)
Pbuf:ref (0x3fff3e50) [00] next(0x3fff3ec8) payload(0x3fff3e8a) tlen: 0b7 len: 036 type:00 flags: 00 ref:01 eb(0xa5a5a5a5)
ref exit
Pbuf:   loop next (0x3fff3ec8) [00] next(0) payload(0x3fff3ee0) tlen: 081 len: 081 type:01 flags: 00 ref:01 eb(0xa5a5a5a5) <- pbuf before sdk_ieee80211_output_pbuf
Pbuf:ref (0x3fff3ec8) [00] next(0x26000000) payload(0x9c893844) tlen: 0312e len: 03120 type:030 flags: 031 ref:05448 eb(0x312f5054)  <-- trashed pbuf after sdk_ieee80211_output_pbuf
ref exit
Pbuf:   loop next (0x26000000) [00] next(0xa65149c) payload(0xa65149c) tlen: 0149c len: 0a65 type:09c flags: 014 ref:0a65 eb(0xa65149c)
Second inherited bug:  In api_lib.c
at netconn_write_partly()
the following two lines are obviously wrong. The "!=" and "==" should be swapped in these two lines.
LWIP_ERROR("netconn_write: invalid conn",  (conn != NULL), return ERR_ARG;);
LWIP_ERROR("netconn_write: invalid conn->type",  (conn->type == NETCONN_TCP), return ERR_VAL;);
This error is inheried from the official FREERTOS release.

Looking at the code (disassembled) it appears to allocate an esf_buf via sdk_esf_buf_alloc which is passed the pbuf, this copies the payload pointer and the length of this buffer into the esf_buf slots and it later calls ppTxPkt with this esf_buf. Perhaps it sends one packet for each call, and can not build a packet from multiple pbufs?
I don't even know what the fields of the esf_buf are. The first slot gets the pbuf pointer, but the next slot points to an object into which the payload pointers is stored and seems to be statically allocated when these pools are initialized.
What would that mean for the lwip integration? Is it always wrong for low_level_output to send the linked pbufs as separate packets? Should it be copying them into one packet?
fwiw I have been working away slowly trying to get the code working with a recent lwip version and if you think that would help or you could contribute then please see https://github.com/ourairquality/esp-open-rtos

Thanks, for that rapid response.  I will take a look at the new lwip
code and pound on it.  Tracking down the pbuf related problem was rather
tedious and I was quite disappointed when it  lead to places I could not
fallow.  I also took a look at the disassembled code but did not see
anything obvious.
Regards. Ernest
On 4/15/2017 1:55 PM, Our Air Quality wrote:

Looking at the code (disassembled) it appears to allocate an |esf_buf|
via |sdk_esf_buf_alloc| which is passed the pbuf, this copies the
payload pointer and the length of this buffer into the esf_buf slots
and it later calls |ppTxPkt| with this |esf_buf|. Perhaps it sends one
packet for each call, and can not build a packet from multiple pbufs?
I don't even know what the fields of the esf_buf are. The first slot
gets the pbuf pointer, but the next slot points to an object into
which the payload pointers is stored and seems to be statically
allocated when these pools are initialized.
What would that mean for the lwip integration? Is it always wrong for
|low_level_output| to send the linked pbufs as separate packets?
Should it be copying them into one packet?
fwiw I have been working away slowly trying to get the code working
with a recent lwip version and if you think that would help or you
could contribute then please see
https://github.com/ourairquality/esp-open-rtos
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/SuperHouse/esp-open-rtos/issues/365#issuecomment-294291615,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADjanUiPGgKq3LuDzCyOwhCBXXjRJP-Vks5rwL5JgaJpZM4M-Uiw.

@ourairquality thank you for your efforts.  I dig lwip, and I hope your efforts can get it more solid in this project

Fwiw tried to address this in https://github.com/SuperHouse/esp-open-rtos/pull/389 using the approach noted in the lwip v2 source code which uses pbuf_clone to create a single pbuf from multiple pbufs. Obviously this is not efficient, and deeper understanding and integration might be needed to implement something efficient, but would it happen to solve the bug?

Perhaps related to #322


mysql model can not exit without warning

mysqldump always warns if a password is passed in via cli args
[root@mysql1 models]# /usr/bin/backup perform -t provisioning
[2015/07/14 02:53:07][info] Performing Backup for 'Description for provisioning (provisioning)'!
[2015/07/14 02:53:07][info] [ backup 4.1.10 : ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux] ]
[2015/07/14 02:53:07][info] Database::MySQL (provisioning) Started...
[2015/07/14 02:53:07][info] Using Compressor::Gzip for compression.
[2015/07/14 02:53:07][info]   Command: '/bin/gzip'
[2015/07/14 02:53:07][info]   Ext: '.gz'
[2015/07/14 02:53:23][warn]   Pipeline STDERR Messages:
[2015/07/14 02:53:23][warn]   (Note: may be interleaved if multiple commands returned error messages)
[2015/07/14 02:53:23][warn]
[2015/07/14 02:53:23][warn]   Warning: Using a password on the command line interface can be insecure.
[2015/07/14 02:53:23][info] Database::MySQL (provisioning) Finished!
[2015/07/14 02:53:23][info] Packaging the backup files...
[2015/07/14 02:53:23][info] Using Encryptor::OpenSSL to encrypt the archive.
[2015/07/14 02:53:24][info] Packaging Complete!
[2015/07/14 02:53:24][info] Cleaning up the temporary files...
[2015/07/14 02:53:24][info] Storage::S3 Started...
[2015/07/14 02:53:24][info] Storing 'backups/backups/tropo-provisioning-database/provisioning/2<IP_ADDRESS>.53.07/provisioning.tar.enc'...
[2015/07/14 02:53:24][info]   Initiate Multipart 'backups/backups/tropo-provisioning-database/provisioning/2<IP_ADDRESS>.53.07/provisioning.tar.enc'
[2015/07/14 02:53:24][info]   Uploading 22 Parts...
[2015/07/14 02:53:25][info]   ...10% Complete...
[2015/07/14 02:53:26][info]   ...20% Complete...
[2015/07/14 02:53:26][info]   ...30% Complete...
[2015/07/14 02:53:27][info]   ...40% Complete...
[2015/07/14 02:53:28][info]   ...50% Complete...
[2015/07/14 02:53:29][info]   ...60% Complete...
[2015/07/14 02:53:30][info]   ...70% Complete...
[2015/07/14 02:53:30][info]   ...80% Complete...
[2015/07/14 02:53:31][info]   ...90% Complete...
[2015/07/14 02:53:32][info]   Complete Multipart 'backups/backups/tropo-provisioning-database/provisioning/2<IP_ADDRESS>.53.07/provisioning.tar.enc'
[2015/07/14 02:53:33][info] Storage::S3 Finished!
[2015/07/14 02:53:33][info] Cleaning up the package files...
[2015/07/14 02:53:33][warn] Backup for 'Description for provisioning (provisioning)' Completed Successfully (with Warnings) in 00:00:26
[2015/07/14 02:53:33][info] Sending notification using Notifier::Slack...
[root@mysql1 models]#

This means the dump can never pass without a warn

mysqldump gives that warning yes. That's because it's not the safest way to send passwords, the CLI. It's the only way you can set it with backup unfortunately.
You can use a mysql option file to set your password there and then you wouldn't need to set it in the backup config.
https://dev.mysql.com/doc/refman/5.1/en/option-files.html

I merged the fix for issue https://github.com/backup/backup/issues/618 on master. This touches on using the options file and using that approach will remove the warning for your logs.


change styles exports to support easy import in apps.

This project uses semver, please check the scope of this pr:

[x] #patch# - backwards-compatible bug fix
[ ] #minor# - adding functionality in a backwards-compatible manner
[ ] #major# - incompatible API change

Changelog
Based on (https://github.com/aexmachina/ember-cli-sass/issues/107#issuecomment-188283583), moved core styles into app for easier consumption and/or overriding in apps.
Apps can import these styles via @import 'ember-frost-core/frost-theme';
Addons can import styles via @import 'node_modules/ember-frost-core/app/styles/ember-frost-core/frost-theme';

Approved - let's document app/theme as the imports

Approved


Side Note
compile_flags.txt isn't working on my setup

Thanks for the detailed report! A couple things:

Your compile_flags.txt file isn't working because -x assembler-with-cpp needs to be split into two separate lines.
While reproducing your issue, I found an issue where our parsing code doesn't handle compiler output with column numbers correctly (the column number is treated as the line number). I'll work on a fix for that first.
As to why the server is crashing , I honestly have no idea. I'll try to get Helix set up locally so I can dig into this more.

I have a fix up for the compile_flags.txt-related issues in #149. If you could test that out to make sure it resolves those issues (along with the change to your compile_flags.txt) that would be greatly appreciated. I don't have an answer for the crash yet, but am actively looking into it. :)

Crash should be fixed by #150

@WillLillis thanks for the quick response and fix. With the two prs, the server doesn't crash anymore, and compile_flags.txt is working. But even though I see the arm register set loaded in my helix.log, I still don't get any information when I hover on the registers.

@WillLillis thanks for the quick response and fix. With the two prs, the server doesn't crash anymore, and compile_flags.txt is working.

Glad to hear it!

But even though I see the arm register set loaded in my helix.log, I still don't get any information when I hover on the registers.

Hmmm. Which registers are you trying to hover over specifically? The supported registers supported by asm-lsp are listed here: https://github.com/bergercookie/asm-lsp/blob/master/docs_store/registers/raw/arm.xml. It's possible I missed some when adding the architecture.

Hmmm. Which registers are you trying to hover over specifically? The supported registers supported by asm-lsp are listed here: https://github.com/bergercookie/asm-lsp/blob/master/docs_store/registers/raw/arm.xml. It's possible I missed some when adding the architecture.

All the registers I see in that file are arm 32 registers, I am working on arm64(aarch64). The registers are x0 - x31 and xzr

Hmmm. Which registers are you trying to hover over specifically? The supported registers supported by asm-lsp are listed here: https://github.com/bergercookie/asm-lsp/blob/master/docs_store/registers/raw/arm.xml. It's possible I missed some when adding the architecture.

All the registers I see in that file are arm 32 registers, I am working on arm64(aarch64). The registers are x0 - x31 and xzr with their 32-bit variant w0 - w32 and wzr.
Maybe a new option arm64 should be added to .asm-lsp.toml and in that mode these registers would be made available. aarch64 doesn't have a PC register, LR is replaced by X30, arm64 has a dedicated SP register and doesn't use a general purpose register and I'm sure there are a bunch of others. I'm still in the process of learning aarch64 so this list is not exhaustive.

:+1:

But in my opinion, the issue I reported in this tracker has been resolved so your PRs can be merged and this issue closed so that I create a new issue to track that.

Sounds good! I should be able to give those a final look over later tonight!

I am willing to work on this.

That would be greatly appreciated! If you could get the JSON file pulled together, I can take it the rest of the way to add arm64 support to the server.

@WillLillis you mean xml file right?

@WillLillis you mean xml file right?

Oops, yeah xml. 🤦♂


Scan CLI and AtClientImpl _getAtKeys implementation

PR = #19
There is a lot of hidden work behind _getAtKeys in AtClientImpl such as string parsing from scan and AtKey object instantiating via KeyBuilders.

PR = #19
Key discussion (from Engineering Google Chat thread)
https://docs.google.com/document/d/1E-FIHySO73k036kaXnkuUESkOQ2PUOLex2t7fMovc4I/edit?usp=sharing

#19

@public API notes
https://docs.google.com/document/d/1DaH23kW7gpPdFiTIWxScIovJXiGqJS9eNnRcs7m_wY0/edit#heading=h.53nl6rorpt9q

About 5SP worth of work this sprint


[SPARK-3162] [MLlib] Add local tree training for decision tree regressors

What changes were proposed in this pull request?
WIP, DO NOT MERGE
Overview
This PR adds local tree training for decision tree regressors as a first step for addressing SPARK-3162: train decision trees locally when possible.
See this design doc (in particular the local tree training section) for detailed discussion of the proposed changes.
Distributed training logic has been refactored but only minimally modified; the local tree training implementation leverages existing distributed training logic for computing impurities and splits. This shared logic has been refactored into ...Utils objects (e.g. SplitUtils.scala, ImpurityUtils.scala).
How to Review
Each commit in this PR adds non-overlapping functionality, so the PR can be reviewed commit-by-commit.
Changes introduced by each commit:

Adds new data structures for local tree training (FeatureVector, TrainingInfo)
Adds shared utility methods for computing splits/impurities (SplitUtils, ImpurityUtils, AggUpdateUtils), largely copied from existing distributed training code in RandomForest.scala.
Unit tests for split/impurity utility methods (TreeSplitUtilsSuite)
Updates distributed training code in RandomForest.scala to depend on the utility methods introduced in 2.
Adds local tree training logic (LocalDecisionTree)
Local tree unit/integration tests (LocalTreeUnitSuite, LocalTreeIntegrationSuite)

How was this patch tested?
No existing tests were modified. The following new tests were added (also described above):

Unit tests for new data structures specific to local tree training (LocalTreeDataSuite, LocalTreeUtilsSuite)
Unit tests for impurity/split utility methods (TreeSplitUtilsSuite)
Unit tests for local tree training logic (LocalTreeUnitSuite)
Integration tests verifying that local & distributed tree training produce the same trees (LocalTreeIntegrationSuite)

Can one of the admins verify this patch?

@WeichenXu123 would you be able to take an initial look at this?

@smurching Does it still WIP ? If done remove "[WIP]", I will begin review, thanks!

Thanks! I'll remove the WIP. To clear things up for the future, I'd thought [WIP] was the appropriate tag for a PR that's ready for review but not ready to be merged (based on https://spark.apache.org/contributing.html) -- have we stopped using the WIP tag?

add to whitelist

Test build #82557 has started for PR 19433 at commit 9a7174e.

Test build #82557 has finished for PR 19433 at commit 9a7174e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class LocalTreeIntegrationSuite extends SparkFunSuite with MLlibTestSparkContext
class LocalTreeUtilsSuite extends SparkFunSuite

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82557/
Test FAILed.

The failing tests (in DecisionTreeSuite) fail because we've historically handled
a) splits that have 0 gain
differently from
b) splits that fail to achieve user-specified minimum gain (metadata.minInfoGain) or don't meet minimum instance-counts per node (metadata.minInstancesPerNode).
Previously we'd create a leaf node with valid impurity stats in case a) and invalid impurity stats in case b). This PR creates a leaf node with invalid impurity stats in both cases.
As a fix I'd suggest creating a LeafNode with correct impurity stats in case a), but with the stats.valid member set to false to indicate that the node should not be split.

Test build #82570 has started for PR 19433 at commit abc86b2.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82570/
Test FAILed.

The failing SparkR test (which compares RandomForest predictions to hardcoded values) fails not due to a correctness issue but (AFAICT) because of an implementation change in best-split selection.
In this PR we recompute parent node impurity stats when considering each split for a feature, instead of computing parent impurity stats once per feature (see this by comparing RandomForest.calculateImpurityStats in Spark master and ImpurityUtils.calculateImpurityStats in this PR).
The process of repeatedly computing parent impurity stats results in slightly different results at each iteration due to Double precision limitations. This in turn can cause different splits to be selected (e.g. if two splits have mathematically equal gains, Double precision limitations can cause one split to have a higher/smaller gain than the other, influencing tiebreaking).

Test build #82652 has started for PR 19433 at commit 5c29d3d.

Test build #82652 has finished for PR 19433 at commit 5c29d3d.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82652/
Test FAILed.

I made a rough pass. I have only a few issues for now, I haven't go into code details:

The colStoreInit currently ignore the subsampleWeights, it should be used, isn't it ? I read your doc, in the higher level, the local training will be used to train sub-trees as parts of the global distributed training, subsampleWeights should be important info. and here it will train only single tree so subsampleWeights only contains one element, does we still need use BaggedPoint structure ?

The logic of training for regression and for classification will be the same I think, only impurity difference but do not influence the code logic.

The key idea is to use the columnar storage format for features, is the purpose to improve memory cost & cache locality when finding best splits ? I see the code will do some reordering operation on feature values and use indices, but I haven't go into details. It's a complex part I need more time to review.

Maybe we can support multithreads in local training, what do you think about it ?

Thanks for the comments!

Yep, feature subsampling is necessary for using local tree training in distributed training. I was thinking of adding subsampling in a follow-up PR. You're right that we don't need to pass an array of BaggedPoints to local tree training; we should just pass an array of subsampleWeights (weights for the current tree) and an array of TreePoints. I'll push an update for this.

Agreed that the logic for classification will be the same but with a different impurity metric. I can add support for classification & associated tests in a follow-up PR.

IMO the primary advantage of the columnar storage format is that it'll eventually enable improvements to best split calculations; specifically, for continuous features we could sort the unbinned feature values and consider every possible threshold.  There are also the locality & memory advantages described in the design doc. In brief, DTStatsAggregator stores a flat array partitioned by (feature x bin). If we can iterate through all values for a single feature at once, most updates to DTStatsAggregatorwill occur within the same subarray.

Multithreading could be a nice way to increase parallelism since we don't use Spark during local tree training. I think we could add it in a follow-up PR.

Sorry, realized I conflated feature subsampling and subsampleWeights (instance weights for training examples). IMO feature subsampling can be added in a follow-up PR, but subsampleWeights should go in this PR.

Test build #82717 has started for PR 19433 at commit c9a8e01.

Test build #82717 has finished for PR 19433 at commit c9a8e01.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82717/
Test PASSed.

Test build #82721 has started for PR 19433 at commit 93e17fc.

Test build #82721 has finished for PR 19433 at commit 93e17fc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82721/
Test PASSed.

@smurching I found some issues and have some thoughts on the columnar features format:

In your doc, you said "Specifically, we only need to store sufficient stats for each bin of a single feature, as opposed to each bin of every feature", BUT, current implementation, you still allocate space for all features when computing: -- see DTStatsAggregator implementation, you pass featureSubset = None so DTStatsAggregator will allocate space for every features. According to your purpose, you should pass featureSubset = Some(Array(currentFeatureIndex)).

Current implementation still use binnedFeatures. You said in future it will be improved to sort feature values for continuous feature (for more precise tree training), if you want to consider every possible thresholds, you need hold rawFeatures instead of binnedFeatures in the columnar feature array, and in each split range offset, you need sort every continuous features. Is this the thing you want to do in the future ? This will increase calculation amount.

For current implementation(using binnedFeature) , there is no need to sort continuous features inside each split offset. So the indices for each feature is exactly the same. In order to save memory, I think these indices should be shared, no need to create separate indices array for each features. Even if you add the improvements for continuous features mentioned above, you can create separate indices array for only continuous features, the categorical features can still share the same indices array.

About locality advantage of columnar format, I have some doubts. Current implementation, you do not reorder the label and weight array, access label and weight value need use indices, when calculating DTStat, this break locality. (But I'm not sure how much impact to perf this will bring).

About the overhead of columnar format: when making reordering (when get new split, we need reorder left sub-tree samples into front), so you need reordering on each column, and at the same time, update the indices array. But, if we use row format, like:
Array[(features, label, weight)], reordering will be much easier, and do not need indices.
So, I am considering, whether we can use row format, but at the time when we need DTStatsAggregator computation, copy the data we need from the row format into columnar format array (only need to copy rows between sub-node offset and only copy the sampled features if using feature subsampling).

Test build #83025 has started for PR 19433 at commit fd6cdbb.

Test build #83025 has finished for PR 19433 at commit fd6cdbb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83025/
Test PASSed.

Merged build finished. Test PASSed.

@WeichenXu123 Thanks for the comments! I'll respond inline:

In your doc, you said "Specifically, we only need to store sufficient stats for each bin of a single feature, as opposed to each bin of every feature", BUT, current implementation, you still allocate space for all features when computing: -- see DTStatsAggregator implementation, you pass featureSubset = None so DTStatsAggregator will allocate space for every features. According to your purpose, you should pass featureSubset = Some(Array(currentFeatureIndex)).

I like your proposed solution (pass featureSubset = Some(Array(currentFeatureIndex))). I'll go ahead & implement it.

Current implementation still use binnedFeatures. You said in future it will be improved to sort feature values for continuous feature (for more precise tree training), if you want to consider every possible thresholds, you need hold rawFeatures instead of binnedFeatures in the columnar feature array, and in each split range offset, you need sort every continuous features. Is this the thing you want to do in the future ? This will increase calculation amount.

Yep, we'll have to pass raw (Double) continuous features to the local tree training methods, which will require them to accept an Array[LabeledPoint] instead of an Array[TreePoint] as input & increase memory usage (along with requiring us to store additional indices).
We'll actually only have to run an O(n log n) sort on continuous feature values once (i.e. in the FeatureVector constructor), since once the continuous features are sorted we can update them as we would for categorical features when splitting nodes (in O(n) time) and they'll remain sorted.

For current implementation(using binnedFeature) , there is no need to sort continuous features inside each split offset. So the indices for each feature is exactly the same. In order to save memory, I think these indices should be shared, no need to create separate indices array for each features. Even if you add the improvements for continuous features mentioned above, you can create separate indices array for only continuous features, the categorical features can still share the same indices array.

Agreed, I'll make this change.

About locality advantage of columnar format, I have some doubts. Current implementation, you do not reorder the label and weight array, access label and weight value need use indices, when calculating DTStat, this break locality. (But I'm not sure how much impact to perf this will bring).

Yeah, I'm not sure if it'd be better to reorder the labels/weights arrays to achieve improved locality.
I think we could experiment with both, but I'd prefer to save that for a follow-up PR unless you or another reviewer think it'll make a big perf difference.

About the overhead of columnar format: when making reordering (when get new split, we need reorder left sub-tree samples into front), so you need reordering on each column, and at the same time, update the indices array. But, if we use row format, like:
Array[(features, label, weight)], reordering will be much easier, and do not need indices.
So, I am considering, whether we can use row format, but at the time when we need DTStatsAggregator computation, copy the data we need from the row format into columnar format array (only need to copy rows between sub-node offset and only copy the sampled features if using feature subsampling).

This is an interesting idea, my main concern is that on the first iteration of local tree training we'd need to copy the entire training data matrix from row -> columnar format, which negates any memory savings we get from not using indices. I'm also concerned about the overhead of repeatedly copying data from row -> columnar format.

We'll actually only have to run an O(n log n) sort on continuous feature values once (i.e. in the FeatureVector constructor), since once the continuous features are sorted we can update them as we would for categorical features when splitting nodes (in O(n) time) and they'll remain sorted.

Nice! so only one pass global sort, and then each split only need O(n) time copy.

Test build #83093 has started for PR 19433 at commit ebade23.

Test build #83093 has finished for PR 19433 at commit ebade23.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83093/
Test PASSed.

Merged build finished. Test PASSed.

Made a few updates, here’s a quick summary/what I’d propose moving forward:
Right now:

Shared row indices for all (categorical & continuous) features are stored & updated in TrainingInfo
LocalDecisionTree.computeBestSplits computes best splits/sufficient stats for a single feature at a time
A utility method (LocalDecisionTreeUtils.updateArrayForSplit) is used to sort both feature values and shared row indices

When we add support for raw continuous feature values:

Add a subclass of FeatureColumn (e.g. ContinuousFeatureColumn) that stores and sorts its own array of row indices, pass these row indices to methods requiring them.

I also renamed FeatureVector to FeatureColumn since the former seemed like it’d confuse developers (FeatureVector sounds like a single data point)

Test build #83104 has started for PR 19433 at commit 9cc6333.

Test build #83104 has finished for PR 19433 at commit 9cc6333.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83104/
Test PASSed.

After discussion and modifications, I approve this PR overall. Ping @jkbradley Can you take a look now ?

Test build #83219 has started for PR 19433 at commit 7efb1e0.

Test build #83219 has finished for PR 19433 at commit 7efb1e0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83219/
Test PASSed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83310/
Test FAILed.

jenkins retest this please

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83353/
Test FAILed.

jenkins retest this please

Test build #83464 has started for PR 19433 at commit 3f72cc0.

Test build #83464 has finished for PR 19433 at commit 3f72cc0.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83464/
Test FAILed.

jenkins retest this please

Test build #83503 has started for PR 19433 at commit 3f72cc0.

Test build #83503 has finished for PR 19433 at commit 3f72cc0.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83503/
Test FAILed.

Test build #83507 has started for PR 19433 at commit b7e6e40.

Test build #83507 has finished for PR 19433 at commit b7e6e40.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83507/
Test FAILed.

CC @dbtsai  in case you're interested b/c of Sequoia forests

Test build #3983 has started for PR 19433 at commit b7e6e40.

Test build #83873 has started for PR 19433 at commit 0b27c56.

Test build #83874 has started for PR 19433 at commit d86dd18.

Test build #83873 has finished for PR 19433 at commit 0b27c56.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83873/
Test FAILed.

Test build #83874 has finished for PR 19433 at commit d86dd18.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83874/
Test FAILed.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3156/
Test PASSed.

Test build #97977 has started for PR 19433 at commit d86dd18.

Test build #97977 has finished for PR 19433 at commit d86dd18.

This patch fails to generate documentation.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97977/
Test FAILed.

Is this still a thing you are actively working on?

Thank you for your contribution! We've used this code extensively as a basis for our @cisco/oraf library, which incorporates local training into the existing decision tree and random forest APIs, and managed to significantly speed-up the training process.

That's cool @rstarosta . Does having it in a library meet the needs of folks and we can close this issue?

Test build #110569 has started for PR 19433 at commit d86dd18.

Test build #110569 has finished for PR 19433 at commit d86dd18.

This patch fails to generate documentation.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110569/
Test FAILed.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15790/
Test PASSed.


[SMALLFIX][branch-1.5] Fix a bug that mounting UFS should connect to UFS only when replay=true

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/15523/Failed Tests: 1org.alluxio:alluxio-tests: 1alluxio.client.MultiUfsMountIntegrationTest.mountAfterMasterRestart
Test FAILed.

Merged build finished. Test FAILed.

Jenkins, test this please

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/15525/Failed Tests: 1org.alluxio:alluxio-tests: 1alluxio.client.MultiUfsMountIntegrationTest.mountAfterMasterRestart
Test FAILed.

Jenkins, test this please

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/15533/Build result: FAILURE[...truncated 2643 lines...][JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/core/client/hdfs/pom.xml to org.alluxio/alluxio-core-client-hdfs/1.5.0-RC1/alluxio-core-client-hdfs-1.5.0-RC1.pom[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/core/client/hdfs/target/alluxio-core-client-hdfs-1.5.0-RC1.jar to org.alluxio/alluxio-core-client-hdfs/1.5.0-RC1/alluxio-core-client-hdfs-1.5.0-RC1.jar[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/core/client/hdfs/target/alluxio-core-client-hdfs-1.5.0-RC1-sources.jar to org.alluxio/alluxio-core-client-hdfs/1.5.0-RC1/alluxio-core-client-hdfs-1.5.0-RC1-sources.jar[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/core/client/hdfs/target/alluxio-core-client-hdfs-1.5.0-RC1-tests.jar to org.alluxio/alluxio-core-client-hdfs/1.5.0-RC1/alluxio-core-client-hdfs-1.5.0-RC1-tests.jar[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/keyvalue/common/pom.xml to org.alluxio/alluxio-keyvalue-common/1.5.0-RC1/alluxio-keyvalue-common-1.5.0-RC1.pom[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/core/client/fs/pom.xml to org.alluxio/alluxio-core-client-fs/1.5.0-RC1/alluxio-core-client-fs-1.5.0-RC1.pom[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/core/client/fs/target/alluxio-core-client-fs-1.5.0-RC1.jar to org.alluxio/alluxio-core-client-fs/1.5.0-RC1/alluxio-core-client-fs-1.5.0-RC1.jar[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/core/client/fs/target/alluxio-core-client-fs-1.5.0-RC1-sources.jar to org.alluxio/alluxio-core-client-fs/1.5.0-RC1/alluxio-core-client-fs-1.5.0-RC1-sources.jar[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/core/client/fs/target/alluxio-core-client-fs-1.5.0-RC1-tests.jar to org.alluxio/alluxio-core-client-fs/1.5.0-RC1/alluxio-core-client-fs-1.5.0-RC1-tests.jar[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/core/common/pom.xml to org.alluxio/alluxio-core-common/1.5.0-RC1/alluxio-core-common-1.5.0-RC1.pom[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/core/common/target/alluxio-core-common-1.5.0-RC1.jar to org.alluxio/alluxio-core-common/1.5.0-RC1/alluxio-core-common-1.5.0-RC1.jar[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/core/common/target/alluxio-core-common-1.5.0-RC1-sources.jar to org.alluxio/alluxio-core-common/1.5.0-RC1/alluxio-core-common-1.5.0-RC1-sources.jar[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/core/common/target/alluxio-core-common-1.5.0-RC1-tests.jar to org.alluxio/alluxio-core-common/1.5.0-RC1/alluxio-core-common-1.5.0-RC1-tests.jar[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/core/server/worker/pom.xml to org.alluxio/alluxio-core-server-worker/1.5.0-RC1/alluxio-core-server-worker-1.5.0-RC1.pom[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/underfs/glusterfs/dependency-reduced-pom.xml to org.alluxio/alluxio-underfs-glusterfs/1.5.0-RC1/alluxio-underfs-glusterfs-1.5.0-RC1.pom[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/underfs/glusterfs/target/alluxio-underfs-glusterfs-1.5.0-RC1.jar to org.alluxio/alluxio-underfs-glusterfs/1.5.0-RC1/alluxio-underfs-glusterfs-1.5.0-RC1.jar[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/underfs/glusterfs/target/alluxio-underfs-glusterfs-1.5.0-RC1-sources.jar to org.alluxio/alluxio-underfs-glusterfs/1.5.0-RC1/alluxio-underfs-glusterfs-1.5.0-RC1-sources.jar[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/core/client/pom.xml to org.alluxio/alluxio-core-client/1.5.0-RC1/alluxio-core-client-1.5.0-RC1.pom[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/examples/pom.xml to org.alluxio/alluxio-examples/1.5.0-RC1/alluxio-examples-1.5.0-RC1.pom[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/core/pom.xml to org.alluxio/alluxio-core/1.5.0-RC1/alluxio-core-1.5.0-RC1.pom[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/core/server/proxy/pom.xml to org.alluxio/alluxio-core-server-proxy/1.5.0-RC1/alluxio-core-server-proxy-1.5.0-RC1.pom[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/shell/pom.xml to org.alluxio/alluxio-shell/1.5.0-RC1/alluxio-shell-1.5.0-RC1.pom[JENKINS] Archiving /home/jenkins/workspace/Alluxio-Pull-Request-Builder/tests/pom.xml to org.alluxio/alluxio-tests/1.5.0-RC1/alluxio-tests-1.5.0-RC1.pomchannel stopped
Test FAILed.

@gpang  PTAL

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/15534/
Test PASSed.


I successfully ran node main.js on composer, but the result of the test is this

My cpu share has exceeded 100%!! Why?
What is the difference between dev-peer0.org1.example.co...0.1.0 and peer0.org1.example.com? What do they mean? What does Traffic In and Traffic Out and Disc Read mean? What do the names in docker mean? Why is there a difference between CPU(max) and CPU(avg)? Please explain

@zcyzhu For general questions please use the Rocket.Chat #caliper channel or the Caliper mailing list. GitHub issues should only be raised for confirmed bugs or enhancement suggestions.


refactoring of the S3 sync operations

Modify the env variables
Only sync dirs that had changes
Don't move the submissions dir when a form is archived or restored

👍 tested and looks good to me


SubForm DestroyWorkaround Optimizations

Adds better handling for destroyWorkaround and non-persisted objects.
Non-persisted objects that are removed will return null instead of _destroy: true.

Closing to target branch 2.0.0


Add cross-env for cross platform project start

Hello there,
Following some discussion in #157, these changes simply introduce the cross-env package to make npm start execute seamlessly across platforms when any npm scripts require setting of environment variables.
Currently, for anybody trying to run npm start in Command Prompt in Windows, they'll likely hit a message like:
'GENERATE_SOURCEMAP' is not recognized as an internal or external command,
operable program or batch file.

If it's not worthwhile enough (i.e. telling such users to use a different shell, or work around it another way), I don't mind if these changes are just closed off.

Separately, while on the topic of Windows environments, the ESLint configuration will go bonkers in reporting warnings under the linebreak-style rule for every single file in the repository (assuming Git has been installed to check in as LF and check out as CRLF, which I believe is the default setting).
src\utils\fragment-tools.ts
  Line 1:27:     Expected linebreaks to be 'LF' but found 'CRLF'  linebreak-style
  Line 2:39:     Expected linebreaks to be 'LF' but found 'CRLF'  linebreak-style
  Line 3:34:     Expected linebreaks to be 'LF' but found 'CRLF'  linebreak-style
  ...

Currently, the relevant configuration is:
https://github.com/IBM/carbon-ui-builder/blob/5af3177ad750411c4b87d73744d489c23554da73/.eslintrc.js#L30-L33
There's some relevant information from the ESLint website about this problem specifically.
https://eslint.org/docs/latest/rules/linebreak-style#using-this-rule-with-version-control-systems
One suggestion would be to replace 'unix' with process.platform === 'win32' ? 'windows' : 'unix' (can confirm this works, at least for me).
Again, the suggestion can be ignored if the developer experience of non-Unix users isn't a priority.

Thanks for the PR, Michael (@miqh)!
If I understand the line break issue, this is on purpose. Line breaks in files should remain consistent and unix type makes sense.
It they don't, we get (well, other than inconsistency) diffs where lines look the same and are hard to review.
IDEs should be able to use a specific line break regardless of the platform.

@zvonimirfras, no worries.

If I understand the line break issue, this is on purpose.

As long as you're aware that typical checkouts on Windows (i.e. auto-conversion of LF to CRLF) of this codebase will trigger warnings against the linebreak-style lint for every single file, as I wrote previously.
In case I wasn't clear, I wasn't advocating for the removal of the rule altogether. It's good to ensure line-ending consistency. Rather, I was suggesting an adjustment such that editing on Windows environments won't bury the reporting of other legitimate linting issues with all these ones against linebreak-style.

As an aside, what's with those failing build notifications? I noticed it happened in my other recent pull request too.

@miqh here's a simple write-up on how to fix LF/CRLF in VSCode if that's what you're using. Other IDEs have similar settings.
Our .editorconfig has end_of_file = lf, so I'm not sure why your IDE wouldn't pick that up and do the right thing 🤔
What's your IDE?

Failing build notifications are due to deploy failing - and we don't deploy from PRs, only when the code is merged.
So it's a false negative because the build itself was fine. We should fix it to not try deploying if it's not a merge to main, but it wasn't a priority.
If you wanna take a stab at it, you're very welcome. You seem to be doing great with everything else so far :)

@zvonimirfras, yes, I've been using VSCode.
Ah, wasn't aware of the .editorconfig present. I don't have the EditorConfig extension installed, and I think you're assuming anybody who checks out this codebase has it (and obviously in my case I didn't). That would explain why I see all the warnings, given that VSCode hasn't applied the configuration settings. A potential documentation note for contributors?

Understood, thanks for clarifying the latter.

@miqh oh my, I thought the editorconfig detection was built into VSCode :sweat_smile:
Must've been a while since I installed it.
Documentation note makes sense. How would you word it to keep it general enough but clear for people in your situation?


Update policyGenerator.yaml

few minor edits into the descriptions.

@berenss could you please add a signoff to your commit?

@berenss You have to add it to the commit directly:
git commit --amend --signoff
git push --force

😂😂 you guys - I'm struggling here obviously 😂😂

You're not the first--DCO is a little confusing (and not widely used outside of ACM). We're here to help! 🙂


Feature: dynamic queues/bindings registry

This is related to dynamic queues and their registration in runtime.
So far, if queue is declared as a @Bean it will be detected by RabbitAdmin and declared in case there is a connection problem (or rabbit failover etc.). But in case that it is created in runtime, it has to be manually registered as a singleton bean, otherwise on rabbit failover, queue will not exist anymore in rabbit and listener container will fail.
I guess this should be taken care of? E.g. when registering the queue to container it should be registered as a bean, so rabbit can later redeclare if needed. Just don't know what to do with bindings, since those also have to be in place on Rabbit failover.
Other option is to have something like:
ContainerDeclarables cdec = new ContainerDeclarablesBuilder
                .container(messageListenerContiner)
                .declarables(declarablesSupplier);

where you would have all declarables which would be bound to container at a runtime, and at the same time handle bean registration at runtime and expose methods as removeQueue (that would remove the queue from registry and unbind it from container) etc.
What do you think about this?

It's not clear what you are trying to achieve. What is wrong with registering the bean with the application context?
And, as you say, the queue alone is not enough and the container has no knowledge of the exchange and/or routing key.
I could envision something like CompoundDeclarable containing queue, exchange and binding and
RabbitAdmin.declareCompound(String beanName, CompoundDeclarable dec) {...}

Since the admin has a reference to the application context it could register the bean automatically.
But I question the value, given that you can register the bean(s) yourself.

Was thinking if possible to decrease code you need to write, but to have it already in amqp-spring lib.
So use case is to register queue to container at runtime as easy as possible. This queue will possibly be bound to some exchanges. I guess exchange in runtime is not that common requirement, but queue and binding is.
So far container.addQueue only adds queue to container, but it is not tracked by context and not automatically declared.
So, something like container.addQueueWithBindings(Map<Queue, Set<Binding>> qb) that would automatically register queues and bindings with context and declare it with rabbit admin (and re-declare as needed) and start consuming from queues in runtime.

I don't think we should think about mixing responsibilities. The container is really not about declaration and registration. It is only for connecting to the specified queue and consuming messages.
A declaration, registration and binding is really up to a RabbitAdmin.
I think we need just add an additional functionality into a RabbitAdmin something like declareAndRegistrerQueue(). So, target application will be able to declare a queue at runtime and have it registered as as a bean.
We also could do a revers logic like BeanPostProcessor to catch runtime beans and declare them on RabbitMQ, but that is not going to be as high level API like the mentioned single function like declareAndRegistrerQueue().
Anyway I'm against adding such a functionality into a listener container...


[Bug report] Doc: "COLUMN" doesn't have owner, please remove it from metadataObjectType possible values in set-owner/get-owner api doc

Version
main branch
Describe what's wrong
https://gravitino.apache.org/docs/0.7.0-incubating/api/rest/set-owner
https://gravitino.apache.org/docs/0.7.0-incubating/api/rest/get-owner

Error message and/or stacktrace
N/A
How to reproduce
N/A
Additional context
No response

Working on this


Add more options

Hi. I like you module a lot. It's great.
Is it possible to add 3 and 5? It's for a pbta game... (impulse drive)
Thank you.

Adding to this suggestion would it be possible to support customizable clocks? Would love to be able to use these in Foundry.

Adding to this suggestion would it be possible to support customizable clocks? Would love to be able to use these in Foundry.


Add the support of the bloom BF.INFO command

This PR introduce the bloom BF.INFO command.
[1] https://redis.io/commands/bf.info/

@zncleon BTW, according to the Redis documentation, the BF.EXISTS command returns 0 if the key doesn't exist. But I see that currently, the code returns an error {Status::RedisExecErr, "key is not found"}. Is it intentional?

@zncleon BTW, according to the Redis documentation, the BF.EXISTS command returns 0 if the key doesn't exist. But I see that currently, the code returns an error {Status::RedisExecErr, "key is not found"}. Is it intentional?

Sure, I think I made a mistake. It should return 0 in this case. I will fix it.

Thanks all, merging


Use timezone when iterating over timeline

Use timezone when iterating over timeline
Issue #38

Codecov Report

Merging #123 (c9148d4) into main (6ea1606) will decrease coverage by 0.00%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #123      +/-   ##
==========================================
- Coverage   96.22%   96.21%   -0.01%
==========================================
  Files          10       10
  Lines         848      846       -2
==========================================
- Hits          816      814       -2
  Misses         32       32

Impacted Files
Coverage Δ

gcal_sync/timeline.py
100.00% <ø> (ø)

gcal_sync/api.py
98.05% <100.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


East/west travel on keyboard is interrupted when VoiceOver Live Region active

Steps to reproduce:
go to /a11y-map/index.html
navigate with keyboard to area over pop-ups
I went 7 arrows west, then 5 arrows south from the starting position to sit over a region of three pop-ups
turn on VoiceOver for OS X (cmd-f5)
try to rapidly travel east with keyboard (right arrow)
Expected result:
Travels east
Context:
Initial assumption is that the ARIA Live Region is taking priority over the JS to move the map east and west. This doesn't happen with north and south though.
Supporting Info:
Google Chrome Version 68.0.3440.106
OS X High Sierra
localhost..././a11y-map/index.html

See original at: https://github.com/kellyhutchins/a11y-map/issues/15

I suspect this might be an issue with the keydown function needing to be throttled to compensate for the VoiceOver running.

Bug occurs with the MapView animation when VoiceOver is active, and is not caused by this config map.


Remove legacy crown assets from Frontend (v6.0)

What
Remove remaining instances of St. Edward's crown from GOV.UK Frontend.
Why
These assets are no longer needed, having been superceded by the Tudor crown.
Who needs to work on this

Developer

Who needs to review this

Developer

Done when
### Tasks
- [ ] The St. Edward's crown has been removed from the header component
- [ ] The feature flag to show St. Edward's crown has been removed
- [ ] Any remaining assets featuring the St. Edward's crown have been removed or replaced

Have realised that this ended up being duplicated. Closing this in favour of the newer issue as it contains more details. #4623


fix(obstacle_avoidance_planner): solution of changing the sampling distance of behavior velocity planner causing obstacle avoidance planner to die

Description
When changing the sampling length in the"behavior_velocity_planner" node, the "obstacle_avoidance_planner"node will die.
The length in the "obstacle_avoidance_planner" node is 5 (fixed). When it is changed to a value greater than 5 in the "behavior_velocity_planner"node, the "obstacle_avoidance_planner" node will die when the vehicle reaches the destination.

Related links
https://github.com/autowarefoundation/autoware.universe/issues/798

Tests performed
Current simulation:

Modified simulation:

Notes for reviewers

Pre-review checklist for the PR author
The PR author must check the checkboxes below when creating the PR.

[x] I've confirmed the contribution guidelines.
[x] The PR follows the pull request guidelines.

In-review checklist for the PR reviewers
The PR reviewers must check the checkboxes below before approval.

[ ] The PR follows the pull request guidelines.
[ ] The PR has been properly tested.
[ ] The PR has been reviewed by the code owners.

Post-review checklist for the PR author
The PR author must check the checkboxes below before merging.

[ ] There are no open discussions or they are tracked via tickets.
[ ] The PR is ready for merge.

After all checkboxes are checked, anyone who has write access can merge the PR.

@kosuke55

@shulanbushangshu
Thank you for your PR. Your branch looks this PR is not included https://github.com/autowarefoundation/autoware.universe/pull/1320
can you include this PR and test this again?

@taikitanaka3 .Thank you for your review,I  will confirm the code and retest

@taikitanaka3 ,I have confirmed the code and retest:

Modified simulation:

@shulanbushangshu
Thank you. Modified simulation will not die at obstacle avoidance planner but dies at behavior velocity planner?

@shulanbushangshu
Could you please re-test including my fix PR
with / without your fix PR with sampling interval greater than 5.0 ?
Is this comment doing above?

@shulanbushangshu I've tested with this PR as you suggests ego can't goal
without this PR

with this PR

Do you have any good suggestion to arrive goal?

Thank you for your discussion.
I'll look into the problem to realize both reaching the goal and the node keeping alive. Please wait a bit.

@taikitanaka3 @takayuki5168
Thank you for your review.I fix the problem of not reaching the goal.Can you check it?

@shulanbushangshu
Thank you , but build is failed because of writing const value.
@takayuki5168
FYI  with this change I confirm this problem is fixed

Yes,I forget to change the "const".I fix it.

@shulanbushangshu
If i am not wrong this bug is only reproduces with setting behavior velocity planner sampling interval more than  5m,
and that setting is not suitable for autonomous driving. If this problem happens for normal driving case with sampling interval 1.0m this bug is high priority, if not I think we can set this PR low priority. what do you think?

@taikitanaka3 OK,we can set this PR low priority

@shulanbushangshu @xmfcx @Sharrrrk cc @mitsudome-r
Sorry to be late. Your change can solve the issue, but I guess there may be a better code. Let me implement it, and I'll let you know in a few days.

@shulanbushangshu @taikitanaka3
I created another PR to make commit history clear. Could you review this.
https://github.com/autowarefoundation/autoware.universe/pull/1616

@shulanbushangshu @xmfcx @Sharrrrk
We can close this PR since I merged another PR for the issue. Let me know if there is another bug.
https://github.com/autowarefoundation/autoware.universe/pull/1616

@takayuki5168
Yes,I have simulated the new code and it works well.I will close this PR.